Abnormal volume detection method, electronic equipment, readable storage medium and program product

Abstract

The invention provides an abnormal volume detection method, electronic equipment, a readable storage medium and a program product. The method comprises the following steps: acquiring left channel data and right channel data of first audio data; performing sound channel alignment processing on the left sound channel data and the right sound channel data to obtain sound channel aligned left and right sound channel data; performing mid-mixing processing on the left and right sound channel data with the aligned sound channels to obtain mid-audio data; performing time-frequency transformation processing on the mid-set audio data to obtain time-frequency joint distribution data; performing target peak searching on the time-frequency joint distribution data based on the target frequency interval to obtain one or more target frequencies and time point arrays corresponding to the target frequencies; performing audio interception on the first audio data based on each time point in the time point array to obtain an audio clip corresponding to each time point; performing abnormal volume detection on the audio clip corresponding to each time point to obtain a detection result of whether the first audio data is an abnormal volume audio; the detection efficiency is improved.

Description

Technical Field

[0001] This application relates to video technology, and more particularly to an abnormal volume detection method, electronic device, readable storage medium, and program product. Background Technology

[0002] In the field of audio and video processing, audio volume detection is a crucial step in ensuring content quality. With the rapid growth and diversification of video content, efficiently and accurately identifying video resources with abnormal audio volume has become a key issue in video platform operations. Insufficiently low audio volume not only affects user experience but may also prevent content from being properly disseminated or cause it to be ignored by users.

[0003] In existing technologies, a common approach is to process audio data frame by frame and combine it with silence detection to identify abnormal volume. This method typically includes steps such as filtering, framing, and mapping the audio to mark potentially problematic audio segments. Another mainstream approach uses deep learning models to train and analyze audio features to identify whether the volume of a human voice meets the standard. While these methods improve detection accuracy to some extent, they generally rely on frame-by-frame analysis, resulting in low overall processing efficiency. Summary of the Invention

[0004] This application provides an abnormal volume detection method, electronic device, readable storage medium, and program product, which can significantly improve detection efficiency and reduce computing resource consumption.

[0005] The technical solution of this application embodiment is implemented as follows: This application provides an abnormal volume detection method, the method comprising: Obtain the left and right channel data of the first audio data; The left channel data and the right channel data are processed to perform channel alignment to obtain channel-aligned left and right channel data; The left and right channel data that are aligned with the audio channels are mixed with a center channel to obtain center channel audio data. The center audio data is subjected to time-frequency transformation processing to obtain time-frequency joint distribution data; Based on the target frequency range, a target peak search is performed on the time-frequency joint distribution data to obtain one or more target frequencies and an array of time points corresponding to the target frequencies. The array of time points includes the time points corresponding to each target frequency. Based on each time point in the time point array, the first audio data is extracted to obtain the audio segment corresponding to each time point; Abnormal volume detection is performed on the audio segment corresponding to each time point to obtain the detection result, which indicates whether the first audio data is abnormal volume audio.

[0006] This application provides an electronic device, the electronic device comprising: Memory is used to store executable instructions for a computer; When the processor executes computer-executable instructions stored in the memory, it implements the abnormal volume detection method provided in the embodiments of this application.

[0007] This application provides a computer-readable storage medium storing a computer program or computer-executable instructions, which, when executed by a processor, implements the abnormal volume detection method provided in this application.

[0008] This application provides a computer program product, including a computer program or computer executable instructions. When the computer program or computer executable instructions are executed by a processor, they implement the abnormal volume detection method provided in this application.

[0009] In the technical solution of this application embodiment, the left and right channel data of the first audio data are acquired; the left and right channel data are aligned to obtain channel-aligned left and right channel data; the channel-aligned left and right channel data are mixed with a center channel to obtain center audio data; the center audio data is transformed by time and frequency to obtain time-frequency joint distribution data; a target peak search is performed on the time-frequency joint distribution data based on the target frequency range to obtain one or more target frequencies and an array of time points corresponding to the target frequencies, the time point values ​​including the time point corresponding to each target frequency; audio is extracted from the first audio data based on each time point in the time point array to obtain an audio segment corresponding to each time point; abnormal volume detection is performed on the audio segment corresponding to each time point to obtain a detection result, the detection result indicating whether the first audio data is abnormal volume audio. Thus, by aligning and centering the left and right channel data, human voice interference in the left and right channels can be effectively eliminated, thereby improving the accuracy of subsequent feature extraction. By performing time-frequency transformation on the center audio and searching for peaks within a specific frequency range, key time points that may affect volume judgment can be located. Combined with decibel parameters for detection, abnormal volume identification can be achieved quickly and efficiently without traversing all audio frames. Compared with the frame-by-frame analysis method in existing technologies, this significantly improves detection efficiency and reduces computational resource consumption. Attached Figure Description

[0010] Figure 1 This is a schematic diagram of the abnormal volume detection system architecture provided in the embodiments of this application; Figure 2 This is a schematic diagram of the structure of the device provided in the embodiments of this application; Figure 3This is a schematic diagram illustrating the principle of the abnormal volume detection method provided in the embodiments of this application; Figure 4 This is a first flowchart illustrating the abnormal volume detection method provided in this application embodiment; Figure 5 This is a second flowchart illustrating the abnormal volume detection method provided in this application embodiment; Figure 6 This is a schematic diagram of the third process of the abnormal volume detection method provided in the embodiments of this application; Figure 7 This is a schematic diagram of the fourth process of the abnormal volume detection method provided in the embodiments of this application; Figure 8 This is a schematic diagram of the fifth process of the abnormal volume detection method provided in the embodiments of this application.

[0011] It should be noted that the terms "first" and "second" mentioned above are only used to distinguish between different options and do not represent the degree of superiority or inferiority of the options or their priority in the implementation process. Detailed Implementation

[0012] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0013] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0014] In the following description, the terms "first, second, third" are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first, second, third" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0015] In the embodiments of this application, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0016] Unless otherwise defined, all technical and scientific terms used in the embodiments of this application have the same meaning as commonly understood by one of ordinary skill in the art. The terminology used in the embodiments of this application is for the purpose of describing the embodiments of this application only and is not intended to limit this application.

[0017] In the implementation of this application, the collection and processing of relevant data should strictly comply with the requirements of relevant national laws and regulations, obtain the informed consent or separate consent of the personal information subject, and carry out subsequent data use and processing within the scope of laws and regulations and the authorization of the personal information subject.

[0018] Before providing a further detailed description of the embodiments of this application, the nouns and terms involved in the embodiments of this application will be explained, and the nouns and terms involved in the embodiments of this application shall be interpreted as follows.

[0019] 1) Left and right channel data: This refers to the data of the audio signals corresponding to the left and right sound channels of stereo sound. In two-channel (stereo) audio, there are usually two independent audio channels, called the left channel and the right channel. These two channels may have time offsets or amplitude differences due to factors such as recording equipment and transmission delays.

[0020] 2) Channel alignment processing: This refers to the process of eliminating the time difference between the left and right channels through algorithmic means, so that the waveforms of the two channels are synchronized on the time axis.

[0021] 3) Center channel mixing: This technique involves adjusting the phase of the left and right channels and then superimposing them to extract the sound components located in the center position (i.e., not separated from the left and right channels). This method can enhance background music features and remove interference information such as vocals, thereby more accurately analyzing the overall volume performance of the audio.

[0022] 4) Time-frequency transformation processing: This is the process of converting audio signals from the time domain to the time-frequency joint distribution domain. For example, using methods such as short-time Fourier transform (STFT) to analyze the energy distribution of audio at different frequencies.

[0023] Low audio volume in videos is one of the most significant and common problems faced by major video companies. Recording, encoding, transmission, and noise can all affect audio volume. Low audio volume results in a poor user experience during playback on the client side, significantly testing users' tolerance. Therefore, before releasing video resources online, video companies need to check the audio quality of video resources in their media asset library and increase the volume of videos with low audio before releasing them. Existing technology processes the audio filtering data into two sets of audio data by performing two separate frame-by-frame processing steps. One set of audio data is used for silence detection, and the detection result is marked and mapped to the other set of audio data to obtain audio mapping data; this method can detect video sources with abnormal audio volume. Another method uses deep learning or large models to train and analyze the audio data in the video to find the corresponding decibel level of human voices. While this is also based on frame-by-frame analysis, although the detection accuracy is improved, the time efficiency is low. Moreover, for silent videos or videos with few human voices and low average decibel levels, a large amount of training data is required in the early stages, increasing the difficulty of promotion.

[0024] Existing methods for detecting abnormal volume have significant limitations when dealing with large-scale video libraries. Frame-by-frame analysis significantly increases processing time, especially for long videos, where detection efficiency falls short of practical requirements. Furthermore, deep learning methods rely on large amounts of training data, have limited effectiveness with videos containing little or no human voice, and increase deployment and maintenance costs.

[0025] To address the aforementioned issues, embodiments of this application provide an abnormal volume detection method, electronic device, computer program product, and computer-readable storage medium, which can improve the efficiency and accuracy of abnormal volume detection.

[0026] The following describes exemplary applications of the devices provided in the embodiments of this application. The electronic devices provided in the embodiments of this application can be implemented as various types of terminals such as laptops, tablets, desktop computers, set-top boxes, smartphones, smart speakers, smartwatches, smart TVs, and vehicle terminals, or they can be implemented as servers.

[0027] See Figure 1 , Figure 1 This is a schematic diagram of the abnormal volume detection system architecture provided in an embodiment of this application, for example. Figure 1 The system involves server 100, terminal device 200, and network 300. Terminal device 200 is connected to server 100 through network 300, which can be a wide area network (WAN), a local area network (LAN), or a combination of both.

[0028] In some embodiments, the present application embodiments can be implemented collaboratively by a server and a terminal device. For example, the terminal device 200 sends first audio data to the server 100 via the network 300. The server 100 uses the abnormal volume detection method provided in the present application embodiments to perform abnormal volume detection on the first audio data, obtains the detection result, and sends the detection result to the terminal device 200 via the network 300. The terminal device 200 determines whether the first audio data is abnormal volume audio based on the detection result, and improves the audio and video when abnormal volume is present.

[0029] In other embodiments, the embodiments of this application can be implemented independently by a terminal device. Terminal device 200 sends an abnormal volume detection request to a server. Server 100 receives the abnormal volume detection request and sends first audio data for abnormal volume detection provided in the embodiments of this application to terminal device 200. Terminal device 200 receives the first audio data sent by the server and obtains audio segments at suspected distortion locations in the first audio data. Abnormal volume detection is performed on the audio segments. Terminal device 200 determines whether abnormal volume exists in the first audio data by judging the audio segments. When abnormal volume exists, the audio and video are improved.

[0030] In some embodiments, the terminal device or server can implement the abnormal volume detection method provided in this application by running various computer-executable instructions or computer programs. For example, computer-executable instructions can be microprogram-level commands, machine instructions, or software instructions. Computer programs can be native programs or software modules in an operating system. In summary, the aforementioned computer-executable instructions can be any form of instruction, and the aforementioned computer programs can be any form of application program, module, or plug-in. Terminal devices include, but are not limited to, mobile phones, computers, smart voice interaction devices, smart home appliances, vehicle terminals, and aircraft.

[0031] See Figure 2 , Figure 2 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Figure 2 The electronic device 400 shown can be either the server 100 or the terminal device 200 mentioned above. Figure 2 The illustrated electronic device 400 includes at least one processor 410, a memory 430, and at least one network interface 420. The various components of the electronic device 400 are coupled together via a bus system 440. It is understood that the bus system 440 is used to implement communication between these components. In addition to a data bus, the bus system 440 also includes a power bus, a control bus, and a status signal bus. However, for clarity, ... Figure 2 The general designated all buses as Bus System 440.

[0032] Processor 410 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0033] The memory 430 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard disk drives, optical disk drives, etc. The memory 430 may optionally include one or more storage devices physically located away from the processor 410.

[0034] The memory 430 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), and the volatile memory may be random access memory (RAM). The memory 430 described in this application embodiment is intended to include any suitable type of memory.

[0035] In some embodiments, memory 430 is capable of storing data to support various operations, examples of which include programs, modules, and data structures or subsets or supersets thereof, as illustrated below.

[0036] Operating system 431 includes system programs for handling various basic system services and performing hardware-related tasks, such as the framework layer, core library layer, driver layer, etc., for implementing various basic business functions and handling hardware-based tasks; The network communication module 432 is used to reach other electronic devices via one or more (wired or wireless) network interfaces 420, exemplary network interfaces 420 including: Bluetooth, WiFi, and Universal Serial Bus (USB), etc. In some embodiments, the apparatus provided in this application can be implemented in software. Figure 2 An abnormal volume detection device 433 stored in memory 430 is shown. This device can be software in the form of programs and plugins, and includes the following software modules: a data acquisition module 4331 and a data processing module 4332. These modules are logically linked and can therefore be arbitrarily combined or further separated according to their implemented functions. The functions of each module will be described below.

[0037] In other embodiments, the apparatus provided in this application can be implemented in hardware. For example, the apparatus provided in this application can be a processor in the form of a hardware decoding processor, which is programmed to execute the image anomaly detection method for virtual scenes provided in this application. For example, the processor in the form of a hardware decoding processor can be one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic components.

[0038] See Figure 3 , Figure 3 This is a schematic diagram illustrating the principle of the abnormal volume detection method provided in this application embodiment. First, the left and right channel audio data of the audio / video to be detected for abnormal volume are acquired. Peak alignment is performed on the left and right channel audio / video data. Then, the left and right channels are phase-centered. The positions of the centered audio notes are detected, and an array of time points indicating suspected volume distortion is output. The audio volume within the suspected distortion location range is detected to obtain a detection result indicating either normal or abnormal volume audio. See also... Figure 4 , Figure 4 This is a first flowchart of the abnormal volume detection method provided in the embodiments of this application, which will be described in detail below.

[0039] Step 401: Obtain the left and right channel data of the first audio data.

[0040] In some embodiments, the first audio data can be an audio stream extracted from a video for abnormal volume detection, containing two independent audio channels, left and right. For example, left and right channel data acquisition of the first audio data can be implemented using the Python libraries SciPy, LibreSta, and NumPy. The left and right channel data correspond to the left and right sound channels in the stereo signal, respectively. Due to factors such as recording environment and transmission delay, there may be time offsets or amplitude differences between the left and right channels, thus requiring further processing to ensure the consistency of the audio information.

[0041] In some embodiments, the first audio data may be video data suspected of having abnormal audio volume after preliminary screening. Specifically, before performing anomaly detection on the first audio data, the audio data to be detected, as well as the maximum and average decibel levels of the audio and video data to be detected, can be obtained first. By establishing a preliminary screening mechanism and determining whether the audio data is suspected of having abnormal volume based on a preset threshold, most audio data with normal volume can be quickly filtered out, thereby significantly improving the efficiency of volume anomaly detection, reducing system load, and enabling efficient volume detection for large-scale video libraries. See [link to relevant documentation] for details. Figure 5 In execution Figure 4 Before step 401 shown, steps 501 to 502 are also performed, which will be explained in detail below.

[0042] Step 501: Obtain the audio data to be detected and the first decibel parameter and the second decibel parameter of the audio data to be detected; the first decibel parameter is the maximum decibel of the audio data to be detected, and the second decibel parameter is the average decibel of the audio data to be detected.

[0043] Here, the audio data can be sound information contained in a video file, typically stored in digital form and extracted using audio processing tools such as FFmpeg. In this application, the audio data is used to determine whether the volume of the audio data is within the normal range, thereby deciding whether further detection of volume anomalies is needed. The audio data can originate from the audio track of a video file or from independently recorded audio files.

[0044] The first decibel parameter refers to the maximum decibel value (db_max) in the audio data, which is the highest loudness found in the audio data. The first decibel parameter reflects the most intense part of the audio data and can provide a preliminary indication of whether there are extremely low volumes or no sound at all. The second decibel parameter refers to the average decibel value (db_avg) of the audio data, which is the average loudness of all sample points in the entire audio. The second decibel parameter reflects the average volume level of the overall audio data and helps determine whether the overall audio data is too low.

[0045] Step 502: When the first decibel parameter is greater than the first preset threshold and the second decibel parameter is less than the second preset threshold, the audio data to be detected is determined to be the first audio data, and the first audio data is the audio data to be detected for abnormal volume.

[0046] Here, the first and second preset thresholds are numerical standards set according to business requirements. For example, the first preset threshold can be set to -5dB, and the second preset threshold can be set to -20dB. These two thresholds correspond to the boundary conditions for the maximum and average volume of the audio data, respectively. If the maximum decibel of an audio data point is higher than the first preset threshold (e.g., -5dB), it indicates that the audio data has a strong instantaneous volume; while if the average decibel of the audio data is lower than the second preset threshold (e.g., -20dB), it indicates that the overall volume of the audio data is too low. The system can then determine the possibility of abnormal audio volume.

[0047] The method of judging based on the combination of the first and second decibel parameters is more robust than using the maximum decibel or average decibel alone. For example, some audio data may have high-volume segments in certain areas, but the overall volume is still low. In this case, using the maximum decibel alone may not be accurate, but combining it with the average decibel can more effectively identify audio data with abnormal volume.

[0048] For example, ffmpeg can be used to obtain the maximum and average decibel levels of the audio data to be detected. When the maximum decibel level (db_max) is higher than a first preset threshold (e.g., max_volume > -5dB, obtained via ffmpeg), and the average decibel level (db_avg) is lower than a second preset threshold (e.g., mean_volume < -20dB, obtained via ffmpeg), the current video is judged to have abnormal audio volume. The preset thresholds, such as -5dB and -20dB, are business data and can be set according to different user tolerance levels. Specific data can be dynamically adjusted based on business needs. By obtaining the maximum and average decibel levels, the overall volume of the audio data can be more comprehensively evaluated, avoiding misjudgments caused by relying on a single indicator. For example, an audio dataset may contain brief high-volume segments, but the overall average volume of the audio data may still be low. Even if the maximum decibel level exceeds the first preset threshold and the average decibel level is lower than the second preset threshold, this situation is still judged as abnormal volume.

[0049] Step 402: Perform channel alignment processing on the left and right channel data to obtain channel-aligned left and right channel data.

[0050] In some embodiments, the purpose of channel alignment is to eliminate timing differences between the left and right channels, synchronizing them on the time axis. For example, by calculating the correlation between the left and right channels at different offsets, the maximum correlation is identified. The offset corresponding to the highest correlation value is the peak value for channel alignment. The left channel is offset by the current value to align with the peak value of the right channel, thereby achieving synchronization with the right channel. Channel alignment improves the accuracy of subsequent center channel mixing.

[0051] In some embodiments, see Figure 6 , Figure 4 Step 402 shown can be implemented through steps 601 to 603, as explained in detail below.

[0052] Step 601: Set the offset sample interval, calculate the correlation coefficient between the left channel data and the right channel data based on the offset sample interval, and obtain the correlation coefficient set.

[0053] Here, the offset sample interval refers to the sample range used to calculate the time difference between the left and right channel data. For example, in this embodiment, the system sets the offset sample interval to [-20, 20], indicating that the maximum allowed number of offset samples is 20 sampling points. By setting the offset sample interval, the system can evaluate the similarity between the left and right channels at different time offsets. The correlation coefficient is a numerical value that measures the degree of matching between two signals at a given offset. For example, by choosing to use cosine similarity as the calculation method, a set of correlation coefficients can be generated for each offset, and this set can be used for subsequent operations to determine the best alignment point. Setting the offset sample interval and calculating the set of correlation coefficients can quickly find the best alignment position of the left and right channels, thereby improving audio processing efficiency.

[0054] For example, if the offset sample value of the audio frequency is set to 20, then the offset sample interval is [-20, 20]. Find the maximum correlation between the left and right channels within the offset sample interval. The cosine similarity can be calculated using formula (1) to obtain the correlation between the left and right channels.

[0055] (1) Here, A represents the left channel data sample. The left channel is selected based on the offset sample interval [-20, 20], where the highest correlation is found. The offset corresponding to the highest correlation value is the aligned peak value. The left channel is offset by the current value to align with the peak value of the right channel. The current 20 sample offset values ​​are the optimal offset samples in business testing, and this application does not impose specific limitations on this.

[0056] Step 602: Determine the maximum correlation coefficient based on the set of correlation coefficients, where the offset corresponding to the maximum correlation coefficient is the channel alignment peak.

[0057] Here, the maximum correlation coefficient refers to the similarity value of the left and right channel data that best matches among all offsets. The offset corresponding to the maximum correlation coefficient is the channel alignment peak, indicating how many samples should be moved for the left and right channels to achieve optimal alignment. Determining the channel alignment peak can effectively identify the time difference between the left and right channels, thus providing a basis for subsequent channel alignment operations. The method of calculating the correlation coefficient and determining the channel alignment peak based on the offset sample interval avoids the inefficiency of traditional frame-by-frame comparison, improving processing speed and accuracy.

[0058] Step 603: Perform channel alignment processing on the left channel data and right channel data based on the channel alignment peak to obtain the channel-aligned left channel data and the channel-aligned right channel data.

[0059] Here, channel alignment processing refers to adjusting the positions of the left and right channel data based on the offset corresponding to the channel alignment peak, aligning the left and right channel data in time. Specifically, when the left channel is detected to lag behind the right channel, the left channel is moved forward by the corresponding number of samples; when the right channel is detected to lag behind the left channel, the system moves the right channel backward by the corresponding number of samples. After channel alignment processing is completed based on the channel alignment peak, the left and right channel data are kept consistent on the time axis. The result of channel alignment processing facilitates subsequent center channel processing and frequency analysis, ensuring the consistency and integrity of audio information.

[0060] In this embodiment, by setting an offset sample interval and calculating a set of correlation coefficients, the optimal alignment position of the left and right channels can be quickly determined, thereby improving the efficiency and accuracy of audio processing. The method of calculating correlation coefficients based on offset sample intervals and determining channel alignment peaks avoids the high computational overhead of traditional frame-by-frame comparison, enabling the system to perform volume anomaly detection more efficiently and further improving the overall effect of video and audio quality control.

[0061] Step 403: Perform center channel mixing processing on the left and right channel data that are aligned with the channel to obtain center channel audio data.

[0062] In some embodiments, center-channel mixing refers to the process of inverting the phases of the aligned left and right channels and then superimposing them to extract the sound component located in the center of the audio. Center-channel mixing effectively removes interference information such as vocals, enhances background music characteristics, and makes the overall volume of the audio clearer. Furthermore, center-channel mixing can reduce redundant information that may exist in the left and right channels, improving the efficiency of audio analysis.

[0063] In some embodiments, see Figure 7 , Figure 4Step 403 shown can be implemented through steps 701 to 702, as explained in detail below.

[0064] Step 701: Perform phase inversion processing on the right channel data after channel alignment to obtain the phase-inverted right channel data.

[0065] Here, phase inversion processing refers to flipping the waveform of the audio signal by 180 degrees on the time axis, that is, the peak value of the sine wave becomes negative, and the negative value becomes positive. Phase inversion processing is often used to eliminate asymmetric interference or vocal components in stereo, thereby preserving background music or other common sound components. In this application, by performing phase inversion processing on the right channel audio data, the right channel audio data and the left channel audio can achieve a center-channel effect when superimposed. The system concentrates the sound originally distributed in the left and right channels to the center position to extract common audio information.

[0066] In practical applications, such as in video volume anomaly detection, phase inversion processing of the right channel followed by superposition with the left channel can remove some human voice components, thereby more accurately extracting key frequency components such as background music or ambient sounds. This phase inversion processing of the right channel followed by superposition with the left channel improves the accuracy of subsequent frequency analysis and avoids misjudgments caused by decibel fluctuations in human voices.

[0067] Step 702: Perform center channel mixing processing on the phase-inverted right channel data and the channel-aligned left channel data to obtain center channel audio data.

[0068] Here, center channel mixing is a process that merges left and right channel audio into a single center channel audio using a specific algorithm. The core purpose of center channel mixing is to extract common audio information from both channels, thereby removing differences between the left and right channels (such as vocals). In this application, by performing a phase inversion on the right channel data and then superimposing it with the left channel data, the common components of both can be effectively extracted to form center channel audio data. This center channel audio data can be used for further frequency analysis to identify any abnormal volume levels.

[0069] For example, center-channel mixing can significantly improve the efficiency and accuracy of audio analysis. For instance, during video playback, if there are few human voices but a lot of background music, center-channel mixing can more clearly extract the frequency characteristics of the background music. The system can then quickly determine whether the audio is within the normal volume range using these characteristics. First, phase inversion processing is used to make the right channel data and left channel data more consistent in phase. Then, center-channel mixing processing is used to superimpose the two, extracting common components and removing differing components. Performing phase inversion processing on the right channel data and then center-mixing it with the left channel data not only improves the signal-to-noise ratio of the audio processing but also makes subsequent frequency analysis more accurate. This center-channel mixing processing enhances the overall performance of volume anomaly detection.

[0070] In summary, in this embodiment, by performing phase inversion processing on the right channel data and then performing center-mixing processing on the resulting audio signal with the left channel data, the common audio components of the left and right channels can be effectively extracted, thereby removing interference and improving the accuracy of audio analysis. The center-mixing process reduces unnecessary computational resource consumption and improves the efficiency of the volume anomaly detection task. This task can quickly filter out video content with abnormal volume, thus optimizing the quality review process before video upload.

[0071] Step 404: Perform time-frequency transformation on the center audio data to obtain time-frequency joint distribution data.

[0072] In some embodiments, time-frequency transformation processing is a method for converting audio signals from the time domain to the time-frequency joint distribution domain. For example, time-frequency transformation processing is achieved through short-time Fourier transform (STFT). For instance, the center audio data is subjected to short-time Fourier transform using the librosa library to obtain time-frequency joint distribution data, as detailed in formula (2).

[0073] (2) Here, M1 represents the center audio data, and F1 represents the frequency data, i.e., the combined time-frequency distribution data. This allows observation of the frequency energy distribution of the audio at different time points, providing basic data for subsequent target peak search. The time-frequency transformation process helps identify frequency points with significant energy in the audio, thereby determining the possibility of abnormal volume.

[0074] Step 405: Perform target peak search on the time-frequency joint distribution data based on the target frequency range to obtain one or more target frequencies and the corresponding time point array. The time point array includes the time point corresponding to each target frequency.

[0075] In some embodiments, the target frequency range is determined by a target frequency and a frequency tolerance. The target frequency can be a specific frequency set according to the audio content. For example, the target frequency could be the frequency corresponding to the note "FA." The position of the note "FA" in background music typically occurs at the climax of the melody and in transitional sections, adding color, depth, and emotional expression to the video; therefore, the corresponding audio volume will be louder than other frequency bands. Frequency tolerance refers to the range of acceptable fluctuations in the target frequency, used to avoid misjudgments caused by noise interference or measurement errors.

[0076] In some embodiments, by performing a target peak T search on the time-frequency joint distribution data, the frequency points of target notes appearing within the target frequency range can be identified, and the time points corresponding to the frequency points of the target notes can be recorded. Target peak search helps to locate potentially abnormal volume areas in the audio, providing a basis for subsequent audio extraction and detection. For example, see... Figure 8 , Figure 4 Step 405 shown can be implemented through steps 801 to 802, as explained in detail below.

[0077] Step 801: Search for target peaks that match the target frequency range in the time-frequency joint distribution data to obtain one or more target frequencies.

[0078] Here, the target frequency can be set to the frequency corresponding to the note "FA," with a default frequency of 349.23 Hz. The selection of the target frequency is based on common background music characteristics found in the video content, and this selection method effectively reflects the key frequency bands of audio loudness. Frequency tolerance refers to the allowable range of fluctuation in the target frequency, used to avoid misjudgments caused by noise interference or measurement errors. Frequency tolerance is usually expressed in positive or negative values, such as ±20 Hz. In practical applications, the tolerance is dynamically adjusted based on factors such as audio quality and signal-to-noise ratio to improve detection accuracy and robustness.

[0079] The target frequency range is a frequency range jointly determined by the target frequency and the frequency tolerance, used to limit the area for peak search. For example, if the target frequency is 349.23Hz and the frequency tolerance is ±20Hz, then the target frequency range is [329.23Hz, 369.23Hz]. By setting a reasonable frequency range, the most representative frequency bands in the audio can be focused on, thereby improving the efficiency and accuracy of abnormal volume detection. For example, the numpy.where() function can be used to search within the F1 frequency range of the current "FA" frequency range [329.23Hz, 369.23Hz]. In this way, by precisely controlling the search range, reducing unnecessary calculations, and improving processing speed, key frequency information can be quickly located, further improving the efficiency of judging whether the audio is abnormal.

[0080] Step 802: Record the frequency index corresponding to each target frequency, and determine the time point corresponding to each target frequency based on the frequency index.

[0081] Here, the frequency index F_index refers to the position number of each frequency value on the frequency axis in the time-frequency joint distribution data. Using the frequency index, the system can quickly locate the specific position of the target frequency without traversing the entire frequency spectrum. The time point t refers to the specific moment when the target frequency appears in the audio signal, usually calculated based on the sampling rate sr. By converting the frequency index into a time point, the occurrence time of the target frequency in the audio is clearly defined, providing a basis for subsequent time window selection and analysis.

[0082] Step 803: Obtain a time point array based on the time point corresponding to each target frequency.

[0083] Here, the time point array N refers to an ordered set of time points corresponding to all target frequencies, which can be represented as N=[t1,t2,…], used to store the occurrence times of all events in the audio that meet the conditions. By traversing the time point array, the decibel level of the audio segment can be further analyzed to determine whether the audio segment belongs to an abnormal volume.

[0084] The mapping relationship between frequency indices and time points is fundamental to achieving accurate time positioning. Only by ensuring the accuracy of the frequency index can the reliability of the time points be guaranteed, thus ensuring the effectiveness of subsequent analysis. By systematically organizing and utilizing the time information of target frequencies, more refined audio segment analysis can be supported. Audio processing systems can achieve efficient volume anomaly detection, enabling rapid detection of audio volume abnormalities and thus widespread application in quality screening of large-scale video libraries.

[0085] Step 406: Extract audio segments from the first audio data based on each time point in the time point array to obtain the audio segments corresponding to each time point.

[0086] In some embodiments, the frequency extraction operation is performed around a time point corresponding to each target frequency, typically extracting audio data for a certain duration before and after the time point corresponding to the target frequency. For example, 3 seconds of audio can be extracted before and after each time point, for a total of 6 seconds of data. The purpose of the audio extraction operation is to obtain audio segments containing potentially abnormal volumes for more detailed analysis.

[0087] In some embodiments, audio segmentation of the first audio data is performed based on each time point in the time point array, including: for each time point in the time point array, extracting an audio segment of a preset time length before and after that time point in the first audio data, wherein the preset time length is determined based on business requirements.

[0088] Here, the preset time length refers to a fixed time period set according to specific application scenarios or business needs, used to extract audio segments within a specific time window from the original audio. For example, in a video volume anomaly detection scenario, the preset time length can be set to 3 seconds, meaning that 3 seconds of audio content is extracted before and after each time point in the time point array, for a total of 6 seconds of audio segments. The preset time length can be adjusted according to different types of video content (such as short videos and long videos) and actual processing efficiency requirements. A shorter preset time length is suitable for scenarios with high real-time requirements; a longer preset time length helps to more comprehensively evaluate volume change trends.

[0089] An audio segment is a continuous audio signal extracted from the original audio data, used for subsequent decibel calculations, silence detection, or other audio feature analyses. In this embodiment, the audio segment is determined by each time point in the time point array, ensuring that the data extracted from the audio segment can cover key audio events, thereby improving detection accuracy.

[0090] The time point array is a set of timestamps generated by the frequency peak search operation. The time points in the time point array correspond to the locations of significant sonic features that may exist in the audio, such as the climax of music or the location of human voices. These time points serve as reference points for audio extraction, allowing the extraction process to focus on potentially problematic areas and avoiding the waste of computational resources caused by indiscriminate analysis of the entire audio segment.

[0091] In this embodiment, by extracting audio segments of a preset duration from each time point in the time point array, key areas of abnormal volume in the audio can be efficiently located and extracted. This reduces unnecessary global audio analysis, thereby improving the efficiency and accuracy of audio processing and enabling rapid determination of whether a video has abnormal volume issues.

[0092] Step 407: Perform abnormal volume detection on the audio segment corresponding to each time point, and obtain the detection result. The detection result indicates whether the first audio data is abnormal volume audio.

[0093] In some embodiments, abnormal volume detection is based on decibel parameters (such as maximum and average decibels) of audio segments. If a segment contains an audio segment where the maximum decibel is higher than a preset threshold and the average decibel is lower than a predetermined threshold, the video exhibiting these characteristics is classified as an abnormal volume video; otherwise, the video is classified as a non-abnormal volume video. By performing abnormal volume detection, efficient and accurate volume anomaly judgment can be made in videos without relying on deep learning models.

[0094] In some embodiments, before performing step 407, the method further includes: obtaining a third decibel parameter and a fourth decibel parameter for each audio segment; the third decibel parameter is the maximum decibel level for each audio segment, and the fourth decibel parameter is the average decibel level for each audio segment.

[0095] Here, the third decibel parameter refers to the maximum decibel value (db_max) of the sound signal in the audio data segment, used to measure the loudest part of the sound in the audio data segment. The higher the maximum decibel value, the more obvious the high volume part in the audio data segment, which may represent background music or sudden loud voices. The fourth decibel parameter refers to the average decibel value (db_avg) of all sound signals in the audio data segment, reflecting the overall volume level of the entire audio data segment. The lower the average decibel value, the lower the overall volume, which may indicate weak voices or missing background sounds. These two parameters can quickly determine whether there are localized excessively loud or overall weak audio segments, thus serving as a basis for detecting videos with abnormal volume. By collecting these two parameters, it is not necessary to process the entire audio data frame by frame during audio detection, thereby significantly reducing computational complexity and improving detection efficiency.

[0096] In some embodiments, when the third decibel parameter of at least one audio segment is greater than or equal to the third preset threshold and the fourth decibel parameter is less than or equal to the fourth preset threshold, the first audio data is determined to be abnormal volume audio; when the third decibel parameter of each audio segment is less than the third preset threshold and the fourth decibel parameter is greater than the fourth preset threshold, the first audio data is determined to be non-abnormal volume audio.

[0097] Here, the judgment logic is set based on actual business experience. For example, the third preset threshold can be set to -5dB, and the fourth preset threshold can be set to -20dB. If the maximum decibel level of a certain audio data segment in the first audio data is higher than -5dB and the average decibel level is lower than -20dB, it indicates that the audio data segment has a phenomenon of excessively high local volume but low overall volume, which may be due to background noise or unclear voice recording during the recording process, and is judged as abnormal volume audio. Conversely, if the maximum decibel level of all audio data segments in the first audio data is lower than -5dB and the average decibel level is higher than -20dB, it indicates that the overall audio data volume of the first audio data is moderate and there is no obvious problem, and the first audio data is judged as non-abnormal volume audio. In this way, the high time cost of the traditional frame-by-frame scanning method is avoided. At the same time, by combining the judgment of the maximum decibel level and the average decibel level, the normal and abnormal volume situations can be more accurately distinguished, improving the detection accuracy and execution efficiency.

[0098] In this embodiment, by obtaining the maximum and average decibel levels of an audio data segment and performing a joint judgment, it is possible to efficiently identify whether there are local volume abnormalities or overall low volume issues in the audio data. This method of obtaining the maximum and average decibel levels of an audio data segment and performing a joint judgment reduces the need for frame-by-frame processing of audio data, thereby significantly reducing the time overhead of audio data processing and enabling rapid volume detection of a large number of video resources.

[0099] As described above, the abnormal volume detection method mentioned in this application identifies videos suspected of having abnormal volume by comparing the maximum decibel parameter of the audio in the video data with a preset threshold. Then, it acquires and aligns the left and right channel data of the suspected abnormal volume video, centers the right and left channels, performs a short-time Fourier transform on the centered audio to obtain frequency data F1, and performs peak search based on this frequency data F1 to obtain one or more notes with the frequency "FA". Finally, it determines whether the current video is a video with abnormal volume based on the threshold range of audio decibel values ​​within a preset duration corresponding to the "FA" notes. Thus, through a pre-screening process, using the maximum and average decibels as preliminary judgment criteria, obviously normal audio data can be quickly excluded, and only suspected abnormal audio is further processed, thereby significantly reducing invalid calculations and improving overall detection efficiency. By acquiring left and right channel data from the pre-screened first audio data and aligning and centering the left and right channel data, human voice interference in the left and right channels can be effectively eliminated, thereby improving the accuracy of subsequent feature extraction. By performing time-frequency transformation on the center audio and searching for peaks within a specific frequency range, key time points that may affect volume judgment can be located. Combined with decibel parameters for detection, this method can achieve fast and efficient abnormal volume identification without traversing all audio frames. Compared with the existing frame-by-frame analysis method, this method significantly improves detection efficiency and reduces computational resource consumption.

[0100] The following description continues to illustrate the exemplary structure of the abnormal volume detection device 433 provided in the embodiments of this application as a software module. In some embodiments, such as Figure 2 As shown, the software module stored in the abnormal volume detection device 433 in the memory 430 may include: The data acquisition module 4331 is used to acquire the left channel data and right channel data of the first audio data.

[0101] The data processing module 4332 is used to perform channel alignment processing on the left channel data and the right channel data to obtain channel-aligned left and right channel data.

[0102] In some embodiments, the data processing module 4332 is further configured to perform center-channel mixing processing on the channel-aligned left and right channel data to obtain center-channel audio data.

[0103] In some embodiments, the data processing module 4332 is further configured to perform time-frequency transformation processing on the center audio data to obtain time-frequency joint distribution data.

[0104] In some embodiments, the data processing module 4332 is further configured to perform target peak search on the time-frequency joint distribution data based on the target frequency range to obtain one or more target frequencies and an array of time points corresponding to the target frequencies, wherein the array of time points includes the time points corresponding to each target frequency.

[0105] In some embodiments, the data processing module 4332 is further configured to perform audio segmentation on the first audio data based on each time point in the time point array to obtain an audio segment corresponding to each time point.

[0106] In some embodiments, the data processing module 4332 is further configured to perform abnormal volume detection on the audio segment corresponding to each time point and obtain a detection result, wherein the detection result indicates whether the first audio data is abnormal volume audio.

[0107] In some embodiments, the data processing module 4332 is further configured to set an offset sample interval, calculate the correlation coefficient between the left channel data and the right channel data based on the offset sample interval, and obtain a set of correlation coefficients; determine the maximum correlation coefficient based on the set of correlation coefficients, wherein the offset corresponding to the maximum correlation coefficient is the channel alignment peak value; and perform channel alignment processing on the left channel data and the right channel data based on the channel alignment peak value to obtain channel-aligned left channel data and channel-aligned right channel data.

[0108] In some embodiments, the data processing module 4332 is further configured to perform phase inversion processing on the right channel data after channel alignment to obtain phase-inverted right channel data; and to perform center-channel mixing processing on the phase-inverted right channel data and the left channel data after channel alignment to obtain the center-channel audio data.

[0109] In some embodiments, the target frequency range is determined by the target frequency and frequency tolerance; the data processing module 4332 is further configured to search for target peaks matching the target frequency range in the time-frequency joint distribution data to obtain one or more target frequencies; record the frequency index corresponding to each target frequency, determine the time point corresponding to each target frequency based on the frequency index; and obtain the time point array based on the time point corresponding to each target frequency.

[0110] In some embodiments, the data processing module 4332 is further configured to extract an audio segment of a preset time length before and after each time point in the time point array, wherein the preset time length is determined based on business requirements.

[0111] In some embodiments, before the data acquisition module 4331 acquires the left channel data and right channel data of the first audio data, the data acquisition module 4331 is further configured to acquire the audio data to be detected and a first decibel parameter and a second decibel parameter of the audio data to be detected; the first decibel parameter is the maximum decibel of the audio data to be detected, and the second decibel parameter is the average decibel of the audio data to be detected.

[0112] In some embodiments, the data processing module 4332 is further configured to determine the audio data to be detected as the first audio data when the first decibel parameter is greater than the first preset threshold and the second decibel parameter is less than the second preset threshold, wherein the first audio data is the audio data to be detected for abnormal volume.

[0113] In some embodiments, before the data processing module 4332 performs abnormal volume detection on the audio segment corresponding to each time point, the data acquisition module 4331 is further configured to acquire a third decibel parameter and a fourth decibel parameter for each audio segment; the third decibel parameter is the maximum decibel of each audio segment, and the fourth decibel parameter is the average decibel of each audio segment.

[0114] In some embodiments, the data processing module 4332 is further configured to determine that the first audio data is abnormal volume audio when the third decibel parameter of at least one audio segment is greater than or equal to the third preset threshold and the fourth decibel parameter is less than or equal to the fourth preset threshold; and to determine that the first audio data is non-abnormal volume audio when the third decibel parameter of each audio segment is less than the third preset threshold and the fourth decibel parameter is greater than the fourth preset threshold.

[0115] This application provides a computer program product, which includes a computer program or computer-executable instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer-executable instructions from the computer-readable storage medium and executes the computer-executable instructions, causing the electronic device to perform the abnormal volume detection method described above in this application.

[0116] This application provides a computer-readable storage medium storing computer-executable instructions or a computer program. When the computer-executable instructions or the computer program are executed by a processor, the processor will execute the image anomaly detection method for a virtual scene provided in this application. For example, ... Figure 4 The abnormal volume detection method is shown.

[0117] In some embodiments, the computer-readable storage medium may be a memory such as RAM, ROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or it may be a variety of devices including one or any combination of the above-mentioned memories.

[0118] In some embodiments, computer-executable instructions may take the form of programs, software, software modules, scripts, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as stand-alone programs or as modules, components, subroutines, or other units suitable for use in a computing environment.

[0119] As an example, computer-executable instructions may, but do not necessarily, correspond to files in a file system. They may be stored as part of a file that holds other programs or data, for example, in one or more scripts in a Hyper Text Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple co-located files (e.g., files that store one or more modules, subroutines, or code sections).

[0120] As an example, computer-executable instructions can be deployed to execute on a single electronic device, or on multiple electronic devices located at one location, or on multiple electronic devices distributed across multiple locations and interconnected via a communication network.

[0121] In summary, through the embodiments of this application, on the one hand, by using the maximum and average decibel levels as preliminary judgment criteria for pre-screening, obviously normal audio data can be quickly excluded, and only suspected abnormal audio is further processed, thereby significantly reducing invalid calculations and improving overall detection efficiency. This method significantly reduces system load while ensuring detection quality, making it suitable for batch detection scenarios of large-scale video content. On the other hand, by aligning and centering the left and right channel data, human voice interference in the left and right channels can be effectively eliminated, thereby improving the accuracy of subsequent feature extraction. By performing time-frequency transformation on the center audio and searching for peaks within a specific frequency range, key time points that may affect volume judgment can be located. Combined with decibel parameters for detection, rapid and efficient abnormal volume identification can be achieved without traversing all audio frames. Compared with the frame-by-frame analysis method in the prior art, this significantly improves detection efficiency and reduces computational resource consumption.

[0122] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and scope of this application are included within the scope of protection of this application.

Claims

1. A method for detecting abnormal volume, characterized in that, The method includes: Obtain the left and right channel data of the first audio data; The left channel data and the right channel data are processed to perform channel alignment to obtain channel-aligned left and right channel data; The left and right channel data that are aligned with the audio channels are mixed with a center channel to obtain center channel audio data. The center audio data is subjected to time-frequency transformation processing to obtain time-frequency joint distribution data; Based on the target frequency range, a target peak search is performed on the time-frequency joint distribution data to obtain one or more target frequencies and an array of time points corresponding to the target frequencies. The array of time points includes the time points corresponding to each target frequency. Based on each time point in the time point array, the first audio data is extracted to obtain the audio segment corresponding to each time point; Abnormal volume detection is performed on the audio segment corresponding to each time point to obtain the detection result, which indicates whether the first audio data is abnormal volume audio.

2. The method according to claim 1, characterized in that, The process of aligning the left channel data and the right channel data includes: Set an offset sample interval, and calculate the correlation coefficient between the left channel data and the right channel data based on the offset sample interval to obtain a set of correlation coefficients; The maximum correlation coefficient is determined based on the set of correlation coefficients, wherein the offset corresponding to the maximum correlation coefficient is the channel alignment peak value; Based on the channel alignment peak, the left channel data and the right channel data are aligned to obtain the channel-aligned left channel data and the channel-aligned right channel data.

3. The method according to claim 2, characterized in that, The process of performing center channel mixing on the left and right channel data to obtain center channel audio data includes: The right channel data after channel alignment is phase-inverted to obtain the phase-inverted right channel data. The right channel data after phase inversion and the left channel data after channel alignment are mixed in the center to obtain the center audio data.

4. The method according to claim 1, characterized in that, The target frequency range is determined by the target frequency and frequency tolerance; the target peak search based on the time-frequency joint distribution data based on the target frequency range includes: Search for target peak values ​​that match the target frequency range in the time-frequency joint distribution data to obtain one or more target frequencies; Record the frequency index corresponding to each target frequency, and determine the time point corresponding to each target frequency based on the frequency index; The time point array is obtained based on the time point corresponding to each target frequency.

5. The method according to claim 4, characterized in that, The step of extracting audio from the first audio data based on each time point in the time point array includes: For each time point in the time point array, an audio segment of a preset time length is extracted from the first audio data before and after that time point. The preset time length is determined based on business requirements.

6. The method according to any one of claims 1 to 5, characterized in that, Before acquiring the left and right channel data of the first audio data, the method further includes: Acquire the audio data to be detected and a first decibel parameter and a second decibel parameter of the audio data to be detected; the first decibel parameter is the maximum decibel of the audio data to be detected, and the second decibel parameter is the average decibel of the audio data to be detected; When the first decibel parameter is greater than the first preset threshold and the second decibel parameter is less than the second preset threshold, the audio data to be detected is determined to be the first audio data, and the first audio data is the audio data to be detected for abnormal volume.

7. The method according to any one of claims 1 to 5, characterized in that, Before performing abnormal volume detection on the audio segment corresponding to each time point, the method further includes: Obtain the third and fourth decibel parameters for each audio segment; the third decibel parameter is the maximum decibel level for each audio segment, and the fourth decibel parameter is the average decibel level for each audio segment. The abnormal volume detection for the audio segment corresponding to each time point includes: When there is at least one audio segment where the third decibel parameter is greater than or equal to the third preset threshold and the fourth decibel parameter is less than or equal to the fourth preset threshold, the first audio data is determined to be abnormal volume audio. When the third decibel parameter of each audio segment is less than the third preset threshold and the fourth decibel parameter is greater than the fourth preset threshold, the first audio data is determined to be non-abnormal volume audio.

8. An electronic device, characterized in that, include: The processor and the memory used to store computer programs that can run on the processor. When the processor is used to run the computer program, it performs the steps of the method according to any one of claims 1 to 7.

9. A storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images