GPU hardware decoding fault detection method and device, electronic equipment and storage medium

Abstract

The invention provides a GPU hardware decoding fault detection method and device, electronic equipment and a storage medium, and relates to the technical field of GPU testing. The method comprises the following steps: collecting an audio output signal of an audio / video under the condition that the audio / video adopts a hardware decoding function of a graphics processing unit (GPU); performing audio anomaly analysis on the frequency spectrum of the audio output signal to obtain an audio analysis result; determining video state data corresponding to a video frame in the audio and video, wherein the video state data represents video processing stability of the video frame in a GPU hardware decoding process; and based on the audio analysis result and the video state data, carrying out fault positioning on the GPU hardware decoding process, and determining a GPU hardware decoding fault point. According to the scheme, fault positioning can be accurately carried out on the GPU hardware decoding process, and then the GPU hardware decoding fault point can be accurately determined.

Description

Technical Field

[0001] This disclosure relates to the field of GPU testing technology, and more specifically, to a method, apparatus, electronic device, and storage medium for detecting GPU hardware decoding faults. Background Technology

[0002] With the rapid development of high-definition video, immersive media, and real-time streaming services, hardware decoding technology of graphics processing units (GPUs) has become a core supporting capability of modern multimedia playback systems, enabling efficient decoding of high-resolution, high-bitrate video streams through dedicated hardware units. As high-resolution, high-bitrate video content becomes more widespread, GPU hardware decoding has become a standard feature of player software.

[0003] However, during the GPU hardware decoding process, the stability issues of the GPU have become increasingly prominent, causing phenomena such as stuttering, screen tearing, audio-visual asynchrony, and popping sounds to occur during audio and video playback.

[0004] Traditional GPU hardware decoding fault detection methods mainly rely on monitoring performance counters exposed by the GPU driver layer (such as decoder utilization, frame processing time, temperature, etc.) and combining them with external indicators such as network throughput and system load for indirect inference. However, they cannot accurately locate faults in GPU hardware decoding.

[0005] It should be noted that the information disclosed in the background section above is only used to enhance the understanding of the background of this disclosure, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0006] The purpose of this disclosure is to provide a GPU hardware decoding fault detection method, apparatus, electronic device, and storage medium, which can accurately locate faults in the GPU hardware decoding process and thus accurately determine the GPU hardware decoding fault point.

[0007] Other features and advantages of this disclosure will become apparent from the following detailed description, or may be learned in part from practice of this disclosure.

[0008] According to a first aspect of the present disclosure, a GPU hardware decoding fault detection method is provided, the method comprising: When the audio and video use the hardware decoding function of the graphics processing unit (GPU), the audio output signal of the audio and video is acquired; Audio anomaly analysis is performed on the spectrum of the audio output signal to obtain audio analysis results; Determine the video state data corresponding to the video frame in the audio and video, wherein the video state data characterizes the video processing stability of the video frame during the GPU hardware decoding process; Based on the audio analysis results and the video status data, fault location is performed on the GPU hardware decoding process to determine the GPU hardware decoding fault point.

[0009] In some exemplary embodiments of this disclosure, based on the foregoing scheme, the step of performing audio anomaly analysis on the spectrum of the audio output signal to obtain audio analysis results includes: Based on the spectrum of the audio output signal, determine the target spectral energy of the spectrum; The target spectral energy is compared with the preset reference spectral energy to obtain the audio analysis result.

[0010] In some example embodiments of this disclosure, based on the foregoing scheme, the target spectral energy is characterized by a first spectral energy within the target frequency band; determining the target spectral energy of the spectrum based on the spectrum of the audio output signal includes: For at least one target frequency band within the spectral range of the audio output signal, obtain the first spectral energy within the target frequency band; The step of comparing the target spectral energy with a preset reference spectral energy to obtain the audio analysis result includes: The second spectral energy of the reference audio signal within the target frequency band is obtained, and the first spectral energy and the second spectral energy corresponding to at least one of the target frequency bands are compared to obtain the audio analysis result.

[0011] In some exemplary embodiments of this disclosure, based on the foregoing scheme, comparing the first spectral energy and the second spectral energy corresponding to at least one of the target frequency bands to obtain the audio analysis result includes: For at least one of the target frequency bands, determine the ratio between the first spectral energy and the second spectral energy; If the ratio is less than the lower limit threshold of energy fluctuation or greater than the upper limit threshold of energy fluctuation, the audio analysis sub-result of the target frequency band is determined to be spectral abnormal. If the ratio is greater than or equal to the lower limit threshold of energy fluctuation and less than or equal to the upper limit threshold of energy fluctuation, then the audio analysis sub-result of the target frequency band is determined to be normal in terms of spectrum. The audio analysis results are determined based on the audio analysis sub-results of each target frequency band.

[0012] In some example embodiments of this disclosure, based on the foregoing scheme, the spectral anomaly includes one of frequency component missing anomaly and energy rise anomaly; If the ratio is less than the lower threshold of energy fluctuation or greater than the upper threshold of energy fluctuation, the audio analysis sub-result of the target frequency band is determined to be spectrally abnormal, including: If the ratio is less than the lower limit threshold of energy fluctuation, the audio analysis sub-result of the target frequency band is determined to be an abnormal frequency component missing value. If the ratio is greater than the upper limit threshold of energy fluctuation, the audio analysis sub-result of the target frequency band is determined to be an abnormal energy rise.

[0013] In some example embodiments of this disclosure, based on the foregoing scheme, determining the audio analysis result according to the audio analysis sub-results of each target frequency band includes: If the audio analysis sub-results for each target frequency band are all normal, then the audio analysis results are determined to be normal in terms of spectrum. If at least one audio analysis sub-result of the target frequency band is a frequency band anomaly, then the audio analysis result is determined to be a spectrum anomaly.

[0014] In some example embodiments of this disclosure, based on the foregoing scheme, the step of locating faults in the GPU hardware decoding process based on the audio analysis results and the video status data, and determining the GPU hardware decoding fault point, includes: If the audio analysis result shows an abnormal spectrum, then the GPU hardware decoding process is determined to be faulty. If the audio analysis result shows a normal spectrum and the video status data indicates that the video output signal is choppy, then the GPU hardware decoding process is determined to be normal, and the video processing link is determined to be faulty.

[0015] In some example embodiments of this disclosure, based on the foregoing scheme, the hardware decoding function is implemented by a hardware structure including a video decoding unit and an audio decoding unit; the step of determining that the GPU hardware decoding process has a fault if the audio analysis result is a spectral abnormality includes: If the audio analysis result indicates an abnormal spectrum and the video status data indicates that the video decoding is in an unstable state, then it is determined that the video decoding unit in the GPU hardware decoding process is faulty. If the audio analysis result shows an abnormal spectrum and the video status data indicates that the video decoding is in a stable state, then it is determined that the audio decoding unit in the GPU hardware decoding process is faulty.

[0016] In some example embodiments of this disclosure, based on the foregoing scheme, the method for determining the reference spectrum energy includes: A pre-built baseline model is invoked, and its frequency domain statistical characteristics are obtained; the baseline model is constructed based on a fault-free reference audio signal. Based on the frequency domain statistical characteristics, the reference spectral energy is determined.

[0017] In some exemplary embodiments of this disclosure, based on the foregoing scheme, the acquisition of the audio output signal of the audio and video includes: The audio output signal of the audio and video is acquired at the final node of the audio rendering process; the audio output signal is a pulse code modulation format audio signal.

[0018] In some example embodiments of this disclosure, based on the foregoing scheme, determining the video state data corresponding to the video frame in the audio and video includes: By intercepting the application's calls to the graphics rendering interface, the context information of at least one video frame in the audio and video is extracted; Obtain the hardware decoding status of the GPU associated with at least one of the video frames, and determine the video status data corresponding to the video frame based on the hardware decoding status associated with at least one of the video frames and the context information.

[0019] According to a second aspect of the present disclosure, a GPU hardware decoding fault detection device is provided, comprising: The audio and video signal acquisition module is used to acquire the audio output signal of the audio and video when the audio and video use the hardware decoding function of the graphics processor (GPU). The audio spectrum analysis module is used to perform audio anomaly analysis on the spectrum of the audio output signal and obtain audio analysis results; The video status data determination module is used to determine the video status data corresponding to the video frame in the audio and video, wherein the video status data characterizes the video processing stability of the video frame during the GPU hardware decoding process. The fault location module is used to locate faults in the GPU hardware decoding process based on the audio analysis results and the video status data, and to determine the fault points in the GPU hardware decoding.

[0020] According to a third aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory storing computer-readable instructions that, when executed by the processor, implement the GPU hardware decoding fault detection method as described in the first aspect.

[0021] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the GPU hardware decoding fault detection method as described in the first aspect.

[0022] The technical solutions provided in this disclosure may have the following beneficial effects: The GPU hardware decoding fault detection method in the example embodiments of this disclosure, when audio and video use the hardware decoding function of the graphics processing unit (GPU), collects the audio output signal after hardware decoding processing. That is, the audio output signal is the signal directly output after hardware decoding processing. This can avoid the uncertainty caused by the traditional method of indirect inference based solely on GPU driver layer performance counters (such as decoder utilization, frame processing time, and temperature) and peripheral system indicators (such as network throughput and CPU load). Then, by performing audio anomaly analysis on the spectrum of the audio output signal, the audio analysis result can be obtained. This audio analysis result can truly reflect the quality of hardware decoding. At the same time, combined with video status data that characterizes the stability of video processing during GPU hardware decoding, the fault location of the GPU hardware decoding process can be accurately determined, thereby accurately identifying the GPU hardware decoding fault point.

[0023] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0024] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure. It is obvious that the drawings described below are merely some embodiments of this disclosure, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0025] Figure 1 The illustration shows a schematic diagram of a GPU hardware decoding fault detection method according to some embodiments of the present disclosure.

[0026] Figure 2 The schematic diagram illustrates a process for acquiring audio output signals according to some embodiments of the present disclosure.

[0027] Figure 3 The illustration shows a flowchart of determining video stream state data according to some embodiments of the present disclosure.

[0028] Figure 4 The illustration shows a flowchart of the steps for performing audio anomaly analysis on the spectrum of an audio output signal according to some embodiments of the present disclosure.

[0029] Figure 5 A schematic diagram illustrating a system framework for GPU hardware decoding fault detection according to some embodiments of the present disclosure is provided.

[0030] Figure 6 The illustration shows a schematic diagram of a GPU hardware decoding fault detection method according to some other embodiments of the present disclosure.

[0031] Figure 7 A schematic diagram of a GPU hardware decoding fault detection apparatus according to some embodiments of the present disclosure is shown.

[0032] Figure 8 The schematic diagram illustrates the structural schematic of a computer system of an electronic device according to some embodiments of the present disclosure.

[0033] Figure 9 A schematic diagram of a computer-readable storage medium according to some embodiments of the present disclosure is shown.

[0034] In the accompanying drawings, the same or corresponding reference numerals indicate the same or corresponding parts. Detailed Implementation

[0035] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this specification. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this specification as detailed in the appended claims.

[0036] Furthermore, the accompanying drawings are for illustrative purposes only and are not necessarily drawn to scale. The block diagrams shown in the drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0037] In this example embodiment, a GPU hardware decoding fault detection method is first provided. This GPU hardware decoding fault detection method can be applied to electronic devices. For example, electronic devices can be various computing devices with audio and video processing capabilities, system-level signal acquisition capabilities, and fault analysis capabilities, including but not limited to: multimedia automated testing terminals, GPU driver verification platforms, intelligent operation and maintenance servers, streaming media quality monitoring nodes, operating system diagnostic modules, and dedicated hardware testing devices with audio loopback acquisition interfaces, graphics API interception capabilities, and high-precision time synchronization mechanisms.

[0038] Figure 1 The illustration schematically shows a flowchart of a GPU hardware decoding fault detection method according to some embodiments of the present disclosure. (Reference) Figure 1As shown, the GPU hardware decoding fault detection method may include the following steps: Step S110: When the audio and video use the hardware decoding function of the graphics processing unit (GPU), the audio output signal of the audio and video is acquired. Step S120: Perform audio anomaly analysis on the spectrum of the audio output signal to obtain the audio analysis results; Step S130: Determine the video status data corresponding to the video frame in the audio and video. The video status data characterizes the video processing stability of the video frame during the GPU hardware decoding process. Step S140: Based on the audio analysis results and video status data, perform fault location on the GPU hardware decoding process to determine the GPU hardware decoding fault point.

[0039] According to the GPU hardware decoding fault detection method in this example embodiment, when audio and video use the hardware decoding function of the graphics processing unit (GPU), the audio output signal after hardware decoding is acquired. That is, the audio output signal is the signal directly output after hardware decoding, which can avoid the uncertainty caused by the traditional method of relying solely on GPU driver layer performance counters (such as decoder utilization, frame processing time, and temperature) and peripheral system indicators (such as network throughput and CPU load) for indirect inference. Then, by performing audio anomaly analysis on the spectrum of the audio output signal, the audio analysis result can be obtained. This audio analysis result can truly reflect the quality of the hardware decoding output signal. At the same time, combined with video status data that characterizes the stability of video processing during GPU hardware decoding, the fault location of the GPU hardware decoding process can be accurately determined, thereby accurately identifying the GPU hardware decoding fault point.

[0040] Furthermore, the aforementioned GPU hardware decoding fault detection method can eliminate reliance on human experience, improve testing efficiency and consistency, significantly enhance fault detection accuracy, diagnostic efficiency, and system reliability, and provide a scientific basis for high-quality verification of GPU audio and video hardware decoding functions.

[0041] The GPU hardware decoding fault detection method in this example embodiment will be further described below.

[0042] Step S110: When the audio and video use the hardware decoding function of the graphics processor GPU, the audio output signal of the audio and video is acquired.

[0043] GPU hardware decoding refers to the ability to decode compressed audio and video streams using the multimedia hardware units integrated within the GPU chip. This GPU hardware decoding function can be applied to various operating environments, including but not limited to local multimedia players (such as MPV, VLC, and Windows Media Player), web browsers (such as Chrome, Edge, and Safari), mobile video applications (such as streaming media clients on iOS or Android platforms), and cloud gaming or remote desktop systems. In these scenarios, the decoding task of audio and video data is handled by the GPU's internal hardware decoding engine, thereby significantly reducing system load and improving the smoothness of high-bitrate content playback.

[0044] In some embodiments, it is detected whether the audio and video use the hardware decoding function of the GPU; if the audio and video use the hardware decoding function of the graphics processor GPU, the audio output signal of the audio and video is acquired; if the audio and video do not use the hardware decoding function of the GPU, the GPU hardware decoding fault detection is stopped.

[0045] Specifically, different methods are used to detect whether audio and video are using the GPU's hardware decoding function, depending on the application scenario. These different application scenarios can include local video player scenarios, browser scenarios, and mobile application scenarios.

[0046] For example, in a local video player scenario, control and querying can be performed via injected commands or API interfaces to detect whether audio and video are using GPU hardware decoding. Local video players can include MPV, VLC, Windows Media Player, etc. MPV player can use the command "mpv --hwdec=vaapi --vo=gpu Test.mp4" to detect whether audio and video are using GPU hardware decoding. VLC player can confirm whether audio and video are using GPU hardware decoding by calling its user interface elements and information windows. Windows Media Player, a system-level player, can call the system performance counter interface Performance Counter to monitor the utilization of the GPU's video decoding engine in real time. If the utilization is higher than the baseline value, it is determined that audio and video are using GPU hardware decoding; if the utilization is lower than or equal to the baseline value, it is determined that audio and video are not using GPU hardware decoding.

[0047] For example, in a browser scenario, one can detect whether audio and video are using the GPU's hardware decoding capabilities by monitoring the browser's graphics media interface or system-level performance counters. Browser environments include Chrome, Edge, and Safari.

[0048] For example, in mobile application scenarios, hardware decoding data can be captured through system-provided audio graph routing or GPU debugging interfaces to determine whether the hardware decoding function is enabled. Examples of mobile applications include video apps on iOS and Android.

[0049] The audio output signal refers to the audio signal output after being decoded by the GPU's audio hardware decoding unit. In some embodiments, the audio output signal can be the audio signal directly output after being decoded by the GPU's audio hardware decoding unit. In other embodiments, the audio output signal can also be a Pulse Code Modulation (PCM) format audio signal that has been decoded by the GPU's audio hardware decoding unit, then processed by system audio post-processing (such as resampling, reverb, and volume adjustment), mixed by a mixer, and finally sent to the audio driver and output to the speaker or recording loopback interface.

[0050] In some embodiments, the audio output signal of the audio and video can be acquired at the final node of the audio rendering process; the audio output signal is a pulse code modulation format audio signal.

[0051] The audio rendering process refers to the process by which the operating system's audio subsystem renders the raw audio data obtained after the audio and video content is decoded by the GPU hardware. For example, the audio rendering process may include format conversion (such as sampling rate and bit depth adaptation), multi-channel audio mixing, sound effects processing (such as equalization and noise reduction), buffer scheduling, and outputting an audio signal in pulse code modulation format through the sound card driver.

[0052] The final node in the audio rendering process refers to the last data acquisition point, which is either the output buffer of the sound card driver or the audio loopback endpoint provided by the operating system. The signal output by this final node contains all the results of the upper-layer processing and has not been transmitted through external physical channels, thus accurately reflecting the audio quality after GPU hardware decoding.

[0053] In Windows operating systems, the final node of audio rendering can be accessed through the Shared-Mode Loopback mechanism of the Windows AudioSession API (WASAPI), allowing for the cyclic acquisition of audio output signals. This mechanism allows applications to capture the mixed output of all active audio streams on the default audio rendering device (player) in read-only mode, without modifying player behavior or injecting intermediate proxies, thus ensuring that the acquired data is completely consistent with what the user hears.

[0054] In the Linux operating system, the final node for audio rendering can be implemented through the loopback device (snd-aloop) of the Advanced Linux Sound Architecture (ALSA), or by monitoring the final node for audio rendering and capturing the audio output signal through a monitor source provided by the PulseAudio audio service (e.g., alsa_output.pci-0000_00_1f.3.analog-stereo.monitor). These mechanisms can intercept the system's main audio output stream, covering all pulse-code modulation data that has undergone hardware decoding and is sent to the audio service.

[0055] In macOS, audio output signals can be captured using the AudioQueue or AudioUnit interfaces in the Core Audio framework. Alternatively, a loopback channel can be created using the system's built-in Multi-Output Device or third-party virtual audio drivers (such as BlackHole or Loopback) to capture the final mixed audio stream's audio output signal.

[0056] By collecting data at the final node of the audio rendering process, interference introduced by obtaining data from the player's internal buffer, decoder output queue, or intermediate audio processing module can be avoided. This ensures that the analyzed audio signal truly reflects the actual output quality of the GPU hardware decoding and the entire system's audio link, providing a high-fidelity and high-reliability data foundation for subsequent spectrum anomaly detection and fault root cause localization.

[0057] In specific implementation methods, such as Figure 2 As shown, the audio stream monitoring task is started to obtain the audio device and its default format, and detect the operating system platform. If the operating system platform is Windows, the shared listen mode is initialized through the Windows Audio Session API (WASAPI). If the operating system platform is Linux, the ALSA loopback interface is initialized. If the operating system platform is macOS, the Core Audio output unit callback is initialized.

[0058] The acquisition thread is started, reading the audio data block of the final node in the audio rendering process. A high-precision timestamp is added to the current audio data block, which is then encapsulated into a data packet and stored in the audio data queue. It is then determined whether the monitoring task needs to continue. If it is detected that the monitoring task for the audio output signal still needs to continue monitoring, the process proceeds to the next iteration. In the next iteration, the audio data block of the final node in the audio rendering process continues to be read until it is detected that the monitoring task for the audio output signal no longer needs to be monitored. At this point, acquisition stops, and resources are released. Thus, the audio output signal can be constructed from the audio data blocks in the audio data queue.

[0059] In other embodiments, a hook module can be injected into the audio / video playback process to intercept the audio output signals passed when it calls the operating system's audio output interface (such as IAudioRenderClient::GetBuffer for Windows, snd_pcm_writei for Linux, and AudioQueueEnqueueBuffer for macOS).

[0060] It is understood that the above-described method for acquiring audio output signals is merely an illustrative example. Other audio signal capture methods may be used in other embodiments, such as kernel-level audio driver filters (e.g., Windows WDMAudio Filter Driver), virtual audio device redirection, remote debugging interfaces (e.g., Android ADB audiodump, iOS Core Audio Remote IO callback capture), or dedicated test fixtures based on hardware loopback, etc. The specific implementation is not limited to these methods.

[0061] In step S120, audio anomaly analysis is performed on the spectrum of the audio output signal to obtain the audio analysis results.

[0062] The spectrum refers to the frequency component distribution of the audio output signal after it has been converted from the time domain to the frequency domain.

[0063] Specifically, the audio output signal is a one-dimensional pulse-code modulated digital signal, represented as a time-varying sequence of sampling points. By performing frame segmentation, windowing (such as the Hanning window), and Fast Fourier Transform (FFT) on this audio output signal, the amplitude value corresponding to each frequency point can be obtained, forming an amplitude spectrum. This spectrum reflects the energy intensity distribution of the audio signal in different frequency ranges (such as low frequency, mid frequency, and high frequency), and is a key feature characterizing the quality and integrity of audio content.

[0064] For example, normal audio has a continuous and stable energy distribution in the frequency bands sensitive to the human ear (such as 1 kHz–8 kHz), while hardware decoding failures may lead to energy loss in specific frequency bands or abnormal enhancement of high-frequency noise. Therefore, the spectrum, as an objective mathematical expression of the audio output signal, is the fundamental data for subsequent anomaly detection.

[0065] Audio analysis results refer to the results obtained by analyzing the spectrum of the audio output signal. These results characterize whether the audio output signal exhibits any abnormalities and the types of these abnormalities. Specifically, they can include audio spectrum anomalies and normal audio spectrum. Audio spectrum anomalies can further include the detection of abnormal phenomena such as missing frequency components, increased noise floor, silence fragments, or transient distortion.

[0066] In other embodiments, the audio analysis results may also include at least one of the following: the start time stamp and duration of the anomaly, the frequency range involved, quantitative indicators of the severity of the anomaly (such as energy ratio R, deviation from standard deviation multiple, etc.), and the final anomaly classification label (such as "audio decoding interruption", "high frequency distortion", "suspected popping sound"), and are not limited thereto.

[0067] Specifically, the audio output signal can be frequency domain converted to obtain the audio spectrum signal, and the audio spectrum signal can be used for audio anomaly analysis to obtain the audio analysis results.

[0068] The audio output signal can be divided into frames with a fixed frame length (e.g., 1024 sampling points) to obtain multi-frame signal data. The Hanning window function is applied to each frame of signal data to reduce spectral leakage, and then a fast Fourier transform (e.g., a fast Fourier transform with N=2048) is performed to convert the audio output signal from a time domain signal to a frequency domain signal, thus obtaining the audio spectrum signal.

[0069] In some embodiments, audio anomaly analysis of audio spectrum signals can be achieved using various techniques. For example, audio anomaly analysis can be performed using statistical modeling methods (such as setting dynamic thresholds based on the mean and standard deviation of the baseline spectrum), energy comparison methods (calculating the degree of energy attenuation in a specific frequency band), noise floor monitoring, spectral flatness, or harmonic-to-noise ratio analysis. Machine learning methods, such as support vector machines (SVM), isolation forests, or deep neural networks (such as convolutional autoencoders and Transformers), can also be used to classify and identify normal and abnormal spectral patterns. The above methods can be used individually or in combination, and the specific implementation is not limited to these. As long as it can objectively identify audio output quality degradation based on the spectral signal, it falls within the scope of this solution.

[0070] In other embodiments, a dynamic threshold adjustment mechanism may be introduced to adaptively adjust the threshold used to judge anomalies based on the type of the current audio output signal (such as speech, music, or silence) to reduce the false alarm rate; or it may be combined with time-domain features (such as zero-crossing rate or short-time energy change) to assist in judging transient anomalies such as popping sounds and clicking sounds.

[0071] In step S130, the video state data corresponding to the video frame in the audio and video is determined. The video state data characterizes the video processing stability of the video frame during the GPU hardware decoding process.

[0072] In this context, a video frame refers to a single image unit output by the GPU's video decoding unit after decoding a compressed video stream. Video frames can be further processed by the graphics rendering system and submitted to the display device for presentation.

[0073] Video state data characterizes the video processing stability of video frames during GPU hardware decoding. It can be used to describe the temporal continuity, scheduling regularity, and resource usage of video frames during GPU hardware decoding and subsequent processing.

[0074] In some embodiments, video state data may include at least one of the following: presentation timestamp (PTS) of a video frame, decoding timestamp (DTS) of a video frame, a dropped frame count calculated based on consecutive presentation time intervals, and the instantaneous occupancy of the GPU video decoding engine associated with the presentation timestamp of a frame.

[0075] The presentation timestamp refers to the system time when a video frame is submitted to the operating system's display compositor or directly displayed, and can be used to calculate the time interval between adjacent frames. The dropped frame count refers to the number of video frames that failed to be presented on time within the expected playback time, which can be statistically analyzed by comparing the deviation between the actual frame interval and the nominal frame rate (e.g., 30 fps corresponds to 33.3 milliseconds / frame). The GPU video decoding engine refers to the hardware module inside the GPU chip dedicated to video decoding; its instantaneous utilization rate can be read in real time through the operating system's performance monitoring interface (such as Windows' GPU performance counters, Linux's NVML, or Android's GPU debugging tools).

[0076] In other embodiments, the video status data may also include at least one of the following: video decoding completion time, frame type identifier (I-frame / P-frame / B-frame), resolution change event, or hardware decoding error flag reported by the driver layer.

[0077] In some embodiments, video status data can be collected by intercepting applications' calls to the graphics rendering interface.

[0078] Specifically, by intercepting the application's calls to the graphics rendering interface, the context information of at least one video frame in the audio and video is extracted; the hardware decoding status of the GPU associated with at least one video frame is obtained; and the video status data corresponding to the video frame is determined based on the hardware decoding status and context information associated with at least one video frame.

[0079] Among them, the graphics rendering interface refers to the standard API interface used by applications to submit graphics or video rendering instructions to the GPU, including but not limited to Direct3D, OpenGL, Vulkan, Metal or platform-specific media framework interfaces (such as Android MediaCodec, Windows Media Foundation, FFmpeg hardware acceleration interface, etc.).

[0080] Context information may include, but is not limited to: timestamps of video frames, frame types (I-frames, P-frames, B-frames), resolution, color format (such as NV12, YUV420), target display surface (Surface or Texture handle), submission queue ID, and associated audio and video stream identifiers.

[0081] The hardware decoding status of a GPU refers to the internal operating metrics and feedback signals generated by the GPU's hardware decoding engine when processing a corresponding video frame. In some embodiments, the GPU's hardware decoding status can be obtained by querying the GPU driver's performance counters or diagnostic registers, including: hardware decoding unit utilization, decoding latency (time from submission to completion), frame buffer wait time, command queue depth, error codes (such as stream parsing failure, missing reference frame, memory access anomaly), etc. In other embodiments, the hardware decoding status may also include callback events from the operating system's media subsystem, such as successful decoding, frame drop notifications, or resource shortage warnings.

[0082] In some embodiments, a hook mechanism can be injected into the driver layer or runtime library to transparently intercept the application's calls to the aforementioned interfaces, thereby capturing key context information related to video frames.

[0083] In some embodiments, hardware decoding status and context information can be used as video status data corresponding to a video frame. In other embodiments, the hardware decoding status and context information can be filtered to obtain video status data corresponding to a video frame. In other embodiments, other methods can be used to determine video status data, which are not limited here.

[0084] For example, with the Vulkan API, you can hook the `vkQueuePresentKHR` function to record the presentation timestamp and related context information of each video frame submitted, thus obtaining video state data. Understandably, since this function is the necessary interface for the application to finally submit a frame to the display queue for presentation, intercepting it here allows for precise capture of the final submission event of each frame, ensuring the extracted frame information has frame-level accuracy.

[0085] For example, with the OpenGL API, functions such as eglSwapBuffers or glXSwapBuffers can be called to swap the frame buffers before and after the video to capture video state data. These calls directly correspond to the screen rendering timing of a single video frame; intercepting these calls can also achieve frame-level precision data acquisition.

[0086] In some embodiments, such as Figure 3 As shown, the video status data acquisition process includes the following steps: start the video stream status data monitoring task to begin observing the GPU hardware decoding process; initialize the interception environment and configure the call listening mechanism for graphics rendering interfaces (such as Direct3D, Vulkan, or MediaCodec); enter the waiting state and wait for the frame rendering instruction to be called.

[0087] When a frame rendering command is invoked, the command is intercepted or a corresponding event is triggered to obtain the context information of the current video frame. This context information may include the video frame's timestamp, frame type, resolution, color format, target display surface, submission queue ID, and associated audio and video stream identifiers. The DTS, PTS, and frame drop count of the current video frame are extracted, and a high-precision timestamp is added to the DTS, PTS, and frame drop count of the current video frame.

[0088] The system queries the GPU hardware decoding status, including whether it is in decoding mode, whether decoding was successful, error codes, queue latency, etc. If the GPU hardware decoding status is in decoding mode, it calculates the instantaneous utilization rate of the GPU video decoding engine associated with the PTS and adds a high-precision timestamp to this instantaneous utilization rate. Then, the timestamped DTS, PTS, frame drop count, and timestamped instantaneous utilization rate can generate the video status data of the current video frame and store it in the queue. It determines whether the monitoring task needs to continue. If it is detected that monitoring still needs to continue, it enters the next loop round and continues to wait for the frame rendering instruction to be called in the next loop round until it is detected that the monitoring task no longer needs to be monitored. At this point, the monitoring task ends and the video status data is output.

[0089] Furthermore, in some embodiments, to ensure the temporal consistency of audio and video analysis, each video state data is stamped with a high-precision system timestamp and synchronized with the acquisition timeline of the audio output signal. The high-precision system timestamp can be generated using a monotonic clock provided by the operating system (such as Linux's CLOCK_MONOTONIC, Windows' QueryPerformanceCounter, or macOS's mach_absolute_time).

[0090] By synchronously collecting and correlating audio spectrum and video decoding timestamps, it is possible to accurately distinguish whether the audio decoding itself is faulty or whether the video decoding problem is dragging down the audio, thus achieving fine-grained fault location.

[0091] In other implementations, auxiliary metrics such as GPU command buffer submission latency, video memory bandwidth utilization, or video decoding queue depth can be combined to further enhance the ability to assess the stability of video processing.

[0092] In step S140, based on the audio analysis results and video status data, the GPU hardware decoding process is fault located to determine the GPU hardware decoding fault point.

[0093] The GPU hardware decoding process refers to the complete processing chain that uses the GPU's internal multimedia hardware units to decode compressed audio and video streams, including video decoding, audio decoding, post-processing, synchronization, and output. This process involves the coordinated work of multiple functional modules, and any abnormality in any of these steps may lead to playback malfunctions perceived by the user, such as stuttering, audio-video desynchronization, audio popping, muting, or screen tearing.

[0094] GPU hardware decoding failure points refer to the specific functional modules (nodes) or processing stages that cause audio and video anomalies during the GPU hardware decoding process. GPU hardware decoding failure points can be located in the GPU audio decoding unit, GPU video decoding unit, driver layer scheduling logic, system resource contention mechanisms, or external interference paths, and are not limited to these.

[0095] In some embodiments, if the audio analysis results indicate an audio spectrum anomaly (such as missing frequency components, increased noise floor, or silent segments), while the video status data shows continuous timestamps, zero dropped frame count, and stable GPU video decoding engine utilization, then the fault point is determined to be located in the GPU audio decoding unit, which may be caused by audio decoder firmware defects, driver configuration errors, or audio stream parsing failures.

[0096] In other embodiments, if the video status data shows significant jitter in the inter-frame time interval, a continuous increase in the number of dropped frames, or a periodic sharp drop in the GPU video decoding engine utilization, while the audio analysis results do not detect any obvious abnormalities (normal spectral energy distribution, no noise rise), then the fault point is determined to be located in the GPU video decoding unit or its scheduling link, which may be due to insufficient video decoding capability, memory bandwidth bottleneck, or driver frame submission blockage.

[0097] In other embodiments, if the audio analysis results and the video status data both show abnormalities, and the time of the abnormality is highly synchronized (the time deviation is within a preset tolerance range, such as ±10 milliseconds), then it is further determined whether there is shared resource contention or system-level interference: if the system CPU load, memory bandwidth, or power management status fluctuates drastically during the abnormality period, it is attributed to external system interference; if the abnormality only reproduces under specific high bitrate video sources, and the overall GPU temperature or power consumption reaches the threshold, it is determined to be caused by overall GPU hardware decoding overload or thermal throttling.

[0098] The fault location logic can be implemented through a predefined rule engine. In some embodiments, a decision tree model is constructed, which takes audio anomaly type, video status indicators and their temporal correlation as input features and outputs corresponding fault point classification labels; in other embodiments, a state machine model is used, which jumps to different fault state nodes according to the anomaly combination pattern and triggers corresponding diagnostic suggestions.

[0099] Understandably, the fault localization process supports multi-level refinement. For example, after initially determining that the GPU audio hardware decoding unit is faulty, it can be further refined by combining audio encoding format (such as AAC vs. AC-3), channel configuration (stereo vs. 5.1 surround sound), or sampling rate information to narrow it down to the specific encoding and decoding sub-module; after determining that "video decoding is unstable", it can be used to help identify whether it is a compatibility problem triggered by specific video characteristics based on frame type (abnormal ratio of I / P / B frames), resolution switching behavior, or chroma sampling format.

[0100] In some implementations, a historical fault knowledge base can be introduced for matching to more accurately identify GPU hardware decoding fault points. For example, the combined features of the current audio analysis results and video status data can be compared with a known fault case library. If a match is found, existing root cause conclusions and repair suggestions can be directly reused.

[0101] Other implementation methods can also generate structured diagnostic reports, including fault type, root cause, timestamp of occurrence, duration and related evidence data, for developers, testers or operations and maintenance platforms to further verify.

[0102] The contents of steps S110 to S140 will be described in detail below.

[0103] In an example embodiment of this disclosure, reference is made to Figure 4 The audio anomaly analysis of the audio output signal spectrum in step S120 can be performed through the following steps to obtain the audio analysis results, specifically steps S410 to S420: Step S410: Determine the target spectral energy of the spectrum based on the spectrum of the audio output signal.

[0104] The target spectral energy refers to the sum of spectral energy of the audio output signal calculated within a specific frequency band, which is used to characterize the activity level or integrity of the audio content within that frequency band.

[0105] In some embodiments, the squared amplitude values ​​of each frequency point in the spectrum of the audio output signal can be calculated to obtain the power spectrum. The power spectrum can then be integrated or summed within a preset target frequency band to obtain the target spectral energy of the spectrum.

[0106] In other embodiments, the target spectral energy can also be calculated based on psychoacoustic scales such as Mel-scaled spectrograms or Bark spectrograms to better align with human auditory perception characteristics. Specifically, the linear spectrum is first mapped to a Mel filter bank, and then the energy of each Mel channel is weighted and summed to obtain the perceptually weighted target spectral energy.

[0107] Understandably, the calculation of target spectral energy can be performed on a single frame or multiple frames of audio signal. In some implementations, the target spectral energy can be averaged over M consecutive frames (e.g., M=50) to suppress transient fluctuations and obtain a more stable baseline reference; in other implementations, a sliding window can be used to update the target spectral energy in real time for online anomaly detection.

[0108] Step S420: Compare the target spectral energy with the preset reference spectral energy to obtain the audio analysis results.

[0109] Among them, the reference spectrum energy refers to the baseline energy value pre-established for the same specific frequency band under normal GPU hardware decoding function. The reference spectrum energy can represent the expected audio output level under fault-free scenario.

[0110] In some embodiments, the determination of the reference spectral energy includes: invoking a pre-built baseline model and obtaining the frequency domain statistical characteristics of the baseline model; the baseline model is constructed based on a fault-free reference audio signal; and the reference spectral energy is determined based on the frequency domain statistical characteristics.

[0111] The baseline model refers to a benchmark model formed after frequency domain feature extraction and statistical modeling of multiple sets of reference audio signals. In some embodiments, the baseline model can store the energy distribution characteristics of each frequency band on a target frequency band basis. In other embodiments, the baseline model can include richer statistical information, such as mean, variance, higher-order moments, or confidence intervals.

[0112] The reference audio signal refers to a standard audio sample acquired under conditions where the GPU hardware decoding function is operating normally and there is no external interference. In some embodiments, the reference audio signal may be a pre-recorded high-quality test audio source (such as full-band white noise, pink noise, or a standard audio test file), which is acquired and stored after being played through the same hardware decoding link; in other embodiments, the reference audio signal may be automatically calibrated and generated by the system during the initialization phase, and multiple sets of reference data may be saved according to content type (such as voice, music, silence).

[0113] The process of constructing a baseline model may include: performing frame segmentation, windowing, and FFT processing on multiple fault-free reference audio signals to obtain corresponding reference spectrum signals; calculating the spectral energy of each sample within the target frequency band for each target frequency band; and then statistically analyzing the mean μ and standard deviation σ of the spectral energy of the target frequency band as the frequency domain statistical characteristics of the band.

[0114] For example, M frames of reference audio signals in a fault-free state can be continuously acquired, and the reference audio signals can be converted from the time domain to the frequency domain to obtain a reference spectrum signal; for each frequency point k in the reference spectrum signal, a sample set consisting of M amplitude values ​​can be constructed: {X_1, X_2, ..., X_M}, where.

[0115] Therefore, the average value can be calculated using the following formula (1): (1) Among them, the average value This indicates the central trend of the amplitude at that frequency point, i.e., the typical amplitude value under normal conditions. Let represent the spectral amplitude of the i-th frequency point k, where i ranges from 1 to M.

[0116] The standard deviation can be calculated using the following formula (2): (2) Among them, standard deviation This indicates the range or dispersion of amplitude at that frequency point, quantifying the amplitude value's position around the average value during normal operation. The range of fluctuations.

[0117] In other embodiments, the baseline model may also express frequency domain statistical characteristics in the form of probability density functions (such as Gaussian mixture models), histogram distributions, or machine learning embedding vectors, while the reference spectral energy is derived through expected value, mode, or maximum likelihood estimation.

[0118] In other embodiments, the baseline model can be constructed according to content type. For example, separate baseline sub-models can be established for media types such as audio, music, and silence. At runtime, the matching baseline model can be automatically selected based on the metadata of the currently playing content, thereby improving the accuracy of the reference spectral energy adaptation.

[0119] Understandably, the baseline model can be pre-configured before the device leaves the factory, or it can be generated online through a self-calibration process during the initial deployment of the system or periodic maintenance. In some implementations, the baseline model supports incremental updates, meaning that when the system continuously detects long-term stable high-quality audio output, new data can be incorporated into the statistics to dynamically optimize the frequency domain statistical characteristics.

[0120] Specifically, based on frequency domain statistical characteristics, the reference spectral energy is determined, including: the mean μ can be directly used as the reference spectral energy; or, depending on the sensitivity requirements of the application scenario, μ ± kσ (where k is an adjustment coefficient, such as k = 0, 0.5, or 1) can be used as a dynamic reference spectral energy. For example, in high-reliability detection scenarios, μ - 0.5σ can be taken as a conservative reference spectral energy to improve the detection sensitivity to energy attenuation.

[0121] In some embodiments, the reference spectral energy can be obtained through a calibration process during system initialization. Specifically, assuming that GPU hardware decoding is enabled and the playback content is a reference audio signal, multiple frames of audio output signals are acquired, their average spectral energy within the target frequency band is calculated, and this average spectral energy is stored as the reference spectral energy in a local configuration file or memory cache. The reference audio signal is a fault-free standard test audio source, such as full-band white noise, pink noise, or high-quality reference audio.

[0122] In other embodiments, different reference spectrum energies can be dynamically adapted according to different audio and video content types. For example, for voice content, a reference energy with 300 Hz–3.4 kHz as the main frequency band can be used; for music content, a reference energy with the entire frequency band of 20 Hz–20 kHz can be used; and a matching reference spectrum energy template can be automatically selected based on media metadata (such as MIME type, sampling rate, number of channels) or content feature recognition module.

[0123] In some embodiments, the reference spectrum energy supports online updates. When a long-term stable high-fidelity playback state is detected, the reference spectrum energy can be automatically refreshed to adapt to different device aging characteristics or environmental changes.

[0124] The comparison operation may include calculating the relative deviation between the target spectral energy and the reference spectral energy. This could involve calculating the ratio between the target spectral energy and the reference spectral energy, or calculating the difference between the target spectral energy and the reference spectral energy.

[0125] In other embodiments, the comparison process may also incorporate time-based constraints. For example, an anomaly may be confirmed as valid only if the target spectral energy is below a threshold for N consecutive frames (e.g., N≥5) to avoid false alarms caused by instantaneous fluctuations; or the short-term energy change rate of the audio signal may be used to determine whether there are sudden popping or clicking sounds.

[0126] In some embodiments, the target spectral energy is characterized by a first spectral energy within a target frequency band; determining the target spectral energy of the spectrum based on the spectrum of the audio output signal includes: acquiring the first spectral energy within at least one target frequency band within the spectral range of the audio output signal; comparing the target spectral energy with a preset reference spectral energy to obtain an audio analysis result includes: acquiring the second spectral energy of the reference audio signal within the target frequency band, and comparing the first spectral energy and the second spectral energy corresponding to at least one target frequency band to obtain an audio analysis result.

[0127] Specifically, the audio output signal can be frequency-domain converted to obtain an audio spectrum signal. Then, for at least one target frequency band within the spectral range of the audio spectrum signal, the first spectral energy within that target spectrum is obtained. The audio spectrum signal refers to the signal characterizing the energy distribution of audio at various frequency components, obtained by frequency-domain conversion of the time-domain audio output signal. The audio spectrum signal can be plotted with frequency on the horizontal axis and amplitude or power on the vertical axis to reflect the energy strength of different frequency bands (such as low frequency, mid frequency, and high frequency), and is used to analyze the integrity, distortion, noise, and other characteristics of audio content.

[0128] The audio output signal can be frequency domain transformed by using Short-Time Fourier Transform (STFT) or Fast Fourier Transform (FFT) to convert the time-domain sampling sequence of the audio output signal into a complex spectrum containing amplitude and phase information. Then, the power spectrum, i.e., the audio spectrum signal, can be obtained by calculating the square of its amplitude.

[0129] A target frequency band refers to a pre-defined key frequency range used for fault detection. In some embodiments, the target frequency band may include at least one of a low-frequency band (e.g., 20 Hz–250 Hz), a mid-frequency band (e.g., 250 Hz–4 kHz), and a high-frequency band (e.g., 4 kHz–22 kHz), corresponding to bass, vocal body, and detail / noise components in audio, respectively. In other embodiments, the target frequency band may also be dynamically divided according to the actual application scenario. For example, for voice call scenarios, the 300 Hz–3.4 kHz band may be the focus of monitoring; for high-fidelity music playback scenarios, a finer-grained equal-ratio frequency band division (e.g., 1 / 3 octave) may be used; when detecting high-frequency distortion or noise rise, a high-frequency silent segment (e.g., 16 kHz to 22 kHz) may be selected as the frequency band.

[0130] Understandably, the thresholds for each target frequency band can be configured independently to accommodate the sensitivity differences of different frequency bands. In addition, the comparison process can also introduce time continuity constraints, such as requiring the abnormal state to last for more than N frames (e.g., N=10) before triggering the final alarm, in order to suppress false judgments caused by momentary jitter.

[0131] The first spectral energy refers to the sum or integral of the power spectrum values ​​of the current audio output signal at all frequency points within the target frequency band, used to characterize the actual output intensity of the target frequency band. The second spectral energy refers to the energy value of the reference audio signal within the same target frequency band, serving as a benchmark under normal conditions.

[0132] Specifically, for each target frequency band within the spectral range of the audio spectrum signal, the amplitude values ​​in the target frequency band are summed to obtain the first spectral energy within that target frequency band.

[0133] The first spectral energy of the audio spectrum signal within the target frequency band can be calculated using the following formula (3): (3) in, It is the first spectral energy of the audio spectrum signal. It is the audio spectrum signal number The amplitude value of each frequency component, It is the highest frequency component of the audio spectrum signal within the target frequency band. It is the lowest frequency component of the audio spectrum signal within the target frequency band. Integer Start by taking... , Until .

[0134] Similarly, the reference audio signal is frequency-domain transformed to obtain the reference spectrum signal. Specifically, this can be achieved through Short-Time Fourier Transform (STFT) or Fast Fourier Transform (FFT), which converts the time-domain sampling sequence of the reference audio signal into a complex spectrum containing amplitude and phase information, and then calculates the square of its amplitude to obtain the power spectrum, i.e., the reference spectrum signal.

[0135] For each target frequency band within the spectral range of the reference spectrum signal, the amplitude values ​​in the target frequency band are summed to obtain the second spectral energy within that target frequency band.

[0136] The second spectral energy of the reference spectral signal within the target frequency band can be calculated using the following formula (4): (4) in, It is the second spectral energy of the reference spectrum signal. It is the average amplitude value of the reference spectrum signal within the target frequency band. It is the highest frequency component of the reference spectrum signal within the target frequency band. It is the lowest frequency component of the reference spectrum signal within the target frequency band. Integer Start by taking... , Until .

[0137] In other embodiments, the second spectral energy can be dynamically updated. When the system is in a stable playback state for a long period of time, the reference value can be updated based on a sliding average of historical normal data, improving its adaptability to equipment aging or environmental changes.

[0138] The comparison operation may include calculating the relative deviation between the first spectral energy and the second spectral energy. For example, calculating the ratio between the first spectral energy and the second spectral energy, or calculating the difference between the first spectral energy and the second spectral energy.

[0139] In some embodiments, comparing the first spectral energy and the second spectral energy corresponding to at least one target frequency band to obtain an audio analysis result includes: for at least one target frequency band, determining the ratio between the first spectral energy and the second spectral energy; if the ratio is less than the lower limit threshold of energy fluctuation or greater than the upper limit threshold of energy fluctuation, then determining that the audio analysis sub-result of the target frequency band is spectrally abnormal; if the ratio is greater than or equal to the lower limit threshold of energy fluctuation and less than or equal to the upper limit threshold of energy fluctuation, then determining that the audio analysis sub-result of the target frequency band is spectrally normal; and determining the audio analysis result based on the audio analysis sub-result of each target frequency band.

[0140] Both the lower and upper thresholds for energy fluctuation can be configured independently based on the characteristics of different target frequency bands. For example, for the mid-frequency band where human voices are sensitive (e.g., 300 Hz–3.4 kHz), the lower threshold for energy fluctuation can be set to 0.4 and the upper threshold to 1.5 to tolerate moderate dynamic changes but exclude silence or distortion; for the high-frequency silent band (e.g., 16 kHz–22 kHz), the lower threshold can be set to 0.1 and the upper threshold to 0.8 to prevent noise rise from being misjudged as normal.

[0141] In some embodiments, the lower limit threshold and the upper limit threshold of energy fluctuation can be derived based on historical normal data statistics, for example, by taking the reference energy mean ± 2 times the standard deviation as the upper and lower limits; or they can be dynamically adjusted through manual calibration or adaptive learning mechanisms to adapt to different playback content, device models or environmental noise levels.

[0142] Specifically, the ratio between the first spectral energy and the second spectral energy can be determined using the following formula (5): (5) in, It's a ratio. It is the first spectrum energy. It is the second spectrum energy.

[0143] In other implementations, a multi-level threshold strategy or a fuzzy logic controller can be used to perform nonlinear mapping on the energy ratio to improve the robustness of anomaly detection.

[0144] Furthermore, in some embodiments, the spectral anomaly includes one of frequency component missing anomaly and energy rise anomaly; if the ratio is less than the lower threshold of energy fluctuation or greater than the upper threshold of energy fluctuation, the audio analysis sub-result of the target frequency band is determined to be spectral anomaly, including: if the ratio is less than the lower threshold of energy fluctuation, the audio analysis sub-result of the target frequency band is determined to be frequency component missing anomaly; if the ratio is greater than the upper threshold of energy fluctuation, the audio analysis sub-result of the target frequency band is determined to be energy rise anomaly.

[0145] Among them, frequency component loss anomaly refers to the audio output signal having energy lower than the normal reference level within the target frequency band, manifesting as the loss or attenuation of audio content in that frequency band. This may be caused by GPU audio hardware decoder decoding interruption, channel mute, driver configuration errors, or bitstream parsing failure. Energy rise anomaly refers to the abnormal increase in energy of the audio output signal within the target frequency band, which can manifest as increased noise, pops, or jarring noise. This may originate from internal errors in the hardware decoder, post-processing distortion, buffer overflow, or system interference.

[0146] In some embodiments, the lower limit threshold and the upper limit threshold of energy fluctuation can be determined based on the second spectral energy and a preset sensitivity control coefficient, respectively. In some embodiments, the preset sensitivity control coefficient can be a sigma multiple of the control sensitivity, for example, a sigma multiple of the control sensitivity n=3 (corresponding to a 99.7% confidence interval).

[0147] Specifically, the lower limit threshold for energy fluctuations can be determined using the following formula (6): (6) in, It is the lower limit threshold of energy fluctuation. It is the second spectrum energy. It is a preset sensitivity control coefficient.

[0148] The upper limit threshold for energy fluctuations can be determined using the following formula (7): (7) in, It is the upper limit threshold for energy fluctuations.

[0149] like If so, it is determined that there is an abnormality in the frequency component of the audio spectrum signal, which may be caused by audio decoding interruption, channel mute, or packet loss in the hardware decoder; if If the target frequency band is located in the high-frequency quiet zone (such as 16 kHz–22 kHz), it is determined that there is an abnormal energy rise in the audio spectrum signal, which may be caused by internal errors, driver crashes or post-processing distortion due to the performance degradation of the hardware decoder.

[0150] In some embodiments, determining the audio analysis result based on the audio analysis sub-results of each target frequency band includes: if the audio analysis sub-results of each target frequency band are all normal, then the audio analysis result is determined to be spectrally normal; if the audio analysis sub-results of at least one target frequency band are abnormal, then the audio analysis result is determined to be spectrally abnormal.

[0151] After obtaining the audio analysis sub-results for each target frequency band, if the audio analysis sub-result for any target frequency band is spectrally abnormal (including abnormal frequency component loss or abnormal energy rise), the overall audio analysis result of the audio spectrum signal is determined to be spectrally abnormal; if the audio analysis sub-results for each target frequency band are all spectrally normal, the overall audio analysis result of the audio spectrum signal is determined to be spectrally normal.

[0152] The audio analysis results include either abnormal audio spectrum or normal audio spectrum. Abnormal audio spectrum refers to a significant deviation of the energy distribution of the audio output signal within at least one target frequency band from the normal reference state, which may be caused by factors such as GPU hardware decoding failure, driver anomalies, audio post-processing errors, or system resource contention. Normal audio spectrum means that the energy distribution within each target frequency band is within the expected reasonable range, indicating that the GPU audio hardware decoding unit is working stably and without significant quality degradation.

[0153] In other implementations, the location of the frequency band to which the audio spectrum anomaly belongs can be combined to generate more detailed conclusions. For example, if there is a missing frequency component in the mid-frequency band while the low-frequency band is normal, it can be inferred that the human voice channel is abnormal; if there is an energy boost in the high-frequency band while the mid- and low-frequency bands are normal, it can be preliminarily determined to be output noise of the hardware decoder or distortion of the driver layer.

[0154] It is understood that the above method of dynamically setting thresholds and distinguishing anomaly types based on sensitivity control coefficients is merely an illustrative example. In other embodiments, fixed thresholds, adaptive percentile thresholds, or anomaly classification mechanisms based on machine learning models may also be used, and no limitation is made here.

[0155] In other embodiments, logarithmic energy difference, normalization error, confidence interval test, etc. can also be used to determine whether the spectrum is abnormal, and this is not limited here.

[0156] In some implementations, the location, ratio deviation, and duration of abnormal frequency bands can also be recorded to generate fine-grained diagnostic information and assist in subsequent fault root cause localization.

[0157] In other embodiments, audio analysis results can be generated based on multi-band joint judgment. For example, when the mid-frequency energy drops significantly while the low-frequency energy is normal, it may indicate an abnormality in the human voice channel; when the energy of all frequency bands decays synchronously, it may indicate overall audio mute or complete failure of the hardware decoder; when only the high-frequency energy rises abnormally, it may be due to a driving error or audio post-processing distortion.

[0158] It is understood that the above comparison method is not limited to the energy ratio form. In other embodiments, Euclidean distance, KL divergence, or similarity scoring based on machine learning models can also be used to quantify the difference between the target spectral energy and the reference spectral energy. As long as it can ultimately generate audio analysis results that can be used to determine whether the audio output is abnormal, it falls within the protection scope of this solution.

[0159] Furthermore, in some embodiments, based on audio analysis results and video status data, fault location is performed on the GPU hardware decoding process to determine the GPU hardware decoding fault point, including: if the audio analysis result shows an abnormal spectrum, it is determined that there is a fault in the GPU hardware decoding process; if the audio analysis result shows a normal spectrum and the video status data indicates that the video output signal is stuttering, it is determined that the GPU hardware decoding process is normal and that there is a fault in the video processing link.

[0160] The GPU hardware decoding process refers to the complete processing flow of synchronously decoding audio and video compressed streams using the GPU's internal hardware decoding units (such as NVDEC, VCN, or Intel Quick Sync Video), which can cover audio decoding, video decoding, frame scheduling, audio-visual synchronization, and output.

[0161] In some embodiments, when the audio analysis results indicate a spectral anomaly (including frequency component loss anomalies or energy boost anomalies), it can be attributed to a fault in the GPU hardware decoding process itself. It is understood that spectral anomalies can reflect overall hardware decoder malfunctions or driver layer errors, constituting a core hardware decoding link failure.

[0162] In other embodiments, when the audio analysis results indicate that the spectrum is normal, it means that the audio hardware decoding output is stable and the spectrum energy distribution is in line with expectations, indicating that the GPU audio decoding unit and the shared hardware decoding infrastructure are working normally. At this time, if the video status data shows significant jitter in the inter-frame time interval, an increase in the number of consecutive dropped frames, or periodic interruptions in the GPU video decoding engine utilization rate, it is determined that the GPU hardware decoding process is generally normal, but there is an independent fault in the video processing link.

[0163] The video processing link may include a video hardware decoding post-processing module, video memory frame buffer management, graphics rendering submission path or display compositor interface. Its failure may be caused by video memory bandwidth bottleneck, rendering API call blocking, driver frame scheduling logic defects or system-level resource contention.

[0164] For example, if the audio analysis results indicate that the spectrum is normal, and the video frame PTS in the video status data shows a significant jump or an increase in the number of dropped frames, it can be determined that the audio and video stuttering is caused by the video processing link, and the possibility of a failure in the GPU audio hardware decoding unit can be ruled out.

[0165] Furthermore, in some embodiments, the hardware decoding function is implemented by a hardware structure including a video decoding unit and an audio decoding unit; if the audio analysis result is an abnormal spectrum, it is determined that there is a fault in the GPU hardware decoding process, including: if the audio analysis result is an abnormal spectrum and the video status data indicates that the video decoding is in an unstable state, it is determined that there is a fault in the video decoding unit in the GPU hardware decoding process; if the audio analysis result is an abnormal spectrum and the video status data indicates that the video decoding is in a stable state, it is determined that there is a fault in the audio decoding unit in the GPU hardware decoding process.

[0166] The video decoding unit refers to the hardware module within the GPU used to process video compressed streams (such as H.264, H.265, AV1, etc.), responsible for operations such as frame parsing, motion compensation, inverse transform, and pixel reconstruction. The audio decoding unit refers to the hardware module within the GPU used to process audio compressed streams, capable of decoding audio encoding formats such as AAC, AC-3, and DTS. Physically, they can be integrated on the same chip, logically sharing the driver control interface, command scheduling queue, and some memory resources, but they are relatively independent in their functional paths.

[0167] In some embodiments, an unstable video decoding state refers to video status data displaying at least one of the following characteristics: consecutive frame drop counts exceeding a threshold, variance of the time interval between video decoding timestamps in the DTS sequence exceeding a preset threshold, periodic sharp drops in GPU video decoding engine utilization or the occurrence of idle periods, or the driver reporting a video hardware decoding error flag. Such phenomena indicate that the video decoding process is interrupted, delayed, or experiencing abnormal resource scheduling.

[0168] If the audio analysis results indicate a spectrum anomaly (such as missing frequency components or increased energy), and the video status data shows unstable video decoding, then it is determined that there is a fault in the video decoding unit during the GPU hardware decoding process, and the possibility of a fault in the GPU audio decoding unit is ruled out.

[0169] Understandably, although the audio anomaly was observed, the video decoding was also abnormal, indicating that the problem originated from an anomaly in the GPU hardware decoder's common control logic, firmware scheduler, power management unit, memory access channel, or global state of the driver layer. As a high-load module, the video decoding unit is more likely to expose such common faults, so the problem was attributed to a fault in the video decoding unit.

[0170] A stable video decoding state means that the variance of the video decoding timestamp time interval displayed by the video status data is less than or equal to a preset threshold, the frame drop count is zero or extremely low, the GPU video decoding engine utilization is stable, and there are no scheduling interruptions or error reports. If, at this point, the audio analysis results still indicate a spectrum anomaly, it means that the video hardware decoding link is operating normally, but the audio output anomaly cannot be explained by the video side. Therefore, it is determined that there is a fault in the audio decoding unit during the GPU hardware decoding process, which may be caused by audio decoder firmware defects, incorrect channel configuration, sampling rate mismatch, DMA transfer interruption, or audio post-processing module malfunction.

[0171] For example, when the audio analysis results indicate an abnormal spectrum (such as missing frequency components or increased energy), further correlation analysis is performed on the decoding timestamp sequence of video frames in the video status data. The variance between the decoding timestamps of consecutive video frames is calculated. If the variance is extremely large, it indicates that the video decoding is unstable, leading to audio problems, and the fault is located in the video decoding unit. If the variance is small, it indicates that the video decoding time is stable but the audio spectrum is abnormal, and the fault is located in the audio decoding unit itself.

[0172] In other embodiments, further verification can be achieved by combining the GPU's internal performance counters. For example, if the audio hardware decoding dedicated register reports a decoding timeout or bitstream verification failure when the audio is abnormal but the video is stable, the conclusion of audio decoding unit failure is strengthened; if the video is abnormal and accompanied by memory bandwidth saturation or command buffer overflow, the determination of video decoding unit or shared resource bottleneck is supported.

[0173] In some implementations, the timing consistency of the anomaly can be further used for verification. For example, if the audio spectrum anomaly and the video stuttering occur strictly synchronously, it is still likely to be judged as a common fault of the GPU hardware decoding; if the two occur asynchronously (such as only the video stuttering while the audio remains stable), the conclusion that the video link is an independent fault is strengthened.

[0174] In other embodiments, auxiliary criteria can be introduced to improve positioning accuracy. For example, when the audio is normal but the video is choppy, check the overall GPU temperature, power consumption, or memory usage: if there are no signs of overload, thermal throttling or resource exhaustion can be ruled out, further supporting the judgment of partial faults in the video processing link.

[0175] It is understood that the above binary joint criterion based on audio analysis results and video state data is only an exemplary implementation. In other implementations, it can also be extended to a multi-level decision tree or introduce a confidence weighting mechanism, which is not limited here.

[0176] In one embodiment, such as Figure 5The diagram shows a system framework for GPU hardware decoding fault detection. This framework includes an input layer, an analysis layer, a decision layer, and an output layer. These layers collaborate through data flow to complete GPU hardware decoding fault detection.

[0177] The input layer supports various audio / high-definition video file inputs, including local players (such as MPV, VLC, Windows Media Player), browser environments (such as Chrome, Edge, Safari), and mobile terminal applications (such as iOS or Android Apps). The audio and video streams are all connected to the system through a unified interface.

[0178] In the analysis layer, responding to driver commands and data streams, the GPU hardware decoding engine and driver process the data, outputting hardware-decoded audio and video signals. The audio decoded signal is input to the audio rendering device, where the final node in the audio rendering process is acquired. This audio output signal is captured by the audio stream acquisition module and sent to the audio spectrum analysis module for FFT transformation. The energy of each target frequency band is calculated, and anomaly comparison is performed based on a baseline model to obtain the audio analysis results. Simultaneously, the video decoded signal is input to the video rendering device, extracting the DTS, PTS, and frame drop counts of the current video frame, and adding high-precision timestamps to these metrics. The video stream acquisition module calculates the instantaneous occupancy rate of the GPU video decoding engine associated with the PTS and adds a high-precision timestamp to this instantaneous occupancy rate. Therefore, the timestamped DTS, PTS, and frame drop counts, along with the timestamped instantaneous occupancy rate, constitute timestamped video state data.

[0179] At the decision-making level, timestamped video status data and audio analysis results are input into the root cause comprehensive decision module. The root cause comprehensive decision module performs audio-video status correlation analysis to locate faults in the GPU hardware decoding process and determine the fault points in the GPU hardware decoding.

[0180] At the output layer, the results output module can output diagnostic conclusions in the form of structured data (such as JSON or XML format) or visual analysis reports for use by maintenance personnel or automated systems, enabling rapid identification and response to faults.

[0181] In some embodiments, such as Figure 6 As shown, a GPU hardware decoding fault detection method is also provided, including the following steps: Start the audio spectrum analysis task to begin anomaly detection of the audio signal output by the GPU hardware decoding; play the reference audio signal, collect 500 frames of spectrum data in a loop, calculate the average value and standard deviation of each frequency point, and build a baseline model Baseline(k) as the spectral energy benchmark for normal audio signals.

[0182] Entering the continuous anomaly detection phase, the following operations are performed in a loop: acquire the current one or more frames of audio output signal, apply Hanning window for windowing processing to reduce spectral leakage; perform N-point FFT transformation to obtain the real-time audio spectrum signal S_real(k), and perform anomaly feature detection on the audio spectrum signal S_real(k).

[0183] During the anomaly detection process, the audio spectrum signal S_real(k) is compared with the baseline model Baseline(k), and the energy ratio of the target frequency band R = E_real / E_baseline is calculated, where E_real is the first spectral energy of the current target frequency band, and E_baseline is the second spectral energy of the current target frequency band.

[0184] Determine if the energy ratio R is less than the lower limit threshold T_missing_low for energy fluctuation. If so, it is determined that the target frequency band has an abnormality of missing frequency components. If not, continue to determine if R is greater than the upper limit threshold T_missing_high for energy fluctuation. If R is greater than the upper limit threshold T_missing_high for energy fluctuation, it is determined that the target frequency band has an abnormality of energy rise. If R is greater than or equal to the lower limit threshold T_missing_low for energy fluctuation and less than or equal to the upper limit threshold T_missing_high for energy fluctuation, it is determined that the spectrum of the target frequency band is normal.

[0185] Anomaly reports are generated from the anomaly detection results of each target frequency band, and the reports are stored in the analysis queue. Then, the next frame or multiple frames of audio output signals are retrieved, and GPU hardware decoding fault detection is performed on the next frame or multiple frames of audio output signals.

[0186] It is understandable that the above embodiments can automatically and accurately locate the root cause of stuttering, distortion and other fault phenomena to the underlying cause of unstable video decoding, audio decoding failure or system resource bottleneck by performing high-precision synchronous signal acquisition at the audio rendering and video driving layers and combining audio spectrum modeling and multi-dimensional state correlation analysis. It can not only directly, in real time and automatically detect the GPU hardware decoding output quality, but also accurately distinguish whether the abnormal audio and video playback is caused by the GPU hardware decoding itself or by external factors.

[0187] It should be noted that although the steps of the method in this disclosure are described in a specific order in the accompanying drawings, this does not require or imply that the steps must be performed in that specific order, or that all the steps shown must be performed to achieve the desired result. Additional or alternative steps may be omitted, multiple steps may be combined into one step, and / or a step may be broken down into multiple steps.

[0188] Furthermore, this example embodiment also provides a GPU hardware decoding fault detection device. (Refer to...) Figure 7 As shown, the GPU hardware decoding fault detection device 700 includes: an audio / video signal acquisition module 710, an audio spectrum analysis module 720, a video status data determination module 730, and a fault location module 740. Wherein: The audio and video signal acquisition module 710 is used to acquire the audio output signal of audio and video when the audio and video use the hardware decoding function of the graphics processor GPU. The audio spectrum analysis module 720 is used to perform audio anomaly analysis on the spectrum of the audio output signal and obtain audio analysis results. The video status data determination module 730 is used to determine the video status data corresponding to the video frame in the audio and video. The video status data characterizes the video processing stability of the video frame during the GPU hardware decoding process. The fault location module 740 is used to locate faults in the GPU hardware decoding process based on audio analysis results and video status data, and to determine the fault point of the GPU hardware decoding.

[0189] In some example embodiments of this disclosure, based on the foregoing scheme, the audio spectrum analysis module 720 is configured as follows: Determine the target spectral energy of the audio output signal based on its spectrum. The target spectral energy is compared with the preset reference spectral energy to obtain the audio analysis results.

[0190] In some example embodiments of this disclosure, based on the foregoing scheme, the target spectral energy is characterized by the first spectral energy within the target frequency band; the aforementioned audio spectrum analysis module 720 is configured as follows: For at least one target frequency band within the spectral range of the audio output signal, obtain the first spectral energy within the target frequency band; The second spectral energy of the reference audio signal within the target frequency band is obtained, and the first and second spectral energies corresponding to at least one target frequency band are compared to obtain the audio analysis results.

[0191] In some example embodiments of this disclosure, based on the foregoing scheme, the audio spectrum analysis module 720 is configured as follows: For at least one target frequency band, determine the ratio between the first spectral energy and the second spectral energy; If the ratio is less than the lower limit threshold of energy fluctuation or greater than the upper limit threshold of energy fluctuation, the audio analysis sub-result of the target frequency band is determined to be spectral abnormal. If the ratio is greater than or equal to the lower limit threshold of energy fluctuation and less than or equal to the upper limit threshold of energy fluctuation, then the audio analysis sub-result of the target frequency band is determined to be normal. Based on the audio analysis sub-results of each target frequency band, the audio analysis results are determined.

[0192] In some example embodiments of this disclosure, based on the foregoing scheme, the spectral anomaly includes one of frequency component loss anomaly and energy rise anomaly; the aforementioned audio spectrum analysis module 720 is configured as follows: If the ratio is less than the lower limit threshold of energy fluctuation, the audio analysis sub-result of the target frequency band is determined to be an abnormal frequency component missing value. If the ratio is greater than the upper limit threshold of energy fluctuation, the audio analysis sub-result of the target frequency band is determined to be an energy rise anomaly.

[0193] In some example embodiments of this disclosure, based on the foregoing scheme, the audio spectrum analysis module 720 is configured as follows: If the audio analysis sub-results for each target frequency band are all normal, then the audio analysis results are determined to be normal in terms of spectrum. If at least one audio analysis sub-result for a target frequency band is a frequency band anomaly, then the audio analysis result is determined to be a spectral anomaly.

[0194] In some example embodiments of this disclosure, based on the foregoing scheme, the fault location module 740 is configured as follows: If the audio analysis results show an abnormal spectrum, it indicates a fault in the GPU hardware decoding process. If the audio analysis results show a normal spectrum and the video status data indicates that the video output signal is choppy, then the GPU hardware decoding process is considered normal, and the video processing link is considered to have a fault.

[0195] In some example embodiments of this disclosure, based on the foregoing scheme, the hardware decoding function is implemented by a hardware structure including a video decoding unit and an audio decoding unit; the fault location module 740 is configured as follows: If the audio analysis results show an abnormal spectrum and the video status data indicates that the video decoding is in an unstable state, then it is determined that there is a fault in the video decoding unit during the GPU hardware decoding process. If the audio analysis results show an abnormal spectrum and the video status data indicates that the video decoding is in a stable state, then it is determined that there is a fault in the audio decoding unit during the GPU hardware decoding process.

[0196] In some example embodiments of this disclosure, based on the foregoing scheme, the audio spectrum analysis module 720 is configured as follows: Invoke a pre-built baseline model and obtain its frequency domain statistical properties; the baseline model is built based on a fault-free reference audio signal. The reference spectrum energy is determined based on frequency domain statistical characteristics.

[0197] In some example embodiments of this disclosure, based on the foregoing scheme, the audio and video signal acquisition module 710 is configured as follows: The audio output signal of the audio and video is acquired at the final node of the audio rendering process; the audio output signal is a pulse code modulation audio signal.

[0198] In some example embodiments of this disclosure, based on the foregoing scheme, the video state data determination module 730 is configured as follows: By intercepting the application's calls to the graphics rendering interface, the context information of at least one video frame in the audio and video can be extracted. Obtain the hardware decoding status of the GPU associated with at least one video frame, and determine the video status data corresponding to the video frame based on the hardware decoding status associated with at least one video frame and context information.

[0199] The specific details of each module of the GPU hardware decoding fault detection device have been described in detail in the corresponding GPU hardware decoding fault detection method, so they will not be repeated here.

[0200] It should be noted that although several modules or units of the GPU hardware decoding fault detection device have been mentioned in the detailed description above, this division is not mandatory. In fact, according to embodiments of this disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0201] Furthermore, in an exemplary embodiment of this disclosure, an electronic device capable of implementing the above-described GPU hardware decoding fault detection method is also provided.

[0202] Those skilled in the art will understand that various aspects of this disclosure can be implemented as a system, method, or program product. Therefore, various aspects of this disclosure can be embodied in the following forms: a completely hardware embodiment, a completely software embodiment (including firmware, microcode, etc.), or an embodiment combining hardware and software aspects, collectively referred to herein as a "circuit," "module," or "system."

[0203] The following reference Figure 8To describe an electronic device 800 according to such an embodiment of the present disclosure. Figure 8 The electronic device 800 shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments disclosed herein.

[0204] like Figure 8 As shown, the electronic device 800 is presented in the form of a general-purpose computing device. The components of the electronic device 800 may include, but are not limited to: at least one processing unit 810, at least one storage unit 820, a bus 830 connecting different system components (including storage unit 820 and processing unit 810), and a display unit 840.

[0205] The storage unit stores program code that can be executed by the processing unit 810, causing the processing unit 810 to perform the steps described in the "Exemplary Methods" section of this specification according to various exemplary embodiments of this disclosure. For example, the processing unit 810 can perform actions such as... Figure 1 In step S110, when the audio and video are decoded using the hardware decoding function of the graphics processing unit (GPU), the audio output signal of the audio and video is acquired; in step S120, the audio output signal spectrum is analyzed for audio anomalies to obtain audio analysis results; in step S130, the video status data corresponding to the video frame in the audio and video is determined, and the video status data characterizes the video processing stability of the video frame during the GPU hardware decoding process; in step S140, based on the audio analysis results and the video status data, the GPU hardware decoding process is fault located to determine the GPU hardware decoding fault point.

[0206] Storage unit 820 may include readable media in the form of volatile storage units, such as random access memory (RAM) 821 and / or cache memory 822, and may further include read-only memory (ROM) 823.

[0207] The storage unit 820 may also include a program / utility 824 having a set (at least one) of program modules 825, including but not limited to: an operating system, one or more application programs, other program modules, and program data, each or some combination of these examples may include an implementation of a network environment.

[0208] Bus 830 can represent one or more of several types of bus structures, including a memory cell bus or memory cell controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of the various bus structures.

[0209] Electronic device 800 can also communicate with one or more external devices 870 (e.g., keyboard, pointing device, Bluetooth device, etc.), and with one or more devices that enable a user to interact with electronic device 800, and / or with any device that enables electronic device 800 to communicate with one or more other computing devices (e.g., router, modem, etc.). This communication can be performed via input / output (I / O) interface 850. Furthermore, electronic device 800 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 860. As shown, network adapter 860 communicates with other modules of electronic device 800 via bus 830. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 800, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0210] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, terminal device, or network device, etc.) to execute the methods according to the embodiments of this disclosure.

[0211] In exemplary embodiments of this disclosure, a computer-readable storage medium is also provided, on which a program product capable of implementing the methods described above is stored. In some possible embodiments, various aspects of this disclosure may also be implemented as a program product including program code that, when the program product is run on a terminal device, causes the terminal device to perform the steps described in the "Exemplary Methods" section of this specification according to various exemplary embodiments of this disclosure.

[0212] refer to Figure 9 As shown, a program product 900 for implementing the GPU hardware decoding fault detection method described in embodiments of the present disclosure is illustrated. This product may be a portable compact disc read-only memory (CD-ROM) and includes program code, and may run on a terminal device, such as a personal computer. However, the program product of this disclosure is not limited thereto. In this document, a readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system, apparatus, or device.

[0213] The program product may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0214] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium, capable of sending, propagating, or transmitting programs for use by or in conjunction with an instruction execution system, apparatus, or device.

[0215] The program code contained on the readable medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0216] Program code for performing the operations of this disclosure can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0217] Furthermore, the above figures are merely illustrative of the processes included in the method according to exemplary embodiments of this disclosure and are not intended to be limiting. It is readily understood that the processes shown in the above figures do not indicate or limit the temporal order of these processes. Additionally, it is readily understood that these processes may be executed synchronously or asynchronously, for example, in multiple modules.

[0218] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, touch terminal, or network device, etc.) to execute the methods according to the embodiments of this disclosure.

[0219] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and embodiments are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the claims.

[0220] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A method for detecting GPU hardware decoding faults, characterized in that, The method includes: When the audio and video use the hardware decoding function of the graphics processing unit (GPU), the audio output signal of the audio and video is acquired; Audio anomaly analysis is performed on the spectrum of the audio output signal to obtain audio analysis results; Determine the video state data corresponding to the video frame in the audio and video, and the video state data is used to characterize the video processing stability of the video frame during the GPU hardware decoding process; Based on the audio analysis results and the video status data, fault location is performed on the GPU hardware decoding process to determine the GPU hardware decoding fault point.

2. The GPU hardware decoding fault detection method according to claim 1, characterized in that, The audio anomaly analysis of the spectrum of the audio output signal, to obtain the audio analysis results, includes: Based on the spectrum of the audio output signal, determine the target spectral energy of the spectrum; The target spectral energy is compared with the preset reference spectral energy to obtain the audio analysis result.

3. The GPU hardware decoding fault detection method according to claim 2, characterized in that, The target spectral energy is characterized by a first spectral energy within the target frequency band; determining the target spectral energy of the spectrum based on the spectrum of the audio output signal includes: For at least one target frequency band within the spectral range of the audio output signal, obtain the first spectral energy within the target frequency band; The step of comparing the target spectral energy with a preset reference spectral energy to obtain the audio analysis result includes: The second spectral energy of the reference audio signal within the target frequency band is obtained, and the first spectral energy and the second spectral energy corresponding to at least one of the target frequency bands are compared to obtain the audio analysis result.

4. The GPU hardware decoding fault detection method according to claim 3, characterized in that, The step of comparing the first spectral energy and the second spectral energy corresponding to at least one of the target frequency bands to obtain the audio analysis result includes: For at least one of the target frequency bands, determine the ratio between the first spectral energy and the second spectral energy; If the ratio is less than the lower limit threshold of energy fluctuation or greater than the upper limit threshold of energy fluctuation, the audio analysis sub-result of the target frequency band is determined to be spectral abnormal. If the ratio is greater than or equal to the lower limit threshold of energy fluctuation and less than or equal to the upper limit threshold of energy fluctuation, then the audio analysis sub-result of the target frequency band is determined to be normal in terms of spectrum. The audio analysis results are determined based on the audio analysis sub-results of each target frequency band.

5. The GPU hardware decoding fault detection method according to claim 4, characterized in that, The spectral anomaly includes one of the following: frequency component missing anomaly and energy rise anomaly. If the ratio is less than the lower threshold of energy fluctuation or greater than the upper threshold of energy fluctuation, the audio analysis sub-result of the target frequency band is determined to be spectrally abnormal, including: If the ratio is less than the lower limit threshold of energy fluctuation, the audio analysis sub-result of the target frequency band is determined to be an abnormal frequency component missing value. If the ratio is greater than the upper limit threshold of energy fluctuation, the audio analysis sub-result of the target frequency band is determined to be an abnormal energy rise.

6. The GPU hardware decoding fault detection method according to claim 4, characterized in that, The step of determining the audio analysis result based on the audio analysis sub-results of each target frequency band includes: If the audio analysis sub-results for each target frequency band are all normal, then the audio analysis results are determined to be normal in terms of spectrum. If at least one audio analysis sub-result of the target frequency band is a frequency band anomaly, then the audio analysis result is determined to be a spectrum anomaly.

7. The GPU hardware decoding fault detection method according to claim 4, characterized in that, The step of locating faults in the GPU hardware decoding process based on the audio analysis results and the video status data, and determining the GPU hardware decoding fault points, includes: If the audio analysis result shows an abnormal spectrum, then the GPU hardware decoding process is determined to be faulty. If the audio analysis result shows a normal spectrum and the video status data indicates that the video output signal is choppy, then the GPU hardware decoding process is determined to be normal, and the video processing link is determined to be faulty.

8. The GPU hardware decoding fault detection method according to claim 7, characterized in that, The hardware decoding function is implemented by a hardware structure including a video decoding unit and an audio decoding unit; if the audio analysis result is an abnormal spectrum, the GPU hardware decoding process is determined to be faulty, including: If the audio analysis result indicates an abnormal spectrum and the video status data indicates that the video decoding is in an unstable state, then it is determined that the video decoding unit in the GPU hardware decoding process is faulty. If the audio analysis result shows an abnormal spectrum and the video status data indicates that the video decoding is in a stable state, then it is determined that the audio decoding unit in the GPU hardware decoding process is faulty.

9. The GPU hardware decoding fault detection method according to claim 2, characterized in that, The method for determining the energy of the reference spectrum includes: A pre-built baseline model is invoked, and its frequency domain statistical characteristics are obtained; the baseline model is constructed based on a fault-free reference audio signal. Based on the frequency domain statistical characteristics, the reference spectral energy is determined.

10. The GPU hardware decoding fault detection method according to any one of claims 1 to 9, characterized in that, The acquisition of the audio output signal of the audio and video includes: The audio output signal of the audio and video is acquired at the final node of the audio rendering process; the audio output signal is a pulse code modulation format audio signal.

11. The GPU hardware decoding fault detection method according to any one of claims 1 to 9, characterized in that, Determining the video state data corresponding to the video frame in the audio and video includes: By intercepting the application's calls to the graphics rendering interface, the context information of at least one video frame in the audio and video is extracted; Obtain the hardware decoding status of the GPU associated with at least one of the video frames, and determine the video status data corresponding to the video frame based on the hardware decoding status associated with at least one of the video frames and the context information.

12. A GPU hardware decoding fault detection device, characterized in that, include: The audio and video signal acquisition module is used to acquire the audio output signal of the audio and video when the audio and video use the hardware decoding function of the graphics processor (GPU). The audio spectrum analysis module is used to perform audio anomaly analysis on the spectrum of the audio output signal and obtain audio analysis results; The video status data determination module is used to determine the video status data corresponding to the video frame in the audio and video, wherein the video status data characterizes the video processing stability of the video frame during the GPU hardware decoding process. The fault location module is used to locate faults in the GPU hardware decoding process based on the audio analysis results and the video status data, and to determine the fault points in the GPU hardware decoding.

13. An electronic device, characterized in that, include: processor; as well as A memory storing computer-readable instructions that, when executed by the processor, implement the GPU hardware decoding fault detection method as described in any one of claims 1 to 11.

14. A computer-readable storage medium, characterized in that, It stores a computer program that, when executed by a processor, implements the GPU hardware decoding fault detection method as described in any one of claims 1 to 11.

Images