Audio and video processing system and method supporting AI intelligent repair technology

Abstract

The application discloses an audio and video processing system and method supporting AI intelligent repair technology, and relates to the technical field of audio and video data processing.The application comprises the following steps: scene recognition based on the playing source type, user interaction mode, audio and video code rate, audio sampling rate and network transmission delay to generate scene labels of live broadcast, movie watching and singing; according to the scene labels, the upper limit of the delay, the resolution target, the detail restoration level, the audio fidelity index and the beat alignment threshold are determined to form a performance constraint vector. The application aims at the differentiated needs of live broadcast, movie watching and singing, and constructs a closed-loop mechanism of scene recognition, performance constraint setting, repair link arrangement, cloud and terminal task allocation, real-time adaptive adjustment and parameter iterative optimization, realizes low delay, high quality, high sound quality and power self-adaption, and improves the system stability and continuous optimization ability through versioning and phased release.

Description

Technical Field

[0001] This invention relates to the field of audio and video data processing technology, and more specifically to an audio and video processing system and method that supports AI intelligent repair technology. Background Technology

[0002] AI-powered intelligent repair technology in audio and video processing refers to a comprehensive approach that utilizes artificial intelligence to repair and optimize the quality of audio and video content within a digital, networked, and intelligent home entertainment system. Its core idea is to automatically analyze collected or stored audio and video data using a deep learning model within a unified Y·OS entertainment system architecture, combining cloud and terminal collaborative operation mechanisms. This identifies quality defects such as noise, distortion, blurriness, stuttering, color shift, and audio-visual asynchrony. Based on the type and severity of the defects, targeted intelligent repair strategies are implemented, such as detail texture reconstruction, inter-frame loss compensation, color restoration, audio waveform reshaping, and synchronization correction, thereby significantly improving playback picture and sound quality. Furthermore, this processing is seamlessly adaptable across different hardware models and continuously optimized using big data and IoT technologies, enabling users to obtain a higher fidelity and more immersive audio-visual experience in diverse entertainment scenarios such as singing, watching, listening, playing games, and live streaming.

[0003] The existing technology has the following shortcomings:

[0004] Existing AI-based audio and video restoration methods mostly employ uniform, generalized restoration strategies, failing to establish differentiated processing mechanisms for different types of entertainment scenarios. For example, in live streaming scenarios, audio and video restoration must prioritize low latency and real-time synchronization, but the complex computational processes in general strategies may cause latency accumulation, affecting the continuity of the live stream. In movie-watching scenarios, users value high resolution and detail restoration, but general strategies do not deeply optimize image reconstruction algorithms, resulting in insufficient detail representation. In singing scenarios, audio restoration should focus on maintaining high fidelity and accompaniment synchronization, but general strategies lack sufficient precision in audio waveform reconstruction and beat alignment, easily producing slight audio-visual misalignment or sound quality loss. Due to the lack of scenario-specific differentiated optimization, the restoration effects of existing methods are unstable in multi-scenario entertainment applications, making it difficult to simultaneously meet the performance requirements and user experience needs of different scenarios.

[0005] The information disclosed in the background section is only intended to enhance the understanding of the background of this disclosure, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0006] The purpose of this invention is to provide an audio and video processing system and method that supports AI intelligent repair technology, so as to solve the problems mentioned in the background art.

[0007] To achieve the above objectives, the present invention provides the following technical solution: an audio and video processing method supporting AI intelligent repair technology, comprising the following steps:

[0008] Scene recognition is performed based on playback source type, user interaction pattern, audio and video bitrate, audio sampling rate, and network transmission latency to generate scene tags for live streaming, movie watching, and singing.

[0009] Based on scene labels, determine the latency limit, resolution target, detail restoration level, audio fidelity index, and beat alignment threshold to form a performance constraint vector;

[0010] The audio and video restoration link is arranged into nodes according to the performance constraint vector, and the noise reduction, super-resolution reconstruction, frame interpolation compensation, color restoration, audio waveform reconstruction and audio-visual synchronization correction models and parameters are configured.

[0011] Based on the computational load of the repair link and the computing power of the terminal, the entertainment system architecture divides the tasks into cloud and terminal execution, and distributes the plugins and parameters from the cloud to the terminal for loading.

[0012] During task execution, the link parameters are dynamically adjusted and adaptively corrected based on real-time collected playback latency, stuttering rate, picture quality score, audio distortion rate, and audio-visual synchronization error.

[0013] The adjusted parameters are stored in the scene feature profile and version library, and then deployed to the terminal application according to the phased release strategy to achieve continuous iterative optimization of the scene-based audio and video restoration baseline.

[0014] Preferably, the specific steps for scene recognition based on playback source type, user interaction pattern, audio / video bitrate, audio sampling rate, and network transmission latency are as follows:

[0015] The playback source is parsed by reading the media file header information or streaming media session initialization data, parsing the media encapsulation format identifier, transmission protocol type and media metadata, and determining whether it is a live broadcast type, movie viewing type or singing type playback source;

[0016] Acquire user interaction patterns during playback, record operation commands, number of times, interval time and content, and analyze behavioral patterns corresponding to the scenario;

[0017] During the acquisition and playback process, the video encoding method, frame rate, resolution, bit rate, audio encoding method, sampling rate, number of channels, network first frame delay time, segment first delay time, buffer duration, and end-to-end transmission latency are recorded.

[0018] The parsing results, interaction pattern data, audio and video technical parameters, and network transmission characteristics are input into the scene determination rule set to generate live streaming scene tags, movie viewing scene tags, or singing scene tags and then pass them to the subsequent audio and video processing flow.

[0019] Preferably, the matching conditions of the scene determination rule set include: generating a live streaming scene tag when the playback source type is a real-time stream, the user interaction frequency is higher than five times per minute, and the end-to-end transmission latency is less than two seconds; generating a movie viewing scene tag when the playback source type is a local high-definition file, the video resolution is not less than 1920×1080 pixels, the video bitrate is not less than 10 megabits per second, and the user interaction frequency is less than twice per minute; and generating a singing scene tag when the playback source contains independent vocal tracks and accompaniment tracks, the audio sampling rate is not less than 48 kHz, the lyrics timestamp corresponds to the playback audio, and the user triggers lyrics synchronization and pitch adjustment multiple times.

[0020] Preferably, the specific steps for forming a performance constraint vector by quantizing the latency upper limit, resolution target, detail restoration level, audio fidelity index, and beat alignment threshold based on scene recognition labels are as follows:

[0021] Read the initial performance target values ​​corresponding to the live streaming scene tag, movie viewing scene tag, and singing scene tag from the preset scene baseline parameter table;

[0022] Collect 10 to 30 seconds of runtime data, including video bitrate, frame rate, frame rate fluctuation range, keyframe interval, audio sampling rate, audio peak level, noise floor level, network round-trip latency, latency jitter, packet loss ratio, first frame rendering time, and segment start waiting time.

[0023] The baseline parameters are compared with the acquired data, and the performance target values ​​are adjusted according to the available processing margin to determine the latency limit, resolution target, detail restoration level, audio fidelity index and beat alignment threshold.

[0024] All adjusted performance target values ​​are written into the performance constraint vector and a consistency check is performed. Once the check passes, the values ​​are submitted to the audio and video repair link as input for repair strategy selection and parameter configuration.

[0025] Preferably, the specific steps for arranging the audio and video restoration link into nodes according to the performance constraint vector are as follows:

[0026] Set the denoising processing node as the starting node, and perform denoising processing on video and audio based on the signal-to-noise ratio and total harmonic distortion plus noise limit of the performance constraint vector;

[0027] Configure super-resolution reconstruction nodes and frame interpolation compensation nodes in sequence to improve video resolution and frame rate while maintaining temporal continuity;

[0028] Configure the color restoration node and audio waveform reconstruction node to perform color space conversion and audio sampling rate and phase difference correction;

[0029] Configure an audio-visual synchronization correction node to ensure that the audio-visual synchronization error does not exceed the performance constraint vector limit by adjusting the playback buffer queue depth, and output the repair result after all consistency checks pass.

[0030] Preferably, in the audio-visual synchronization correction node, the difference between the video frame display timestamp and the audio playback timestamp is compared with the beat alignment threshold in the performance constraint vector. If the difference exceeds the threshold range, synchronization correction is performed by precisely discarding or repeating audio sampling points and adjusting the video frame output time, so that the absolute value of the final audio-visual synchronization error does not exceed the limit value of the performance constraint vector.

[0031] Preferably, the specific steps for task partitioning based on the computational load of the repair link and the computing power of the terminal device are as follows:

[0032] The average processing time per unit frame, peak processing time, required memory capacity, video memory capacity, and number of floating-point operations for each link in the repair link are measured. The processor clock speed, number of cores, floating-point operation capability, acceleration unit capability, available memory, video memory capacity, power consumption limit, bandwidth, and round-trip latency of the terminal device are collected.

[0033] Based on the task division criteria, the processing steps are allocated to cloud execution or terminal execution, and the execution location of each step is fixedly configured according to different entertainment scenarios;

[0034] Generate a distribution list containing plugin file information, runtime requirements, and parameter files; distribute the distribution through an encrypted channel and perform signature and hash value verification.

[0035] Real-time loading and rollback control are performed before and during playback to ensure processing performance and playback stability, and the running data is recorded to a log file and uploaded to the cloud.

[0036] Preferably, the specific steps for dynamically adjusting and adaptively correcting the parameters of each processing node in the link based on real-time collected playback latency, stuttering rate, picture quality score, audio distortion rate, and audio-visual synchronization error during task execution are as follows:

[0037] Playback latency, stuttering rate, picture quality score, audio distortion rate, and audio-visual synchronization error are continuously collected according to a fixed sampling period and a rolling statistical window, and compared with the target values ​​corresponding to different entertainment scenarios.

[0038] The processing parameters for denoising, super-resolution reconstruction, frame interpolation compensation, color restoration, audio waveform reconstruction, and audio-visual synchronization correction are adjusted according to performance deviation judgment and parameter mapping rules, and step-wise and hysteresis control is adopted to avoid frequent oscillations.

[0039] After parameter adjustment, performance metrics are continuously monitored through two statistical windows. If performance deteriorates, the previous parameter set is rolled back. If performance is stable, the current parameter set is marked as a stable version and cached.

[0040] At the end of the session, the performance metric trajectory, parameter adjustment records, rollback records, stable version parameters, and corresponding performance constraint vectors are uploaded to the version repository for subsequent optimization.

[0041] Preferably, the specific steps for writing the dynamically adjusted parameters into the scene feature profile and version repository, and gradually applying them to different types of terminals according to a phased release strategy are as follows:

[0042] The dynamically adjusted and stably converged parameters for denoising, super-resolution reconstruction, frame interpolation compensation, color restoration, audio waveform reconstruction, and audio-visual synchronization correction, along with session performance indicators and device and content characteristics, are written into the scene feature profile as structured data.

[0043] The parameter set is versioned and stored, generating a unique version number and metadata list, and integrity verification and offline reproduction experiments are completed.

[0044] The application scope of the version will be gradually expanded in accordance with the phased release strategy, and real-time loading tests will be conducted on the terminal.

[0045] During operation, a rollback is triggered based on performance anomalies. After the phase ends, the operation logs, anomaly causes, rollback records, and final stable version parameters are written to the version repository, forming a closed-loop optimization mechanism.

[0046] An audio and video processing system supporting AI-powered intelligent repair technology includes a scene recognition module, a performance constraint generation module, a repair link orchestration module, a task allocation and loading module, a real-time adaptive adjustment module, and a parameter accumulation and iterative optimization module.

[0047] The scene recognition module identifies scenes based on playback source type, user interaction pattern, audio and video bitrate, audio sampling rate, and network transmission latency, and generates scene tags for live streaming, movie watching, and singing.

[0048] The performance constraint generation module determines the latency limit, resolution target, detail restoration level, audio fidelity index and beat alignment threshold based on scene labels, forming a performance constraint vector.

[0049] The repair link orchestration module arranges the audio and video repair links into nodes according to the performance constraint vector, and configures the noise reduction, super-resolution reconstruction, frame interpolation compensation, color restoration, audio waveform reconstruction and audio-visual synchronization correction models and parameters.

[0050] The task allocation and loading module, based on the computational load of the repair link and the computing power of the terminal, divides the execution tasks in the cloud and the terminal in the entertainment system architecture, and distributes the plugins and parameters from the cloud to the terminal for loading;

[0051] The real-time adaptive adjustment module dynamically adjusts and repairs link parameters and adaptively corrects them based on real-time collected playback latency, stuttering rate, picture quality score, audio distortion rate, and audio-visual synchronization error during task execution.

[0052] The parameter accumulation and iterative optimization module stores the adjusted parameters in the scene feature profile and version library, and releases them to the terminal application according to the phased release strategy, so as to realize the continuous iterative optimization of the scene-based audio and video restoration baseline.

[0053] The technical effects and advantages provided by the present invention in the above technical solution are as follows:

[0054] This invention addresses the differentiated performance requirements of various entertainment scenarios such as live streaming, movie watching, and karaoke. It constructs a closed-loop processing mechanism encompassing scene identification, performance constraint setting, refined orchestration of the repair process, cloud and terminal task allocation, real-time adaptive adjustment, and parameter accumulation and iterative optimization. This overcomes the bottleneck of existing generalized repair strategies that cannot simultaneously address performance and user experience across multiple scenarios. The solution effectively controls latency and maintains real-time audio-visual synchronization in live streaming, significantly improves resolution and detail reproduction in movie watching, and ensures high audio fidelity and precise alignment with accompaniment rhythm in karaoke. Furthermore, cloud distribution and instant loading enable self-adaptation of computing power across different hardware terminals, ensuring stable repair results for both high- and low-performance devices. Simultaneously, the versioned parameter storage and phased release mechanism allows for rapid dissemination of validated optimization results and continuous improvement of the repair baseline's stability and performance consistency through multiple iterations. This endows the system with long-term self-evolution capabilities, significantly enhancing the quality and stability of audio-visual experiences across multiple scenarios. Attached Figure Description

[0055] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this invention. For those skilled in the art, other drawings can be obtained based on these drawings.

[0056] Figure 1 This is a flowchart of the audio and video processing method that supports AI intelligent repair technology according to the present invention.

[0057] Figure 2 This is a schematic diagram of the audio and video processing system that supports AI intelligent repair technology according to the present invention. Detailed Implementation

[0058] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, they are provided so that the description of this disclosure will be more complete and fully convey the concept of the exemplary embodiments to those skilled in the art.

[0059] This invention provides, for example Figure 1 The audio and video processing method shown, which supports AI intelligent repair technology, includes the following steps:

[0060] A scene recognition process is established based on playback source type, user interaction pattern, audio and video bitrate, audio sampling rate, and network transmission latency to generate live streaming scene tags, movie viewing scene tags, and singing scene tags to clarify the subsequent audio and video processing targets.

[0061] To achieve accurate identification of different entertainment scenarios and thus clarify performance targets for subsequent audio and video processing, the following steps are adopted to establish a scenario identification process based on playback source type, user interaction mode, audio and video bitrate, audio sampling rate, and network transmission latency.

[0062] The playback source is classified. The playback source can be a video file stored on local storage media, or a real-time or on-demand audio / video stream transmitted via network protocols. During the classification process, the media file header information or session initialization data of the streaming media transmission is first read. By parsing the media encapsulation format identifier (e.g., MP4, MKV, TS, FLV), the transmission protocol type (e.g., Real-time Transport Protocol (RTP), Hypertext Transfer Protocol (HTTP), Real-time Messaging Protocol (RTMP), and the media file's internal metadata, the content source and transmission characteristics of the playback source are determined. When the transmission protocol is detected as RTMP and the media stream has continuous small data packet updates, it is determined to be a live streaming playback source; when the playback source is detected as a locally stored high-resolution movie file, and the video frame rate, resolution, and bitrate reach preset high-definition standard values, it is determined to be a movie viewing playback source; when the playback source contains independent vocal and accompaniment tracks, and the metadata contains word-by-word lyrics timestamp information, it is determined to be a singing playback source.

[0063] The system captures user interaction patterns during playback. These patterns are determined by recording user commands, frequency, intervals, and content. Specifically, they include commands to start playback, pause, fast forward, rewind, switch channels, input real-time comments, trigger synchronized lyrics, and adjust volume. During live stream playback, users typically trigger channel switching and real-time comment input multiple times within a short period. During movie playback, user interactions are fewer, primarily focused on start playback, pause, and a small number of fast forward or rewind actions. During karaoke playback, users frequently trigger synchronized lyrics display, pitch adjustment, rating triggers, and accompaniment channel on / off operations. Statistical analysis of these specific interactions reveals behavioral patterns corresponding to different playback scenarios.

[0064] The system collects technical parameters and network transmission characteristics of the audio and video streams during playback. Technical parameters include video encoding methods (e.g., H.264, H.265), video frame rates (e.g., 24 fps, 30 fps, 60 fps), video resolutions (e.g., 1920×1080 pixels, 3840×2160 pixels), video bitrates (e.g., 5 Mbps, 15 Mbps), audio encoding methods (e.g., AAC, FLAC), audio sampling rates (e.g., 44.1 kHz, 48 kHz, 96 kHz), and the number of audio channels (mono, stereo, 5.1 channels). Network transmission characteristics include first-frame latency (measured in milliseconds, the time from when the user issues a playback command to when the first frame is displayed), segment latency during playback (the waiting time for each video segment switch), buffering duration (the total time the video is paused due to buffering), and end-to-end transmission latency (the total time from source encoding to terminal decoding). These data are calculated using the player's decoding statistics interface, network transmission monitoring interface, and system timestamps.

[0065] The playback source type parsing results, user interaction pattern data, audio and video technical parameters, and network transmission characteristics are input into a pre-established set of scene determination rules. This set of rules matches based on multiple conditions. For example, when the playback source type is detected as a real-time stream, the user interaction frequency is higher than five times per minute, and the end-to-end network transmission latency is less than two seconds, a live streaming scene label is output; when the playback source type is detected as a local high-definition file, the video resolution is not less than 1920×1080 pixels, the video bitrate is not less than 10 megabits per second, and the user interaction frequency is less than twice per minute, a movie viewing scene label is output; when the playback source contains independent vocal and accompaniment tracks, the audio sampling rate is not less than 48 kHz, the lyrics timestamp strictly corresponds to the playback audio, and the user triggers lyrics synchronization and pitch adjustment multiple times during playback, a singing scene label is output. The generated scene labels are directly passed to subsequent audio and video processing flows to clarify performance goals and drive the selection of repair strategies, ensuring that the audio and video processing process is optimized differently for the characteristics of different entertainment scenarios, thereby improving picture quality, sound quality, and the user's immersive experience.

[0066] The purpose of this step is to accurately identify the entertainment scenario of the current playback activity by collecting, parsing, and comprehensively analyzing multi-dimensional information such as playback source type, user interaction pattern, audio and video bitrate, audio sampling rate, and network transmission latency. This information is then output as a tag for a live streaming scenario, movie-watching scenario, or singing scenario, providing clear and highly targeted performance goals and optimization directions for subsequent audio and video processing. Specifically, different entertainment scenarios have significant differences in latency tolerance, image quality detail requirements, audio fidelity requirements, and synchronization accuracy. Using a uniform audio and video restoration strategy without differentiation will lead to problems such as latency accumulation, insufficient detail, or synchronization inaccuracies. Through scenario identification in this step, scenario characteristics can be locked in before the processing flow begins, thereby driving the generation of subsequent performance constraints and the configuration of the restoration link. This allows the system to rationally allocate computing resources, select appropriate restoration algorithms and parameters based on scenario differences, and divide tasks between the cloud and the terminal. This approach not only avoids the performance waste and inconsistent experience issues caused by generic strategies, but also ensures low latency and smooth synchronization in live streaming, high resolution and detail reproduction in movie watching, and high-fidelity sound quality and accurate accompaniment synchronization in singing, fundamentally improving audio and video processing effects and user immersion experience.

[0067] Based on scene recognition labels, the latency limit, resolution target, detail restoration level, audio fidelity index and beat alignment threshold are quantified to form a performance constraint vector that drives the selection of audio and video restoration strategies.

[0068] To quantize the latency ceiling, resolution target, detail restoration level, audio fidelity index, and beat alignment threshold based on scene recognition labels, thereby forming a performance constraint vector to drive the selection of audio and video restoration strategies, the following steps are specifically included:

[0069] Based on the live streaming, movie viewing, and singing scene tags, the corresponding initial performance target values ​​are read from the pre-defined scene baseline parameter table. For the live streaming scene, the end-to-end latency is capped at 2000 milliseconds, the pre-processing time budget for rendering does not exceed 120 milliseconds, the target resolution is set at 1920 pixels multiplied by 1080 pixels (allowed to be reduced to 1280 pixels multiplied by 720 pixels when network conditions are insufficient), the detail reproduction level is set to medium, and the audio fidelity requirements are a sampling rate of at least 48 kHz, a signal-to-noise ratio of at least 60 decibels, and an absolute value of audio-visual synchronization error not exceeding 40 milliseconds. For the movie viewing scene, the end-to-end latency is capped at 5000 milliseconds, the target resolution is set at 3840 pixels multiplied by 2160 pixels, and the detail reproduction level is set to high. For video recording, the timing consistency must be stable. For audio fidelity, the sampling rate must be no less than 48 kHz, the total harmonic distortion plus noise must be no more than 0.005%, the frequency response must cover the range of 20 Hz to 20 kHz, and the absolute value of the audio-visual synchronization error must not exceed 30 milliseconds. For singing scenes, the end-to-end latency limit is set to 1,000 milliseconds, the pre-processing time budget for rendering must not exceed 80 milliseconds, the target resolution is 1,920 pixels by 1,800 pixels, the detail reproduction level is set to medium, and the audio fidelity requirements are a sampling rate of no less than 96 kHz, a signal-to-noise ratio of no less than 70 decibels, stable phase consistency, and a beat alignment threshold of no more than 10 milliseconds.

[0070] After acquiring scene tags, the system collects runtime data for a fixed period during the current playback process, ranging from ten to thirty seconds. The collected data includes average video bitrate (megabits per second), average video frame rate (frames per second), frame rate fluctuation range (difference between maximum and minimum values), keyframe interval (number of frames), audio sampling rate (hertz), audio peak level (decibels), noise floor level (decibels), network round-trip latency (milliseconds), latency jitter (milliseconds), packet loss rate (percentage), first frame rendering time (milliseconds), and segment start latency (milliseconds). This data is acquired in real-time through the playback end's decoding statistics interface and network monitoring interface, and is precisely timed using a local clock.

[0071] The scene baseline parameters are compared with the real-time collected operating data, and the performance target values ​​are adjusted according to the available processing margin. The latency upper limit is the smaller value between the scene baseline value and the available latency budget calculated based on network round-trip latency, jitter amplitude, and packet loss ratio. The resolution target is determined by comprehensively considering the physical resolution of the display panel, the average video bitrate, and the frame rate fluctuation range. The target remains unchanged when the panel resolution is met and the bitrate is stable. When the bitrate is insufficient and the frame rate fluctuates greatly, the resolution is lowered by one level, while ensuring that the frame rate is not less than 30 frames per second. The detail reproduction level is selected from three levels: low, medium, and high. The high level is selected when the bitrate is high and the noise is low, and the medium or low level is selected when the bitrate is low or the noise is high. The audio fidelity index is set to a target of not less than the scene baseline value based on the sampling rate, signal-to-noise ratio, total harmonic distortion plus noise, and phase consistency. The beat alignment threshold is set based on the time offset of the lyrics timestamp and the measurement results of the accompaniment beat stability. It is fixed at no more than 10 milliseconds in the singing scenario and no more than 30 milliseconds in the live streaming and movie watching scenarios.

[0072] The adjusted latency cap, resolution target, detail restoration level, audio fidelity index, and beat alignment threshold are written into the performance constraint vector in a ternary format (key name, unit, value), along with scene tags, acquisition timestamps, and descriptions of acquisition device capabilities. A consistency check is then performed, confirming that the latency cap is not less than the pre-processing budget plus a safety margin, the resolution target does not exceed the physical resolution of the display panel, the detail restoration level matches the video frame rate, and the beat alignment threshold does not exceed the audio-visual synchronization error limit. After all checks pass, the performance constraint vector is submitted to the subsequent audio-visual restoration process, serving as the sole input for restoration strategy selection and parameter configuration, thus forming a closed-loop process from scene recognition to performance quantification and structured output.

[0073] The purpose of this step is to transform abstract scene labels into quantifiable and actionable technical objectives after scene identification, providing clear and binding performance parameters for the selection and execution of subsequent audio and video restoration strategies. Different entertainment scenarios exhibit significant differences in latency tolerance, image quality priority, detail recovery requirements, audio fidelity, and audio-visual synchronization accuracy. If only scene labels are available without quantitative standards, subsequent processing stages cannot accurately match the optimal restoration strategy, potentially leading to unreasonable allocation of performance resources or processing results deviating from user expectations. This step combines scene baseline parameters and real-time operational data to generate a performance constraint vector containing latency limits, resolution targets, detail recovery levels, audio fidelity metrics, and beat alignment thresholds. This vector not only clarifies the numerical targets and allowable ranges for each performance dimension but also closely correlates with the current network status, device capabilities, and content characteristics, ensuring that the restoration process has operable input conditions during execution. The introduction of performance constraint vectors allows the restoration process to be adjusted as needed in different scenarios. For example, in live streaming, priority can be given to reducing latency; in watching movies, priority can be given to improving resolution and detail; and in singing, priority can be given to ensuring sound quality and beat synchronization. This establishes a direct mapping relationship between scene recognition and processing strategies, enabling overall optimization of audio and video restoration effects and user experience in multiple scenarios.

[0074] Based on the performance constraint vector, the audio and video restoration link is arranged in a node-based manner. Each node is configured with a denoising model, a super-resolution reconstruction model, a frame interpolation compensation model, a color restoration model, an audio waveform reconstruction model, and an audio-visual synchronization correction model and their parameters, so that all nodes meet the requirements of the performance constraint vector.

[0075] To orchestrate the audio-visual restoration pipeline according to the performance constraint vector, and to configure denoising models, super-resolution reconstruction models, frame interpolation compensation models, color restoration models, audio waveform reconstruction models, and audio-visual synchronization correction models and their parameters at each node, ensuring that the processing results of all nodes meet the requirements of the performance constraint vector, the following steps are specifically included:

[0076] Based on the latency cap, resolution target, and detail restoration level in the performance constraint vector, the node order and processing priority of the repair link are determined, and the denoising processing node is set as the starting node of the link. Video denoising uses a fixed-window pixel block statistical method, calculating the local noise energy value for each frame of video in an 8x8 pixel region. This value is compared with the target signal-to-noise ratio in the performance constraint vector. If the noise energy exceeds the target threshold, the high-frequency components in that region are reduced according to preset filtering coefficients, while preserving grayscale gradient changes to maintain edge details. Audio denoising measures the background noise power through full-band spectrum analysis and, based on the total harmonic distortion plus noise limit in the performance constraint vector, attenuation processing is performed on frequency bands where the noise power exceeds the limit to ensure that the signal-to-noise ratio meets the requirements after denoising.

[0077] Following the denoising node, a super-resolution reconstruction node and a frame interpolation compensation node are configured sequentially. The super-resolution reconstruction node resamples the pixel matrix of each frame of the video according to the resolution target set in the performance constraint vector. When the resolution target is higher than the original frame resolution, an interpolation method based on pixel neighborhood gradients is used to increase the image size, and sharpening weights are added to edge regions to improve the contrast of details and textures by at least 10%. The frame interpolation compensation node calculates the intermediate frame based on the motion vector information of two adjacent frames. When the output frame rate required by the performance constraint vector is higher than the original frame rate, compensation frames are accurately generated according to the number of frames added per second, and motion trajectory prediction is performed in the moving object region to ensure that the temporal continuity error after interpolation does not exceed two milliseconds.

[0078] After the video resolution and frame rate stabilize, configure the color restoration node and the audio waveform reconstruction node. The color restoration node performs color space conversion on the video frames, converting the input YUV color values ​​to the target color space (e.g., Rec.709 or DCI-P3) specified by the performance constraint vector, and corrects the offset of each color channel by comparing it with standard color chart data to ensure that the color ΔE value is no greater than two. The audio waveform reconstruction node re-interpolates the sampled points of the audio signal according to the sampling rate, signal-to-noise ratio, and frequency response range specified by the performance constraint vector. When the sampling rate is lower than the target value, high-precision interpolation is performed; when the sampling rate is higher than the target value, resampling is performed. After resampling, the phase difference between the two stereo channels is corrected so that the phase offset does not exceed the target threshold.

[0079] At the end of the repair link, an audio-visual synchronization correction node is configured. This node compares the video frame display timestamp with the audio playback timestamp, calculates the time difference, and compares it with the beat alignment threshold in the performance constraint vector. If the difference exceeds the threshold, the fill depth of the playback buffer queue is dynamically adjusted. By precisely discarding or repeating audio sampling points and adjusting the video frame output time, the absolute value of the final audio-visual synchronization error is ensured to not exceed the performance constraint vector limit. After synchronization correction is completed, consistency checks are performed on the output results of all nodes in the link. These checks include whether the signal-to-noise ratio of the denoised video meets the performance constraint vector setting, whether the output resolution and frame rate are consistent with the target, whether the color difference after color restoration is less than the limit, whether the signal-to-noise ratio and frequency response after audio reconstruction meet the standards, and whether the audio-visual synchronization error meets the constraints. After all checks pass, the repair result is output to the playback end or storage medium, realizing a closed-loop processing process from performance constraint vector-driven to node-based orchestration execution to high-quality audio and video output.

[0080] This step transforms the abstract numerical targets in the performance constraint vector into a concrete, executable processing chain structure. It precisely decomposes and systematically arranges the audio-visual restoration tasks through a node-based approach, ensuring that the entire restoration process simultaneously meets predetermined requirements across multiple performance dimensions, including latency, resolution, detail restoration, audio fidelity, and synchronization accuracy. Different performance targets correspond to different processing priorities and resource allocation strategies. For example, under high-resolution and high-detail requirements, super-resolution reconstruction and detail sharpening should be prioritized in the chain, while under low-latency requirements, computationally complex nodes should be reduced or their execution order optimized. This step allocates processing functions such as denoising, super-resolution reconstruction, frame interpolation, color restoration, audio waveform reconstruction, and audio-visual synchronization correction to independent nodes, configuring processing parameters for each node that match the performance constraint vector, ensuring that the output of each stage conforms to the overall target. This not only technically achieves step-by-step assurance of complex, multi-dimensional performance indicators but also improves the controllability and adjustability of the restoration process. When a performance indicator deviates during execution, the corresponding node can be directly located and adjusted. This step actually establishes a direct mapping relationship from performance goals to specific processing procedures, enabling the repair chain to be repeatedly deployed and optimized iteratively, laying the foundation for achieving consistency and stability of audio and video repair effects in multiple scenarios.

[0081] Based on the computational load of the repair link and the computing power of the terminal device, the cloud execution task and the terminal execution task are divided in the unified entertainment system architecture, and the matching plugins and parameters are distributed to the terminal through the cloud for real-time loading.

[0082] To divide tasks based on the computational load of the repair link and the computing power of the terminal devices, and to distribute matching plugins and parameters to the terminals for real-time loading via the cloud within a unified entertainment operation framework, the specific steps include:

[0083] A quantitative evaluation process for the computational load of the repair link and the computing power of the terminal was established. For each of the six processing stages in the repair link—denoising, super-resolution reconstruction, frame interpolation, color restoration, audio waveform reconstruction, and audio-visual synchronization correction—the average processing time per frame (in milliseconds), peak processing time, required memory capacity (in megabytes), required video memory capacity (in megabytes), and required floating-point operations (in billions of floating-point operations per second) were measured. On the terminal side, the following data were collected: CPU clock speed (in megahertz), number of cores, GPU single-precision floating-point operation capability (in trillions of floating-point operations per second), dedicated video acceleration unit processing capability (in operations per second), available memory capacity, available video memory capacity, device thermal limit trigger temperature (in degrees Celsius), continuous power consumption limit (in watts), and current network uplink and downlink bandwidth (in megabits per second). The unit frame time budget corresponding to playing 30 frames per second is set to 33 milliseconds, with a 3-millisecond safety margin. The peak processing time of each stage is compared with this budget to filter out the candidate list of stages that need to be moved to the cloud for execution.

[0084] Based on task allocation criteria, each stage of the repair process is assigned to either cloud or terminal execution. Stages with peak processing time exceeding half the budget and whose processing data can be pre-fetched are assigned to cloud execution; stages with response times below 50 milliseconds and directly impacting playback clock synchronization are assigned to terminal execution. When network uplink bandwidth is greater than 10 megabits per second and end-to-end round-trip latency is less than 150 milliseconds, super-resolution reconstruction and color restoration are permitted to be performed in the cloud; when network uplink bandwidth is less than 10 megabits per second or end-to-end round-trip latency is greater than 150 milliseconds, super-resolution reconstruction must be performed on the terminal using degraded parameters. For singing scenarios, audio waveform reconstruction and audio-visual synchronization correction are fixed to terminal execution; for movie watching scenarios, cloud-based super-resolution reconstruction is prioritized when playback buffer time exceeds three seconds; for live streaming scenarios, adding cloud processing stages that would significantly accumulate latency is prohibited when buffer depth is less than 500 milliseconds.

[0085] A distribution list of plugins and parameters is generated. This list includes the plugin's filename, version number, target processing stage, binary file size (in megabytes), required floating-point arithmetic capability, memory capacity, video memory capacity, supported graphics processor architecture, supported dedicated acceleration unit instruction sets, and key names, units, and value ranges from the parameter file. The cloud packages the plugin and parameter files into a compressed file, attaches an elliptic curve digital signature and hash value to ensure data integrity, and distributes it to the terminal via an encrypted transmission channel. Upon receiving the file, the terminal verifies the signature and hash value. After confirming compatibility with hardware capabilities, the plugin is stored in a local isolated directory. During playback idle periods, pre-decompression and loading tests are performed, recording the initial loading time and memory increment. When the initial loading time does not exceed 100 milliseconds and the memory increment does not exceed 250 megabytes, it is marked as ready for immediate loading.

[0086] The system implements real-time loading and runtime rollback control. Before the playback session begins, the terminal loads plugins that need to be executed locally in sequence, including noise reduction, frame interpolation compensation, audio waveform reconstruction, and audio-visual synchronization correction. For processes requiring cloud execution, a task request is sent before playback, and processing and transmission are initiated after buffered data preparation. During playback, the system monitors unit frame processing time, buffer depth, end-to-end round-trip latency, and packet loss ratio in real time. When the unit frame processing time exceeds the budget three times consecutively, the processing parameters of the corresponding plugin are automatically reduced. If the parameters are reduced but still insufficient, the system rolls back to the previous lower-version plugin. When the network uplink bandwidth is continuously below 5 megabits per second, the system switches from cloud-based super-resolution reconstruction to terminal-based downgrade reconstruction and simultaneously disables cloud-based color restoration. After playback, the system records the plugin versions used, loading time, downgrade counts, rollback counts, network anomaly duration, and user-perceptible interruption counts in a log file and uploads it to the cloud. This log file serves as a reference for subsequent version optimization and phased release strategies, achieving a closed-loop process based on computing power and computational load for task allocation, plugin distribution, real-time loading, and stable operation.

[0087] The purpose of this step is to rationally distribute the complex audio and video restoration process between the cloud and the terminal before execution. This ensures the restoration effect while maximizing the use of available computing resources, avoiding latency accumulation, stuttering, or image quality degradation caused by excessive computational pressure on one side. Audio and video restoration involves computationally intensive steps such as noise reduction, super-resolution reconstruction, frame interpolation, color restoration, audio waveform reconstruction, and audio-visual synchronization correction. Different steps have different requirements for latency, bandwidth, and real-time performance, directly affecting the overall performance of the processing chain. If all steps are executed on the terminal, low-performance devices may not be able to complete the processing within the time limit; if everything relies on the cloud, it will be constrained by network bandwidth and latency fluctuations, resulting in playback asynchrony or unstable image quality. Through this step, the system first quantifies the computational load and device computing power of each step, and then combines it with the real-time network status. It allocates computationally intensive and pre-fetchable data processing steps to the cloud for execution, while keeping latency-sensitive steps on the terminal. By distributing and loading plugins and parameters in real time through the cloud, it ensures that task switching is fast and does not affect playback continuity. This division of labor based on computational load and processing power can leverage the advantages of local computing power on high-performance terminals while making up for deficiencies on low-performance terminals by relying on cloud computing power. This enables stable low-latency, high-definition, and high-audio-definition restoration effects in different scenarios such as live streaming, movie watching, and singing, improving the overall user experience and providing sustainable operational data support for the long-term iterative optimization of the system.

[0088] During the task execution, based on real-time collected playback latency, stuttering rate, picture quality score, audio distortion rate and audio-visual synchronization error, the parameters of each node in the repair link are dynamically adjusted and adaptively corrected to ensure that the output effect continuously meets the requirements of the performance constraint vector.

[0089] To dynamically adjust and adaptively correct the parameters of each processing node in the repair link based on real-time acquired playback latency, stuttering rate, picture quality score, audio distortion rate, and audio-visual synchronization error during task execution, ensuring that the overall output effect always meets the requirements of the performance constraint vector, the following steps are specifically included:

[0090] A continuous data acquisition and quantization process was established, with a sampling period set to 200 milliseconds and a statistical window set to a 10-second scrolling interval. Playback latency was calculated as the display timestamp of the current frame minus the request initiation time corresponding to that frame. The stuttering rate was calculated as the cumulative stuttering duration divided by the total playback duration and multiplied by 100 to obtain a percentage. Image quality scoring used a 0-100 scale, weighted based on four objective indicators: image sharpness (resolution detail retention), contrast, texture continuity, and color shift. Audio distortion rate was expressed as total harmonic distortion plus noise percentage (THD+N). Audio-visual synchronization error was the absolute value of the difference between the audio playback timestamp and the video display timestamp. Specific target values ​​are set for different scenarios. For example, for live streaming scenarios, the image quality score is required to be ≥75 points, the audio-visual synchronization error is required to be ≤40 milliseconds, and the stuttering rate is required to be ≤1%; for movie watching scenarios, the image quality score is required to be ≥85 points, the audio-visual synchronization error is required to be ≤30 milliseconds, and the stuttering rate is required to be ≤0.5%; for singing scenarios, the image quality score is required to be ≥80 points, the audio-visual synchronization error is required to be ≤10 milliseconds, and the audio distortion rate is required to be ≤0.005%.

[0091] Establish performance deviation judgment and parameter mapping rules. When the average playback latency reaches 90% of the latency upper limit in the performance constraint vector, reduce the processing intensity of computationally expensive nodes. For example, lower the super-resolution magnification by one level, reduce the interpolation compensation output frame rate from 60 frames / second to 30 frames / second, adjust the noise reduction window size from 8×8 pixels to 4×4 pixels, and reduce the filtering coefficient by 20%. When the stuttering rate exceeds the target value, increase the video buffer depth by one frame and simultaneously reduce the local correction frequency of color reproduction. When the picture quality score is more than 5 points lower than the target value, increase the detail reproduction level by one level and increase the edge sharpening weight by 10% without exceeding the latency budget. When the audio distortion rate is higher than the target value, reduce the dynamic range compression ratio and increase the quantization accuracy of waveform reconstruction. When the audio-visual synchronization error exceeds the threshold for three consecutive samples, adjust the audio and video playback phase, for example, increase the audio buffer by 2 milliseconds or delay the video output by 2 milliseconds.

[0092] During parameter adjustments, a stepped and hysteresis control strategy is employed to avoid frequent oscillations. Each adjustment interval is no less than 1 second, and the magnitude of a single adjustment does not exceed twice the previous one. Furthermore, all parameter changes must be limited to the maximum and minimum values ​​defined by the performance constraint vector. For live streaming scenarios, low latency and synchronization accuracy are prioritized, allowing one switch between 1920×1080 pixels and 1280×720 pixels in resolution. For movie viewing scenarios, resolution and image quality rating are prioritized, allowing one switch between 30 frames per second and 60 frames per second in frame interpolation compensation. For singing scenarios, audio fidelity and beat alignment are prioritized, and lowering the audio sampling rate is prohibited; only minor adjustments to video denoising and color reproduction parameters are permitted.

[0093] Establish a rapid rollback and effect confirmation mechanism. After each parameter adjustment, continue monitoring two statistical windows. If any core performance indicator deteriorates after adjustment and exceeds the target value by 10%, immediately roll back to the previous parameter set, and record the rollback reason, trigger threshold, and rollback time. If the monitoring results of two consecutive windows reach the target value, mark the current parameter set as the stable version and cache it locally. When entering the same scenario again, prioritize loading this stable version to shorten the system convergence time. At the end of the session, upload the entire performance indicator trajectory, parameter adjustment records, rollback records, final stable version parameters, and corresponding performance constraint vectors to the version repository for subsequent phased releases and baseline updates. This completes the closed-loop control from real-time detection to parameter adjustment and long-term optimization, ensuring that the set performance goals are always met during playback.

[0094] The purpose of this step is to establish a dynamic closed-loop control mechanism based on real-time performance data throughout the entire audio and video restoration task. This mechanism enables each processing node in the restoration chain to adjust parameters and adaptively correct itself in real time according to the actual playback status, thereby ensuring that the output effect always meets the multi-dimensional performance goals set by the performance constraint vector. Because audio and video restoration involves many processing stages, a large amount of computation, and interdependencies, performance fluctuations in any stage during playback can lead to a decline in the overall experience, such as increased latency, decreased image quality, audio distortion, or audio-visual asynchrony. This step collects key indicators such as playback latency, stuttering rate, image quality score, audio distortion rate, and audio-visual synchronization error in real time and compares them item by item with the target values ​​corresponding to the scene. Once a deviation is detected, the processing intensity, execution parameters, or data flow control strategies of the relevant nodes are immediately adjusted to bring performance back to the target range. Simultaneously, this step incorporates hysteresis control, stepped adjustment, and rapid rollback mechanisms to effectively avoid system oscillations caused by frequent fluctuations and ensure a smooth and controllable adjustment process. By continuously recording and analyzing the adjustment results, stable parameter combinations can be established as a version baseline, which can then be directly applied in subsequent similar or identical scenarios, enabling rapid convergence and long-term optimization of the repair process. This step not only improves the stability and consistency during playback but also addresses the differentiated needs of various scenarios for low latency, high image quality, high audio quality, and precise synchronization, thereby significantly enhancing the overall user experience.

[0095] The dynamically adjusted parameters are written into the scene feature profile and version repository, and gradually applied to different types of terminals according to the phased release strategy, so as to form a stable scene-based audio and video repair baseline and realize a closed loop of continuous iterative optimization.

[0096] To write the dynamically adjusted parameters into the scene feature profile and version repository, and to gradually apply them to different types of terminals according to a phased release strategy, thereby forming a stable scene-based audio and video restoration baseline and achieving a closed loop of continuous iterative optimization, the following steps are taken:

[0097] Execution parameter accumulation and scene feature profile update. During a playback session, a dynamically adjusted and stably converged set of parameters includes: denoising window size (e.g., 4×4 pixels, 8×8 pixels), denoising intensity coefficient, super-resolution magnification (e.g., 1.5x, 2x), interpolation compensation output frame rate (e.g., 30 fps, 60 fps), upper limit of interpolation time continuity error (in milliseconds), target color space standard (e.g., BT.709, BT.2020), skin tone protection weight (percentage), luminance and chroma correction thresholds (percentage), audio sampling rate (e.g., 44.1kHz, 48kHz), target audio signal-to-noise ratio (dB), waveform reconstruction quantization level, and allowed parameters. The system records the maximum phase difference (milliseconds), audio-visual synchronization buffer depth (milliseconds), maximum allowable synchronization error (milliseconds), and session performance metrics, including average picture quality score, 95th percentile picture quality score, average stuttering rate, average playback latency, audio distortion rate, unit frame processing time, average power consumption, peak temperature, end-to-end round-trip latency distribution, and average and fluctuation range of uplink and downlink bandwidth. It also records device and content characteristic data, including terminal model, physical resolution of the display panel, panel refresh rate, audio hardware sampling capability, available storage capacity, thermal limit trigger temperature, content type tags (e.g., movie, live stream, karaoke), and playback duration range. This information is written into the scene feature profile in the form of structured data.

[0098] The aforementioned parameter sets and scene feature profiles are versioned and stored. A unique version number is generated for each parameter set in the version repository, following the rule of "scene tag + date and timestamp + incrementing sequence number," and a metadata list is created. The metadata list includes the range of compatible devices, minimum available memory capacity, minimum available video memory capacity, required computing power range, a description of parameter differences from the previous version, expected improvement goals, a list of potential performance risks, rollback trigger conditions, and log field definitions. The parameter sets and metadata are packaged into an archive file, a SHA256 checksum and digital signature are generated, and integrity verification is performed. After verification, an offline reproducible experiment is conducted, processing content segments consistent with the session. Once it is confirmed that the image quality score, stuttering rate, audio-visual synchronization error, and unit frame processing time all meet the target range, the archive file is written to the version repository.

[0099] Implement a phased release strategy and prepare for immediate loading. Expand the application scope of the version in stages: 5%, 20%, 50%, and 100%. First, test on high-performance terminals, then expand to medium-performance terminals, and finally cover low-power terminals. Each stage should run for at least 24 hours, ensuring coverage of at least 30 minutes for each of the three scenarios: live streaming, movie watching, and karaoke. The passing criteria for each stage are: image quality score not lower than the target value, stuttering rate not higher than the target value, audio-visual synchronization error not exceeding the set limit, no more than one rollback, and no more than two user-perceived interruptions. For versions that pass the test, their plugins and parameters will undergo pre-download, decompression, and cold start tests on the terminal. The initial loading time should not exceed 100 milliseconds, and the increase in memory usage should not exceed 250MB before it can be marked as a version that can be loaded immediately.

[0100] Establish a rollback and closed-loop optimization mechanism. At any stage, if any of the following occurs: a drop in image quality score exceeding the target value by more than 10%, an increase in stuttering rate exceeding the target value by more than 10%, or an audio-visual synchronization error exceeding the limit for three consecutive sampling periods, immediately switch to the previous stable version according to the rollback conditions recorded in the version repository, and record the rollback time, reason, and relevant performance data. After the stage ends, write the runtime log, anomaly reasons, rollback records, final stable version parameters, and corresponding performance constraint vectors to the version repository. After all stages are completed, the finally verified version is set as the new scenario-based audio-visual restoration baseline, and this baseline is prioritized for loading in subsequent similar scenarios. Simultaneously, the runtime data of new sessions will be used to update the scene feature profile, compare and analyze it with the existing baseline, and trigger the next round of parameter iteration and baseline upgrade when conditions are met, thus forming a long-term stable and sustainable closed-loop optimization mechanism.

[0101] This step aims to solidify the dynamically adjusted and stably converged repair link parameters during task execution into a structured and versioned format within the scene feature profile and version repository. A phased release strategy enables orderly application across different types of terminals, forming a reusable and continuously optimizable scenario-based audio and video repair baseline. This process allows for the optimization results of processing node parameters such as noise reduction intensity, super-resolution magnification, frame interpolation compensation frame rate, color restoration standards, audio waveform reconstruction accuracy, and audio-visual synchronization buffer depth within a single session, linked and stored with corresponding operating conditions such as image quality score, stuttering rate, playback latency, audio distortion rate, network bandwidth status, and device hardware characteristics, constructing a comprehensive scene feature profile. Subsequently, version numbers, metadata lists, and verification mechanisms ensure consistent availability and reliability of this parameter set under different hardware and network conditions. The phased release strategy allows for verification of version stability and performance on a small number of devices, gradually expanding the application scope and reducing the risk of large-scale updates. Simultaneously, a rollback mechanism allows for rapid restoration to the previous stable version when performance anomalies are detected, ensuring continuity and security of the user experience. This step not only ensures the rapid loading and application of the repair link parameters in subsequent similar scenarios, but also enables long-term closed-loop optimization and baseline upgrades by continuously accumulating and comparing the running data of different versions in multiple scenarios, giving the audio and video repair system the ability to adapt and evolve.

[0102] This invention addresses the differentiated performance requirements of various entertainment scenarios such as live streaming, movie watching, and karaoke. It constructs a closed-loop processing mechanism encompassing scene identification, performance constraint setting, refined orchestration of the repair process, cloud and terminal task allocation, real-time adaptive adjustment, and parameter accumulation and iterative optimization. This overcomes the bottleneck of existing generalized repair strategies that cannot simultaneously address performance and user experience across multiple scenarios. The solution effectively controls latency and maintains real-time audio-visual synchronization in live streaming, significantly improves resolution and detail reproduction in movie watching, and ensures high audio fidelity and precise alignment with accompaniment rhythm in karaoke. Furthermore, cloud distribution and instant loading enable self-adaptation of computing power across different hardware terminals, ensuring stable repair results for both high- and low-performance devices. Simultaneously, the versioned parameter storage and phased release mechanism allows for rapid dissemination of validated optimization results and continuous improvement of the repair baseline's stability and performance consistency through multiple iterations. This endows the system with long-term self-evolution capabilities, significantly enhancing the quality and stability of audio-visual experiences across multiple scenarios.

[0103] This invention provides, for example Figure 2 The audio and video processing system shown, which supports AI-powered intelligent repair technology, includes a scene recognition module, a performance constraint generation module, a repair link orchestration module, a task allocation and loading module, a real-time adaptive adjustment module, and a parameter accumulation and iterative optimization module.

[0104] The scene recognition module identifies scenes based on playback source type, user interaction pattern, audio and video bitrate, audio sampling rate, and network transmission latency, and generates scene tags for live streaming, movie watching, and singing.

[0105] The performance constraint generation module determines the latency limit, resolution target, detail restoration level, audio fidelity index and beat alignment threshold based on scene labels, forming a performance constraint vector.

[0106] The repair link orchestration module arranges the audio and video repair links into nodes according to the performance constraint vector, and configures the noise reduction, super-resolution reconstruction, frame interpolation compensation, color restoration, audio waveform reconstruction and audio-visual synchronization correction models and parameters.

[0107] The task allocation and loading module, based on the computational load of the repair link and the computing power of the terminal, divides the execution tasks in the cloud and the terminal in the entertainment system architecture, and distributes the plugins and parameters from the cloud to the terminal for loading;

[0108] The real-time adaptive adjustment module dynamically adjusts and repairs link parameters and adaptively corrects them based on real-time collected playback latency, stuttering rate, picture quality score, audio distortion rate, and audio-visual synchronization error during task execution.

[0109] The parameter accumulation and iterative optimization module stores the adjusted parameters in the scene feature profile and version library, and releases them to the terminal application according to the phased release strategy, so as to realize the continuous iterative optimization of the scene-based audio and video restoration baseline.

[0110] The audio and video processing method supporting AI intelligent repair technology provided in this embodiment of the invention is implemented through the aforementioned audio and video processing system supporting AI intelligent repair technology. For details of the specific methods and processes of the audio and video processing system supporting AI intelligent repair technology, please refer to the embodiments of the audio and video processing method supporting AI intelligent repair technology described above, which will not be repeated here.

[0111] The foregoing has only described certain exemplary embodiments of the present invention by way of illustration. Undoubtedly, those skilled in the art can modify the described embodiments in various ways without departing from the spirit and scope of the present invention. Therefore, the foregoing drawings and descriptions are illustrative in nature and should not be construed as limiting the scope of protection of the claims of the present invention.

Claims

1. An audio and video processing method supporting AI intelligent repair technology, characterized in that, Includes the following steps: Scene recognition is performed based on playback source type, user interaction pattern, audio and video bitrate, audio sampling rate, and network transmission latency to generate scene tags for live streaming, movie watching, and singing. Based on scene labels, determine the latency limit, resolution target, detail restoration level, audio fidelity index, and beat alignment threshold to form a performance constraint vector; The audio and video restoration link is arranged into nodes according to the performance constraint vector, and the noise reduction, super-resolution reconstruction, frame interpolation compensation, color restoration, audio waveform reconstruction and audio-visual synchronization correction models and parameters are configured. Based on the computational load of the repair link and the computing power of the terminal, the cloud and terminal execution tasks are divided in the entertainment system architecture, and the plugins and parameters are distributed from the cloud to the terminal for loading. During task execution, the link parameters are dynamically adjusted and adaptively corrected based on real-time collected playback latency, stuttering rate, picture quality score, audio distortion rate, and audio-visual synchronization error. The adjusted parameters are stored in the scene feature profile and version library, and then deployed to the terminal application according to the phased release strategy to achieve continuous iterative optimization of the scene-based audio and video restoration baseline.

2. The audio and video processing method supporting AI intelligent repair technology according to claim 1, characterized in that, The specific steps for scene recognition based on playback source type, user interaction pattern, audio / video bitrate, audio sampling rate, and network transmission latency are as follows: The playback source is parsed by reading the media file header information or streaming media session initialization data, parsing the media encapsulation format identifier, transmission protocol type and media metadata, and determining whether it is a live broadcast type, movie viewing type or singing type playback source; Acquire user interaction patterns during playback, record operation commands, number of times, interval time and content, and analyze behavioral patterns corresponding to the scenario; During the acquisition and playback process, the video encoding method, frame rate, resolution, bit rate, audio encoding method, sampling rate, number of channels, network first frame delay time, segment first delay time, buffer duration, and end-to-end transmission latency are recorded. The parsing results, interaction pattern data, audio and video technical parameters, and network transmission characteristics are input into the scene determination rule set to generate live streaming scene tags, movie viewing scene tags, or singing scene tags and then pass them to the subsequent audio and video processing flow.

3. The audio and video processing method supporting AI intelligent repair technology according to claim 2, characterized in that, The matching conditions for the set of scene determination rules include: generating a live streaming scene tag when the playback source is a real-time stream, the user interaction frequency is higher than five times per minute, and the end-to-end transmission latency is less than two seconds; generating a movie viewing scene tag when the playback source is a local high-definition file, the video resolution is not lower than 1920×1080 pixels, the video bitrate is not lower than 10 megabits per second, and the user interaction frequency is less than twice per minute; and generating a singing scene tag when the playback source contains independent vocal tracks and accompaniment tracks, the audio sampling rate is not lower than 48 kHz, the lyrics timestamp corresponds to the played audio, and the user triggers lyrics synchronization and pitch adjustment multiple times.

4. The audio and video processing method supporting AI intelligent repair technology according to claim 1, characterized in that, The specific steps for forming a performance constraint vector based on scene recognition labels, including quantization of latency upper limit, resolution target, detail restoration level, audio fidelity index, and beat alignment threshold, are as follows: Read the initial performance target values ​​corresponding to the live streaming scene tag, movie viewing scene tag, and singing scene tag from the preset scene baseline parameter table; Collect 10 to 30 seconds of runtime data, including video bitrate, frame rate, frame rate fluctuation range, keyframe interval, audio sampling rate, audio peak level, noise floor level, network round-trip latency, latency jitter, packet loss ratio, first frame rendering time, and segment start waiting time. The baseline parameters are compared with the acquired data, and the performance target values ​​are adjusted according to the available processing margin to determine the latency limit, resolution target, detail restoration level, audio fidelity index and beat alignment threshold. All adjusted performance target values ​​are written into the performance constraint vector and a consistency check is performed. Once the check passes, the values ​​are submitted to the audio and video repair link as input for repair strategy selection and parameter configuration.

5. The audio and video processing method supporting AI intelligent repair technology according to claim 1, characterized in that, The specific steps for node-based orchestration of the audio / video restoration link according to the performance constraint vector are as follows: Set the denoising processing node as the starting node, and perform denoising processing on video and audio based on the signal-to-noise ratio and total harmonic distortion plus noise limit of the performance constraint vector; Configure super-resolution reconstruction nodes and frame interpolation compensation nodes in sequence to improve video resolution and frame rate while maintaining temporal continuity; Configure the color restoration node and audio waveform reconstruction node to perform color space conversion and audio sampling rate and phase difference correction; Configure an audio-visual synchronization correction node to ensure that the audio-visual synchronization error does not exceed the performance constraint vector limit by adjusting the playback buffer queue depth, and output the repair result after all consistency checks pass.

6. The audio and video processing method supporting AI intelligent repair technology according to claim 5, characterized in that, In the audio-visual synchronization correction node, the difference between the video frame display timestamp and the audio playback timestamp is compared with the beat alignment threshold in the performance constraint vector. If the difference exceeds the threshold range, synchronization correction is performed by precisely discarding or repeating audio sampling points and adjusting the video frame output time, so that the absolute value of the final audio-visual synchronization error does not exceed the limit value of the performance constraint vector.

7. The audio and video processing method supporting AI intelligent repair technology according to claim 1, characterized in that, The specific steps for task partitioning based on the computational load of the repair link and the computing power of the terminal devices are as follows: The average processing time per unit frame, peak processing time, required memory capacity, video memory capacity, and number of floating-point operations for each link in the repair link are measured. The processor clock speed, number of cores, floating-point operation capability, acceleration unit capability, available memory, video memory capacity, power consumption limit, bandwidth, and round-trip latency of the terminal device are collected. Based on the task division criteria, the processing steps are allocated to cloud execution or terminal execution, and the execution location of each step is fixedly configured according to different entertainment scenarios; Generate a distribution list containing plugin file information, runtime requirements, and parameter files; distribute the distribution through an encrypted channel and perform signature and hash value verification. Real-time loading and rollback control are performed before and during playback to ensure processing performance and playback stability, and the running data is recorded to a log file and uploaded to the cloud.

8. The audio and video processing method supporting AI intelligent repair technology according to claim 1, characterized in that, The specific steps for dynamically adjusting and repairing the parameters of each processing node in the link and performing adaptive correction based on real-time collected playback latency, stuttering rate, picture quality score, audio distortion rate, and audio-visual synchronization error during task execution are as follows: Playback latency, stuttering rate, picture quality score, audio distortion rate, and audio-visual synchronization error are continuously collected according to a fixed sampling period and a rolling statistical window, and compared with the target values ​​corresponding to different entertainment scenarios. The processing parameters for denoising, super-resolution reconstruction, frame interpolation compensation, color restoration, audio waveform reconstruction, and audio-visual synchronization correction are adjusted according to performance deviation judgment and parameter mapping rules, and step-wise and hysteresis control is adopted to avoid frequent oscillations. The step-by-step and hysteresis control includes at least the following: the interval between two adjacent adjustments is not less than a preset time, the magnitude of a single adjustment does not exceed a preset range, and the parameter changes are limited to the upper and lower limits defined by the performance constraint vector; After parameter adjustment, performance metrics are continuously monitored through two statistical windows. If performance deteriorates, the previous parameter set is rolled back. If performance is stable, the current parameter set is marked as a stable version and cached. At the end of the session, the performance metric trajectory, parameter adjustment records, rollback records, stable version parameters, and corresponding performance constraint vectors are uploaded to the version repository for subsequent optimization.

9. The audio and video processing method supporting AI intelligent repair technology according to claim 1, characterized in that, The specific steps for writing the dynamically adjusted parameters into the scene feature profile and version repository, and gradually applying them to different types of terminals according to a phased release strategy, are as follows: The dynamically adjusted and stably converged parameters for denoising, super-resolution reconstruction, frame interpolation compensation, color restoration, audio waveform reconstruction, and audio-visual synchronization correction, along with session performance indicators and device and content characteristics, are written into the scene feature profile as structured data. The parameter set is versioned and stored, generating a unique version number and metadata list, and integrity verification and offline reproduction experiments are completed. The application scope of the version will be gradually expanded in accordance with the phased release strategy, and real-time loading tests will be conducted on the terminal. During operation, a rollback is triggered based on performance anomalies. After the phase ends, the operation logs, anomaly causes, rollback records, and final stable version parameters are written to the version repository, forming a closed-loop optimization mechanism.

10. An audio-visual processing system supporting AI intelligent repair technology, used to implement the audio-visual processing method supporting AI intelligent repair technology as described in any one of claims 1-9, characterized in that, It includes a scene recognition module, a performance constraint generation module, a repair link orchestration module, a task allocation and loading module, a real-time adaptive adjustment module, and a parameter accumulation and iterative optimization module. The scene recognition module identifies scenes based on playback source type, user interaction pattern, audio and video bitrate, audio sampling rate, and network transmission latency, and generates scene tags for live streaming, movie watching, and singing. The performance constraint generation module determines the latency limit, resolution target, detail restoration level, audio fidelity index and beat alignment threshold based on scene labels, forming a performance constraint vector. The repair link orchestration module arranges the audio and video repair links into nodes according to the performance constraint vector, and configures the noise reduction, super-resolution reconstruction, frame interpolation compensation, color restoration, audio waveform reconstruction and audio-visual synchronization correction models and parameters. The task allocation and loading module, based on the computational load of the repair link and the computing power of the terminal, divides the execution tasks in the cloud and the terminal in the entertainment system architecture, and distributes the plugins and parameters from the cloud to the terminal for loading; The real-time adaptive adjustment module dynamically adjusts and repairs link parameters and adaptively corrects them based on real-time collected playback latency, stuttering rate, picture quality score, audio distortion rate, and audio-visual synchronization error during task execution. The parameter accumulation and iterative optimization module stores the adjusted parameters in the scene feature profile and version library, and releases them to the terminal application according to the phased release strategy, so as to realize the continuous iterative optimization of the scene-based audio and video restoration baseline.

Images