A smart TV image quality optimization method based on AI chip and smart TV

Abstract

The present invention provides a method for optimizing the image quality of a smart TV based on an AI chip and a smart TV. The method comprises: obtaining an input video signal and performing decoding and frame extraction to obtain an original video frame sequence; performing super-resolution reconstruction using a pre-established feature processing model to obtain a super-resolution video frame; establishing a brightness mapping model based on the super-resolution video frame, and optimizing the brightness parameter set using an asynchronous batch Bayesian optimization model to convert the super-resolution video frame into a high dynamic range video frame; and performing dynamic image quality optimization on the high dynamic range video frame to obtain a video frame signal for display. The present invention improves the image quality effect through multi-level optimization processing, and adopts a heterogeneous computing architecture to improve processing efficiency.

Description

Technical Field

[0001] The present invention belongs to the technical field of smart televisions, and in particular relates to a smart television image quality optimization method based on an AI chip and a smart television. Background Art

[0002] With the rapid development of display technology, users' requirements for smart TV picture quality experience are constantly increasing. High-quality image display not only requires consideration of resolution reconstruction and dynamic range expansion, but also requires consideration of picture detail optimization and real-time processing performance.

[0003] Currently, smart TVs on the market mainly use traditional digital signal processing methods to optimize image quality, such as linear interpolation, motion estimation and other algorithms. These traditional methods often have poor effects when processing complex scenes and are difficult to meet users' demand for high-quality images.

[0004] In recent years, some high-end TVs have begun to introduce AI technology to enhance picture quality. Such solutions usually use a single convolutional neural network for picture quality processing. Although it can achieve good results in certain specific scenarios, it still has obvious technical limitations. Specifically, the existing solutions use a single neural network structure, which makes it difficult to handle multiple picture quality optimization tasks at the same time, and it is difficult to achieve a good balance between super-resolution reconstruction and detail optimization. The optimization effect is poor, affecting the user experience. Summary of the invention

[0005] In view of this, the present application provides a smart TV image quality optimization method and a smart TV based on an AI chip, which solves the problem in the prior art that a single neural network is difficult to simultaneously handle multiple image quality optimization tasks, resulting in poor image quality optimization effects.

[0006] The embodiment of the present application provides a method for optimizing the image quality of a smart TV based on an AI chip, including:

[0007] Acquire an input video signal, decode and extract frames from the input video signal, and obtain an original video frame sequence;

[0008] Performing super-resolution reconstruction on the original video frame sequence using a pre-established feature processing model to obtain a super-resolution video frame;

[0009] Establishing a brightness mapping model based on the super-resolution video frame, optimizing an initial brightness parameter set indicated by the brightness mapping model using a pre-established asynchronous batch Bayesian optimization model to obtain an optimized brightness parameter set, and converting the super-resolution video frame into a high dynamic range video frame based on the optimized brightness parameter set;

[0010] Dynamic image quality optimization is performed on the high dynamic range video frame to obtain a video frame signal for display.

[0011] In some embodiments, the feature processing model includes a downsampling network and an upsampling network, the downsampling network includes a plurality of first feature layers arranged in cascade order; the upsampling network includes a plurality of second feature layers arranged in cascade order; at least one layer between the first feature layer and the second feature layer has a lateral connection structure;

[0012] The steps performed by the first feature layer of each level include: performing feature extraction and downsampling processing on the original video frame sequence or the feature map output by the first feature layer of the previous level to obtain the feature map output by the first feature layer of the current level;

[0013] The steps performed by the second feature layer of each level include: performing feature fusion and upsampling processing on the feature map output by the first feature layer connected to it and / or the feature map output by the second feature layer of the previous level, so as to obtain the feature map output by the second feature layer of the current level, and to use the feature map output by the second feature layer of the last level as the super-resolution video frame.

[0014] In some embodiments, the first feature layer includes an autoregressive feature extraction module, and the autoregressive feature extraction module includes a convolutional layer and an attention layer;

[0015] The first feature layer further performs the following steps:

[0016] Inputting the feature map output by the original video frame sequence or the first feature layer of the previous level into the convolution layer to perform feature extraction to obtain corresponding spatial features;

[0017] The spatial features are input into the attention layer to calculate adaptive weights based on a preset sliding window mechanism, and the specified area in the feature map output by the original video frame sequence or the first feature layer of the previous level is enhanced based on the adaptive weights to obtain corresponding temporal features, wherein the sliding window mechanism is used to indicate the extraction of contextual information consisting of the current frame and the specified frame in the feature map output by the original video frame sequence or the first feature layer of the previous level, and the spatiotemporal features are used to indicate the temporal correlation of each specified area.

[0018] In some embodiments, the autoregressive feature extraction module further includes a dual-branch network structure, wherein the dual-branch network structure includes a main branch unit and an auxiliary branch unit;

[0019] The dual-branch network structure performs the following steps:

[0020] Reconstructing the time series features through the main branch unit to obtain overall features of each designated area, wherein the overall features include a facial structure and a facial contour for the facial area;

[0021] Extracting features from the time series features by the auxiliary branch unit to obtain detail features of each designated area, wherein the detail features include facial features, facial wrinkles and facial pores for the facial area;

[0022] Introducing an attention mechanism into the residual block of the main branch to adaptively adjust the weight coefficient of each channel according to the importance of the channel;

[0023] Adaptively weighting and fusing the overall feature with the detail feature to generate a corresponding feature map;

[0024] Among them, a preset channel attention mechanism is introduced in the main branch unit and the auxiliary branch unit, and the channel attention mechanism is used to indicate that the weight of the feature channel related to the acquired feature is enhanced when acquiring any overall feature or detail feature.

[0025] In some embodiments, the step of establishing a brightness mapping model based on the super-resolution video frame and optimizing the initial brightness parameter set indicated by the brightness mapping model using a pre-established asynchronous batch Bayesian optimization model to obtain an optimized brightness parameter set includes:

[0026] Based on the brightness distribution characteristics of the super-resolution video frame, a piecewise linear brightness mapping curve is constructed, and an adaptive tone mapping function is determined;

[0027] Calculating an initial brightness parameter set according to a preset local contrast preservation constraint, wherein the local contrast preservation constraint is used to prevent loss of detail information;

[0028] Constructing an asynchronous batch Bayesian optimization model, wherein the asynchronous batch Bayesian optimization model includes a multi-objective Bayesian optimization framework, wherein the multi-objective Bayesian optimization framework includes a Gaussian process regression model for predicting parameter optimization directions, a collection function set, and an asynchronous parallel evaluation mechanism for simultaneously processing multiple sets of parameter configurations, wherein the collection function set includes an expected improvement value and an upper confidence bound value;

[0029] The initial brightness parameter set is input into the asynchronous batch Bayesian optimization model for optimization to obtain an optimized brightness parameter set.

[0030] In some embodiments, converting the super-resolution video frame into a high dynamic range video frame based on the optimized brightness parameter set includes:

[0031] Based on the optimized brightness parameter set, an adversarial network is generated, wherein the adversarial network includes a generator, a discriminator, and a perceptual loss function, wherein the generator is used to instruct the extraction of material texture features and illumination distribution features, the discriminator is used to instruct the authenticity evaluation of multiple local areas divided by the picture, and the perceptual loss function is used to make the video frame to be output by the adversarial network conform to the preset high dynamic range imaging perceptual features;

[0032] Based on the adversarial network, converting the super-resolution video frame into a first video frame;

[0033] Based on the first video frame, analyzing local brightness features of the picture, and calculating a dynamic range extension coefficient based on the local brightness features of the picture, so as to adjust the dynamic range of the first video frame based on the dynamic range extension coefficient to obtain a second video frame;

[0034] The second video frame is optimized for temporal consistency to obtain a high dynamic range video frame.

[0035] In some embodiments, the step of dynamically optimizing the image quality of the high dynamic range video frame to obtain a video frame signal for display includes:

[0036] Encoding the image quality parameters of the high dynamic range video frame to generate parameter codes, and determining an initial population based on the parameter codes, wherein the initial population is used to indicate a parameter combination determined by parameter codes of a plurality of image quality parameters; determining a dynamic constraint range of each parameter in the parameter code sequence according to the picture complexity and motion characteristics of the high dynamic range video frame;

[0037] Constructing a multi-objective fitness function based on the dynamic constraint range, evaluating the population based on the multi-objective fitness function, and outputting high-quality individuals in each population for entering the next generation;

[0038] Performing crossover and mutation operations on the high-quality individuals to obtain an evolved population;

[0039] Based on the parameter encoding of the image quality parameters indicated by the evolved population, a video frame signal for display is determined.

[0040] In some embodiments, the method further comprises:

[0041] Constructing a task dependency graph, the task dependency graph comprising a plurality of processing tasks, the processing tasks comprising a first processing task for indicating obtaining the original video frame sequence, a second processing task for indicating obtaining the super-resolution video frame, a third processing task for indicating obtaining the high dynamic range video frame, and a fourth processing task for indicating obtaining the video frame signal;

[0042] Decomposing the processing tasks in the task dependency graph into subtasks with dependency relationships;

[0043] Based on the task characteristics of the subtasks, evaluating the performance characteristics of the computing units, the computing unit characteristics including the neural network reasoning performance of the NPU, the parallel computing capability of the GPU, and the general computing performance of the CPU;

[0044] Establish a task mapping strategy based on the task characteristics and computing unit performance characteristics;

[0045] According to the task mapping strategy, a double buffer data transmission mechanism is established, wherein the double buffer data transmission mechanism is used to indicate a data flow strategy between computing units;

[0046] The task mapping strategy is optimized by monitoring the status of the computing unit.

[0047] In some embodiments, establishing a task mapping strategy according to the task characteristics and the computing unit performance characteristics includes:

[0048] Allocating the first processing task to the NPU for processing;

[0049] Allocating the third processing task to the GPU for processing;

[0050] Allocating an operation for instructing task scheduling and parameter optimization between the processing tasks to a CPU for processing;

[0051] Establishing a task priority evaluation mechanism, wherein the task priority evaluation mechanism is used to indicate the evaluation of a corresponding priority for each processing task during the processing;

[0052] According to the resource utilization status monitored in real time, the priority of the processing task and the resource allocation strategy are adjusted, and the resource allocation strategy is used to indicate the resource allocation to each computing unit.

[0053] The embodiment of the present application also provides a smart TV, which includes: a preprocessing module, used to obtain an input video signal, decode and extract frames of the input video signal, and obtain an original video frame sequence; a super-resolution reconstruction module, used to perform super-resolution reconstruction on the original video frame sequence using a pre-established feature processing model to obtain a super-resolution video frame; an HDR conversion module, used to establish a brightness mapping model based on the super-resolution video frame, and use a pre-established asynchronous batch Bayesian optimization model to optimize the initial brightness parameter set indicated by the brightness mapping model to obtain an optimized brightness parameter set, and based on the optimized brightness parameter set, convert the super-resolution video frame into a high dynamic range video frame; an image quality optimization module, used to dynamically optimize the image quality of the high dynamic range video frame to obtain a video frame signal for display.

[0054] This application has the following technical effects:

[0055] By adopting the residual neural network structure with autoregressive characteristics for super-resolution reconstruction, the super-resolution reconstruction effect is effectively improved, the accuracy of detail restoration is improved, and the picture details are made clearer and more natural. The SDR to HDR processing solution based on asynchronous batch Bayesian optimization realizes efficient and accurate dynamic range expansion, enhances the picture layering, and improves the efficiency of picture quality parameter optimization, so as to obtain video frame signals for display after dynamic picture quality optimization of high dynamic range video frames. The problem in the prior art that a single neural network is difficult to handle multiple picture quality optimization tasks at the same time, resulting in poor picture quality optimization effect, is solved, and a balance between super-resolution reconstruction and detail optimization is achieved, which significantly improves the overall processing efficiency of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the following is a brief introduction to the drawings required for use in the embodiments. The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present disclosure and are used together with the specification to illustrate the technical solutions of the present disclosure. It should be understood that the following drawings only illustrate certain embodiments of the present disclosure and should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can also be obtained based on these drawings without creative work.

[0057] Figure 1 A flowchart of a method for optimizing image quality of a smart TV based on an AI chip provided in an embodiment of the present application;

[0058] Figure 2 This is a schematic diagram of the structure of the feature processing model in the embodiment of the present application;

[0059] Figure 3 This is a schematic diagram of the structure of the autoregressive feature extraction module in the embodiment of the present application;

[0060] Figure 4 This is a schematic diagram of the architecture of the asynchronous batch Bayesian optimization model in the embodiment of the present application;

[0061] Figure 5 A schematic diagram of the structure of a smart TV provided in an embodiment of the present application. DETAILED DESCRIPTION

[0062] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below in conjunction with the accompanying drawings.

[0063] The present application embodiment provides a method for optimizing the image quality of a smart TV based on an AI chip, see Figure 1 A flowchart of a method for optimizing the image quality of a smart TV based on an AI chip provided in an embodiment of the present application, the method comprising steps S1 to S4. S1: obtaining an input video signal, decoding and frame extracting the input video signal, and obtaining an original video frame sequence;

[0064] Among them, the specific implementation process of S1 is as follows: First, video signal acquisition, the video signal interface support can be to support the HDMI 2.1 digital video interface to achieve a maximum bandwidth of 48Gbps video signal transmission, or to support the digital TV tuner to receive broadcast signals of DVB-T / T2, DVB-C, DVB-S / S2 and other standards, or to support the video stream input of the USB interface and the network interface; signal format recognition, including: automatic detection of the video standard of the input signal (such as PAL, NTSC, ATSC, etc.), identification of video resolution (such as 1920×1080, 3840×2160, etc.), detection of frame rate (such as 24fps, 50fps, 60fps, etc.), identification of color space (such as YUV, RGB, etc.) and bit depth (8bit, 10bit, etc.).

[0065] Second, video decoding processing, including: coding format support, such as support for H.264 / AVC decoding, support for H.265 / HEVC decoding, support for VP9 decoding, support for AV1 decoding, etc.; decoding process, that is, performing syntax analysis on the input compressed video stream, extracting coding parameters, performing entropy decoding, restoring quantization coefficients, performing inverse quantization and inverse transform operations, performing intra-frame prediction or inter-frame prediction reconstruction, and applying decoding loop filters, including deblocking filtering and sample adaptive offset; decoding acceleration, that is, using hardware decoders for acceleration, realizing multi-threaded parallel decoding, and adopting frame-level parallel processing strategies.

[0066] Third, frame extraction processing, including: video frame cache management, that is, establishing a double buffering mechanism to achieve parallel processing of decoding and display, maintaining a frame cache pool, dynamically allocating and recycling memory resources, and realizing timing management of frame sequences to ensure the continuity of video frames; key frame extraction, that is, identifying the types of I frames, P frames, and B frames, giving priority to key frames (I frames), ensuring picture quality, and maintaining GOP (group of pictures) structure information; frame data organization, that is, organizing the decoded video frame data into a unified data structure, including metadata information such as frame type, timestamp, resolution, and recording color components and bit depth information.

[0067] Fourth, quality pre-inspection, including: basic quality assessment, i.e. detecting video noise level, evaluating picture clarity, and analyzing color saturation; scene analysis, i.e. performing scene switching detection, identifying picture types (such as static, motion, portrait, etc.), and evaluating picture complexity; exception handling, i.e. detecting and handling frame loss, identifying and marking damaged video frames, and implementing error recovery mechanisms.

[0068] Fifth, output processing, including: data format conversion, converting the decoded video frames into the standard format required for subsequent processing, performing color space conversion (if necessary), and adjusting the data bit width; frame sequence organization, that is, establishing a frame sequence buffer queue, assigning a unique identifier to each frame, and recording the dependencies between frames; synchronization control, that is, achieving precise timing control of video frames, maintaining the playback clock reference, and ensuring audio and video synchronization.

[0069] S2: Perform super-resolution reconstruction on the original video frame sequence using a pre-established feature processing model to obtain super-resolution video frames.

[0070] Among them, Figure 2 As shown, the feature processing model includes a downsampling network and an upsampling network, the downsampling network includes a plurality of first feature layers arranged in cascade order; the upsampling network includes a plurality of second feature layers arranged in cascade order; at least one layer between the first feature layer and the second feature layer has a lateral connection structure;

[0071] The core architecture of the feature processing model adopts a symmetrical structure design of downsampling network and upsampling network. The downsampling network consists of multiple cascaded first feature layers, which are mainly responsible for feature extraction and compression; the upsampling network contains multiple cascaded second feature layers, which are used for feature reconstruction and enhancement. A horizontal connection structure is set between the two networks. This design can ensure that low-level detail features are effectively utilized in the upsampling process.

[0072] The steps performed by the first feature layer of each level include: performing feature extraction and downsampling processing on the original video frame sequence or the feature map output by the first feature layer of the previous level to obtain the feature map output by the first feature layer of the current level;

[0073] The steps performed by the second feature layer of each level include: performing feature fusion and upsampling processing on the feature map output by the first feature layer connected to it and / or the feature map output by the second feature layer of the previous level, so as to obtain the feature map output by the second feature layer of the current level, and to use the feature map output by the second feature layer of the last level as the super-resolution video frame.

[0074] In the actual operation of the model, the original video frame sequence is first processed by the first feature layer. Each level of the first feature layer performs feature extraction and downsampling operations on the input feature map. Depthwise separable convolution is used in the feature extraction process to reduce the computational complexity while maintaining the effect of feature extraction. Downsampling is achieved through convolution operations with a stride of 2. Each downsampling reduces the spatial dimension of the feature map by half, but increases the number of channels to maintain the amount of information.

[0075] In one embodiment, in the downsampling network, each first feature layer is equipped with an autoregressive feature extraction module, such as Figure 3 As shown in the figure, this is a composite structure consisting of a convolutional layer and an attention layer. The convolutional layer uses a 3×3 convolution kernel to extract spatial features and a 1×1 convolution to adjust the channel dimension. The attention layer introduces an adaptive weight mechanism that can dynamically adjust the weights of different regions of the feature map according to the importance of the content. This design enables the network to better focus on and retain key details in the image.

[0076] In an embodiment of the present application, the first feature layer includes an autoregressive feature extraction module, and the autoregressive feature extraction module includes a convolutional layer and an attention layer.

[0077] It should be noted that another important feature of the feature processing model is the design of its autoregressive feature extraction module. This module has a unique advantage when processing video sequences because it can utilize the temporal correlation information between frames. Through the feature extraction of the convolution layer and the weight adjustment of the attention layer, the system can more accurately capture the motion information and temporal change characteristics in the video. This autoregressive mechanism enables the model to maintain good temporal consistency when processing consecutive video frames, effectively reducing jitter and flicker between frames.

[0078] During the upsampling process, each second feature layer receives the feature map from the corresponding first feature layer and performs feature fusion with the output of the second feature layer of the previous level. The fusion process uses an adaptive weighting method to ensure that features from different levels can be reasonably integrated. This multi-scale feature fusion strategy can take into account both global semantic information and local detail features, thereby achieving better super-resolution reconstruction effects.

[0079] In order to improve the efficiency of feature extraction, the model implements a parallel computing mechanism in each feature layer. On the GPU, features of different channels can be processed simultaneously, greatly improving the computing efficiency. At the same time, the model also adopts a progressive training strategy, first training at a low resolution and then gradually increasing the resolution. This strategy not only speeds up the convergence of the model, but also improves the quality of the final reconstruction results.

[0080] In terms of model structure optimization, a residual learning framework is adopted, and each feature layer contains multiple residual blocks. The use of residual connections not only helps to solve the gradient vanishing problem in deep network training, but also promotes feature reuse and improves the network's expressiveness. In addition, by adding a batch normalization layer to the residual block, the stability and generalization ability of the model training are further improved.

[0081] In this embodiment, the first feature layer also performs the following steps: inputting the feature map output by the original video frame sequence or the first feature layer of the previous level into the convolution layer for feature extraction to obtain corresponding spatial features; inputting the spatial features into the attention layer to calculate adaptive weights based on a preset sliding window mechanism, and enhancing the specified area in the feature map output by the original video frame sequence or the first feature layer of the previous level based on the adaptive weights to obtain corresponding temporal features, wherein the sliding window mechanism is used to indicate the extraction of context information consisting of the current frame and the specified frame in the feature map output by the original video frame sequence or the first feature layer of the previous level, and the spatiotemporal features are used to indicate the temporal correlation of each specified area.

[0082] In the processing flow of the first feature layer, the original video frame sequence or the output of the previous feature layer first enters the convolution layer for feature extraction. This stage uses multi-scale convolution operations, including a combination of standard 3×3 convolution, dilated convolution, and depthwise separable convolution to capture spatial features under different receptive fields. The extracted spatial features are then input to the attention layer, where the system calculates adaptive weights based on a preset sliding window mechanism. The sliding window mechanism extracts contextual features by setting a fixed-size time window and analyzing the information of the current frame and several frames before and after it at the same time. This mechanism can effectively capture the temporal correlation in video sequences, especially for video clips with intense motion or complex scenes.

[0083] The adaptive weights calculated by the attention layer are used to enhance the specified areas in the original feature map. This enhancement process is selective, and the system assigns different attention weights according to the importance of the content. For example, for portrait areas, facial details will receive higher weights; for motion scenes, the edge areas of moving objects will receive special attention. This adaptive feature enhancement mechanism ensures that important visual information can be fully preserved in subsequent processing.

[0084] Specifically, in one of the embodiments, the autoregressive feature extraction module further includes a dual-branch network structure, and the dual-branch network structure includes a main branch unit and an auxiliary branch unit;

[0085] The dual-branch network structure performs the following steps: reconstructing the temporal features through the main branch unit to obtain the overall features of each specified area, wherein the overall features include the facial structure and facial contour of the facial area; extracting the temporal features through the auxiliary branch unit to obtain the detailed features of each specified area, wherein the detailed features include facial features, facial wrinkles and facial pores of the facial area; introducing an attention mechanism (such as a channel attention mechanism) in the residual block of the main branch to adaptively adjust the weight coefficient of each channel according to the importance of the channel; adaptively weightingly fusion the overall features and the detailed features to generate a corresponding feature map; a preset channel attention mechanism is introduced in the main branch unit and the auxiliary branch unit, and the channel attention mechanism is used to indicate that when acquiring any overall feature or detailed feature, the weight of the feature channel related to the acquired feature is enhanced.

[0086] The feature processing model adopts an innovative dual-branch network structure, including a main branch unit and an auxiliary branch unit. The main branch unit is mainly responsible for feature reconstruction, focusing on extracting and reconstructing the overall features of the image, such as the structure and contour of the facial area. This branch uses a larger receptive field and a deep network structure to ensure that the global semantic information of the image can be accurately grasped. The auxiliary branch unit focuses on the extraction of detailed features, including fine texture information such as facial features, wrinkles, and pores. This branch uses a smaller receptive field and a multi-scale feature extraction strategy to ensure accurate capture of detailed information.

[0087] In the residual block of the main branch, a channel attentiveness mechanism is introduced, which can adaptively adjust the weight coefficients according to the importance of different channels. By learning the correlation between channels, the system can strengthen the influence of important feature channels while suppressing the interference of irrelevant or redundant channels. This adaptive channel weight adjustment ensures that the essential characteristics of the image can be better maintained during feature reconstruction.

[0088] In order to achieve more accurate feature extraction, a preset channel attention mechanism is introduced in both the main branch unit and the auxiliary branch unit. This mechanism dynamically enhances the weights of feature channels related to the current task during feature extraction. For example, when extracting facial details, channels related to skin texture will receive higher weights; when processing motion areas, channels related to edge and motion information will be given priority.

[0089] Finally, the system adaptively weights and fuses the overall features extracted by the main branch with the detail features extracted by the auxiliary branch. The fusion process uses a dynamic weight allocation strategy to determine the fusion ratio of the two features based on the complexity and importance of the image content. This adaptive fusion mechanism ensures that while maintaining the overall structure of the image, the detail information can be retained to the greatest extent. The fused feature map contains both accurate overall structural information and rich local details, providing a high-quality feature foundation for subsequent super-resolution reconstruction.

[0090] In actual operation, the parallel processing mechanism of the dual-branch network not only improves computational efficiency, but also enhances the model's feature extraction capabilities. By rationally allocating computing resources, the system can achieve optimal feature extraction effects with limited computing overhead. At the same time, the weight parameters of each branch are continuously adjusted through the feedback mechanism to ensure that the entire feature processing process can adapt to different types of video content and maintain a stable processing effect.

[0091] For example, suppose a frame of 1080p resolution (1920×1080) video is input, and the content of the picture is a person giving a speech against a complex background. The system first constructs a feature pyramid and downsamples the original frame to obtain three lower resolution feature maps of 960×540, 480×270, and 240×135. For each resolution level, the system uses two 3×3 convolutional layers to extract spatial features, and uses the Sobel operator to calculate the edge gradient map to retain detail information at different scales.

[0092] After the feature pyramid is constructed, the autoregressive feature extraction module begins to process the timing information. The system maintains a 7-frame processing window, centered on the current frame, while taking into account the information of the 3 frames before and after. Taking the speaker's facial area as an example, by calculating the optical flow information between adjacent frames, the system can accurately track subtle changes in facial expressions. The spatiotemporal attention mechanism automatically assigns higher feature weights to the facial area because this is the most important content area in the picture. At the same time, static areas in the background (such as walls, furniture, etc.) are given lower weights because these areas have weaker temporal correlation.

[0093] Entering the advantage weighted residual learning stage, the system divides the features into two processing branches. When processing human faces, the main branch is responsible for reconstructing the basic facial structure and contours, while the auxiliary branch focuses on extracting facial details, such as wrinkles, pores and other high-frequency features. Through the channelization mechanism, the system can adaptively adjust the importance of different feature channels. For example, when processing skin color, the feature channels related to skin color will be strengthened; when processing hair texture, the weight of the feature channel representing texture will be enhanced.

[0094] Finally, in the feature fusion and upsampling stage, the system uses a progressive strategy to increase the resolution of the feature map to its original size. First, the 240×135 feature map is enlarged to 480×270 and fused with the jump connection features of this scale. An adaptive weighting method is used during fusion. For important areas such as the face of a person, a higher weight is given to the jump connection features to retain more original details; for secondary areas such as the background, the upsampled features are used more. This process is carried out step by step until the resolution is restored to 1080p. After each level of upsampling, the system adjusts the importance of the features by recalibrating the feature channels to ensure that the final output image maintains both overall clarity and good detail features.

[0095] Through this multi-scale, multi-stage processing method, the system can effectively improve the image quality while retaining the original image details. In particular, for important areas such as the face of a person, it can achieve more detailed image quality optimization, making the image present a better visual effect.

[0096] Exemplarily, according to television-related standards, the super-resolution reconstruction multiple ranges from 1.1 to 16 times, preferably:

[0097] 1.1x to 2x: used for 720p to 1080p conversion scenarios;

[0098] 2x to 4x: for 1080p to 4K conversion scenarios;

[0099] 4x to 8x: used for 1080p to 8K conversion scenarios;

[0100] 8x to 16x: For future expansion needs with higher resolutions.

[0101] S3: Establishing a brightness mapping model based on the super-resolution video frame, and optimizing the initial brightness parameter set indicated by the brightness mapping model using a pre-established asynchronous batch Bayesian optimization model to obtain an optimized brightness parameter set, and converting the super-resolution video frame into a high dynamic range video frame based on the optimized brightness parameter set.

[0102] In step S3, a brightness mapping model is established based on the super-resolution video frame, and the initial brightness parameter set indicated by the brightness mapping model is optimized using a pre-established asynchronous batch Bayesian optimization model to obtain an optimized brightness parameter set, including the following steps:

[0103] Based on the brightness distribution characteristics of the super-resolution video frame, a piecewise linear brightness mapping curve is constructed, and an adaptive tone mapping function is determined; an initial brightness parameter set is calculated according to a preset local contrast preservation constraint, and the local contrast preservation constraint is used to prevent the loss of detail information; an asynchronous batch Bayesian optimization model is constructed, and the asynchronous batch Bayesian optimization model includes a multi-objective Bayesian optimization framework, and the multi-objective Bayesian optimization framework includes a Gaussian process regression model for predicting parameter optimization directions, an acquisition function set, and an asynchronous parallel evaluation mechanism for simultaneously processing multiple groups of parameter configurations, and the acquisition function set includes an expected improvement value and a confidence upper bound value; the initial brightness parameter set is input into the asynchronous batch Bayesian optimization model for optimization to obtain an optimized brightness parameter set.

[0104] The converting of the super-resolution video frame into a high dynamic range video frame based on the optimized brightness parameter set comprises the following steps:

[0105] Based on the optimized brightness parameter set, an adversarial network is generated, the adversarial network includes a generator, a discriminator and a perceptual loss function, the generator is used to indicate the extraction of material texture features and lighting distribution features, the discriminator is used to indicate the authenticity evaluation of multiple local areas divided by the picture, and the perceptual loss function is used to make the video frame to be output by the adversarial network conform to the preset high dynamic range imaging perceptual features; based on the adversarial network, the super-resolution video frame is converted into a first video frame; based on the first video frame, the local brightness features of the picture are analyzed, and the dynamic range expansion coefficient is calculated based on the local brightness features of the picture, so as to adjust the dynamic range of the first video frame based on the dynamic range expansion coefficient to obtain a second video frame; the second video frame is optimized for time domain consistency to obtain a high dynamic range video frame.

[0106] Combined with the above process, the HDR conversion process starts with the construction of the brightness mapping model. Based on the brightness distribution characteristics of the super-resolution video frame, the system constructs a piecewise linear brightness mapping curve. This segmented processing method can more accurately adapt to the mapping requirements of different brightness ranges and avoid the loss of details caused by a single mapping function. At the same time, the system establishes an adaptive tone mapping function, which can dynamically adjust the mapping parameters according to the scene content to ensure that the natural performance of the picture is maintained while the brightness is improved.

[0107] When determining the initial brightness parameters, the system introduces a local contrast preservation constraint mechanism. This constraint ensures that while the overall dynamic range is improved, the detail information of the local area is not lost. Specifically, the system calculates and maintains the contrast relationship between adjacent pixels, which is important for preserving texture details and edge features. The construction of the initial parameter set takes into account multiple factors, including scene brightness distribution, local contrast characteristics, and human visual perception characteristics.

[0108] In order to optimize the brightness parameters, the system builds an innovative asynchronous batch Bayesian optimization model. Figure 4 As shown in the figure, the core of the model is the multi-objective Bayesian optimization framework, which includes the Gaussian process regression model for predicting the parameter optimization direction. The Gaussian process regression model can predict the optimal parameter adjustment direction by learning the distribution characteristics of the parameter space. At the same time, the system uses a collection function set to balance the relationship between exploration and utilization, including two key indicators: expected improvement value and confidence upper bound value.

[0109] The introduction of the asynchronous parallel evaluation mechanism greatly improves the optimization efficiency. This mechanism allows multiple sets of parameter configurations to be evaluated simultaneously, making full use of parallel computing resources. The system inputs the initial brightness parameter set into this optimization model, and finally obtains the optimized brightness parameter set through multiple rounds of iterative optimization. This optimization process not only takes into account the image quality improvement effect, but also takes into account the computing efficiency and convergence speed.

[0110] After obtaining the optimized set of brightness parameters, the system builds a special generative adversarial network (GAN) to handle HDR conversion. The network contains a carefully designed generator, discriminator, and perceptual loss function. The generator is responsible for extracting material texture features and lighting distribution features. This dual feature extraction mechanism ensures that material details and lighting effects can be maintained while improving the dynamic range. The discriminator evaluates the authenticity of local areas of the picture. This local evaluation strategy helps to maintain the authenticity and naturalness of local details.

[0111] The design of the perceptual loss function is based on the perceptual characteristics of the human eye to HDR images, ensuring that the generated image conforms to the laws of visual perception. Through this GAN network, the system converts the super-resolution video frame into a preliminary HDR image (the first video frame). Subsequently, the system performs an in-depth analysis of the local brightness characteristics of the first video frame and calculates the dynamic range expansion coefficient. The calculation of this coefficient takes into account the brightness distribution characteristics of the local area to ensure the adaptability of the expansion process.

[0112] Finally, the second video frame with dynamic range adjustment is optimized for temporal consistency. This step is particularly important because it ensures a smooth transition between consecutive video frames and avoids flickering or jitter in the temporal dimension. The optimization process takes into account the temporal correlation between adjacent frames and achieves a stable HDR effect through techniques such as temporal filtering and motion compensation. This comprehensive processing flow ultimately produces high-quality HDR video frames that ensure both the visual impact of the picture and the continuity and stability in the temporal domain.

[0113] For example, consider a typical scene: an indoor living room scene, which includes a bright view outside the window and a darker indoor environment. The system first analyzes the brightness distribution characteristics of the input SDR video frame. Through histogram analysis, it is found that the brightness values ​​of the scene are mainly concentrated in two intervals: the part outside the window is concentrated in 180-220 (taking 8-bit quantization as an example), and the indoor part is mainly distributed between 40-100. Based on this, the system constructs a segmented brightness mapping curve, using a larger mapping slope for the dark area of ​​the room to improve the visibility of details, and a smaller slope for the bright area outside the window to avoid overexposure. At the same time, a smooth transition interval is set at the junction of light and dark to ensure that the brightness changes naturally.

[0114] In the asynchronous batch Bayesian optimization stage, the system evaluates multiple sets of HDR parameter configurations simultaneously. For example, the system will test different combinations of brightness expansion parameters in parallel: {dark enhancement: 1.8, bright compression: 0.7}, {dark enhancement: 2.0, bright compression: 0.8}, etc. The Gaussian process regression model is used to predict the optimization direction of each set of parameters, and the system gradually narrows the search range for the optimal parameters. During the evaluation process, focus on whether the material details of the indoor furniture are displayed reasonably and whether the layering of the clouds outside the window remains natural. In the end, the system may determine that the optimal parameter combination is {dark enhancement: 1.9, bright compression: 0.75}.

[0115] Entering the conditional generative adversarial network processing stage, the system uses optimized parameter configuration for HDR conversion. The generator network adopts an improved U-Net structure. When processing indoor areas, the shallow features of the network are mainly responsible for extracting material texture information, while the deep features focus on the overall lighting distribution. The discriminator adopts the PatchGAN architecture to divide the picture into multiple local areas for authenticity assessment. Especially in the light and dark transition areas around the windows, the discriminator will focus on the naturalness of the tonal transition. By introducing the perceptual loss function, it is ensured that the converted picture visually conforms to the perceptual characteristics of the human eye for real HDR content.

[0116] In the dynamic range adaptive adjustment, the system adopts an adaptive processing strategy for different areas. For indoor areas such as sofas and carpets, the system calculates the local contrast gain coefficient to appropriately improve the visibility of material details; for the sky and buildings outside the window, dynamic range compression is used to retain the cloud layers while avoiding detail loss in the highlight area. This process takes into account the spatial continuity of the picture content to ensure a natural transition between processing effects between areas.

[0117] Finally, in the temporal consistency optimization stage, the system focuses on the stability of the HDR effect between consecutive frames. For example, when the outdoor light changes due to the movement of clouds, the system will smoothly adjust the HDR parameters to avoid sudden changes in the brightness of the picture. For slow-moving objects indoors, the system will track their movement trajectory to ensure the consistency of the HDR enhancement effect. At the same time, when the screen switches, the system will gradually adjust the HDR parameters in about 4-5 frames to achieve smooth scene transitions.

[0118] Therefore, in step S3, through this multi-level, adaptive HDR processing flow, the system can convert the original SDR content into an HDR picture with a wider dynamic range while maintaining the naturalness and temporal stability of the picture. Whether it is a bright view outside the window or a dark indoor environment, it can present rich details and layering.

[0119] S4: Dynamically optimize the image quality of the high dynamic range video frame to obtain a video frame signal for display.

[0120] Specifically, step S4 includes:

[0121] Encoding the image quality parameters of the high dynamic range video frame to generate parameter codes, and determining an initial population based on the parameter codes, wherein the initial population is used to indicate a parameter combination determined by the parameter codes of the plurality of image quality parameters;

[0122] Determining the dynamic constraint range of each parameter in the parameter coding sequence according to the picture complexity and motion characteristics of the high dynamic range video frame;

[0123] Constructing a multi-objective fitness function based on the dynamic constraint range, evaluating the population based on the multi-objective fitness function, and outputting high-quality individuals in each population for entering the next generation;

[0124] Performing crossover and mutation operations on the high-quality individuals to obtain an evolved population; and determining a video frame signal for display based on parameter encoding of image quality parameters indicated by the evolved population.

[0125] Specifically, the first step in image quality optimization is to encode the image quality parameters of high dynamic range video frames. The system uses a multi-dimensional parameter encoding scheme to map various image quality parameters (such as brightness, contrast, sharpness, color saturation, etc.) into standardized numerical sequences. This encoding method not only facilitates subsequent optimization processing, but also effectively expresses the relationship between parameters. Based on these parameter encodings, the system generates an initial population, and each individual represents a specific set of image quality parameter combinations. The diversity of the initial population is ensured by combining random sampling and prior knowledge to ensure that the search space is fully covered.

[0126] When determining the dynamic constraint range of the parameters, the system will deeply analyze the characteristics of the high dynamic range video frame. The evaluation of picture complexity includes multiple dimensions such as texture complexity, edge density, color change, etc.; motion characteristics are obtained through motion vector analysis, scene switching detection, etc. Based on these feature analysis results, the system sets an adaptive constraint range for each parameter. For example, in scenes with intense motion, the range of change of the sharpening parameters will be appropriately narrowed to avoid artifacts; while in static scenes, a larger parameter adjustment space can be allowed.

[0127] The construction of a multi-objective fitness function is the core of the optimization process. This function comprehensively considers multiple evaluation indicators, including objective image quality indicators (such as PSNR, SSIM), subjective perception characteristics (such as clarity, naturalness), and computational efficiency. These indicators are assigned different weights to form a comprehensive scoring mechanism. The system uses this fitness function to evaluate each individual in the population and selects high-quality individuals to enter the next generation based on the scoring results. The selection process adopts the multi-objective Pareto optimization principle to ensure a good balance between different objectives.

[0128] The evolutionary operation of high-quality individuals includes two main steps: crossover and mutation. The crossover operation combines the excellent features of different individuals to produce offspring with new characteristics. The system adopts an adaptive crossover strategy to dynamically adjust the crossover probability and crossover mode according to the fitness value of the parent individuals. The mutation operation introduces new feature combinations by randomly changing certain parameters of the individuals to maintain the diversity of the population. The mutation intensity will be dynamically adjusted according to the optimization process, maintaining a large exploration range in the early stage and gradually converging to local fine adjustments in the later stage.

[0129] In order to improve the optimization efficiency, the system uses a parallel computing framework to process population evaluation. Multiple individuals can be evaluated on different processing units at the same time, which greatly reduces the calculation time. At the same time, the system also maintains an elite individual library and saves the historical optimal solution to ensure that the optimization process does not lose the discovered excellent solution.

[0130] After each generation of evolution is completed, the system will evaluate the convergence of the population. If the optimal solution for multiple consecutive generations does not improve significantly, or reaches the preset number of iterations, the optimization process will terminate. Finally, the system selects the optimal parameter combination from the evolved population to generate a video frame signal for display. This signal not only has excellent objective image quality indicators, but also ensures a good subjective viewing experience.

[0131] The entire optimization process forms a closed-loop adaptive system. Through the real-time feedback mechanism, the system can dynamically adjust the parameters of each link according to the optimization effect, including population size, crossover mutation probability, termination conditions, etc. This adaptive mechanism ensures that the optimization process can adapt to different types of video content and always maintain a stable optimization effect. At the same time, the system will also record the parameter combinations and their effects during the optimization process to form a knowledge base to provide a reference for subsequent optimization tasks.

[0132] The following is a specific example to explain the dynamic image quality optimization process in detail:

[0133] Consider a live sports event scene, which includes fast-moving athletes, spectators, and stadium environments. In the parameter encoding stage, the system first initializes the image quality parameter chromosome based on the scene characteristics. For example, for this sports event scene, a typical parameter individual may be encoded as: sharpness enhancement is set to 180 (moderate to high, highlighting the outline of the athlete), color saturation is set to 150 (maintaining the vividness of sportswear), contrast gain is set to 160 (enhancing the layering of the picture), and noise suppression threshold is set to 120 (balancing detail preservation and noise removal). The system generates 50 similar parameter combinations as the initial population, each individual is slightly different to cover a reasonable parameter search space.

[0134] In the construction of the dynamic search space, the system adjusts the parameter range according to the real-time picture characteristics. When fast-moving picture clips (such as sprint races) are detected, the system will narrow the search range of the sharpness parameter to 120-200 to avoid motion smear caused by over-sharpening; at the same time, the range of the noise suppression threshold is expanded to 100-180 to better handle motion blur. For background areas such as the auditorium, the system allows color saturation to search in a wider range (100-200) to create a more vivid on-site atmosphere. These search ranges will be dynamically adjusted as the content of the picture changes.

[0135] Entering the fitness evaluation stage, the system conducts a multi-dimensional evaluation of each parameter combination. Taking a close-up shot of an athlete as an example, the system will focus on evaluating the clarity score of the athlete's outline (weight 0.4), the accuracy of skin color restoration (weight 0.3), the degree of preservation of motion details (weight 0.2) and the overall noise level (weight 0.1). Through the fast evaluation algorithm, the system can complete the evaluation of a generation of populations within 16.7ms. Using the tournament selection mechanism, the best 2 individuals are selected from 5 random individuals to enter the next generation each time, while retaining the current best 3 individuals.

[0136] During the genetic operation phase, the system performs intelligent crossover and mutation operations. For example, when two excellent individuals are found to be complementary in sharpness and noise suppression parameters (one with high sharpness and low noise, and the other with the opposite), the system will prioritize these parameter positions for crossover operations. The mutation operation will refer to the gradient information of the picture and prioritize small adjustments to motion-related parameters (such as sharpness and noise suppression) in areas with intense motion. The system will also perform local searches after each generation of population updates to fine-tune and optimize the best performing parameter combinations.

[0137] Finally, in the real-time application stage, the system applies the optimized parameters to video processing. When the lens switches from a close-up of an athlete to a panoramic view, the system smoothly transitions the parameter settings within about 8-10 frames: gradually reducing the sharpness enhancement intensity (because the panoramic view has low requirements for details), increasing the color saturation (to highlight the overall atmosphere of the stadium), and appropriately reducing the contrast gain (to avoid the distant view being too rigid). This gradual parameter adjustment ensures the continuity of the picture quality and avoids visual discomfort caused by sudden changes.

[0138] Therefore, in this step, through this adaptive genetic algorithm optimization process, the system can adjust the image quality parameters in real time to keep it in the best state at all times. Whether it is a fast-moving close-up shot or a slowly panning panoramic picture, it can present the image quality that best suits the current scene characteristics. This optimization not only improves the viewing experience, but also can automatically adapt to different types of sports event content, showing strong scene adaptability.

[0139] In some embodiments, the present application also includes the following steps:

[0140] Constructing a task dependency graph, the task dependency graph comprising a plurality of processing tasks, the processing tasks comprising a first processing task for indicating obtaining the original video frame sequence, a second processing task for indicating obtaining the super-resolution video frame, a third processing task for indicating obtaining the high dynamic range video frame, and a fourth processing task for indicating obtaining the video frame signal;

[0141] Decomposing the processing tasks in the task dependency graph into subtasks with dependency relationships;

[0142] Based on the task characteristics of the subtasks, evaluating the performance characteristics of the computing units, the computing unit characteristics including the neural network reasoning performance of the NPU, the parallel computing capability of the GPU, and the general computing performance of the CPU;

[0143] Establish a task mapping strategy based on the task characteristics and computing unit performance characteristics;

[0144] According to the task mapping strategy, a double buffer data transmission mechanism is established, wherein the double buffer data transmission mechanism is used to indicate a data flow strategy between computing units;

[0145] The task mapping strategy is optimized by monitoring the status of the computing unit.

[0146] Specifically, the task dependency graph is constructed using a directed acyclic graph (DAG) structure, which clearly depicts the dependencies and execution order between tasks in the entire video processing flow. This graph structure not only shows the order of tasks, but also reflects the path of data flow, providing an important basis for parallel processing and resource scheduling. In actual implementation, the system uses an adjacency matrix to store the connection relationship between task nodes and determines the execution order of tasks through topological sorting.

[0147] The first processing task is the starting point of the entire processing flow and is mainly responsible for the acquisition and decoding of video signals. This task node contains multiple submodules, such as signal reception, format conversion, decoding processing, etc. The system will assign priorities to these submodules and establish corresponding execution queues. In order to improve processing efficiency, this task node adopts a pipeline structure, allowing different submodules to process different data blocks at the same time.

[0148] The second processing task focuses on super-resolution processing, which directly depends on the output of the first processing task. This task node builds a complete super-resolution processing pipeline, including steps such as feature extraction, network inference, and image reconstruction. The system uses a task splitting mechanism to decompose large super-resolution processing tasks into multiple subtasks that can be executed in parallel to improve processing efficiency. At the same time, the task node also includes a cache mechanism for storing intermediate processing results.

[0149] The third processing task is responsible for HDR conversion processing, which uses the output of the second processing task as input. This task node implements a complex HDR processing flow, including brightness mapping, tone mapping, and detail enhancement operations. The system configures an adaptive resource allocation strategy for this task node, which can dynamically adjust the allocation of computing resources according to the complexity of the processing content. In addition, the task node also includes a quality monitoring module for real-time evaluation of processing effects.

[0150] The fourth processing task is the end point of the entire processing flow and is responsible for the generation and output of the final video frame signal. This task node integrates the results of the previous processing and performs the final quality optimization and format conversion. The system implements a multi-level buffering mechanism in this task node to ensure the stability and continuity of the output signal. At the same time, the task node also contains an adaptation layer to handle the special needs of different display devices.

[0151] An important feature of the task dependency graph is its flexible parallel processing capability. The system can process multiple frames at different time points at the same time as long as there is no direct dependency between them. For example, when the second processing task is processing the nth frame, the first processing task can simultaneously process the n+1th frame. This parallel mechanism significantly improves the overall processing efficiency.

[0152] In addition, the system also implements a dynamic load balancing mechanism in the task dependency graph. By real-time monitoring of the execution status and resource usage of each processing task, the system can dynamically adjust the task priority and resource allocation. When a task node encounters a processing bottleneck, the system will automatically adjust the resource allocation strategy to ensure the balance of the entire processing flow.

[0153] To ensure processing reliability, the task dependency graph also includes a complete exception handling mechanism. Each task node is equipped with an error detection and recovery module. When a processing exception occurs, the system can quickly locate the problem and recover. At the same time, the system also maintains a processing log to record the execution status and performance data of each task node, providing a basis for system optimization.

[0154] In actual operation, the system will dynamically adjust the structure of the task dependency graph according to different application scenarios. For example, in high-performance processing mode, the system may enable more parallel processing units; while in low-power mode, some processing steps may be simplified to reduce resource consumption. This flexible architecture design ensures that the system can adapt to different application requirements.

[0155] In one embodiment, establishing a task mapping strategy according to the task characteristics and the computing unit performance characteristics includes:

[0156] Allocating the first processing task to the NPU for processing;

[0157] Allocating the third processing task to the GPU for processing;

[0158] Allocating an operation for instructing task scheduling and parameter optimization between the processing tasks to a CPU for processing;

[0159] Establishing a task priority evaluation mechanism, wherein the task priority evaluation mechanism is used to indicate the evaluation of a corresponding priority for each processing task during the processing;

[0160] According to the resource utilization status monitored in real time, the priority of the processing task and the resource allocation strategy are adjusted, and the resource allocation strategy is used to indicate the resource allocation to each computing unit.

[0161] The task scheduling system first performs a fine-grained decomposition of the task dependency graph. Each main processing task is decomposed into multiple subtasks with clear dependencies to form a finer-grained task network. For example, a super-resolution processing task can be decomposed into subtasks such as feature extraction, network reasoning, and image reconstruction. The system analyzes the data dependencies of these subtasks and constructs a detailed task execution graph, providing a basis for subsequent task scheduling.

[0162] When evaluating the performance characteristics of computing units, the system will conduct a comprehensive analysis of the processing capabilities of NPU, GPU, and CPU. For NPU, the focus is on evaluating the reasoning performance of its neural network model, including indicators such as computing latency, throughput, and energy consumption; for GPU, the main focus is on its parallel computing capabilities, including the number of computing cores, memory bandwidth, and scheduling efficiency; for CPU, its general computing performance is evaluated, including instruction execution efficiency, cache hit rate, and task switching overhead. These performance characteristic data are used for subsequent task mapping decisions.

[0163] Based on the task characteristics and computing unit performance characteristics, the system establishes a dynamic task mapping strategy. This strategy takes into account the computing characteristics, data dependencies, and hardware resource characteristics of the task, and selects the most suitable execution unit for each subtask. For example, tasks that require a large number of matrix operations are preferentially assigned to the GPU, while tasks with more control flow are assigned to the CPU for processing.

[0164] To optimize data transmission efficiency, the system implements a double-buffered data transmission mechanism. This mechanism establishes an efficient data channel between computing units, allowing data transmission to proceed in parallel with computing processing. When one buffer is processing data, the other buffer can read or write data at the same time, effectively reducing the delay caused by data transmission.

[0165] The system continuously optimizes the task mapping strategy through a real-time monitoring mechanism. The monitoring content includes indicators such as the load status, processing delay, and energy consumption of each computing unit. Based on these monitoring data, the system can dynamically adjust the task allocation plan to ensure the balance and efficiency of resource utilization. When a performance bottleneck is found in a computing unit, the system will automatically adjust the task allocation and transfer part of the load to other available resources.

[0166] In the specific task allocation, the system assigns the first processing task (video frame acquisition and processing) to the NPU, making full use of its efficient neural network reasoning capabilities; assigns the third processing task (HDR processing) to the GPU, using its powerful parallel computing capabilities to handle complex image conversion operations; and assigns control operations such as task scheduling and parameter optimization to the CPU, leveraging its advantages in general computing and task coordination.

[0167] The task priority evaluation mechanism dynamically evaluates the importance of each processing task at runtime. Evaluation factors include the timeliness requirements of the task, resource consumption, and the impact on the final output quality. Based on these factors, the system assigns a corresponding priority to each task to ensure that critical tasks can obtain sufficient processing resources.

[0168] The system also establishes a real-time resource utilization monitoring mechanism. By continuously monitoring the load status, memory usage, energy consumption level and other indicators of each computing unit, the system can promptly detect imbalances in resource usage. Based on the monitoring results, the system will dynamically adjust the task priority and resource allocation strategy to ensure the optimal use of system resources.

[0169] In order to improve the adaptability of the system, the resource allocation strategy adopts a flexible scheduling mechanism. When some tasks require more computing resources, the system can dynamically adjust the resource allocation ratio; when the processing load is light, the operating frequency of some computing units can be reduced to save energy. This flexible resource management strategy ensures that the system can maintain efficient operation under different workloads.

[0170] In actual operation, the system also implements a task prediction mechanism. By analyzing historical processing data and current task queues, the system can predict future processing needs and adjust resource allocation strategies in advance. This forward-looking resource management method helps reduce task scheduling delays and improve overall processing efficiency. At the same time, the system also maintains a detailed performance log to record the effects of various optimization adjustments, providing a basis for subsequent system optimization.

[0171] The following is a specific example to explain the heterogeneous computing task scheduling process in detail:

[0172] Consider a 4K TV real-time playback scenario, the system needs to handle tasks such as image quality optimization, HDR conversion, and motion compensation at the same time. In the task decomposition stage, the system first decomposes the entire processing flow into atomic tasks. For example, the super-resolution reconstruction task is decomposed into three subtasks: feature extraction, autoregressive processing, and upsampling; HDR conversion is decomposed into subtasks such as brightness mapping, GAN processing, and time domain optimization. Through analysis, the system constructs a complete task dependency graph: the feature extraction task must be performed before the autoregressive processing, while the HDR conversion can be processed in parallel with motion compensation. At the same time, the system calculates the resource requirements of each atomic task, such as feature extraction requires 2GB of video memory and 30% of NPU computing power, GAN processing requires 3GB of video memory and 50% of GPU resources, etc.

[0173] In the analysis of computing unit characteristics, the system conducts a detailed evaluation of the NPU, GPU, and CPU on the device. For example, the NPU on this TV has 8TOPS computing power and is particularly good at 8-bit fixed-point quantization of neural network operations, but has limited support for irregular convolutions; the GPU provides 384 CUDA cores and a shared memory bandwidth of 200GB / s, which is particularly suitable for parallel image processing tasks; the CPU is an 8-core processor with powerful general computing power and flexible task scheduling capabilities. Through actual testing, the system found that the NPU performs best when processing standard convolutions, with a delay of only 2ms; the GPU performs outstandingly when processing HDR conversions, with a single processing delay of about 4ms; and the CPU is most efficient when coordinating tasks and processing complex logic.

[0174] Entering the dynamic task mapping stage, the system formulates the optimal allocation plan based on task characteristics and hardware capabilities. For example, the feature extraction task is assigned to the NPU because the task mainly involves standard convolution operations; the GAN processing of HDR conversion is assigned to the GPU to take advantage of its parallel computing advantages; task scheduling and parameter optimization are assigned to the CPU. The system also considers data transmission overhead and tries to assign data-related tasks to the same computing unit. When a task is detected to be on the critical path (such as autoregressive processing), the system will increase its priority and even preempt resources for other non-critical tasks if necessary.

[0175] In the optimization of asynchronous data transmission, the system implements an efficient data stream processing mechanism. Taking video frame processing as an example, when the NPU is processing the feature extraction of the current frame, the system has already started transferring the next frame data from the memory to the NPU dedicated memory, while the GPU is processing the HDR conversion of the previous frame. Through this pipeline mechanism, the system is able to achieve a perfect overlap of computing and data transmission. Especially when processing 4K resolution video, the system divides each frame of the image into multiple 2048×1080 size data blocks, and these block sizes are optimized to maximize cache hit rate and reduce memory access latency.

[0176] Finally, in the real-time monitoring and adjustment stage, the system continuously monitors the performance status of each computing unit. Assuming that the system detects that the NPU temperature rises when processing feature extraction, resulting in a decrease in frequency, and the processing delay increases from 2ms to 3ms, the system will immediately start the load balancing mechanism: temporarily transfer some feature extraction tasks to the GPU for processing, while reducing the priority of non-critical tasks to ensure the real-time performance of the main processing flow. If it is detected that the computing load of a certain video clip is particularly high (such as a scene containing a lot of fast motion), the system will automatically adjust the degree of parallelism, such as increasing the number of data blocks that the GPU participates in processing. When an abnormal situation occurs (such as a temporary failure of a computing unit), the system can complete task migration at the millisecond level to ensure that the processing flow is not interrupted.

[0177] Through this sophisticated management of heterogeneous computing resources, the system can fully utilize the advantages of various computing resources to achieve optimal processing performance. Even in heavy-load scenarios such as processing 4K high-definition video, the system can still maintain a stable processing frame rate and excellent image quality optimization. This dynamic and adaptive scheduling mechanism not only improves the processing efficiency of the system, but also ensures stability and reliability under various working conditions.

[0178] Figure 5 The present invention provides a schematic diagram of the structure of a smart TV according to an embodiment of the present invention. The present invention also provides a smart TV according to an embodiment of the present invention. The smart TV 200 includes:

[0179] The preprocessing module 201 is used to obtain an input video signal, decode and extract frames of the input video signal, and obtain an original video frame sequence;

[0180] A super-resolution reconstruction module 202 is used to perform super-resolution reconstruction on the original video frame sequence using a pre-established feature processing model to obtain a super-resolution video frame;

[0181] An HDR conversion module 203 is used to establish a brightness mapping model based on the super-resolution video frame, and optimize the initial brightness parameter set indicated by the brightness mapping model using a pre-established asynchronous batch Bayesian optimization model to obtain an optimized brightness parameter set, and convert the super-resolution video frame into a high dynamic range video frame based on the optimized brightness parameter set;

[0182] The image quality optimization module 204 performs dynamic image quality optimization on the high dynamic range video frame to obtain a video frame signal for display.

[0183] The smart TV of the embodiment of the present application can execute the method provided by the embodiment of the present application, and the implementation principle is similar. The actions performed by each module in the smart TV of each embodiment of the present application correspond to the steps in the method of each embodiment of the present application. For the detailed functional description of each module of the smart TV, please refer to the description in the corresponding method shown in the previous text, which will not be repeated here.

Claims

1. A method for optimizing the image quality of a smart TV based on an AI chip, characterized in that: The method comprises: Acquire an input video signal, decode and extract frames from the input video signal, and obtain an original video frame sequence; Performing super-resolution reconstruction on the original video frame sequence using a pre-established feature processing model to obtain a super-resolution video frame; Establishing a brightness mapping model based on the super-resolution video frame, optimizing an initial brightness parameter set indicated by the brightness mapping model using a pre-established asynchronous batch Bayesian optimization model to obtain an optimized brightness parameter set, and converting the super-resolution video frame into a high dynamic range video frame based on the optimized brightness parameter set; Dynamic image quality optimization is performed on the high dynamic range video frame to obtain a video frame signal for display.

2. The method according to claim 1, characterized in that The feature processing model includes a downsampling network and an upsampling network, the downsampling network includes a plurality of first feature layers arranged in cascade order; the upsampling network includes a plurality of second feature layers arranged in cascade order; at least one layer between the first feature layer and the second feature layer has a lateral connection structure; The steps performed by the first feature layer of each level include: performing feature extraction and downsampling processing on the original video frame sequence or the feature map output by the first feature layer of the previous level to obtain the feature map output by the first feature layer of the current level; The steps performed by the second feature layer of each level include: performing feature fusion and upsampling processing on the feature map output by the first feature layer connected to it and / or the feature map output by the second feature layer of the previous level, so as to obtain the feature map output by the second feature layer of the current level, and to use the feature map output by the second feature layer of the last level as the super-resolution video frame.

3. The method according to claim 2, characterized in that The first feature layer includes an autoregressive feature extraction module, and the autoregressive feature extraction module includes a convolutional layer and an attention layer; The first feature layer further performs the following steps: Inputting the feature map output by the original video frame sequence or the first feature layer of the previous level into the convolution layer to perform feature extraction to obtain corresponding spatial features; The spatial features are input into the attention layer to calculate adaptive weights based on a preset sliding window mechanism, and the specified area in the feature map output by the original video frame sequence or the first feature layer of the previous level is enhanced based on the adaptive weights to obtain corresponding temporal features, wherein the sliding window mechanism is used to indicate the extraction of contextual information consisting of the current frame and the specified frame in the feature map output by the original video frame sequence or the first feature layer of the previous level, and the spatiotemporal features are used to indicate the temporal correlation of each specified area.

4. The method according to claim 3, characterized in that: The autoregressive feature extraction module also includes a dual-branch network structure, which includes a main branch unit and an auxiliary branch unit; The dual-branch network structure performs the following steps: Reconstructing the time series features through the main branch unit to obtain overall features of each designated area, wherein the overall features include a facial structure and a facial contour for the facial area; Extracting features from the time series features by the auxiliary branch unit to obtain detail features of each designated area, wherein the detail features include facial features, facial wrinkles and facial pores for the facial area; Introducing an attention mechanism into the residual block of the main branch to adaptively adjust the weight coefficient of each channel according to the importance of the channel; Adaptively weighting and fusing the overall feature with the detail feature to generate a corresponding feature map; Among them, a preset channel attention mechanism is introduced in the main branch unit and the auxiliary branch unit, and the channel attention mechanism is used to indicate that the weight of the feature channel related to the acquired feature is enhanced when acquiring any overall feature or detail feature.

5. The method according to claim 4, characterized in that The step of establishing a brightness mapping model based on the super-resolution video frame and optimizing the initial brightness parameter set indicated by the brightness mapping model using a pre-established asynchronous batch Bayesian optimization model to obtain an optimized brightness parameter set includes: Based on the brightness distribution characteristics of the super-resolution video frame, a piecewise linear brightness mapping curve is constructed, and an adaptive tone mapping function is determined; Calculating an initial brightness parameter set according to a preset local contrast preservation constraint, wherein the local contrast preservation constraint is used to prevent loss of detail information; Constructing an asynchronous batch Bayesian optimization model, wherein the asynchronous batch Bayesian optimization model includes a multi-objective Bayesian optimization framework, wherein the multi-objective Bayesian optimization framework includes a Gaussian process regression model for predicting parameter optimization directions, a collection function set, and an asynchronous parallel evaluation mechanism for simultaneously processing multiple sets of parameter configurations, wherein the collection function set includes an expected improvement value and an upper confidence bound value; The initial brightness parameter set is input into the asynchronous batch Bayesian optimization model for optimization to obtain an optimized brightness parameter set.

6. The method according to claim 5, characterized in that The converting the super-resolution video frame into a high dynamic range video frame based on the optimized brightness parameter set includes: Based on the optimized brightness parameter set, an adversarial network is generated, wherein the adversarial network includes a generator, a discriminator, and a perceptual loss function, wherein the generator is used to instruct the extraction of material texture features and illumination distribution features, the discriminator is used to instruct the authenticity evaluation of multiple local areas divided by the picture, and the perceptual loss function is used to make the video frame to be output by the adversarial network conform to the preset high dynamic range imaging perceptual features; Based on the adversarial network, converting the super-resolution video frame into a first video frame; Based on the first video frame, analyzing local brightness features of the picture, and calculating a dynamic range extension coefficient based on the local brightness features of the picture, so as to adjust the dynamic range of the first video frame based on the dynamic range extension coefficient to obtain a second video frame; The second video frame is optimized for temporal consistency to obtain a high dynamic range video frame.

7. The method according to claim 6, characterized in that The step of dynamically optimizing the high dynamic range video frame to obtain a video frame signal for display includes: Encoding the image quality parameters of the high dynamic range video frame to generate parameter codes, and determining an initial population based on the parameter codes, wherein the initial population is used to indicate a parameter combination determined by the parameter codes of the plurality of image quality parameters; Determining the dynamic constraint range of each parameter in the parameter coding sequence according to the picture complexity and motion characteristics of the high dynamic range video frame; Constructing a multi-objective fitness function based on the dynamic constraint range, evaluating the population based on the multi-objective fitness function, and outputting high-quality individuals in each population for entering the next generation; Performing crossover and mutation operations on the high-quality individuals to obtain an evolved population; Based on the parameter encoding of the image quality parameters indicated by the evolved population, a video frame signal for display is determined.

8. The method according to claim 7, characterized in that The method further comprises: Constructing a task dependency graph, the task dependency graph comprising a plurality of processing tasks, the processing tasks comprising a first processing task for indicating obtaining the original video frame sequence, a second processing task for indicating obtaining the super-resolution video frame, a third processing task for indicating obtaining the high dynamic range video frame, and a fourth processing task for indicating obtaining the video frame signal; Decomposing the processing tasks in the task dependency graph into subtasks with dependency relationships; Based on the task characteristics of the subtasks, evaluating the performance characteristics of the computing units, the computing unit characteristics including the neural network reasoning performance of the NPU, the parallel computing capability of the GPU, and the general computing performance of the CPU; Establish a task mapping strategy based on the task characteristics and computing unit performance characteristics; According to the task mapping strategy, a double buffer data transmission mechanism is established, wherein the double buffer data transmission mechanism is used to indicate a data flow strategy between computing units; The task mapping strategy is optimized by monitoring the status of the computing unit.

9. The method according to claim 8, characterized in that The step of establishing a task mapping strategy according to the task characteristics and the computing unit performance characteristics includes: Allocating the first processing task to the NPU for processing; Allocating the third processing task to the GPU for processing; Allocating an operation for instructing task scheduling and parameter optimization between the processing tasks to a CPU for processing; Establishing a task priority evaluation mechanism, wherein the task priority evaluation mechanism is used to indicate the evaluation of a corresponding priority for each processing task during the processing; According to the resource utilization status monitored in real time, the priority of the processing task and the resource allocation strategy are adjusted, and the resource allocation strategy is used to indicate the resource allocation to each computing unit.

10. A smart TV, characterized in that: The smart TV comprises: A preprocessing module is used to obtain an input video signal, decode and extract frames of the input video signal, and obtain an original video frame sequence; A super-resolution reconstruction module, used to perform super-resolution reconstruction on the original video frame sequence using a pre-established feature processing model to obtain a super-resolution video frame; An HDR conversion module is used to establish a brightness mapping model based on the super-resolution video frame, and optimize the initial brightness parameter set indicated by the brightness mapping model using a pre-established asynchronous batch Bayesian optimization model to obtain an optimized brightness parameter set, and convert the super-resolution video frame into a high dynamic range video frame based on the optimized brightness parameter set; The image quality optimization module is used to dynamically optimize the image quality of the high dynamic range video frame to obtain a video frame signal for display.

Images