A CNN-based method and system for extracting region backlight

Abstract

This invention discloses a CNN-based method and system for region backlight extraction. The method includes: acquiring a first sample image; preprocessing the first sample image to obtain a second sample image; inputting the second sample image into a CNN-based backlight extraction network, which outputs first sample backlight data corresponding to the second sample image through a three-step serial downsampling process; inputting the second sample image and the first sample backlight data into a simulator, which outputs simulation results; calculating the error based on the simulation results and training and adjusting the backlight extraction network based on the error to obtain a target backlight extraction network; and performing backlight extraction processing on the input video stream based on the target backlight extraction network to obtain the target video stream. This invention can be flexibly applied to backlight modules with different numbers of partitions and different application scenarios, avoiding manual adjustment of algorithm details and saving manpower; it adopts a pipeline-like approach, reducing the demand for storage units.

Description

Technical Field

[0001] This invention relates to the field of display technology, and in particular to a method and system for extracting regional backlight based on CNN. Background Technology

[0002] Currently, many electronic products containing LCD screens, such as televisions and automotive displays, typically utilize local dimming technology to enhance screen contrast, thereby providing viewers with higher-quality images while reducing power consumption. However, traditional backlight extraction techniques usually only consider the values ​​of the current backlight area and adjacent backlight areas. Furthermore, because different backlight modules have different parameters, traditional backlight extraction techniques suffer from problems such as significant halo effects and inconsistent results for different backlight modules and application scenarios. To address these issues, current local dimming techniques often employ a "patching" approach for manual correction. This involves further processing the backlight data after extraction using traditional backlight extraction techniques and performing detailed image compensation. These methods significantly alleviate the aforementioned problems, but require substantial manual adjustments, resulting in a large workload.

[0003] In addition, the ultra-high-definition images targeted by local dimming technology contain a huge amount of data, which leads to a high demand for storage units on the FPGA (Field-Programmable Gate Array) platform currently used to process ultra-high-definition images. At the same time, the deployment of neural networks on the FPGA platform also requires a large number of storage units.

[0004] Therefore, existing technologies still need to be improved and developed. Summary of the Invention

[0005] The main objective of this invention is to provide a CNN-based method and system for extracting regional backlight, which aims to solve the problems of traditional backlight extraction technology requiring manual adjustment of algorithm details, large workload, and high storage requirements when facing different backlight modules and different application scenarios.

[0006] To achieve the above-mentioned objective, the present invention provides a CNN-based method for region backlight extraction, the CNN-based method for region backlight extraction comprising:

[0007] A first sample image is acquired, and the first sample image is preprocessed to obtain a second sample image;

[0008] The second sample image is input into a CNN-based backlight extraction network. The backlight extraction network outputs the first sample backlight data corresponding to the second sample image through a serial three-step downsampling process. The three downsampling rates corresponding to the serial three-step downsampling process are three factors of the rounded image resolution and the multiple of the number of backlight partitions.

[0009] The second sample image and the first sample backlight data are input into the simulator for quantization, power limiting, diffusion simulation, image compensation and simulation processing, and the simulator outputs the simulation results.

[0010] The error is calculated based on the simulation results, and the backlight extraction network is trained and adjusted based on the error to obtain the target backlight extraction network.

[0011] Based on the target backlight extraction network, the input video stream is processed to extract the backlight, thereby obtaining the target video stream.

[0012] Optionally, obtaining the first sample image and preprocessing the first sample image to obtain the second sample image specifically includes:

[0013] Acquire a first sample image; wherein the first sample image includes an RGB image, an HDR image, an image with a resolution greater than a first set threshold, or an image with a contrast greater than a second set threshold;

[0014] The first sample image is preprocessed to obtain the second sample image;

[0015] The preprocessing includes image normalization and Gamma correction.

[0016] Optionally, the CNN-based backlight extraction network includes a first convolutional module, a second convolutional module, and a third convolutional module. The first convolutional module includes two dilated convolutional layers and one max pooling layer. The second convolutional module includes four dilated convolutional layers and one max pooling layer. The third convolutional module includes two convolutional layers and one sampling layer.

[0017] The step of inputting the second sample image into a CNN-based backlight extraction network, wherein the backlight extraction network outputs the first sample backlight data corresponding to the second sample image through a serial three-step downsampling process, specifically includes:

[0018] The second sample image is used to extract features using two layers of dilated convolution and ReLU activation function in the first convolution module, and the first feature map is obtained by using max pooling in the first convolution module for the first downsampling; wherein, the downsampling rate of the first downsampling is α1;

[0019] The first feature map is extracted using four layers of dilated convolution and ReLU activation function in the second convolution module, and the second feature map is obtained by a second downsampling using max pooling in the second convolution module; wherein the downsampling rate of the second downsampling is α2.

[0020] The second feature map is used to extract features using two convolutional layers and the ReLU activation function in the third convolutional module. Then, the convolution in the third convolutional module is used in conjunction with the stride to perform a third downsampling to obtain the initial first sample backlight data. The downsampling rate of the third downsampling is α3.

[0021] The initial first sample backlight data is normalized using the Sigmoid activation function to obtain the normalized first sample backlight data.

[0022] The formula for dilated convolution is:

[0023]

[0024] Among them, c out Indicates the output channel index. Indicates the output channel index c out The corresponding convolution output, i' represents the feature map row index, j' represents the feature map column index, C in Indicates the number of input channels, c represents the channel index, and bias c The input represents the convolution bias corresponding to channel index c, ks represents half the kernel size rounded down, kernel_size represents the kernel size, u represents the kernel row index, v represents the kernel column index, and input... c This represents the input feature map corresponding to channel index c, ds represents the interval between adjacent pixels, and weight... c This represents the convolution kernel data corresponding to channel index c;

[0025] When the image resolution and the number of backlight zones are integer multiples of each other, the downsampling rates α1, α2, and α3 are three factors of the multiples of the image resolution and the number of backlight zones, and the ratio is the product of the three downsampling rates:

[0026] ratio = α1 × α2 × α3;

[0027] When the image resolution and the number of backlight partitions are not integer multiples, the multiples are first rounded, then the three downsampling rates are calculated, and finally the size error is repaired by the padding parameters in the convolution process.

[0028] Optionally, the step of inputting the second sample image and the first sample backlight data into the simulator, and the simulator outputting simulation results, specifically includes:

[0029] The second sample image and the first sample backlight data are input into the simulator for quantization, power limiting, diffusion simulation, image compensation and simulation processing.

[0030] In the simulator, the first sample backlight data is quantized and power-limited to obtain the second sample backlight data; wherein, the power-limiting method includes curve mapping or lookup table mapping.

[0031] The backlight diffusion of the second sample backlight data is simulated to obtain the third sample backlight data; wherein the backlight diffusion simulation method includes interpolation, convolution or Gaussian blur.

[0032] Based on the third sample backlight data, the second sample image is subjected to gain-limited image compensation to obtain the third sample image:

[0033]

[0034] Among them, T i,j This represents the normalized pixel value of the third sample image, clip(·) represents the truncation function, γ represents the maximum compensation ratio, and I i,j B represents the normalized pixel value of the second sample image. i,j This represents the backlight data of the third sample, where i represents the row index and j represents the column index.

[0035] Based on the third sample backlight data and the third sample image, the image perceived by the human eye is simulated to obtain the simulation result; wherein, the simulation result is the simulated image brightness perceived by the human eye:

[0036]

[0037] in, ε represents the image brightness perceived by the human eye in the simulation, and ε represents the light leakage rate.

[0038] Optionally, the error is the loss of the simulated human-perceived image relative to the second sample image;

[0039] The step of calculating the error based on the simulation results and training and adjusting the backlight extraction network based on the error to obtain the target backlight extraction network specifically includes:

[0040] Based on the simulated image perceived by the human eye, the second sample image, and the first sample backlight data, a loss function is used to calculate the loss (Loss) of the simulated image perceived by the human eye relative to the second sample image:

[0041]

[0042] Where Loss represents the loss of the simulated human eye perception of the image relative to the second sample image, MSE(·) is the root mean square error function, is the backlight data of the first sample, β is the power adjustment ratio, and mean(·) is the function for calculating the average value.

[0043] The backlight extraction network is trained and adjusted based on the loss to obtain the target backlight extraction network.

[0044] Optionally, the step of performing backlight extraction processing on the input video stream based on the target backlight extraction network to obtain the target video stream specifically includes:

[0045] The input video stream is acquired and decoded to obtain the first image;

[0046] The first image is preprocessed to obtain the second image; wherein the preprocessing includes image normalization and Gamma correction;

[0047] The second image is input into the target backlight extraction network for backlight data extraction, and the target backlight extraction network outputs the first backlight data corresponding to the second image;

[0048] The first backlight data is subjected to backlight diffusion preprocessing to obtain second backlight data, and the second backlight data is stored in the on-chip RAM; wherein, the backlight diffusion preprocessing includes quantization, power limiting, and calculation of the mutual influence between different partitions;

[0049] A bilinear interpolation method is used to diffuse the second backlight data to make it the same size as the second image, thus obtaining the third backlight data;

[0050] Based on the third backlight data of the previous frame image, the second image is subjected to gain-limited image compensation to obtain the third image;

[0051] The third image is encoded to obtain the target video stream;

[0052] Specifically, the process of inputting the second image into the target backlight extraction network for backlight data extraction and the process of performing gain-limited image compensation on the second image based on the third backlight data of the previous frame image are performed simultaneously.

[0053] Optionally, the step of using bilinear interpolation to perform backlight diffusion on the second backlight data, diffusing the second backlight data to the same size as the second image to obtain the third backlight data, specifically includes:

[0054] A bilinear interpolation method is used to diffuse the second backlight data to make it the same size as the second image, thus obtaining the third backlight data;

[0055] The third backlight data is calculated based on the left linear interpolation result and the right linear interpolation result. The left linear interpolation result is calculated based on the second backlight data at the top left corner and the second backlight data at the bottom left corner. The right linear interpolation result is calculated based on the second backlight data at the top right corner and the third backlight data at the bottom right corner.

[0056]

[0057]

[0058] Where D(x,y) represents the third backlight data, M1(x1,y) represents the left linear interpolation result, M2(x2,y) represents the right linear interpolation result, R1(x1,y1), R2(x2,y1), R3(x1,y2), and R4(x2,y2) represent the second backlight data in the upper left corner, the second backlight data in the upper right corner, the second backlight data in the lower left corner, and the third backlight data in the lower right corner, respectively, x' represents the row index of the second backlight data before interpolation, and y' represents the row index of the first backlight data before interpolation. The column indices of the second backlight data are: x1 represents the row index of the second backlight data in the top left and bottom left corners, x2 represents the row index of the second backlight data in the top right and bottom right corners, y1 represents the column index of the second backlight data in the top left and top right corners, y2 represents the column index of the second backlight data in the bottom left and bottom right corners, x represents the row index of the third backlight data, y represents the column index of the third backlight data, and ratio' represents the interpolation ratio.

[0059] To achieve the above-mentioned objectives, the present invention also provides a CNN-based region backlight extraction system for implementing the steps of the CNN-based region backlight extraction method described above. The CNN-based region backlight extraction system includes a backlight extraction training system and a backlight extraction application system. The backlight extraction training system is used to train and adjust the backlight extraction network based on sample images and simulation results from a simulator to obtain a target backlight extraction network. The backlight extraction application system is deployed on an FPGA platform and is used to perform backlight extraction processing on the input video stream based on the target backlight extraction network to obtain a target video stream.

[0060] Optionally, the backlight extraction training system includes a preprocessing unit, a backlight extraction network, and a simulator connected in sequence. The output of the preprocessing unit is also connected to the input of the simulator, and the output of the simulator is also connected to the input of the backlight extraction network. The simulator includes a quantization module, a power limiting module, a backlight diffusion simulation module, an image compensation module with gain limiting, a simulation module, and an error calculation module connected in sequence.

[0061] Optionally, the backlight extraction application system includes a controller, and a video stream decoding module, a preprocessing module, a backlight data extraction module, a backlight diffusion preprocessing module, a backlight data storage module, a backlight diffusion module, an image compensation module, and a video stream encoding module, which are respectively connected to the controller and connected in sequence. The output of the preprocessing module is also connected to the input of the image compensation module. The backlight data extraction module includes the target backlight extraction network.

[0062] In this invention, a first sample image is acquired, and the first sample image is preprocessed to obtain a second sample image. The second sample image is input into a backlight extraction network based on a CNN (Convolutional Neural Network). The backlight extraction network outputs the first sample backlight data corresponding to the second sample image through a serial three-step downsampling process. The three downsampling rates corresponding to the serial three-step downsampling process are three factors of the rounded image resolution and the multiple of the number of backlight partitions. The second sample image and the first sample backlight data are input into a simulator for quantization, power limiting, diffusion simulation, image compensation, and simulation processing. The simulator outputs simulation results. The error is calculated based on the simulation results, and the backlight extraction network is trained and adjusted based on the error to obtain a target backlight extraction network. Based on the target backlight extraction network, the input video stream is subjected to backlight extraction processing to obtain the target video stream. This invention employs a serial three-step downsampling method to extract backlight data. The three downsampling rates are factors of the image resolution and the number of backlight partitions. If the factor is not an integer, it is rounded down. The padding parameter of the convolution is then used to fill the gap. This method can be flexibly applied to backlight modules with different numbers of partitions and different application scenarios, avoiding manual adjustment of algorithm details and saving manpower. The calculations between and within modules are performed in a pipeline-like manner, reducing the demand for storage units and eliminating the need for off-chip DDR (Double Data Rate) storage units. Quantization, power limiting, and image compensation modules are added to the simulator to more realistically simulate the human eye's perception. The simulation results are used to train and adjust the backlight extraction network, improving its performance. Attached Figure Description

[0063] Figure 1 This is a flowchart of a preferred embodiment of the CNN-based region backlight extraction method of the present invention;

[0064] Figure 2 This is a structural diagram of the backlight extraction network of the present invention;

[0065] Figure 3 This is a schematic diagram of the dilated convolution and the required buffer in this invention;

[0066] Figure 4 This is a schematic diagram of the bilinear interpolation method of the present invention;

[0067] Figure 5 This is a flowchart of a preferred embodiment of the CNN-based region backlight extraction system of the present invention;

[0068] Figure 6 This is a structural diagram of the backlight extraction training system of the present invention;

[0069] Figure 7 This is a structural diagram of the simulator of the present invention;

[0070] Figure 8 This is a structural diagram of the backlight extraction application system of the present invention. Detailed Implementation

[0071] To make the objectives, technical solutions, and advantages of this invention clearer and more explicit, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0072] Traditional backlight extraction techniques typically only consider the values ​​of the current backlight area and adjacent backlight areas. Furthermore, because different backlight modules have different parameters, traditional techniques suffer from problems such as large halos and the need for manual adjustment of algorithm details based on different backlight modules and application scenarios. To address this issue, current local dimming techniques often employ a "patching" approach for manual correction. This involves further processing the backlight data extracted using traditional techniques and performing detailed image compensation. These methods significantly alleviate the problems, but require substantial manual adjustments, resulting in a large workload.

[0073] Furthermore, due to the massive amount of data contained in ultra-high-definition images, a large number of storage units are often required for their transmission and processing; similarly, the operation of neural network algorithms often requires a significant amount of storage. Therefore, deploying neural network algorithms on an FPGA requires not only high computing power but also a large number of storage units. To address this issue, most current products employ methods such as optimizing neural network algorithms and designing dedicated computing units for neural network computation. These methods alleviate the huge demand for storage units by reducing the size of the neural network algorithm and increasing the device's computing power and storage capacity. However, many gaps exist between the various computational stages, resulting in the device's computing and storage capabilities not being fully utilized.

[0074] In summary, current backlight extraction technologies suffer from several problems. Firstly, the algorithm only considers the current backlight and adjacent backlight data. Secondly, different backlight modules have different parameters. Therefore, current backlight extraction technologies suffer from excessive halo effects and inconsistent performance across different backlight modules and application scenarios. Currently, solving these problems typically involves manually adding different processing steps at certain stages from backlight extraction to display, which is labor-intensive. Furthermore, current FPGA platforms for processing ultra-high-definition images have high requirements for storage units (e.g., off-chip DDR), and the deployment of neural networks on FPGAs also requires a large number of storage units.

[0075] To address the aforementioned technical issues, this invention provides a CNN-based method for region backlight extraction. It employs a serial three-step downsampling method to extract backlight data. The three downsampling rates are factors of the image resolution and the number of backlight zones. If the factor is not an integer, it is rounded down. The padding parameters of the convolution are then used to correct the discrepancies. This method can be flexibly applied to backlight modules with different numbers of zones and different application scenarios, avoiding manual adjustment of algorithm details and saving manpower. The calculations between and within modules use a pipeline-like approach, reducing the demand for storage units and eliminating the need for external DDR storage. Quantization, power limiting, and image compensation modules are added to the simulator to more realistically simulate the human eye's perception. The simulation results are used to train and adjust the backlight extraction network, improving its performance.

[0076] The application content will be further explained below with reference to the accompanying drawings and the description of the embodiments.

[0077] A preferred embodiment of the CNN-based region backlight extraction method of the present invention is as follows: Figure 1 As shown, it specifically includes:

[0078] S1. Obtain the first sample image, preprocess the first sample image to obtain the second sample image.

[0079] In one implementation of this embodiment, obtaining a first sample image and preprocessing the first sample image to obtain a second sample image specifically includes:

[0080] Acquire a first sample image; wherein the first sample image includes an RGB image, an HDR image, an image with a resolution greater than a first set threshold, or an image with a contrast greater than a second set threshold;

[0081] The first sample image is preprocessed to obtain the second sample image;

[0082] The preprocessing includes image normalization and Gamma correction.

[0083] S2. Input the second sample image into a CNN-based backlight extraction network. The backlight extraction network outputs the first sample backlight data corresponding to the second sample image through a serial three-step downsampling process. The three downsampling rates corresponding to the serial three-step downsampling process are three factors of the rounded image resolution and the multiple of the number of backlight partitions.

[0084] In one implementation of this embodiment, such as Figure 2 As shown, the CNN-based backlight extraction network includes a first convolutional module 121, a second convolutional module 122, and a third convolutional module 123. The first convolutional module 121 includes two layers of dilated convolution and one layer of max pooling. The second convolutional module 122 includes four layers of dilated convolution and one layer of max pooling. The third convolutional module 123 includes two convolutional layers and one sampling layer.

[0085] The step of inputting the second sample image into a CNN-based backlight extraction network, wherein the backlight extraction network outputs the first sample backlight data corresponding to the second sample image through a serial three-step downsampling process, specifically includes:

[0086] The second sample image is used to extract features using two layers of dilated convolution and ReLU activation function in the first convolution module, and the first feature map is obtained by using max pooling in the first convolution module for the first downsampling; wherein, the downsampling rate of the first downsampling is α1;

[0087] The first feature map is extracted using four layers of dilated convolution and ReLU activation function in the second convolution module, and the second feature map is obtained by a second downsampling using max pooling in the second convolution module; wherein the downsampling rate of the second downsampling is α2.

[0088] The second feature map is used to extract features using two convolutional layers and the ReLU activation function in the third convolutional module. Then, the convolution in the third convolutional module is used in conjunction with the stride to perform a third downsampling to obtain the initial first sample backlight data. The downsampling rate of the third downsampling is α3.

[0089] The initial first sample backlight data is normalized using the Sigmoid activation function to obtain the normalized first sample backlight data.

[0090] The formula for dilated convolution is:

[0091]

[0092] Among them, c out Indicates the output channel index. Indicates the output channel index c outThe corresponding convolution output, i' represents the feature map row index, j' represents the feature map column index, C in Indicates the number of input channels, c represents the channel index, and bias c The input represents the convolution bias corresponding to channel index c, ks represents half the kernel size rounded down, kernel_size represents the kernel size, u represents the kernel row index, v represents the kernel column index, and input... c This represents the input feature map corresponding to channel index c, ds represents the interval between adjacent pixels, and weight... c This represents the convolution kernel data corresponding to channel index c;

[0093] When the image resolution and the number of backlight zones are integer multiples of each other, the downsampling rates α1, α2, and α3 are three factors of the multiples of the image resolution and the number of backlight zones, and the ratio is the product of the three downsampling rates:

[0094] ratio = α1 × α2 × α3;

[0095] When the image resolution and the number of backlight partitions are not integer multiples, the multiples are first rounded, then the three downsampling rates are calculated, and finally the size error is repaired by the padding parameters in the convolution process.

[0096] Specifically, the backlight extraction network of the present invention uses a serial three-step downsampling method to extract backlight data. The three downsampling rates are factors of the relationship between the image resolution and the number of backlight partitions. If the multiple relationship is not an integer, it is rounded down. Then, the padding parameter of the convolution is used to repair the gap. It can be flexibly applied to backlight systems with different numbers of partitions. The backlight extraction network does not contain complex modules such as residual connections and attention, and can be more easily deployed on hardware platforms.

[0097] To reduce the demand for storage units, especially off-chip DDR resources, this invention optimizes the overall process and internal modules through pipeline optimization. The backlight extraction network proposed in this invention has only a serial structure, thus allowing computation to be performed while the image is being input without waiting for the entire image to be transmitted. Figure 3 As shown, since each convolution operation only involves a portion of the data, only a few rows need to be cached to begin the operation, and then the convolution operation is performed simultaneously with the input of image pixel data. Similarly, pooling operations do not involve the complete feature map input, so a pipelined operation similar to convolution can also be performed.

[0098] S3. Input the second sample image and the first sample backlight data into the simulator for quantization, power limiting, diffusion simulation, image compensation and simulation processing, and the simulator outputs the simulation results.

[0099] In one implementation of this embodiment, the step of inputting the second sample image and the first sample backlight data into a simulator for quantization, power limiting, diffusion simulation, image compensation, and simulation processing, and the simulator outputting simulation results, specifically includes:

[0100] The second sample image and the backlight data of the first sample are input into the simulator;

[0101] In the simulator, the first sample backlight data is quantized and power-limited to obtain the second sample backlight data; wherein, the power-limiting method includes curve mapping or lookup table mapping.

[0102] The backlight diffusion of the second sample backlight data is simulated to obtain the third sample backlight data; wherein the backlight diffusion simulation method includes interpolation, convolution or Gaussian blur.

[0103] Based on the third sample backlight data, the second sample image is subjected to gain-limited image compensation to obtain the third sample image:

[0104]

[0105] Among them, T i,j This represents the normalized pixel value of the third sample image, clip(·) represents the truncation function, γ represents the maximum compensation ratio, and I i,j B represents the normalized pixel value of the second sample image. i,j This represents the backlight data of the third sample, where i represents the row index and j represents the column index. The data is normalized, so a clip function (·) is needed to prevent overflow.

[0106] Based on the third sample backlight data and the third sample image, the image perceived by the human eye is simulated to obtain the simulation result; wherein, the simulation result is the simulated image brightness perceived by the human eye:

[0107]

[0108] in, ε represents the image brightness perceived by the human eye in the simulation, and ε represents the light leakage rate.

[0109] Specifically, in the simulator, since the backlight value of the backlight module is often an integer value, while the backlight value obtained by the backlight extraction network is often a floating-point number, a quantization module is used to quantize the backlight value obtained by the backlight extraction network into an integer value in order to better simulate the display effect. The final integer backlight value has bits including, but not limited to, 8 bits, 12 bits, etc. When the number of zones is too large, the overall power is insufficient to drive all LEDs to maximum brightness. To obtain better simulation results and calculate the actual backlight value transmitted to the backlight module, a power limiting module is used to limit the overall power. Power limiting methods include, but are not limited to, curve mapping, lookup table mapping, etc. Since different backlight zones will affect each other, a backlight diffusion simulation module is used to simulate the backlight that ultimately acts on the LCD screen. Backlight diffusion simulation can use methods including, but not limited to, interpolation, convolution, Gaussian blur, etc. Since a decrease in backlight will lead to a decrease in the actual display brightness, the image needs to be compensated to increase the transmittance of the LCD screen, so that the final display effect is closer to the original image. In practical use, overexposure often occurs due to excessively high compensation ratios caused by excessively low backlight data. Therefore, the image compensation module proposed in this invention limits the maximum compensation ratio to γ, with a value of (0, 20]. The maximum compensation ratio can be adjusted according to the module. The calculation method for the compensation value is as follows: This invention uses a simulation module to simulate the image perceived by the human eye. The brightness perceived by the human eye can be calculated from the backlight brightness and the LCD screen transmittance. These two values ​​can be obtained from the backlight data and the pixel data of the compensated image. Simultaneously, since the LCD module exhibits a certain degree of light leakage, the light leakage rate is assumed to be ε, and the final brightness perceived by the human eye is...

[0110] S4. Calculate the error based on the simulation results, and train and adjust the backlight extraction network based on the error to obtain the target backlight extraction network.

[0111] In one implementation of this embodiment, the error is the loss of the simulated human-perceived image relative to the second sample image;

[0112] The step of calculating the error based on the simulation results and training and adjusting the backlight extraction network based on the error to obtain the target backlight extraction network specifically includes:

[0113] Based on the simulated image perceived by the human eye, the second sample image, and the first sample backlight data, a loss function is used to calculate the loss (Loss) of the simulated image perceived by the human eye relative to the second sample image:

[0114]

[0115] Where Loss represents the loss of the simulated human eye perception of the image relative to the second sample image, MSE(·) is the root mean square error function, is the backlight data of the first sample, β is the power adjustment ratio, and mean(·) is the function for calculating the average value.

[0116] The backlight extraction network is trained and adjusted based on the loss to obtain the target backlight extraction network.

[0117] Specifically, this invention uses an error calculation module to guide the training and adjustment of the backlight extraction network. The backlight values ​​extracted by the backlight extraction network should ideally approximate the image perceived by the human eye as close to the original image, while also maintaining low power. Therefore, the loss function used in this invention is...

[0118] S5. Based on the target backlight extraction network, perform backlight extraction processing on the input video stream to obtain the target video stream.

[0119] In one implementation of this embodiment, the step of performing backlight extraction processing on the input video stream based on the target backlight extraction network to obtain the target video stream specifically includes:

[0120] The input video stream is acquired and decoded to obtain the first image;

[0121] The first image is preprocessed to obtain the second image; wherein the preprocessing includes image normalization and Gamma correction;

[0122] The second image is input into the target backlight extraction network for backlight data extraction, and the target backlight extraction network outputs the first backlight data corresponding to the second image;

[0123] The first backlight data is subjected to backlight diffusion preprocessing to obtain second backlight data, and the second backlight data is stored in on-chip RAM (Random Access Memory); wherein, the backlight diffusion preprocessing includes quantization, power limiting, and calculation of the mutual influence between different partitions;

[0124] A bilinear interpolation method is used to diffuse the second backlight data to make it the same size as the second image, thus obtaining the third backlight data;

[0125] Based on the third backlight data of the previous frame image, the second image is subjected to gain-limited image compensation to obtain the third image;

[0126] The third image is encoded to obtain the target video stream;

[0127] Specifically, the process of inputting the second image into the target backlight extraction network for backlight data extraction and the process of performing gain-limited image compensation on the second image based on the third backlight data of the previous frame image are performed simultaneously (described here from an image perspective).

[0128] Specifically, this invention uses a video stream decoding module to receive video streams, including but not limited to HDMI, DP, and V-By-One video stream data. The video stream data is then decoded into common image data, referred to as the first image, which includes a data validity signal (DE), a column synchronization signal (HS), and a row synchronization signal (VS). The first image is input to a preprocessing module to perform image preprocessing, including but not limited to image normalization and Gamma correction, and outputs a second image. The second image is input to a backlight data extraction module, which extracts suitable backlight data, referred to as the first backlight data, based on the second image using a convolutional neural network (CNN). The CNN is implemented in a pipelined manner, using only on-chip RAM to temporarily store some data, eliminating the need for additional off-chip DDR storage of intermediate data. The first backlight data is input to a backlight diffusion preprocessing module, which performs backlight diffusion preprocessing, mainly including quantization, power limiting, and Gaussian blurring, and outputs the second backlight data. The second backlight data is input to the backlight diffusion module, which diffuses the second backlight data to the same size as the second image using bilinear interpolation, and outputs the third backlight data. Backlight diffusion and image compensation are implemented synchronously using a pipelined approach. The second image and third backlight data are input into the image compensation module. The second image is compensated based on the third backlight data, and the third image is output using the gain-limited image compensation method described above. The image compensation module is implemented using a pipelined approach, eliminating the need to wait for all data to be prepared before calculation, and completes the backlight diffusion and image compensation work together with the backlight diffusion module. When the number of partitions is low, the backlight data volume is small; therefore, the second backlight data is stored in the on-chip RAM of the backlight data storage module for calculation by the backlight diffusion module. Finally, the video stream encoding module encodes the third image into HDMI, DP, V-By-One, etc., video streams for screen display.

[0129] It should be noted that image compensation requires backlight calculation, but since the difference between two video frames is small, this invention uses the backlight data of the previous frame (i.e., the third backlight data) in conjunction with the image of the current frame (i.e., the second image) for display to complete the overall pipeline operation. The specific steps are as follows:

[0130] While receiving each line of image data (V_active), preprocessing and backlight extraction are performed simultaneously. The image (second image) is compensated using the third backlight data from the previous frame. Then, the compensated image (third image) of this frame and the first backlight data from the previous frame are output and displayed (described from a pixel perspective).

[0131] During the image data reception interval (V_blank) of each line, the backlight diffusion preprocessing of the first backlight data of the current frame is completed and stored in the on-chip RAM for processing in the next frame.

[0132] It should be noted that backlight extraction has two meanings: a narrow one and a broad one. In a narrow sense, it refers to backlight data extraction, while in a broad sense, it includes processes such as video stream decoding, preprocessing, backlight data extraction, backlight diffusion preprocessing, backlight data storage, backlight diffusion, image compensation, and video stream encoding.

[0133] In one implementation of this embodiment, the step of using bilinear interpolation to diffuse the second backlight data until it matches the size of the second image, thereby obtaining the third backlight data, specifically includes:

[0134] A bilinear interpolation method is used to diffuse the second backlight data to make it the same size as the second image, thus obtaining the third backlight data;

[0135] The third backlight data is calculated based on the left linear interpolation result and the right linear interpolation result. The left linear interpolation result is calculated based on the second backlight data at the top left corner and the second backlight data at the bottom left corner. The right linear interpolation result is calculated based on the second backlight data at the top right corner and the third backlight data at the bottom right corner.

[0136]

[0137] Where D(x,y) represents the third backlight data, M1(x1,y) represents the left linear interpolation result, M2(x2,y) represents the right linear interpolation result, R1(x1,y1), R2(x2,y1), R3(x1,y2), and R4(x2,y2) represent the second backlight data in the upper left corner, the second backlight data in the upper right corner, the second backlight data in the lower left corner, and the third backlight data in the lower right corner, respectively, x' represents the row index of the second backlight data before interpolation, and y' represents the row index of the first backlight data before interpolation. The column indices of the second backlight data are: x1 represents the row index of the second backlight data in the top left and bottom left corners, x2 represents the row index of the second backlight data in the top right and bottom right corners, y1 represents the column index of the second backlight data in the top left and top right corners, y2 represents the column index of the second backlight data in the bottom left and bottom right corners, x represents the row index of the third backlight data, y represents the column index of the third backlight data, and ratio' represents the interpolation ratio.

[0138] Specifically, the algorithm used in the backlight diffusion module of this invention is bilinear interpolation, such as... Figure 4As shown, the third backlight data is calculated based on the left and right linear interpolation results. The left linear interpolation result is calculated based on the second backlight data in the upper left and lower left corners, while the right linear interpolation result is calculated based on the second backlight data in the upper right and lower right corners. This demonstrates that backlight diffusion does not require complete second backlight data to complete the pipeline operation.

[0139] The following examples are provided to aid in the explanation.

[0140] This invention first trains and adjusts the backlight extraction network using simulated images on a device with a GPU (Graphics Processing Unit) so that the backlight extraction network can extract appropriate backlight data. The specific steps are as follows:

[0141] Step 1: Read batch_size sample images from the sample data, for example, 32 sample images, and perform preprocessing operations on the read sample images, including image normalization and Gamma correction, to obtain preprocessed sample images (second sample images).

[0142] Step 2: Use the parameters of the current network (backlight extraction network) to extract appropriate backlight data from the second sample image to obtain the first sample backlight data; this process can be accelerated using a GPU.

[0143] Step 3: Simulate the image perceived by the human eye using the first sample backlight data and the second sample image. Specifically, this includes: quantizing the first sample backlight data, for example, to 8 bits; power limiting the quantized sample backlight data, specifically by: first calculating the mean of the sample backlight data, then using a pre-designed curve, looking up the mean to obtain the scaling factor (range (0,1]) of the sample backlight data and applying it to the sample backlight data to obtain the second backlight data; diffusing the second sample backlight data using bilinear interpolation to obtain the third sample backlight data; compensating the second sample image with gain limitation based on the third sample backlight data, with a maximum compensation ratio of 5, to obtain the compensated image (the third sample image); simulating the result based on the third sample backlight data and the third sample image, with a light leakage ratio of 0.05, to obtain the simulated image (i.e., the simulated image perceived by the human eye); calculating the loss based on the simulated image, the second sample image, and the first sample backlight data; and finally training and adjusting the backlight extraction network based on the loss.

[0144] Step 4: If not all sample data has been read, return to Step 1 to continue training; if all sample data has been read, proceed to Step 5.

[0145] Step 5: Repeat steps 1 to 4 several times, for example, 1000 times.

[0146] After training the backlight extraction network, the hardware system is deployed on an FPGA platform. For example... Figure 8 As shown, the implemented hardware system includes a video stream decoding module 21, a video stream encoding module 28, a preprocessing module 22, a backlight data extraction module 23, a backlight diffusion preprocessing module 24, a backlight diffusion module 26, an image compensation module 27, a backlight data storage module 25, and a controller 29. Each module and the modules between modules are implemented using a pipelined optimization approach, as detailed below:

[0147] The video stream decoding module and video stream encoding module are used for decoding and encoding video streams, such as the decoder and encoder of HDMI 2.1 IP; the video signal includes the pixel clock, data valid signal DE, horizontal sync signal HS, vertical sync signal VS, and pixel data.

[0148] The controller can be a hard core embedded in the FPGA platform, such as an ARM core, or a soft core built using FPGA resources, such as a NIOS core. The controller is mainly used for the control and scheduling of various modules, especially the video stream decoding module and the video stream encoding module.

[0149] The preprocessing module is mainly used for image preprocessing, such as gamma correction, which can be accomplished using a lookup table method. Since the calculation process for each pixel does not involve other pixels, preprocessing can be completed immediately after receiving each pixel without waiting for the complete image input.

[0150] The backlight data extraction module is mainly used for backlight extraction and consists of three layers of convolutional neural networks. Since each convolutional layer only requires a portion of the input data, computation can begin once the required data is input. Taking a 4K image and the first convolutional layer as an example, the first convolutional layer uses dilated convolution with a dilation rate of 3, a stride of 1, and a padding parameter of 3. Therefore, each convolution requires 9 data points; due to the dilation rate of 3, the required data spans 7 rows. Because the padding parameter is 3, and the image data is input row by row through a line scan, the actual number of pixels that need to be waited for is 3 rows + 3 data points. After that, a result data point can be calculated for each incoming pixel.

[0151] The backlight diffusion preprocessing module uses a Gaussian blur kernel, which performs blurring through convolution, similar to the pipelined approach of the backlight extraction network. The processed second backlight data is then stored in the on-chip RAM of the backlight data storage module for use in the next frame's image compensation. Because there are intervals between each line and frame of image data transmission for video signal synchronization—for example, a typical 4K signal has a 560-pixel clock interval between line synchronization signals, meaning that 4400 pixels out of 3840 pixels per line are actually transmitted; similarly, there is a 90-line interval between frames, meaning that 2250 pixels out of 2160 lines per frame are actually transmitted—there is sufficient time to calculate the necessary data before the next line or frame of pixel data arrives.

[0152] The backlight diffusion module uses a bilinear interpolation method, which requires a maximum of four pixels to calculate each pixel value. Since the second backlight data is stored in the on-chip RAM, a diffused backlight data (third backlight data) can be calculated when each pixel data is input. Then, the gain-limited image compensation module can use the input third backlight data and the preprocessed image data (second image) to complete the image compensation.

[0153] Furthermore, this invention also provides a CNN-based region backlight extraction system for implementing the steps of the CNN-based region backlight extraction method described above. The CNN-based region backlight extraction system includes a backlight extraction training system and a backlight extraction application system. The backlight extraction training system is used to train and adjust the backlight extraction network based on sample images and simulation results from a simulator to obtain a target backlight extraction network. The backlight extraction application system is deployed on an FPGA platform and is used to perform backlight extraction processing on the input video stream based on the target backlight extraction network to obtain a target video stream.

[0154] Specifically, the operation flow of the CNN-based region backlight extraction system is as follows: Figure 5As shown, the system is divided into three modes: "training," "testing," and "deployment." In "training" mode, after image preprocessing, the backlight extraction network extracts suitable backlight data from the input sample image. This backlight data and the preprocessed sample image are then fed into the simulator for simulation. The simulation results are used to calculate the error and adjust the backlight extraction network. In this mode, the sample image (image dataset) is preferably an HDR image or another image with high contrast. In "testing" mode, after image preprocessing, the backlight extraction network extracts suitable backlight data from the input sample image. This backlight data and the preprocessed sample image are then fed into the simulator for simulation. The simulation results are output and saved. In "deployment" mode, after image preprocessing, the backlight extraction network extracts suitable backlight data from the input image. This backlight data is transmitted to the backlight module to control the brightness of each LED in its respective zone. The simulator's simulation module calculates a compensation image, which is then transmitted to the LCD module for display.

[0155] In one implementation of this embodiment, such as Figure 6 As shown, the backlight extraction training system includes a preprocessing unit 11, a backlight extraction network 12, and a simulator 13 connected in sequence. The output of the preprocessing unit 11 is also connected to the input of the simulator 13, and the output of the simulator 13 is also connected to the input of the backlight extraction network 12. Figure 7 As shown, the simulator 13 includes a quantization module 131, a power limiting module 132, a backlight diffusion simulation module 133, an image compensation module 134 with gain limiting, a simulation module 135, and an error calculation module 136 connected in sequence.

[0156] Specifically, the backlight extraction training system provided by this invention mainly comprises three parts: a preprocessing unit, a backlight extraction network, and a simulator. The preprocessing unit preprocesses the input sample images (e.g., RGB images), including but not limited to normalization and gamma correction. The backlight extraction network extracts suitable backlight data from the sample images. The simulator simulates the human eye's perception effect based on the sample images and backlight data, and trains and adjusts the backlight extraction network under certain conditions. This invention can utilize information from image datasets of different types and the simulation results of the simulator to train and adjust the backlight extraction network, ultimately extracting suitable backlight data from the input image. This method can be deployed on an FPGA platform, has low storage requirements, and does not require external storage. The functions of each module in the simulator have been described above and will not be repeated here.

[0157] In one implementation of this embodiment, such as Figure 8As shown, the backlight extraction application system includes a controller 29, and a video stream decoding module 21, a preprocessing module 22, a backlight data extraction module 23, a backlight diffusion preprocessing module 24, a backlight data storage module 25, a backlight diffusion module 26, an image compensation module 27, and a video stream encoding module 28, which are connected to the controller 29 and connected in sequence. The output of the preprocessing module 22 is also connected to the input of the image compensation module 27. The backlight data extraction module 23 includes the target backlight extraction network.

[0158] Specifically, the backlight extraction application system of this invention is deployed on an FPGA platform, eliminating the need for off-chip DDR cache images and feature maps in neural network operations. The controller is used to control and schedule the various modules within the entire backlight extraction application system; the functions of the other modules have already been described above and will not be repeated here.

[0159] It should be understood that the application of the present invention is not limited to the examples above. Those skilled in the art can make improvements or modifications based on the above description, and all such improvements and modifications should fall within the protection scope of the appended claims.

Claims

1. A CNN-based method for extracting region backlighting, characterized in that, The CNN-based region backlight extraction method includes: A first sample image is acquired, and the first sample image is preprocessed to obtain a second sample image; The second sample image is input into a CNN-based backlight extraction network. The backlight extraction network outputs the first sample backlight data corresponding to the second sample image through a serial three-step downsampling process. The three downsampling rates corresponding to the serial three-step downsampling process are three factors of the rounded image resolution and the multiple of the number of backlight partitions. The CNN-based backlight extraction network includes a first convolutional module, a second convolutional module, and a third convolutional module. The first convolutional module includes two dilated convolutional layers and one max pooling layer. The second convolutional module includes four dilated convolutional layers and one max pooling layer. The third convolutional module includes two convolutional layers and one sampling layer. The step of inputting the second sample image into a CNN-based backlight extraction network, wherein the backlight extraction network outputs the first sample backlight data corresponding to the second sample image through a serial three-step downsampling process, specifically includes: Feature extraction is performed on the second sample image using two layers of dilated convolution and the ReLU activation function in the first convolution module. A first downsampling is then performed using max pooling in the first convolution module to obtain a first feature map; wherein the downsampling rate of the first downsampling is... α 1; The first feature map is extracted using four layers of dilated convolution and ReLU activation function in the second convolution module, and a second downsampling is performed using max pooling in the second convolution module to obtain the second feature map; wherein, the downsampling rate of the second downsampling is... α 2; Feature extraction is performed on the second feature map using two convolutional layers and the ReLU activation function in the third convolutional module. A third downsampling is then performed using the convolutional layers and stride in the third convolutional module to obtain the initial first sample backlight data. The downsampling rate of the third downsampling is... α 3; The initial first sample backlight data is normalized using the Sigmoid activation function to obtain the normalized first sample backlight data. The formula for dilated convolution is: in, C OUT Indicates the output channel index. Indicates the output channel index C OUT The corresponding convolution output, Indicates the feature map row index. Indicates the column index of the feature map. This indicates the number of input channels, where c represents the channel index. This represents the convolution bias corresponding to channel index c. ks This represents half the kernel size rounded down. kernel_size This represents the kernel size, u represents the kernel row index, and v represents the kernel column index. This represents the input feature map corresponding to channel index c, and ds represents the interval between adjacent pixels. This represents the convolution kernel data corresponding to channel index c; When the image resolution and the number of backlight zones are integer multiples of each other, the downsampling rate α 1. α 2 and α 3 represents three factors that are multiples of the image resolution and the number of backlight zones, and these multiples are the product of the three downsampling rates: ； When the image resolution and the number of backlight partitions are not integer multiples, the multiples are first rounded, then the three downsampling rates are calculated, and finally the size error is repaired by the padding parameters in the convolution process. The second sample image and the first sample backlight data are input into the simulator for quantization, power limiting, diffusion simulation, image compensation and simulation processing, and the simulator outputs the simulation results. The error is calculated based on the simulation results, and the backlight extraction network is trained and adjusted based on the error to obtain the target backlight extraction network. Based on the target backlight extraction network, the input video stream is processed to extract the backlight, thereby obtaining the target video stream.

2. The CNN-based region backlight extraction method according to claim 1, characterized in that, The step of obtaining a first sample image and preprocessing the first sample image to obtain a second sample image specifically includes: Acquire a first sample image; wherein the first sample image includes an RGB image, an HDR image, an image with a resolution greater than a first set threshold, or an image with a contrast greater than a second set threshold; The first sample image is preprocessed to obtain the second sample image; The preprocessing includes image normalization and Gamma correction.

3. The CNN-based region backlight extraction method according to claim 1, characterized in that, The process of inputting the second sample image and the first sample backlight data into the simulator for quantization, power limiting, diffusion simulation, image compensation, and simulation processing, and the simulator outputting simulation results, specifically includes: The second sample image and the backlight data of the first sample are input into the simulator; In the simulator, the first sample backlight data is quantized and power-limited to obtain the second sample backlight data; wherein, the power-limiting method includes curve mapping or lookup table mapping. The backlight diffusion of the second sample backlight data is simulated to obtain the third sample backlight data; wherein the backlight diffusion simulation method includes interpolation, convolution or Gaussian blur. Based on the third sample backlight data, the second sample image is subjected to gain-limited image compensation to obtain the third sample image: ： in, This represents the normalized pixel value of the third sample image. This represents the truncation function. This indicates the maximum compensation ratio. This represents the normalized pixel value of the second sample image. This represents the backlight data of the third sample, where i represents the row index and j represents the column index. Based on the third sample backlight data and the third sample image, the image perceived by the human eye is simulated to obtain the simulation result; wherein, the simulation result is the simulated image brightness perceived by the human eye: ； in, This represents the simulated image brightness as perceived by the human eye. This indicates the light leakage rate.

4. The CNN-based region backlight extraction method according to claim 3, characterized in that, The error is the loss of the simulated human eye's perception of the image relative to the second sample image; The step of calculating the error based on the simulation results and training and adjusting the backlight extraction network based on the error to obtain the target backlight extraction network specifically includes: Based on the simulated image perceived by the human eye, the second sample image, and the first sample backlight data, the loss function is used to calculate the loss (LOSS) of the simulated image perceived by the human eye relative to the second sample image: ； in, LOSS The loss represents the image perceived by the human eye in the simulation relative to the second sample image, where MSE(·) is the root mean square error function. This is the backlight data of the first sample. The power adjustment ratio is denoted by mean(·), which is a function for calculating the average value. The backlight extraction network is trained and adjusted based on the loss to obtain the target backlight extraction network.

5. The CNN-based region backlight extraction method according to claim 1, characterized in that, The process of performing backlight extraction processing on the input video stream based on the target backlight extraction network to obtain the target video stream specifically includes: The input video stream is acquired and decoded to obtain the first image; The first image is preprocessed to obtain the second image; wherein the preprocessing includes image normalization and Gamma correction; The second image is input into the target backlight extraction network for backlight data extraction, and the target backlight extraction network outputs the first backlight data corresponding to the second image; The first backlight data is subjected to backlight diffusion preprocessing to obtain second backlight data, and the second backlight data is stored in the on-chip RAM; wherein, the backlight diffusion preprocessing includes quantization, power limiting, and calculation of the mutual influence between different partitions; A bilinear interpolation method is used to diffuse the second backlight data to make it the same size as the second image, thus obtaining the third backlight data; Based on the third backlight data of the previous frame image, the second image is subjected to gain-limited image compensation to obtain the third image; The third image is encoded to obtain the target video stream; Specifically, the process of inputting the second image into the target backlight extraction network for backlight data extraction and the process of performing gain-limited image compensation on the second image based on the third backlight data of the previous frame image are performed simultaneously.

6. The CNN-based region backlight extraction method according to claim 5, characterized in that, The method employs bilinear interpolation to diffuse the second backlight data, expanding it to match the size of the second image to obtain the third backlight data. Specifically, this includes: A bilinear interpolation method is used to diffuse the second backlight data to make it the same size as the second image, thus obtaining the third backlight data; The third backlight data is calculated based on the left linear interpolation result and the right linear interpolation result. The left linear interpolation result is calculated based on the second backlight data at the top left corner and the second backlight data at the bottom left corner. The right linear interpolation result is calculated based on the second backlight data at the top right corner and the third backlight data at the bottom right corner. in, This indicates the third backlight data. This represents the result of left linear interpolation. This represents the result of right-linear interpolation. , , and These represent the second backlight data at the top left, the second backlight data at the top right, the second backlight data at the bottom left, and the third backlight data at the bottom right, respectively. This indicates the row index of the second backlight data before interpolation. This indicates the column index of the second backlight data before interpolation. This indicates the row index of the second backlight data in the top left corner and the second backlight data in the bottom left corner. This indicates the row index of the second backlight data in the upper right corner and the second backlight data in the lower right corner. This indicates the column index of the second backlight data in the top left and top right corners. This indicates the column index of the second backlight data in the bottom left and bottom right corners. This indicates the row index of the third backlight data. The column index representing the third backlight data. Indicates the interpolation ratio.

7. A CNN-based region backlight extraction system, used to implement the steps of the CNN-based region backlight extraction method as described in any one of claims 1-6, characterized in that, The CNN-based regional backlight extraction system includes a backlight extraction training system and a backlight extraction application system. The backlight extraction training system is used to train and adjust the backlight extraction network based on sample images and simulation results of the simulator to obtain the target backlight extraction network. The backlight extraction application system is deployed on an FPGA platform. The backlight extraction application system is used to perform backlight extraction processing on the input video stream based on the target backlight extraction network to obtain the target video stream.

8. The CNN-based region backlight extraction system according to claim 7, characterized in that, The backlight extraction training system includes a preprocessing unit, a backlight extraction network, and a simulator connected in sequence. The output of the preprocessing unit is also connected to the input of the simulator, and the output of the simulator is also connected to the input of the backlight extraction network. The simulator includes a quantization module, a power limiting module, a backlight diffusion simulation module, an image compensation module with gain limiting, a simulation module, and an error calculation module connected in sequence.

9. The CNN-based region backlight extraction system according to claim 7, characterized in that, The backlight extraction application system includes a controller, and a video stream decoding module, a preprocessing module, a backlight data extraction module, a backlight diffusion preprocessing module, a backlight data storage module, a backlight diffusion module, an image compensation module, and a video stream encoding module, which are connected to the controller in sequence. The output of the preprocessing module is also connected to the input of the image compensation module. The backlight data extraction module includes the target backlight extraction network.

Images