Regional backlight extraction method and system based on CNN

Abstract

The invention discloses a CNN-based regional backlight extraction method and system, and the method comprises the steps: obtaining a first sample image, and carrying out the preprocessing of the first sample image, and obtaining a second sample image; inputting the second sample image into a CNN-based backlight extraction network, wherein the backlight extraction network outputs first sample backlight data corresponding to the second sample image through serial three-step downsampling; inputting the second sample image and the first sample backlight data into a simulator, and outputting a simulation result by the simulator; calculating an error according to the simulation result, and training and adjusting the backlight extraction network according to the error to obtain a target backlight extraction network; and performing backlight extraction processing on the input video stream based on the target backlight extraction network to obtain a target video stream. The method can be flexibly applied to backlight modules with different partition numbers and different application scenes, manual adjustment of algorithm details is avoided, and manpower is saved; a similar assembly line mode is adopted, and the requirement for a storage unit is reduced.

Description

Technical Field

[0001] The present invention relates to the field of display technology, and in particular to a CNN-based regional backlight extraction method and system. Background Art

[0002] At present, many electronic products containing LCD screens, such as televisions and car screens, usually use regional dimming technology to improve the contrast of the screen, thereby providing viewers with higher quality images while reducing power consumption. However, since traditional backlight extraction technology usually only considers the values ​​of the current backlight area and the adjacent backlight area, and since different backlight modules have different parameters, traditional backlight extraction technology has the problem of large halo and different effects on different backlight modules and different application scenarios. To solve the above problems, the current regional dimming technology often uses a "patching" method for manual correction, such as further processing the backlight data after extracting the backlight data using traditional backlight extraction technology, and performing detailed compensation on the image. These methods can greatly alleviate the above problems, but require a lot of manpower for manual adjustment, which is a lot of work.

[0003] In addition, the amount of data contained in the ultra-high-definition images targeted by local dimming technology is huge, resulting in a high demand for storage units on the FPGA (Field-Programmable Gate Array) platform currently used to process ultra-high-definition images. At the same time, the deployment of neural networks on the FPGA platform also requires a large number of storage units.

[0004] Therefore, the prior art still needs to be improved and developed. Summary of the invention

[0005] The main purpose of the present invention is to provide a CNN-based regional backlight extraction method and system, aiming to solve the problems of traditional backlight extraction technology requiring manual adjustment of algorithm details when facing different backlight modules and different application scenarios, large workload and high demand for storage units.

[0006] To achieve the above-mentioned object of the invention, the present invention provides a method for extracting area backlight based on CNN, and the method for extracting area backlight based on CNN includes:

[0007] Acquire a first sample image, and preprocess the first sample image to obtain a second sample image;

[0008] Inputting the second sample image into a CNN-based backlight extraction network, the backlight extraction network outputs first sample backlight data corresponding to the second sample image through serial three-step downsampling, wherein three downsampling rates corresponding to the serial three-step downsampling are three factors of the rounded image resolution and multiples of the number of backlight partitions;

[0009] inputting the second sample image and the first sample backlight data into a simulator for quantization, power limitation, diffusion simulation, image compensation and simulation processing, the simulator outputting a simulation result;

[0010] Calculating an error according to the simulation result, and training and adjusting the backlight extraction network according to the error to obtain a target backlight extraction network;

[0011] Based on the target backlight extraction network, backlight extraction processing is performed on the input video stream to obtain a target video stream.

[0012] Optionally, the acquiring the first sample image and preprocessing the first sample image to obtain the second sample image specifically includes:

[0013] Acquire a first sample image; wherein the first sample image includes an RGB image, an HDR image, an image with a resolution greater than a first set threshold, or an image with a contrast greater than a second set threshold;

[0014] Preprocessing the first sample image to obtain a second sample image;

[0015] The preprocessing includes image normalization and Gamma correction.

[0016] Optionally, the CNN-based backlight extraction network includes a first convolution module, a second convolution module and a third convolution module, the first convolution module includes two layers of dilated convolution and one layer of maximum pooling, the second convolution module includes four layers of dilated convolution and one layer of maximum pooling, and the third convolution module includes two layers of convolution and one layer of sampling;

[0017] The step of inputting the second sample image into a CNN-based backlight extraction network, wherein the backlight extraction network outputs first sample backlight data corresponding to the second sample image through serial three-step downsampling, specifically includes:

[0018] The second sample image is feature extracted using the two-layer dilated convolution and ReLU activation function in the first convolution module, and the first downsampling is performed using the maximum pooling in the first convolution module to obtain a first feature map; wherein the downsampling rate of the first downsampling is α 1 ;

[0019] The first feature map is extracted by using the four-layer hole convolution and ReLU activation function in the second convolution module, and the second feature map is obtained by using the maximum pooling in the second convolution module; wherein the downsampling rate of the second downsampling is α 2 ;

[0020] The second feature map is extracted by using the two-layer convolution and ReLU activation function in the third convolution module, and the convolution in the third convolution module is used to perform a third downsampling with a step size to obtain the initial first sample backlight data; wherein the downsampling rate of the third downsampling is α 3 ;

[0021] Normalizing the initial first sample backlight data using a Sigmoid activation function to obtain normalized first sample backlight data;

[0022] Among them, the formula of the dilated convolution is:

[0023]

[0024] Among them, c out represents the output channel index, Indicates the output channel index c out The corresponding convolution output, i' represents the feature map row index, j' represents the feature map column index, C in represents the number of input channels, c represents the channel index, bias c represents the convolution bias corresponding to the channel index c, ks represents half of the convolution kernel size rounded down, kernel_size represents the convolution kernel size, u represents the convolution kernel row index, v represents the convolution kernel column index, input c represents the input feature map corresponding to the channel index c, ds represents the interval between adjacent pixels, and weight c Represents the convolution kernel data corresponding to the channel index c;

[0025] When the image resolution and the number of backlight partitions are integer multiples, the downsampling rate α 1 , α 2 and α 3 are three factors of the multiples of the image resolution and the number of backlight partitions, and the multiple ratio is the product of three downsampling rates:

[0026] ratio = α 1 ×α 2 ×α 3 ;

[0027] When the image resolution and the number of backlight partitions are not integer multiples, the multiple is rounded first, then the three downsampling rates are calculated, and finally the size error is repaired by the padding parameter in the convolution process.

[0028] Optionally, the step of inputting the second sample image and the first sample backlight data into a simulator, and the simulator outputting a simulation result, specifically includes:

[0029] inputting the second sample image and the first sample backlight data into a simulator for quantization, power limitation, diffusion simulation, image compensation and simulation processing;

[0030] In the simulator, the first sample backlight data is quantized and power-limited to obtain second sample backlight data; wherein the power-limiting method includes a curve mapping method or a table lookup mapping method;

[0031] Performing backlight diffusion simulation on the second sample backlight data to obtain third sample backlight data; wherein the backlight diffusion simulation method includes an interpolation method, a convolution method or a Gaussian blur method;

[0032] According to the third sample backlight data, image compensation with gain limitation is performed on the second sample image to obtain a third sample image:

[0033]

[0034] Among them, T i,j represents the normalized pixel value of the third sample image, clip(·) represents the truncation function, γ represents the maximum compensation ratio, I i,j represents the normalized pixel value of the second sample image, B i,j represents the third sample backlight data, i represents the row index, and j represents the column index;

[0035] According to the third sample backlight data and the third sample image, the image perceived by the human eye is simulated to obtain a simulation result; wherein the simulation result is the brightness of the image perceived by the simulated human eye:

[0036]

[0037] in, It represents the brightness of the image perceived by the simulated human eye, and ε represents the light leakage rate.

[0038] Optionally, the error is a loss of an image perceived by a simulated human eye relative to the second sample image;

[0039] The calculating the error according to the simulation result, and training and adjusting the backlight extraction network according to the error to obtain a target backlight extraction network specifically includes:

[0040] According to the simulated image perceived by human eyes, the second sample image and the first sample backlight data, a loss Loss of the simulated image perceived by human eyes relative to the second sample image is calculated using a loss function:

[0041]

[0042] Wherein, Loss represents the loss of the simulated image perceived by the human eye relative to the second sample image, MSE(·) is the root mean square error function, is the backlight data of the first sample, β is the power adjustment ratio, and mean(·) is the function for calculating the average value;

[0043] The backlight extraction network is trained and adjusted according to the loss to obtain a target backlight extraction network.

[0044] Optionally, performing backlight extraction processing on the input video stream based on the target backlight extraction network to obtain the target video stream specifically includes:

[0045] Acquire an input video stream and decode the input video stream to obtain a first image;

[0046] Preprocessing the first image to obtain a second image; wherein the preprocessing includes image normalization and gamma correction;

[0047] Inputting the second image into the target backlight extraction network to extract backlight data, and the target backlight extraction network outputs first backlight data corresponding to the second image;

[0048] Performing backlight diffusion preprocessing on the first backlight data to obtain second backlight data and storing the second backlight data in an on-chip RAM; wherein the backlight diffusion preprocessing includes quantization, power limitation, and calculation of mutual influence between different partitions;

[0049] Using a bilinear interpolation method, performing backlight diffusion on the second backlight data, diffusing the second backlight data to a size consistent with that of the second image, and obtaining third backlight data;

[0050] performing image compensation with gain limitation on the second image according to the third backlight data of the previous frame image to obtain a third image;

[0051] Encoding the third image to obtain a target video stream;

[0052] The step of inputting the second image into the target backlight extraction network to extract backlight data and the step of performing image compensation with gain limitation on the second image according to the third backlight data of the previous frame image are performed simultaneously.

[0053] Optionally, the adopting a bilinear interpolation method to perform backlight diffusion on the second backlight data, diffusing the second backlight data to a size consistent with that of the second image, and obtaining the third backlight data specifically includes:

[0054] Using a bilinear interpolation method, performing backlight diffusion on the second backlight data, diffusing the second backlight data to a size consistent with that of the second image, and obtaining third backlight data;

[0055] The third backlight data is calculated based on the left linear interpolation result and the right linear interpolation result, the left linear interpolation result is calculated based on the second backlight data in the upper left corner and the second backlight data in the lower left corner, and the right linear interpolation result is calculated based on the second backlight data in the upper right corner and the third backlight data in the lower right corner:

[0056]

[0057]

[0058] Wherein, D(x, y) represents the third backlight data, M 1 (x 1 ,y) represents the left linear interpolation result, M 2 (x 2 ,y) represents the right linear interpolation result, R 1 (x 1 ,y 1 ), R 2 (x 2 ,y 1 ), R 3 (x 1 ,y 2 ) and R 4 (x 2 ,y 2 ) represent the second backlight data in the upper left corner, the second backlight data in the upper right corner, the second backlight data in the lower left corner and the third backlight data in the lower right corner, x' represents the row index of the second backlight data before interpolation, y' represents the column index of the second backlight data before interpolation, x 1 Indicates the row index of the upper left corner second backlight data and the lower left corner second backlight data, x 2 Indicates the row index of the upper right corner second backlight data and the lower right corner second backlight data, y 1 Indicates the column index of the upper left corner second backlight data and the upper right corner second backlight data, y 2 represents the column index of the lower left corner second backlight data and the lower right corner second backlight data, x represents the row index of the third backlight data, y represents the column index of the third backlight data, and ratio' represents the interpolation ratio.

[0059] In order to achieve the above-mentioned invention object, the present invention also provides a CNN-based regional backlight extraction system, which is used to implement the steps of the CNN-based regional backlight extraction method as described above, and the CNN-based regional backlight extraction system includes a backlight extraction training system and a backlight extraction application system. The backlight extraction training system is used to train and adjust the backlight extraction network according to the sample image and the simulation results of the simulator to obtain the target backlight extraction network; the backlight extraction application system is deployed on an FPGA platform, and the backlight extraction application system is used to perform backlight extraction processing on the input video stream based on the target backlight extraction network to obtain the target video stream.

[0060] Optionally, the backlight extraction training system includes a preprocessing unit, a backlight extraction network and a simulator connected in sequence, the output end of the preprocessing unit is also connected to the input end of the simulator, the output end of the simulator is also connected to the input end of the backlight extraction network, and the simulator includes a quantization module, a power limiting module, a backlight diffusion simulation module, an image compensation module with gain limitation, a simulation module and an error calculation module connected in sequence.

[0061] Optionally, the backlight extraction application system includes a controller, and a video stream decoding module, a preprocessing module, a backlight data extraction module, a backlight diffusion preprocessing module, a backlight data storage module, a backlight diffusion module, an image compensation module and a video stream encoding module, which are respectively connected to the controller and connected in sequence, and the output end of the preprocessing module is also connected to the input end of the image compensation module, wherein the backlight data extraction module includes the target backlight extraction network.

[0062] In the present invention, a first sample image is obtained, and the first sample image is preprocessed to obtain a second sample image; the second sample image is input into a backlight extraction network based on CNN (Convolutional Neural Network), and the backlight extraction network outputs first sample backlight data corresponding to the second sample image through serial three-step downsampling, wherein the three downsampling rates corresponding to the serial three-step downsampling are three factors of the rounded image resolution and the multiple of the number of backlight partitions; the second sample image and the first sample backlight data are input into a simulator for quantization, power limitation, diffusion simulation, image compensation and simulation processing, and the simulator outputs a simulation result; the error is calculated according to the simulation result, and the backlight extraction network is trained and adjusted according to the error to obtain a target backlight extraction network; based on the target backlight extraction network, the input video stream is subjected to backlight extraction processing to obtain a target video stream. The present invention adopts a serial three-step downsampling method to extract backlight data. The three downsampling rates are three factors of the image resolution and the multiples of the backlight partition number. If the multiple is not an integer, the multiple is rounded down, and the gap is filled by using the convolution filling parameter. It can be flexibly applied to backlight modules with different numbers of partitions and different application scenarios, avoiding manual adjustment of algorithm details and saving manpower. A similar pipeline method is used for each step of calculation between and within modules, reducing the demand for storage units and eliminating the need for off-chip DDR (Double Data Rate) storage unit assistance. Quantization, power limitation, image compensation and other modules are added to the simulator to more realistically simulate the human eye perception effect, and the simulation results are used to train and adjust the backlight extraction network to improve the performance of the backlight extraction network. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] Figure 1 It is a flow chart of a preferred embodiment of the regional backlight extraction method based on CNN of the present invention;

[0064] Figure 2 is a structural diagram of a backlight extraction network of the present invention;

[0065] Figure 3 Schematic diagram of the dilated convolution and the required buffering of the present invention;

[0066] Figure 4 is a schematic diagram of the bilinear interpolation method of the present invention;

[0067] Figure 5 It is a work flow chart of a preferred embodiment of the regional backlight extraction system based on CNN of the present invention;

[0068] Figure 6 is a structural diagram of the backlight extraction training system of the present invention;

[0069] Figure 7 It is a structural diagram of the simulator of the present invention;

[0070] Figure 8 It is a structural diagram of the backlight extraction application system of the present invention. DETAILED DESCRIPTION

[0071] In order to make the purpose, technical solution and advantages of the present invention clearer and more specific, the present invention is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0072] Traditional backlight extraction technology usually only considers the values ​​of the current backlight area and the adjacent backlight area. Since different backlight modules have different parameters, traditional backlight extraction technology has problems such as large halo and the need to manually adjust the algorithm details according to different backlight modules and different application scenarios. In order to solve this problem, the current regional dimming technology often uses a "patching" method for manual correction, such as further processing the backlight data after extracting it using traditional backlight extraction technology, and performing detailed compensation on the image. These methods can greatly alleviate the existing problems, but require a lot of manpower for manual adjustment, which is a lot of work.

[0073] In addition, due to the huge amount of data contained in ultra-high-definition images, a large number of storage units are often required to assist in the transmission and processing of ultra-high-definition images; the operation of neural network algorithms often requires the assistance of a large number of storage units. Therefore, deploying neural network algorithms on FPGAs requires not only high computing power, but also a large number of storage units. In order to solve this problem, most products currently use methods such as optimizing neural network algorithms and designing dedicated computing units for neural network calculations. These methods alleviate the huge demand for storage units by reducing the scale of neural network algorithms and increasing the computing power and storage capacity of the device, but there are many gaps between each computing link, and the computing power and storage capacity of the device are not fully utilized.

[0074] In summary, the current backlight extraction technology has problems such as excessive halo and different effects on different backlight modules and different application scenarios because the algorithm only considers the current backlight and adjacent backlight data, and because different backlight modules have different parameters. The current solution to the problem of excessive halo and different effects on different backlight modules and different application scenarios is usually to manually add different processing in certain links from backlight extraction to display, which is a lot of work. In addition, the current FPGA platform for processing ultra-high-definition images has a high demand for storage units (such as off-chip DDR), and the deployment of neural networks on FPGAs also requires a large number of storage units.

[0075] In order to solve the above technical problems, the present invention provides a regional backlight extraction method based on CNN, which extracts backlight data by a serial three-step downsampling method. The three downsampling rates are three factors of the image resolution and the multiple of the number of backlight partitions. If the multiple is not an integer, the multiple is rounded down, and the gap is filled by using the filling parameter of the convolution. The method can be flexibly applied to backlight modules with different numbers of partitions and different application scenarios, avoiding manual adjustment of algorithm details and saving manpower. A similar pipeline method is used for each step of calculation between and within modules, reducing the demand for storage units and eliminating the need for external DDR storage units. Quantization, power limitation, image compensation and other modules are added to the simulator to more realistically simulate the human eye perception effect, and the simulation results are used to train and adjust the backlight extraction network to improve the performance of the backlight extraction network.

[0076] The application content is further explained below through the description of embodiments in conjunction with the accompanying drawings.

[0077] A preferred embodiment of the regional backlight extraction method based on CNN of the present invention is as follows: Figure 1 As shown, specifically including:

[0078] S1. Acquire a first sample image, and preprocess the first sample image to obtain a second sample image.

[0079] In an implementation of this embodiment, the acquiring the first sample image and preprocessing the first sample image to obtain the second sample image specifically includes:

[0080] Acquire a first sample image; wherein the first sample image includes an RGB image, an HDR image, an image with a resolution greater than a first set threshold, or an image with a contrast greater than a second set threshold;

[0081] Preprocessing the first sample image to obtain a second sample image;

[0082] The preprocessing includes image normalization and Gamma correction.

[0083] S2. Input the second sample image into a CNN-based backlight extraction network, and the backlight extraction network outputs the first sample backlight data corresponding to the second sample image through serial three-step downsampling, wherein the three downsampling rates corresponding to the serial three-step downsampling are three factors of the rounded image resolution and multiples of the number of backlight partitions.

[0084] In one implementation of this embodiment, Figure 2As shown, the CNN-based backlight extraction network includes a first convolution module 121, a second convolution module 122 and a third convolution module 123, wherein the first convolution module 121 includes two layers of dilated convolution and one layer of maximum pooling, the second convolution module 122 includes four layers of dilated convolution and one layer of maximum pooling, and the third convolution module 123 includes two layers of convolution and one layer of sampling;

[0085] The step of inputting the second sample image into a CNN-based backlight extraction network, wherein the backlight extraction network outputs first sample backlight data corresponding to the second sample image through serial three-step downsampling, specifically includes:

[0086] The second sample image is feature extracted using the two-layer dilated convolution and ReLU activation function in the first convolution module, and the first downsampling is performed using the maximum pooling in the first convolution module to obtain a first feature map; wherein the downsampling rate of the first downsampling is α 1 ;

[0087] The first feature map is extracted by using the four-layer hole convolution and ReLU activation function in the second convolution module, and the second feature map is obtained by using the maximum pooling in the second convolution module; wherein the downsampling rate of the second downsampling is α 2 ;

[0088] The second feature map is extracted by using the two-layer convolution and ReLU activation function in the third convolution module, and the convolution in the third convolution module is used to perform a third downsampling with a step size to obtain the initial first sample backlight data; wherein the downsampling rate of the third downsampling is α 3 ;

[0089] Normalizing the initial first sample backlight data using a Sigmoid activation function to obtain normalized first sample backlight data;

[0090] Among them, the formula of the dilated convolution is:

[0091]

[0092] Among them, c out represents the output channel index, Indicates the output channel index c out The corresponding convolution output, i' represents the feature map row index, j' represents the feature map column index, C in represents the number of input channels, c represents the channel index, bias crepresents the convolution bias corresponding to the channel index c, ks represents half of the convolution kernel size rounded down, kernel_size represents the convolution kernel size, u represents the convolution kernel row index, v represents the convolution kernel column index, input c represents the input feature map corresponding to the channel index c, ds represents the interval between adjacent pixels, and weight c Represents the convolution kernel data corresponding to the channel index c;

[0093] When the image resolution and the number of backlight partitions are integer multiples, the downsampling rate α 1 , α 2 and α 3 are three factors of the multiples of the image resolution and the number of backlight partitions, and the multiple ratio is the product of three downsampling rates:

[0094] ratio = α 1 ×α 2 ×α 3 ;

[0095] When the image resolution and the number of backlight partitions are not integer multiples, the multiple is rounded first, then the three downsampling rates are calculated, and finally the size error is repaired by the padding parameter in the convolution process.

[0096] Specifically, the backlight extraction network of the present invention adopts a serial three-step downsampling method to extract backlight data. The three downsampling rates are factors of the multiple relationship between the image resolution and the number of backlight partitions. If the multiple relationship is not an integer, the multiple relationship is rounded down and the gap is filled using the convolution filling parameters. It can be flexibly applied to backlight systems with different numbers of partitions. The backlight extraction network does not contain complex modules such as residual connections and attention, and can be more easily deployed on a hardware platform.

[0097] In order to reduce the demand for storage units, especially off-chip DDR resources, the present invention has optimized the pipeline of the overall process and the internal modules. The backlight extraction network proposed in the present invention is only a serial structure, so it can be calculated while inputting the image without waiting for the entire image to be transmitted. Figure 3 As shown in the figure, since each convolution operation only involves part of the data, only a few lines need to be cached to start the operation, and then the convolution operation is performed simultaneously with the input of image pixel data. Similarly, the pooling operation does not involve the complete feature map input, so a pipeline operation similar to convolution can also be completed.

[0098] S3, inputting the second sample image and the first sample backlight data into a simulator for quantization, power limitation, diffusion simulation, image compensation and simulation processing, and the simulator outputs a simulation result.

[0099] In an implementation of this embodiment, the second sample image and the first sample backlight data are input into a simulator for quantization, power limitation, diffusion simulation, image compensation and simulation processing, and the simulator outputs a simulation result, specifically including:

[0100] inputting the second sample image and the first sample backlight data into a simulator;

[0101] In the simulator, the first sample backlight data is quantized and power-limited to obtain second sample backlight data; wherein the power-limiting method includes a curve mapping method or a table lookup mapping method;

[0102] Performing backlight diffusion simulation on the second sample backlight data to obtain third sample backlight data; wherein the backlight diffusion simulation method includes an interpolation method, a convolution method or a Gaussian blur method;

[0103] According to the third sample backlight data, image compensation with gain limitation is performed on the second sample image to obtain a third sample image:

[0104]

[0105] Among them, T i,j represents the normalized pixel value of the third sample image, clip(·) represents the truncation function, γ represents the maximum compensation ratio, I i,j represents the normalized pixel value of the second sample image, B i,j represents the third sample backlight data, i represents the row index, and j represents the column index; the data is normalized, so a truncation function clip(·) is also required to prevent overflow.

[0106] According to the third sample backlight data and the third sample image, the image perceived by the human eye is simulated to obtain a simulation result; wherein the simulation result is the brightness of the image perceived by the simulated human eye:

[0107]

[0108] in, It represents the brightness of the image perceived by the simulated human eye, and ε represents the light leakage rate.

[0109] Specifically, in the simulator, since the backlight value of the backlight module is often an integer value, and the backlight value obtained by the backlight extraction network is often a floating point number, in order to better simulate the display effect, a quantization module is used to quantize the backlight value obtained by the backlight extraction network into an integer value, and the number of bits of the final integer backlight value includes but is not limited to 8 bits, 12 bits, etc. When the number of partitions is too large, it is difficult for the whole machine power to drive all lamp beads to the maximum brightness. In order to obtain better simulation results, the backlight value actually transmitted to the backlight module is calculated at the same time, and a power limiting module is used to limit the overall power. The power limiting method includes but is not limited to curve mapping, table lookup mapping, etc. Since different backlight partitions will affect each other, a backlight diffusion simulation module is used to simulate the backlight that ultimately acts on the LCD screen. The backlight diffusion simulation can use methods including but not limited to interpolation, convolution, Gaussian blur, etc. Since the reduction of the backlight will lead to a reduction in the brightness of the actual display, it is necessary to compensate the image to increase the transmittance of the LCD screen so that the final display effect is closer to the original image. In actual use, overexposure is often caused by too low backlight data, which leads to too high compensation multiples. Therefore, the image compensation module proposed in the present invention limits the maximum compensation ratio to γ, which is (0,20]. The maximum compensation ratio can be adjusted according to the module. The calculation method of the compensation value is: The present invention uses a simulation module to simulate the image perceived by the human eye. The brightness perceived by the human eye can be calculated from the backlight brightness and the transmittance of the LCD screen. These two values ​​can be obtained from the backlight data and the pixel data of the compensation image. At the same time, since the LCD module has a certain amount of light leakage, it is assumed that the light leakage rate is ε, and the final brightness perceived by the human eye is

[0110] S4. Calculate an error according to the simulation result, and train and adjust the backlight extraction network according to the error to obtain a target backlight extraction network.

[0111] In an implementation of this embodiment, the error is a loss of an image perceived by a simulated human eye relative to the second sample image;

[0112] The calculating the error according to the simulation result, and training and adjusting the backlight extraction network according to the error to obtain a target backlight extraction network specifically includes:

[0113] According to the simulated image perceived by human eyes, the second sample image and the first sample backlight data, a loss Loss of the simulated image perceived by human eyes relative to the second sample image is calculated using a loss function:

[0114]

[0115] Wherein, Loss represents the loss of the simulated image perceived by the human eye relative to the second sample image, MSE(·) is the root mean square error function, is the backlight data of the first sample, β is the power adjustment ratio, and mean(·) is the function for calculating the average value;

[0116] The backlight extraction network is trained and adjusted according to the loss to obtain a target backlight extraction network.

[0117] Specifically, the present invention uses an error calculation module to guide the backlight extraction network training adjustment. The backlight value extracted by the backlight extraction network is expected to make the image perceived by the human eye close to the original image, and at the same time, the power is expected to be low. Therefore, the loss function used in the present invention is

[0118] S5. Based on the target backlight extraction network, perform backlight extraction processing on the input video stream to obtain a target video stream.

[0119] In an implementation of this embodiment, performing backlight extraction processing on the input video stream based on the target backlight extraction network to obtain the target video stream specifically includes:

[0120] Acquire an input video stream and decode the input video stream to obtain a first image;

[0121] Preprocessing the first image to obtain a second image; wherein the preprocessing includes image normalization and gamma correction;

[0122] Inputting the second image into the target backlight extraction network to extract backlight data, and the target backlight extraction network outputs first backlight data corresponding to the second image;

[0123] Performing backlight diffusion preprocessing on the first backlight data to obtain second backlight data and storing the second backlight data in an on-chip random access memory (RAM); wherein the backlight diffusion preprocessing includes quantization, power limitation, and calculation of mutual influence between different partitions;

[0124] Using a bilinear interpolation method, performing backlight diffusion on the second backlight data, diffusing the second backlight data to a size consistent with that of the second image, and obtaining third backlight data;

[0125] performing image compensation with gain limitation on the second image according to the third backlight data of the previous frame image to obtain a third image;

[0126] Encoding the third image to obtain a target video stream;

[0127] Among them, the inputting of the second image into the target backlight extraction network for backlight data extraction and the image compensation with gain limitation for the second image based on the third backlight data of the previous frame image are performed simultaneously (described here from the image perspective).

[0128] Specifically, the present invention uses a video stream decoding module to receive a video stream, including but not limited to HDMI, DP, V-By-One and other video stream data, and then decodes the video stream data into common image data, referred to as a first image, including a data valid signal DE, a column synchronization signal HS, and a row synchronization signal VS. The first image is input into a preprocessing module to complete image preprocessing work, including but not limited to image normalization, Gamma correction, etc., and output a second image. The second image is input into a backlight data extraction module, and appropriate backlight data is extracted according to the second image through a convolutional neural network CNN, referred to as the first backlight data; the convolutional neural network is implemented in a pipeline manner, and only part of the data is temporarily stored in the on-chip RAM, without the need for additional off-chip DDR to store intermediate data. The first backlight data is input into a backlight diffusion preprocessing module, and the first backlight data is subjected to backlight diffusion preprocessing, mainly including quantization, power limitation and Gaussian blur, and the second backlight data is output. The second backlight data is input into a backlight diffusion module, and the second backlight data is diffused to be consistent with the size of the second image, and the diffusion is completed using a bilinear interpolation method, and the third backlight data is output. Backlight diffusion and image compensation are used together in a pipeline manner to achieve synchronous operation of backlight diffusion and image compensation. The second image and the third backlight data are input into the image compensation module, and the second image is compensated to a certain extent according to the third backlight data. The third image is output using the above-mentioned image compensation method with gain limitation. The image compensation module is implemented in a pipeline manner, and there is no need to wait for all data to be prepared before starting the calculation. It completes the backlight diffusion and image compensation work together with the backlight diffusion module. When the number of partitions is not high, the amount of backlight data is not large, so the backlight data storage module, that is, the on-chip RAM, is used to store the second backlight data for the calculation of the backlight diffusion module. Finally, the video stream encoding module is used to encode the third image into HDMI, DP, V-By-One and other video streams for dot screen.

[0129] It should be noted that image compensation requires backlight calculation, but since the difference between two video frames is small, the present invention uses the backlight data of the previous frame (i.e., the third backlight data) to display with the current frame image (i.e., the second image) to complete the overall pipeline operation. The specific steps are:

[0130] When receiving each line of image data (V_active), preprocessing, backlight extraction and other operations are completed, and the image (second image) is compensated by using the third backlight data of the previous frame, and then the compensated image (third image) of this frame and the first backlight data of the previous frame are output and displayed (described here from the perspective of pixels);

[0131] During the interval (V_blank) of receiving each line of image data, the backlight diffusion preprocessing of the first backlight data of the current frame is completed and stored in the on-chip RAM for calculation in the next frame.

[0132] It should be noted that backlight extraction has both narrow and broad meanings. In a narrow sense, it refers to backlight data extraction, and in a broad sense, it includes processes such as video stream decoding, preprocessing, backlight data extraction, backlight diffusion preprocessing, backlight data storage, backlight diffusion, image compensation and video stream encoding.

[0133] In an implementation of this embodiment, the bilinear interpolation method is used to perform backlight diffusion on the second backlight data, and the second backlight data is diffused to a size consistent with the size of the second image to obtain the third backlight data, which specifically includes:

[0134] Using a bilinear interpolation method, performing backlight diffusion on the second backlight data, diffusing the second backlight data to a size consistent with that of the second image, and obtaining third backlight data;

[0135] The third backlight data is calculated based on the left linear interpolation result and the right linear interpolation result, the left linear interpolation result is calculated based on the second backlight data in the upper left corner and the second backlight data in the lower left corner, and the right linear interpolation result is calculated based on the second backlight data in the upper right corner and the third backlight data in the lower right corner:

[0136]

[0137] Wherein, D(x, y) represents the third backlight data, M 1 (x 1 ,y) represents the left linear interpolation result, M 2 (x 2 ,y) represents the right linear interpolation result, R 1 (x 1 ,y 1 ), R 2 (x 2 ,y 1 ), R 3 (x 1 ,y 2 ) and R 4 (x 2 ,y 2 ) represent the second backlight data in the upper left corner, the second backlight data in the upper right corner, the second backlight data in the lower left corner and the third backlight data in the lower right corner, x' represents the row index of the second backlight data before interpolation, y' represents the column index of the second backlight data before interpolation, x 1 Indicates the row index of the upper left corner second backlight data and the lower left corner second backlight data, x2 Indicates the row index of the upper right corner second backlight data and the lower right corner second backlight data, y 1 Indicates the column index of the upper left corner second backlight data and the upper right corner second backlight data, y 2 represents the column index of the lower left corner second backlight data and the lower right corner second backlight data, x represents the row index of the third backlight data, y represents the column index of the third backlight data, and ratio' represents the interpolation ratio.

[0138] Specifically, the algorithm used by the backlight diffusion module of the present invention is a bilinear interpolation method, such as Figure 4 As shown, the third backlight data is calculated based on the left linear interpolation result and the right linear interpolation result, the left linear interpolation result is calculated based on the second backlight data in the upper left corner and the second backlight data in the lower left corner, and the right linear interpolation result is calculated based on the second backlight data in the upper right corner and the third backlight data in the lower right corner. It can be seen that backlight diffusion does not require complete second backlight data and can also complete pipeline operation.

[0139] To aid in illustration, the following examples are provided.

[0140] The present invention firstly trains and adjusts the backlight extraction network using a simulated image on a device with a GPU (Graphics Processing Unit) so that the backlight extraction network can extract appropriate backlight data. The specific steps are as follows:

[0141] Step 1: Read batch_size sample images from the sample data, for example, 32 sample images, and perform preprocessing operations on the read sample images, including image normalization and gamma correction, to obtain preprocessed sample images (second sample images).

[0142] Step 2: Use the current network (backlight extraction network) parameters to extract appropriate backlight data from the second sample image to obtain the first sample backlight data; this process can be accelerated by using a GPU.

[0143] Step 3: Use the first sample backlight data and the second sample image to simulate the image perceived by the human eye, specifically including: quantizing the first sample backlight data, for example, quantizing to 8 bits; power limiting the quantized sample backlight data, specifically: first calculate the mean of the sample backlight data, then according to a pre-designed curve, look up the table according to the mean value to obtain the scaling factor of the sample backlight data (in the range of (0,1]) and apply it to the sample backlight data to obtain the second backlight data; diffuse the second sample backlight data using bilinear interpolation to obtain the third sample backlight data; compensate the second sample image with gain limitation according to the third sample backlight data, the highest compensation ratio can be 5, and obtain the compensated image (third sample image); simulate the result according to the third sample backlight data and the third sample image, the light leakage ratio can be 0.05, and obtain a simulated image (i.e., the simulated image perceived by the human eye); calculate the loss according to the simulated image, the second sample image and the first sample backlight data, and finally train and adjust the backlight extraction network according to the loss.

[0144] Step 4: If all sample data have not been read, return to step 1 to continue training; if all sample data have been read, jump to step 5.

[0145] Step 5: Repeat steps 1 to 4 and train several times, for example, 1000 times.

[0146] After completing the training of the backlight extraction network, the hardware system is deployed on the FPGA platform. Figure 8 As shown, the implemented hardware system includes a video stream decoding module 21, a video stream encoding module 28, a preprocessing module 22, a backlight data extraction module 23, a backlight diffusion preprocessing module 24, a backlight diffusion module 26, an image compensation module 27, a backlight data storage module 25 and a controller 29. The pipeline optimization method is adopted within and between each module, as follows:

[0147] The video stream decoding module and the video stream encoding module are used for decoding and encoding of video streams, such as the decoder and encoder of HDMI 2.1IP; the video signal includes pixel clock, data valid signal DE, horizontal synchronization signal HS, vertical synchronization signal VS and pixel data.

[0148] The controller can be a hard core embedded in the FPGA platform, such as an ARM core, or a soft core built using FPGA resources, such as a NIOS core. The controller is mainly used for the control and scheduling of each module, especially the control and scheduling of the video stream decoding module and the video stream encoding module.

[0149] The preprocessing module is mainly used for image preprocessing, such as gamma correction, which can be completed by table lookup. Since the calculation process of each pixel does not involve other pixels, the preprocessing can be completed after receiving each pixel without waiting for the complete image input.

[0150] The backlight data extraction module is mainly used for backlight extraction and consists of three layers of convolutional neural networks. Since each layer of convolution only requires a part of the input data, the operation can be started after waiting for the required data to be input. Taking the 4K image and the first layer of convolution as an example, the first layer of convolution uses a dilated convolution with a dilated rate of 3, a step size of 1, and a padding parameter of 3. Therefore, each convolution requires 9 data; since the dilated rate is 3, the required data spans 7 rows. Since the padding parameter is 3 and the image data is input row by row through line scanning, the actual number of pixels that need to be waited for is 3 rows + 3 data, and then a result data can be calculated for each pixel data that arrives.

[0151] The Gaussian blur kernel used in the backlight diffusion preprocessing module uses convolution to complete the blurring, and the pipeline method is similar to the backlight extraction network. The processed second backlight data is then stored in the backlight data storage module, i.e., the on-chip RAM, for use in the next frame of image compensation. Since there are intervals between each row and each frame for video signal synchronization during image data transmission, for example, there will be an interval of 560 pixel clocks between the line synchronization signals of a typical 4K signal, which means that a row of 3840 pixels will actually transmit 4400 pixels; similarly, there will be an interval of 90 rows between each frame, which means that a frame of 2160 rows of pixels will actually transmit 2250 rows of pixels. Therefore, there is enough time to calculate the required data before the pixel data of the next row or frame arrives.

[0152] The backlight diffusion module adopts a bilinear interpolation method, and only four pixels are needed at most to calculate each pixel value. Since the second backlight data has been stored in the on-chip RAM, a diffused backlight data (third backlight data) can be calculated when each pixel data is input. After that, the image compensation module with gain limitation can use the input third backlight data and the preprocessed image data (second image) to complete the image compensation.

[0153] In addition, the present invention also provides a CNN-based regional backlight extraction system, which is used to implement the steps of the CNN-based regional backlight extraction method as described above, and the CNN-based regional backlight extraction system includes a backlight extraction training system and a backlight extraction application system. The backlight extraction training system is used to train and adjust the backlight extraction network according to the sample image and the simulation results of the simulator to obtain a target backlight extraction network; the backlight extraction application system is deployed on an FPGA platform, and the backlight extraction application system is used to perform backlight extraction processing on the input video stream based on the target backlight extraction network to obtain a target video stream.

[0154] Specifically, the operation process of the CNN-based regional backlight extraction system is as follows: Figure 5 As shown, the system is divided into three modes: "training", "testing" and "deployment". When the system is in the "training" mode, after the input sample image is preprocessed, the backlight extraction network extracts appropriate backlight data, and the backlight data and the preprocessed sample image are sent to the simulator for simulation. The error is calculated according to the simulation result and the backlight extraction network is trained and adjusted. After the image preprocessing, the backlight extraction network extracts appropriate backlight data. At this time, the sample image (image data set) is preferably an HDR image or other image with high contrast. When the system is in the "testing" mode, after the input sample image is preprocessed, the backlight extraction network extracts appropriate backlight data, and the backlight data and the preprocessed sample image are sent to the simulator for simulation, and the simulation results are output and saved. When the system is in the "deployment" mode, after the input image is preprocessed, the backlight extraction network extracts appropriate backlight data, and the backlight data can be transmitted to the backlight module to control the brightness of each partition lamp bead; the simulation module of the simulator calculates the compensation image, and the image can be transmitted to the liquid crystal module to display the image.

[0155] In one implementation of this embodiment, Figure 6 As shown, the backlight extraction training system includes a pre-processing unit 11, a backlight extraction network 12 and a simulator 13 connected in sequence, the output end of the pre-processing unit 11 is also connected to the input end of the simulator 13, and the output end of the simulator 13 is also connected to the input end of the backlight extraction network 12, as shown in FIG. Figure 7 As shown, the simulator 13 includes a quantization module 131, a power limiting module 132, a backlight diffusion simulation module 133, an image compensation module with gain limitation 134, a simulation module 135 and an error calculation module 136 which are connected in sequence.

[0156] Specifically, the backlight extraction training system provided by the present invention mainly includes three parts: a preprocessing unit, a backlight extraction network and a simulator. The preprocessing unit preprocesses the input sample image (such as an RGB image), including but not limited to normalization, Gamma correction, etc.; the backlight extraction network can extract appropriate backlight data from the sample image; the simulator simulates the human eye perception effect according to the sample image and the backlight data, and trains and adjusts the backlight extraction network under certain conditions. The present invention can use information containing different types of image data sets and the simulation results of the simulator to train and adjust the backlight extraction network, and finally can extract appropriate backlight data according to the input image; this method can be deployed on an FPGA platform, with low requirements for storage units, and does not require the assistance of off-chip storage units. The functions of each module in the simulator have been described above and will not be repeated here.

[0157] In one implementation of this embodiment, Figure 8 As shown, the backlight extraction application system includes a controller 29, and a video stream decoding module 21, a preprocessing module 22, a backlight data extraction module 23, a backlight diffusion preprocessing module 24, a backlight data storage module 25, a backlight diffusion module 26, an image compensation module 27 and a video stream encoding module 28, which are respectively connected to the controller 29 and connected in sequence, and the output end of the preprocessing module 22 is also connected to the input end of the image compensation module 27, wherein the backlight data extraction module 23 includes the target backlight extraction network.

[0158] Specifically, the backlight extraction application system of the present invention is deployed on the FPGA platform, and there is no need to use off-chip DDR to cache images and feature maps in neural network operations. Among them, the controller is used to control and schedule each module in the entire backlight extraction application system. The functions of other modules have been described above and will not be repeated here.

[0159] It should be understood that the application of the present invention is not limited to the above examples. For ordinary technicians in this field, improvements or changes can be made based on the above description. All these improvements and changes should fall within the scope of protection of the claims attached to the present invention.

Claims

1. A CNN-based regional backlight extraction method, characterized in that: The CNN-based regional backlight extraction method comprises: Acquire a first sample image, and preprocess the first sample image to obtain a second sample image; Inputting the second sample image into a CNN-based backlight extraction network, the backlight extraction network outputs first sample backlight data corresponding to the second sample image through serial three-step downsampling, wherein three downsampling rates corresponding to the serial three-step downsampling are three factors of the rounded image resolution and multiples of the number of backlight partitions; inputting the second sample image and the first sample backlight data into a simulator for quantization, power limitation, diffusion simulation, image compensation and simulation processing, the simulator outputting a simulation result; Calculating an error according to the simulation result, and training and adjusting the backlight extraction network according to the error to obtain a target backlight extraction network; Based on the target backlight extraction network, backlight extraction processing is performed on the input video stream to obtain a target video stream.

2. The CNN-based regional backlight extraction method according to claim 1, characterized in that: The acquiring of the first sample image and preprocessing of the first sample image to obtain the second sample image specifically includes: Acquire a first sample image; wherein the first sample image includes an RGB image, an HDR image, an image with a resolution greater than a first set threshold, or an image with a contrast greater than a second set threshold; Preprocessing the first sample image to obtain a second sample image; The preprocessing includes image normalization and Gamma correction.

3. The CNN-based regional backlight extraction method according to claim 1, characterized in that: The CNN-based backlight extraction network includes a first convolution module, a second convolution module and a third convolution module, wherein the first convolution module includes two layers of dilated convolution and one layer of maximum pooling, the second convolution module includes four layers of dilated convolution and one layer of maximum pooling, and the third convolution module includes two layers of convolution and one layer of sampling; The step of inputting the second sample image into a CNN-based backlight extraction network, wherein the backlight extraction network outputs first sample backlight data corresponding to the second sample image through serial three-step downsampling, specifically includes: Using the two-layer dilated convolution and ReLU activation function in the first convolution module to extract features from the second sample image, and using the maximum pooling in the first convolution module to perform a first downsampling to obtain a first feature map; wherein the downsampling rate of the first downsampling is α1; Using the four-layer dilated convolution and ReLU activation function in the second convolution module to extract features from the first feature map, and using the maximum pooling in the second convolution module to perform a second downsampling to obtain a second feature map; wherein the downsampling rate of the second downsampling is α2; Using the two-layer convolution and ReLU activation function in the third convolution module to extract features from the second feature map, and using the convolution in the third convolution module with the step size to perform a third downsampling to obtain initial first sample backlight data; wherein the downsampling rate of the third downsampling is α3; Normalizing the initial first sample backlight data using a Sigmoid activation function to obtain normalized first sample backlight data; Among them, the formula of the dilated convolution is: Among them, c out represents the output channel index, Indicates the output channel index c out The corresponding convolution output, i' represents the feature map row index, j' represents the feature map column index, C in represents the number of input channels, c represents the channel index, bias c represents the convolution bias corresponding to the channel index c, ks represents half of the convolution kernel size rounded down, kernel_size represents the convolution kernel size, u represents the convolution kernel row index, v represents the convolution kernel column index, input c represents the input feature map corresponding to the channel index c, ds represents the interval between adjacent pixels, and weight c Represents the convolution kernel data corresponding to the channel index c; When the image resolution and the number of backlight partitions are integer multiples, the downsampling rates α1, α2 and α3 are three factors of the multiples of the image resolution and the number of backlight partitions, and the multiple ratio is the product of the three downsampling rates: ratio = α1 × α2 × α3; When the image resolution and the number of backlight partitions are not integer multiples, the multiple is rounded first, then the three downsampling rates are calculated, and finally the size error is repaired by the padding parameter in the convolution process.

4. The CNN-based regional backlight extraction method according to claim 1, characterized in that: The inputting of the second sample image and the first sample backlight data into the simulator for quantization, power limitation, diffusion simulation, image compensation and simulation processing, wherein the simulator outputs a simulation result, specifically includes: inputting the second sample image and the first sample backlight data into a simulator; In the simulator, the first sample backlight data is quantized and power-limited to obtain second sample backlight data; wherein the power-limiting method includes a curve mapping method or a table lookup mapping method; Performing backlight diffusion simulation on the second sample backlight data to obtain third sample backlight data; wherein the backlight diffusion simulation method includes an interpolation method, a convolution method or a Gaussian blur method; According to the third sample backlight data, image compensation with gain limitation is performed on the second sample image to obtain a third sample image: Among them, T i,j represents the normalized pixel value of the third sample image, clip(·) represents the truncation function, γ represents the maximum compensation ratio, I i,j represents the normalized pixel value of the second sample image, B i,j represents the third sample backlight data, i represents the row index, and j represents the column index; According to the third sample backlight data and the third sample image, the image perceived by the human eye is simulated to obtain a simulation result; wherein the simulation result is the brightness of the image perceived by the simulated human eye: in, It represents the brightness of the image perceived by the simulated human eye, and ε represents the light leakage rate.

5. The method for extracting area backlight based on CNN according to claim 4, characterized in that: The error is the loss of the image perceived by the simulated human eye relative to the second sample image; The calculating the error according to the simulation result, and training and adjusting the backlight extraction network according to the error to obtain a target backlight extraction network specifically includes: According to the simulated image perceived by human eyes, the second sample image and the first sample backlight data, a loss Loss of the simulated image perceived by human eyes relative to the second sample image is calculated using a loss function: Wherein, Loss represents the loss of the simulated human eye image relative to the second sample image, MSE(·) is the root mean square error function, and A i,j is the first sample backlight data, β is the power adjustment ratio, mean(·) is the function for calculating the average value; The backlight extraction network is trained and adjusted according to the loss to obtain a target backlight extraction network.

6. The CNN-based regional backlight extraction method according to claim 1, characterized in that: The step of performing backlight extraction processing on the input video stream based on the target backlight extraction network to obtain the target video stream specifically includes: Acquire an input video stream and decode the input video stream to obtain a first image; Preprocessing the first image to obtain a second image; wherein the preprocessing includes image normalization and gamma correction; Inputting the second image into the target backlight extraction network to extract backlight data, and the target backlight extraction network outputs first backlight data corresponding to the second image; Performing backlight diffusion preprocessing on the first backlight data to obtain second backlight data and storing the second backlight data in an on-chip RAM; wherein the backlight diffusion preprocessing includes quantization, power limitation, and calculation of mutual influence between different partitions; Using a bilinear interpolation method, backlight diffusion is performed on the second backlight data, and the second backlight data is diffused to a size consistent with that of the second image, so as to obtain third backlight data; performing image compensation with gain limitation on the second image according to the third backlight data of the previous frame image to obtain a third image; Encoding the third image to obtain a target video stream; The step of inputting the second image into the target backlight extraction network to extract backlight data and the step of performing image compensation with gain limitation on the second image according to the third backlight data of the previous frame image are performed simultaneously.

7. The CNN-based regional backlight extraction method according to claim 6, characterized in that: The adopting a bilinear interpolation method to perform backlight diffusion on the second backlight data, diffusing the second backlight data to a size consistent with that of the second image, and obtaining third backlight data specifically includes: Using a bilinear interpolation method, performing backlight diffusion on the second backlight data, diffusing the second backlight data to a size consistent with that of the second image, and obtaining third backlight data; The third backlight data is calculated based on the left linear interpolation result and the right linear interpolation result, the left linear interpolation result is calculated based on the second backlight data in the upper left corner and the second backlight data in the lower left corner, and the right linear interpolation result is calculated based on the second backlight data in the upper right corner and the third backlight data in the lower right corner: Wherein, D(x,y) represents the third backlight data, M1(x1,y) represents the left linear interpolation result, M2(x2,y) represents the right linear interpolation result, R1(x1,y1), R2(x2,y1), R3(x1,y2) and R4(x2,y2) represent the second backlight data in the upper left corner, the second backlight data in the upper right corner, the second backlight data in the lower left corner and the third backlight data in the lower right corner respectively, x' represents the row index of the second backlight data before interpolation, y' represents the row index of the third backlight data before interpolation. The column index of the second backlight data is as follows: x1 represents the row index of the second backlight data in the upper left corner and the lower left corner, x2 represents the row index of the second backlight data in the upper right corner and the lower right corner, y1 represents the column index of the second backlight data in the upper left corner and the upper right corner, y2 represents the column index of the second backlight data in the lower left corner and the lower right corner, x represents the row index of the third backlight data, y represents the column index of the third backlight data, and ratio' represents the interpolation ratio.

8. A CNN-based regional backlight extraction system, used to implement the steps of the CNN-based regional backlight extraction method according to any one of claims 1 to 7, characterized in that: The CNN-based regional backlight extraction system includes a backlight extraction training system and a backlight extraction application system. The backlight extraction training system is used to train and adjust the backlight extraction network according to the sample image and the simulation results of the simulator to obtain a target backlight extraction network. The backlight extraction application system is deployed on an FPGA platform, and is used to perform backlight extraction processing on an input video stream based on the target backlight extraction network to obtain a target video stream.

9. The CNN-based regional backlight extraction system according to claim 8, characterized in that: The backlight extraction training system includes a preprocessing unit, a backlight extraction network and a simulator connected in sequence, the output end of the preprocessing unit is also connected to the input end of the simulator, the output end of the simulator is also connected to the input end of the backlight extraction network, and the simulator includes a quantization module, a power limiting module, a backlight diffusion simulation module, an image compensation module with gain limitation, a simulation module and an error calculation module connected in sequence.

10. The CNN-based regional backlight extraction system according to claim 8, characterized in that: The backlight extraction application system includes a controller, and a video stream decoding module, a preprocessing module, a backlight data extraction module, a backlight diffusion preprocessing module, a backlight data storage module, a backlight diffusion module, an image compensation module and a video stream encoding module, which are respectively connected to the controller and connected in sequence, and the output end of the preprocessing module is also connected to the input end of the image compensation module, wherein the backlight data extraction module includes the target backlight extraction network.

Images