Dynamic backlight regulation and control method and system of display device

Abstract

The invention discloses a dynamic backlight regulation and control method and system for a display device, and the method comprises the steps: forming a grayscale image according to the feature value of each pixel point in a current to-be-displayed original input image; inputting the grayscale image into a preset convolutional neural network encoder model to obtain a compressed feature map; inputting the compressed feature map into a preset coupling diffusion algorithm decoder model to obtain an updated image; adjusting and controlling the backlight of the display based on the gray value of each pixel point of the updated image; according to the method, the neural network method is adopted to extract the compression features of the image backlight, the adaptability of the algorithm to different display scenes is improved, the operation in the expansion stage is not too complex through the coupling of bilinear interpolation and nearest neighbor interpolation in the decoder stage, the relatively reliable and smooth image effect is provided, and the method is suitable for large-scale popularization and application. A ping-pong design and a pipeline architecture design are adopted for video stream processing, the degree of parallelism and the data throughput rate are increased, and the data processing efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of display backlight adjustment, and in particular to a dynamic backlight adjustment method and system for a display device. Background Art

[0002] Dynamic backlight control technology is an important application in modern display technology, especially in improving display effects and reducing energy consumption. With the continuous advancement of display technology, especially the rise of high dynamic range content, traditional backlight control methods mainly rely on fixed algorithms and parameters to adjust backlight brightness, which cannot provide the best visual experience and has poor generalization ability when facing different display scenarios.

[0003] For example, the patent with publication number CN110796988A discloses a backlight adjustment method and device, including obtaining the ambient light brightness; if the ambient light brightness is greater than a brightness threshold, adjusting the screen backlight brightness to be greater than a maximum preset brightness; detecting a temperature value through a first temperature sensor; when the temperature value detected by the first temperature sensor is greater than the temperature threshold, adjusting the screen backlight brightness to be less than or equal to the maximum preset brightness.

[0004] The above patents all have the problems raised by this background technology: the existing traditional display technology mostly adjusts the display backlight brightness based on preset backlight algorithms and parameters, which leads to its poor adaptability in the face of complex display scenes, and may result in low color contrast of the picture, or even distortion of some details, which is obviously not enough to meet the increasing display needs; to solve the above problems, the present invention proposes a dynamic backlight control method and system for a display device. Summary of the invention

[0005] In view of the shortcomings of the prior art, the main purpose of the present invention is to provide a method and system for dynamically adjusting the backlight of a display device, which can effectively solve the problems in the background technology. The specific technical solutions of the present invention are as follows:

[0006] A dynamic backlight control method for a display device, comprising:

[0007] A grayscale image is formed according to the characteristic value of each pixel in the original input image to be displayed;

[0008] Inputting the grayscale image into a preset convolutional neural network encoder model to obtain a compressed feature map;

[0009] Inputting the compressed feature map into a preset coupled diffusion algorithm decoder model to obtain an updated image, wherein the image size of the updated image is the same as the image size of the original input image;

[0010] Based on the grayscale value of each pixel of the updated image, the backlight of the display is adjusted.

[0011] Specifically, before forming a grayscale image according to the characteristic value of each pixel in the original input image to be displayed, the dynamic backlight control method further includes:

[0012] The characteristic value of each pixel is determined according to the pixel value of each sub-pixel of each pixel of the original input image to be displayed.

[0013] Specifically, the characteristic value of each pixel is the maximum sub-pixel value of each pixel, and forming a grayscale image according to the characteristic value of each pixel in the original input image to be displayed currently includes:

[0014] Taking the maximum sub-pixel value of each pixel as the grayscale value of the pixel;

[0015] The grayscale image is formed according to the grayscale value of each pixel.

[0016] Specifically, the convolutional neural network encoder model includes multiple convolutional pooling groups, the convolution kernels and / or pooling kernels of each convolutional pooling group are the same or different in size, and the convolution mode in at least one convolutional pooling group is a hole convolution, and the grayscale image is input into a preset convolutional neural network encoder model to obtain a compressed feature map, including:

[0017] A compressed feature map is obtained by inputting the grayscale image into the multiple convolution pooling groups, wherein the multiple convolution pooling groups are in a hierarchical structure, and the output of the adjacent previous convolution pooling group serves as the input of the adjacent next convolution pooling group.

[0018] Specifically, the step of inputting the compressed feature map into a preset coupled diffusion algorithm decoder model to obtain an updated image includes:

[0019] Perform edge copy filling on the compressed feature map according to the sliding window size to obtain a filled compressed feature map;

[0020] Performing mean filtering on the padded compressed feature map using a sliding window to obtain a smoothed feature map;

[0021] Performing interpolation and expansion processing on the smoothed feature map by using a bilinear interpolation method to obtain a first extended feature map;

[0022] The first extended feature map is interpolated and extended using a nearest neighbor interpolation method to obtain an updated image.

[0023] Specifically, the interpolation and expansion processing is performed on the smoothed feature map by using a bilinear interpolation method to obtain a first extended feature map, including:

[0024] Calculate the neighboring points of the interpolation point in the smoothed feature map;

[0025] The interpolation coefficient is calculated according to the relationship between the coordinates of the interpolation point in the interpolated image and the coordinates in the corresponding smoothed feature map;

[0026] Calculating the pixel value of the interpolation point according to the adjacent points and the interpolation coefficient;

[0027] The first extended feature map is formed according to the pixel values ​​of the interpolation points.

[0028] A dynamic backlight control system for a display device, used to implement a dynamic backlight control method for a display device, comprising:

[0029] Preprocessing unit: forms a grayscale image according to the feature value of each pixel in the original input image to be displayed;

[0030] Convolutional neural network encoder: input the grayscale image into a preset convolutional neural network encoder model to obtain a compressed feature map;

[0031] Coupled diffusion algorithm decoder: input the compressed feature map into a preset coupled diffusion algorithm decoder model to obtain an updated image;

[0032] A backlight control unit: based on the gray value of each pixel of the updated image, controls the backlight of the display;

[0033] Pipeline control unit: Based on a fine-grained pipeline architecture design, the calculation process of each module is controlled by control signals, enabling the modules to process multiple rows or frames of data in parallel.

[0034] Specifically, the convolutional neural network encoder includes:

[0035] Pooling unit: including a coordinate counter and a first register group, storing the maximum value data in an output row in the first register group, judging the current original data point coordinates and the output point coordinates through the coordinate counter, and continuously comparing the input data with the maximum value stored in the first register group according to the coordinate flow, updating the maximum value stored in the first register group, until the pooling calculation of a row is completed, and the maximum value result is output;

[0036] The hole convolution unit includes a first RAM memory, a first MAC calculation unit, and a second register group. The first RAM memory is divided into two parts through a ping-pong design. In the same ping-pong cycle, one part of the first RAM memory is used to store the maximum value result of the previous pooling unit, and the other part of the first RAM memory sends the maximum value result to the first MAC calculation unit for calculation to obtain a hole convolution result. The second register group is used to store the convolution kernel weight of the hole convolution unit.

[0037] Traditional convolution unit: includes a first FIFO cache unit, a second MAC calculation unit and a third register group, wherein the first FIFO cache unit is used to store the maximum value result of the previous pooling unit, and the second MAC calculation unit obtains data from the first FIFO cache unit for calculation according to the control signal of the pipeline control unit to obtain the traditional convolution result, and the third register group is used to store the convolution kernel weight of the traditional convolution unit.

[0038] Specifically, the coupled diffusion algorithm decoder includes:

[0039] Mean filtering module: including a fourth register group, a counter and a divider, the compressed feature map is shifted and stored in the fourth register group, the counter is used to indicate the position of the current calculation point in the output matrix, when performing filtering calculation, the values ​​corresponding to the sliding window are respectively input into the corresponding fourth register group, and after addition, they are sent to the divider to calculate the average value to obtain the smoothed feature map;

[0040] Bilinear interpolation module: comprising a storage module and an interpolation calculation module, wherein the storage module stores the smoothed feature map transmitted by the mean filter module and provides the data to the interpolation calculation module, wherein the interpolation calculation module is used to calculate the pixel value of the interpolation point to obtain the first extended feature map;

[0041] The nearest neighbor interpolation module includes a dual-port memory block, which instantiates the dual-port memory block, receives the first extended feature map from the bilinear interpolation module according to the control signal of the pipeline control unit, copies the first extended feature map, and stores it in the dual-port memory block according to the address until each row is copied and stored to obtain an updated image.

[0042] Specifically, the bilinear interpolation module includes:

[0043] Storage module: comprising a second FIFO cache unit, a second RAM memory and a calculation control unit, wherein the second FIFO cache unit stores the data transmitted by the mean filter module in rows, the calculation control unit obtains the adjacent points of the interpolation point in the smoothed feature map according to the register delay between the input and output points, stores the adjacent points in the second RAM memory, and divides the second RAM memory into two parts by using a ping-pong design, wherein one part of the second RAM memory receives new data, and the other part of the second RAM memory sends data to the interpolation calculation module;

[0044] Interpolation calculation module: includes a third FIFO cache unit, an address generator, a third RAM memory, a multiplier and a trigger. According to the continuous flow of interpolation pixel point coordinates, the coordinates of the interpolation point corresponding to the smoothed feature map are calculated by instantiating the multiplier and the trigger, and the calculation results are stored in the third FIFO cache unit. The address generator generates a read address according to a rule to read the corresponding data in the storage module. The third RAM memory is divided into two parts by using a ping-pong design. One part of the third RAM memory receives new data, and the other part of the third RAM memory sends data to the multiplier. The multiplier is used to calculate the pixel value of the interpolation point, and the calculation result is stored in the third FIFO cache unit.

[0045] Compared with the prior art, the present invention has the following beneficial effects:

[0046] The present invention extracts the maximum value of three sub-pixels of each pixel of the original input image through the maximum value method to form a grayscale image, thereby reducing the computational complexity; a neural network is used instead of the traditional method based on preset values ​​to extract the compression features of the image backlight, thereby improving the adaptability of the algorithm to different display scenarios; by introducing the dilated convolution to extract the compression features, the receptive field is expanded without significantly increasing the computational burden, which is more conducive to the extraction of pixel features around a pixel; a coupled two-step interpolation algorithm of bilinear interpolation and nearest neighbor interpolation is used in the decoder stage, so that the calculation in the expansion stage is not too complicated, and a relatively more reliable and smooth image effect is provided; a ping-pong design and a pipeline architecture design are used for the processing of the video stream, thereby increasing the parallelism and data throughput, improving the data processing efficiency, and reducing the delay. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 A flowchart of a method for dynamically controlling backlight of a display device according to the present invention;

[0048] Figure 2 It is an overall architecture diagram of a dynamic backlight control method for a display device of the present invention;

[0049] Figure 3 This is a diagram of the network layer structure in Embodiment 1 of the present invention;

[0050] Figure 4 4 is a hardware architecture diagram of a dilated convolution unit in Embodiment 2 of the present invention;

[0051] Figure 5 4 is a hardware architecture diagram of a bilinear interpolation unit in Embodiment 2 of the present invention;

[0052] Figure 6 It is a visualized backlight diagram of the data in Example 3 of the present invention at different stages;

[0053] Figure 7 This is the flow design diagram in Example 3 of the present invention. DETAILED DESCRIPTION

[0054] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the accompanying drawings.

[0055] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein, and those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0056] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The term "in one embodiment" that appears in different places in this specification does not necessarily refer to the same embodiment, nor does it refer to a separate or selective embodiment that is mutually exclusive with other embodiments.

[0057] Example 1

[0058] This embodiment provides a method for dynamic backlight control of a display device, and the specific scheme is as follows: Figure 1 As shown, the dynamic backlight control method of a display device includes:

[0059] S101, forming a grayscale image according to the feature value of each pixel in the original input image to be displayed;

[0060] S102, inputting the grayscale image into a preset convolutional neural network encoder model to obtain a compressed feature map;

[0061] S103, inputting the compressed feature map into a preset coupled diffusion algorithm decoder model to obtain an updated image, wherein the image size of the updated image is the same as the image size of the original input image;

[0062] S104: Adjust the backlight of the display based on the grayscale value of each pixel of the updated image.

[0063] In this embodiment, the original input image is a three-channel RGB image, in which each pixel is composed of three colors: red, green, and blue. For each pixel, the maximum value method is used to extract the maximum value of the three sub-pixels of each pixel of the original input image to form a grayscale image. Converting the RGB image to a grayscale image can greatly simplify subsequent calculations and reduce computational complexity. By selecting the maximum value, the stronger brightness part of the image can be retained, thereby improving the overall contrast of the image. The grayscale image is input into a preset convolutional neural network encoder model. The encoder extracts image features through multi-layer convolution and pooling operations, gradually reduces the spatial dimension, and compresses important features into low-dimensional feature maps. The convolutional neural network can automatically learn important features in the image without manually designing features. The compressed feature map reduces the dimension of the data, facilitates subsequent processing, and improves computational efficiency. The compressed feature map is input into the preset coupled diffusion algorithm decoder model, which expands the size of the compressed feature map to the same size as the original input image through filtering and interpolation algorithms. The coupled diffusion algorithm improves the accuracy of the interpolation process by considering the relationship between features. At the same time, the process can adjust the backlight to improve the visual effect of the image. The coupled diffusion algorithm can better capture the details of the image and reduce the distortion in the interpolation process. In this way, the backlight can be adjusted for different image scenes to enhance the visibility of the image, which is suitable for a variety of applications.

[0064] like Figure 2 The original input is a 1536*960*3 RGB image, and the maximum value of the three sub-pixels of each pixel is extracted using the maximum value method to form a 1536*960*1 grayscale image; the 1536*960*1 grayscale image is input into the convolutional neural network encoder model, and multi-layer convolution and pooling operations are performed to obtain a compressed feature map with a size of 16*10; the 16*10 compressed feature map is input into the coupled diffusion algorithm decoder model, and after filtering and interpolation, the grayscale image is restored to the original size of 1536*960, and the backlight is adjusted.

[0065] The present invention extracts the maximum value of three sub-pixels of each pixel of the original input image through the maximum value method to form a grayscale image, thereby reducing the computational complexity; a neural network is used instead of the traditional method based on preset values ​​to extract the compression features of the image backlight, thereby improving the adaptability of the algorithm to different display scenarios; by introducing the dilated convolution to extract the compression features, the receptive field is expanded without significantly increasing the computational burden, which is more conducive to the extraction of pixel features around a pixel; a coupled two-step interpolation algorithm of bilinear interpolation and nearest neighbor interpolation is used in the decoder stage, so that the calculation in the expansion stage is not too complicated, and a relatively more reliable and smooth image effect is provided; a ping-pong design and a pipeline architecture design are used for the processing of the video stream, thereby increasing the parallelism and data throughput, improving the data processing efficiency, and reducing the delay.

[0066] Furthermore, before forming a grayscale image according to the characteristic value of each pixel in the original input image to be displayed, the dynamic backlight control method further includes:

[0067] The characteristic value of each pixel is determined according to the pixel value of each sub-pixel of each pixel of the original input image to be displayed.

[0068] Specifically, the original input image is composed of pixels, and each pixel is represented by its pixel value. Sub-pixels refer to the ability to subdivide each pixel into finer rendering units. This helps improve the resolution and display quality of the image. The feature value of each pixel can be obtained by aggregating the pixel values ​​of its sub-pixels. Common methods include mean value, standard deviation, and the specific feature value is the maximum value.

[0069] Furthermore, the characteristic value of each pixel is the maximum sub-pixel value of each pixel, and the grayscale image is formed according to the characteristic value of each pixel in the original input image to be displayed, including:

[0070] S301, taking the maximum sub-pixel value of each pixel as the grayscale value of the pixel;

[0071] S302: Form the grayscale image according to the grayscale value of each pixel.

[0072] In this embodiment, based on the representation method of 8-bit color depth, for each pixel, its RGB value is 8 bits, that is, the value range of each color channel is 0-255. The RGB value of each pixel is sequentially combined into a 24-bit wide integer. By combining the RGB values ​​into an integer, memory usage can be effectively reduced, which helps to improve data processing efficiency; from the combined 24-bit wide integer, it is split into three 8-bit integers according to [23:16], [15:8] and [7:0], and each color channel can be accessed and operated independently, which can quickly process data and avoid complex operations of strings or arrays; the three 8-bit integers obtained from the previous step are compared for the maximum value, and the maximum value is taken as the corresponding pixel value in the grayscale image, and a single grayscale value is generated for each pixel. By selecting the maximum value, the brightest part of the image can be effectively highlighted. Compared with the traditional grayscale conversion method, the clarity and contrast of the image can be better preserved to ensure that the display brightness requirements of each sub-pixel are met.

[0073] Furthermore, the convolutional neural network encoder model includes multiple convolutional pooling groups, the convolution kernels and / or pooling kernels of each convolutional pooling group are the same or different in size, and the convolution mode in at least one convolutional pooling group is a dilated convolution, and the grayscale image is input into a preset convolutional neural network encoder model to obtain a compressed feature map, including:

[0074] A compressed feature map is obtained by inputting the grayscale image into the multiple convolution pooling groups, wherein the multiple convolution pooling groups are in a hierarchical structure, and the output of the adjacent previous convolution pooling group serves as the input of the adjacent next convolution pooling group.

[0075] In this embodiment, the input grayscale image is compressed by the first maximum pooling layer. The pooling kernel of size 8*8 is slid on the input image, and the maximum pixel value is taken in each area to compress the image. The step size is set to 8, and each pooling operation moves once on 8 pixels. Finally, it is compressed into a smaller feature map to obtain the first compression result. By downsampling, the spatial size of the image is significantly reduced, the amount of calculation and memory usage are reduced, and the main features are retained, which is helpful to filter the noise in the image and retain only the important feature information. The first compression result is further compressed by the first convolution layer using the convolution kernel of size 3*3. A dilated convolution is performed, combined with an edge zero padding method of size 3 to keep the image size unchanged. Eight convolution kernels are used to perform convolution operations on the input image respectively. Each convolution kernel extracts different features to obtain the first convolution result. The dilated convolution can enhance the capture of contextual information, help to obtain information more comprehensively in the high-dimensional feature space, and expand the receptive field without significantly increasing the computational burden. It is more conducive to the extraction of pixel features around a pixel point; the convolution result is input into the activation function relu layer to obtain the activation result. ReLU is simple to calculate and converges quickly, avoiding the problem of gradient disappearance.

[0076] The activation result is input into the second maximum pooling layer. By sliding the pooling kernel of size 12*12, the maximum value in each area is taken for downsampling to obtain the second compression result, which further reduces the size of the feature map, reduces the computational complexity, ensures the retention of the main feature information, and further resists noise; the second convolution layer uses the convolution kernel of size 1*1*8 to perform traditional convolution on the second compression result to obtain a compressed feature map. The use of traditional convolution can effectively integrate information from different channels, enhance the feature representation capability, avoid a significant increase in the dimension of the feature map, and help improve the computational efficiency of the model. Through multiple pooling and convolution operations, the key features in the image can be gradually extracted, allowing the model to better understand the image content. Through downsampling, the calculation and memory required for subsequent layer processing are saved. Combined with the dilated convolution and pooling layers, the model can capture more feature information at different levels.

[0077] Specifically, Figure 3, the processing process of the original input image is as follows: first, the first maximum pooling layer, the pooling kernel size is 8*8, the step size is 8, the input image of 1536*960 is compressed, and the output result of 192*120 is obtained; next, after the first convolution layer, in order to increase the receptive field of the convolution kernel and capture a wider range of features, the hollow convolution is used, the convolution kernel size is 3*3, the step size is 1, the expansion coefficient is 3, the number of convolution kernels is 8, and in order to ensure that the image size does not change, the edge 0 padding of size 3 is performed. The data size output by the first convolution layer becomes 192*120*8, and it needs to pass through the activation function relu layer, and the size remains unchanged; after the second maximum pooling layer, the pooling kernel size is 12*12, the step size is 12, and the output result of 16*10*8 is obtained; finally, the second convolution layer is used for traditional convolution, the convolution kernel size is 1*1*8, the step size is 1, and the compressed feature image of 16*10*1 is obtained.

[0078] Furthermore, the step of inputting the compressed feature map into a preset coupled diffusion algorithm decoder model to obtain an updated image includes:

[0079] S401, performing edge copy filling on the compressed feature map according to the sliding window size to obtain a filled compressed feature map;

[0080] S402, performing mean filtering on the padded compressed feature map using a sliding window to obtain a smoothed feature map;

[0081] S403, performing interpolation and expansion processing on the smoothed feature map using a bilinear interpolation method to obtain a first extended feature map;

[0082] S404: Perform interpolation and expansion processing on the first extended feature map using a nearest neighbor interpolation method to obtain an updated image.

[0083] In this embodiment, before the compressed feature map is processed, the edge of the image is copied and filled according to the sliding window size, and rows or columns identical to the boundaries are added to the top, bottom, left, and right to avoid boundary effects during filtering. The filling can ensure that no information is lost when the mean filter is performed on the edge of the image, ensuring that all pixels can participate in the filtering process, helping to improve the overall quality of the image; the mean filter is performed on the filled compressed feature map through the sliding window, the average value of the pixels in the sliding window is calculated, and the mean value is assigned to the central pixel. The mean filter can effectively remove small noise, and at the same time, the feature map is smoothed, and the high-frequency components are reduced, so that the feature extraction results are more consistent; the smoothed feature map is expanded by using the bilinear interpolation method, and the new pixel value is obtained by weighted averaging the values ​​of the four pixels around it to obtain the first extended feature map, generating a smoother image expansion effect; the first extended feature map is interpolated and expanded by using the nearest neighbor interpolation method, and the image size is expanded to the size of the original input image. The hard edges and features in the original image are retained by the nearest neighbor interpolation method. The quality of the feature map is enhanced through edge replication and smoothing filtering. The combination of bilinear interpolation and nearest neighbor interpolation can balance the smoothness and detail retention of image expansion, making the calculations in the expansion stage not too complicated and providing a relatively more reliable and smoother image effect.

[0084] Specifically, Figure 3 The mean filter layer receives the 16*10 compressed feature map from the upper layer. To ensure that the output size is still 16*10, the algorithm first performs edge copying and padding of the 16*10 input to 18*12, and then performs mean filtering with a sliding window size of 3*3 and a step size of 1; a coupled diffusion algorithm of bilinear interpolation and nearest neighbor interpolation is used, and the 16*10 smoothed image received by the upper layer is diffused to a size of 768*480 using the bilinear interpolation method, and the 768*480 grayscale image is interpolated and expanded to the original image size of 1536*960 using the nearest neighbor interpolation method. The length and width of the image are both expanded to twice the original size, and each pixel should be copied to four adjacent points.

[0085] Furthermore, the interpolation and expansion processing is performed on the smoothed feature map by using a bilinear interpolation method to obtain a first extended feature map, including:

[0086] S501, calculating the neighboring points of the interpolation point in the smoothed feature map;

[0087] S502, calculating an interpolation coefficient according to a relationship between the coordinates of the interpolation point in the interpolated image and the coordinates in the corresponding smoothed feature map;

[0088] S503, calculating the pixel value of the interpolation point according to the adjacent points and the interpolation coefficient;

[0089] S504: Form the first extended feature map according to the pixel values ​​of the interpolation points.

[0090] In this embodiment, the core idea of ​​the bilinear interpolation algorithm is to perform linear interpolation in the horizontal and vertical directions in sequence. The calculation process can be simplified as follows:

[0091] P_inter=W1P1+W2P2+W3P3+W4P4

[0092] W1=(1-dx)(1-dy)

[0093] W2=dx(1-dy)

[0094] W3=(1-dx)dy

[0095] W4=dxdy

[0096] Among them, P_inter represents the pixel value of the target interpolation point; P1, P2, P3 and P4 represent the four points around the corresponding position of the target interpolation point in the original image; the four interpolation coefficients W1, W2, W3, W4 represent the contribution weights of the four points to the interpolation point; dx and dy represent the distance ratio of the interpolation point relative to P1 and P2 in the horizontal direction and relative to P1 and P3 in the vertical direction.

[0097] According to the above formula, the calculation process of bilinear interpolation can be divided into three parts: 1. Obtaining the four adjacent points P1, P2, P3 and P4 of the interpolation point in the original image; 2. Obtaining the interpolation ratios dx and dy; 3. Calculating the four interpolation coefficients and the final interpolation point P-inter.

[0098] The interpolation coefficient and the final interpolation result are calculated by inverting the original coordinates through the scaling coordinate transformation of the target interpolation point. The conversion formula is as follows:

[0099] srcx=(dstx+0.5) * (src_width / dst_width)-0.5

[0100] srcy=(dsty+0.5) * (src_height / dst_height)-0.5

[0101] Among them, srcx and srcy represent the horizontal and vertical coordinates of the interpolation point in the original image respectively; dstx and dsty represent the horizontal and vertical coordinates of the interpolation point in the interpolated image respectively; src_width, src_height, dst_width and dst_height represent the length and width of the original image and the interpolated image respectively, which are 16, 10, 768, and 480. Split the obtained srcx and srcy into the form of integer part plus integer part, and then: srcx = i + u, srcy = j + v. Among them, i and j are the integer part, u and v are the decimal part, and the four adjacent points of the original image corresponding to the interpolation coordinates are calculated using the coordinates (i, j); the decimal parts u and v represent dx and dy in the bilinear interpolation calculation formula respectively, and then the four interpolation coefficients W1, W2, W3, and W4 can be calculated. So far, for an interpolation point, the interpolation coefficients W1, W2, W3, W4 and the four adjacent points P1, P2, P3, P4 have been obtained. According to the formula, the pixel value of the interpolation point of the final target image can be obtained to obtain the first extended feature map.

[0102] Example 2

[0103] A dynamic backlight control system for a display device, used to implement a dynamic backlight control method for a display device, comprising:

[0104] Preprocessing unit: forms a grayscale image according to the feature value of each pixel in the original input image to be displayed;

[0105] Convolutional neural network encoder: input the grayscale image into a preset convolutional neural network encoder model to obtain a compressed feature map;

[0106] Coupled diffusion algorithm decoder: input the compressed feature map into a preset coupled diffusion algorithm decoder model to obtain an updated image;

[0107] A backlight control unit: based on the gray value of each pixel of the updated image, controls the backlight of the display;

[0108] Pipeline control unit: Based on a fine-grained pipeline architecture design, the calculation process of each module is controlled by control signals, enabling the modules to process multiple rows or frames of data in parallel.

[0109] In this embodiment, the preprocessing unit receives the original three-channel RGB image, and extracts the maximum value of the three sub-pixels of each pixel of the original input image by the maximum value method to form a grayscale image; the convolutional neural network encoder performs multi-layer convolution and pooling operations on the grayscale image, continuously extracts high-level features of the image, and obtains a compressed feature map, each layer gradually reduces the spatial dimension of the image and enhances its information features; the coupled diffusion algorithm decoder filters and interpolates the compressed feature map, restores the size of the compressed feature map to the original image size, and adjusts the backlight of the image in the process.

[0110] In this embodiment, in order to increase the parallelism and data throughput, a fine-grained pipeline architecture design is used to process the video stream. The pipeline control unit controls the pipelines through SOF (Start of Frame), EOF (End of Frame), SOL (Start of Line), EOL (End of Line) and de (enable signal). By using these signals, the processing module can accurately know when to start processing a new frame or line of data, which allows the module to process multiple lines or frames of data in parallel. In addition, this pipeline control method usually only needs to calculate the output of a few lines in each cycle, which can greatly reduce the storage resource pressure of the data cache area.

[0111] Furthermore, the convolutional neural network encoder comprises:

[0112] Pooling unit: including a coordinate counter and a first register group, storing the maximum value data in an output row in the first register group, judging the current original data point coordinates and the output point coordinates through the coordinate counter, and continuously comparing the input data with the maximum value stored in the first register group according to the coordinate flow, updating the maximum value stored in the first register group, until the pooling calculation of a row is completed, and the maximum value result is output;

[0113] The hole convolution unit includes a first RAM memory, a first MAC calculation unit, and a second register group. The first RAM memory is divided into two parts through a ping-pong design. In the same ping-pong cycle, one part of the first RAM memory is used to store the maximum value result of the previous pooling unit, and the other part of the first RAM memory sends the maximum value result to the first MAC calculation unit for calculation to obtain a hole convolution result. The second register group is used to store the convolution kernel weight of the hole convolution unit.

[0114] Traditional convolution unit: includes a first FIFO cache unit, a second MAC calculation unit and a third register group, wherein the first FIFO cache unit is used to store the maximum value result of the previous pooling unit, and the second MAC calculation unit obtains data from the first FIFO cache unit for calculation according to the control signal of the pipeline control unit to obtain the traditional convolution result, and the third register group is used to store the convolution kernel weight of the traditional convolution unit.

[0115] In this embodiment, the first maximum pooling unit divides the matrix information into 8*8 blocks, uses the coordinate counter to determine the current original data point coordinates and the output point coordinates, and establishes an 8-bit wide register group with a number of 192 to store the 192 maximum value data in an output row. According to the coordinate flow, the input data is continuously judged against the stored maximum value to update the maximum value until the pooling calculation of one row is completed, and then a write request is sent to the RAM in the void convolution unit to select the RAM for writing according to the address and row position.

[0116] like Figure 4 The dilated convolution unit contains 6 RAMs for accessing the output data of the pooling unit. To increase the parallelism, a ping-pong design is adopted, in which 3 RAMs are ping areas and 3 RAMs are pong areas. In the same ping-pong cycle, 3 RAMs are used to receive the result data of the pooling layer, and the other 3 RAMs output the data to MAC for convolution calculation. Data is stored in rows, where the storage order of RAM0 is 1, 2, 3, 10, 11, 12, 19, 20, 21, etc., the storage order of RAM1 is 4, 5, 6, 13, 14, 15, 22, 23, 24, etc., and the storage order of RAM2 is 7, 8, 9, 16, 17, 18, 25, 26, 27, etc. The purpose of such storage is to cooperate with the dilated convolution algorithm. After the 3*3 convolution kernel is expanded by a dilation factor of 3, a multiplication and addition operation is performed every 3 rows in the same convolution sliding window (taking the first row of sliding window as an example, the 1st, 4th, and 7th rows of data need to be read). In this way, the three rows in the same sliding window operation can be taken out from three RAMs in parallel. The convolution calculation module contains three MACs, which can simultaneously calculate the multiplication and addition operations of three rows in a sliding window. The weight of the convolution kernel is stored by the register group through external input and fed into the MAC operation according to the sliding window and calculation point. During the sliding window operation, the data fed into the MAC is determined to be 0 by judging whether the sliding window position exceeds the original image position, so as to achieve the effect of edge filling. The three MACs are the multiplication and addition results of the three rows of the convolution kernel. The three results need to be added to obtain the value of a point in the final output image after convolution. Whenever a single sliding window calculation end signal is received, a single point result is output to the register group in the next pooling unit. Here, by judging whether the output result of a single point is a positive number, it is considered that the output result is 0 or itself, thus realizing the function of the Relu layer.

[0117] In this embodiment, the second maximum pooling unit receives a 192*120*8 image, compresses it into a 16*10*8 image, divides the single-channel image into 12*12 blocks, and takes the maximum value within the block. According to the output, a 16*8 8-bit wide register group is established to store the 16 maximum value data in an output row in the order of the channels. According to the coordinate flow, the input data is continuously judged against the stored maximum value, thereby updating the maximum value until the pooling calculation of one row of 8 channels is completed, and a row of data is sent to the FIFO of the next convolution layer.

[0118] Specifically, the calculation of the traditional convolution unit only needs to consider the multiplication and addition of a single channel because the convolution kernel size is 1. A row of 8-channel data output by the second maximum pooling unit is stored in the FIFO cache. Whenever the video stream control signal EOL given by the front layer is received, the data is taken from the FIFO and convolution operation is performed with the preset convolution kernel weight. After calculating the result of each point, it is sent to the mean filter module. When the output of a row of results is completed, the EOL signal is sent backward.

[0119] Furthermore, the coupled diffusion algorithm decoder comprises:

[0120] Mean filtering module: including a fourth register group, a counter and a divider, the compressed feature map is shifted and stored in the fourth register group, the counter is used to indicate the position of the current calculation point in the output matrix, when performing filtering calculation, the values ​​corresponding to the sliding window are respectively input into the corresponding fourth register group, and after addition, they are sent to the divider to calculate the average value to obtain the smoothed feature map;

[0121] Bilinear interpolation module: comprising a storage module and an interpolation calculation module, wherein the storage module stores the smoothed feature map transmitted by the mean filter module and provides the data to the interpolation calculation module, wherein the interpolation calculation module is used to calculate the pixel value of the interpolation point to obtain the first extended feature map;

[0122] The nearest neighbor interpolation module includes a dual-port memory block, which instantiates the dual-port memory block, receives the first extended feature map from the bilinear interpolation module according to the control signal of the pipeline control unit, copies the first extended feature map, and stores it in the dual-port memory block according to the address until each row is copied and stored to obtain an updated image.

[0123] In this embodiment, the mean filter module shifts and stores the input data in an 8-bit wide register group with a size of 160. The data of one point can be stored in each clock cycle to facilitate the use of data of a specific cycle for calculation. Two counters are used inside the module to indicate the position h and w of the current calculation point in the output matrix. When performing filtering calculations, the 9 values ​​corresponding to the sliding window are respectively input into the corresponding registers, and then added and sent to the divider to obtain the average value to obtain the smoothed feature map. In the sliding window process, only the boundaries need to be specially considered. For example, the positions of the four vertex angles need to copy the corresponding vertex angle values ​​three times and input them into the corresponding registers. For example, the edge position instead of the vertex angle position only needs to be copied once, so that the edge copy and fill operation is completed without expanding the storage resources. For the control signal, it should be noted that after each row of calculation is completed, the de enable signal will be pulled high for 16 cycles, and the data of one point will be output in each cycle. During this period, sol will be pulled high for one beat while de is pulled high, eol will be pulled high for one beat while de is pulled low, sof will be pulled high for one beat at the beginning of the current frame, and eof will reach cnt_h='d15,cnt_w='d9 at the counter at the indicated position, and the sliding window will reach the last point, and the last row will be output, and eof and eol will be pulled high for one beat at the same time. The bilinear interpolation module includes a storage module and an interpolation calculation module, the storage module stores the smoothed feature map, and provides data to the interpolation calculation module, the interpolation calculation module is used to calculate the interpolation coefficients and the pixel values ​​of the interpolation points of the final target image to obtain the first extended feature map.

[0124] Specifically, the nearest neighbor interpolation module instantiates a dual-port memory block, receives 8-bit wide data from the FIFO of the bilinear interpolation module, copies and binds it into a 16-bit wide data in the form of {data_in, data_in}, and stores it in the memory block according to the address control. This is reflected in the image as a copy in the horizontal row direction. Next, through the control of the end-of-line signal eol, each time a row of data is written, the previous row of input should be repeated and written into the memory block, so as to achieve a copy of the image in the vertical column direction. Each address of the memory block read port takes 8-bit data, and splits the two pixel point data bound when storing. The whole process is pipelined through the video stream control signal to achieve nearest neighbor interpolation and obtain an updated image.

[0125] Furthermore, the bilinear interpolation module includes:

[0126] Storage module: comprising a second FIFO cache unit, a second RAM memory and a calculation control unit, wherein the second FIFO cache unit stores the data transmitted by the mean filter module in rows, the calculation control unit obtains the adjacent points of the interpolation point in the smoothed feature map according to the register delay between the input and output points, stores the adjacent points in the second RAM memory, and divides the second RAM memory into two parts by using a ping-pong design, wherein one part of the second RAM memory receives new data, and the other part of the second RAM memory sends data to the interpolation calculation module;

[0127] Interpolation calculation module: includes a third FIFO cache unit, an address generator, a third RAM memory, a multiplier and a trigger. According to the continuous flow of interpolation pixel point coordinates, the coordinates of the interpolation point corresponding to the smoothed feature map are calculated by instantiating the multiplier and the trigger, and the calculation results are stored in the third FIFO cache unit. The address generator generates a read address according to a rule to read the corresponding data in the storage module. The third RAM memory is divided into two parts by using a ping-pong design. One part of the third RAM memory receives new data, and the other part of the third RAM memory sends data to the multiplier. The multiplier is used to calculate the pixel value of the interpolation point, and the calculation result is stored in the third FIFO cache unit.

[0128] In this embodiment, the storage module includes a FIFO cache unit, a RAM memory and a calculation control unit. When the mean filter module outputs 16*10 image data, the data first enters the FIFO cache unit for cache. These data will be input by row. The FIFO cache unit only stores 16 8-bit wide data of one row at the same time, and there is a one-beat register delay for the input and output data and enable. The purpose of this is to make the data at the input and output ends of the FIFO at the same time just two points that span 1 row and have the same horizontal coordinates. Combined with the last input and output point retained by the register delay, a 2*2 adjacent point matrix window can be formed in the image, that is, P1, P2, P3 and P4 required for interpolation calculation. The data of these four points are combined into a data packet with a bit width of 32 bits in the order of {P1, P2, P3, P4}, and stored in RAM for subsequent interpolation calculation module to use. In particular, because we are expanding a 16*10 image to 768*480, there will be an uneven number of interpolation points at the boundary. Algorithmically, we will copy the points to fill the edge. In hardware design, there will be no new data to read at the input of the FIFO cache unit. We will copy the data of the previous cycle and pack it into RAM in the order of {P1, P1, P3, P3}.

[0129] Specifically, the RAM access adopts a ping-pong design to ensure the continuity of the video stream and high data throughput. The RAM is divided into the first 16*10=160 addresses as the ping area and the last 160 addresses as the pong area. Each time the frame end signal EOF is received, the base address for writing data is switched between 0 and 160. In this way, at the same time, half of the RAM area receives the new data input in a package, and the other half area outputs 2*2 point packaged data to the subsequent interpolation calculation module.

[0130] The interpolation calculation module is used to obtain the interpolation coefficients and calculate the final interpolation results. Through the current interpolation pixel point coordinates (dstx, dsty) that are constantly circulating, it is only necessary to instantiate the multiplier and trigger, and calculate the corresponding srcx and srcy. Here, a fixed-point number with 11 integer bits and 13 decimal bits, a total of 24 bits, is used to store the calculated coordinate data. The generated coordinate data is packed into 48-bit wide data in the order of {srcx, srcy} and input into the FIFO cache unit for caching so that it can be used in the subsequent interpolation calculation process. Through the transformation of the current interpolation coordinates, the address generator continuously generates the address corresponding to the current interpolation point in the storage module RAM0 according to the rules, and then sends a read request to the storage module RAM1 to read the corresponding 32-bit adjacent point data packet and store it in the interpolation calculation module RAM0. The RAM1 in the interpolation calculation module also adopts ping-pong operation, divides the ping area and pong area according to the address, and fetches data from RAM0 at the same time, and fetches the data stored in the previous ping-pong cycle for subsequent interpolation calculation. By taking the decimal places from the data packet obtained from the FIFO cache unit, dx and dy can be obtained, and the interpolation coefficients W1, W2, W3, and W4 can be calculated; the adjacent point data packet taken from RAM1 is bit-splitting to obtain P1, P2, P3, and P4, and the pixel value of the interpolation point of the final target image is calculated by the multiplier and stored in the FIFO cache unit, waiting for the control signal to enter the next module calculation.

[0131] like Figure 5 It is the architecture of the bilinear interpolation unit, which is completed by the cooperation of the storage module and the interpolation calculation module. The front layer data is input into the FIFO in the storage module for cache. By intercepting the data of the FIFO input and output ends and the cache of the previous row, the adjacent point matrix is ​​obtained, and the four point data are packaged and sent to the RAM for storage according to the coordinates; the control module of the interpolation calculation module first calculates the original image coordinates {srcx, srcy} corresponding to the interpolation point according to the coordinate flow according to the formula, calculates the address according to the coordinates, and fetches the data from the RAM in the storage module to obtain the adjacent points P1, P2, P3, P4. At the same time, the srcx and srcy cached in the FIFO are taken out to take the decimal part to calculate the interpolation coefficient. Finally, the multiplication and addition are completed according to the formula to obtain the interpolation point value, and finally sent to the FIFO cache to wait for output.

[0132] Example 3

[0133] This embodiment provides a process of processing video stream data using a dynamic backlight control method and system for a display device.

[0134] The data first enters the preprocessing module, which splits the RGB three-channel data into [23:16], [15:8], and [7:0], and takes the maximum value of the three sub-pixels to obtain a 1536*960*1 grayscale image to ensure that the display brightness requirements of each sub-pixel are met.

[0135] The encoder accepts a grayscale image of 1536*960*1, and first compresses the grayscale image using maximum pooling with a pooling kernel size of 8*8 and a step size of 8. According to the flow of the coordinate counter, each data is compared with the maximum value in the 8*8 pooling block where it is located and the maximum value data is updated. The data results are saved using a set of 192 8-bit registers. Whenever a row of the output image is calculated, it is written to the 3 RAMs of the post-layer void convolution unit according to the corresponding coordinates, and finally the image is compressed to a size of 192*120*1.

[0136] The dilated convolution unit receives 192*120*1 data, performs dilated convolution with a kernel size of 3*3, a step size of 1, a filling factor of 3, and a dilation factor of 3, with a total of 8 convolution kernels, and the resulting output size is 192*120*8 data. It contains 6 RAMs, 3 of which are ping-pong areas and 3 are pong areas. In the same ping-pong cycle, 3 are used to receive pooling layer data, and the other 3 send data of the points for convolution operations to MAC. According to the movement of the sliding window, the coordinate control logic gives the RAM read address, and at the same time, in conjunction with the convolution kernel weight data stored in the register, the calculation result of MAC is fed into the next maximum pooling layer calculation.

[0137] The second maximum pooling unit receives 192*120*8 data, performs maximum pooling with a pooling kernel size of 12*12 and a step size of 12, and the output size is 16*10*8 data. According to the coordinate flow, the maximum value is taken and updated in each 12*12 pooling block, and the pooling result is stored in a group of 16*8 8-bit wide registers. After completing one line of calculation, one line of data is sent to the FIFO of the post-convolution layer.

[0138] The second convolution unit in the encoder receives 16*10*8 data, performs convolution operation with a convolution kernel size of 1*1, a step size of 1, and a number of convolution kernels of 1, and outputs a compressed feature image with a size of 16*10*1. The convolution kernel weights are stored in registers from external inputs. Since the convolution kernel size is 1*1, only the 8 channels at a single point need to be multiplied and added, and the operation results are directly input to the shift register group of the post-mean filter layer for storage. At this point, the encoding and compression process of the backlight image is completed.

[0139] The decoder stage starts with the mean filter unit, which receives a 16*10*1 compressed feature image. It first performs edge replication and padding with a padding coefficient of 1, and the image becomes 18*12*1 in size. Then, it performs a mean filter with a filter size of 3*3 and a step size of 1. The final result is still 16*10*1 in size. The mean filter unit uses a set of registers of size 160 to shift and register the output results of the front layer. During the filter sliding window process, if an edge part is encountered, the edge point corresponding to the original image is copied to realize the edge replication and filling function. For the rest, the nine points are summed according to the 3*3 sliding window and then divided by nine by the divider to obtain the average value. The result is sent to the FIFO of the post-bilinear interpolation unit for storage.

[0140] The coupled interpolation algorithm is first bilinear interpolation. The bilinear interpolation unit receives 16*10*1 data and expands it to 768*480*1 size through bilinear interpolation. The function of this unit is completed by the collaboration of the image_2*2 and interpolation calculation modules. The calculation results of the front layer are output row by row to the FIFO in the bilinear interpolation unit image_2*2 module. The data at the input and output ends of the FIFO can form a 2*2 adjacent point matrix window, which contains the four adjacent point values ​​P1, P2, P3 and P4 required to calculate the interpolation point value. The four values ​​are bound into a data packet and stored in the RAM in the module according to the coordinate point for access. Here, the RAM is ping-pong operated and divided into two areas according to the address. In the same ping-pong cycle, half receives the packaged data and the other half sends data to the interpolation calculation module. The interpolation calculation module calculates the reverse coordinates srcx and srcy according to the coordinate flow, uses the integer part to calculate the address, takes the corresponding data packet from the RAM of image_2*2, and uses the decimal part to calculate the difference coefficient, that is, to obtain all the required parameters, and performs multiplication and addition operations according to the formula to obtain the value of the corresponding interpolation point, and the result is stored in the FIFO.

[0141] The nearest neighbor interpolation unit receives 768*380*1 data, expands it to 1536*960*1 through interpolation, and finally restores the original video backlight size. This unit receives the data given by the front bilinear interpolation unit FIFO, stores it into the memory according to the address in the form of {data_in, data_in}, and repeats it once for each line, thus completing the function of expanding one point into four connected points. Finally, the image is cached and output at a given frequency, such as Figure 6 , which is the backlight display simulation after neural network extraction, mean filtering, bilinear interpolation, and nearest neighbor interpolation.

[0142] In this implementation, SOF (Start of Frame), EOF (End of Frame), SOL (Start of Line), EOL (End of Line) and de (enable signal) are used to control the pipeline processing of the video stream. SOF and EOF signals are pulled high when each layer receives a new frame of data and ends a frame of calculation.

[0143] like Figure 7 For the first pooling layer, each time the calculation result in the first 8*8 pooling window in an output row is completed, SOL is pulled high. When all pooling windows in this output row are calculated, EOL is pulled high.

[0144] For the atrous convolution layer, for the operation of each convolution kernel, when the calculation of the first output point of a row of sliding windows is completed, SOL is pulled high, and when all the points in this row are calculated, EOL is pulled high.

[0145] For the second pooling layer, each time the calculation result in the first 12*12 pooling window in an output row is completed, SOL is pulled high. When the calculation of all pooling windows in this output row is completed, EOL is pulled high.

[0146] For the second convolution layer, there is only one convolution kernel. When the calculation results of the first sliding window 8 channels of each row are completed, SOL is pulled high. When all the points in this row are calculated, EOL is pulled high.

[0147] For the mean filter layer, when the calculation of the first filter window of each row is completed, SOL is pulled high, and when the calculation of all filter windows of this row is completed, EOL is pulled high.

[0148] For the bilinear interpolation layer, when the interpolation calculation of the first point in each output row is completed, SOL is pulled high, and when the interpolation of all points in an output row is completed, EOL is pulled high.

[0149] For the nearest neighbor interpolation layer, since it is the last layer, there is no need to send SOL and EOL signals.

[0150] Example 4

[0151] This embodiment uses a dynamic backlight control method for a display device to map a neural network application in a functionally reconfigurable on-chip network architecture. The experiment uses the MINIST data set and a fully connected network model. The accuracy of training 900 images on software is 98.22%, and the accuracy of training 900 images on on-chip network hardware is 98.00%. For weight and configuration initialization, the communication delay using this embodiment is 5371 cycles. After that, according to the network mapping and the communication and calculation design between layers, one picture can be trained every 2100 cycles.

[0152] The above shows and describes the basic principles and main features of the present invention and the advantages of the present invention. It should be understood by those skilled in the art that the present invention is not limited to the above embodiments. The above embodiments and descriptions are only for explaining the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, which fall within the scope of the present invention to be protected. The scope of protection of the present invention is defined by the attached claims and their equivalents.

Claims

1. A method for dynamic backlight control of a display device, characterized in that: include: A grayscale image is formed according to the characteristic value of each pixel in the original input image to be displayed; Inputting the grayscale image into a preset convolutional neural network encoder model to obtain a compressed feature map; Inputting the compressed feature map into a preset coupled diffusion algorithm decoder model to obtain an updated image, wherein the image size of the updated image is the same as the image size of the original input image; Based on the grayscale value of each pixel of the updated image, the backlight of the display is adjusted.

2. The method for dynamic backlight control of a display device according to claim 1, characterized in that: Before forming a grayscale image according to the characteristic value of each pixel in the original input image to be displayed, the dynamic backlight control method further includes: The characteristic value of each pixel is determined according to the pixel value of each sub-pixel of each pixel of the original input image to be displayed.

3. The method for dynamic backlight control of a display device according to claim 2, characterized in that: The characteristic value of each pixel is the maximum sub-pixel value of each pixel, and the grayscale image is formed according to the characteristic value of each pixel in the original input image to be displayed, including: Taking the maximum sub-pixel value of each pixel as the grayscale value of the pixel; The grayscale image is formed according to the grayscale value of each pixel.

4. The method for dynamic backlight control of a display device according to claim 1, characterized in that: The convolutional neural network encoder model includes multiple convolutional pooling groups, the convolution kernels and / or pooling kernels of each convolutional pooling group are the same or different in size, and the convolution mode in at least one convolutional pooling group is a dilated convolution, and the grayscale image is input into a preset convolutional neural network encoder model to obtain a compressed feature map, including: A compressed feature map is obtained by inputting the grayscale image into the multiple convolution pooling groups, wherein the multiple convolution pooling groups are in a hierarchical structure, and the output of the adjacent previous convolution pooling group serves as the input of the adjacent next convolution pooling group.

5. The method for dynamic backlight control of a display device according to claim 1, characterized in that: The step of inputting the compressed feature map into a preset coupled diffusion algorithm decoder model to obtain an updated image comprises: Perform edge copy filling on the compressed feature map according to the sliding window size to obtain a filled compressed feature map; Performing mean filtering on the padded compressed feature map using a sliding window to obtain a smoothed feature map; Performing interpolation and expansion processing on the smoothed feature map by using a bilinear interpolation method to obtain a first extended feature map; The first extended feature map is interpolated and extended using a nearest neighbor interpolation method to obtain an updated image.

6. The method for dynamic backlight control of a display device according to claim 5, characterized in that: The method of performing interpolation and expansion processing on the smoothed feature map by using a bilinear interpolation method to obtain a first extended feature map includes: Calculate the neighboring points of the interpolation point in the smoothed feature map; The interpolation coefficient is calculated according to the relationship between the coordinates of the interpolation point in the interpolated image and the coordinates in the corresponding smoothed feature map; Calculating the pixel value of the interpolation point according to the adjacent points and the interpolation coefficient; The first extended feature map is formed according to the pixel values ​​of the interpolation points.

7. A dynamic backlight control system for a display device, used to implement a dynamic backlight control method for a display device as claimed in claims 1 to 6, characterized in that: include: Preprocessing unit: forms a grayscale image according to the feature value of each pixel in the original input image to be displayed; Convolutional neural network encoder: input the grayscale image into a preset convolutional neural network encoder model to obtain a compressed feature map; Coupled diffusion algorithm decoder: input the compressed feature map into a preset coupled diffusion algorithm decoder model to obtain an updated image; A backlight control unit: based on the gray value of each pixel of the updated image, controls the backlight of the display; Pipeline control unit: Based on a fine-grained pipeline architecture design, the calculation process of each module is controlled by control signals, enabling the modules to process multiple rows or frames of data in parallel.

8. The dynamic backlight control system of a display device according to claim 7, characterized in that: The convolutional neural network encoder comprises: Pooling unit: including a coordinate counter and a first register group, storing the maximum value data in an output row in the first register group, judging the current original data point coordinates and the output point coordinates through the coordinate counter, and continuously comparing the input data with the maximum value stored in the first register group according to the coordinate flow, updating the maximum value stored in the first register group, until the pooling calculation of a row is completed, and the maximum value result is output; The hole convolution unit includes a first RAM memory, a first MAC calculation unit, and a second register group. The first RAM memory is divided into two parts through a ping-pong design. In the same ping-pong cycle, one part of the first RAM memory is used to store the maximum value result of the previous pooling unit, and the other part of the first RAM memory sends the maximum value result to the first MAC calculation unit for calculation to obtain a hole convolution result. The second register group is used to store the convolution kernel weight of the hole convolution unit. Traditional convolution unit: includes a first FIFO cache unit, a second MAC calculation unit and a third register group, wherein the first FIFO cache unit is used to store the maximum value result of the previous pooling unit, and the second MAC calculation unit obtains data from the first FIFO cache unit for calculation according to the control signal of the pipeline control unit to obtain the traditional convolution result, and the third register group is used to store the convolution kernel weight of the traditional convolution unit.

9. The dynamic backlight control system of a display device according to claim 7, characterized in that: The coupled diffusion algorithm decoder comprises: Mean filtering module: including a fourth register group, a counter and a divider, the compressed feature map is shifted and stored in the fourth register group, the counter is used to indicate the position of the current calculation point in the output matrix, when performing filtering calculation, the values ​​corresponding to the sliding window are respectively input into the corresponding fourth register group, and after addition, they are sent to the divider to calculate the average value to obtain the smoothed feature map; Bilinear interpolation module: comprising a storage module and an interpolation calculation module, wherein the storage module stores the smoothed feature map transmitted by the mean filter module and provides the data to the interpolation calculation module, wherein the interpolation calculation module is used to calculate the pixel value of the interpolation point to obtain the first extended feature map; The nearest neighbor interpolation module includes a dual-port memory block, which instantiates the dual-port memory block, receives the first extended feature map from the bilinear interpolation module according to the control signal of the pipeline control unit, copies the first extended feature map, and stores it in the dual-port memory block according to the address until each row is copied and stored to obtain an updated image.

10. The dynamic backlight control system of a display device according to claim 8, characterized in that: The bilinear interpolation module comprises: Storage module: comprising a second FIFO cache unit, a second RAM memory and a calculation control unit, wherein the second FIFO cache unit stores the data transmitted by the mean filter module in rows, the calculation control unit obtains the adjacent points of the interpolation point in the smoothed feature map according to the register delay between the input and output points, stores the adjacent points in the second RAM memory, and divides the second RAM memory into two parts by using a ping-pong design, wherein one part of the second RAM memory receives new data, and the other part of the second RAM memory sends data to the interpolation calculation module; Interpolation calculation module: includes a third FIFO cache unit, an address generator, a third RAM memory, a multiplier and a trigger. According to the continuous flow of interpolation pixel point coordinates, the coordinates of the interpolation point corresponding to the smoothed feature map are calculated by instantiating the multiplier and the trigger, and the calculation results are stored in the third FIFO cache unit. The address generator generates a read address according to a rule to read the corresponding data in the storage module. The third RAM memory is divided into two parts by using a ping-pong design. One part of the third RAM memory receives new data, and the other part of the third RAM memory sends data to the multiplier. The multiplier is used to calculate the pixel value of the interpolation point, and the calculation result is stored in the third FIFO cache unit.

Images