Liquid crystal display area dynamic dimming method based on image saliency ranking model

Abstract

The application discloses a liquid crystal display area dynamic dimming method based on an image saliency ranking model, which comprises the following steps: step 1, obtaining initial backlight values of each backlight partition of an input image; using an instance-level saliency ranking detection model based on graph reasoning to predict saliency scores of different salient objects in the input image and perform saliency ranking to obtain a saliency ranking graph; step 2, dividing the saliency ranking graph obtained in step 1 into a plurality of backlight partitions according to the number of backlight source partitions, giving different weights to the backlight partitions where each salient object is located according to the saliency ranking, and then performing weighted calculation and correction on the initial backlight values according to the weights to obtain final partition backlight values; step 3, using a BMA light mixing and diffusion algorithm to simulate the distribution of the partition backlight values on a liquid crystal panel, and determining pixel compensation values, and displaying the input image according to the backlight values and the pixel compensation values. The application can improve the display effect perceived by human eyes and reduce energy consumption.

Description

Technical Field

[0001] This invention relates to liquid crystal display control methods, specifically a dynamic dimming method for liquid crystal display areas based on an image saliency ranking model. Background Technology

[0002] LCD displays have become the mainstream flat panel display technology. In recent years, with the development of LCD display technology, backlight modules have also been continuously improving. Ordinary LED backlights are moving towards small-pitch LEDs and gradually developing towards Mini-LED. Compared with traditional direct-lit LED backlights, Mini-LED backlights can achieve more precise local light control, making the screen brightness more uniform. At the same time, more local dimming zones can be achieved, enabling the display to achieve better contrast, deeper blacks, and lower power consumption.

[0003] To achieve lower power consumption and higher display quality, dynamic dimming algorithms are crucial. These algorithms dynamically adjust backlight brightness and liquid crystal pixel opening based on the displayed image, effectively improving contrast and reducing power consumption. While current dynamic dimming algorithms improve image quality to some extent, they still have limitations. For example, the maximum value method primarily focuses on the brightest parts of the image and consumes more power; the average value method offers better energy savings, but for pixel values ​​above the average value, display quality degrades at the given backlight brightness.

[0004] Image display quality is not only related to objective evaluation metrics but also closely linked to human visual perception. For an image, high objective metrics such as PSNR, SSIM, and information entropy do not necessarily indicate better image display quality; subjective human evaluation is also crucial. Currently, many metrics incorporate subjective evaluation, and some researchers are beginning to study backlighting algorithms based on human visual characteristics. While these algorithms may not have the same objective image quality metrics as others, they offer excellent display results and a more comfortable subjective experience. Image feature-based dimming algorithms can classify images based on their detailed features and apply optimal algorithms to different zones, improving image display quality and saving energy. However, this method is only related to human visual perception and is unrelated to the importance of image content or the brain's attention mechanisms. Therefore, a dynamic dimming algorithm is needed that can select the optimal dimming method based on the importance of image content. This algorithm should divide the image based on content analysis and weight distribution, improving the fidelity of key areas and maximizing visual perception while minimizing energy consumption, thus ensuring a better subjective visual experience.

[0005] The 2022 IEEE paper by Liu et al., "Instance-Level Relative SaliencyRanking With Graph Reasoning," proposed an instance-level saliency ranking detection model based on graph reasoning. First, an improved Mask R-CNN network is used to segment salient instances. Then, a saliency ranking branch is added to infer relative saliency. Relative saliency ranking constructs a novel graph reasoning module, and a new loss function is proposed to efficiently train the saliency ranking branch. Finally, the authors constructed a new saliency ranking benchmark dataset using the MS-COCO and SALICON datasets to train the saliency ranking model and validated the model using a validation set.

[0006] Based on this, the present invention considers applying it to the image preprocessing part of the backlighting algorithm. First, the size of the input image is adjusted, then the salience scores of different salient objects in the input image are predicted and ranked, and finally the size of the output image is restored to obtain the final salience ranking map. This model can simulate the attention mechanism of the human brain, that is, the gaze time of the human eye on different salient objects in an image, to distinguish the importance of each part, and to optimize the dimming method of the present invention. Summary of the Invention

[0007] This invention provides a dynamic dimming method for liquid crystal display areas based on an image saliency ranking model, in order to solve the problem that existing dynamic dimming methods for liquid crystal displays only consider the human eye's perception effect and cannot perform dynamic dimming of areas based on the human brain's attention level.

[0008] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0009] A dynamic dimming method for liquid crystal display areas based on an image saliency ranking model includes the following steps:

[0010] Step 1: Convert the input image into a grayscale image, partition the grayscale image according to the number of partitions of the backlight, extract the partition backlight value of each backlight partition using different methods, and calculate the initial backlight value of each backlight partition based on the partition backlight value.

[0011] Furthermore, the input image is preprocessed using an instance-level saliency ranking detection model based on graph reasoning. First, the size of the input image is adjusted, then the saliency scores of different salient objects in the input image are predicted for saliency ranking, and finally the size of the output image is restored to obtain the final saliency ranking map.

[0012] Step 2: Divide the saliency ranking map obtained in Step 1 into blocks corresponding to the backlight partitions determined in Step 1. Assign different weights to the blocks containing each salient object according to the saliency ranking of each salient object obtained in Step 1. Then, perform weighted calculation and correction on the initial backlight value of each block according to the weights, thereby determining the final block backlight value of each block.

[0013] Step 3: Use the BMA light mixing diffusion algorithm to simulate the distribution of the block backlight value of each block obtained in Step 2 on the LCD panel, and determine the pixel compensation value. Display the input image based on the block backlight value and pixel compensation value of each block.

[0014] In further step 1, the sub-pixel maximum value method is used. First, the value of the channel with the largest brightness information among the R, G, and B channels of each pixel in the input image is taken as the brightness value of that pixel. Then, the maximum brightness value of the pixels in the partition is taken as the backlight value BL1 of the first partition. The average value method is used to accumulate the grayscale values ​​of each pixel in each partition, and then the average value is taken as the backlight value BL2 of the second partition. Based on the backlight values ​​BL1 and BL2 of the first partition, the initial backlight value BL3 of each backlight partition is calculated. The formula for calculating the initial backlight BL3 is as follows:

[0015] BL3 = BL1×1 + BL2×0.

[0016] Furthermore, in step 1, the instance-level saliency ranking detection model based on graph reasoning uses an improved MaskR-CNN network for salient instance segmentation and adds a saliency ranking branch to rank the relative saliency of each segmented instance. Different salient objects in the input image are ranked according to their importance in the image, i.e., the length of time the human eye gazes at different contents in the image. The input image is preprocessed before backlight extraction is performed.

[0017] In the further step 2, the partitions containing significant objects with different significance rankings in the significance ranking graph are weighted. The highest significance level has a weight of 1. As the significance ranking decreases, the weight value is calculated using the following formula:

[0018] α = 1 - β*(n-1),

[0019] Where n is the rank number and β is a test value.

[0020] Furthermore, if several salient objects with different salient rankings coexist within a partition, the weight value of the backlight partition is calculated by summing the weight values ​​assigned to each pixel of the different salient objects, and then averaging the summations. The calculation formula is as follows:

[0021]

[0022] in:

[0023] M×N represents the resolution of the input image;

[0024] R is the number of columns of the corresponding backlight partitions in the input image, which are divided according to the number of partitions of the backlight, and S is the number of rows of the corresponding backlight partitions.

[0025] α(R, S) is the weight value of the backlight partition in row S and column R, α(i, j) is the weight value of the pixel in row j and column i of the image, and M / R and N / S are the number of pixels in the backlight partition.

[0026] In the further step 2, the final backlight value of each block is calculated according to the following formula:

[0027] BL4 = BL1*α + BL2*(1-α),

[0028] in,

[0029] BL1 is the backlight value for the first zone;

[0030] BL2 is the backlight value for the second zone;

[0031] α represents the weight value assigned to different partitions;

[0032] BL4 represents the final backlight value of each zone, and all the obtained backlight values ​​of each zone form an R×S backlight value matrix, where R represents the number of columns of the backlight zone and S represents the number of rows of the backlight zone.

[0033] In the further step 3, the dimming factor of the corresponding block backlight value module is determined according to the final backlight value of the partition. The pulse width modulation (PWM) signal of the backlight LED in each block backlight value module is set according to the dimming factor and sent to the LED driver module of the corresponding backlight module. Furthermore, the backlight matrix composed of all partition backlight values ​​obtained by the above backlight algorithm is blurred and diffused using the BMA light mixing diffusion algorithm to obtain a blurred diffusion matrix. The backlight value corresponding to each pixel on the LCD panel is obtained from the blurred diffusion matrix. Then, the LCD compensation algorithm is used to obtain the LCD compensation signal of each pixel and sent to the LCD control module to display the input image.

[0034] Furthermore, the liquid crystal compensation algorithm employs either a linear or nonlinear compensation algorithm.

[0035] Compared with existing dynamic dimming methods for liquid crystal displays, this invention is a dynamic dimming algorithm that selects the optimal dimming method based on the importance of the image content. It divides the image into weighted distributions based on the analysis of the image content, improves the fidelity of key areas, and enhances the display effect perceived by the human eye while reducing energy consumption, all while ensuring the subjective visual effect. Attached Figure Description

[0036] Figure 1 This is a flowchart of the dynamic dimming method according to an embodiment of the present invention.

[0037] Figure 2 This is the significance ranking detection model suggested in the embodiments of the present invention. Detailed Implementation

[0038] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0039] like Figure 1 As shown, this embodiment discloses a dynamic dimming method for liquid crystal displays based on image content analysis, including the following steps:

[0040] Step 1: Convert the input image to a grayscale image. Divide the grayscale image into partitions according to the number of backlight partitions. Extract the partition backlight value of each backlight partition using different methods, and calculate the initial backlight value of each backlight partition based on the partition backlight value. Furthermore, an instance-level saliency ranking detection model based on graph reasoning processes the input image, ranking the attention levels of different salient objects in the input image to obtain a saliency ranking map.

[0041] In this embodiment, for an input image A with a resolution of 1920×1080, the image is converted into a grayscale image I. Based on the 64×32 backlight partition of the direct-lit LED backlight module, the number of pixels within the corresponding partition of grayscale image I is either 30×33 or 30×34. Here, R is the number of columns in the corresponding backlight partition of the input image, and S is the number of rows in the corresponding backlight partition.

[0042] The sub-pixel maximum value method is used, where the value of the channel with the highest brightness information among the R, G, and B channels of each pixel in the input image is taken as the brightness value of that pixel. The maximum brightness value of pixels within a given partition is then used as the backlight value BL1 for the first partition. Simultaneously, the average value method is used, where the grayscale values ​​of all pixels within each partition are accumulated, and the average value is taken as the backlight value BL2 for the second partition. Based on the backlight values ​​BL1 and BL2, the initial backlight value BL3 for each backlight partition is calculated using the following formula:

[0043] BL1(i,j)=Max(Max(R(i,j),G(i,j),B(i,j)),

[0044]

[0045] BL3 = BL1 × 1 + BL2 × 0;

[0046] in:

[0047] i and j are the row and column positions of pixels in the input image;

[0048] R(i,j), G(i,j), and B(i,j) are the R, G, and B channel pixel values ​​of a pixel at a certain location in the image.

[0049] L gray (i, j) is the brightness matrix of the image grayscale;

[0050] M and N are the number of rows and columns of pixels in the input image.

[0051] like Figure 2 As shown, in this embodiment, an instance-level saliency ranking detection model based on graph reasoning is used to process the input image A, which simulates the visual mechanism of the human brain. That is, based on the length of time humans gaze at objects in an image, the attention levels of different salient objects are ranked saliency. Salient objects include humans, animals, objects, and backgrounds, etc., and are ranked from 1 to n from the most salient to the least salient, thus obtaining the saliency ranking graph B.

[0052] like Figure 2 As shown, the specific processing method for the input image is as follows: First, the input image size is adjusted to 480×640. An improved Mask R-CNN network is used to perform object detection and segment salient instances of the image. Then, a branch is added to infer the salient ranking of each instance. In this branch, a feature graph from the network backbone is first constructed. Then, salient features and local contextual features of each instance are extracted to obtain appearance information and local contrast cues. At the same time, a set of global contextual features and person-sensitive features are extracted to combine global contrast and person priors. Afterward, a graph inference module is used to propagate the competitive relationships between different instances and obtain updated features. Finally, we predict the salient score of each instance and obtain a salient ranking map. Then, linear interpolation is used to adjust the image size to 1920×1080 to obtain the final salient ranking map B.

[0053] In the image processing described above: the input image is processed by a Mask R-CNN network to obtain instance segmentation results. Box detection, person detection, and mask detection are added in parallel to determine whether each instance is a person. Then, ResNet features are fused to obtain a fused salient feature map, from which instance features, as well as local and global contextual features, are extracted for salient ranking inference. The graph inference model integrates instance interaction relationships, local and global comparative inference, and human prior knowledge to obtain updated instance features. This operation has been used in classic slot carving image retargeting algorithms, utilizing saliency ranking to adjust the priority of regions to be retained or deleted.

[0054] Step 2: Divide the backlight partitions into blocks corresponding to the backlight partitions as determined in Step 1. Assign different weights to the blocks containing each significant object based on the significant ranking of each significant object obtained in Step 1. Then, perform weighted calculation and correction on the initial backlight value of each block based on the weights, thereby determining the final block backlight value of each block.

[0055] Specifically, after dividing the image into backlit blocks, the blocks containing salient objects of different salient rankings in the salient ranking map B are weighted and ranked from 1 to n according to their salientness from high to low. The highest salient ranking has a weight of 1. As the salient ranking decreases, the weight is calculated using the following formula:

[0056] α = 1 - β*(n-1),

[0057] Where n is the sequence number and β is a test value.

[0058] In this embodiment, the specific testing method for the test value β is as follows: Twenty participants (16 men and 4 women) with normal or corrected vision were selected for the experiment, ranging in age from 20 to 27 years old. A 25-inch LCD monitor with a resolution of 1920×1080 was used, and the participants observed from a distance of 45 cm. First, the original image was shown to the observers for 3 seconds, followed by a blank image for 1.5 seconds, and finally the image after dimming was shown, at which point the β value was set to 0.05. The β value was then continuously varied, and different dimmed images were displayed, allowing the observers to score them. The β value with the highest average score was then recorded. Therefore, this test value β reflects the optimal value at which the weight of the segmented backlight brightness decreases as the ranking decreases, resulting in the best display effect of the input image after this dimming method.

[0059] If several salient objects with different salient rankings coexist within a block, the weight value of the backlight block is calculated by summing the weight values ​​assigned to each pixel of the different salient objects, and then averaging the summations. The formula is as follows:

[0060]

[0061] in:

[0062] M×N represents the resolution of the input image;

[0063] α(R, S) is the weight value of the backlight partition in row S and column R, α(i, j) is the weight value of the pixel in row j and column i of the image, and M / R and N / S are the number of pixels in the backlight partition.

[0064] Finally, the weighted values ​​of the obtained blocks are weighted together with the backlight values ​​BL1 of the first partition and BL2 of the second partition, and the initial backlight value BL3 is corrected. The final calculation formula for the block backlight value BL4 is as follows:

[0065] BL4 = BL1*α + BL2*(1-α),

[0066] in,

[0067] BL1 is the backlight value for the first zone;

[0068] BL2 is the backlight value for the second zone;

[0069] α represents the weight value assigned to different partitions;

[0070] BL4 represents the final partition backlight value obtained, and all partition backlight values ​​obtained by this method form an R×S backlight value matrix, where R represents the number of columns of the backlight partition and S represents the number of rows of the backlight partition.

[0071] The final backlight values ​​BL4 obtained from all the blocks are combined into an R×S backlight value matrix.

[0072] Step 3: Use the BMA light mixing diffusion algorithm to simulate the distribution of the block backlight value of each block obtained in Step 2 on the LCD panel, and determine the pixel compensation value. Display the input image based on the block backlight value and pixel compensation value of each block.

[0073] Specifically, the dimming factor K of the segmented backlight module is first determined based on the segmented backlight value BL4. R,S The calculation formula is:

[0074]

[0075] Where n is the bit width of grayscale image I; BL R,S The final backlight value for the block in row S and column R.

[0076] According to the dimming factor K R,S Configure the pulse width modulation (PWM) signal for the backlight LEDs in each backlight module and send it to the LED driver module of the corresponding backlight module.

[0077] Then, the R×S backlight value matrix composed of the backlight values ​​BL4 of all backlight zones is finally transformed into an M×N fuzzy diffusion matrix through fuzzy diffusion based on the BMA light mixing diffusion algorithm. In this embodiment, the diffusion method is the sliding window method. A window template is selected, and the diffusion window template is (2t+1)×(2t+1), where t≥1 and t is a positive integer. The specific values ​​within the diffusion template need to be determined through actual light diffusion measurements.

[0078] In this embodiment, the diffusion template is a 3×3 window template, as shown below:

[0079]

[0080] The sliding window method is operated as follows:

[0081] (a) The edges of the initially obtained 64×32 brightness matrix are mirrored and filled with rows and columns to obtain a 66×34 brightness matrix. The matrix is ​​then convolved with the diffusion template F to obtain a new 64×32 brightness matrix.

[0082] (b) The 64×32 brightness matrix output in step (a) is expanded to a 128×64 brightness matrix using bilinear interpolation. The edges of the 128×64 brightness matrix are then mirrored and filled with rows and columns to obtain a 130×66 brightness matrix. A new 128×64 backlight matrix is ​​obtained by convolving it with the diffusion template F.

[0083] (c) The 128×64 brightness matrix output in step (b) is expanded to a 256×128 brightness matrix using bilinear interpolation. The edges of the 256×128 brightness matrix are then mirrored in rows and columns to obtain a 258×130 brightness matrix. A new 256×128 backlight matrix is ​​obtained by convolving it with the diffusion template F.

[0084] (d) The 256×128 brightness matrix output in step (c) is expanded to a 512×256 brightness matrix using bilinear interpolation. The edges of the 512×256 brightness matrix are then mirrored in rows and columns to obtain a 514×258 brightness matrix. A new 512×256 brightness matrix is ​​obtained by convolving it with the diffusion template F.

[0085] (e) The new 512×256 brightness matrix is ​​subjected to bilinear interpolation, so that the backlight matrix can directly simulate diffusion to a size of 1920×1080.

[0086] The brightness of each pixel in the image to be processed is obtained from the fuzzy diffusion matrix. A liquid crystal compensation algorithm is then used to obtain a brightness adjustment signal for each pixel, which is sent to the liquid crystal control module. In this embodiment, the liquid crystal compensation algorithm can be either a linear or nonlinear compensation algorithm.

[0087] Through the above process, the pulse width modulation (PWM) signal of the backlight LED is sent to the LED driver module of the corresponding backlight module, and the brightness adjustment signal of each pixel is sent to the LCD control module, thereby completing the control dimming, and finally the input image is displayed on the LCD screen.

[0088] The preferred embodiments of the present invention have been described in detail above with reference to the accompanying drawings. These embodiments are merely descriptions of preferred embodiments and are not intended to limit the scope or concept of the invention. The specific technical features described in the above embodiments can be combined in any suitable manner without contradiction. Such combinations, as long as they do not violate the spirit of the present invention, should also be considered as part of this disclosure. To avoid unnecessary repetition, the present invention will not further describe the various possible combinations.

[0089] This invention is not limited to the specific details of the above embodiments. Within the scope of the technical concept of this invention and without departing from the design idea of ​​this invention, all modifications and improvements made by those skilled in the art to the technical solutions of this invention should fall within the protection scope of this invention. The technical content for which protection is sought in this invention has been fully described in the claims.

Claims

1. A dynamic dimming method for liquid crystal display areas based on an image saliency ranking model, characterized in that, Includes the following steps: Step 1: Convert the input image into a grayscale image, partition the grayscale image according to the number of partitions of the backlight, extract the partition backlight value of each backlight partition using different methods, and calculate the initial backlight value of each backlight partition based on the partition backlight value. Furthermore, the input image is preprocessed using an instance-level saliency ranking detection model based on graph reasoning. First, the size of the input image is adjusted, then the saliency scores of different salient objects in the input image are predicted for saliency ranking, and finally the size of the output image is restored to obtain the final saliency ranking map. Step 2: Divide the saliency ranking map obtained in Step 1 into blocks corresponding to the backlight partitions determined in Step 1. Assign different weights to the blocks containing each salient object according to the saliency ranking of each salient object obtained in Step 1. Then, perform weighted calculation and correction on the initial backlight value of each block according to the weights, thereby determining the final block backlight value of each block. Step 3: Use the BMA light mixing diffusion algorithm to simulate the distribution of the block backlight value of each block obtained in Step 2 on the LCD panel, and determine the compensation value of the pixel. Display the input image according to the block backlight value of each block and the compensation value of the pixel. In step 1, the sub-pixel maximum value method is used. First, the value of the channel with the largest brightness information among the R, G, and B channels of each pixel in the input image is taken as the brightness value of that pixel. Then, the maximum brightness value of the pixels in the partition is taken as the backlight value BL1 of the first partition. The average value method is used to accumulate the grayscale values ​​of each pixel in each partition, and then the average value is taken as the backlight value BL2 of the second partition. Based on the backlight values ​​BL1 and BL2 of the first and second partitions, the initial backlight value BL3 of each backlight partition is calculated. The formula for calculating the initial backlight BL3 is as follows: ； In step 1, the instance-level saliency ranking detection model based on graph reasoning uses an improved Mask R-CNN network for salient instance segmentation and adds a saliency ranking branch to rank the relative saliency of each segmented instance. Different salient objects in the input image are ranked according to their importance in the image, i.e., the length of time the human eye gazes at different contents in the image. The input image is preprocessed before backlight extraction is performed. In step 2, the partitions containing significant objects with different significance rankings in the significance ranking graph are weighted. The partition with the highest significance has a weight of 1. As the significance ranking decreases, the weight value is calculated using the following formula: ， Where n is the rank number and β is a test value; If several salient objects with different salient rankings coexist within a partition, the weight value of the backlight partition is calculated by summing the weight values ​​assigned to each pixel of the different salient objects, and then averaging the summations. The formula is as follows: ， in: M×N represents the resolution of the input image; R is the number of columns of the corresponding backlight partitions in the input image, which are divided according to the number of partitions of the backlight, and S is the number of rows of the corresponding backlight partitions. The weight value of the backlight partition in row S and column R is given. Let M / R and N / S be the weight values ​​of the pixel in the j-th row and i-th column of the image, where M / R and N / S are the number of pixels in the backlight partition. In step 2, the final backlight value for each block is calculated using the following formula: ， in, BL1 is the backlight value for the first zone; BL2 is the backlight value for the second zone; Weight values ​​assigned to different partitions; BL4 represents the final backlight value of each zone, and all the backlight values ​​of each zone are combined into an R×S backlight value matrix, where R represents the number of columns of the backlight zone and S represents the number of rows of the backlight zone. In step 3, based on the final partitioned backlight values, the dimming factor of the corresponding partitioned backlight value module is determined. The pulse width modulation (PWM) signal of the backlight LED in each partitioned backlight value module is set according to the dimming factor and sent to the LED driver module of the corresponding backlight module. Furthermore, the backlight matrix composed of all the final partitioned backlight values ​​is blurred and diffused using the BMA light mixing diffusion algorithm to obtain a blurred diffusion matrix. The backlight value corresponding to each pixel on the LCD panel is obtained from the blurred diffusion matrix. Then, the LCD compensation algorithm is used to obtain the LCD compensation signal for each pixel and sent to the LCD control module to display the input image.

2. The dynamic dimming method for liquid crystal display areas based on an image saliency ranking model according to claim 1, characterized in that, The liquid crystal compensation algorithm can employ either a linear or nonlinear compensation algorithm.

Images