Backlight display method, device and equipment based on gazing point and readable storage medium

Abstract

The invention provides a backlight display method, device and equipment based on a gaze point and a readable storage medium, and the method comprises the steps: traversing each pixel point in a compressed image to be decompressed, and determining the statistical information of each liquid crystal partition in a liquid crystal image corresponding to the compressed image according to the coordinate of each pixel point and an image compression rule corresponding to the compressed image; determining liquid crystal backlight distribution information according to the statistical information of each liquid crystal partition; for each pixel point in the traversed compressed image, according to the coordinate of the pixel point, an image compression rule and a corresponding backlight and liquid crystal partition rule, determining the coordinate of a target sampling point on a liquid crystal backlight distribution matrix corresponding to the pixel point; and according to the coordinates of the target sampling points on the liquid crystal backlight distribution matrix and the liquid crystal backlight distribution information, determining values of pixel points in the compressed image through an interpolation algorithm. According to the method provided by the invention, on the basis of the smaller-resolution image compressed by the gaze point, the display effect is not lost, and meanwhile, the computing resources are saved.

Description

Technical Field

[0001] The present application relates to the field of display, and in particular to a gaze point-based backlight display method, device, equipment, and readable storage medium. Background Art

[0002] With the increasing demand for display effects and the rapid development of software and hardware technologies in the display field, more and more new display technologies have emerged to improve the contrast, color gamut and other effects of displayed images. Among them, miniLED display technology has now been widely used.

[0003] Currently, existing miniLED display technology uses 2D backlight display technology. In 2D backlight display technology, the area of ​​the original image (uncompressed image) corresponding to each backlight source is generally fixed and uniform. Therefore, it is impossible to directly use 2D backlight display technology to process compressed images (non-uniform images), and a large amount of computing power is required to achieve it. Therefore, although this solution can achieve good display effects, it also consumes a lot of computing resources.

[0004] Therefore, the existing technology cannot reduce the occupied computing resources while ensuring the display effect. Summary of the Invention

[0005] The present application provides a gaze point-based backlight display method, device, equipment and readable storage medium to solve the problem that the existing technology cannot reduce the occupied computing resources while ensuring the display effect.

[0006] In a first aspect, embodiments of the present application provide a gaze point-based backlight display method, which is applied to a gaze point-based display device, wherein the gaze point-based display device is configured in a head-mounted device or an electronic device; the method comprises:

[0007] Traversing each pixel point in the compressed image to be decompressed, and determining statistical information of each liquid crystal partition in the liquid crystal image corresponding to the compressed image according to the coordinates of each pixel point and the image compression rule corresponding to the compressed image;

[0008] Determining liquid crystal backlight distribution information according to statistical information of each liquid crystal partition; the liquid crystal backlight distribution information includes coordinates of each sampling point in each liquid crystal partition and corresponding values ​​of each sampling point;

[0009] For each pixel in the traversed compressed image, the coordinates of the target sampling point corresponding to the pixel on the liquid crystal backlight distribution matrix are determined based on the coordinates of the pixel, the image compression rule, and the corresponding backlight and liquid crystal partitioning rules; the compressed image is compressed based on the gaze point;

[0010] According to the coordinates of the target sampling point on the liquid crystal backlight distribution matrix and the liquid crystal backlight distribution information, the value of the pixel point in the compressed image is determined by an interpolation algorithm.

[0011] In one possible design, determining the coordinates of a target sampling point corresponding to the pixel point on the liquid crystal backlight distribution matrix based on the coordinates of the pixel point, the image compression rule, and corresponding backlight and liquid crystal partitioning rules includes:

[0012] Determining the coordinates of a liquid crystal pixel corresponding to the pixel on a liquid crystal backlight distribution matrix according to an image compression rule corresponding to the compressed image and the coordinates of the pixel;

[0013] The coordinates of the target sampling point corresponding to the liquid crystal pixel point are determined according to the coordinates of the liquid crystal pixel point and the backlight and liquid crystal partitioning rules; wherein the target sampling point corresponding to the liquid crystal pixel point on the liquid crystal backlight distribution matrix is ​​the target sampling point corresponding to the pixel point.

[0014] In one possible design, determining the value of a pixel in the compressed image by an interpolation algorithm based on the coordinates of the target sampling point on the liquid crystal backlight distribution matrix and the liquid crystal backlight distribution information includes:

[0015] Determining a neighborhood sampling point for interpolation according to the coordinates of the target sampling point;

[0016] Determining a value of the neighborhood sampling point according to the coordinates of the neighborhood sampling point and the liquid crystal backlight distribution information;

[0017] According to the coordinates of the neighborhood sampling points and the values ​​of the neighborhood sampling points, the value of the target sampling point is calculated by an interpolation algorithm;

[0018] The value of the target sampling point is used as the value of the pixel point.

[0019] In one possible design, determining the neighborhood sampling points for interpolation based on the coordinates of the target sampling point includes:

[0020] According to the coordinates of the target sampling point, the coordinates of the four pixel points in the area corresponding to the target sampling point are determined by a bilinear interpolation method;

[0021] The four pixel points in the area are used as neighborhood sampling points for interpolation.

[0022] In one possible design, determining the liquid crystal backlight distribution information based on the statistical information of each liquid crystal partition includes:

[0023] Determining a liquid crystal backlight matrix based on the statistical information of each liquid crystal partition and using a backlight display algorithm; the liquid crystal backlight matrix is ​​composed of the values ​​of the backlights of each liquid crystal partition;

[0024] According to the liquid crystal backlight matrix, liquid crystal backlight distribution information is determined by a preset upsampling resolution.

[0025] In one possible design, determining statistical information of each liquid crystal partition in the liquid crystal image corresponding to the compressed image based on the coordinates of each pixel point and an image compression rule corresponding to the compressed image includes:

[0026] For each pixel point, determining a compression ratio of the pixel point in the horizontal and vertical directions corresponding to the current liquid crystal partition according to the image compression rule;

[0027] According to the compression ratio corresponding to each pixel point and the coordinates of each pixel point, statistical information of each liquid crystal partition in the liquid crystal image corresponding to the compressed image is determined; the statistical information includes an average value, a maximum value, and a minimum value corresponding to each liquid crystal partition.

[0028] In a second aspect, an embodiment of the present application provides a gaze point-based backlight display device, wherein the gaze point-based display device is configured in a head-mounted device or an electronic device; the device includes:

[0029] a statistical information acquisition module, configured to traverse each pixel point in the compressed image to be decompressed, and determine statistical information of each liquid crystal partition in the liquid crystal image corresponding to the compressed image based on the coordinates of each pixel point and the image compression rule corresponding to the compressed image;

[0030] A first display processing module, configured to determine liquid crystal backlight distribution information based on statistical information of each liquid crystal partition; the liquid crystal backlight distribution information includes coordinates of each sampling point in each liquid crystal partition and corresponding values ​​of each sampling point;

[0031] a second display processing module, configured to determine, for each pixel in the traversed compressed image, the coordinates of a target sampling point on the liquid crystal backlight distribution matrix corresponding to the pixel based on the coordinates of the pixel, the image compression rule, and corresponding backlight and liquid crystal partitioning rules; the compressed image being compressed based on the gaze point;

[0032] The third display processing module is configured to determine the value of a pixel in the compressed image by an interpolation algorithm according to the coordinates of the target sampling point on the liquid crystal backlight distribution matrix and the liquid crystal backlight distribution information.

[0033] In a third aspect, an embodiment of the present application provides a head-mounted device, comprising: a computing processing unit and a display processing unit; wherein the computing processing unit is used to execute the method described in any one of the first aspects; and the display processing unit is used to display a corresponding image based on the value of each pixel point.

[0034] In a fourth aspect, an embodiment of the present application provides an electronic device, including: a processor and a memory;

[0035] The memory stores computer-executable instructions;

[0036] The processor executes the computer-executable instructions stored in the memory, so that the processor performs the method as described in any one of the first aspects.

[0037] In a fifth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions. When a processor executes the computer-executable instructions, the method described in any one of the first aspects is implemented.

[0038] In a sixth aspect, an embodiment of the present application provides a computer program product, comprising a computer program, which, when executed by a processor, implements the method as described in any one of the first aspects.

[0039] The gaze point-based backlight display method, apparatus, device and readable storage medium provided in this embodiment first traverse each pixel point in the compressed image to be decompressed, and determine the statistical information of each liquid crystal partition in the liquid crystal image corresponding to the compressed image based on the coordinates of each pixel point and the image compression rule corresponding to the compressed image; further, determine the liquid crystal backlight distribution information based on the statistical information of each liquid crystal partition; the liquid crystal backlight distribution information includes the coordinates of each sampling point in each liquid crystal partition and the corresponding values ​​of each sampling point; further, for each pixel point in the traversed compressed image, determine the coordinates of the target sampling point corresponding to the pixel point on the liquid crystal backlight distribution matrix based on the coordinates of the pixel point, the image compression rule and the corresponding backlight and liquid crystal partition rules; the compressed image is compressed based on the gaze point; further, according to the coordinates of the target sampling point on the liquid crystal backlight distribution matrix and the liquid crystal backlight distribution information, the value of the pixel point in the compressed image is determined by an interpolation algorithm. Therefore, the present application directly processes the compressed image (non-uniform symmetrical relationship) without decompressing and restoring the original image. That is, first, based on the traversed pixel points and the corresponding compression rules, statistical information is determined (the statistical information is a uniform symmetrical relationship), and then based on the statistical information, the coordinates and values ​​of the sampling points in the liquid crystal partition corresponding to the compressed image are determined through sampling, and then the compensated value of the pixel point is obtained through interpolation calculation to achieve compressed resolution (non-uniform) output. There is no need for decompression and restoration and then display processing and then compression processing, which solves the unevenness problem caused by the non-uniformity caused by the compressed image. On the image with smaller resolution based on gaze point compression, computing resources are saved without losing the display effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0041] Figure 1 A schematic diagram of a scene of a backlight display method based on a gaze point provided in an embodiment of the present application;

[0042] Figure 2 A schematic diagram of a flow chart of a gaze point-based backlight display method provided in an embodiment of the present application;

[0043] Figure 3 A schematic diagram of a scene of a backlight display method based on a gaze point provided in another embodiment of the present application;

[0044] Figure 4A schematic flow chart of a backlight display method based on a gaze point according to another embodiment of the present application;

[0045] Figure 5 A schematic diagram of gaze point compression provided in an embodiment of the present application;

[0046] Figure 6 A schematic diagram of bilinear interpolation provided in an embodiment of the present application;

[0047] Figure 7 A schematic structural diagram of a gaze point-based backlight display device provided in an embodiment of the present application;

[0048] Figure 8 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0049] To make the purpose, technical solutions, and advantages of the embodiments of this application more clear, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0050] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of this application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the numbers used in this way are interchangeable where appropriate, so that the embodiments of the application described herein can, for example, be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0051] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0052] Currently, existing miniLED display technology uses 2D backlight display technology. In 2D backlight display technology, the area in the original image (uncompressed image) corresponding to each backlight source is generally fixed and uniform. Therefore, it is impossible to directly use 2D backlight display technology to process compressed images (non-uniform images), and a large amount of computing power is required to achieve this. Therefore, although this solution can achieve a good display effect, it also takes up a lot of computing power resources. Therefore, the existing technology cannot reduce the occupied computing power resources while ensuring the display effect (or without losing the display effect).

[0053] Therefore, in response to the above problems, the technical concept of the present application is to directly process the compressed image (non-uniform symmetrical relationship) without decompressing and restoring the original image, that is, first determine the statistical information based on the traversed pixel points and the corresponding compression rules (the statistical information is a uniform symmetrical relationship), and then determine the coordinates and values ​​of the sampling points in the liquid crystal partition corresponding to the compressed image based on the statistical information through sampling, and then obtain the compensated value of the pixel point through interpolation calculation to achieve compressed resolution (non-uniform) output, without the need for decompression and restoration and then display processing and then compression processing, to solve the unevenness problem caused by the non-uniformity caused by the compressed image, and save computing resources while not losing the display effect on the smaller resolution image based on gaze point compression.

[0054] In practical applications, the backlight display method based on gaze point can be applied to fields such as VR, and is not specifically limited here. Figure 1 As shown, Figure 1A scene diagram of the gaze point-based backlight display method provided in an embodiment of the present application, wherein the scene includes a gaze point-based backlight display device, which can be a head-mounted device, including a computing processing unit 101 (e.g., a CPU) and a display processing unit 102 (e.g., a GPU, which can be configured on a monitor or display screen). The calculation processing unit 101 traverses each pixel in the compressed image to be decompressed, and determines the statistical information of each liquid crystal partition in the liquid crystal image corresponding to the compressed image based on the coordinates of each pixel and the image compression rule corresponding to the compressed image; then determines the liquid crystal backlight distribution information based on the statistical information of each liquid crystal partition; the liquid crystal backlight distribution information includes the coordinates of each sampling point in each liquid crystal partition and the corresponding value of each sampling point; for each pixel in the traversed compressed image, the coordinates of the target sampling point on the liquid crystal backlight distribution matrix corresponding to the pixel are determined based on the coordinates of the pixel, the image compression rule, and the corresponding backlight and liquid crystal partition rules; the compressed image is compressed based on the gaze point; based on the coordinates of the target sampling point on the liquid crystal backlight distribution matrix and the liquid crystal backlight distribution information, the value of the pixel in the compressed image is determined by an interpolation algorithm. The display processing unit 102 is used to display the corresponding image based on the value of each pixel.

[0055] Specifically, combined Figure 2 As shown in , firstly, the image data is input, and the gaze point image coordinates are output through the eye tracking system. The image will be compressed according to the gaze point before being displayed, that is, the image data is compressed based on the gaze point, as shown in Figure 3 The figure shows a common compression method, which compresses pixels at different ratios based on their distance from the gaze point, resulting in a compressed image with a smaller resolution. However, compressed image processing introduces a problem: in 2D backlight display technology, the area in the uncompressed image corresponding to each backlight source is generally fixed and uniform. This makes it impossible to directly apply 2D backlight algorithms to the compressed image, and the non-uniformity caused by the compressed image must be addressed.

[0056] Therefore, to address the non-uniformity caused by the compressed image, image information statistics are completed through independent hardware acceleration (such as DSP, etc.), and dynamic decompression is achieved, so that the image statistical information of the backlight partition can be losslessly restored, thereby not affecting the calculation results of the 2D backlight display algorithm. Similarly, due to the non-uniform symmetry between the compressed image and the actual backlight, the pixel compensation results will also be affected. Because the sampling points and LC pixels (i.e., liquid crystal pixels) and backlight areas calculated by sampling at low-frequency resolution are uniformly symmetrical, if you want to directly output the image at the compressed resolution, you need to perform compression again after uniform upsampling, which will greatly waste computing resources. Therefore, through the liquid crystal pixel compensation unit, a non-uniform upsampling algorithm is adopted. The image at the sampling (SP) resolution can be directly upsampled and interpolated to the compressed resolution, and output through the display pipeline. According to the compensated liquid crystal pixels, the miniLED backlight control unit is used to output the display effect through the miniLED backlight. The effect is basically the same as that after uniform upsampling and then compression, but it greatly saves computing resources.

[0057] Therefore, based on the traversed pixel points and the corresponding compression rules, statistical information is determined (the statistical information is a uniform and symmetrical relationship), and then based on the statistical information, the coordinates and values ​​of the sampling points in the liquid crystal partition corresponding to the compressed image are determined through sampling, and then the compensated value of the pixel point is obtained through interpolation calculation, thereby achieving compressed resolution (uneven) output, without the need for decompression and restoration and then display processing and then compression processing, thereby solving the unevenness problem caused by the non-uniformity brought by the compressed image, and saving computing resources while not losing the display effect on the smaller resolution image based on gaze point compression.

[0058] The following specific embodiments are used to describe the technical solution of the present application in detail. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0059] Figure 4 A flowchart of a gaze point-based backlight display method according to another embodiment of the present application is provided. The gaze point-based backlight display method may include:

[0060] S401 , traversing each pixel point in a compressed image to be decompressed, and determining statistical information of each liquid crystal partition in a liquid crystal image corresponding to the compressed image according to the coordinates of each pixel point and an image compression rule corresponding to the compressed image.

[0061] In this embodiment, the execution entity can be a gaze point-based backlight display device, which is installed in a gaze point-based backlight display device. The gaze point-based backlight display device can be an electronic device or a head-mounted device, which is not specifically limited here.

[0062] For example, using a head-mounted device as an example, the head-mounted device can capture an image to be processed using a camera (including an infrared camera and / or a visible light camera), compress the image to be processed, and obtain a compressed image to be decompressed. For example, the compressed image is 2000*2000. At the same time, the compression rules used when compressing the image are recorded, including the horizontal and vertical compression ratios h and v of each pixel. Based on the coordinates of the currently traversed pixel and the horizontal and vertical compression ratios of the pixel in the current partition, statistical information of each liquid crystal partition corresponding to the compressed image is determined. For example, the statistical information is 20*20.

[0063] Among them, the largest image (the uncompressed original image) is the LCD image, and the pixels on the LCD image are the LCD pixels; the compressed image based on gaze point compression: the resolution is smaller than the original image and becomes non-uniformly smaller; the sampling resolution: the original image or LCD pixels are uniformly reduced; the backlight area is uniformly symmetrical.

[0064] S402 : Determine liquid crystal backlight distribution information according to statistical information of each liquid crystal partition; the liquid crystal backlight distribution information includes coordinates of each sampling point in each liquid crystal partition and corresponding values ​​of each sampling point.

[0065] The LCD backlight distribution information here can be an LCD backlight distribution matrix, which is equivalent to upsampling the statistical information (using the backlight value of a partition to predict the values ​​of other pixels distributed in the same partition, i.e., the value of the sampling point), for example, 200*200. Specifically, based on the statistical information of the LCD partition, the miniLED backlight bead brightness matrix can be calculated, and then the LCD backlight distribution information can be determined, which is a low-resolution LCD backlight distribution matrix.

[0066] S403. For each pixel point in the traversed compressed image, determine the coordinates of the target sampling point corresponding to the pixel point on the liquid crystal backlight distribution matrix based on the coordinates of the pixel point, the image compression rule, and the corresponding backlight and liquid crystal partitioning rules; the compressed image is compressed based on the gaze point.

[0067] Among them, the compressed image based on gaze point compression is a non-uniform symmetric relationship.

[0068] S404 : Determine the value of a pixel in the compressed image by an interpolation algorithm according to the coordinates of the target sampling point on the liquid crystal backlight distribution matrix and the liquid crystal backlight distribution information.

[0069] The values ​​of the pixels in the compressed image can be RGB values ​​or brightness values. Figure 5 Schematic diagram of gaze point compression shown.

[0070] In this embodiment, based on the coordinates of the target sampling point on the liquid crystal backlight distribution matrix and the liquid crystal backlight distribution information, an interpolation algorithm is used to determine the backlight value and compensation value required for the compensation unit to calculate, and then obtain the final compressed image pixel value (here refers to the value of the pixel point in the compressed image) as the display liquid crystal value output.

[0071] First, based on the image compression rules and the coordinates of the currently interpolated pixel, the corresponding LCD pixel coordinates are calculated. Based on the LCD pixel coordinates and the miniLED partitioning rules, the corresponding sampling point coordinates are calculated. Based on the sampling point coordinates, the domain sampling points used for interpolation are found, such as the domain four pixels for bilinear interpolation. Based on the interpolation sampling points and the interpolation method, the value of the current pixel to be interpolated is calculated. This allows for direct upsampling and interpolation of images at the sampled (SP) resolution to the compressed resolution output, achieving essentially the same effect as uniform upsampling followed by compression, but significantly saving computing resources.

[0072] The gaze point-based backlight display method provided in the embodiment of the present application can directly process the compressed image (non-uniform symmetrical relationship) without decompressing and restoring the original image. That is, first, based on the traversed pixel points and the corresponding compression rules, statistical information (the statistical information is a uniform symmetrical relationship) is determined, and then based on the statistical information, the coordinates and values ​​of the sampling points in the liquid crystal partition corresponding to the compressed image are determined through sampling, and then the compensated value of the pixel point is obtained through interpolation calculation to achieve compressed resolution (non-uniform) output. There is no need for decompression and restoration and then display processing and then compression processing, which solves the unevenness problem caused by the non-uniformity caused by the compressed image. On the smaller resolution image compressed based on the gaze point, computing resources are saved without losing the display effect.

[0073] In one possible design, determining statistical information of each liquid crystal partition in the liquid crystal image corresponding to the compressed image based on the coordinates of each pixel point and an image compression rule corresponding to the compressed image includes:

[0074] For each pixel point, determining a compression ratio of the pixel point in the horizontal and vertical directions corresponding to the current liquid crystal partition according to the image compression rule;

[0075] According to the compression ratio corresponding to each pixel point and the coordinates of each pixel point, statistical information of each liquid crystal partition in the liquid crystal image corresponding to the compressed image is determined; the statistical information includes an average value, a maximum value, and a minimum value corresponding to each liquid crystal partition.

[0076] In this embodiment, the statistical information of each liquid crystal partition corresponding to the compressed image is determined based on the coordinates of the currently traversed pixel point and the compression ratio of the pixel point in the horizontal and vertical directions of the current partition. To address the non-uniformity caused by the compressed image, a dynamic decompression algorithm applied to 2D backlight is adopted, so that the image statistical information of the partition where the backlight is located can be losslessly restored, thereby not affecting the calculation results of the 2D backlight algorithm. Specifically, the horizontal and vertical compression ratios h and v of the traversed pixel points are first determined; when calculating the statistical information, the actual value of the current pixel point is restored based on the total compression s = h*v to ensure that it is accurately calculated.

[0077] Taking the average value of each liquid crystal partition as an example, the specific calculation formula is:

[0078]

[0079] Among them, Rx and Ry represent the current pixel point P i,j In the current zone (here refers to the current LCD zone) Zone m,n The compression ratio in the x and y directions.

[0080] It should be noted that the calculation method of the maximum value and the minimum value of each liquid crystal partition is similar to the calculation method of the average value of each liquid crystal partition described above, and will not be repeated here.

[0081] In one possible design, determining the liquid crystal backlight distribution information based on the statistical information of each liquid crystal partition includes:

[0082] Determining a liquid crystal backlight matrix based on the statistical information of each liquid crystal partition and using a backlight display algorithm; the liquid crystal backlight matrix is ​​composed of the values ​​of the backlights of each liquid crystal partition;

[0083] According to the liquid crystal backlight matrix, liquid crystal backlight distribution information is determined by a preset upsampling resolution.

[0084] Among them, the LCD backlight distribution information here can be a LCD backlight distribution matrix, which is equivalent to upsampling of statistical information (using the backlight value of a partition to predict the values ​​of other pixel points distributed in the area, that is, the value of the sampling point), such as 200*200.

[0085] In this embodiment, based on the statistical information of each liquid crystal partition, a liquid crystal backlight matrix is ​​determined through a backlight display algorithm, such as 20*20; according to the liquid crystal backlight matrix, a preset upsampling resolution is used, that is, the value of the backlight of one partition is used to predict the values ​​of other pixel points distributed in the area, that is, the value of the sampling point), such as 200*200. Specifically, based on the statistical information, the backlight display algorithm is used to obtain the value of the liquid crystal backlight corresponding to each liquid crystal partition, and then through upsampling, a multi-value liquid crystal backlight distribution matrix is ​​obtained for each area. Subsequently, the image at the sampling (SP) resolution can be directly upsampled and interpolated to the compressed resolution output, and the effect is basically the same as that of uniform upsampling followed by compression, but it greatly saves computing resources.

[0086] In one possible design, determining the coordinates of a target sampling point corresponding to the pixel point on the liquid crystal backlight distribution matrix based on the coordinates of the pixel point, the image compression rule, and corresponding backlight and liquid crystal partitioning rules includes:

[0087] Determining the coordinates of a liquid crystal pixel corresponding to the pixel on a liquid crystal backlight distribution matrix according to an image compression rule corresponding to the compressed image and the coordinates of the pixel;

[0088] The coordinates of the target sampling point corresponding to the liquid crystal pixel point are determined according to the coordinates of the liquid crystal pixel point and the backlight and liquid crystal partitioning rules; wherein the target sampling point corresponding to the liquid crystal pixel point on the liquid crystal backlight distribution matrix is ​​the target sampling point corresponding to the pixel point.

[0089] In this embodiment, the partitioning logic basis of the backlight and liquid crystal partitioning rules here can be: dividing the size of the partition where the actual backlight lamp bead is located according to the position of the lamp bead on the liquid crystal panel, for example: when the lamp bead is lit alone, it can light up X*Y liquid crystals centered on it, then the X*Y liquid crystal area is the backlight partition of the lamp bead.

[0090] Specifically, first, according to the image compression rules and the coordinates of the currently traversed pixel to be interpolated, the coordinates of the corresponding liquid crystal pixel are calculated; according to the coordinates of the liquid crystal pixel and the miniLED partitioning rules (here refers to the backlight and liquid crystal partitioning rules), the corresponding sampling point coordinates (here refers to the coordinates of the target sampling point) are calculated.

[0091] In one possible design, determining the value of a pixel in the compressed image by an interpolation algorithm based on the coordinates of the target sampling point on the liquid crystal backlight distribution matrix and the liquid crystal backlight distribution information includes:

[0092] Determining a neighborhood sampling point for interpolation according to the coordinates of the target sampling point;

[0093] Determining a value of the neighborhood sampling point according to the coordinates of the neighborhood sampling point and the liquid crystal backlight distribution information;

[0094] According to the coordinates of the neighborhood sampling points and the values ​​of the neighborhood sampling points, the value of the target sampling point is calculated by an interpolation algorithm;

[0095] The value of the target sampling point is used as the value of the pixel point.

[0096] In one possible design, determining the neighborhood sampling points for interpolation based on the coordinates of the target sampling point includes:

[0097] According to the coordinates of the target sampling point, the coordinates of the four pixel points in the area corresponding to the target sampling point are determined by a bilinear interpolation method;

[0098] The four pixel points in the area are used as neighborhood sampling points for interpolation.

[0099] In this embodiment, the domain sampling points used for interpolation are found according to the sampling point coordinates, for example, the domain four pixels in bilinear interpolation; the value of the current pixel to be interpolated, that is, the value of the pixel in the reduced image, is calculated according to the interpolation sampling points and the interpolation method.

[0100] Specifically, according to the above steps, for any point P(i, j) on the upsampled compressed image, its corresponding position on the sampling (SP) resolution is:

[0101] X j+1 =X j +Rx*step row , X0=0

[0102] Y i+1 =Y i +Ry*step col , Y0=0

[0103] Among them, steprow = 1 / SPRow; stepcol = 1 / SPcol; Rx and Ry are the compression ratios of the corresponding pixel points on the compressed image in the x and y directions respectively, (X i , Y i ) represents the corresponding coordinates of any point P(i,j) on the compressed image at the sampling resolution; Steprow represents the step size corresponding to each pixel in the row direction, stepcol represents the step size corresponding to each pixel in the column direction, SPRow represents the total length in the row direction, and SPCol is the total length in the column direction.

[0104] According to the coordinates (X, Y) at the sampling resolution, the interpolation algorithm can be used to evaluate the value. For example, see bilinear interpolation. Figure 6The schematic diagram of bilinear interpolation is shown, and the interpolation calculation of point C(m,n) is performed:

[0105] Among them, the four pixel points in the area of ​​point C are A11(neighbor_x1, neighbor_y1), A12(neighbor_x2, neighbor_y1), A21(neighbor_x1, neighbor_y2), and A22(neighbor_x2, neighbor_y2); neighbor_y1=floor(m); neighbor_y2=ceil(m); neighbor_x1=floor(n); neighbor_x2=ceil(n).

[0106] A11, A12, A21, and A22 represent the four neighborhood points of point C, respectively. neighbor_x1, neighbor_x2, neighbor_y1, and neighbor_y2 are the x and y coordinate values ​​of the four neighborhood points. Floor represents rounding down, and ceil represents rounding up.

[0107] Calculated by interpolation:

[0108] B1 = interp_linear(A11, A12)

[0109] B2 = interp_linear(A21, A22)

[0110] C = interp_linear(B1, B2)

[0111] Among them, B1 and B2 are intermediate results calculated by interpolation (interp_linear represents linear interpolation, other interpolation is also possible, not specifically limited here) according to A11, A12 and A21, A22 in the column direction respectively, and finally the final result C is obtained by interpolation according to B1B2.

[0112] Therefore, compared with traditional 2D backlight solutions, this application's 2D backlight display solution based on gaze points can significantly reduce power consumption while ensuring display quality. The use of independent hardware modules to accelerate the completion of image information statistics can better reduce calculation delays, increase display frame rates, and achieve better display effects.

[0113] In order to implement the backlight display method based on the gaze point, this embodiment provides a backlight display device based on the gaze point. Figure 7 , Figure 7A structural schematic diagram of a gaze point-based backlight display device provided in an embodiment of the present application; the gaze point-based display device is configured in a head-mounted device or an electronic device; the gaze point-based backlight display device includes: a statistical information acquisition module 701, a first display processing module 702, a second display processing module 703 and a third display processing module 704.

[0114] The statistical information acquisition module 701 is configured to traverse each pixel in the compressed image to be decompressed, and determine statistical information of each liquid crystal partition in the liquid crystal image corresponding to the compressed image based on the coordinates of each pixel and the image compression rule corresponding to the compressed image;

[0115] A first display processing module 702 is configured to determine liquid crystal backlight distribution information based on the statistical information of each liquid crystal partition; the liquid crystal backlight distribution information includes the coordinates of each sampling point in each liquid crystal partition and the corresponding value of each sampling point;

[0116] A second display processing module 703 is configured to determine, for each pixel in the traversed compressed image, the coordinates of a target sampling point corresponding to the pixel on the liquid crystal backlight distribution matrix according to the coordinates of the pixel, the image compression rule, and corresponding backlight and liquid crystal partitioning rules; the compressed image is compressed based on the gaze point;

[0117] The third display processing module 704 is configured to determine the value of a pixel in the compressed image by an interpolation algorithm according to the coordinates of the target sampling point on the liquid crystal backlight distribution matrix and the liquid crystal backlight distribution information.

[0118] In this embodiment, the statistical information acquisition module 701, the first display processing module 702, the second display processing module 703 and the third display processing module 704 are used to traverse each pixel point in the compressed image to be decompressed, and determine the statistical information of each liquid crystal partition in the liquid crystal image corresponding to the compressed image based on the coordinates of each pixel point and the image compression rule corresponding to the compressed image; further, the liquid crystal backlight distribution information is determined based on the statistical information of each liquid crystal partition; the liquid crystal backlight distribution information includes the coordinates of each sampling point in each liquid crystal partition and the corresponding value of each sampling point; further, for each pixel point in the traversed compressed image, the coordinates of the target sampling point corresponding to the pixel point on the liquid crystal backlight distribution matrix are determined based on the coordinates of the pixel point, the image compression rule and the corresponding backlight and liquid crystal partition rules; the compressed image is compressed based on the gaze point; further, based on the coordinates of the target sampling point on the liquid crystal backlight distribution matrix and the liquid crystal backlight distribution information, the value of the pixel point in the compressed image is determined by an interpolation algorithm. Therefore, the present application directly processes the compressed image (non-uniform symmetrical relationship) without decompressing and restoring the original image. That is, first, based on the traversed pixel points and the corresponding compression rules, the statistical information is determined (the statistical information is a uniform symmetrical relationship), and then based on the statistical information, the coordinates and values ​​of the sampling points in the liquid crystal partition corresponding to the compressed image are determined by sampling, and then the compensated value of the pixel point is obtained through interpolation calculation to achieve compressed resolution (non-uniform) output. There is no need for decompression and restoration and then display processing and then compression processing, which solves the unevenness problem caused by the non-uniformity brought by the compressed image, ensuring the display effect while saving computing resources.

[0119] The gaze point-based backlight display device provided in this embodiment can be used to implement the technical solution of the above-mentioned gaze point-based backlight display method embodiment. Its implementation principle and technical effects are similar and will not be repeated in this embodiment.

[0120] In one possible design, the second display processing module 703 is specifically configured to:

[0121] Determining the coordinates of a liquid crystal pixel corresponding to the pixel on a liquid crystal backlight distribution matrix according to an image compression rule corresponding to the compressed image and the coordinates of the pixel;

[0122] The coordinates of the target sampling point corresponding to the liquid crystal pixel point are determined according to the coordinates of the liquid crystal pixel point and the backlight and liquid crystal partitioning rules; wherein the target sampling point corresponding to the liquid crystal pixel point on the liquid crystal backlight distribution matrix is ​​the target sampling point corresponding to the pixel point.

[0123] In one possible design, the third display processing module 704 is specifically configured to:

[0124] Determining a neighborhood sampling point for interpolation according to the coordinates of the target sampling point;

[0125] Determining a value of the neighborhood sampling point according to the coordinates of the neighborhood sampling point and the liquid crystal backlight distribution information;

[0126] According to the coordinates of the neighborhood sampling points and the values ​​of the neighborhood sampling points, the value of the target sampling point is calculated by an interpolation algorithm;

[0127] The value of the target sampling point is used as the value of the pixel point.

[0128] In one possible design, the third display processing module 704 is specifically configured to:

[0129] According to the coordinates of the target sampling point, the coordinates of the four pixel points in the area corresponding to the target sampling point are determined by a bilinear interpolation method;

[0130] The four pixel points in the area are used as neighborhood sampling points for interpolation.

[0131] In one possible design, the first display processing module 702 is specifically configured to:

[0132] Determining a liquid crystal backlight matrix based on the statistical information of each liquid crystal partition and using a backlight display algorithm; the liquid crystal backlight matrix is ​​composed of the values ​​of the backlights of each liquid crystal partition;

[0133] According to the liquid crystal backlight matrix, liquid crystal backlight distribution information is determined by a preset upsampling resolution.

[0134] In one possible design, the statistical information acquisition module 701 is specifically configured to:

[0135] For each pixel point, determining a compression ratio of the pixel point in the horizontal and vertical directions corresponding to the current liquid crystal partition according to the image compression rule;

[0136] According to the compression ratio corresponding to each pixel point and the coordinates of each pixel point, statistical information of each liquid crystal partition in the liquid crystal image corresponding to the compressed image is determined; the statistical information includes an average value, a maximum value, and a minimum value corresponding to each liquid crystal partition.

[0137] In order to implement the above-mentioned backlight display method based on gaze point, this embodiment provides a head-mounted device, combined with Figure 1 and Figure 2 As shown, the head-mounted device includes a computing processing unit and a display processing unit; wherein, the computing processing unit is used to execute the method as described in any one of the first aspects; and the display processing unit is used to display the corresponding image according to the value of each pixel point.

[0138] The head-mounted device provided in this embodiment can be used to execute the technical solution of the above-mentioned method embodiment. Its implementation principle and technical effects are similar, and will not be repeated here in this embodiment.

[0139] In order to implement the method of the above embodiment, this embodiment provides an electronic device. Figure 8 This is a schematic diagram of the structure of the electronic device provided in the embodiment of the present application. Figure 8 As shown, the electronic device of this embodiment includes: a processor 801 and a memory 802; wherein the memory 802 is used to store computer-executable instructions; the processor 801 is used to execute the computer-executable instructions stored in the memory to implement the various steps performed in the above embodiment. For details, please refer to the relevant description of the above method embodiment.

[0140] An embodiment of the present application further provides a computer-readable storage medium, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, the above method is implemented.

[0141] An embodiment of the present application also provides a computer program product, including a computer program, which implements the above method when executed by a processor.

[0142] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of modules is only a logical function division. There may be other division methods in actual implementation. For example, multiple modules can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or modules, which can be electrical, mechanical or other forms. In addition, the functional modules in the various embodiments of the present application can be integrated into a processing unit, or each module can exist physically separately, or two or more modules can be integrated into one unit. The unit composed of the above modules can be implemented in the form of hardware or in the form of hardware plus software functional units.

[0143] The above-mentioned integrated module implemented in the form of a software function module can be stored in a computer-readable storage medium. The above-mentioned software function module is stored in a storage medium, including a number of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor (English: processor) to perform some steps of the methods of each embodiment of the present application. It should be understood that the above-mentioned processor can be a central processing unit (English: Central Processing Unit, referred to as: CPU), or other general-purpose processors, digital signal processors (English: Digital Signal Processor, referred to as: DSP), application-specific integrated circuits (English: Application Specific Integrated Circuit, referred to as: ASIC), etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in conjunction with the invention can be directly embodied as being executed by a hardware processor, or can be executed by a combination of hardware and software modules in the processor.

[0144] The memory may include high-speed RAM memory, and may also include non-volatile storage NVM, such as at least one disk storage, and may also be a USB flash drive, a mobile hard disk, a read-only memory, a magnetic disk or an optical disk, etc. The bus may be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the buses in the drawings of this application are not limited to only one bus or one type of bus. The above-mentioned storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The storage medium can be any available medium that can be accessed by a general-purpose or special-purpose computer.

[0145] An exemplary storage medium is coupled to a processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be an integral part of the processor. The processor and storage medium can be located in an application-specific integrated circuit (ASIC). Of course, the processor and storage medium can also exist as discrete components in an electronic device or a main control device.

[0146] Those skilled in the art will appreciate that all or part of the steps in the above-described method embodiments can be implemented using hardware associated with program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0147] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some or all of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A backlight display method based on gaze point, characterized in that: The method is applied to a gaze point-based display device, wherein the gaze point-based display device is configured in a head-mounted device or an electronic device; the method comprises: Traversing each pixel point in the compressed image to be decompressed, and determining statistical information of each liquid crystal partition in the liquid crystal image corresponding to the compressed image according to the coordinates of each pixel point and the image compression rule corresponding to the compressed image; Determining liquid crystal backlight distribution information according to statistical information of each liquid crystal partition; the liquid crystal backlight distribution information includes coordinates of each sampling point in each liquid crystal partition and corresponding values ​​of each sampling point; For each pixel in the traversed compressed image, the coordinates of a target sampling point corresponding to the pixel on the liquid crystal backlight distribution matrix are determined based on the coordinates of the pixel, the image compression rule, and corresponding backlight and liquid crystal partitioning rules; the compressed image is compressed based on the gaze point; According to the coordinates of the target sampling point on the liquid crystal backlight distribution matrix and the liquid crystal backlight distribution information, the values ​​of the pixels in the compressed image are determined by an interpolation algorithm.

2. The method according to claim 1, characterized in that The determining, based on the coordinates of the pixel point, the image compression rule, and the corresponding backlight and liquid crystal partitioning rules, the coordinates of the target sampling point corresponding to the pixel point on the liquid crystal backlight distribution matrix includes: Determining the coordinates of a liquid crystal pixel corresponding to the pixel on a liquid crystal backlight distribution matrix according to an image compression rule corresponding to the compressed image and the coordinates of the pixel; The coordinates of the target sampling point corresponding to the liquid crystal pixel point are determined according to the coordinates of the liquid crystal pixel point and the backlight and liquid crystal partitioning rules; wherein the target sampling point corresponding to the liquid crystal pixel point on the liquid crystal backlight distribution matrix is ​​the target sampling point corresponding to the pixel point.

3. The method according to claim 1, characterized in that The determining, by an interpolation algorithm based on the coordinates of the target sampling point on the liquid crystal backlight distribution matrix and the liquid crystal backlight distribution information, the values ​​of the pixels in the compressed image comprises: Determining a neighborhood sampling point for interpolation according to the coordinates of the target sampling point; Determining a value of the neighborhood sampling point according to the coordinates of the neighborhood sampling point and the liquid crystal backlight distribution information; According to the coordinates of the neighborhood sampling points and the values ​​of the neighborhood sampling points, the value of the target sampling point is calculated by an interpolation algorithm; The value of the target sampling point is used as the value of the pixel point.

4. The method according to claim 3, characterized in that The step of determining a neighborhood sampling point for interpolation according to the coordinates of the target sampling point includes: According to the coordinates of the target sampling point, the coordinates of the four pixel points in the area corresponding to the target sampling point are determined by a bilinear interpolation method; The four pixel points in the area are used as neighborhood sampling points for interpolation.

5. The method according to any one of claims 1 to 4, characterized in that The determining of liquid crystal backlight distribution information according to the statistical information of each liquid crystal partition includes: Determining a liquid crystal backlight matrix based on the statistical information of each liquid crystal partition and using a backlight display algorithm; the liquid crystal backlight matrix is ​​composed of the values ​​of the backlights of each liquid crystal partition; According to the liquid crystal backlight matrix, liquid crystal backlight distribution information is determined by a preset upsampling resolution.

6. The method according to any one of claims 1 to 4, characterized in that The determining, based on the coordinates of each pixel point and the image compression rule corresponding to the compressed image, the statistical information of each liquid crystal partition in the liquid crystal image corresponding to the compressed image includes: For each pixel point, determining a compression ratio of the pixel point in the horizontal and vertical directions corresponding to the current liquid crystal partition according to the image compression rule; According to the compression ratio corresponding to each pixel point and the coordinates of each pixel point, statistical information of each liquid crystal partition in the liquid crystal image corresponding to the compressed image is determined; the statistical information includes an average value, a maximum value, and a minimum value corresponding to each liquid crystal partition.

7. A backlight display device based on gaze point, characterized in that: The gaze point-based display device is configured in a head-mounted device or an electronic device; the device includes: a statistical information acquisition module, configured to traverse each pixel point in the compressed image to be decompressed, and determine statistical information of each liquid crystal partition in the liquid crystal image corresponding to the compressed image based on the coordinates of each pixel point and the image compression rule corresponding to the compressed image; A first display processing module, configured to determine liquid crystal backlight distribution information based on statistical information of each liquid crystal partition; the liquid crystal backlight distribution information includes coordinates of each sampling point in each liquid crystal partition and corresponding values ​​of each sampling point; a second display processing module, configured to determine, for each pixel in the traversed compressed image, the coordinates of a target sampling point on the liquid crystal backlight distribution matrix corresponding to the pixel based on the coordinates of the pixel, the image compression rule, and corresponding backlight and liquid crystal partitioning rules; the compressed image being compressed based on the gaze point; The third display processing module is configured to determine the value of a pixel in the compressed image by an interpolation algorithm according to the coordinates of the target sampling point on the liquid crystal backlight distribution matrix and the liquid crystal backlight distribution information.

8. A head-mounted device, characterized in that: include: A computing processing unit and a display processing unit; wherein the computing processing unit is used to execute the method according to any one of claims 1 to 6; The display processing unit is used to display the corresponding image according to the value of each pixel point.

9. An electronic device, characterized in that: include: a processor, and a memory communicatively connected to the processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the method according to any one of claims 1 to 6.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 6 when executed by a processor.

11. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images