Image display method, device, readable medium and electronic device

Abstract

The present application relates to an image display method, device, readable medium and electronic device. The method includes: obtaining an image to be processed and user's gaze point position information; based on the obtained image to be processed and the gaze point position information, rendering the image to be processed to obtain a first image and a second image; interlacing and integrating the first image and the second image according to a set row ratio to obtain an integrated image; caching at least part of the row of pixel points corresponding to each row of pixel points of the first image, amplifying at least part of the row of pixel points corresponding to each row of pixel points of the second image, combining the amplified part of the row of pixel points and the cached part of the row of pixel points to obtain a combined image. The present application can effectively reduce display delay and reduce the size of the buffer by interlacing and integrating the gaze point area image and the peripheral area image according to a set row ratio and then caching, amplifying, combining and displaying them.

Description

Technical Field

[0001] The present application relates to the field of virtual reality technology, and in particular to an image display method, device, readable medium and electronic device. Background Art

[0002] Virtual Reality (VR) technology is a computer simulation technology that can create a virtual world for users and give them an immersive experience. It can be widely used in medicine, entertainment, military aerospace, exhibitions and other fields.

[0003] When people use VR devices (such as all-in-one VR devices), they have higher and higher requirements for the clarity, realism, and immersion of virtual images. This means that the requirements for the computing and rendering capabilities of the graphics processing unit (GPU) and the image data transmission bandwidth are also getting higher and higher. However, the size, weight, and battery life of VR devices limit the computing power of the GPU and the transmission bandwidth of image data. Usually, the foveated rendering solution can be used to reduce the amount of GPU calculations, but because the image data rendered by the GPU is large, a larger buffer is required to cache the image data, which is bound to bring challenges to the size, weight, battery life, image latency and product cost of VR devices. Summary of the invention

[0004] The embodiments of the present application provide a method, device, readable medium and electronic device for displaying an image. The technical solution of the present application renders the image to be processed based on the gaze point rendering technology after the system side acquires the image to be processed, thereby obtaining a sub-image corresponding to the human eye gaze point area and a sub-image corresponding to the periphery of the human eye gaze point area. The system side interlaces and integrates the sub-image corresponding to the human eye gaze point area and the sub-image corresponding to the periphery of the human eye gaze point area according to a set row ratio to obtain an integrated image, and then sends the integrated image to the display side for display. It can effectively reduce display latency and reduce the size of the buffer.

[0005] In a first aspect, an embodiment of the present application provides a method for displaying an image, comprising:

[0006] Acquire an image to be processed and the user's gaze point position information, wherein the gaze point position information is used to describe the position where the user's visual center falls on the screen of the electronic device; render the image to be processed based on the acquired image to be processed and the gaze point position information to obtain a first image and a second image, wherein the first image corresponds to a first area in the image to be processed, and the second image corresponds to a second area in the image to be processed, the first area is an area within a preset range of the user's gaze point position in the image to be processed, and the second area is an area outside the first area in the image to be processed; interlace and integrate each row of pixel points in the first image and each row of pixel points in the second image according to a set row ratio to obtain an integrated image; cache at least part of the row of pixel points in each row of pixel points in the integrated image corresponding to the first image, amplify at least part of the row of pixel points in each row of pixel points in the integrated image corresponding to the second image, combine the amplified part of the row of pixel points and the cached part of the row of pixel points to obtain a combined image; and display the combined image.

[0007] In a possible implementation of the first aspect, the method further includes: the first image and the second image have the same resolution.

[0008] In a possible implementation of the first aspect, the method further includes: interlacing and integrating each row of pixels in the first image and each row of pixels in the second image according to a set row ratio, and obtaining an integrated image includes:

[0009] Each a row of pixels in the first image is grouped together with each b row of pixels in the second image is grouped together to obtain an integrated image, wherein a:b=X, a>b, and a and b are positive integers.

[0010] In a possible implementation of the first aspect, the method further includes: caching at least part of the rows of pixel points in each row of pixel points corresponding to the first image in the integrated image, amplifying at least part of the rows of pixel points in each row of pixel points corresponding to the second image in the integrated image, combining the amplified part of the rows of pixel points and the cached part of the rows of pixel points, and obtaining the combined image includes:

[0011] Buffering each a row of pixels corresponding to the first image in the integrated image, and amplifying each b row of pixels corresponding to the second image in the integrated image by Y times;

[0012] The pixel points in row a of the first image currently being cached are read, and the pixel points in row b that are enlarged in the second image are combined with the pixel points in row a that are currently being cached in the read first image to obtain a combined image.

[0013] In a possible implementation of the first aspect, the method further includes: characterized in that the value of X is equal to the value of Y.

[0014] In a possible implementation of the first aspect, the method further includes: after the resolution of the second image is magnified Y times, the resolution is the same as the resolution of the combined image.

[0015] In a possible implementation of the first aspect, the method further includes: caching at least part of the rows of pixel points in each row of pixel points corresponding to the first image in the integrated image, amplifying at least part of the rows of pixel points in each row of pixel points corresponding to the second image in the integrated image, combining the amplified part of the rows of pixel points and the cached part of the rows of pixel points, and obtaining the combined image includes:

[0016] Compressing each row of pixel points corresponding to the first image in the integrated image and each row of pixel points corresponding to the second image in the integrated image, wherein a compression ratio corresponding to each row of pixel points in the first image is greater than a compression ratio corresponding to each row of pixel points in the second image;

[0017] caching at least a portion of the compressed rows of pixel points corresponding to the first image and at least a portion of the compressed rows of pixel points corresponding to the second image;

[0018] Reading at least a portion of the compressed rows of pixel points corresponding to the first image and at least a portion of the compressed rows of pixel points corresponding to the second image that are currently cached, and decompressing the read at least a portion of the compressed rows of pixel points corresponding to the first image and at least a portion of the compressed rows of pixel points corresponding to the second image that are currently cached;

[0019] The decompressed at least part of the row of pixel points corresponding to the second image is magnified S times and then combined with the decompressed at least part of the row of pixel points corresponding to the first image to obtain a combined image.

[0020] In a possible implementation of the first aspect, the method further includes: after the resolution of the second image is magnified S times, the resolution is the same as the resolution of the combined image.

[0021] In a possible implementation of the first aspect, the method further includes: caching at least part of the rows of pixel points in each row of pixel points corresponding to the first image in the integrated image, amplifying at least part of the rows of pixel points in each row of pixel points corresponding to the second image in the integrated image, combining the amplified part of the rows of pixel points and the cached part of the rows of pixel points, and obtaining the combined image includes:

[0022] Cache each row of pixel points in the integrated image corresponding to the first image and cache one row of pixel points in the integrated image corresponding to the second image;

[0023] Reading a portion of the rows of pixels corresponding to the first image and one row of pixels corresponding to the second image that are currently cached, and amplifying the read row of pixels corresponding to the second image by T times;

[0024] The enlarged row of pixel points corresponding to the second image is combined with the read partial row of pixel points corresponding to the first image currently cached to obtain a combined image.

[0025] In a possible implementation of the first aspect, the method further includes: after the resolution of the second image is magnified T times, the resolution is the same as the resolution of the combined image.

[0026] In a second aspect, an embodiment of the present application provides an image display device, including:

[0027] An acquisition module, used to acquire the image to be processed and the user's gaze point position information, wherein the gaze point position information is used to describe the position where the user's visual center falls on the screen of the electronic device;

[0028] A first processing module is used to render the image to be processed based on the acquired image to be processed and the gaze point position information to obtain a first image and a second image, wherein the first image corresponds to a first area in the image to be processed, and the second image corresponds to a second area in the image to be processed, the first area is an area within a preset range of the user's gaze point position in the image to be processed, and the second area is an area outside the first area in the image to be processed;

[0029] A second processing module, used for interlacing and integrating each row of pixel points in the first image and each row of pixel points in the second image according to a set row ratio to obtain an integrated image;

[0030] A third processing module is used to cache at least part of the rows of pixel points in each row of pixel points in the integrated image corresponding to the first image, amplify at least part of the rows of pixel points in each row of pixel points in the integrated image corresponding to the second image, and combine the amplified part of the rows of pixel points and the cached part of the rows of pixel points to obtain a combined image; and a display module is used to display the combined image.

[0031] In a third aspect, an embodiment of the present application provides a readable medium having instructions stored thereon, which, when executed on an electronic device, enables the electronic device to execute the above-mentioned first aspect and any one of the image display methods of various possible implementations of the first aspect.

[0032] In a fourth aspect, an embodiment of the present application provides an electronic device, including:

[0033] a memory for storing instructions to be executed by one or more processors of the electronic device, and

[0034] The processor is one of the processors of the electronic device, and is used to execute the above-mentioned first aspect and any one of the image display methods in various possible implementations of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] FIG1( a ) is a schematic diagram showing an application scenario of an image display method according to some embodiments of the present application;

[0036] FIG1( b ) is a schematic diagram showing the position of a human eye's gaze point according to some embodiments of the present application;

[0037] Figure 2 According to some embodiments of the present application, a hardware structure block diagram of the system end shown in FIG. 1( a ) is shown;

[0038] Figure 3 According to some embodiments of the present application, a hardware structure block diagram of the display terminal shown in FIG1( a ) is shown;

[0039] Figure 4 According to some embodiments of the present application, a display path of an image to be processed is shown;

[0040] Figure 5 According to some embodiments of the present application, it is shown that Figure 4 The corresponding processing flow of the image to be processed;

[0041] Figure 6 According to some embodiments of the present application, it is shown Figure 5 A timing diagram of an image to be processed in the embodiment shown;

[0042] Figure 7 According to some embodiments of the present application, a display path of an image to be processed is shown;

[0043] Figure 8 According to some embodiments of the present application, it is shown that Figure 7 The corresponding processing flow of the image to be processed;

[0044] FIG9( a ) shows sub-image A and sub-image B both having a resolution of M*N according to some embodiments of the present application;

[0045] FIG9( b ) shows a method of integrating sub-image A and sub-image B according to some embodiments of the present application;

[0046] FIG9( c ) shows a method of integrating sub-image A and sub-image B according to some embodiments of the present application;

[0047] FIG9( d ) shows an image with a horizontal resolution of M*X and a vertical resolution of N*X according to some embodiments of the present application;

[0048] FIG9( e ) shows another image with a horizontal resolution of M*X and a vertical resolution of N*X according to some embodiments of the present application;

[0049] Fig.10 According to some embodiments of the present application, it is shown Figure 8 The display timing diagram of the image to be processed is shown;

[0050] Fig.11 According to some embodiments of the present application, a display path of an image to be processed in the display method provided by the present application is shown;

[0051] Fig.12 According to some embodiments of the present application, it is shown Fig.11 The display process of the image to be processed is shown;

[0052] Fig.13 According to some embodiments of the present application, another display path of the image to be processed is shown;

[0053] Fig.14 According to some embodiments of the present application, it is shown Fig.13 The display process of the image to be processed is shown;

[0054] Fig.15 According to some embodiments of the present application, it is shown Fig.14 The display timing diagram of the image to be processed is shown.

[0055] Fig.16 According to some embodiments of the present application, a structural block diagram of an image display device is shown. DETAILED DESCRIPTION

[0056] Illustrative embodiments of the present application include, but are not limited to, methods, apparatuses, readable media, and electronic devices for displaying images.

[0057] The technical solution of the present application renders the image to be processed based on the foveation rendering technology after the system end obtains the image to be processed, and obtains a sub-image corresponding to the human eye foveation area (for the convenience of description, hereinafter referred to as sub-image A) and a sub-image corresponding to the periphery of the human eye foveation area (for the convenience of description, hereinafter referred to as sub-image B). The system end interlaces and integrates sub-image A and sub-image B according to the set row ratio. For example, every 3 rows of pixels of sub-image A are grouped as a group (for example, a0, a1, a2) and every 1 row of pixels of sub-image B are grouped as a group (for example, b0), and the order of the combined images is: a0, a1, a2, b0, a3, a4, a5, b1, etc., and so on, to obtain the integrated image. Then the system end outputs the integrated image to the display end, and the display end caches the received interlaced integrated image. Among them, the data currently cached in the buffer of the display end includes part of the rows of sub-image A, for example, the data currently cached in the buffer are rows a0, a1, and a2 of sub-image A. Once the display terminal receives a row of sub-image B corresponding to a partial row of sub-image A currently cached in the buffer, the row of sub-image B can be enlarged according to the resolution of the display screen of the display terminal to fill the row of sub-image B with pixels so that the resolution of sub-image B is consistent with the resolution of the display screen. The enlarged row of image B is combined with the partial row of sub-image A mentioned above, and finally displayed by the display terminal. Among them, the magnification of sub-image B is the same as the value of the set row ratio when the sub-image A and sub-image B are interlaced and integrated. Since the data written into the buffer of the display terminal each time is a partial row of sub-image A, it can be combined with the partial row of sub-image B after enlargement, and then the display screen is driven by the driving circuit to display the image, instead of writing all sub-image A into the buffer of the display terminal at once. Therefore, the display delay can be effectively reduced. And the size of the display terminal buffer can be reduced, which is conducive to reducing the volume and weight of the product, reducing costs, and improving product endurance.

[0058] Fixation Point Rendering is a selective image rendering technology that can select the fixation area of ​​the human eye for full-resolution rendering based on eye tracking technology, and blur the area outside the fixation area, thereby achieving an image with clear fixation areas and blurred non-fixation areas. This can greatly reduce the amount of data calculation and data transmission bandwidth, thereby saving computing resources and reducing power consumption.

[0059] The embodiments of the present application will be described in further detail below in conjunction with the accompanying drawings.

[0060] FIG1(a) shows a schematic diagram of an application scenario 10 of the image display method provided by the present application according to some embodiments of the present application. The scenario 10 includes a system end 200 and a display end 100. The system end 200 can communicate with the display end 100 through various wired (e.g., USB interface) or wireless (e.g., Bluetooth communication link, radio frequency identification technology, short-range wireless communication technology, etc.) methods and perform firmware updates on the display end 100. Users can use the display end 100 to experience three-dimensional stereoscopic vision such as watching movies, shopping, playing games, and remote teaching.

[0061] In the embodiment shown in FIG1(a), the system end 200 can calculate the position information of the user's visual center on the display screen when the user is looking at the display screen of the display end 100 based on the eye tracking technology, that is, the user's gaze point position information. For example, in the embodiment shown in FIG1(b), the user's visual center falls at the center of the display screen, that is, the gaze point position is at the center of the display screen, the area near the gaze point position can be regarded as the gaze point area, and the area outside the gaze point area can be regarded as the periphery of the gaze point area. Then, the image to be processed (that is, the image to be displayed by the display end 100) is rendered by the gaze point rendering technology to obtain sub-image A and sub-image B. The system end 200 can interlace and integrate sub-image A and sub-image B according to the set line ratio. For example, the resolutions of sub-image A and sub-image B are both 1280*1280, that is, sub-image A and sub-image B both have 1280 rows of pixels and 1280 columns of pixels. From top to bottom, the 1280 rows of pixels of sub-image A are recorded as rows a1 to a1280; the 1280 rows of pixels of sub-image B are recorded as rows b1 to b1280. If sub-image A and sub-image B are interlaced and integrated in a ratio of 3:1, the order after integration is: a1, a2, a3, b1, a4, a5, a6, b2, a7, a8, a9, b3, and so on. When the 1280 rows of pixels of sub-image A are integrated, blank pixels are used to interlace and integrate with the remaining rows of sub-image B in the same manner as described above. Then the system end 200 outputs the integrated image to the display end 100, and the display end caches the received interlaced integrated image. The data currently cached in the buffer of the display end includes some rows of sub-image A, for example, the data currently cached in the buffer are rows a1, a2, and a3 of sub-image A. Once the display end receives a row of sub-image B corresponding to the partial rows of sub-image A currently cached in the buffer, for example, row b1 of sub-image B, the row of sub-image B can be enlarged according to the resolution of the display screen of the display end, that is, the row of sub-image B is filled with pixels so that the resolution of sub-image B is consistent with the resolution of the display screen. And it is combined with the partial rows of sub-image A mentioned above, and finally displayed by the display end. The magnification of sub-image B is the same as the value of the row ratio set when sub-image A and sub-image B are interlaced and integrated.

[0062] Since the data written into the buffer of the display end each time is part of the rows of sub-image A, it can be combined with part of the rows of the enlarged sub-image B, and then the display screen is driven by the driving circuit to display the image, rather than writing all the sub-image A into the buffer of the display end at once. For example, the data currently written into the buffer of the display end is the rows a1, a2, and a3 of sub-image A. Once the display end receives the row b1 of sub-image B, the pixels of the row b1 of sub-image B can be filled with 3 times the pixels to obtain b1', b2', and b3'. Then, b1' and a1 are combined, for example, in a rectangular covering manner, and then the image is displayed on the display screen. Similarly, b2' and a2 are combined; b3' and a3 are combined and then the image is displayed on the display screen. And there is no need to cache sub-image B. Therefore, the display delay can be effectively reduced, and the size of the display end buffer can be reduced, which is conducive to reducing the volume and weight of the product, reducing costs, and improving the product endurance.

[0063] In some embodiments of the present application, the received sub-image A and sub-image B may be compressed at different ratios at the display end 100 before being cached, and the compression ratio of sub-image B is greater than that of sub-image A. When the image is displayed, the compressed sub-image A and sub-image B cached in the cache are decompressed, and then the decompressed sub-image B is enlarged and combined with the decompressed sub-image A for image display. Since the display end 100 first compresses sub-image A and sub-image B at different ratios before caching, the cache pressure of the cache is greatly reduced, and the capacity requirement of the cache is reduced.

[0064] In addition, in some embodiments of the present application, the display end 100 can also directly cache the received sub-image A. After the display end 100 receives one line of sub-image B, the line of sub-image B can be enlarged. The enlarged line of sub-image B is combined with sub-image A for image display. Since the display end needs to cache all of sub-image A and one line of sub-image B, it does not need to cache all of sub-image B. Therefore, the cache pressure of the cache can be greatly reduced, and the capacity requirements of the cache can be reduced, which is conducive to reducing the volume and weight of the product, reducing costs, and improving battery life.

[0065] In addition, it is understandable that the display end 100 can be any electronic device or partial module of an electronic device that can generate a three-dimensional virtual world, provide users with simulations of senses such as vision, hearing, and touch, form a virtual reality field of view in the user's brain visual system, and generate stereoscopic vision. This application is not limited to this.

[0066] The display end 100 and the system end 200 may be located in the same virtual reality or augmented reality device, such as an all-in-one VR device. The display end 100 and the system end 200 may also be two independent electronic devices that can communicate with each other. For example, the display end 100 may be VR / AR glasses, VR / AR helmets, etc., and the system end 200 may be a server, a distributed server cluster composed of multiple servers, a mobile phone, a tablet computer, a laptop computer, a desktop computer, etc.

[0067] Figure 2 According to some embodiments of the present application, a hardware structure block diagram of a system end 200 shown in Fig. 1(a) is shown. The system end 200 includes a central processing unit (CPU) 201, a graphics processing unit (GPU) 202, a memory 203, a communication module 204, an interface module 205, and the like.

[0068] The CPU 201 can be used to analyze the acquired image including the human eye and calculate the position information of the gaze point of the human eye. The GPU 202 is used to render the image to be processed according to the position information of the gaze point of the human eye calculated by the CPU 201, and obtain a sub-image of the human eye gaze point area and a sub-image of the periphery of the human eye gaze point area. The GPU 202 can also be used to interlace and integrate the sub-image of the human eye gaze point area and the sub-image of the periphery of the human eye gaze point area according to a set line ratio.

[0069] The memory 203 is used to store software programs and data, and the CPU 201 executes various functional applications and data processing of the system end 20 by running the software programs and data stored in the memory 203. For example, in some embodiments of the present application, the memory 203 can store the image including the human eye acquired by the CPU 201, the human eye gaze point position information calculated based on the image including the human eye, the sub-image of the human eye gaze point area and the sub-image of the periphery of the human eye gaze point area rendered by the graphics processor 202, and the image obtained by interlacing and integrating the sub-image of the human eye gaze point area and the sub-image of the periphery of the human eye gaze point area according to the set line ratio, etc.

[0070] The communication module 204 can be used to enable the system end 200 and the display end 100 to communicate through various wired communication methods or wireless communication methods. For example, the system end 200 communicates with the display end 100 through wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), bluetooth (blue tooth, BT), global navigation satellite system (global navigation satellite system, GNSS), frequency modulation (frequency modulation, FM), near field communication technology (near field communication, NFC), infrared technology (infrared, IR) and other wireless communication methods.

[0071] The interface module 205 includes an external memory interface, a universal serial bus (USB) interface, etc. The external memory interface can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the system end 200. The external memory card communicates with the central processor 201 through the external memory interface to implement the data storage function. The universal serial bus interface can be used for the system end 200 to communicate with the display end 100 or other electronic devices. In some embodiments, when the system end 200 is a smart phone, the interface module 205 can also include a subscriber identification module (SIM) card interface.

[0072] The system end 200 also includes a power supply, a power management system, etc. The power management system is used to connect the power supply, CPU 201, GPU 202, memory 203, communication module 204, and interface module 205. The power management system 106 receives the input of the power supply and supplies power to the CPU 201, GPU 202, memory 203, communication module 204, interface module 205, etc. In some other embodiments, the power management system can also be set in the CPU 201.

[0073] Understandably, Figure 2 The hardware structure shown is only an exemplary structure for realizing the functions of the system end 200 in the technical solution of the present application. The system end 200 having other structures and realizing similar functions is also applicable to the technical solution of the present application and is not limited here.

[0074] Figure 3According to some embodiments of the present application, a hardware structure block diagram of a display terminal 100 shown in Fig. 1(a) is shown. The display terminal 100 includes an interface module 101, a communication module 102, a processor 103, a camera 105, an audio module 106, a display screen 107, a sensor module 108, and a lens 109.

[0075] The processor 103 may include one or more processing units, for example, a processing module or processing circuit including a central processing unit (CPU), an image signal processor (ISP), a digital signal processor (DSP), a microprocessor (MCU), an artificial intelligence (AI) processor, or a field programmable gate array (FPGA). For example, the processor 103 may be used to perform calculations based on the data transmitted by the sensor module 108 to generate spatial position and direction information of the user's head.

[0076] The power module 111 may include a power supply, a power management component, etc. The power supply may be a battery. The power management component is used to manage the charging of the power supply and the power supply of the power supply to other modules. The power management component is used to connect the power supply, the charging management module and the processor 103.

[0077] The lens 109 includes two lenses. Since there is only one screen, the images viewed by the left and right eyes must be separated to achieve stereoscopic vision. The left and right eye images are displayed alternately on the screen continuously, that is, the parallax of the left and right eyes is utilized, and the physiological characteristics of the visual persistence of the human eye are added, so that users can experience stereoscopic visual images.

[0078] The display screen 107 is used to display images and includes a display driver integrated circuit (DDIC) 104 , a display panel 110 , and the like.

[0079] DDIC 104 may include a row drive circuit, a column drive circuit, a timing control circuit, and a buffer. It is used to send drive signals and data to display screen 107 in the form of electrical signals, and to display image information such as letters and pictures on display panel 110 by controlling the brightness and color of display panel 110. For example, in some embodiments of the present application, DDIC 104 is used to Figure 2The GPU 202 of the system end 200 shown in the figure compresses, decompresses, amplifies, and combines the sub-image of the human eye gaze point area and the sub-image of the periphery of the human eye gaze point area to drive the display screen 107 to display the image. In some embodiments of the present application, the DDIC 104 can also be used to Figure 2 The GPU 202 of the system end 200 shown in the figure outputs the sub-image of the human eye gaze point area and the sub-image of the periphery of the human eye gaze point area after interlaced integration according to the set row ratio, and then amplifies and combines them to drive the display screen 107 to display the image. Among them, the timing control circuit controls the row driving circuit to select the pixel points of each row of the pixel array of the display screen 107, and then outputs the sub-image A and sub-image B cached in the buffer to the display screen through the column driving circuit for display. The buffer is used to Figure 2 The GPU 202 of the system end 200 shown in the figure outputs the sub-image of the human eye gaze point area and the sub-image of the periphery of the human eye gaze point area to cache, or Figure 2 The GPU 202 of the system end 200 shown in the figure outputs the sub-image of the human eye gaze point area after interlaced integration according to the set line ratio and the sub-image outside the human eye gaze point area for caching. It should be noted that in some embodiments of the present application, only the sub-image of the human eye gaze point area, or the sub-image of the human eye gaze point area after interlaced integration, can also be cached.

[0080] The display panel 110 may be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a quantum dot light-emitting diode (QLED), etc.

[0081] The camera 105 is used to collect human eye images or videos for Figure 2 The CPU 201 of the system end 200 shown uses eye tracking technology to calculate the position information of the gaze point of the human eye. Then, the GPU 202 uses the gaze point rendering technology to render a sub-image of the human eye gaze point area and a sub-image of the periphery of the human eye gaze point area. In some embodiments, the camera 105 can be an infrared camera, and the position of the infrared light emitted by the infrared light emitting module (not shown) in the cornea of ​​the user's eye is determined to determine the position information of the user's eye gazing at the display screen 107.

[0082] The sensor module 108 may include an inertial measurement unit (IMU) nine-axis sensor, a spatial position sensor, etc., for generating spatial position and direction information of the user's head.

[0083] Audio module 106 can convert digital audio information into analog audio signal output, or convert analog audio input into digital audio signal. Audio module 106 can also be used for audio signal encoding and decoding. In certain embodiments, audio module 106 can include loudspeaker, receiver, microphone and earphone interface.

[0084] The communication module 102 can be used to enable the system end 200 and the display end 100 to communicate through various wired communication methods or wireless communication methods. The interface module 101 includes an external memory interface, a universal serial bus (USB) interface, etc. Figure 2 Similar to the communication module 204 shown in FIG. Figure 2 The interface module 205 shown is similar and will not be described in detail here.

[0085] It is understood that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the display terminal 100. In other embodiments of the present application, the display terminal 100 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0086] First reference Figures 4 to 6 The present invention describes a display scheme using foveated rendering technology in related art.

[0087] exist Figure 4 In the illustrated embodiment, the display path of the image to be processed is illustrated. The data source may include the human eye gaze point position information and the image to be processed. Based on the gaze point rendering technology, the GPU 202 of the system 200 performs region-by-region rendering on the image to be processed according to the human eye gaze point position information to obtain sub-image A and sub-image B. Then the GPU 202 outputs the sub-image A and sub-image B to the display terminal 100. The buffer in the display terminal 100 caches the sub-image A and sub-image B, and then amplifies the sub-image B and combines it with the sub-image A, and then controls the column driving circuit and the row driving circuit through the timing control circuit, so that the pixel array displays the combined sub-image A and sub-image B.

[0088] refer to Figure 5 The display solutions using the foveated rendering technology in the above-mentioned related technologies generally include:

[0089] Step 500: The CPU 201 of the system end 200 calculates and obtains the gaze point information of the human eye based on the eye tracking technology.

[0090] Step 502: The GPU 202 of the system end 200 obtains sub-image A and sub-image B based on the obtained human eye gaze point information and the gaze point rendering technology, and outputs sub-image A and sub-image B.

[0091] Step 504: the display terminal 100 receives and buffers sub-image A and sub-image B, and then amplifies sub-image B and combines it with sub-image A. The purpose of amplification is to make the resolution of sub-image B consistent with that of the display screen.

[0092] Step 506: The timing control circuit controls the column driving circuit and the row driving circuit to make the pixel array display the combined image.

[0093] Figure 6 Shows Figure 5 The timing diagram of the image to be processed in the embodiment shown. The vertical synchronization (V-Sync) signal is generated by the display terminal 100 when a frame of the image on the display screen of the display terminal 100 is completely refreshed from left to right and from top to bottom. When the GPU 202 of the system terminal 200 turns on vertical synchronization, the GPU 202 will wait for the V-Sync signal from the display terminal 100 to be sent before rendering a new frame to solve the image tearing phenomenon and increase the smoothness of the image. Figure 6 It can be seen that since the display terminal 100 needs to cache sub-image A in its buffer before enlarging the received sub-image B and combining it with sub-image A before displaying it, there is a display delay of about half a frame from the time the GPU 202 starts rendering to the time the image is finally displayed on the display screen. For example, if the frame rate of the GPU 202 is 90 Hz, there is a display delay of about 5.5 milliseconds, which affects the user experience. In addition, from Figure 6 It can be seen from the figure that the display time of a frame of image on the display screen starts from when the sub-image B of the current frame of image is written into the buffer and ends when the sub-image B of the next frame of image is written into the buffer. Since the buffer needs to cache both sub-image A and sub-image B, the buffer capacity requirement is relatively high, so that the buffer of DDIC 104 is relatively large, resulting in an increase in the size of DDIC 104, which is not conducive to controlling costs and reducing product size.

[0094] Therefore, in order to solve the problem of long display delay and high buffer capacity requirement in DDIC 104, the embodiment of the present application proposes that after the system end 200 renders sub-image A and sub-image B corresponding to the image to be processed, the system end 200 interleaves and integrates sub-image A and sub-image B according to a set line ratio, and then outputs the integrated image to the display end 100. DDIC 104 in the display end 100 processes the interleaved integrated image and then displays it.

[0095] The following uses the example in which the display end 100 shown in FIG. 1 is a VR pair of glasses and the system end 200 is a computer to introduce the display methods provided in some embodiments of the present application in detail.

[0096] The following will refer to Figures 7 to 10 , explaining the display method provided according to some embodiments of the present application. Figure 7 In the illustrated embodiment, the display path of the image to be processed in the image display method provided by the present application is shown. Among them, the data source includes the image to be processed, and the image including the human eye taken by the camera of the VR glasses 100 received by the computer 200. In some embodiments, the image to be processed can be a picture frame of a movie downloaded by the movie playback application on the computer 200, an image stored in the image database of the computer 200, an image interface displayed on the shopping website of the computer 200, and the like. It can be understood that the movie screen seen by the user is composed of a series of continuous still images. When the computer 200 and the VR glasses 100 perform image processing, each frame of the image performs the same processing method. In order to facilitate the description of the present application scheme, any frame of the above-mentioned continuous frames of images is collectively referred to as the image to be processed. CPU201 calculates the position information of the human eye gaze point based on the image including the human eye. Based on the gaze point rendering technology, GPU 202 renders the image to be processed in different regions according to the position information of the human eye gaze point to obtain sub-image A and sub-image B. Among them, the resolution of sub-image A and sub-image B is the same. Then, the GPU 202 interleaves and integrates the sub-image A and the sub-image B according to the set row ratio to obtain an interleave-integrated image, and then outputs the interleave-integrated image to the VR glasses 100 .

[0097] The VR glasses 100 cache the received interlaced integrated image. The data currently cached in the buffer of the VR glasses 100 includes partial rows of sub-image A, for example, the data currently cached in the buffer are rows a1, a2, and a3 of sub-image A. Once the VR glasses 100 receive a row in sub-image B corresponding to the partial rows of sub-image A currently cached in the buffer, for example, row b1 of sub-image B. The row of sub-image B can be enlarged according to the resolution of the display screen of the VR glasses 100, that is, the row of sub-image B is filled with pixels so that the resolution of sub-image B is consistent with the resolution of the display screen. And it is combined with the partial rows of the aforementioned sub-image A, and finally displayed on the VR glasses 100. The magnification of sub-image B is the same as the value of the row ratio set when the aforementioned sub-image A and sub-image B are interlaced and integrated.

[0098] Since the computer 200 interlaces and integrates the sub-image A and the sub-image B according to the set row ratio before transmitting the image. That is, when the computer 200 transmits the integrated image to the VR glasses 100, the arrangement order of each row of the sub-image A and each row of the sub-image B is determined. Therefore, each time the data written into the buffer of the DDIC 104 in the VR glasses 100 is a partial row of the sub-image A, it can be combined with the partial row of the enlarged sub-image B, and then the pixel array is driven by the DDIC 104 to display the image. Instead of writing all the sub-image A into the buffer of the DDIC 104 in the VR glasses 100 at once. And there is no need to cache the sub-image B. That is, the VR glasses 100 do not need to wait until the buffer in the DDIC 104 has completely cached the sub-image A before processing and displaying the image, which can effectively reduce the display delay and improve the user experience. And it can reduce the capacity requirements of the buffer of the DDIC 104, reduce the size of the DDIC 104, and then when designing the product, it is beneficial to reduce the volume, weight, power consumption, etc. of the product.

[0099] Below Figure 7 The display process of the image to be processed is described in detail. Specifically, Figure 8 As shown, the display process of the image to be processed includes:

[0100] Step 800: VR glasses 100 and computer 200 are connected. For example, if a user wants to experience three-dimensional shopping through VR glasses 100, VR glasses 100 and computer 200 can be connected through a data cable, wireless LAN, Bluetooth, etc., so that VR glasses 100 and computer 200 can communicate. It can be understood that in other embodiments, VR glasses 100 and computer 200 can also be connected in other ways, not limited to the above method.

[0101] Step 802: The VR glasses 100 sends the image of the human eye captured by the camera to the computer 200. The computer 200 performs calculation analysis based on the image of the human eye. In some embodiments, the VR glasses 100 can emit infrared light to the human eye through an infrared light emitter. Correspondingly, after the human eye receives the infrared light emitted by the infrared light emitter, an infrared light reflection point is formed in the cornea of ​​the human eye. The image of the human eye is acquired in real time through the infrared camera, and then the image of the human eye captured by the infrared camera is sent to the computer 200.

[0102] Step 804: The computer 200 performs calculations based on the received image including the human eye to obtain the gaze point position information of the human eye.

[0103] Specifically, in some embodiments, the gaze point position information of the human eye can be calculated in the following manner: the CPU 201 of the computer 200 processes the received image including the human eye to obtain the grayscale gradient value in the set direction, and the position where the grayscale gradient maximum value is located is determined as the pupil edge feature of the human eye image, and then the pupil edge feature can be fitted to determine the center position of the fitted image, and the center position is determined as the pupil center position in the human eye image, and the grayscale image obtained after processing the human eye image can be obtained based on the spot recognition algorithm. The area with a higher grayscale value is the Purchin spot, and the position of the Purchin spot is the light spot, or the "reflection point" position. According to the reflection point position and the corneal curvature radius, the corneal curvature center is obtained, and the gaze point position information of the human eye is determined according to the pupil center position, the corneal curvature center and the preset angle between the eye optical axis and the visual axis. It can be understood that in other embodiments, other methods can also be used to determine the gaze point position information of the human eye. This application is not limited to this.

[0104] Step 806: The computer 200 obtains sub-image A and sub-image B based on the gaze point rendering technology according to the obtained gaze point position information of the human eye. Specifically, after the GPU 202 of the computer 200 obtains the image to be processed and the gaze point position information of the human eye, it performs image rendering through vertex transformation calculation, pixel filling calculation, etc., to obtain sub-image A corresponding to the gaze point position area and sub-image B corresponding to the area outside the gaze point.

[0105] Step 808: The computer 200 interlaces and integrates the sub-image A and the sub-image B according to the set row ratio to determine the arrangement order of each row of pixels in the sub-image A and the sub-image B when the GPU 202 transmits data, so that when the GPU 202 sends the image data, it is not necessary to send the sub-image A completely before sending the sub-image B, and thus the buffer in the DDIC 104 of the VR glasses 100 can only buffer part of the rows of the sub-image A, thereby reducing the display delay and the size of the buffer.

[0106] The set row ratio (denoted as X) can be any value set as needed. For example, in some embodiments, the GPU 202 of the computer 200 renders the sub-image A and sub-image B with a resolution of M*N as shown in FIG9(a). The GPU 202 interleaves and integrates the sub-image A and the sub-image B according to the row ratio of X=3. For example, in some embodiments, the resolution of the sub-image A and the sub-image B are both 1280*1280, that is, the sub-image A and the sub-image B have 1280 rows of pixels and 1280 columns of pixels, and the 1280 rows of pixels of the sub-image A are denoted as rows a1 to a1280 from top to bottom; and the 1280 rows of pixels of the sub-image B are denoted as rows b1 to b1280. If sub-image A and sub-image B are interlaced and integrated in a ratio of 3, the order after integration is: a1, a2, a3, b1, a4, a5, a6, b2, a7, a8, a9, b3, and so on.

[0107] In some embodiments, when the human eye gaze point is located above the screen, every 3 rows of pixels in sub-image A and every row of pixels in sub-image B are integrated in accordance with the integration method shown in FIG9(b). In some embodiments, when the human eye gaze point is located at the center of the screen, every 3 rows of pixels in sub-image A and every row of pixels in sub-image B are integrated in accordance with the integration method shown in FIG9(c). It can be understood that, in the embodiment shown in FIG9(a), the resolutions of sub-image A and sub-image B are both M*N, that is, the resolutions are the same. Therefore, when sub-image A and sub-image B are integrated according to the row ratio of X=3, the pixels of sub-image A are arranged first, and blank spaces can be used to integrate the unarranged portions of sub-image B.

[0108] Step 810 : The computer 200 sends the integrated image to the VR glasses 100 .

[0109] For example, in some embodiments, the computer 200 sends the sub-image A and sub-image B integrated according to the set line ratio to the VR glasses 100 through a wireless local area network, Bluetooth, etc., so that the VR glasses 100 can display the image. It can be understood that when the computer 200 sends the integrated sub-image A and sub-image B to the VR glasses 100, the order of the images sent can be, for example, a1, a2, a3, b1, a4, a5, a6, b2, a7, a8, a9, b3, a10, a11, a12, b4, and so on as shown in FIG. 9(b). After the sub-image A is sent, it is 3 blank lines, b5, 3 blank lines, b6, 3 blank lines, b7, 3 blank lines, b8, 3 blank lines, b9, 3 blank lines, b10, 3 blank lines, b11, 3 blank lines, b12, that is, until all the contents of the sub-image B are sent.

[0110] Step 812: VR glasses 100 receive the integrated image and cache some of the rows corresponding to sub-image A. This is for subsequent processing and display by VR glasses 100. For example, in the combination shown in FIG9(b), the order in which the VR glasses 100 receive the images is a1, a2, a3, b1, a4, a5, a6, b2, a7, a8, a9, b3, a10, a11, a12, b4. After all the rows of sub-image A are received, the order is 3 blank rows, b5, 3 blank rows, b6, 3 blank rows, b7, 3 blank rows, b8, 3 blank rows, b9, 3 blank rows, b10, 3 blank rows, b11, 3 blank rows, b12, until all the rows of sub-image B are received. Each time the data written into the buffer of DDIC 104 in VR glasses 100 is part of the rows of sub-image A, not all of A. For example, the current cached contents in the cache of DDIC 104 are a1, a2, a3. After a1, a2, a3 are displayed, the next cached contents are a4, a5, a6. After a4, a5, a6 are displayed, the next cached contents are a7, a8, a9, and so on.

[0111] Step 814: The VR glasses 100 enlarge a row of pixels in sub-image B corresponding to a portion of the rows of sub-image A currently cached.

[0112] For example, in some embodiments, the contents currently cached in the buffer of DDIC 104 are a1, a2, and a3. After receiving the corresponding b1 in sub-image B, VR glasses 100 can use an interpolation algorithm, such as nearest neighbor interpolation, bilinear interpolation, high-order interpolation algorithm, etc., to fill the pixels of the b1 row of sub-image B (i.e., enlarge them). It should be noted that the magnification of each row in sub-image B is the same as the value of the row ratio X set above. For example, in the embodiment shown in FIG9 (a), the resolutions of sub-image A and sub-image B are both M*N, the resolution of the display screen 107 of VR glasses 100 is H*V, and the magnification of sub-image B is Y, where Y can be 1.33, 2, 2.5, 2.85, 3, 4, etc. Then the row ratio X (i.e., a:b) of each a row of sub-image A and each b row of sub-image B is equal to the magnification Y of sub-image B in VR glasses 100. And a and b are both integers. Generally, in order to minimize the size of the buffer, a and b are the smallest integers. For example, in some embodiments, if the resolution of the display screen 107 is H*V=3840*3840, and the magnification Y of the sub-image B is equal to 3, then M=1280, N=1280, a=3, b=1 or a=6, b=2. Similarly, if H*V=3840*3840, X is equal to 2.5, then M=1536, N=1536, a=5, b=2 or a=10, b=4.

[0113] Step 816: VR glasses 100 combine the magnified row pixel points in sub-image B with the corresponding part of the row in sub-image A, so that sub-image B and sub-image A are finally combined into a complete frame of image. In some embodiments, DDIC 104 of VR glasses can cover sub-image A to the position corresponding to the position information of the human eye gaze point in the enlarged sub-image B (for example, rectangular covering, or processing into a circle and then covering) according to the human eye gaze point position information, to obtain a combined image. Then, the splicing boundary of the combined image is smoothly fused, for example, a low-pass filtering method is used to smoothly fuse the boundary area of ​​the combined image to obtain the final image to be displayed. For example, when the human eye gaze point position is located above the display screen 107, the image with a horizontal resolution of M*X and a vertical resolution of N*X as shown in Figure 9 (d) is finally combined. For another example, when the human eye gaze point position is located at the center of the display screen 107, the image with a horizontal resolution of M*X and a vertical resolution of N*X as shown in Figure 9 (e) is finally combined.

[0114] In addition, it is understandable that the position of sub-image A changes with the position of the human eye's gaze point. And, since sub-image A is not enlarged, only sub-image B is enlarged, the final image presented is that the image at the center of human eye vision (i.e., sub-image A) is clear, while the image outside the visual center (i.e., sub-image B) is relatively blurred. Since the human eye is not sensitive to the freshness of images outside the visual center, it can ensure the user's visual experience while reducing the amount of data calculation.

[0115] Step 818: The timing control circuit of the VR glasses 100 controls the column driving circuit and the row driving circuit to make the pixel array display the combined image. For example, in some embodiments, the timing control circuit in the DDIC 104 of the VR glasses can control the row driving circuit of the DDIC 104 to select the pixel points of each row of the pixel array of the display screen 107, and then the column driving circuit of the DDIC 104 will display the information of each row of pixels after the combination on the display screen 107. When the user looks at the display screen 107 of the VR glasses 100 through the lenses of the VR glasses 100, because the images seen by the left and right eyes of the user are separated from each other, coupled with the physiological characteristics of the visual persistence of the human eye, a three-dimensional stereoscopic image can be seen.

[0116] Fig.10 Shows Figure 8 The display timing diagram of the image to be processed in the embodiment shown. When the GPU 202 of the computer 100 turns on vertical synchronization, the GPU 202 will wait for the V-Sync signal of the VR glasses 100 to be sent before rendering a new frame to solve the image tearing phenomenon and increase the smoothness of the image. Fig.10 It can be seen that when the buffer of DDIC 104 completes caching of some lines of sub-image A, as soon as a line of sub-image B is received, the line of sub-image B can be magnified and combined with the corresponding line of sub-image A for display. Figure 4 and Figure 5 The solution shown greatly reduces the display delay, reduces the requirement on the buffer capacity of DDIC 104, and reduces the size of DDIC 104.

[0117] In order to solve the problem of high requirements for the buffer capacity in the DDIC 104, in some embodiments of the present application, the VR glasses 100 can compress the received sub-image A and sub-image B at different magnifications before caching them. This can greatly reduce the buffer pressure and reduce the requirements for the buffer capacity.

[0118] Below, we will refer to Fig.11 and Fig.12 , explaining the image display method provided according to other embodiments of the present application.

[0119] Specifically, refer to Fig.11 and Fig.12 ,exist Fig.11 In the illustrated embodiment, the display path of the image to be processed in the image display method provided by the present application is illustrated. The data source includes the image to be processed and the image including the human eye taken by the camera of the VR glasses 100 received by the computer 200. For example, in some embodiments, for the user picture preview scene, the image to be processed is an image stored in the image database on the computer 200. The CPU 201 calculates the position information of the gaze point of the human eye based on the image including the human eye. The GPU 202 renders the image to be processed in different regions based on the gaze point rendering technology and the position information of the gaze point of the human eye to obtain sub-image A and sub-image B. The rendering process is similar to the aforementioned rendering process and will not be repeated here. The computer 200 sends sub-image A and sub-image B to the VR glasses 100, and the VR glasses 100 compress the received sub-image A and sub-image B respectively and then cache them, wherein the compression ratio of sub-image B is greater than the compression ratio of sub-image A. Then the VR glasses 100 will read the cached compressed sub-image A and sub-image B from the buffer, decompress them, and then amplify the decompressed sub-image B, and display the image after combining it with sub-image A. Since the VR glasses 100 compress the received sub-image A and sub-image B at different magnifications before caching, and the compression magnification of sub-image B is greater than that of sub-image A, the buffer pressure of the buffer can be greatly reduced, the capacity requirement of the buffer can be reduced, and the size of the DDIC 104 can be reduced.

[0120] Below Fig.12 The display process of the image to be processed is described in detail. Fig.12 As shown, the display process of the image to be processed includes:

[0121] Step 1200: VR glasses 100 and computer 200 are connected. For example, if a user wants to view pictures through VR glasses 100, VR glasses 100 and computer 200 can be connected through a data cable, wireless LAN, Bluetooth, etc., so that VR glasses 100 and computer 200 can communicate. It can be understood that in other embodiments, VR glasses 100 and computer 200 can also be connected in other ways, not limited to the above method.

[0122] Step 1202: The VR glasses 100 sends the image of the human eye captured by the camera to the computer 200, so that the computer 200 performs calculation analysis based on the image of the human eye. For example, the image of the human eye is acquired in real time by an infrared camera, and then the image of the human eye captured by the infrared camera is sent to the computer 200.

[0123] Step 1204: The computer 200 performs calculations based on the received image including the human eye to obtain the gaze point position information of the human eye. The calculation method is similar to step 804 and will not be described in detail here.

[0124] Step 1206: The computer 200 obtains sub-image A and sub-image B based on the foveation rendering technology according to the obtained foveation point position information of the human eye. The rendering process is similar to step 806 and will not be described in detail here.

[0125] Step 1208: The computer 200 sends the sub-image A and the sub-image B to the VR glasses 100. For example, in some embodiments, the computer 200 sends the sub-image A and the sub-image B to the VR glasses 100 via a wireless local area network, Bluetooth, etc., so that the VR glasses 100 can display the images.

[0126] It is understandable that, in some embodiments, the computer 200 may also interlace and integrate the sub-image A and the sub-image B according to a set row ratio in a manner similar to the above step 808 to obtain an integrated image, and then send the integrated image to the VR glasses 100.

[0127] Step 1210: VR glasses 100 receive sub-image A and sub-image B, and compress sub-image A and sub-image B respectively for buffering. Since the human eye is not sensitive to the image quality outside the visual center, in order to reduce the requirement for the capacity of the buffer, the compression ratio of sub-image B can be greater than the compression ratio of sub-image A. For example, the compression ratio of sub-image B is 100, and the compression ratio of sub-image A is 10. In some embodiments, sub-image A and sub-image B can be compressed by block division, prediction, transformation, quantization, arithmetic coding, etc.

[0128] Step 1212: The VR glasses 100 decompress the compressed sub-image A and sub-image B read from the buffer. For example, in some embodiments, the compressed sub-image A and sub-image B may be decompressed by an image decompression algorithm.

[0129] Step 1214: The VR glasses 100 enlarge the decompressed sub-image B. For example, in some embodiments, in order to achieve full-screen display of the sub-image B, an interpolation algorithm, such as nearest neighbor interpolation, bilinear interpolation, high-order interpolation algorithm, etc., may be used to make the resolution of the sub-image B consistent with that of the display screen 107.

[0130] Step 1216: The VR glasses 100 combine the enlarged sub-image B with the sub-image A, so that the sub-image B and the sub-image A are combined into a complete image. The combination method is similar to step 816 and will not be repeated here.

[0131] Step 1218: The timing control circuit of the VR glasses 100 controls the column driving circuit and the row driving circuit so that the pixel array displays the combined image.

[0132] In addition, in order to solve the problem of high requirements for the buffer capacity in the DDIC 104, in some embodiments of the present application, the VR glasses 100 can cache all received sub-images A, and when the first line of sub-image B arrives, it is directly enlarged and then combined with sub-image A for display. Since the buffer in the DDIC 104 of the VR glasses 100 needs to cache all the data of sub-image A, it only needs to cache one line of sub-image B in real time, which can greatly reduce the buffer pressure and reduce the buffer capacity requirements.

[0133] It should be noted that in some embodiments, the horizontal scanning frequency of the display screen 107 of the VR glasses 100 needs to be doubled. For example, the frame rate of the display screen 107 of the VR glasses 100 is 90 Hz. In theory, the display screen 107 of the VR glasses 100 can light up each row of pixels within 11 milliseconds, but in some embodiments of the present application, the display screen 107 needs to light up each row of pixels within 5.5 milliseconds.

[0134] Below, we will refer to Figures 13 to 15 , explaining the display methods provided according to other embodiments of the present application.

[0135] Specifically, first refer to Fig.13 and Fig.14 ,exist Fig.13In the illustrated embodiment, the display path of the image to be processed in the image display method provided by the present application is illustrated. Among them, the data source includes the image to be processed, and the image including the human eye taken by the camera of the VR glasses 100 received by the computer 200. For example, in some embodiments, for the scene in which the user plays a game, the image to be processed can be the game screen of the game application installed on the computer 200. The CPU 201 calculates the position information of the human eye gaze point based on the image including the human eye. Based on the gaze point rendering technology, the GPU 202 renders the image to be processed in different regions according to the position information of the human eye gaze point to obtain sub-image A and sub-image B. The rendering process is similar to the aforementioned rendering process, and will not be repeated here. The computer 200 sends sub-image A and sub-image B to the VR glasses 100, and the VR glasses 100 caches the received sub-image A. As long as one line of sub-image B is received, the line of image can be directly enlarged and combined with sub-image A for image display. Since the buffer in the DDIC 104 of the VR glasses 100 needs to cache all the data of sub-image A, it only needs to cache one line of sub-image B in real time, which can greatly reduce the cache pressure of the buffer and lower the capacity requirement of the buffer.

[0136] Below Fig.14 The display process of the image to be processed is described in detail. Specifically, Fig.14 As shown, the display process of the image to be processed includes:

[0137] Step 1400: The VR glasses 100 and the computer 200 are connected. For example, if the user wants to view pictures through the VR glasses 100, the VR glasses 100 and the computer 200 can be connected through a data cable, a wireless LAN, Bluetooth, etc., so that the VR glasses 100 and the computer 200 can communicate. It can be understood that in other embodiments, the VR glasses 100 and the computer 200 can also be connected in other ways, not limited to the above method.

[0138] Step 1402: The VR glasses 100 sends the image of the human eye captured by the camera to the computer 200, so that the computer 200 performs calculation analysis based on the image of the human eye. For example, the image of the human eye is acquired in real time by an infrared camera, and then the image of the human eye captured by the infrared camera is sent to the computer 200.

[0139] Step 1404: The computer 200 performs calculations based on the received image including the human eye to obtain the gaze point position information of the human eye. The calculation method is similar to step 804 and will not be described in detail here.

[0140] Step 1406: The computer 200 obtains sub-image A and sub-image B based on the obtained gaze point position information of the human eye and the gaze point rendering technology. The rendering process is similar to step 806 and will not be described in detail here.

[0141] Step 1408: The computer 200 sends the sub-image A and the sub-image B to the VR glasses 100. For example, in some embodiments, the computer 200 sends the sub-image A and the sub-image B to the VR glasses 100 via a wireless local area network, Bluetooth, etc., so that the VR glasses 100 can display the images.

[0142] It is understandable that, in some embodiments, the computer 200 may also interlace and integrate the sub-image A and the sub-image B according to a set row ratio in a manner similar to the above step 808 to obtain an integrated image, and then send the integrated image to the VR glasses 100.

[0143] Step 1410: The VR glasses 100 receive sub-image A and sub-image B, and after caching sub-image A, cache one line of sub-image B. For example, in some embodiments, after caching sub-image A, cache the first line of sub-image B. Since the buffer in the DDIC 104 of the VR glasses 100 needs to cache all the data of sub-image A, it only needs to cache one line of sub-image B in real time, which can greatly reduce the buffer pressure and reduce the capacity requirement of the buffer.

[0144] Step 1412: After the VR glasses 100 have cached a row of the sub-image B, the VR glasses 100 magnify the row. For example, an interpolation algorithm is used to magnify a row of the written sub-image B by 3 times.

[0145] Step 1414: The VR glasses 100 combine the enlarged sub-image B with the sub-image A. The combination method is similar to step 816 and will not be described in detail here.

[0146] Step 1416: The timing control circuit of the VR glasses 100 controls the column driving circuit and the row driving circuit to make the pixel array display the combined sub-image A and sub-image B. For example, in some embodiments, the timing control circuit in the DDIC 104 of the VR glasses can control the row driving circuit of the DDIC 104 to select the pixel points of each row of the pixel array of the display screen 107, and then the sub-image A and sub-image B cached in the cache of the DDIC 104 are output on the display screen 107 through the column driving circuit of the DDIC 104 for display.

[0147] Fig.15 Shows Fig.14The display timing diagram of the image to be processed in the embodiment shown. When the GPU 202 of the computer 100 turns on vertical synchronization, the GPU 202 will wait for the V-Sync signal from the VR glasses 100 to be sent before rendering a new frame. Fig.15 It can be seen that when the buffer of DDIC 104 completes caching of sub-image A, as soon as a line of sub-image B is received, the line of sub-image B can be magnified and combined with the corresponding line of sub-image A for display. Figure 4 and Figure 5 The solution shown reduces the requirement on the buffer capacity of DDIC 104 and reduces the size of DDIC 104 .

[0148] Fig.16 According to some embodiments of the present application, a device 1600 for displaying an image is provided, specifically comprising:

[0149] An acquisition module 1602 is used to acquire the image to be processed and the user's gaze point position information, wherein the gaze point position information is used to describe the position where the user's visual center falls on the screen of the electronic device;

[0150] The first processing module 1604 is used to render the image to be processed based on the acquired image to be processed and the gaze point position information to obtain a first image and a second image, wherein the first image corresponds to a first area in the image to be processed, and the second image corresponds to a second area in the image to be processed, the first area is an area within a preset range of the user's gaze point position in the image to be processed, and the second area is an area outside the first area in the image to be processed;

[0151] The second processing module 1606 is used to interlace and integrate each row of pixel points in the first image and each row of pixel points in the second image according to a set row ratio to obtain an integrated image;

[0152] A third processing module 1608 is used to cache at least part of each row of pixel points in the integrated image corresponding to the first image, amplify at least part of each row of pixel points in the integrated image corresponding to the second image, and combine the amplified part of the row of pixel points with the cached part of the row of pixel points to obtain a combined image; and

[0153] The display module 1610 is used to display the combined image.

[0154] Understandably, Fig.16 The image display device 1600 shown corresponds to the image display method provided by the present application, and the technical details in the above specific description of the image display method provided by the present application are still applicable to Fig.16The display device 1600 of the image shown is described above in detail and will not be repeated here.

[0155] The various embodiments of the mechanism disclosed in the present application can be implemented in hardware, software, firmware or a combination of these implementation methods. The embodiments of the present application can be implemented as a computer program or program code executed on a programmable system, which includes at least one processor, a storage system (including volatile and non-volatile memory and / or storage elements), at least one input device and at least one output device.

[0156] Program code can be applied to input instructions to perform the functions described in this application and generate output information. The output information can be applied to one or more output devices in a known manner. For the purposes of this application, a processing system includes any system having a processor such as, for example, a digital signal processor (DSP), a microcontroller, an application specific integrated circuit (ASIC), or a microprocessor.

[0157] Program code can be implemented with high-level programming language or object-oriented programming language to communicate with the processing system. When necessary, program code can also be implemented with assembly language or machine language. In fact, the mechanism described in this application is not limited to the scope of any specific programming language. In either case, the language can be a compiled language or an interpreted language.

[0158] In some cases, the disclosed embodiments may be implemented in hardware, firmware, software, or any combination thereof. The disclosed embodiments may also be implemented as instructions carried or stored on one or more temporary or non-temporary machine-readable (e.g., computer-readable) storage media, which may be read and executed by one or more processors. For example, instructions may be distributed over a network or through other computer-readable media. Therefore, machine-readable media may include any mechanism for storing or transmitting information in a machine (e.g., computer) readable form, including, but not limited to, floppy disks, optical disks, optical disks, read-only memories (CD-ROMs), magneto-optical disks, read-only memories (ROM), random access memories (RAM), erasable programmable read-only memories (EPROM), electrically erasable programmable read-only memories (EEPROM), magnetic cards or optical cards, flash memory, or tangible machine-readable memories for transmitting information (e.g., carrier waves, infrared signal digital signals, etc.) using the Internet in electrical, optical, acoustic, or other forms of propagation signals. Accordingly, machine-readable media include any type of machine-readable media suitable for storing or transmitting electronic instructions or information in a form readable by a machine (eg, a computer).

[0159] In the accompanying drawings, some structural or method features may be shown in a specific arrangement and / or order. However, it should be understood that such a specific arrangement and / or order may not be required. Instead, in some embodiments, these features may be arranged in a manner and / or order different from that shown in the illustrative drawings. In addition, the inclusion of structural or method features in a particular figure does not mean that such features are required in all embodiments, and in some embodiments, these features may not be included or may be combined with other features.

[0160] It should be noted that the units / modules mentioned in the various device embodiments of the present application are all logical units / modules. Physically, a logical unit / module can be a physical unit / module, or a part of a physical unit / module, or can be implemented as a combination of multiple physical units / modules. The physical implementation method of these logical units / modules themselves is not the most important. The combination of functions implemented by these logical units / modules is the key to solving the technical problems proposed by the present application. In addition, in order to highlight the innovative part of the present application, the above-mentioned device embodiments of the present application do not introduce units / modules that are not closely related to solving the technical problems proposed by the present application, which does not mean that there are no other units / modules in the above-mentioned device embodiments.

[0161] It should be noted that in the examples and description of this patent, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "including one" do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0162] Although the present application has been illustrated and described with reference to certain preferred embodiments thereof, it will be apparent to those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the present application.

Claims

1. A method for displaying an image, used in an electronic device as a display terminal, characterized in that: include: Acquire an image to be processed and user's gaze point position information, wherein the gaze point position information is used to describe the position where the user's visual center falls on the screen of the electronic device; Based on the acquired image to be processed and the gaze point position information, the image to be processed is rendered to obtain a first image and a second image, wherein the first image corresponds to a first area in the image to be processed, and the second image corresponds to a second area in the image to be processed, the first area is an area within a preset range of the user's gaze point position in the image to be processed, and the second area is an area outside the first area in the image to be processed; Perform interlaced integration of a rows of pixels in the first image as a group and b rows of pixels in the second image as a group to obtain an integrated image, wherein a:b=X, a>b, and a and b are positive integers; caching at least part of the rows of pixel points in each row of pixel points in the integrated image corresponding to the first image, amplifying at least part of the rows of pixel points in each row of pixel points in the integrated image corresponding to the second image, and combining the amplified part of the rows of pixel points with the cached part of the rows of pixel points to obtain a combined image; and The combined image is displayed.

2. The method according to claim 1, characterized in that The first image and the second image have the same resolution.

3. The method according to claim 1, characterized in that: The step of caching at least a portion of the rows of pixel points in each row of pixel points in the integrated image corresponding to the first image, amplifying at least a portion of the rows of pixel points in each row of pixel points in the integrated image corresponding to the second image, and combining the amplified portion of the rows of pixel points with the cached portion of the rows of pixel points to obtain the combined image comprises: Buffering each a row of pixels corresponding to the first image in the integrated image, and amplifying each b row of pixels corresponding to the second image in the integrated image by Y times; Reading a row of pixels of the first image currently cached, and combining the amplified b row of pixels in the second image with the a row of pixels currently cached in the read first image to obtain a combined image.

4. The method according to claim 3, characterized in that The value of X is equal to the value of Y.

5. The method according to claim 3, characterized in that: The resolution of the second image is the same as the resolution of the combined image after being magnified Y times.

6. The method according to claim 1 or 2, characterized in that: The step of caching at least a portion of the rows of pixel points in each row of pixel points in the integrated image corresponding to the first image, amplifying at least a portion of the rows of pixel points in each row of pixel points in the integrated image corresponding to the second image, and combining the amplified portion of the rows of pixel points with the cached portion of the rows of pixel points to obtain the combined image comprises: compressing each row of pixel points corresponding to the first image in the integrated image and each row of pixel points corresponding to the second image in the integrated image, wherein a compression ratio corresponding to each row of pixel points in the first image is greater than a compression ratio corresponding to each row of pixel points in the second image; caching at least a portion of the compressed rows of pixel points corresponding to the first image and at least a portion of the compressed rows of pixel points corresponding to the second image; Reading at least a portion of the compressed rows of pixel points corresponding to the first image and at least a portion of the compressed rows of pixel points corresponding to the second image that are currently cached, and decompressing the read at least a portion of the compressed rows of pixel points corresponding to the first image and at least a portion of the compressed rows of pixel points corresponding to the second image that are currently cached; The decompressed at least part of the rows of pixel points corresponding to the second image are magnified S times and then combined with the decompressed at least part of the rows of pixel points corresponding to the first image to obtain a combined image.

7. The method according to claim 6, characterized in that The resolution of the second image is magnified S times to be the same as the resolution of the combined image.

8. The method according to claim 1 or 2, characterized in that: The step of caching at least a portion of the rows of pixel points in each row of pixel points in the integrated image corresponding to the first image, amplifying at least a portion of the rows of pixel points in each row of pixel points in the integrated image corresponding to the second image, and combining the amplified portion of the rows of pixel points with the cached portion of the rows of pixel points to obtain the combined image comprises: Cache each row of pixel points in the integrated image corresponding to the first image and cache one row of pixel points in the integrated image corresponding to the second image; Reading a portion of the rows of pixels corresponding to the first image and one row of pixels corresponding to the second image that are currently cached, and amplifying the read row of pixels corresponding to the second image by T times; The enlarged row of pixel points corresponding to the second image is combined with the read partial row of pixel points corresponding to the first image currently cached to obtain a combined image.

9. The method according to claim 8, characterized in that The resolution of the second image is the same as the resolution of the combined image after being magnified T times.

10. An image display device, characterized in that: include: An acquisition module, used to acquire the image to be processed and the user's gaze point position information, wherein the gaze point position information is used to describe the position where the user's visual center falls on the screen of the electronic device; A first processing module is used to render the image to be processed based on the acquired image to be processed and the gaze point position information to obtain a first image and a second image, wherein the first image corresponds to a first area in the image to be processed, and the second image corresponds to a second area in the image to be processed, the first area is an area within a preset range of the user's gaze point position in the image to be processed, and the second area is an area outside the first area in the image to be processed; A second processing module is used to perform interlaced integration of each a row of pixel points in the first image as a group and each b row of pixel points in the second image as a group to obtain an integrated image, wherein a:b=X, a>b, and a and b are positive integers; a third processing module, configured to cache at least a portion of the rows of pixel points in each row of pixel points in the integrated image corresponding to the first image, amplify at least a portion of the rows of pixel points in each row of pixel points in the integrated image corresponding to the second image, and combine the amplified portion of the rows of pixel points with the cached portion of the rows of pixel points to obtain a combined image; and The display module is used to display the combined image.

11. A readable medium, characterized in that: The readable medium stores instructions, which, when executed on an electronic device, enable the electronic device to execute the image display method according to any one of claims 1 to 9.

12. An electronic device, characterized in that: include: a memory for storing instructions to be executed by one or more processors of the electronic device, and The processor is one of the processors of the electronic device, and is used to execute the image display method described in any one of claims 1-9.

Images