Resolution adjusting method, device and equipment based on eye tracking technology

Abstract

The invention discloses a resolution adjustment method, device and equipment based on an eye tracking technology. The bandwidth can be reduced while the definition is improved. The resolution adjustment method based on the eye tracking technology comprises the following steps: acquiring real-time eye tracking data based on the eye tracking technology; dividing a sight focus area and a non-focus area of the user based on a preset area range and the real-time eye tracking data; and rendering the sight focus area and the non-focus area according to different resolutions.

Description

Technical Field

[0001] The present application relates to the field of video processing technology, and in particular to a resolution adjustment method, device and equipment based on eye tracking technology. Background Art

[0002] The rapid development of virtual reality (VR) technology has brought an immersive experience to users, but the accompanying demand for high-resolution video streaming has also put forward higher requirements for wireless transmission bandwidth. In order to maintain high-quality visual effects, traditional VR video streaming usually needs to transmit very high-resolution images, resulting in excessive bandwidth usage. Existing VR streaming technology usually saves bandwidth by reducing video resolution when bandwidth is limited, but this will seriously affect the user experience, especially in the area where the user's line of sight is focused, where the impact of reduced resolution on perception is particularly obvious. Summary of the invention

[0003] In order to solve the above technical problems, the present application is proposed. The embodiments of the present application provide a method, device and apparatus for adjusting resolution based on eye tracking technology, which can reduce bandwidth while improving definition.

[0004] According to a first aspect of the present application, a resolution adjustment method based on eye tracking technology is provided, comprising: obtaining real-time eye tracking data based on eye tracking technology; dividing the user's line of sight focus area and non-focus area based on a preset area range and the real-time eye tracking data; rendering the line of sight focus area and the non-focus area at different resolutions.

[0005] As a possible implementation method, the line of sight focus area and the non-focus area are rendered at different resolutions at the same time, including: rendering the image of the line of sight focus area according to a first preset resolution; rendering the image of the non-focus area according to a second preset resolution; wherein the first preset resolution is higher than the second preset resolution.

[0006] As a possible implementation manner, the second preset resolution includes an original resolution; wherein rendering the picture of the non-focus area according to the second preset resolution includes: rendering the picture of the non-focus area according to the original resolution.

[0007] As a possible implementation method, after acquiring real-time eye tracking data based on eye tracking technology, the resolution adjustment method based on eye tracking technology includes: predicting the eye movement area based on real-time eye tracking data and historical eye tracking data; wherein, based on a preset area range and the real-time eye tracking data, the user's line of sight is divided into a focus area and a non-focus area, and also includes: based on the eye movement area and the preset area range, the user's line of sight is divided into a focus area and a non-focus area.

[0008] As a possible implementation method, the preset area range includes the range size of the line of sight focus area, and based on the preset area range and the real-time eye tracking data, the user's line of sight focus area and non-focus area are divided, including: based on the real-time eye tracking data, determining the user's eye center gaze position; with the eye center gaze position as the center point, determining the user's line of sight focus area based on the range size of the line of sight focus area, and other areas except the line of sight focus area are non-focus areas.

[0009] As a possible implementation manner, rendering the sight focus area and the non-focus area according to different resolutions also includes: aligning the boundaries of the sight focus area and the non-focus area during rendering.

[0010] As a possible implementation method, real-time eye tracking data is obtained based on eye tracking technology, including: obtaining eye images collected by a camera or an infrared sensor; extracting features related to eye movement through the eye images to obtain eye movement data; and processing the eye movement data in real time to obtain continuous real-time eye tracking data.

[0011] According to a second aspect of the present application, a resolution adjustment device based on eye tracking technology is provided, including: an acquisition module for acquiring real-time eye tracking data based on eye tracking technology; a division module for dividing the user's line of sight focus area and non-focus area based on a preset area range and the real-time eye tracking data; a rendering module for rendering the line of sight focus area and the non-focus area at different resolutions.

[0012] According to a third aspect of the present application, a device is provided, comprising: a resolution adjustment device based on eye tracking technology as described in the second aspect; the resolution adjustment device based on eye tracking technology is communicatively connected to eye tracking hardware, and the eye tracking hardware is used to collect real-time eye tracking data.

[0013] As a possible implementation manner, the device includes a virtual reality device or a mobile device.

[0014] The resolution adjustment method, device and equipment based on eye tracking technology provided by the present application render the focus area and non-focus area at different resolutions, saving more bandwidth resources than traditional global high-definition rendering in terms of bandwidth occupation, and under the same definition improvement requirements, the use of different resolution rendering can achieve the visual effect of high-bandwidth global rendering through low bandwidth. Through this differentiated processing, the user's visual experience is greatly improved, while reducing the device's requirements for wireless transmission bandwidth. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] By describing the embodiments of the present application in more detail in conjunction with the accompanying drawings, the above and other purposes, features and advantages of the present application will become more apparent. The accompanying drawings are used to provide a further understanding of the embodiments of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the present application and do not constitute a limitation of the present application. In the accompanying drawings, the same reference numerals generally represent the same components or steps.

[0016] Figure 1 It is a flowchart of a resolution adjustment method based on eye tracking technology provided by an exemplary embodiment of the present application.

[0017] Figure 2 It is a structural schematic diagram of a resolution adjustment device based on eye tracking technology provided by an exemplary embodiment of the present application.

[0018] Figure 3 It is a structural diagram of an electronic device provided by an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0019] Below, the exemplary embodiments according to the present application will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application, and it should be understood that the present application is not limited to the exemplary embodiments described here.

[0020] The rapid development of virtual reality (VR) technology has brought an immersive experience to users, but the accompanying demand for high-resolution video streaming has also put forward higher requirements for wireless transmission bandwidth. In order to maintain high-quality visual effects, traditional VR video streaming usually needs to transmit very high-resolution images, resulting in excessive bandwidth usage. Existing VR streaming technology usually saves bandwidth by reducing video resolution when bandwidth is limited, but this will seriously affect the user experience, especially in the area where the user's line of sight is focused, where the impact of reduced resolution on perception is particularly obvious.

[0021] The visual perception of the human eye is not globally consistent. Only in the area where the user is directly looking does the human eye need a higher resolution, while in the surrounding areas, the human eye is insensitive to resolution changes. Therefore, based on this visual characteristic, this application proposes a resolution adjustment method, device and equipment based on eye tracking technology, which combines eye tracking technology to perform regional resolution control, collect the user's eye tracking data in real time, identify the area the user is currently looking at, and render the area in high definition, while rendering the surrounding unlooked-at area at a lower resolution, thereby improving the user's visual experience under limited bandwidth resources.

[0022] Eye tracking technology Eye tracking technology mainly tracks the changes in the features of the eyeball and its surroundings, such as the reflection of the sclera and iris edge, the position and movement of the pupil, etc. These feature changes will generate information that can be extracted. Computers can extract these features through image capture or scanning, thereby tracking the changes in the eyes in real time. The specific implementation of eye tracking technology mainly depends on eye tracking hardware. Eye tracking hardware includes devices with eye tracking functions, such as professional eye trackers or cameras with integrated eye tracking technology. Eye images are captured by the device, and image processing algorithms analyze these images to extract features related to eye movement. Using image processing and data collection technology, eye movement data is extracted from the captured eye images. These data may include pupil position, iris edge position, eye rotation angle, etc. These data are processed in real time to generate continuous eye tracking information. At present, eye tracking hardware can track the eyeball in real time and has a certain function of predicting the future movement position of the eyeball. The resolution adjustment method, device and equipment based on eye tracking technology provided in the present application can rely on this eye tracking technology and eye tracking hardware. In the device, the resolution adjustment device based on eye tracking technology is connected to the eye tracking hardware for communication, and the real-time eye tracking data collected from the eye tracking hardware is read and used for area division and resolution adjustment.

[0023] Based on the problem of high global rendering bandwidth requirements, a resolution adjustment method based on eye tracking technology is proposed below. Figure 1 is a flowchart of a method for adjusting resolution based on eye tracking technology provided by an exemplary embodiment of the present application. Figure 1 For example, firstly, based on the eye tracking technology, real-time eye tracking data is obtained (reference Figure 1 Then, based on the preset area range and the real-time eye tracking data, the user's sight focus area and non-focus area are divided (reference Figure 1 S200 in ), that is, determining the screen area currently being gazed at by the user, and finally rendering the focus area and the non-focus area at different resolutions (reference Figure 1S300 in the figure), for example, a high-resolution rendering is used for the screen area currently being gazed at by the user to ensure that the visual area being gazed at by the user obtains a clear picture, while a low-resolution rendering is used for other areas to reduce bandwidth requirements.

[0024] Combined with the following Figure 1 , a more detailed introduction is given to the resolution adjustment method based on eye tracking technology provided in an embodiment of the present application.

[0025] In S100, real-time eye tracking data is obtained based on eye tracking technology. Eye tracking technology mainly tracks changes in the features of the eyeball and its surroundings, such as reflections of the sclera and iris edges, and the position and movement of the pupil. These feature changes will generate information that can be extracted, and the computer can extract these features through image capture or scanning, thereby tracking changes in the eyes in real time. Eye tracking technology can be used to locate the area where the user's eyes are looking, thereby determining the area to be rendered.

[0026] In some embodiments, S100 may be implemented by: acquiring eye images captured by a camera or an infrared sensor; extracting features related to eye movement through the eye images to obtain eye movement data; and processing the eye movement data in real time to obtain continuous real-time eye tracking data.

[0027] When using eye tracking technology to collect user eye movement data, the position and movement of the user's eyes can be monitored through a camera or infrared sensor, such as by capturing eye images, and then analyzing these images through image processing algorithms to extract features related to eye movement, including pupil position, iris edge position, eye rotation angle, etc., and process these data in real time to generate continuous eye tracking information. After determining the sight focus area through real-time eye tracking data, when the eye moves, the sight focus area will also move with it, and the sight focus area will be determined in real time following the user's eye movements.

[0028] For example, the screen resolution is 1000×1000, and the focus area is set to 500×500. When the user looks at the upper left corner of the screen, the 500×500 area in the upper left corner of the screen is divided into the visual focus area, and the other areas are divided into non-focus areas.

[0029] In some embodiments, since the eye movement speed is fast and the eye movement amplitude is not large, in order to ensure the user's visual experience, eye movement prediction can be introduced to render the area that the user is about to look at in advance to ensure the overall continuity of the picture. For example, based on real-time eye tracking data and historical eye tracking data, the eye movement area is predicted; based on the eye movement area and the preset area range, the user's line of sight focus area and non-focus area are divided. Through real-time eye tracking data and historical eye tracking data, the user's eye movement path in the process can be analyzed, including the area of ​​gaze, the order of gaze, the length of gaze dwell time, and whether the vision is smooth, so that certain predictions can be made based on historical eye tracking data. For example, when the eye tracking hardware continuously records the trajectory of eye movement, when the eye is at one-half of the screen, but the eye continues to move to the right, the area slightly to the right of one-half (for example, four-thirds) is predicted as the line of sight focus area, and other areas are non-focus areas.

[0030] In S200, based on the preset area range and the real-time eye tracking data, the user's sight focus area and the non-focus area are divided. The area range of the sight focus area is preset, and after determining the position where the user's eyes are looking at in real time, the sight focus area is divided according to the preset area range.

[0031] For example, the preset area range includes the range size of the sight focus area, and the user's eye center gaze position is determined based on real-time eye tracking data; the user's sight focus area is determined based on the range size of the sight focus area with the eye center gaze position as the center point, and other areas except the sight focus area are non-focus areas. Based on the eye tracking hardware, the screen position that the user is currently looking at can be directly determined, and combined with the pre-set range size, the division of the sight focus area can be completed. Other areas are classified as non-focus areas.

[0032] In S300, the focus area and the non-focus area are rendered at different resolutions. For the focus area, high resolution rendering can be used to ensure that the user obtains a clear picture in the visual area being gazed. For the area not being gazed by the user, a lower resolution rendering is used to reduce bandwidth consumption while ensuring the overall coherence of the picture.

[0033] In some embodiments, an implementation of S300 may be: rendering the image in the sight focus area according to a first preset resolution; rendering the image in the non-focus area according to a second preset resolution; wherein the first preset resolution is higher than the second preset resolution.

[0034] Taking the resolution of the left and right eyes as an example, assuming that the traditional global high-definition rendering requires the left and right eyes to reach 2000×1000 pixels respectively, if you want to double the clarity, you usually need to increase the resolution to 4000×2000 pixels, which will double the bandwidth demand. However, if the focus area and non-focus area are rendered at different resolutions, for example, the focus area that the user is looking at is rendered at high resolution, the resolution only needs to be increased to 2000×2000 pixels to achieve the same visual effect as the global rendering of 4000×2000 pixels. In this way, by precisely controlling the focus area, the system significantly reduces the transmission bandwidth while ensuring image clarity. For example, a 1000×1000 picture needs to render a low-resolution area at a height of 1000×200, then use 200 to render the non-focus area, and the remaining 800 to render the clear area, that is, the focus area.

[0035] The first preset resolution and the second preset resolution can be freely set according to the required clarity. The non-focus area can be rendered at a low resolution, or the original resolution can be maintained on the original basis when the clarity is required to be improved. For example, the second preset resolution can include the original resolution. When the overall clarity is improved, the picture of the non-focus area is rendered according to the original resolution. Alternatively, when the overall picture needs to maintain the original clarity and reduce the bandwidth demand, the first preset resolution maintains the original resolution and the second preset resolution is lowered. The first preset resolution and the second preset resolution can be reduced or increased at the same time. For example, assuming that the bandwidth required for high-definition global rendering is 100%, the bandwidth demand will increase to 400% when the clarity is doubled. However, if the sight focus area and non-focus area are rendered at different resolutions, the resolution of the focus area is doubled, and the non-focus area maintains the original resolution, the overall bandwidth demand can be reduced to 200%. Compared with the solution of global resolution improvement, the bandwidth demand is reduced by nearly 50%. Moreover, under the same definition improvement demand, the traditional global high-definition rendering needs to increase the bandwidth to 400%, while the solution of the present invention only needs 200% of the bandwidth to achieve the same visual effect.

[0036] In some embodiments, during rendering, the boundaries of the sight focus area and the non-focus area are aligned. That is, during the rendering process, in order to ensure the coherence and consistency of the video stream, the boundaries of the sight focus area and the non-focus area can be visually kept smoothly transitioned to avoid obvious visual breaks or jumps due to resolution differences. Since the sight focus area is rendered at high resolution, the user can perceive a clearer picture when watching, while the non-focus area uses low resolution, and the user can hardly perceive the difference during actual viewing. Through this differentiated processing, the user's visual experience is greatly improved, while the device's requirements for wireless transmission bandwidth are reduced.

[0037] The resolution adjustment method based on eye tracking technology provided above can be applied to virtual reality (VR) devices, and can also be applied to mobile devices that are limited by bandwidth and hardware performance, so that a wider range of users can obtain high-quality VR (virtual reality) experience under wireless networks. Traditional high-resolution global rendering will cause greater bandwidth pressure, which is prone to increased latency and stuttering. Applying the resolution adjustment method based on eye tracking technology provided above to virtual reality devices (or mobile devices) can reduce the overall data volume, thereby improving the real-time and smoothness of VR (or mobile device) wireless transmission, and avoiding poor experience caused by insufficient bandwidth. In addition, since the system no longer needs to perform high-definition rendering of the global picture, the computing load of the virtual reality device (or mobile device) can be significantly reduced, thereby reducing the power consumption and heating problems of the hardware, while extending the battery life of the virtual reality device (or mobile device).

[0038] Figure 2 is a schematic diagram of a resolution adjustment device based on eye tracking technology provided by an exemplary embodiment of the present application. Figure 2 As shown, the resolution adjustment device 2 based on eye tracking technology includes: an acquisition module 21, used to acquire real-time eye tracking data based on eye tracking technology; a division module 22, used to divide the user's line of sight focus area and non-focus area based on a preset area range and real-time eye tracking data; a rendering module 23, used to render the line of sight focus area and non-focus area according to different resolutions.

[0039] The resolution adjustment device based on eye tracking technology provided by the present application renders the focus area and non-focus area at different resolutions, which saves more bandwidth resources than the traditional global high-definition rendering in terms of bandwidth occupation. Moreover, under the same definition improvement requirements, the use of different resolution rendering can achieve the visual effect of high-bandwidth global rendering through low bandwidth. Through this differentiated processing, the user's visual experience is greatly improved, while reducing the device's requirements for wireless transmission bandwidth.

[0040] As a possible implementation method, the rendering module 23 can be configured to: render the image of the line of sight focus area according to a first preset resolution; render the image of the non-focus area according to a second preset resolution; wherein the first preset resolution is higher than the second preset resolution.

[0041] As a possible implementation manner, the second preset resolution includes the original resolution; wherein the rendering module 23 may also be configured to render the image of the non-focus area according to the original resolution.

[0042] As a possible implementation method, the resolution adjustment device 2 based on eye tracking technology can be configured to: predict the eye movement area based on real-time eye tracking data and historical eye tracking data; wherein the division module 22 can be configured to: divide the user's line of sight into focus area and non-focus area based on the eye movement area and the preset area range.

[0043] As a possible implementation method, the preset area range includes the range size of the line of sight focus area, and the division module 22 can be configured as: based on real-time eye tracking data, determining the user's eye center gaze position; with the eye center gaze position as the center point, determining the user's line of sight focus area based on the range size of the line of sight focus area, and other areas except the line of sight focus area are non-focus areas.

[0044] As a possible implementation manner, the rendering module 23 may be configured to align the boundaries of the sight focus area and the non-focus area during rendering.

[0045] As a possible implementation method, the acquisition module 21 can be configured to: acquire eye images captured by a camera or infrared sensor; extract features related to eye movement through the eye images to obtain eye movement data; and process the eye movement data in real time to obtain continuous real-time eye tracking data.

[0046] Exemplary Electronic Devices

[0047] An electronic device, the electronic device comprising: a processor; a memory for storing instructions executable by the processor; and the processor, for executing the resolution adjustment method based on eye tracking technology described in the embodiment provided in the present application.

[0048] Below, reference Figure 3 The electronic device according to the embodiment of the present application is described. The electronic device may be any one or both of the first device and the second device, or a stand-alone device independent of them, and the stand-alone device may communicate with the first device and the second device to receive the collected input signal from them.

[0049] Figure 3 A block diagram of an electronic device according to an embodiment of the present application is illustrated.

[0050] like Figure 3 As shown, the electronic device 10 includes one or more processors 11 and a memory 12 .

[0051] The processor 11 may be a central processing unit (CPU) or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 10 to perform desired functions.

[0052] The memory 12 may include one or more computer program products, and the computer program product may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory (cache), etc. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 11 may run the program instructions to implement the resolution adjustment method based on eye tracking technology and / or other desired functions of the various embodiments of the present application described above. Various contents such as input signals, signal components, noise components, etc. may also be stored in the computer-readable storage medium.

[0053] In one example, the electronic device 10 may further include: an input device 13 and an output device 14, and these components are interconnected via a bus system and / or other forms of connection mechanisms (not shown).

[0054] When the electronic device is a stand-alone device, the input device 13 may be a communication network connector, which is used to receive the collected input signals from the first device and the second device.

[0055] In addition, the input device 13 may also include, for example, a keyboard, a mouse, and the like.

[0056] The output device 14 can output various information to the outside, including the determined distance information, direction information, etc. The output device 14 can include, for example, a display, a speaker, a printer, a communication network and a remote output device connected thereto, and the like.

[0057] Of course, to simplify, Figure 3 Only some of the components related to the present application in the electronic device 10 are shown, and components such as a bus, an input / output interface, etc. are omitted. In addition, according to specific application situations, the electronic device 10 may also include any other appropriate components.

[0058] The computer program product may be written in any combination of one or more programming languages ​​to write program codes for performing the operations of the embodiments of the present application, including object-oriented programming languages, such as Java, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as an independent software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0059] A computer-readable storage medium stores a computer program, and the computer program is used to execute the resolution adjustment method based on eye tracking technology described in the embodiment provided in the present application.

[0060] The computer readable storage medium can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can include, for example, but is not limited to, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0061] The above description has been given for the purpose of illustration and description. In addition, this description is not intended to limit the embodiments of the present application to the forms disclosed herein. Although multiple example aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, changes, additions and sub-combinations thereof.

Claims

1. A resolution adjustment method based on eye tracking technology, characterized in that: include: Based on eye tracking technology, obtain real-time eye tracking data; Divide the user's sight focus area and non-focus area based on the preset area range and the real-time eye tracking data; The sight focus area and the non-focus area are rendered according to different resolutions.

2. The method for adjusting the resolution based on eye tracking technology according to claim 1, characterized in that: Rendering the sight focus area and the non-focus area at different resolutions simultaneously includes: Rendering the image of the sight focus area according to a first preset resolution; Rendering the image of the non-focus area according to a second preset resolution; The first preset resolution is higher than the second preset resolution.

3. The resolution adjustment method based on eye tracking technology according to claim 1, characterized in that: The second preset resolution includes the original resolution; The step of rendering the image of the non-focus area according to the second preset resolution includes: The image of the non-focus area is rendered according to the original resolution.

4. The method for adjusting the resolution based on eye tracking technology according to claim 1, characterized in that: After obtaining real-time eye tracking data based on the eye tracking technology, the resolution adjustment method based on the eye tracking technology includes: Predict eye movement areas based on real-time eye tracking data and historical eye tracking data; Wherein, based on the preset area range and the real-time eye tracking data, dividing the user's sight focus area and non-focus area also includes: Based on the eye movement area and the preset area range, the user's sight focus area and non-focus area are divided.

5. The method for adjusting resolution based on eye tracking technology according to claim 1, characterized in that: The preset area range includes the range size of the sight focus area. Based on the preset area range and the real-time eye tracking data, the user's sight focus area and non-focus area are divided, including: Determining the user's eye center gaze position based on the real-time eye tracking data; Taking the eyeball center gaze position as the center point, the user's sight focus area is determined based on the range size of the sight focus area, and other areas except the sight focus area are non-focus areas.

6. The method for adjusting resolution based on eye tracking technology according to claim 1, characterized in that: Rendering the sight focus area and the non-focus area according to different resolutions also includes: During rendering, the boundaries of the sight focus area and the non-focus area are aligned.

7. The method for adjusting resolution based on eye tracking technology according to claim 1, characterized in that: Based on eye tracking technology, obtain real-time eye tracking data, including: Obtaining eye images collected by a camera or infrared sensor; Extracting features related to eye movement through the eye image to obtain eye movement data; The eye movement data is processed in real time to obtain continuous real-time eye tracking data.

8. A resolution adjustment device based on eye tracking technology, characterized in that: include: An acquisition module, used for acquiring real-time eye tracking data based on eye tracking technology; A division module, used to divide the user's sight focus area and non-focus area based on a preset area range and the real-time eye tracking data; A rendering module is used to render the sight focus area and the non-focus area according to different resolutions.

9. A device, characterized in that the device include: A resolution adjustment device based on eye tracking technology as claimed in claim 8; The eye tracking technology-based resolution adjustment device is in communication with eye tracking hardware, and the eye tracking hardware is used to collect real-time eye tracking data.

10. The device according to claim 9, characterized in that The device includes a virtual reality device or a mobile device.

Images