Image rendering method and device, electronic equipment, storage medium and program product

Abstract

The invention relates to an image rendering method and device, electronic equipment, a storage medium and a program product. The image rendering method comprises: receiving fixation point information sent by a glasses device, the fixation point information being information formed by a user wearing the glasses device watching a display screen in a display device; based on the fixation point information, a display area of the display screen is divided into a fixation area and a non-fixation area, the fixation area is an area where the user watches the display screen, and the non-fixation area is an area outside the fixation area in the display area; respectively rendering the rendering objects of the gazing area and the non-gazing area based on different resolutions; wherein the resolution of the fixation area after rendering is greater than the resolution of the non-fixation area after rendering. Through the embodiment of the invention, the rendering consumption can be reduced under the condition that the user experience is not influenced.

Description

Technical Field

[0001] This disclosure relates to the field of graphics rendering, and in particular to image rendering methods and apparatus, electronic devices, storage media, and program products. Background Technology

[0002] Foveative rendering technology leverages the perceptual characteristics of human vision to achieve excellent visual quality with low computational and power consumption. Specifically, when observing, human vision does not obtain an image that is equally clear across the entire field of vision; instead, it captures the clearest image near the fovea center, while acquiring lower-resolution images at the periphery, with the image becoming increasingly blurry towards the edges. However, current foveative rendering technology faces limitations in its application to extended reality devices. Summary of the Invention

[0003] To overcome the problems existing in related technologies, this disclosure provides an image rendering method and apparatus, electronic device, storage medium, and program product that can reduce rendering consumption without affecting user experience.

[0004] According to a first aspect of the present disclosure, an image rendering method is provided, comprising:

[0005] The system receives gaze point information sent by the glasses device, wherein the gaze point information is information formed on the display screen of the display device by the user wearing the glasses device.

[0006] Based on the gaze point information, the display area of ​​the display screen is divided into a gaze area and a non-gaze area. The gaze area is the area where the user gazes at the display screen, and the non-gaze area is the area outside the gaze area in the display area.

[0007] The rendering objects in the gaze region and the non-gaze region are rendered separately based on different resolutions;

[0008] The resolution of the gaze region after rendering is greater than the resolution of the non-gaze region after rendering.

[0009] In some embodiments, the method further includes:

[0010] When there are multiple non-focused regions, the rendering objects of the multiple non-focused regions are rendered separately based on different resolutions;

[0011] The resolution of the non-focused region after rendering is negatively correlated with the distance from the outer edge of the non-focused region to the focused region.

[0012] In some embodiments, the gaze point information includes the coordinates of the gaze point formed on the display screen by the user gazing at the display screen;

[0013] The step of dividing the display area of ​​the screen into a fixational area and a non-fixational area based on the fixational point information includes:

[0014] Based on the coordinates of the gaze point, the display area of ​​the display screen is divided into the gaze area and the non-gaze area.

[0015] In some embodiments, dividing the display area of ​​the display screen into the gaze region and the non-gaze region based on the coordinates of the gaze point includes:

[0016] Based on the coordinates of the gaze point, determine the object the user is gazing at on the display screen;

[0017] The region enclosed by the outline of the object being gazed upon is defined as the gaze region;

[0018] The area outside the gaze area in the display area is defined as the non-gaze area.

[0019] In some embodiments, dividing the display area of ​​the display screen into a gaze region and a non-gaze region based on the coordinates of the gaze point includes:

[0020] The display area is divided into multiple display sub-areas;

[0021] Based on the distances from the outer edges of the multiple display sub-regions to the coordinates of the gaze point, the display area of ​​the display screen is divided into the gaze area and the non-gaze area;

[0022] The display sub-regions whose distance is less than a preset distance threshold are defined as the gaze region;

[0023] The display sub-regions whose distance is greater than or equal to the preset distance threshold are designated as the non-focused regions.

[0024] In some embodiments, rendering the rendering objects of the gaze region and the non-gaze region separately based on different resolutions includes:

[0025] The rendering object in the gaze area is rendered based on a preset first resolution;

[0026] The rendering object in the non-focused region is rendered based on a preset second resolution;

[0027] Wherein, the first resolution is the maximum resolution that the display screen can support;

[0028] The second resolution is smaller than the first resolution.

[0029] According to a second aspect of the present disclosure, an image rendering method is provided, comprising:

[0030] The first data is obtained by capturing the display device that the user is looking at using the color camera of the glasses device;

[0031] The user's eyes are captured by the eye-tracking camera of the glasses device to obtain second data.

[0032] Based on the first collected information and the second collected information, determine the gaze point information of the user gazing at the display screen in the display device;

[0033] Send the gaze point information to the display device.

[0034] In some embodiments, determining the gaze point information of the user gazing at the display screen in the display device based on the first acquired information and the second acquired information includes:

[0035] The position of the display screen relative to the user is determined based on the first collected information;

[0036] The user's gaze direction is determined based on the second collected information;

[0037] The coordinates of the gaze point are determined based on the position of the display screen relative to the user and the gaze direction.

[0038] According to a third aspect of the present disclosure, an image rendering apparatus is provided, comprising:

[0039] The receiving module is configured to receive gaze point information sent by the glasses device, wherein the gaze point information is information formed on the display screen of the display device by the user wearing the glasses device.

[0040] The segmentation module is configured to divide the display area of ​​the display screen into a gaze area and a non-gaze area based on the gaze point information. The gaze area is the area where the user gazes at the display screen, and the non-gaze area is the area outside the gaze area in the display area.

[0041] The first rendering module is configured to render the rendering objects of the foveated region and the non-foveated region separately based on different resolutions; wherein, the resolution of the foveated region after rendering is greater than the resolution of the non-foveated region after rendering.

[0042] According to a fourth aspect of the present disclosure, an image rendering apparatus is provided, comprising:

[0043] The first acquisition module is configured to acquire first acquisition information by using the color camera of the glasses device to capture the display device that the user is looking at;

[0044] The second acquisition module is configured to acquire second acquisition information by acquiring data from the user's eyes through the eye-tracking camera of the glasses device;

[0045] The gaze determination module is configured to determine the gaze point information of the user gazing at the display screen of the display device based on the first collected information and the second collected information;

[0046] The sending module is configured to send the gaze point information to the display device.

[0047] According to a fifth aspect of the present disclosure, an electronic device is provided, the electronic device comprising:

[0048] processor;

[0049] Memory used to store computer programs or instructions;

[0050] The processor executes the computer program or instructions to implement the steps of the methods described in the first and second aspects.

[0051] According to a sixth aspect of the present disclosure, a non-transitory computer-readable storage medium is provided, the storage medium storing a computer program or instructions that, when executed by a processor, implement the steps of the methods in the first and second aspects.

[0052] According to a seventh aspect of the present disclosure, a computer program product is provided, wherein when the computer program or instructions are executed by a processor, the steps of the methods in the first and second aspects are implemented.

[0053] The technical solutions provided by the embodiments of this disclosure may include the following beneficial effects:

[0054] The display device of this disclosure, through interaction with glasses, can take into account the human eye's focus of visual attention based on gaze point information. Instead of rendering all display areas of the screen at the same resolution, it renders the gaze area and non-gaze area separately at different resolutions, resulting in a higher resolution for the rendered gaze area than for the rendered non-gaze area. This allows for the allocation of rendering resources according to the user's gaze, reducing rendering overhead and improving application frame rates without affecting user experience.

[0055] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0056] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0057] Figure 1 This is a flowchart of an image rendering method according to an exemplary embodiment. Figure 1 .

[0058] Figure 2 This is a schematic diagram illustrating a scenario where a user is looking at the display screen of a display device according to an exemplary embodiment.

[0059] Figure 3 This is a schematic diagram illustrating the distribution of multiple non-focused regions according to an exemplary embodiment.

[0060] Figure 4 This is a flowchart of an image rendering method according to an exemplary embodiment. Figure 2 .

[0061] Figure 5 This is a schematic diagram of the structure of an eyeglass device according to an exemplary embodiment.

[0062] Figure 6 This is a schematic diagram of the structure of an image rendering apparatus according to an exemplary embodiment. Figure 1 .

[0063] Figure 7 This is a schematic diagram of the structure of an image rendering apparatus according to an exemplary embodiment. Figure 2 .

[0064] Figure 8 This is a block diagram illustrating an electronic device according to an exemplary embodiment. Detailed Implementation

[0065] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0066] This disclosure provides an image rendering method. Figure 1 This is a flowchart illustrating an image rendering method according to an exemplary embodiment. Figure 1 .like Figure 1 As shown, this image rendering method is applied to a display device, which executes the image rendering method by interacting with a glasses device. The image rendering method mainly includes the following steps:

[0067] S101, Receive gaze point information sent by the glasses device;

[0068] S102. Based on the gaze point information, the display area of ​​the display screen is divided into a gaze area and a non-gaze area;

[0069] S103. Render the objects in the focal region and the non-focal region separately based on different resolutions;

[0070] The resolution of the focal region after rendering is greater than the resolution of the non-focal region after rendering.

[0071] In this embodiment of the disclosure, such as Figure 2 As shown, this image rendering method is adapted to scenarios where a user wearing glasses 11 is looking at the display screen 12 of the display device. When a user wearing glasses is looking at the display screen of the display device, the display device can interact with the glasses based on this image rendering method to render the objects in the gaze area and the non-gaze area separately based on different resolutions, thereby reducing rendering overhead without affecting the user's viewing experience.

[0072] The aforementioned display device includes a display screen. This display device includes fixed display devices and mobile display devices. Fixed display devices include televisions and desktop computers; mobile display devices include smartphones, laptops, tablets, and wearable devices (such as watches or wristbands).

[0073] The aforementioned glasses device has at least a color camera and an eye-tracking camera, capable of acquiring gaze point information through the color camera and the eye-tracking camera and sending it to a display device. The glasses device includes Extended Reality (XR) glasses, Virtual Reality (VR) glasses, Augmented Reality (AR) glasses, or Mixed Reality (MR) glasses, etc., and this disclosure does not impose any limitations.

[0074] For example, the display device is a television set, and the glasses device is an XR glasses device. When a user wearing the XR glasses device looks at the television screen, the XR glasses device acquires gaze point information and sends it to the television set. The television set can then divide its display area into a gaze area and a non-gaze area based on the gaze point information sent by the XR glasses device, and render the objects in the gaze area and the non-gaze area separately based on different resolutions. In this way, the rendering overhead of the television set can be reduced without affecting the user's viewing experience.

[0075] In step 101, the glasses device can directly send the gaze point information to itself, or it can first send the gaze point information to an intermediate device, which then forwards it to the display device. Here, the intermediate device is a device that has established communication connections with both the glasses device and the display device. When the display device is a television or a computer, the intermediate device may include a smartphone or a smartwatch, etc., and this embodiment of the disclosure is not limited thereto.

[0076] In this embodiment of the disclosure, a user wearing glasses can gaze at any of a plurality of display devices in the vicinity. The glasses device is equipped with a color camera, which can capture an image of the scene in front of it and analyze the captured image to determine whether it is a display screen image. If it is determined to be a display screen image, key point analysis can be performed on the display screen image to obtain first key point information. The glasses device can also receive multiple second key point information related to the current display screen sent by the plurality of surrounding display devices. The first key point information is matched with the multiple second key point information, and the display device corresponding to the successfully matched second key point information is used to communicate with the glasses device to receive gaze point information sent by the glasses device.

[0077] It should be noted that the display device receiving gaze information is the display device that the user wearing glasses is looking at. This display device has established a communication connection with the glasses. Here, the communication connection includes Bluetooth, Wi-Fi, ZigBee, etc., and this disclosure does not limit this type of connection.

[0078] In this embodiment of the disclosure, after establishing a communication connection between the display device and the glasses device, the display device can receive gaze point information sent by the glasses device worn by the user based on the communication connection. The glasses device can be worn on the user's eyes, allowing the user to gaze at the display screen of the display device through the glasses device.

[0079] Here, the gaze point information refers to the information formed when a user wearing glasses gazes at the display screen of the display device. This gaze point information includes at least the user's gaze direction at the display screen and the position of the gaze point formed on the display screen, etc., but this embodiment does not limit this.

[0080] In step 102, the aforementioned gaze area is the area of ​​the display screen that the user is gazing at. It is understood that when a user gazes at the display screen, they will focus on a portion of the screen, while another portion will not be focused on.

[0081] In this embodiment of the disclosure, when the gaze point information includes the user's gaze direction on the display screen, the display area can be divided into a gaze area and a non-gaze area based on the user's gaze direction on the display screen. The area corresponding to the gaze direction can be the gaze area.

[0082] In this embodiment of the disclosure, when the gaze point information includes the location of the gaze point formed on the display screen by the user's gaze, the display area can be divided into a gaze area and a non-gaze area based on the location of the gaze point, wherein the location of the gaze point is located within the gaze area and outside the non-gaze area.

[0083] In step 103, the display device first obtains the resolution corresponding to the gaze area and the resolution corresponding to the non-gaze area, and then renders the rendering object of the gaze area based on the resolution corresponding to the gaze area, and renders the rendering object of the non-gaze area based on the resolution corresponding to the non-gaze area.

[0084] It should be noted that the embodiments of this disclosure can synthesize images rendered at different resolutions to obtain the final display image.

[0085] Here, the display area of ​​the screen can be multiple or a single non-focused area. When there are multiple non-focused areas, the resolution of the multiple non-focused areas after rendering can be the same or different, and this disclosure does not limit this.

[0086] In this embodiment of the disclosure, the resolution of the gaze area after rendering is greater than the resolution of the non-gaze area after rendering. That is, this embodiment of the disclosure no longer renders all display areas of the display screen at the same resolution, but renders the gaze area at a resolution that is relatively higher than the resolution of the non-gaze area, so that the gaze area can be rendered at a high resolution.

[0087] As can be seen, the foveated rendering employed in the display device of this disclosure takes into account the human eye's focus of visual attention. When rendering an image, it applies a higher resolution to the observer's gaze location (i.e., the gaze area), while using a lower resolution for non-gaze areas. This allows for the allocation of rendering resources according to the user's gaze, reducing rendering overhead and improving application frame rates without affecting the user experience.

[0088] In this embodiment of the disclosure, rendering objects in the gaze region and the non-gaze region are rendered separately based on different resolutions, including:

[0089] Render the objects in the gaze area based on a preset first resolution;

[0090] Render objects in non-focused areas based on a preset second resolution;

[0091] The first resolution is the maximum resolution that the display screen can support; the second resolution is less than the first resolution.

[0092] In other words, the embodiments of this disclosure can render the objects in the gaze area at the maximum resolution supported by the display screen, so that the high resolution is applied to the gaze area that the user is looking at, and the low resolution is applied to the non-gaze area that the user is not looking at. In scenarios such as playing games or watching video playback, this can not only not affect the user's viewing experience, but also improve the game frame rate or video playback frame rate.

[0093] For example, when the display supports 8K 30FPS video playback, this embodiment of the disclosure can perform 8K quality video rendering on the gaze area and 4K quality video rendering on the non-gaze area, increasing the frame rate of the played video from 30FPS to 45FPS, or even to 60FPS.

[0094] For example, in related technologies, foveated rendering is deployed on extended reality devices (such as extended reality glasses devices). Display devices (such as televisions) typically use frame interpolation or deep learning super sampling (DLSS) to improve the application frame rate when playing videos and games, which cannot achieve the effect of reducing rendering overhead and not affecting the user's viewing experience.

[0095] Based on this, embodiments of this disclosure propose a display device receiving gaze point information sent by a glasses device, dividing the display area of ​​the screen into a gaze area and a non-gaze area based on the gaze point information, and rendering objects in the gaze area and the non-gaze area separately based on different resolutions, so that the resolution of the gaze area after rendering is greater than the resolution of the non-gaze area after rendering. In other words, by interacting with the glasses device, the display device of this disclosure can take into account the human eye's focus on visual attention based on the gaze point information, and instead of rendering all display areas of the screen at the same resolution, it makes the resolution of the gaze area after rendering greater than the resolution of the non-gaze area after rendering. In this way, rendering resources can be allocated accordingly based on the user's gaze, reducing rendering consumption and improving application frame rate without affecting the user experience.

[0096] In some embodiments, the display device performing the image rendering method further includes:

[0097] When there are multiple non-focused regions, render the rendering objects of each non-focused region separately based on different resolutions;

[0098] Among them, the resolution of the non-focal region after rendering is negatively correlated with the distance from the outer edge of the non-focal region to the focal region.

[0099] In this embodiment of the disclosure, the peripheral area outside the gaze area in the display area can be divided into multiple non-gaze areas. Here, the gaze area is nested within the non-gaze areas, and adjacent gaze areas within the multiple non-gaze areas are nested and sequentially moved away from the gaze area. Exemplarily, the shape of the non-gaze area includes a ring, such as a circular ring or a square ring, etc., and this embodiment of the disclosure does not limit this.

[0100] It should be noted that the number of non-focused regions in this embodiment can be set according to the actual situation. The number of non-focused regions can be two or three, and this embodiment does not limit this.

[0101] In this embodiment of the disclosure, the resolution of the rendered non-focal region is negatively correlated with the distance from the outer edge of the non-focal region to the focal region. That is, the farther the non-focal region is from the focal region, the lower the resolution of the rendered non-focal region.

[0102] For example, such as Figure 3As shown, multiple non-focused regions include a first non-focused region A and a second non-focused region B. A focusing region C is located within the first non-focused region A and the second non-focused region B. The distance from the outer edge of the second non-focused region B to the focusing region C is greater than the distance from the outer edge of the first non-focused region A to the focusing region C. Here, the resolution of the rendered second non-focused region B can be set to be less than the resolution of the rendered first non-focused region A.

[0103] It is understood that, compared to rendering multiple non-focused regions at the same resolution, the embodiments of this disclosure render the rendering objects of multiple non-focused regions separately based on different resolutions, so that the resolution of the rendered non-focused region is negatively correlated with the distance from the outer edge of the non-focused region to the focusing region. In this way, it can better match the focus of the user's visual attention, and further reduce rendering consumption without affecting the user experience.

[0104] In some embodiments, gaze point information includes the coordinates of the gaze point formed on the display screen by the user's gaze; based on the gaze point information, the display area of ​​the display screen is divided into a gaze area and a non-gaze area, including:

[0105] Based on the coordinates of the gaze point, the display area of ​​the screen is divided into a gaze area and a non-gaze area.

[0106] The display area of ​​the aforementioned display screen can form a rectangular coordinate system. The center of the coordinate circle of the rectangular coordinate system can be the center of the display area or the connection point of the two sides of the display area. This disclosure does not limit this.

[0107] Here, the object being gazed at by the user on the display screen can be located using the gaze point coordinates. This object of gaze includes people, vehicles, etc., but this embodiment does not impose any limitations on this.

[0108] In this embodiment of the disclosure, dividing the display area of ​​the display screen into a gaze area and a non-gaze area based on the coordinates of the gaze point may include: determining the object being gazed at by the user on the display screen based on the coordinates of the gaze point, and dividing the gaze area and the non-gaze area based on the outline of the gaze object; it may also include: obtaining the gaze area and the non-gaze area based on the distance from the outer edge of the multiple display sub-regions obtained by dividing the display area to the coordinates of the gaze point; it may also include: drawing a circle with the center of the circle at the coordinates of the gaze point and a radius of a preset distance, designating the circular area as the gaze area, and designating the area outside the circular area in the display area as the non-gaze area.

[0109] In some embodiments, the display area of ​​the screen is divided into a gazing region and a non-gazing region based on the coordinates of the gaze point, including:

[0110] Based on the coordinates of the gaze point, determine the object the user is gazing at on the display screen;

[0111] The area enclosed by the outline of the object being gazed upon is defined as the gaze area;

[0112] The area outside the viewing area will be displayed as the non-viewing area.

[0113] In this embodiment of the disclosure, the display area of ​​the screen can display different objects of gaze, and the object of the user's gaze on the screen can be determined based on the coordinates of the gaze point. By performing image analysis on the image of the object of gaze, the area enclosed by the outline of the object of gaze can be obtained.

[0114] For example, the object of gaze may include a flower. When a user gazes at a flower displayed on a screen, the area enclosed by the outline of the flower can be used as the gaze area. This conforms to the focus of the user's visual attention and does not affect the viewing quality of the flower.

[0115] It should be noted that the object being gazed upon can be in the next frame of the image to be rendered. The gazed and non-gaze regions of the next frame of the image to be rendered can be obtained through the coordinates of the gaze point.

[0116] In this embodiment of the disclosure, the display area is divided based on the region surrounded by the outline of the object being gazed upon to obtain the gaze area and the non-gaze area. This takes into account the user's gaze object's need for high resolution, making the divided gaze area more accurate, thereby improving the user's viewing experience.

[0117] In some embodiments, the display area of ​​the screen is divided into a gazing region and a non-gazing region based on the coordinates of the gaze point, including:

[0118] The display area is divided into multiple display sub-areas;

[0119] The display area of ​​the screen is divided into a fixation area and a non-fixation area based on the distances from the outer edges of multiple display sub-regions to the coordinates of the fixation point.

[0120] Among them, the display sub-regions with a distance less than a preset distance threshold are designated as the gaze region;

[0121] Display sub-regions with a distance greater than or equal to a preset distance threshold are designated as non-focused regions.

[0122] In this embodiment of the disclosure, the display area is divided into multiple display sub-regions. The division rule can be to draw circles around the center of the display area, with the area between two adjacent circles serving as a single display sub-region. Alternatively, the division rule can be to draw squares around the center of the display area, with the area between two adjacent squares serving as a single display sub-region.

[0123] It should be noted that the preset distance threshold can be set according to the actual situation, and this embodiment does not limit it.

[0124] In this embodiment of the present disclosure, the display area is first divided into multiple display sub-regions. Then, based on the distance, the display sub-regions with a distance less than a preset distance threshold are taken as the fixation region, and the display sub-regions with a distance greater than or equal to the preset distance threshold are taken as the non-fixation region. This can take into account the human eye's focus on visual attention, making the divided fixation region more accurate.

[0125] This disclosure also proposes an image rendering method. Figure 4 This is a flowchart of an image rendering method according to an exemplary embodiment. Figure 2 .like Figure 4 As shown, this image rendering method is applied to an eyeglass device, which executes the image rendering method by interacting with a display device, mainly including the following steps:

[0126] S201. Collect the first collection information by using the color camera of the glasses device to capture the display device that the user is looking at;

[0127] S202. Collect the user's eyes through the eye-tracking camera of the glasses device to obtain the second collection information;

[0128] S203. Based on the first and second collected information, determine the gaze point information of the display screen in the user's gaze display device;

[0129] S204. Send gaze point information to the display device.

[0130] In this embodiment of the disclosure, such as Figure 5 As shown, the glasses device 50 has a color camera 51 and an eye-tracking camera 52. When a user wears the glasses device and looks at a display device, the color camera can collect first information from the display screen of the display device that the user is looking at, and the eye-tracking camera can collect second information from the user's eyes.

[0131] Here, the first acquired information includes the display screen image captured by the color camera, and the second acquired information includes the eye image captured by the eye-tracking camera.

[0132] In this embodiment of the disclosure, after obtaining the first acquisition information and the second acquisition information, the gaze point information of the user's gaze on the display screen in the display device can be determined based on the first acquisition information and the second acquisition information. Here, the gaze point information may include the gaze direction.

[0133] Since different display devices have different sizes, the first collected information can determine which of the multiple display devices the user is looking at. Because the eye-tracking camera can track eye movements, the second collected information can determine the direction of the user's gaze on the display device.

[0134] It should be noted that the gaze point information may also include: the coordinates of the gaze point. In some embodiments, determining the gaze point information of the user gazing at the display screen in the display device based on the first acquisition information and the second acquisition information includes:

[0135] The position of the display screen relative to the user is determined based on the first collected information;

[0136] The user's gaze direction is determined based on the second collected information;

[0137] The coordinates of the gaze point are determined based on the position of the display screen relative to the user and the direction of gaze.

[0138] In this embodiment of the disclosure, determining the position of the display screen relative to the user based on the first acquired information may include: performing keypoint matching on the first acquired information to determine the proportion of the display screen image in the display screen image, and determining the distance of the display screen relative to the user based on the proportion and the preset calibration information of the color camera. Here, the six degrees of freedom (6DOF) motion function of the glasses device can be used to determine the real-time position of the display screen relative to the user.

[0139] The above-mentioned determination of the user's gaze direction based on the second acquired information may include: determining eye feature information based on the second acquired information, and inputting the eye feature information into a preset gaze point estimation model to obtain the gaze direction.

[0140] Here, eye features may include: pupil position, pupil shape, iris position, iris shape, eyelid position, corner of the eye position, spot position, myofascial current, capacitance value, etc.

[0141] The aforementioned gaze point estimation model can be obtained by training a gaze training model using sample eye feature information and sample gaze direction. The gaze point estimation model takes eye feature information as input and outputs the gaze direction. Here, the gaze training model can include any neural network model, and this embodiment of the disclosure does not impose any limitations on it.

[0142] In this embodiment of the disclosure, determining the coordinates of the gaze point based on the gaze direction and the position of the display screen relative to the user may include: obtaining the coordinates of the gaze point on the display area of ​​the display screen based on the distance of the display screen relative to the user and the angle of the gaze direction relative to the display screen. Here, the gaze point may include the intersection of the gaze direction and the display screen.

[0143] It is understood that the embodiments of this disclosure determine the gaze point information of the user's gaze on the display screen of the display device by the first collection information and the second collection information and send the gaze point information to the display device, which enables the display device to render at different resolutions based on the gaze point information, thereby reducing rendering consumption and improving the application frame rate without affecting the user experience.

[0144] This disclosure also proposes an image rendering apparatus, such as... Figure 6 As shown, the image rendering apparatus includes:

[0145] The receiving module 1001 is configured to receive gaze point information sent by the glasses device, wherein the gaze point information is the information formed on the display screen of the display device by the user wearing the glasses device.

[0146] The segmentation module 1002 is configured to divide the display area of ​​the display screen into a gaze area and a non-gaze area based on gaze point information. The gaze area is the area where the user gazes at the display screen, and the non-gaze area is the area outside the gaze area in the display area.

[0147] The first rendering module 1003 is configured to render the objects in the foveated region and the non-foveated region separately based on different resolutions; wherein, the resolution of the foveated region after rendering is greater than the resolution of the non-foveated region after rendering.

[0148] In some embodiments, the display device further includes:

[0149] The second rendering module is configured to render the rendering objects of multiple non-focused regions separately based on different resolutions when there are multiple non-focused regions.

[0150] Among them, the resolution of the non-focal region after rendering is negatively correlated with the distance from the outer edge of the non-focal region to the focal region.

[0151] In some embodiments, the partitioning module is further configured to divide the display area of ​​the display screen into a gaze region and a non-gaze region based on the coordinates of the gaze point.

[0152] In some embodiments, the segmentation module is further configured to determine the object of the user's gaze on the display screen based on the coordinates of the gaze point;

[0153] The area enclosed by the outline of the object being gazed upon is defined as the gaze area;

[0154] The area outside the viewing area will be displayed as the non-viewing area.

[0155] In some embodiments, the partitioning module is further configured to partition the display area to obtain multiple display sub-regions; and to partition the display area of ​​the display screen into a viewing area and a non-viewing area based on the distances from the outer edges of the multiple display sub-regions to the coordinates of the viewing point; wherein, the display sub-regions with a distance less than a preset distance threshold are designated as viewing areas; and the display sub-regions with a distance greater than or equal to the preset distance threshold are designated as non-viewing areas.

[0156] In some embodiments, the first rendering module is further configured to render the rendering object in the gaze area based on a preset first resolution; and to render the rendering object in the non-gaze area based on a preset second resolution; wherein the first resolution is the maximum resolution that the display screen can support; and the second resolution is less than the first resolution.

[0157] This disclosure also proposes an image rendering apparatus, such as... Figure 7 As shown, the image rendering apparatus includes:

[0158] The first acquisition module 2001 is configured to acquire first acquisition information by using the color camera of the glasses device to acquire information from the display device that the user is looking at.

[0159] The second acquisition module 2002 is configured to acquire second acquisition information by acquiring the user's eyes through the eye-tracking camera of the glasses device;

[0160] The gaze determination module 2003 is configured to determine the gaze point information of the display screen in the user's gaze display device based on the first acquisition information and the second acquisition information;

[0161] The sending module 2004 is configured to send gaze point information to the display device.

[0162] In some embodiments, the gaze determination module is further configured to determine the position of the display screen relative to the user based on the first acquisition information; determine the user's gaze direction based on the second acquisition information; and determine the coordinates of the gaze point based on the position of the display screen relative to the user and the gaze direction.

[0163] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0164] Figure 8This is a structural block diagram illustrating an electronic device 800 according to an exemplary embodiment. For example, the electronic device 800 may be a mobile phone, computer, digital broadcasting terminal, messaging device, game console, tablet device, medical device, fitness equipment, personal digital assistant, etc.

[0165] Reference Figure 8 The electronic device 800 may include one or more of the following components: processing component 802, memory 804, power supply component 806, multimedia component 808, audio component 810, input / output (I / O) interface 812, sensor component 814, and communication component 816.

[0166] Processing component 802 typically controls the overall operation of electronic device 800, such as operations associated with at least one of display, telephone call, data communication, camera operation, and recording operation. Processing component 802 may include one or more processors 820 to execute instructions to perform all or part of the steps of the methods described above. Furthermore, processing component 802 may include one or more modules to facilitate interaction between processing component 802 and other components. For example, processing component 802 may include a multimedia module to facilitate interaction between multimedia component 808 and processing component 802.

[0167] Memory 804 is configured to store various types of data to support operation on electronic device 800. Examples of such data include at least one of the following: instructions for any application or method operating on electronic device 800, contact data, phonebook data, messages, pictures, and videos. Memory 804 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read Only Memory (EEPROM), Erasable Programmable Read-Only Memory (EPROM), Programmable Read Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0168] Power supply component 806 provides power to various components of electronic device 800. Power supply component 806 may include at least one of the following: a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to electronic device 800.

[0169] Multimedia component 808 includes a screen that provides an output interface between electronic device 800 and user. In some embodiments, the screen may include a Liquid Crystal Display (LCD) and a Touch Panel (TP). If the screen includes a Touch Panel, the screen may be implemented as a touchscreen to receive input signals from the user. The Touch Panel includes one or more touch sensors to sense touches, swipes, and gestures on the Touch Panel. The touch sensors may sense not only the boundaries of touch or swipe actions but also the duration and pressure associated with the touch or swipe operation. In some embodiments, multimedia component 808 includes a front-facing camera and / or a rear-facing camera. When electronic device 800 is in an operating mode, such as a shooting mode or video mode, the front-facing camera and / or rear-facing camera may receive external multimedia data. Each front-facing camera and rear-facing camera may be a fixed optical lens system or have focal length and optical zoom capabilities.

[0170] Audio component 810 is configured to output and / or input audio signals. For example, audio component 810 includes a microphone (MIC) configured to receive external audio signals when electronic device 800 is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 804 or transmitted via communication component 816. In some embodiments, audio component 810 also includes a speaker for outputting audio signals.

[0171] I / O interface 812 provides an interface between processing component 802 and peripheral interface modules, such as keyboards, click wheels, and buttons. These buttons may include, but are not limited to, home buttons, volume buttons, power buttons, and lock buttons.

[0172] Sensor assembly 814 includes one or more sensors for providing state assessments of various aspects of electronic device 800. For example, sensor assembly 814 may detect the on / off state of electronic device 800, the relative positioning of components such as the display and keypad of electronic device 800, changes in position of electronic device 800 or one of its components, the presence or absence of user contact with electronic device 800, orientation or acceleration / deceleration of electronic device 800, and temperature changes of electronic device 800. Sensor assembly 814 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 814 may also include an optical sensor, such as a complementary metal-oxide-semiconductor (CMOS) or charge-coupled device (CCD) image sensor, for use in imaging applications. In some embodiments, sensor assembly 814 may also include, but is not limited to, at least one of the following: an accelerometer, a gyroscope, a magnetometer, a pressure sensor, and a temperature sensor.

[0173] Communication component 816 is configured to facilitate wired or wireless communication between electronic device 800 and other devices. Electronic device 800 can access wireless networks based on communication standards, such as Wi-Fi, 4G, 5G, or combinations thereof. In one exemplary embodiment, communication component 816 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, communication component 816 also includes a Near Field Communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on Radio Frequency Identification (RFID), Infrared Data Association (IrDA), Ultra Wide Band (UWB), Bluetooth (BT), and other technologies.

[0174] In an exemplary embodiment, the electronic device 800 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components.

[0175] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 804 including executable instructions or a computer program, which can be executed by a processor 820 of a display device or glasses device included in the electronic device 800 to perform the above-described method. For example, the non-transitory computer-readable storage medium may be a ROM, random access memory (RAM), a compact disc read-only memory (CD-ROM), magnetic tape, floppy disk, and optical data storage device, etc.

[0176] A non-transitory computer-readable storage medium, when the instructions in the storage medium are executed by a processor of an electronic device including a display device or eyeglasses, enables the processor to perform any of the image rendering methods described above in the embodiments of this disclosure. For example, the method includes: receiving gaze point information sent by the eyeglasses device, the gaze point information being information formed by a user wearing the eyeglasses device gazing at a display screen on the display device; dividing the display area of ​​the display screen into a gaze area and a non-gaze area based on the gaze point information, the gaze area being the area where the user gazes at the display screen, and the non-gaze area being the area outside the gaze area in the display area; rendering rendering objects for the gaze area and the non-gaze area respectively based on different resolutions; wherein, after rendering, the resolution of the gaze area is greater than the resolution of the non-gaze area after rendering.

[0177] This disclosure provides a computer program product comprising a computer program or executable instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer program or executable instructions from the computer-readable storage medium and executes the computer program or executable instructions, causing the computer device to perform any of the image rendering methods described in this disclosure.

[0178] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the claims.

[0179] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. An image rendering method, characterized in that, include: The system receives gaze point information sent by the glasses device, wherein the gaze point information is information formed on the display screen of the display device by the user wearing the glasses device. Based on the gaze point information, the display area of ​​the display screen is divided into a gaze area and a non-gaze area. The gaze area is the area where the user gazes at the display screen, and the non-gaze area is the area outside the gaze area in the display area. The rendering objects in the gaze region and the non-gaze region are rendered separately based on different resolutions; The resolution of the gaze region after rendering is greater than the resolution of the non-gaze region after rendering.

2. The method according to claim 1, characterized in that, The method further includes: When there are multiple non-focused regions, the rendering objects of the multiple non-focused regions are rendered separately based on different resolutions; The resolution of the non-focused region after rendering is negatively correlated with the distance from the outer edge of the non-focused region to the focused region.

3. The method according to claim 1 or 2, characterized in that, The gaze point information includes the coordinates of the gaze point formed on the display screen when the user gazes at the screen; the step of dividing the display area of ​​the display screen into a gaze area and a non-gaze area based on the gaze point information includes: Based on the coordinates of the gaze point, the display area of ​​the display screen is divided into the gaze area and the non-gaze area.

4. The method according to claim 3, characterized in that, The division of the display area of ​​the screen into the gaze region and the non-gaze region based on the coordinates of the gaze point includes: Based on the coordinates of the gaze point, determine the object the user is gazing at on the display screen; The region enclosed by the outline of the object being gazed upon is defined as the gaze region; The area outside the gaze area in the display area is defined as the non-gaze area.

5. The method according to claim 3, characterized in that, The division of the display area of ​​the screen into the gaze region and the non-gaze region based on the coordinates of the gaze point includes: The display area is divided into multiple display sub-areas; Based on the distances from the outer edges of the multiple display sub-regions to the coordinates of the gaze point, the display area of ​​the display screen is divided into the gaze area and the non-gaze area; The display sub-regions whose distance is less than a preset distance threshold are defined as the gaze region; The display sub-regions whose distance is greater than or equal to the preset distance threshold are designated as the non-focused regions.

6. The method according to claim 1 or 2, characterized in that, The rendering of the objects in the gaze region and the non-gaze region based on different resolutions includes: The rendering object in the gaze area is rendered based on a preset first resolution; The rendering object in the non-focused region is rendered based on a preset second resolution; Wherein, the first resolution is the maximum resolution that the display screen can support; The second resolution is smaller than the first resolution.

7. An image rendering method, characterized in that, include: The first data is obtained by capturing the display device that the user is looking at using the color camera of the glasses device; The user's eyes are captured by the eye-tracking camera of the glasses device to obtain second data. Based on the first collected information and the second collected information, determine the gaze point information of the user gazing at the display screen in the display device; Send the gaze point information to the display device.

8. The method according to claim 7, characterized in that, The step of determining the user's gaze point information on the display screen of the display device based on the first collected information and the second collected information includes: The position of the display screen relative to the user is determined based on the first collected information; The user's gaze direction is determined based on the second collected information; The coordinates of the gaze point are determined based on the position of the display screen relative to the user and the gaze direction.

9. An image rendering apparatus, characterized in that, include: The receiving module is configured to receive gaze point information sent by the glasses device, wherein the gaze point information is information formed on the display screen of the display device by the user wearing the glasses device. The segmentation module is configured to divide the display area of ​​the display screen into a fixation area and a non-fixation area based on the fixation point information. The fixation area is the area where the user fixates on the display screen, and the non-fixation area is the area outside the fixation area in the display area. The first rendering module is configured to render the rendering objects of the foveated region and the non-foveated region separately based on different resolutions; wherein, the resolution of the foveated region after rendering is greater than the resolution of the non-foveated region after rendering.

10. An image rendering apparatus, characterized in that, include: The first acquisition module is configured to acquire first acquisition information by using the color camera of the glasses device to capture the display device that the user is looking at; The second acquisition module is configured to acquire second acquisition information by acquiring data from the user's eyes through the eye-tracking camera of the glasses device; The gaze determination module is configured to determine the gaze point information of the user gazing at the display screen of the display device based on the first collected information and the second collected information; The sending module is configured to send the gaze point information to the display device.

11. An electronic device, characterized in that, include: processor; Memory used to store computer programs or instructions; The processor executes the computer program or instructions to implement the steps of the method according to any one of claims 1 to 6 and the steps of the method according to any one of claims 7 and 8.

12. A non-transitory computer-readable storage medium storing a computer program or instructions, characterized in that, When the computer program or instructions in the storage medium are executed by a processor, the steps of the method according to any one of claims 1 to 6 and the steps of the method according to any one of claims 7 and 8 are implemented.

13. A computer program product, comprising a computer program or instructions, characterized in that, When the computer program or instructions are executed by a processor, they implement the steps of the method according to any one of claims 1 to 6 and the steps of the method according to any one of claims 7 and 8.

Images