Image rendering method, head-mounted display device, and readable storage medium

Abstract

The application discloses an image rendering method, a head-mounted display device and a readable storage medium. The image rendering method comprises the following steps: predicting an image frame edge of a predicted window image according to head motion posture information, a frame rate refresh time interval and a maximum observation visual angle; executing a rendering thread to render pixels in the image frame edge; if the pixels in the image frame edge are not rendered completely at a preset time node before a next clock cycle arrives, executing an asynchronous time warping thread to perform directional warping processing on a current window image which is rendered completely last time to obtain a transition window image; displaying the transition window image until the pixels in the image frame edge are rendered completely, and displaying the predicted window image which is rendered. The application can reduce the dizziness caused by time delay in the process of rendering the image.

Description

Technical Field

[0001] This application relates to the field of wearable device technology, and more particularly to an image rendering method, a head-mounted display device, and a readable storage medium. Background Technology

[0002] Extended Reality (XR), as an emerging technology, is gradually entering people's field of vision and is being applied and popularized in various industries. Specifically, Extended Reality includes Virtual Reality (VR), Augmented Reality (AR), and Mixed Reality (MR).

[0003] With the development of extended reality technology, resolution and refresh rates have further improved, meaning that the amount of signal output per frame during image transmission is larger, and the requirements for transmission bandwidth are higher. This poses a significant challenge to the system's rendering capabilities and end-to-end transmission capabilities. Currently, for ultra-high-resolution extended reality applications, image rendering technology is used to refresh the rendered virtual images onto the display device, allowing users to experience the effects of virtual reality / augmented reality through head-mounted displays.

[0004] Because the rendering process takes time, it creates a time delay between reality and perception. For example, during rendering, the user's head or the headset the user is wearing may move, causing a time delay between the user's head posture information and the image data output by the headset. If the time delay is too large, it can cause dizziness. Summary of the Invention

[0005] The main objective of this application is to provide an image rendering method, a head-mounted display device, and a readable storage medium, aiming to solve the technical problem that time delays easily occur during the image rendering process of extended reality devices, causing dizziness.

[0006] To achieve the above objectives, this application provides an image rendering method applied to a head-mounted display device, the method comprising:

[0007] The system dynamically detects the user's head movement posture information, obtains the pre-stored frame rate refresh time interval and maximum observation angle, and predicts the image frame edge of the prediction window image in the next time step based on the head movement posture information, the frame rate refresh time interval and the maximum observation angle.

[0008] Execute the rendering thread to render the pixels within the edges of the image frame;

[0009] If the pixels within the edge of the image frame are not rendered by the preset time node before the next clock cycle arrives, an asynchronous time warp thread is executed to perform orientation warp processing on the most recently rendered current window image to obtain a transition window image.

[0010] The transition window image is displayed until the pixels within the edge of the image frame are rendered, and then the rendered prediction window image is displayed.

[0011] Optionally, after the step of executing the rendering thread to render pixels within the image frame edge, the method further includes:

[0012] If the pixels within the edge of the image frame are not rendered by the preset time node before the next clock cycle arrives, the rendered prediction window image will be displayed.

[0013] Optionally, the step of executing the rendering thread to render the prediction window image includes:

[0014] The system detects the user's predicted eye observation point in the predicted view window image, and uses the predicted eye observation point as a reference to divide the predicted view window image into at least two partition images from near to far according to a preset partitioning rule.

[0015] The partitioned image is rendered according to the rendering resolution corresponding to the partitioned image, wherein the rendering resolution of the partitioned image decreases sequentially from near to far.

[0016] Optionally, the step of detecting the user's predicted eye observation point in the predicted viewport image includes:

[0017] Collect the user's current eye image, determine the eye model with the highest matching degree to the current eye image, and use the eye model with the highest matching degree as the current actual eye model;

[0018] The eye observation point mapped by the current actual eye model is obtained from the preset eye model mapping database, and the mapped eye observation point is used as the user's predicted eye observation point in the prediction window image.

[0019] Optionally, the detection of the user's predicted eye observation point in the predicted viewport image includes:

[0020] Acquire the user's current eye image and perform grayscale processing on the current eye image;

[0021] The pupil region image is determined based on the current eyeball image after grayscale processing, and the pupil region image is then binarized.

[0022] Edge detection is performed on the binarized pupil region image to obtain pupil edge points, and then ellipse fitting is performed on the pupil edge points to obtain the current pupil center.

[0023] The predicted eye observation point of the user in the predicted view window image is determined based on the current pupil center.

[0024] Optionally, the step of determining the user's predicted eye observation point in the predicted view window image based on the current pupil center includes:

[0025] From the pre-calibrated pupil center mapping data table, retrieve the predicted eye observation point of the current pupil center mapping;

[0026] The mapped eye observation point is used as the user's predicted eye observation point in the prediction window image.

[0027] Optionally, the step of dividing the prediction window image into at least two partition images from near to far further includes:

[0028] The brightness display of the partitioned image is controlled according to the backlight brightness of the corresponding area of ​​the partitioned image, wherein the backlight brightness of the area of ​​the partitioned image decreases sequentially from near to far.

[0029] Optionally, the step of dividing the predicted visual window image into at least two partition images from near to far, based on the predicted eye observation point and according to a preset partitioning rule, includes:

[0030] Based on the predicted eye observation point, determine the predicted fixation area;

[0031] The intention observation image within the predicted gaze area in the predicted viewport image is divided into a first partition image;

[0032] The non-intended observation image outside the predicted gaze region in the predicted viewport image is divided into a second partition image;

[0033] The step of controlling the brightness display of the partitioned image according to the backlight brightness of the corresponding area of ​​the partitioned image includes:

[0034] The brightness display of the first partition image is controlled by the backlight brightness of the first region, and the brightness display of the second partition image is controlled by the backlight brightness of the second region, wherein the backlight brightness of the first region is greater than the backlight brightness of the second region.

[0035] This application also provides a head-mounted display device, which is a physical device. The head-mounted display device includes: a memory, a processor, and a program of the image rendering method stored in the memory and executable on the processor. When the program of the image rendering method is executed by the processor, it can implement the steps of the image rendering method as described above.

[0036] This application also provides a readable storage medium, which is a computer-readable storage medium, on which a program implementing an image rendering method is stored. The program implementing the image rendering method is executed by a processor to implement the steps of the image rendering method as described above.

[0037] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the image rendering method described above.

[0038] This application dynamically detects the user's head movement posture information and obtains the pre-stored frame rate refresh interval and maximum viewing angle. Based on the head movement posture information, the frame rate refresh interval, and the maximum viewing angle, it predicts the image frame edge of the prediction window image for the next time step and executes a rendering thread to render the pixels within the image frame edge. If the pixels within the image frame edge are not rendered completely at a preset time node before the next clock cycle arrives, an asynchronous time warp thread is executed to perform orientation warping processing on the most recently rendered current window image to obtain a transition window image. This transition window image is displayed until the pixels within the image frame edge are rendered completely, and then the rendered prediction window image is displayed. By combining position information prediction with asynchronous time warp technology, a virtual compensation frame (i.e., the transition window image) is generated and displayed. This can improve the image frame rate, reduce the delay at a certain stage in the process of acquiring and processing the image frame, reduce the delay in displaying the image frame, and effectively prevent screen jitter due to the current frame not being rendered completely. This solves the technical problem of dizziness caused by time delay during the rendering process of extended reality devices. Attached Figure Description

[0039] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0040] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0041] Figure 1 This is a flowchart illustrating the first embodiment of the image rendering method of this application;

[0042] Figure 2 This is a flowchart illustrating the second embodiment of the image rendering method of this application;

[0043] Figure 3 This is a schematic diagram of the image frame edge of the predicted viewport image in an embodiment of this application.

[0044] Figure 4 This is a schematic diagram illustrating the head posture information of a user wearing a head-mounted display device in one embodiment of this application;

[0045] Figure 5 This is a schematic diagram of a scene observing the perspective of identifying the user's intent in one embodiment of this application;

[0046] Figure 6 This is a schematic diagram illustrating the user's predicted gaze direction in one embodiment of this application;

[0047] Figure 7 This is a schematic diagram of the hardware operating environment involved in the head-mounted display device in the embodiments of this application.

[0048] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0049] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0050] In this embodiment, the head-mounted display device of this application may be, for example, a Mixed Reality (MR) device (e.g., MR glasses or MR helmet), an Augmented Reality (AR) device (e.g., AR glasses or AR helmet), a Virtual Reality (VR) device (e.g., VR glasses or VR helmet), an Extended Reality (XR) device (e.g., XR glasses or XR helmet), or some combination thereof.

[0051] Example 1

[0052] Currently, ultra-high resolution extended reality applications utilize image rendering technology to refresh rendered virtual images onto display devices, allowing users to experience virtual reality / augmented reality effects through head-mounted displays. However, the rendering process takes time, creating a time delay between reality and perception. For example, during rendering, the user's head or the head-mounted device may move, causing a time delay between the user's head posture information and the image data output by the head-mounted device. If this time delay is too large, it can cause dizziness.

[0053] Based on this, please refer to Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the image rendering method of this application. In this embodiment, the image rendering method is applied to a head-mounted display device, and the method includes:

[0054] Step S10: Dynamically detect the user's head motion posture information, and obtain the pre-stored frame rate refresh time interval and maximum observation angle. Based on the head motion posture information, the frame rate refresh time interval and the maximum observation angle, predict the image frame edge of the prediction window image in the next time step.

[0055] In this embodiment, the head-mounted display device is worn on the user's head.

[0056] In this embodiment, the head motion posture information may include head displacement values ​​and angle change values, wherein the angle change values ​​may include pitch angle, yaw angle, and roll angle change values, which can be referred to as... Figure 4 .

[0057] It is easy to understand that the system dynamically detects the user's head posture information (i.e., the current head posture). The camera can be one or more of a TOF (Time of Flight) camera, an infrared camera, a millimeter-wave camera, and an ultrasonic camera. In another embodiment, the user's head posture information can be sent to the head-mounted display device in real time via another device that is communicatively connected to the head-mounted display device, thus completing the dynamic detection of this head posture information. For example, a camera installed in the activity area where the head-mounted display device is used can track and locate the head-mounted display device (or the user's head), obtain the user's head posture information, and send this head posture information to the head-mounted display device in real time, thereby enabling the head-mounted display device to acquire the dynamically detected head posture information in real time.

[0058] In this embodiment, as those skilled in the art will understand, for extended reality technology, in order to simulate changes in human visual perception in the real world and enhance the user's immersion in extended reality content, the field of view that the user can see is often different under different head posture information. The current field of view environment image refers to the field of view image that can be seen under the current head posture (different head posture information corresponds to different field of view environment images). That is to say, under the current head posture (i.e., a specific eye position), the maximum range of XR content images that the user can see is the user's current field of view environment image. Those skilled in the art will readily understand that during the content display process of a head-mounted display device, the user's head posture information can change in real time, and the head-mounted display device can collect or acquire the user's head posture information in real time to update the current field of view environment image.

[0059] In this embodiment, the frame rate refresh interval refers to the screen's frame rate refresh interval, which can characterize the refresh frequency of the head-mounted display device. For example, the refresh frequency of the head-mounted display device can be 60Hz, 120Hz, etc., that is, refreshed 60 times per second, 120 times per second, etc., and the frame rate refresh interval is 1 / 60 second, 1 / 120 second, etc. The maximum viewing angle refers to the user's FOV (Field of View) corresponding to the largest window image displayed by the head-mounted display device.

[0060] To aid understanding, let's consider an example: Taking a single-eye camera with a resolution of 1920*1080 as an example, and a dual-eye camera with a resolution of 3840*1080. Based on the detected head motion posture information A, the user's head rotation speed to the right is determined to be approximately 180° in 0.4 seconds, or 450° / s. At a 60fps refresh rate, the frame rate refresh interval is 16.6ms. This means that after a frame of data is displayed, the head can rotate 450° * 0.0166s = 7.47°, or approximately 1.86°. With a maximum viewing angle of 38°, this requires 1.86 / 38 * 1920 = 93 pixels. Reserving 93 pixels on each side of the normal camera margin is sufficient; any more would be meaningless. If the maximum angular velocity of each person is θ, then the rotation amount is (θ*(1 / FPS))°. Considering FPS, FOV, and image width, and the maximum head rotation angular velocity being 450° / s, the pixel value is Width*(θ*(1 / FPS))° / FOV. Based on this obtained pixel value, the size of the image frame edge of the prediction window image in the next time step can be predicted. Specifically, the predicted image frame edge can be obtained as follows: Figure 3As shown, frame blanktime is the frame blank time, frame length lines are the frame length lines, and effective pixel array is the total number of pixels. Because a faster frame rate requires less margin due to smaller frame intervals, camera margin is configured based on frame rate prediction to reduce unnecessary margins, reduce sensor output, and reduce power consumption.

[0061] Step S20: Execute the rendering thread to render the pixels within the edge of the image frame;

[0062] Step S30: If the pixels within the edge of the image frame have not been rendered by the preset time node before the next clock cycle arrives, then execute the asynchronous time warp thread to perform orientation warp processing on the most recently rendered current window image to obtain a transition window image.

[0063] It should be noted that the current viewport image refers to the largest area of ​​XR content image that the user can see after the most recent rendering, given the current head pose. This current head pose can include the current spatial position and angle of the head. The angle can include the pitch angle (based on the X-axis), the yaw angle (based on the Y-axis), and the roll angle (based on the Z-axis), such as... Figure 4 As shown.

[0064] Step S40: Display the transition window image until the pixel rendering within the edge of the image frame is completed, and then display the rendered prediction window image.

[0065] For example, after step S20, which involves executing a rendering thread to render pixels within the image frame edge, the method further includes:

[0066] Step A10: If the pixels within the edge of the image frame have not been rendered by the preset time node before the next clock cycle arrives, then the rendered prediction window image is displayed.

[0067] In this embodiment, to reduce rendering latency in the display scene, some virtual reality devices employ Timewarp (TW) technology. Timewarp is a technique that corrects image frames. Based on changes in user actions after rendering, it warps (or corrects) the rendered scene data to address rendering latency. Since the timewarp process is closer to the display time, the resulting image is closer to what the user expects. Furthermore, because timewarp only processes two-dimensional images, it's similar to affine transformations in image processing, minimizing system overhead. However, timewarp and rendering typically run in the same thread, leading to excessive thread processing time and hindering the resolution of image latency issues.

[0068] To address this issue, embodiments of this application propose an improved image rendering method that utilizes Asynchronous Timewarp (ATW). This method further optimizes image latency by employing asynchronous timewarp technology. Specifically, asynchronous timewarp technology further optimizes the aforementioned timewarp technique by scheduling rendering and timewarp in two separate threads. This allows the rendering and timewarp steps to be executed asynchronously, thereby reducing the overall runtime of rendering and timewarp processing. For example, when a virtual reality application cannot maintain a sufficient frame rate, the asynchronous timewarp thread reprocesses the previously rendered scene data based on the current user posture to generate frames (intermediate frames) that match the current user posture, thus reducing image jitter and further minimizing latency.

[0069] It should be noted that, ideally, the rendering engine uses pre-measured real-time head posture information (such as orientation and position information) before sending content to the user. However, in reality, the rendering process takes time, causing a time delay between the actual situation and the perceived view. This results in screen jitter, meaning the device cannot synchronously render the image corresponding to the head movements, naturally causing dizziness. This embodiment dynamically detects the user's head movement posture information and obtains pre-stored frame rate refresh intervals and maximum viewing angles. Based on these information, the frame rate refresh intervals and maximum viewing angles, it predicts the image frame edge of the prediction window image for the next time step and executes a rendering thread to render the pixels within the image frame edge. If the pixels within the image frame edge are not rendered by a preset time point before the next clock cycle arrives, an asynchronous time warp thread is executed to perform orientation warping on the most recently rendered current window image, resulting in a transition window. The image is displayed in a transition window until the pixels within the edge of the image frame are rendered. The rendered predicted window image is then displayed. By combining positional information prediction with asynchronous time warp technology, a virtual compensation frame (i.e., the transition window image) is generated and displayed. This can improve the image frame rate, reduce the delay in a certain stage of the process of acquiring and processing the image frame, and reduce the delay in displaying the image frame. It can effectively prevent the phenomenon of screen jitter due to the current frame not being fully rendered, thereby solving the technical problem of dizziness caused by time delay during the rendering of images in extended reality devices.

[0070] Furthermore, due to the current ultra-high resolution of extended reality application images, extended reality devices face a high rendering pressure on the images, which can easily lead to insufficient frame rate of the images displayed by the extended reality devices, resulting in stuttering and failing to meet users' requirements for smoothness.

[0071] Based on this, please refer to Figure 2 The step of executing the rendering thread to render the prediction window image includes:

[0072] Step S51: Detect the user's predicted eye observation point in the predicted window image, and based on the predicted eye observation point, divide the predicted window image into at least two partition images from near to far according to a preset partitioning rule.

[0073] In this embodiment, within the maximum range of XR content image visible to the user in the current head posture (i.e., the current viewport image), not all areas are areas of interest to the user; some areas are focused on by the user's eyes, while others are not. This embodiment detects the user's eye observation point within the current viewport image. Based on this eye observation point, it determines that areas closer to the eye observation point are areas of interest, while areas farther away are areas of non-interest. It's easy to understand that, generally, the closer an area is to the eye observation point, the higher the user's visual focus, and the more likely it is an area the user intends to observe; conversely, the farther away an area is, the lower the user's visual focus, and the more likely it is an area the user does not intend to observe.

[0074] In some embodiments, an eye detection device mounted on a head-mounted display device can acquire images of the user's eyes. Based on the eye feature information extracted from these images, the coordinates of the user's gaze point when looking at the display screen are calculated, thus obtaining the eye observation point of the current viewport image. This eye detection device can be a microelectro-mechanical system (MEMS), which includes an infrared scanning mirror, an infrared light source, and an infrared receiver. Currently, this eye detection device can be a capacitive sensor positioned in the user's eye area. Eye movement is detected by utilizing the capacitance between the eyeball and the capacitive plates of the sensor, thereby determining the user's current eye position information. Based on this current eye position information, the user's eye observation point in the current viewport image is determined. Alternatively, the eye detection device can also be an electromyography (EMG) detector. The EMG detector is connected to electrodes placed on the user's nose, forehead, ears, and earlobes. The electrodes collect EMG signals from these areas, and the detected EMG signal patterns are used to detect eye movements, determine the user's current eye position information, and then determine the user's eye observation point in the current viewport image based on the current eye position information.

[0075] Step S52: Render the partition image according to the rendering resolution corresponding to the partition image, wherein the rendering resolution of the partition image decreases sequentially from near to far.

[0076] In one embodiment, the user's head movement posture information can be dynamically detected using inertial sensors and / or cameras built into the head-mounted display device itself. In another embodiment, the user's head movement posture information can be sent to the head-mounted display device in real time via other devices that are communicatively connected to the head-mounted display device, thus completing the dynamic detection of the head movement posture information. For example, a camera installed in the activity venue where the head-mounted display device is used can track and locate the head-mounted display device (or the user's head), obtain the user's head movement posture information, and send the head movement posture information to the head-mounted display device in real time, thereby enabling the head-mounted display device to obtain the dynamically detected head movement posture information in real time.

[0077] It is known that users often make different head movements depending on the viewing angle of their intention. For example, when a user turns their head to the right, it usually means that the user wants to see the right side of the screen, and when a user turns their head to the left, it usually means that the user wants to see the left side of the screen.

[0078] It's easy to understand that within the maximum range of XR content image that a user can see under their current head posture (i.e., the current field of view), not all areas are areas of focus for the user. There are areas the user's eyes are focused on, and areas the user's eyes are not. Generally, areas that match the movement trend corresponding to the head posture tend to represent the areas the user intends to observe (if the user wants to see the right side of the screen, they will often turn their head to the right), while areas that do not match the movement trend tend to represent areas the user does not intend to observe.

[0079] Therefore, this embodiment detects the user's head movement posture information, and then determines the region in the prediction window image that matches the movement trend corresponding to the head movement posture as the region of eye attention (i.e., the region corresponding to the intended observation viewpoint) based on the head movement posture information, and the region in the prediction window image that does not match the movement trend corresponding to the head movement posture as the region of eye non-attention (i.e., the region corresponding to the non-intended observation viewpoint).

[0080] Research into user experience with images in head-mounted displays revealed that the rendering process for predictive field-of-view environments within head-mounted displays typically involves: transferring image data such as triangles and texture maps needed to render the scene image from the CPU (Central Processing Unit) to the GPU (Graphics Processing Unit); rendering the image data using the rendering pipeline to obtain an initial image; then applying post-processing techniques to colorize and render the initial image; and finally, obtaining an image of the current extended reality scene that can be displayed to the user.

[0081] Existing image rendering technologies generally render the image displayed on the entire target screen at a high rendering quality to meet the user's viewing requirements. However, if the area of ​​the user's eye that is primarily focused on within the field of view can only cover a portion of the screen, then under existing technologies, images displayed outside the area of ​​the eye's primary focus are also rendered at a high rendering quality, resulting in a waste of rendering resources.

[0082] With the rise of head-mounted display devices, people are increasingly using VR / AR devices. These augmented reality mobile terminals require a large amount of image rendering computation, resulting in high power consumption and significantly impacting battery life. If it's possible to identify areas where the user isn't actively focusing on a particular section of content, the rendering of that section can be reduced. For example, if it's detected that the user isn't paying attention to area A of the display screen, the quality of image rendering parameters such as pixel repair, noise reduction, and color interpolation for area A can be reduced, or even these rendering effects can be canceled altogether. This reduces the rendering resolution of area A, thereby reducing image processing workload and ultimately lowering power consumption.

[0083] Therefore, the technical solution of this embodiment is to detect the user's predicted eye observation point in the predicted view window image, and based on the predicted eye observation point, divide the predicted view window image into at least two partition images from near to far according to a preset partitioning rule. The partition images are then rendered according to the rendering resolution corresponding to the partition images, wherein the rendering resolution of the partition images decreases sequentially from near to far. This ensures that the rendering resolution of the image is high in the area that the eye is focused on, while the rendering resolution of the image is low in the area that the eye is not focused on (peripheral vision). Without affecting the user experience, or even improving the user experience, GPU resources are saved, achieving more reasonable rendering of the image. This avoids the waste of rendering resources caused by rendering images with higher quality outside the area that the eye is focused on, minimizes redundant rendering in the image rendering process, and thus reduces the GPU's rendering power consumption. While meeting the viewing needs of the human eye, it reduces the requirements of high-resolution and high-frame-rate content on the GPU's rendering capabilities.

[0084] As an example, in step S51, the step of dividing the predicted viewing window image into at least two partition images from near to far, based on the predicted eye observation point and according to a preset partitioning rule, includes:

[0085] Step B10: Determine the predicted gaze area based on the predicted eye observation point;

[0086] In this embodiment, eye-tracking technology can be used to acquire the user's eye image. Based on this image, the user's pupil center and spot position information (the spot being the bright reflection formed by the head-mounted display screen on the user's cornea) can be obtained. The user's predicted eye observation point is then determined based on this information. Finally, based on this predicted eye observation point, the user's predicted gaze area within the predicted viewing window image is determined. (Refer to...) Figure 6 The terminal can calculate the predicted gaze area based on the user's predicted eye observation point and visual field. For example, a circle with the gaze point as the center and a visual field of a preset radius is the user's predicted gaze area. Similarly, a polygon with the gaze point as the center and a visual field of a preset size can be used as the user's predicted gaze area. The shape of the predicted gaze area can be one of a circle, rectangle, square, or polygon.

[0087] It should be understood that the human eye perceives different areas of an image with varying clarity. Within the user's visible range (i.e., within the prediction window image), the image region that the eye is primarily focused on is highly sensitive and clearly imaged, while other image regions are perceived as blurry. The image corresponding to the predicted fixation area represents the region that the user's eye is primarily focused on, while the images corresponding to other parts of the prediction window image represent other image regions that the user's eye is not focused on.

[0088] Step B20: Divide the intention observation image within the predicted gaze area in the predicted viewport image into a first partition image;

[0089] In this embodiment, the first partition image is the main area viewed by the user, and should be rendered at a relatively large rendering resolution. The adjustment parameters of the rendering resolution include, but are not limited to, adjustments to image color, resolution, pixels, lighting effects, and shadow effects.

[0090] Step B30: Divide the non-intended observation image outside the predicted gaze area in the predicted viewport image into a second partition image;

[0091] Step S52, which involves rendering the partition image according to the rendering resolution corresponding to the partition image, includes:

[0092] Step B40: Render the first partition image at a first rendering resolution and render the second partition image at a second rendering resolution, wherein the first rendering resolution is greater than the second rendering resolution.

[0093] In this embodiment, the first partition image can be rendered first, and then the second partition image can be rendered. If the technology allows, the first partition image and the second partition image can also be rendered simultaneously.

[0094] This embodiment determines the predicted gaze area based on the predicted eye observation point, and divides the intended observation image within the predicted gaze area of ​​the predicted view window image into a first partition image, and the non-intended observation image outside the predicted gaze area of ​​the predicted view window image into a second partition image. Then, the first partition image is rendered at a first rendering resolution, and the second partition image is rendered at a second rendering resolution. The first rendering resolution is greater than the second rendering resolution, thereby reducing the occupancy of rendering resources in the non-gaze area, improving the high definition and high refresh rate of the gaze area image. This results in a high rendering resolution in the area the eye is focused on, and a low rendering resolution in areas the eye is not focused on (peripheral vision). Without affecting the user experience, or even improving it, GPU resources are saved, achieving more reasonable image rendering. This avoids the waste of rendering resources caused by high rendering quality of images outside the gaze area, minimizes redundant rendering during image rendering, and reduces GPU power consumption for image rendering. Through this method, hardware resources are rationally allocated to areas that require focused rendering, improving the image display effect in the eye-focused area, while also mitigating image blurring and stuttering, thus enhancing the user viewing experience.

[0095] Furthermore, the user's intended viewing angle can be determined by detecting the user's predicted eye observation point in the predicted viewpoint image and by considering the predicted eye observation point and the head turning trend.

[0096] In this embodiment, an eye detection device mounted on a head-mounted display device can acquire images of the user's eyes. Based on the eye feature information extracted from these images, the coordinates of the user's gaze point when looking at the display screen are calculated, thus obtaining the eye observation point for the predicted viewport image. This eye detection device can be a microelectro-mechanical system (MEMS), which includes an infrared scanning mirror, an infrared light source, and an infrared receiver. Currently, this eye detection device can be a capacitive sensor positioned in the user's eye area. It uses the capacitance between the eyeball and the capacitive plates of the sensor to detect eye movement, determine the user's current eye position and eye movement trend, and then determine the user's predicted eye observation point in the predicted viewport image based on this current eye position information. Alternatively, the eye detection device can also be an electromyography (EMG) detector. The EMG detector is connected to electrodes placed on the user's nose, forehead, ears, and earlobes. The electrodes collect EMG signals from these areas, and the detected EMG signal patterns are used to detect eye movements, determine the user's current eye position information and eye movement trend, and then determine the user's predicted eye observation point in relation to the predicted view window image based on the current eye position information and eye movement trend.

[0097] To aid understanding, consider this example: when a user turns their head to the right, they most likely want to see the right side of the screen. In this case, combined with gaze point recognition—that is, the predicted gaze point (predicted eye observation point) identified by the eye detection device—if it is also located in the right-hand screen (the gaze point is to the right of the display midline), this often indicates that the user wants to see the right-hand screen. In this situation, the intended viewing angle is the viewing angle included in the right-hand screen. Figure 5 As shown, the rendering resolution of the leftmost 1 / a section of the screen can be reduced to decrease the image rendering load / pressure on the head-mounted display device. For example, if a user turns their head to the left, they are likely hoping to see the left side of the screen. If the gaze point detected by the eye detection device is also on the left side of the screen (the gaze point is to the right of the display center line), this usually indicates that the user wants to see the left side of the screen. In this case, the intended viewing angle is the viewing angle included in the left side of the screen. The rendering resolution of the rightmost 1 / a section of the screen can be reduced to decrease the image rendering load / pressure on the head-mounted display device. It should be noted that 'a' is generally greater than 2, for example, a equals 3 or 4. In one example, a equals 4, meaning that if the user turns their head to the left, the rendering resolution of the rightmost 1 / 4 section of the screen is reduced. If the user turns their head to the right, the rendering resolution of the leftmost 1 / 4 section of the screen is reduced.

[0098] In one possible implementation, the step of detecting the user's predicted eye gaze point in the predicted viewport image includes:

[0099] Step C10: Collect the user's current eye image, determine the eye model with the highest matching degree to the current eye image, and use the eye model with the highest matching degree as the current actual eye model;

[0100] In this embodiment, the eye model with the highest matching degree to the current eye image can be identified by performing image recognition based on a preset image recognition algorithm on the current eye image. This preset image recognition algorithm has been extensively studied by those skilled in the art and will not be elaborated upon here.

[0101] Step C20: Query the eye observation point mapped by the current actual eye model from the preset eye model mapping database, and use the mapped eye observation point as the user's predicted eye observation point in the prediction window image.

[0102] Those skilled in the art will understand that different types of eyeball models (e.g., differences in exit pupil distance, pupil shape, pupil region location, and current pupil spot location in the eyeball model) often correspond to different eyeball observation points.

[0103] In this embodiment, the eye model mapping database stores information on multiple types of eye models, as well as a one-to-one mapping relationship between each eye model and an eye observation point.

[0104] This embodiment acquires the user's current eye image, determines the eye model with the highest matching degree to the current eye image, uses the eye model with the highest matching degree as the current actual eye model, and queries the eye observation point mapped by the current actual eye model from the preset eye model mapping database, thereby accurately obtaining the user's predicted eye observation point in the prediction window image.

[0105] In another possible implementation, detecting the user's eye gaze point in the current viewport image includes:

[0106] Step D10: Acquire the user's current eye image and perform grayscale processing on the current eye image;

[0107] In this embodiment, the user's current eye image can be captured by a camera mounted on the head-mounted display device.

[0108] Step D20: Determine the pupil region image based on the current eyeball image after grayscale processing, and perform binarization processing on the pupil region image;

[0109] Step D30: Perform edge detection on the binarized pupil region image to obtain pupil edge points, and fit the pupil edge points to an ellipse to obtain the current pupil center.

[0110] Step D40: Determine the user's predicted eye observation point in the predicted view window image based on the current pupil center.

[0111] As an example, the step of determining the user's eye observation point in the current viewport image based on the current pupil center includes:

[0112] Step E10: Query the eye observation point of the current pupil center mapping from the pre-calibrated pupil center mapping data table;

[0113] Those skilled in the art will understand that pupil centers at different locations often correspond to different eye observation points. It should be noted that the pre-defined pupil center mapping data table stores multiple pupil centers at different locations, as well as a one-to-one mapping relationship between each pupil center and an eye observation point.

[0114] Step E20: The mapped eye observation point is used as the user's predicted eye observation point in the predicted view window image.

[0115] This embodiment acquires the user's current eye image, performs grayscale processing on the current eye image, determines the pupil region image based on the grayscale processed current eye image, performs binarization processing on the pupil region image, performs edge detection on the binarized pupil region image to obtain pupil edge points, performs ellipse fitting on the pupil edge points to obtain the current pupil center, and then accurately obtains the user's predicted eye observation point in the prediction window image based on the current pupil center.

[0116] Example 2

[0117] Based on the above embodiments of this application, in another embodiment of this application, the same or similar content as in Embodiment 1 can be referred to the above description and will not be repeated hereafter. In addition, after the step of dividing the prediction window image into at least two partition images from near to far, the method further includes:

[0118] Step F10: Control the brightness display of the partitioned image according to the backlight brightness of the corresponding area of ​​the partitioned image, wherein the backlight brightness of the area of ​​the partitioned image decreases sequentially from near to far.

[0119] Existing image rendering technologies typically use a high area backlight brightness to control the brightness of the image displayed on the entire target screen to meet the user's viewing requirements. However, if the area that the user's eyes are primarily focused on within the field of view can only cover a portion of the screen, then under existing technologies, the image displayed outside the area that the eyes are primarily focused on is also illuminated with a high area backlight brightness, resulting in wasted power consumption.

[0120] Therefore, the technical solution of this embodiment is to control the brightness display of the partitioned image according to the backlight brightness of the corresponding area of ​​the partitioned image. The backlight brightness of the partitioned image decreases from near to far. This makes the image have high backlight brightness in the area that the eye is focused on, and low backlight brightness in the area that the eye is not focused on (peripheral vision). Without affecting the user experience, or even improving the user experience, it saves display energy consumption and achieves more reasonable brightness display control of the image, avoiding the power consumption waste caused by high backlight brightness of the image outside the area that the eye is focused on.

[0121] In one possible implementation, the step of dividing the current viewport image into at least two partition images from near to far, based on the predicted eye observation point and according to a preset partitioning rule, includes:

[0122] Step G10: Determine the predicted gaze area based on the predicted eye observation point;

[0123] Step G20: Divide the intention observation image within the predicted gaze area in the predicted viewport image into a first partition image;

[0124] In this embodiment, the first partition image is the main area that the user views, and a larger area backlight brightness should be arranged for it.

[0125] Step S43: Divide the non-intended observation image outside the predicted gaze area in the predicted view window image into a second partition image;

[0126] In this embodiment, the second partition image is a non-primary area for the user to view, and should be arranged with lower area backlight brightness.

[0127] The step of controlling the brightness display of the partitioned image according to the backlight brightness of the corresponding area of ​​the partitioned image includes:

[0128] Step G30: The brightness display of the first partition image is controlled by the backlight brightness of the first region, and the brightness display of the second partition image is controlled by the backlight brightness of the second region, wherein the backlight brightness of the first region is greater than the backlight brightness of the second region.

[0129] This embodiment determines the predicted gaze area based on the predicted eye observation point, and divides the intended gaze image within the predicted gaze area in the viewport image into a first partition image, and the non-intended gaze image outside the predicted gaze area in the predicted viewport image into a second partition image. Then, the brightness display of the first partition image is controlled by the backlight brightness of the first region, and the brightness display of the second partition image is controlled by the backlight brightness of the second region. The backlight brightness of the first region is greater than that of the second region, thereby reducing the waste of brightness power consumption in the non-gaze area, improving the brightness and clarity of the image in the gaze area, so that the image has high backlight brightness in the area that the eye is focused on, and low backlight brightness in the area that the eye is not focused on (peripheral vision). Without affecting the user experience, or even improving the user experience, it saves display energy consumption, achieves more reasonable brightness display control of the image, and avoids the energy waste caused by high display brightness of the image outside the eye's gaze area.

[0130] Example 3

[0131] This invention provides a head-mounted display device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, which are executed by the at least one processor to enable the at least one processor to perform the image rendering method in the first embodiment described above.

[0132] The following is for reference. Figure 7 The diagram illustrates a structural schematic suitable for implementing a head-mounted display device according to embodiments of the present disclosure. The head-mounted display device in embodiments of the present disclosure may include, but is not limited to, Mixed Reality (MR) devices, Augmented Reality (AR) devices, Virtual Reality (VR) devices, Extended Reality (XR) devices, or some combination thereof. Figure 7 The head-mounted display device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0133] like Figure 7 As shown, the head-mounted display device may include a processing unit 1001 (e.g., a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM 1002) or a program loaded from a storage device into a random access memory (RAM 1004). The RAM 1004 also stores various programs and data required for the operation of the AR glasses. The processing unit 1001, ROM 1002, and RAM 1004 are interconnected via a bus 1005. Input / output (I / O) interfaces are also connected to the bus 1005.

[0134] Typically, the following systems can be connected to I / O interface 1006: input devices 1007 including, for example, touchscreens, touchpads, keyboards, mice, image sensors, microphones, accelerometers, gyroscopes, etc.; output devices 1008 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 1003 including, for example, magnetic tapes, hard disks, etc.; and communication devices 1009. Communication device 1009 allows the AR glasses to communicate wirelessly or wiredly with other devices to exchange data. Although AR glasses with various systems are shown in the figure, it should be understood that it is not required to implement or possess all the systems shown. More or fewer systems can be implemented alternatively.

[0135] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from ROM 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of embodiments of this disclosure.

[0136] The head-mounted display device provided by this invention, employing the image rendering method described in Embodiment 1 or Embodiment 2 above, can solve the technical problem of dizziness caused by time delays during image rendering in extended reality devices. Compared with the prior art, the beneficial effects of the head-mounted display device provided by this invention are the same as those of the image rendering method provided in Embodiment 1 above, and other technical features of this head-mounted display device are the same as those disclosed in the previous embodiment method, and will not be repeated here.

[0137] It should be understood that various parts of this disclosure can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics may be combined in any suitable manner in one or more embodiments or examples.

[0138] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

[0139] Example 4

[0140] This invention provides a computer-readable storage medium having computer-readable program instructions stored thereon, which are used to execute the image rendering method in the first embodiment described above.

[0141] The computer-readable storage medium provided in this embodiment of the invention may be, for example, a USB flash drive, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination thereof.

[0142] The aforementioned computer-readable storage medium may be included in the head-mounted display device; or it may exist independently and not assembled into the head-mounted display device.

[0143] The aforementioned computer-readable storage medium carries one or more programs that, when executed by a head-mounted display device, cause the head-mounted display device to: dynamically detect the user's head motion posture information and obtain a pre-stored frame rate refresh interval and maximum viewing angle; predict the image frame edge of the prediction window image for the next time step based on the head motion posture information, the frame rate refresh interval, and the maximum viewing angle; execute a rendering thread to render the pixels within the image frame edge; if the pixels within the image frame edge have not been rendered by a preset time node before the next clock cycle arrives, execute an asynchronous time warp thread to perform orientation warping processing on the most recently rendered current window image to obtain a transition window image; display the transition window image until the pixels within the image frame edge are rendered, and then display the rendered prediction window image.

[0144] Computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0145] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0146] The modules described in the embodiments of this disclosure can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0147] The computer-readable storage medium provided by this invention stores computer-readable program instructions for executing the above-described image rendering method, which can solve the technical problem of dizziness caused by time delays during image rendering in extended reality devices. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided by the embodiments of this invention are the same as the beneficial effects of the image rendering method provided in Embodiment 1 or Embodiment 2, and will not be repeated here.

[0148] Example 5

[0149] This invention also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the image rendering method described above.

[0150] The computer program product provided in this application can solve the technical problem of dizziness caused by time delays during image rendering in extended reality devices. Compared with the prior art, the beneficial effects of the computer program product provided in this embodiment are the same as those of the image rendering methods provided in Embodiment 1 or Embodiment 2 above, and will not be repeated here.

[0151] The above are merely preferred embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent scope of this application.

Claims

1. An image rendering method, characterized in that, The image rendering method is applied to a head-mounted display device, and the method includes: The system dynamically detects the user's head movement posture information, obtains the pre-stored frame rate refresh time interval and maximum observation angle, and predicts the image frame edge of the prediction window image in the next time step based on the head movement posture information, the frame rate refresh time interval and the maximum observation angle. Execute the rendering thread to render the pixels within the edges of the image frame; If the pixels within the edge of the image frame are not rendered by the preset time node before the next clock cycle arrives, an asynchronous time warp thread is executed to perform orientation warp processing on the most recently rendered current window image to obtain a transition window image. The transition window image is displayed until the pixels within the edge of the image frame are rendered, and the rendered prediction window image is then displayed. The step of predicting the image frame edge of the prediction window image for the next time step based on the head motion posture information, the frame rate refresh time interval, and the maximum viewing angle includes: The angular velocity of the user's head is determined based on head motion posture information. Then, based on this angular velocity, the frame rate refresh interval, and the maximum viewing angle, the formula Width is used. (θ (1 / FPS))° / FOV is used to obtain reserved pixels, and then the image frame edge of the prediction window image in the next time step is predicted based on the obtained pixels, where Width is the image width, θ is the angular velocity, FOV is the maximum observation angle, and 1 / FPS is the frame rate refresh time interval.

2. The image rendering method as described in claim 1, characterized in that, After the step of executing the rendering thread to render pixels within the image frame edge, the method further includes: If the pixels within the edge of the image frame have been rendered by a preset time node before the next clock cycle arrives, the rendered prediction window image will be displayed.

3. The image rendering method as described in claim 1, characterized in that, The step of executing the rendering thread to render the prediction window image includes: The system detects the user's predicted eye observation point in the predicted view window image, and uses the predicted eye observation point as a reference to divide the predicted view window image into at least two partition images from near to far according to a preset partitioning rule. The partitioned image is rendered according to the rendering resolution corresponding to the partitioned image, wherein the rendering resolution of the partitioned image decreases sequentially from near to far.

4. The image rendering method as described in claim 3, characterized in that, The step of detecting the user's predicted eye observation point in the predicted viewport image includes: Collect the user's current eye image, determine the eye model with the highest matching degree with the current eye image, and use the eye model with the highest matching degree as the current actual eye model; The eye observation point mapped by the current actual eye model is obtained from the preset eye model mapping database, and the mapped eye observation point is used as the user's predicted eye observation point in the prediction window image.

5. The image rendering method as described in claim 3, characterized in that, The detection of the user's predicted eye observation point in the predicted viewpoint image includes: Acquire the user's current eye image and perform grayscale processing on the current eye image; The pupil region image is determined based on the current eyeball image after grayscale processing, and the pupil region image is then binarized. Edge detection is performed on the binarized pupil region image to obtain pupil edge points, and then ellipse fitting is performed on the pupil edge points to obtain the current pupil center. The predicted eye observation point of the user in the predicted view window image is determined based on the current pupil center.

6. The image rendering method as described in claim 5, characterized in that, The step of determining the user's predicted eye observation point in the predicted view window image based on the current pupil center includes: From the pre-calibrated pupil center mapping data table, retrieve the predicted eye observation point of the current pupil center mapping; The mapped eye observation point is used as the user's predicted eye observation point in the prediction window image.

7. The image rendering method as described in claim 3, characterized in that, The step of dividing the prediction window image into at least two partition images from near to far also includes: The brightness display of the partitioned image is controlled according to the backlight brightness of the corresponding area of ​​the partitioned image, wherein the backlight brightness of the area of ​​the partitioned image decreases sequentially from near to far.

8. The image rendering method as described in claim 7, characterized in that, The step of dividing the predicted visual window image into at least two partition images from near to far, based on the predicted eye observation point and according to a preset partitioning rule, includes: Based on the predicted eye observation point, determine the predicted fixation area; The intention observation image within the predicted gaze area in the predicted viewport image is divided into a first partition image; The non-intended observation image outside the predicted gaze region in the predicted viewport image is divided into a second partition image; The step of controlling the brightness display of the partitioned image according to the backlight brightness of the corresponding area of ​​the partitioned image includes: The brightness display of the first partition image is controlled by the backlight brightness of the first region, and the brightness display of the second partition image is controlled by the backlight brightness of the second region, wherein the backlight brightness of the first region is greater than the backlight brightness of the second region.

9. A head-mounted display device, characterized in that, The head-mounted display device includes: At least one processor; and, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the steps of the image rendering method according to any one of claims 1 to 8.

10. A readable storage medium, characterized in that, The readable storage medium is a computer-readable storage medium, on which a program implementing the image rendering method is stored, and the program implementing the image rendering method is executed by a processor to implement the steps of the image rendering method as described in any one of claims 1 to 8.

Images