Image rendering method, readable medium, electronic device and program product

Abstract

The invention relates to the technical field of image processing, and discloses an image rendering method, a storage medium, electronic equipment and a program product.The method comprises the steps that in the process that the electronic equipment carries out image rendering on a to-be-processed image (such as a virtual image), firstly, a fixation point corresponding to the to-be-processed image is determined; and performing image division on the to-be-processed image based on the fixation point to obtain a fixation area and a non-fixation area, and performing different rendering processing on the fixation area and the non-fixation area. For example, the electronic equipment can perform image rendering on the gaze in the to-be-processed image by adopting a rendering mode with relatively high rendering quality and relatively high processing complexity, and perform image rendering on the non-gaze area in the to-be-processed image by adopting a rendering mode with relatively low rendering quality and relatively low processing complexity. Therefore, the rendering effect of the gazing area can be ensured, and the data processing amount and the equipment power consumption can be reduced.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to an image rendering method, a readable medium, an electronic device, and a program product. Background Technology

[0002] Virtual reality (VR) devices, augmented reality (AR) devices, and mixed reality (MR) devices, as well as other extended reality (XR) devices, typically render the displayed images to enhance the visual effect presented to the user, such as a realistic effect.

[0003] However, image rendering typically involves processing large amounts of data, resulting in high device power consumption. Summary of the Invention

[0004] This invention provides an image rendering method, a storage medium, an electronic device, and a program product.

[0005] In a first aspect, the present invention provides an image rendering method applied to an electronic device. The method includes: determining a gaze point of a virtual image to be displayed; determining a first region and a second region of the virtual image based on the gaze point, wherein the gaze point is located in the first region and the second region does not include the gaze point; performing a first rendering process on the first region to obtain a first rendering result, and performing a second rendering process on the second region to obtain a second rendering result, wherein the processing parameters of the first rendering process and the processing parameters of the second rendering process are different; determining a rendered image of the corresponding virtual image based on the first rendering result and the second rendering result; and displaying the rendered image.

[0006] In this embodiment of the invention, the image is divided into a gaze region (an instance of the first region) and a non-gaze region (an instance of the second region) based on the gaze point. Different rendering processes are applied to the gaze region and the non-gaze region. For example, smart glasses (an instance of an electronic device) can use a rendering method with higher rendering quality and higher processing complexity for the gaze region in the image to be processed, and use a rendering method with lower rendering quality and lower processing complexity for the non-gaze region in the image to be processed. In this way, the rendering effect of the gaze region can be ensured, while reducing the amount of data processing and device power consumption.

[0007] In one possible implementation of the first aspect above, corresponding to both the first rendering process and the second rendering process employing ray tracing technology, the processing parameters include: sampling density and / or ray tracing counts, wherein the sampling density of the first rendering process is greater than the sampling density of the second rendering process, and the ray tracing counts of the first rendering process are greater than the ray tracing counts of the second rendering process; corresponding to the first rendering process employing ray tracing technology, the second rendering process employing image-based lighting technology, the processing parameters of the first rendering process include the processing parameters corresponding to the ray tracing technology, and the second rendering process includes the processing parameters corresponding to the image-based lighting technology.

[0008] In one possible implementation of the first aspect described above, both the first rendering process and the second rendering process employ ray tracing technology; the first rendering process is performed on the first region to obtain a first rendering result, and the second rendering process is performed on the second region to obtain a second rendering result, including: performing a first sampling on a plurality of first pixels in the first region to obtain a plurality of first sampling points; performing a first rendering process on the plurality of first pixels in the first region based on the plurality of first sampling points to obtain a first rendering result; performing a second sampling on a plurality of second pixels in the second region to obtain a plurality of second sampling points; and performing a second rendering process on the plurality of second pixels in the second region based on the plurality of second sampling points to obtain a second rendering result; wherein the sampling density of the first sampling is greater than the sampling density of the second sampling.

[0009] In one possible implementation of the first aspect above, a first rendering result is obtained by performing a first rendering process on a plurality of first pixels corresponding to a plurality of first sampling points in a first region based on a plurality of first sampling points, including: determining a first intermediate result for performing the first rendering process on a plurality of first sampling points; and performing rendering process on at least one first pixel that satisfies a first distance condition based on the first intermediate result of each first sampling point to obtain the first rendering result.

[0010] In one possible implementation of the first aspect above, a second rendering result is obtained by performing a second rendering process on a plurality of second pixels corresponding to a plurality of second sampling points in a second region based on a plurality of second sampling points, including: determining a second intermediate result for performing a second rendering process on a plurality of second sampling points; and, based on the second intermediate result of each second sampling point, performing rendering process on a plurality of second pixels for each second sampling point that satisfies a second distance condition to obtain a second rendering result.

[0011] In one possible implementation of the first aspect above, the method further includes: the number of ray tracing operations in the first rendering process is greater than the number of ray tracing operations in the second rendering process.

[0012] In one possible implementation of the first aspect described above, the method further includes: a second region surrounding the first region in a ring shape.

[0013] In one possible implementation of the first aspect above, the method further includes: determining a third region of the virtual image, wherein the third region is located between the first region and the second region; determining a first distance between a plurality of third pixels in the third region and the first region, and a second distance between the plurality of third pixels and the second region; performing a third rendering process on the plurality of third pixels based on the relationship between the first distance and the second distance to obtain a third rendering result; and determining a rendered image of the corresponding virtual image based on the first rendering result, the second rendering result, and the third rendering result.

[0014] In this embodiment of the invention, a visual transition between the fixation area and the non-fixation area is performed based on the transition area (an instance of the third region) to avoid the feeling of visual segmentation.

[0015] In one possible implementation of the first aspect above, a third rendering process is performed on multiple third pixels based on the relationship between the first distance and the second distance to obtain a third rendering result, including: when the first distance is greater than or equal to the second distance, the third pixel is processed by the third rendering process based on the first rendering process to obtain a third rendering result; when the first distance is less than the second distance, the third pixel is processed by the third rendering process based on the second rendering process to obtain a third rendering result.

[0016] In one possible implementation of the first aspect above, a third rendering process is performed on multiple third pixels based on the size relationship between the first distance and the second distance to obtain a third rendering result, including: determining a first ratio of the first distance to the first distance and the second distance, and a second ratio of the second distance to the first distance and the second distance; determining a first result of the first rendering process corresponding to the third pixel, and a second result of the second rendering process corresponding to the third pixel; and determining a third rendering result of the third rendering process corresponding to the third pixel based on the first result of the first ratio and the second result of the second ratio.

[0017] In one possible implementation of the first aspect described above, the method further includes: a third region surrounding the first region in a ring shape, and a second region surrounding the third region in a ring shape.

[0018] In a second aspect, embodiments of the present invention provide a computer-readable storage medium storing instructions that, when executed on an electronic device, cause the electronic device to implement any of the image rendering methods provided by the first aspect and various possible implementations of the first aspect.

[0019] Thirdly, embodiments of the present invention provide an electronic device comprising: a memory for storing instructions executed by one or more processors of the electronic device; and a processor, one of the processors of the electronic device, for executing the instructions stored in the memory to implement any of the image rendering methods provided by the first aspect and various possible implementations of the first aspect.

[0020] Fourthly, embodiments of the present invention provide a program product including instructions that, when executed by an electronic device, enable the electronic device to implement any of the image rendering methods provided by the first aspect and various possible implementations of the first aspect. Attached Figure Description

[0021] Figure 1 According to some embodiments of this application, an example schematic diagram of smart glasses is shown;

[0022] Figure 2 According to some embodiments of this application, an example schematic diagram of ray tracing is shown;

[0023] Figure 3A According to some embodiments of this application, a schematic diagram of an image 30 to be processed is shown;

[0024] Figure 3B According to some embodiments of this application, a schematic diagram of pixel region division is shown;

[0025] Figure 3C According to some embodiments of this application, a schematic diagram of another pixel region division is shown;

[0026] Figure 4 According to some embodiments of this application, a flowchart of an image rendering method is shown;

[0027] Figure 5 According to some embodiments of this application, a schematic diagram of an image 50 to be processed is shown;

[0028] Figure 6 According to some embodiments of this application, a flowchart of an image rendering method is shown;

[0029] Figure 7 According to some embodiments of this application, a structural schematic diagram of a smart glasses 100 is shown. Detailed Implementation

[0030] The illustrative embodiments of the present invention include, but are not limited to, an image rendering method, a storage medium, an electronic device, and a program product.

[0031] The following is in conjunction with the appendix Figures 1 to 7The technical solution of the present invention will be described.

[0032] Figure 1 According to some embodiments of this application, an example schematic diagram of smart glasses is shown.

[0033] For example, such as Figure 1 As shown, when a user wears an XR device such as smart glasses, the smart glasses can display virtual images; alternatively, the smart glasses can display virtual images while the user is viewing a real-world scene, and the user can interact with the virtual images, achieving a combination of reality and virtuality. For example, in the movie example, users can watch virtual scenery by wearing smart glasses, achieving an immersive movie experience. As another example, in the game example, users can interact with virtual pets in a real-world scene by wearing smart glasses, achieving an immersive gaming experience, and so on.

[0034] It is understood that smart glasses are merely one example of an XR device. In other embodiments, the XR device may also be other head-mounted display (HMD) devices such as smart helmets, or other electronic devices (such as terminals and servers) connected to the head-mounted display device. For example, terminals include, but are not limited to, laptops, mobile phones, etc., and servers include, but are not limited to, standalone physical servers, server clusters consisting of one or more physical servers, etc., without specific limitations.

[0035] The following uses smart glasses as an example to illustrate the embodiments of this application.

[0036] In some embodiments, the virtual images that smart glasses can display include any or a combination of virtual subjects and virtual scenes. For example, when a user wears smart glasses and is in a real-world setting such as a bedroom, the smart glasses can display a virtual image of a virtual cat subject, allowing the user to see both the real bedroom and the virtual cat simultaneously. Or, for example, when a user wears smart glasses, the smart glasses can display virtual images of a virtual cat subject and a virtual forest scene, allowing the user to see both a virtual cat and a virtual forest simultaneously, etc., without any specific limitation.

[0037] In some embodiments, smart glasses typically perform image rendering on the displayed virtual images to enhance their visual appeal. For example, in an example where a user wearing smart glasses can simultaneously see a real bedroom and a virtual cat, when the bedroom lights are on, the light not only directly illuminates objects such as walls, ceiling, and floor, but also undergoes multiple reflections, refractions, and diffuse reflections between these objects. The direct illumination and reflection of light alter the color and brightness of the objects in the bedroom. Thus, the smart glasses can use techniques such as ray tracing to simulate the propagation and interaction of light in the bedroom (e.g., direct illumination and reflection) to obtain ray tracing results. Based on these results, the smart glasses can then render the virtual image of the cat, ensuring that the virtual image of the cat displayed by the smart glasses has similar or identical lighting effects to the bedroom, making the virtual cat appear more realistic to the user.

[0038] Understandably, smart glasses can also use ray tracing technology to simulate the ray tracing results of light in virtual scenes, making the virtual scenes appear more realistic to the user. For example, smart glasses can use ray tracing technology to simulate the ray tracing results of light in a virtual forest scene, and then render the virtual image of the forest scene based on the ray tracing results, making the virtual forest appear more realistic to the user. As another example, smart glasses can use ray tracing technology to simulate the ray tracing results of light in a virtual forest scene, and then render the virtual images of the forest scene and the virtual cat subject based on the ray tracing results, making the virtual forest and the virtual cat appear more realistic to the user.

[0039] The following example illustrates the image rendering process of smart glasses using ray tracing technology and real-world scenarios.

[0040] In some embodiments, the image rendering process using ray tracing technology in smart glasses involves tracing every ray of light emitted from the viewpoint, projected onto the real-world scene via the smart glasses' display screen (hereinafter referred to as the glasses display screen), and tracking the multiple bounces of each ray in the real-world scene, including reflections and refractions. For example, in an example where a user can simultaneously see a real bedroom and a virtual cat while wearing smart glasses, the smart glasses, through ray tracing technology, need to track the ray tracing results of every ray projected from the user's eye position (viewpoint) onto the real-world scene, as well as the multiple bounces of each ray in the real-world scene. The ray tracing results are then used to render the virtual image to be displayed on the glasses display screen, ensuring a realistic visual effect for the user.

[0041] In some embodiments, such as Figure 2As shown, the smart glasses are equipped with a virtual camera corresponding to the user's eye position. The smart glasses emit light from the virtual camera position. After the light passes through the pixel A of the virtual image to be displayed on the glasses screen, it intersects with an object in the real scene (such as the bedroom floor) at the first intersection point. At this time, the first intersection point has a color component relative to the influence of the light source. The color value of pixel A, such as red, green, and blue (RGB) value, is rendered through this color component.

[0042] Furthermore, after passing through the first intersection point, light rays typically bounce multiple times in a real-world scene, resulting in several different intersection points. The first intersection point usually has different color components relative to these different intersection points, and each color component influences the color value of pixel A. For example, see [link to example]. Figure 2 As shown, after the light passes through the first intersection point, it reflects and intersects with another object in the real scene (such as the bedroom wall) at the second intersection point. At this time, the first intersection point has two different color components relative to the light source and the second intersection point. Both color components affect the color value of pixel A. That is, the smart glasses need to render the color value of pixel A through these two color components.

[0043] It is understandable that each bounce of light from pixel A onto the real scene will affect the color value of pixel A; that is, for each bounce of light, the smart glasses need to determine the color component of the first intersection point for that bounce, and finally map the color components of the multiple first intersection points obtained from multiple bounces to pixel A to complete the rendering of pixel A.

[0044] In this way, the smart glasses use ray tracing technology to render every pixel in the virtual image that needs to be displayed on the glasses screen, so as to complete the rendering of the virtual image. The rendered image is then displayed on the glasses screen, ensuring that the rendered image has similar or the same lighting effect as the real scene. This improves the realism of the virtual image to the user's vision when the user wears the smart glasses and sees both the real scene and the virtual image at the same time.

[0045] As mentioned earlier, during image rendering using ray tracing technology, smart glasses typically need to track every ray of light projected from the viewpoint through every pixel on the glasses' display screen into the real-world scene, as well as the multiple bounces of each ray. This involves a large amount of data processing and high device power consumption. In particular, for real-time scenes, smart glasses need to render images in response to real-time changes in the scene, further increasing power consumption. For example, in the scenario where a user is simultaneously viewing a real-world scene and a virtual image while wearing smart glasses, as the user moves or rotates, causing a change in position, the real-world scene the user sees will change in real-time. Therefore, the smart glasses need to perform real-time image rendering of the virtual image to correspond to these changes, resulting in high power consumption.

[0046] To address this, the present invention proposes an image rendering method. During the image rendering process of an image to be processed (such as a virtual image), smart glasses first determine the gaze point corresponding to the image to be processed. Based on the gaze point, the image to be processed is divided into a gaze region (an instance corresponding to the first region) and a non-gaze region (an instance corresponding to the second region). Different rendering processes are applied to the gaze region and the non-gaze region. For example, the smart glasses can use a rendering method with higher rendering quality and higher processing complexity for the gaze region of the image to be processed, and use a rendering method with lower rendering quality and lower processing complexity for the non-gaze region of the image to be processed. In this way, the rendering effect of the gaze region can be ensured while reducing data processing volume and device power consumption.

[0047] For example, smart glasses may use different sampling densities when sampling pixels in the gaze area and non-gaze area. For instance, the sampling density of pixels in the gaze area may be greater than that of pixels in the non-gaze area, and the smart glasses may only perform ray tracing on the sampled points. Alternatively, the number of ray tracing operations may differ between the gaze area and non-gaze area. For instance, the number of ray tracing operations on a single sampled point in the gaze area may be greater than that on a single sampled point in the non-gaze area. Or, for instance, smart glasses may use ray tracing technology to render pixels in the gaze area, while using lightweight rendering techniques such as image-based lighting (IBL) to render pixels in the non-gaze area.

[0048] It's understandable that when smart glasses display images, the concentration of cone cells on the retina responsible for observing color and detail varies. The human eye typically focuses more on image details at the point of fixation, while details further away from the fixation point gradually become blurred. Therefore, using different rendering processes for the fixational and non-fixational areas reduces the amount of data the smart glasses need to process for rendering pixels in the non-fixational area. This not only reduces device power consumption but also doesn't affect the visual experience for the user.

[0049] In some embodiments, the smart glasses can further segment the image to be processed to create a transition region (corresponding to an instance of the third region) between the gaze area and the non-gaze area, and perform a visual transition between the gaze area and the non-gaze area based on the transition region. For example, during the rendering process of the gaze area, transition region, and non-gaze area, the smart glasses sequentially reduce the sampling density of pixels in the gaze area, transition region, and non-gaze area, the number of ray tracings per sampling point, etc., to progressively reduce the rendering effect. Or, for example, during the rendering process of the transition region, the smart glasses can determine the rendering result of the corresponding pixels in the transition region based on the distance of the pixels in the transition region from the gaze area and the non-gaze area, so that the pixels in the transition region closer to the gaze area have a more similar visual effect to the pixels in the gaze area, and the pixels in the transition region closer to the non-gaze area have a more similar visual effect to the pixels in the non-gaze area, ensuring a visual transition between the gaze area and the non-gaze area. It is understood that because the smart glasses use different rendering processes for the gaze area and the non-gaze area, a visual segmentation may occur at the boundary between the gaze area and the non-gaze area. In this way, smart glasses use a transition zone to visually transition between the fixation area and the non-fixation area, avoiding the feeling of visual segmentation.

[0050] Example 1

[0051] In Embodiment 1, the smart glasses divide the image to be processed into a gaze region and a non-gaze region. Different rendering processes are applied to the gaze region and the non-gaze region to reduce the amount of data processed by the smart glasses for rendering pixels in the non-gaze region, thereby reducing device power consumption. Embodiment 1 is illustrated below.

[0052] The following example illustrates the process by which smart glasses determine the gaze point of the image to be processed.

[0053] In some embodiments, smart glasses can determine the gaze point based on the user's eye position when looking at the glasses' display screen after the user wears the glasses. For example, the user's eyes may include areas such as the pupil, iris, sclera, upper eyelid, and lower eyelid. The smart glasses can determine the pupil area of ​​the user's eyes using recognition methods such as infrared thermal imaging or eye-tracking algorithms, then determine the coordinates of the center point in the pupil area, and based on the coordinates of the center point, determine the coordinates of the gaze point corresponding to the image to be processed on the display screen, thereby determining the gaze point of the image to be processed.

[0054] In other embodiments, users can also customize the coordinates of the gaze point based on different needs. For example, if a user wants to focus more on the image details in the right area of ​​the display screen, the user can customize the coordinates of the gaze point to be closer to the right side of the display screen so that the user can adjust the viewing angle according to the coordinates of the gaze point, etc. There are no specific restrictions.

[0055] The following example illustrates the process by which smart glasses divide an image into fixational and non-fixational regions.

[0056] In some embodiments, after determining the gaze point of the image to be processed, the smart glasses can take the region containing multiple pixels (instances of the first pixel) in the image to be processed that satisfy a first distance relationship with the gaze point as the gaze region, and the region containing multiple pixels (instances of the second pixel) in the image to be processed that do not satisfy the first distance relationship with the gaze point as the non-gaze region; wherein the gaze point is located in the gaze region, the non-gaze region does not include the gaze point, and the non-gaze region surrounds the gaze region in a ring shape outside the gaze region.

[0057] For example, Figure 3A A schematic diagram of an image 30 to be processed is shown according to some embodiments of this application. For example... Figure 3A As shown, the image shape of the image to be processed 30 is rectangular, and the gaze point of the image to be processed 30 is located at the center of the rectangle (e.g., Figure 3A (The location of the gray dots). For example, the image to be processed 30 includes 8*8 pixels (e.g., the position of the gray dots). Figure 3A (8*8 small squares), the gaze area of ​​the image to be processed (e.g., 30) Figure 3A The gray-filled area corresponds to the gaze point located in the center region of the image to be processed 30, such as the gaze area containing 4*4 pixels; the non-gaze area of ​​the image to be processed 30 (such as...) Figure 3A The white / unfilled area surrounds the gaze area and is located outside the gaze area. The non-gaze area consists of 4*12 pixels.

[0058] In some embodiments, such as Figure 3BAs shown, when the smart glasses determine whether each pixel in the image to be processed 30 satisfies the first distance relationship with the gaze point, they can use the gaze point as the origin to obtain the coordinate system of the corresponding image to be processed 30 based on the gaze point, determine the pixel coordinates of each pixel based on the coordinate system, and determine whether each pixel satisfies the first distance relationship with the gaze point based on the pixel coordinates of each pixel.

[0059] For example, with Figure 3B The image 30 shown is used as an example. The origin of the coordinate system of the image 30 is the gaze point, the y-axis is parallel to the vertical direction, and the x-axis is parallel to the horizontal direction and perpendicular to the y-axis. For example, the smart glasses can determine the pixel coordinates of each pixel based on the center point of each pixel; for example, the pixel coordinates of pixel 31 are (x1, y1). In other embodiments, the smart glasses can determine the pixel coordinates of each pixel based on the nearest connection point (such as the lower left corner of pixel 31) or the farthest connection point (such as the upper right corner of pixel 31) between each pixel and the gaze point; there are no specific limitations.

[0060] When the smart glasses determine that the pixel coordinates of a certain pixel in the image 30 to be processed meet a preset distance threshold, they determine that the pixel and the gaze point satisfy a first distance relationship, and thus classify the pixel into the gaze area of ​​the image 30 to be processed. For example, when the x-axis coordinate x1 of the pixel coordinates (x1, y1) of pixel 31 is less than or equal to the first threshold, and the y-axis coordinate y1 is less than or equal to the second threshold, they determine that the pixel and the gaze point satisfy the first distance relationship, and classify the pixel 31 into the gaze area of ​​the image 30 to be processed.

[0061] When the smart glasses determine that the pixel coordinates of a certain pixel in the image 30 to be processed do not meet a preset distance threshold, they determine that the pixel does not satisfy a first distance relationship with the gaze point, and thus classify the pixel into the non-gaze region of the image 30 to be processed. For example, when the x-axis coordinate x1 of the pixel coordinates (x2, y2) of pixel 32 is greater than the first threshold and the y-axis coordinate y1 is greater than the second threshold, or when the coordinate y1 is greater than the second threshold, or when the coordinate x1 is greater than the first threshold, it is determined that the pixel 32 does not satisfy the first distance relationship with the gaze point, and pixel 32 is classified into the non-gaze region of the image 30 to be processed.

[0062] It is understood that the first threshold and the second threshold mentioned above can be the same threshold or different thresholds, and there are no specific restrictions.

[0063] For example, the first threshold and the second threshold are the same. For instance, taking the center point of the pupil region of the user's eye as the vertex, the smart glasses define the visual field region of the image to be processed 30 mapped to a field of view (FOV) less than or equal to ±15° as the fixation region, and the visual field region of the image to be processed mapped to a field of view greater than ±15° as the non-fixation region. It can be understood that the first threshold or the second threshold can be half the width or length of the visual field region of the image to be processed 30 mapped to a field of view equal to +15° or -15° from the vertex.

[0064] In other embodiments, such as Figure 3C As shown, when the smart glasses determine whether each pixel in the image to be processed 30 satisfies the first distance relationship with the gaze point, they can use the gaze point as the center point to determine a circle that satisfies a preset radius threshold based on the gaze point, and determine whether each pixel satisfies the first distance relationship with the gaze point based on the determined circle.

[0065] For example, with Figure 3C The image 30 shown is used as an example. The smart glasses determine a circle 33 with radius r based on the gaze point, and consider the pixels completely covered and partially covered by the circle 33 as multiple pixels that satisfy a first distance relationship with the gaze point (e.g., ...). Figure 3C (Pixels filled with medium gray), and the pixels not covered by circle 33 are considered as multiple pixels that do not satisfy the first distance relationship with the gaze point (e.g. Figure 3C (White / unfilled pixels).

[0066] It is understandable that when smart glasses divide the pixels not fully covered by circle 33, they can also determine whether the corresponding pixel satisfies the first distance relationship with the gaze point based on the proportion of pixels covered by circle 33. For example, when the pixel covered by circle 33 occupies two-thirds of the total area of ​​the pixel, it is determined that the pixel satisfies the first distance relationship with the gaze point; when the pixel covered by circle 33 occupies one-tenth of the total area of ​​the pixel, it is determined that the pixel does not satisfy the first distance relationship with the gaze point, and so on, without any specific restrictions.

[0067] Understandable. Figure 3A , Figure 3B and Figure 3C The image shapes shown are merely examples. In other embodiments, the image shape of the image to be processed, as well as the shapes of the gaze area and the non-gaze area, can also be other shapes such as triangles, pentagons, circles, annular rings, and square rings, or other irregular shapes. No specific limitations are imposed.

[0068] In other embodiments, the smart glasses may use the area mapped from the user's pupil to the image to be processed as the fixation zone, and the areas mapped from non-pupil areas such as the iris, sclera, upper eyelid, and lower eyelid to the image to be processed as the non-fixation zone, and so on. Alternatively, in other embodiments, the smart glasses may customize the fixation zone and non-fixation zone according to the different needs of different users or different scenarios. For example, if the image details of the scene displayed on the smart glasses' screen are more located in the right area of ​​the screen, such as the right side, the smart glasses may set the fixation zone closer to the right side of the screen based on the image details, so that the user can adjust the viewing angle according to the fixation zone, etc., without any specific limitations.

[0069] The following example illustrates the process of rendering the gaze area and non-gaze area in smart glasses.

[0070] In some embodiments, smart glasses employ different rendering processes for the gaze area and the non-gaze area, thereby reducing device power consumption by decreasing the amount of data processed for rendering pixels in the non-gaze area.

[0071] For example, smart glasses use different sampling densities when sampling pixels in the gaze area and the non-gaze area. For instance, as... Figure 3A As shown, the image to be processed 30 includes 8*8 processing regions, each with the same number of pixels, such as one or more pixels. The specific number is not limited. The central 4*4 processing regions of the image to be processed 30 are the fixation areas (e.g., ...). Figure 3A The area filled with medium gray), the remaining processing area is the non-focus area (such as...). Figure 3A The smart glasses sample each processing area in the gaze area individually to obtain a sampling point (corresponding to the first sampling instance), that is, the gaze area includes 4*4 sampling points (corresponding to the first sampling point instance); the smart glasses sample every four processing areas in the non-gaze area to obtain a sampling point (corresponding to the second sampling instance), that is, the non-gaze area includes 12 sampling points (corresponding to the second sampling point instance); the smart glasses perform ray tracing using the sampled sampling points to obtain the ray tracing results for each processing area, and render the pixels in each processing area using the ray tracing results, that is, the pixels in each processing area in the gaze area are rendered using the ray tracing result of one sampling point, and the pixels in the four processing areas in the non-gaze area share the ray tracing result of one sampling point for rendering.

[0072] In some embodiments, such as Figure 3AAs shown, after the smart glasses sample the gaze area and non-gaze area in the image to be processed 30 to obtain sampling points, they perform ray tracing on each sampling point to obtain the ray tracing result of the corresponding sampling point, and perform rendering processing on each sampling point, or each sampling point and other unsampled pixels based on the ray tracing result of each sampling point.

[0073] For example, for the gaze area, after the smart glasses sample the gaze area to obtain sampling points, they perform ray tracing on the sampling points to obtain the ray tracing result corresponding to the sampling point (corresponding to the first intermediate result), and based on the ray tracing result of the sampling point, they perform rendering processing on at least one pixel that satisfies the first distance condition with respect to the sampling point; for example, Figure 3A The viewing area shown includes 4*4 processing regions (hereinafter referred to as 4*4 small regions). The pixels in the 4*4 small regions are rendered using the ray tracing results of the sampling points of the corresponding 4*4 small regions. That is, the pixels in each small region satisfy the first distance condition with the sampling points in each small region. Each small region includes at least one pixel.

[0074] For example, for the non-focused area, after the smart glasses sample the non-focused area to obtain sampling points, they perform ray tracing on the sampling points to obtain the ray tracing result corresponding to the sampling point (corresponding to the second intermediate result), and then perform rendering processing on multiple pixels that satisfy the second distance condition with respect to the sampling point based on the ray tracing result of the sampling point; for example, Figure 3A The non-focal region shown includes 12 2*2 processing regions (hereinafter referred to as 12 large regions). The pixels in the 12 large regions are rendered using the ray tracing results of the sampling points of the corresponding 12 large regions. That is, the pixels in each large region satisfy the second distance condition with the sampling points in each large region. Each large region includes at least two pixels.

[0075] It is understandable that the implementation of the first distance condition or the second distance condition between the pixel and the sampling point can be found in the above description of the process of the pixel and the gaze point satisfying the first distance relationship, which will not be repeated here.

[0076] For example, the number of ray bounces during ray tracing of sampling points in the gaze area and non-gaze area differs for the smart glasses. For instance, when ray tracing sampling points in the gaze area, the smart glasses stop ray tracing when the number of ray tracings is greater than or equal to a preset first tracing number, or when the light intensity threshold is less than or equal to a preset first intensity threshold, or when the light exceeds a preset scene boundary. Conversely, when ray tracing sampling points in the non-gaze area, the smart glasses stop ray tracing when the number of ray tracings is greater than or equal to a preset second tracing number, or when the light intensity threshold is less than or equal to a preset second intensity threshold. Here, the first tracing number is greater than the second tracing number, and the first intensity threshold is less than the second intensity threshold.

[0077] For example, smart glasses use ray tracing technology to render pixels in the gaze area, and lightweight rendering technology, such as image lighting technology, to render pixels in the non-gaze area.

[0078] In some embodiments, smart glasses can employ different rendering processes for the gaze area and the non-gaze area using any one or a combination of different sampling densities, different ray tracing times, and different rendering techniques. For example, compared to the rendering process for the gaze area, smart glasses reduce both the sampling density and the number of ray tracing times when rendering the non-gaze area to minimize device power consumption.

[0079] In other embodiments, when rendering the gaze area and the non-gaze area, the smart glasses can gradually reduce the sampling density and the number of ray tracing iterations based on the distance of each sampling point in the gaze area and the non-gaze area from the gaze point. For example, the sampling density in the gaze area is greater than that in the non-gaze area, and the sampling density of positions closer to the gaze point in the gaze area is greater than that of positions farther from the gaze point in the gaze area, and the sampling density of positions closer to the gaze point in the non-gaze area is greater than that of positions farther from the gaze point in the non-gaze area, etc., to avoid a visual segmentation at the boundary between the gaze area and the non-gaze area and ensure a smooth visual transition between the gaze area and the non-gaze area.

[0080] Figure 4 According to some embodiments of this application, a flowchart of an image rendering method is shown. It can be understood that... Figure 4 The processes shown are all executed by smart glasses. For simplicity, the following description... Figure 4 The execution entity will not be described again in the process shown. For example... Figure 4 As shown, this process includes, but is not limited to:

[0081] S401: Determine the gaze point of the image to be processed.

[0082] In some embodiments, smart glasses can determine the gaze point based on the user's eye position when looking at the glasses' display screen. For example, the smart glasses can use an eye-tracking algorithm to determine the pupil area of ​​the user's eyes, then determine the coordinates of the center point in the pupil area, and based on the coordinates of the center point, determine the coordinates of the gaze point corresponding to the image to be processed on the display screen, thereby determining the gaze point of the image to be processed. Alternatively, the user can also customize the gaze point according to different needs. For details, please refer to the foregoing description of the gaze point, which will not be repeated here.

[0083] S402: Divide the image to be processed into a fixation region and a non-fixation region based on the fixation point.

[0084] In some embodiments, after determining the gaze point of an image to be processed, the smart glasses can define the region containing multiple pixels in the image that satisfy a first distance relationship with the gaze point as the gaze region, and the region containing multiple pixels in the image that do not satisfy the first distance relationship with the gaze point as the non-gaze region; wherein the gaze point is located within the gaze region, and the non-gaze region surrounds the gaze region and is located outside the gaze region. For details, please refer to the foregoing description of the gaze region and non-gaze region, which will not be repeated here.

[0085] S403: Perform the first rendering process on the foveated area and the second rendering process on the non-foveated area.

[0086] In some embodiments, the processing parameters of the first rendering process and the second rendering process are different. For example, the sampling density of the first rendering process is greater than that of the second rendering process. It is understood that a higher sampling density indicates a larger amount of data to be processed, and vice versa. Or, for example, the number of ray tracing operations in the first rendering process is greater than that in the second rendering process. It is understood that a higher number of ray tracing operations indicates a larger amount of data to be processed, and vice versa. In other embodiments, the first rendering process and the second rendering process employ different rendering techniques. For example, the first rendering process uses ray tracing technology, while the second rendering process uses image-based lighting technology. It is understood that the processing complexity and rendering quality of the first rendering process are higher than those of the image-based lighting technology.

[0087] For example, the smart glasses may sample pixels in the gaze area at a higher density than they sample pixels in the non-gaze area, thereby reducing power consumption by decreasing the amount of data processed in the non-gaze area; or, the smart glasses may perform more ray tracing operations on a single sample point in the gaze area than on a single sample point in the non-gaze area, further reducing power consumption by decreasing the amount of data processed in the non-gaze area; or, the smart glasses may use ray tracing technology to render pixels in the gaze area, while employing lightweight rendering techniques, such as image lighting-based rendering, to render pixels in the non-gaze area, thereby reducing power consumption by decreasing the processing complexity and rendering quality in the non-gaze area. For details, please refer to the aforementioned description of the rendering process for the gaze and non-gaze areas, which will not be repeated here.

[0088] It is understandable that smart glasses use a first rendering process with higher processing complexity, rendering quality, and data volume for the gaze area, and a second rendering process with higher processing complexity, rendering quality, and data volume for the non-gaze area. This can ensure the rendering effect of the gaze area while reducing the amount of data processing and device power consumption.

[0089] In other embodiments, smart glasses may also reduce the sampling density and ray tracing frequency in non-focused areas relative to the gaze area to further reduce device power consumption.

[0090] S404: Based on the processing results of the first and second rendering processes, obtain the rendered image corresponding to the image to be processed.

[0091] In some embodiments, the smart glasses perform a first rendering process on each pixel in the gaze area of ​​the image to be processed, and a second rendering process on each pixel in the non-gaze area of ​​the image to be processed. Based on the processing results of the first and second rendering processes, a rendered image of the image to be processed is obtained, and the rendered image is displayed on the glasses display screen to display a virtual image to the user.

[0092] It's understandable that the rendered images displayed by smart glasses have similar or identical lighting effects to real-world scenes, making the virtual images appear more realistic to the user when they simultaneously view both. Furthermore, during the rendering process, the smart glasses divide the image into foveated and non-foveated regions, applying different rendering techniques to each region. This reduces the amount of data processed for rendering pixels in the non-foveated region, thus lowering device power consumption.

[0093] Example 2

[0094] In Embodiment 2, the smart glasses divide the image to be processed into a fixation area, a transition area, and a non-fixation area. Different rendering processes are applied to the fixation area, transition area, and non-fixation area. While reducing device power consumption, the transition area provides a visual transition between the fixation area and the non-fixation area, avoiding a visual segmentation at the boundary between the two areas. Embodiment 2 is illustrated below.

[0095] In some embodiments, the process by which smart glasses determine the gaze point corresponding to the image to be processed can be found in the description of Embodiment 1 above, and will not be repeated here.

[0096] The following example illustrates the process by which smart glasses divide an image into a fixational region, a non-fixational region, and a transitional region.

[0097] In some embodiments, after determining the gaze point of the image to be processed, the smart glasses can define the region containing multiple pixels (corresponding to instances of the first pixel) in the image to be processed that satisfy a second distance relationship with the gaze point as the gaze region, the region containing multiple pixels (corresponding to instances of the third pixel) in the image to be processed that satisfy a third distance relationship with the gaze point as the transition region, and the region containing multiple pixels (corresponding to instances of the second pixel) in the image to be processed that do not satisfy the second and third distance relationships with the gaze point as the non-gaze region; wherein the gaze point is located in the gaze region, the transition region surrounds the gaze region in a ring shape, and the non-gaze region surrounds the transition region in a ring shape.

[0098] It is understandable that smart glasses can first divide the image to be processed into a fixation area and a non-fixation area, and then divide the transition area between the fixation area and the non-fixation area; or smart glasses can directly divide the image to be processed into a fixation area, a non-fixation area and a transition area, without any specific restrictions.

[0099] For example, Figure 5 According to some embodiments of this application, a schematic diagram of an image 50 to be processed is shown. For example... Figure 5 As shown, the image shape of the image to be processed 50 is rectangular, and the gaze point of the image to be processed 50 is located at the center of the rectangle (e.g., Figure 5 (The position of the gray dot in the middle) The gaze area of ​​the image to be processed 50 corresponds to the gaze point located in the central region of the image to be processed 30. The transition area of ​​the image to be processed 50 surrounds the gaze area and is located outside the gaze area. The non-gaze area surrounds the transition area and is located outside the transition area.

[0100] In some embodiments, smart glasses can determine whether each pixel satisfies a second distance relationship and a third distance relationship with the gaze point based on the pixel coordinates of each pixel in the image to be processed 50. For example:

[0101] When the x-coordinate of a pixel in the image to be processed 50 is less than or equal to the third threshold and the y-coordinate is less than or equal to the fourth threshold, the pixel is determined to satisfy the second distance relationship with the fixation point so that the pixel is assigned to the fixation area in the image to be processed 50.

[0102] When the x-coordinate of a pixel in the image to be processed 50 is greater than the third threshold and less than or equal to the fifth threshold, and the y-coordinate is greater than the third threshold and less than or equal to the sixth threshold, the pixel is determined to satisfy the third distance relationship with the gaze point so that the pixel is classified into the transition area in the image to be processed 50.

[0103] When the x-coordinate of a pixel in the image to be processed 50 is greater than the third threshold and less than or equal to the fifth threshold, and the y-coordinate is greater than the third threshold and less than or equal to the sixth threshold; or, when the x-coordinate of a pixel in the image to be processed 50 is greater than the third threshold and less than or equal to the fifth threshold; or, when the y-coordinate of a pixel in the image to be processed 50 is greater than the third threshold and less than or equal to the sixth threshold, it is determined that the pixel does not satisfy either the second distance relationship or the third distance relationship with the fixation point, so that the pixel is classified into the non-fixation area in the image to be processed 50.

[0104] It is understandable that the third and fifth thresholds are different thresholds, the fourth and sixth thresholds are different thresholds, the third and fourth thresholds can be the same or different thresholds, and the fifth and sixth thresholds can be the same or different thresholds; there are no specific restrictions.

[0105] For example, the third and fourth thresholds are the same, as are the fifth and sixth thresholds. For instance, using the center point of the pupil region of the user's eye as the vertex, the smart glasses map the visual field of the image 50 to be processed with a field of view (FOV) less than or equal to ±15° as the fixation zone, the visual field of the image 50 to be processed with a FOV greater than ±15° and less than or equal to ±20° as the transition zone, and the visual field of the image 50 to be processed with a FOV greater than ±20° as the non-fixation zone. It can be understood that the third or fourth threshold can be half the width or length of the visual field of the image 50 to be processed when the field of view angle is equal to +15° or -15° from the vertex, and the fifth or sixth threshold can be half the width or length of the visual field of the image 50 to be processed when the field of view angle is equal to +20° or -20° from the vertex.

[0106] Understandable. Figure 5 For pixel region division, please refer to the above. Figure 3B and Figure 3C The relevant descriptions will not be elaborated here. Figure 5The image shapes shown are merely examples. In other embodiments, the image shape of the image to be processed, as well as the shapes of the gaze area, transition area, and non-gaze area, can be other shapes such as triangles, pentagons, circles, annular rings, and square rings, or other irregular shapes. No specific limitations are imposed.

[0107] The following example illustrates the process of rendering the gaze area, transition area, and non-gaze area in smart glasses.

[0108] In some embodiments, the smart glasses employ different rendering processes for the gaze area and the non-gaze area, thereby reducing device power consumption by decreasing the amount of data processed for rendering pixels in the non-gaze area. The rendering processes for the gaze area and the non-gaze area are described in the foregoing Embodiment 1 and will not be repeated here.

[0109] In some embodiments, because smart glasses use different rendering processes for the gaze area and the non-gaze area, a visual segmentation may appear at the boundary between the gaze area and the non-gaze area. Therefore, smart glasses can use a transition area to visually transition between the gaze area and the non-gaze area, avoiding this visual segmentation.

[0110] For example, during the rendering process of the gazing area, transition area, and non-gazing area, the smart glasses sequentially reduce the sampling density of pixels and the number of ray bounces tracked by a single sampling point in each area to gradually reduce the rendering effect. For instance, the sampling density of the gazing area is greater than that of the transition area, which in turn is greater than that of the non-gazing area. Furthermore, the sampling density of locations closer to the gazing point in the gazing area is greater than that of locations farther from the gazing point, the sampling density of locations closer to the gazing point in the transition area is greater than that of locations farther from the gazing point, and the sampling density of locations closer to the gazing point in the non-gazing area is greater than that of locations farther from the gazing point, thus gradually reducing the rendering effect and ensuring a smooth visual transition.

[0111] For example, during the rendering process of the transition area, the rendering result of the corresponding pixel in the transition area can be determined based on the distance of the pixel in the transition area from the gaze area and the non-gaze area.

[0112] In some embodiments, an example is provided where the smart glasses employ a rendering method using ray tracing technology for the gaze area and an image-based lighting technology for the non-gaze area. For instance... Figure 5As shown, the boundary of the gaze area near the transition area (such as the outer frame of the gaze area) is used as the gaze area boundary, and the boundary of the non-gaze area near the transition area (such as the inner frame of the non-gaze area) is used as the non-gaze area boundary. During the rendering process of pixel B in the transition area, the smart glasses first render pixel B using ray tracing and image-based lighting techniques to obtain corresponding rendering results Cr and Cg. Then, the shortest connection between pixel B and the gaze area boundary and the non-gaze area boundary is determined to establish the distance ratio between pixel B and the gaze area and the non-gaze area. Finally, pixel B is rendered based on this distance ratio and the rendering results Cr and Cg. For example, if the distance ratio between pixel B and the gaze area and the non-gaze area is 4:6, the smart glasses will fuse 40% of the rendering result Cr and 60% of the rendering result Cg to obtain the rendering result of pixel B. Or, for example, if the smart glasses determine that the distance between pixel B and the gaze area is greater than the distance between pixel B and the non-gaze area, they will use the rendering result Cr as the rendering result of pixel B, etc.

[0113] It is understandable that smart glasses determine the rendering result of the corresponding pixel in the transition zone based on the distance of the pixel in the transition zone from the gaze zone and the non-gaze zone. This makes the pixel in the transition zone closer to the gaze zone visually closer to the pixel in the gaze zone, and the pixel in the transition zone closer to the non-gaze zone visually closer to the pixel in the non-gaze zone. This ensures that the transition zone provides a visual transition between the gaze zone and the non-gaze zone and avoids a sense of visual segmentation.

[0114] Figure 6 According to some embodiments of this application, a flowchart of an image rendering method is shown. It can be understood that... Figure 6 The processes shown are all executed by smart glasses. For simplicity, the following description... Figure 6 The execution entity will not be described again in the process shown. For example... Figure 6 As shown, this process includes, but is not limited to:

[0115] S601: Determine the gaze point of the image to be processed.

[0116] In some embodiments, the smart glasses can determine the gaze point based on the user's eye position when looking at the glasses' display screen after wearing the smart glasses, or the user can customize the gaze point according to different needs. For details, please refer to the foregoing description of the gaze point, which will not be repeated here.

[0117] S602: Based on the foveation point, the image to be processed is divided into a foveation region, a transition region, and a non-foveation region.

[0118] In some embodiments, after determining the gaze point of the image to be processed, the smart glasses can define the region containing multiple pixels in the image to be processed that satisfy a second distance relationship with the gaze point as the gaze region, the region containing multiple pixels in the image to be processed that satisfy a third distance relationship with the gaze point as the transition region, and the region containing multiple pixels in the image to be processed that do not satisfy either the second or third distance relationship with the gaze point as the non-gaze region. The gaze point is located within the gaze region, the transition region surrounds the gaze region and is located outside the gaze region, and the non-gaze region surrounds the transition region and is located outside the transition region. For details, please refer to the foregoing description of the gaze region, transition region, and non-gaze region; further elaboration will not be repeated here.

[0119] S603: Perform the first rendering process on the foveated area and the second rendering process on the non-foveated area.

[0120] In some embodiments, the processing parameters of the first rendering process and the second rendering process are different. For example, the sampling density of the first rendering process is greater than that of the second rendering process, or the number of ray tracing operations in the first rendering process is greater than that in the second rendering process. Reducing the sampling density and / or the number of ray tracing operations reduces the amount of data processed in the non-focal region, thereby reducing device power consumption. Alternatively, the first rendering process and the second rendering process employ different rendering techniques. For example, the first rendering process uses ray tracing technology, while the second rendering process uses image-based lighting technology. This reduces device power consumption by decreasing the processing complexity and rendering quality in the non-focal region. For details, please refer to the foregoing description of the rendering process in the focal and non-focal regions; further elaboration will not be repeated here.

[0121] S604: Based on the distance of each pixel in the transition zone from the foveated area to the non-foveated area, perform third rendering processing on each pixel in the transition zone.

[0122] In some embodiments, the boundary of the gaze region near the transition region (such as the outer frame of the gaze region) is used as the gaze region boundary, and the boundary of the non-gaze region near the transition region (such as the inner frame of the non-gaze region) is used as the non-gaze region boundary. The distance between each pixel in the transition region and the gaze region is determined based on the shortest connection between each pixel in the transition region and the gaze region boundary, and the distance between each pixel in the transition region and the non-gaze region is determined based on the shortest connection between each pixel in the transition region and the non-gaze region boundary.

[0123] For example, during the rendering process of pixels in the transition area, the smart glasses render pixels in the transition area using the first rendering method for the gaze area when the pixels are closer to the boundary of the gaze area, and using the second rendering method for the non-gaze area when the pixels are closer to the boundary of the non-gaze area. For instance, when rendering pixel B in the transition area, if the smart glasses determine that the distance between pixel B and the gaze area is greater than the distance between pixel B and the non-gaze area, they render pixel B using ray tracing technology.

[0124] For example, during the rendering process of each pixel in the transition area, the smart glasses can also render each pixel in the transition area by combining the distance ratio of each pixel in the transition area to the transition area and the non-focus area with the processing methods of the first rendering process and the second rendering process. For instance, the smart glasses render pixel B in the transition area using ray tracing technology and image-based lighting technology respectively, obtaining the corresponding rendering results Cr and Cg, and determining that the distance ratio of pixel B to the focus area and the non-focus area is 4:6. Then, the smart glasses fuse the rendering results of pixel B by using a rendering ratio of 40% of the rendering result Cr and 60% of the rendering result Cg.

[0125] In other embodiments, during the rendering process of the gaze area, transition area and non-gaze area, the smart glasses sequentially reduce the sampling density of pixels in the gaze area, transition area and non-gaze area, the number of light bounces tracked by a single sampling point, etc., thereby gradually reducing the rendering effect.

[0126] It's understandable that smart glasses use different rendering processes for the gaze area and the non-gaze area, reducing power consumption by decreasing the amount of data processed for rendering pixels in the non-gaze area. Furthermore, smart glasses use transition zones to visually transition between the gaze and non-gaze areas. This ensures that pixels closer to the gaze area in the transition zone have a more similar visual effect to pixels in the gaze area, and vice versa, guaranteeing a smooth visual transition between the gaze and non-gaze areas and avoiding a sense of visual segmentation.

[0127] S605: Based on the processing results of the first rendering process, the second rendering process, and the third rendering process, obtain the rendered image corresponding to the image to be processed.

[0128] In some embodiments, the smart glasses perform a first rendering process on each pixel in the gaze area of ​​the image to be processed, a second rendering process on each pixel in the non-gaze area of ​​the image to be processed, and a third rendering process on each pixel in the transition area of ​​the image to be processed. Based on the processing results of the first rendering process, the second rendering process, and the third rendering process, a rendered image of the image to be processed is obtained, and the rendered image is displayed on the glasses display screen to display a virtual image to the user.

[0129] It's understandable that the rendered images displayed by smart glasses have similar or identical lighting effects to real-world scenes, making the virtual images appear more realistic to the user when they simultaneously view both. Furthermore, during the rendering process, the smart glasses divide the image into regions: a fixation zone, a transition zone, and a non-fixation zone. Different rendering processes are applied to these zones. This reduces power consumption by minimizing the amount of data processed for rendering pixels in the non-fixation zone, and provides a visual transition between the fixation and non-fixation zones through the transition zone.

[0130] In other embodiments, when dividing the image to be processed into regions, the smart glasses are not limited to the division into two regions, namely the gaze area and the non-gaze area, as in Embodiment 1, and the division into three regions, namely the gaze area, the transition area, and the non-gaze area, as in Embodiment 2. The smart glasses can also divide the image to be processed into more regions, such as four regions or five regions. Furthermore, during the image rendering process of each region in the image to be processed, the smart glasses reduce the sampling density of the pixels in each region and the number of light bounces tracked by a single sampling point in order of distance from the gaze point from the closest to the furthest point, thereby gradually reducing the rendering effect and achieving a visual transition.

[0131] Figure 7 According to some embodiments of this application, a structural schematic diagram of a smart glasses 100 is shown. For example... Figure 7 As shown, the smart glasses 100 may include a processor 110, a memory 120, a sensor module 130, an audio module 140, a button 150, an input / output interface 160, a communication module 170, a power supply module 180, an eye-tracking module 11, a display module 12, a ray tracing module 13, an image-based lighting module 14, etc.

[0132] The processor 110 is typically used to control the overall operation of the smart glasses 100 and may include one or more processing units. For example, the processor 110 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a video processing unit (VPU) controller, memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural network processing unit (NPU), etc. The different processing units may be independent devices or integrated into one or more processors.

[0133] The processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. This memory can store instructions or data that the processor 110 has just used or that are used repeatedly. If the processor 110 needs to use the instruction or data again, it can retrieve it directly from the memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0134] In some embodiments, the processor 110 can be used to determine the pupil region and non-pupil region of the user's eyes based on the user's eye images acquired by the eye-tracking module 11, and determine the gaze point of the corresponding image to be processed based on the determined pupil region. Furthermore, the processor 110 can also be used to render the image to be processed based on rendering techniques such as ray tracing technology and image illumination technology of the image rendering module 13, and display the rendered image on the display screens 12L and 12R in the display module 12.

[0135] In some embodiments, the processor 110 may include one or more interfaces. Interfaces may include an inter-integrated circuit (I2C) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general purpose input / output (GPIO) interface, and / or a universal serial bus (USB) interface, a serial peripheral interface (SPI) interface, etc.

[0136] The memory 120 can be used to store computer-executable program code, which includes instructions. The processor 110 executes various functional applications and data processing of the smart glasses 100 by running the instructions stored in the memory 120. The memory 120 may include a program storage area and a data storage area. The program storage area may store the operating system, applications required for at least one function (such as image display function, video playback function, etc.). The data storage area may store data created during the use of the smart glasses 100 (such as eye images captured during eye tracking by the smart glasses 100, ray tracing results during ray tracing by the smart glasses 100, etc.). In addition, the memory 120 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, universal flash storage (UFS), etc.

[0137] The sensor module 130 may include sensors for detecting the working and usage status of the smart glasses 100. These include, but are not limited to, proximity sensors / contact sensors for detecting whether the user is wearing the smart glasses 100, and inertial sensors for detecting the user's head movement.

[0138] The audio module 140 may include a speaker, a microphone, etc., for implementing audio functions.

[0139] There may be one or more buttons 150. Buttons 150 may take the form of buttons, switches, dials, and touch or proximity sensing devices (such as touch sensors). The smart glasses 100 can trigger corresponding functions based on the user's operation of one or more of the buttons 150, such as adjusting the volume or controlling the smart glasses 100 to enter calibration mode.

[0140] The input / output interface 160 can connect other devices to the smart glasses 100 via suitable components. Components may include, for example, audio / video jacks, data connectors, etc. For example, in some embodiments, the eye-tracking module 11 may not be part of the VR glasses, but rather an external device connected to the smart glasses 100 via the input / output interface 160.

[0141] Communication module 170 may include a wireless communication module. Wireless communication functionality can be implemented via an antenna (not shown), a modem processor (not shown), and a baseband processor (not shown). The antenna is used to transmit and receive electromagnetic wave signals. The smart glasses 100 may include multiple antennas, each capable of covering one or more communication frequency bands. Different antennas can also be multiplexed to improve antenna utilization. For example, antenna 1 can be multiplexed as a diversity antenna for a wireless local area network. In some other embodiments, the antenna can be used in conjunction with a tuning switch.

[0142] The wireless communication module can provide solutions for wireless communication applications on smart glasses 100, including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), and infrared (IR) technologies. The wireless communication module can be one or more devices integrating at least one communication processing module. The wireless communication module receives electromagnetic waves via an antenna, modulates and filters the electromagnetic wave signals, and sends the processed signal to the processor 110. The wireless communication module can also receive signals to be transmitted from the processor 110, modulate and amplify them, and then convert them into electromagnetic waves for radiation via the antenna.

[0143] The power supply module 180 may include a battery for supplying power to the various modules of the smart glasses 100.

[0144] Understandable, Figure 7 The structure of the smart glasses 100 shown is only an example. In other embodiments, the smart glasses 100 may include more or fewer modules, or some modules may be combined or split, which is not limited here.

[0145] In some embodiments, this application also provides a computer-readable medium storing program code that, when run on a computer, causes the computer to perform the methods described above.

[0146] In some embodiments, this application also provides a computer program product comprising: computer program code that, when run on a computer, causes the computer to perform the methods described above.

[0147] In the accompanying drawings, some structural or methodological features may be shown in a specific arrangement and / or order. However, it should be understood that such a specific arrangement and / or order may not be necessary. Rather, in some embodiments, these features may be arranged in a manner and / or order different from that shown in the illustrative drawings. Furthermore, the inclusion of structural or methodological features in a particular figure does not imply that such features are required in all embodiments, and in some embodiments, these features may be omitted or may be combined with other features.

[0148] It should be noted that the units / modules mentioned in the various device embodiments of the present invention are all logical units / modules. Physically, a logical unit / module can be a physical unit / module, a part of a physical unit / module, or a combination of multiple physical units / modules. The physical implementation of these logical units / modules themselves is not the most important factor; the combination of functions implemented by these logical units / modules is the key to solving the technical problem proposed by the present invention. Furthermore, to highlight the innovative aspects of the present invention, the above-described device embodiments of the present invention have not introduced units / modules that are not closely related to solving the technical problem proposed by the present invention. This does not mean that the above-described device embodiments do not contain other units / modules.

[0149] It should be noted that in the examples and description of this patent, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one" does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0150] Although the invention has been illustrated and described with reference to certain preferred embodiments thereof, those skilled in the art will understand that various changes in form and detail may be made therein without departing from the scope of the invention.

Claims

1. An image rendering method applied to an electronic device, comprising: The method comprises: determining a gaze point of a virtual image to be displayed; determining a first region and a second region of the virtual image based on the gaze point, wherein the gaze point is located in the first region and the second region does not include the gaze point; performing first rendering processing on the first region to obtain a first rendering result and performing second rendering processing on the second region to obtain a second rendering result, wherein the processing parameters of the first rendering processing are different from the processing parameters of the second rendering processing; determining a rendering image corresponding to the virtual image based on the first rendering result and the second rendering result; and displaying the rendering image.

2. The method of claim 1, wherein: corresponding to the first rendering processing and the first rendering processing both employ ray tracing technology, and the processing parameters include sampling density and / or ray tracing times, wherein the sampling density of the first rendering processing is greater than the sampling density of the second rendering processing, and the ray tracing times of the first rendering processing are greater than the ray tracing times of the second rendering processing; corresponding to the first rendering processing employing ray tracing technology and the second rendering processing employing image-based lighting technology, the processing parameters of the first rendering processing include processing parameters corresponding to ray tracing technology, and the second rendering processing includes processing parameters corresponding to image-based lighting technology.

3. The method according to claim 1 or 2, characterized in that, corresponding to the first rendering processing and the first rendering processing both employ ray tracing technology. The first rendering processing on the first region to obtain a first rendering result and the second rendering processing on the second region to obtain a second rendering result comprise: performing first sampling on a plurality of first pixel points in the first region to obtain a plurality of first sampling points; performing first rendering processing on the corresponding plurality of first pixel points in the first region based on the plurality of first sampling points to obtain the first rendering result; performing second sampling on a plurality of second pixel points in the second region to obtain a plurality of second sampling points; performing second rendering processing on the corresponding plurality of second pixel points in the second region based on the plurality of second sampling points to obtain the second rendering result; wherein the sampling density of the first sampling is greater than the sampling density of the second sampling.

4. The method of claim 3, wherein, The first rendering processing on the first region to obtain a first rendering result and the second rendering processing on the second region to obtain a second rendering result comprise: determining a plurality of first intermediate results of the first rendering processing of the plurality of first sampling points; based on the first intermediate result of each first sampling point, performing rendering processing on at least one first pixel point of each first sampling point that satisfies a first distance condition to obtain the first rendering result.

5. The method of claim 3, wherein, The second rendering processing on the second region to obtain a second rendering result comprises: determining a plurality of second intermediate results of the second rendering processing of the plurality of second sampling points; The second rendering result is obtained by performing rendering processing on the second pixel points that satisfy a second distance condition for each of the second sampling points based on the second intermediate result of each of the second sampling points.

6. The method according to any one of claims 3 to 5, characterized in that, The method further includes: The number of ray tracing of the first rendering processing is greater than the number of ray tracing of the second rendering processing.

7. The method according to any one of claims 1 to 6, characterized in that, The method further includes: The second region is a ring region surrounding the first region.

8. The method according to any one of claims 1 to 6, characterized in that, The method further includes: A third region of the virtual image is determined, wherein the third region is located between the first region and the second region; A first distance between the third pixel points in the third region and the first region and a second distance between the third pixel points in the third region and the second region are determined; A third rendering result is obtained by performing third rendering processing on the third pixel points based on the size relationship between the first distance and the second distance; A rendering image corresponding to the virtual image is determined based on the first rendering result, the second rendering result, and the third rendering result.

9. The method of claim 8, wherein, The third rendering result is obtained by performing third rendering processing on the third pixel points based on the size relationship between the first distance and the second distance, including: The third pixel points are processed by the third rendering processing based on the processing method of the first rendering processing to obtain the third rendering result corresponding to the first distance being greater than or equal to the second distance; The third pixel points are processed by the third rendering processing based on the processing method of the second rendering processing to obtain the third rendering result corresponding to the first distance being less than the second distance.

10. The method of claim 8, wherein, The third rendering result is obtained by performing third rendering processing on the third pixel points based on the size relationship between the first distance and the second distance, including: A first ratio of the first distance relative to the first distance and the second distance and a second ratio of the second distance relative to the first distance and the second distance are determined; A first result corresponding to the first rendering processing of the third pixel points and a second result corresponding to the second rendering processing of the third pixel points are determined; A third rendering result corresponding to the third rendering processing of the third pixel points is determined based on the first result of the first ratio and the second result of the second ratio.

11. The method according to any one of claims 8 to 10, characterized in that, The method further includes: The third region is a ring region surrounding the first region, and the second region is a ring region surrounding the third region.

12. A computer-readable storage medium, characterized in that, The readable storage medium has instructions stored thereon, and the instructions, when executed on an electronic device, cause the electronic device to implement the method of any one of claims 1 to 11.

13. An electronic device, comprising: Including: a memory for storing instructions executed by one or more processors of an electronic device; and a processor, which is one of the processors of the electronic device, is configured to execute the instructions stored in the memory to implement the method of any one of claims 1 to 11.

14. A program product, characterized by The program product includes instructions, and the instructions, when executed on an electronic device, cause the electronic device to implement the method of any one of claims 1 to 11.

Images