Image rendering method and device, storage medium and program product

Abstract

The embodiment of the invention provides an image rendering method and device, a storage medium and a program product, and the method comprises the steps: obtaining a plurality of to-be-rendered UI objects, and for each UI object in the plurality of UI objects, carrying out the rendering display of the UI object based on a first processing mode if the feature data of the UI object meets a first preset condition, if the feature data of the UI object does not meet the first preset condition, rendering display is conducted on the UI object based on a second processing mode, the energy consumption of the first processing mode is larger than that of the second processing mode, and the energy consumption of the first processing mode is larger than that of the second processing mode. According to the method provided by the embodiment of the invention, the UI objects are dynamically screened on the basis of the preset condition, and only part of the UI objects meeting the screening condition can be processed in the first processing mode with relatively high energy consumption, so that the resource consumption can be reduced.

Description

Technical Field

[0001] This disclosure relates to the field of extended reality technology, and more particularly to an image rendering method, apparatus, storage medium, and program product. Background Technology

[0002] Extended Reality (XR) headsets, such as Virtual Reality (VR) all-in-one devices, display user interface (UI) objects in high definition, making it easier for users to interact with the VR system.

[0003] In related technologies, pass-through layer technology is usually used to render UI objects in each frame of an image with a higher resolution than that of conventional rendering processes and display them on the screen.

[0004] However, the inventors discovered at least the following technical problems in the related technology: when there are many UI objects, resource consumption will increase, such as significantly increasing the rendering pressure on the GPU, increasing the risk of screen tearing, and consuming too much power. Summary of the Invention

[0005] This disclosure provides an image rendering method, apparatus, storage medium, and program product to reduce resource consumption.

[0006] In a first aspect, embodiments of this disclosure provide an image rendering method, including:

[0007] Obtain multiple user graphical interface (UI) objects to be rendered;

[0008] For each of the plurality of UI objects, if the feature data of the UI object meets the first preset condition, the UI object is rendered and displayed based on the first processing method; if the feature data of the UI object does not meet the first preset condition, the UI object is rendered and displayed based on the second processing method; the resource consumption of the first processing method is greater than the resource consumption of the second processing method.

[0009] In a second aspect, embodiments of this disclosure provide an image rendering apparatus, comprising:

[0010] The retrieval module is used to retrieve multiple UI objects to be rendered;

[0011] The rendering module is used to render and display each UI object based on a first processing method if the feature data of the UI object meets a first preset condition; and to render and display the UI object based on a second processing method if the feature data of the UI object does not meet the first preset condition. The resource consumption of the first processing method is greater than the resource consumption of the second processing method.

[0012] Thirdly, embodiments of this disclosure provide an electronic device, including: a processor and a memory;

[0013] The memory stores computer-executed instructions;

[0014] The processor executes computer execution instructions stored in the memory, causing the at least one processor to perform the image rendering method as described in the first aspect and various possible designs of the first aspect.

[0015] Fourthly, embodiments of this disclosure provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the image rendering method described in the first aspect and various possible designs of the first aspect.

[0016] Fifthly, embodiments of this disclosure provide a computer program product, including a computer program that, when executed by a processor, implements the image rendering method described in the first aspect and various possible designs of the first aspect.

[0017] The image rendering method, device, storage medium, and program product provided in this embodiment first acquire multiple UI objects to be rendered. For each UI object, if its feature data meets a first preset condition, the UI object is rendered and displayed based on a first processing method. If the feature data of the UI object does not meet the first preset condition, the UI object is rendered and displayed based on a second processing method. The energy consumption of the first processing method is greater than that of the second processing method. The method provided in this embodiment, by dynamically filtering UI objects based on preset conditions, can process only a portion of UI objects that meet the filtering conditions using the more energy-intensive first processing method, such as high-definition display. This reduces resource consumption, such as reducing graphics card resource consumption, reducing GPU rendering pressure, and reducing the risk of screen tearing. Simultaneously, it also reduces power consumption and improves the battery life of the all-in-one machine. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a schematic diagram illustrating an application scenario of the image rendering method provided in the embodiments of this disclosure;

[0020] Figure 2 Schematic diagram of the image rendering method provided in the embodiments of this disclosure Figure 1 ;

[0021] Figure 3 This is a schematic diagram illustrating the principle of determining UI objects within the field of view, provided in an embodiment of this disclosure.

[0022] Figure 4 Schematic diagram of the image rendering method provided in the embodiments of this disclosure Figure 2 ;

[0023] Figure 5 Schematic diagram of the image rendering method provided in the embodiments of this disclosure Figure 3 ;

[0024] Figure 6 Schematic diagram of the image rendering method provided in the embodiments of this disclosure Figure 4 ;

[0025] Figure 7 Schematic diagram of the image rendering method provided in the embodiments of this disclosure Figure 5 ;

[0026] Figure 8 This is a structural block diagram of an image rendering device provided in an embodiment of the present disclosure;

[0027] Figure 9 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this disclosure. Detailed Implementation

[0028] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only one element of the embodiments of this disclosure, and not all of them. Based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0029] User interface (UI) serves as the medium for interaction and information exchange between a system and its users, facilitating the conversion between the internal form of information and a human-readable format. UI enables users to operate hardware conveniently and effectively, achieving two-way interaction and completing desired tasks. The definition of a user interface is broad, encompassing both human-computer interaction (HCI) and graphical user interfaces (GUIs); a user interface exists in any field involving information exchange between humans and machines. In XR headsets, displaying UI objects with high resolution enhances the user experience and facilitates interaction between the user and the headset's system.

[0030] In related technologies, during the VR rendering process, scene content can be rendered by the left and right eye cameras and drawn onto the left and right eye rendering buffers. After rendering, the eye buffers undergo distortion and sampling processing by the ATW thread before finally being rendered onto the VR screen. In this method, since the canvas size of the eye buffer is usually smaller than the screen size, when UI objects are first drawn on the canvas and then rendered onto the screen, texture sampling is required to stretch the image to fit the screen size. This reduces the display clarity of UI objects. To improve clarity, pass-through layer technology (VR Compositor Layers) can be used to render UI objects on the screen in high definition. Specifically, based on this technology, the scene content is no longer directly rendered onto the eye buffer, but is instead "passed through" to the ATW thread for distortion, sampling, and compositing processing. The advantage of this is that it avoids an extra texture sampling and improves the clarity of textures and video.

[0031] However, rendering all UI objects in high definition significantly increases the rendering load on the graphics processing unit (GPU). At a constant screen refresh rate, the GPU's rendering speed becomes out of sync with the screen's pixel refresh rate, increasing the risk of screen tearing. Additionally, when the head-mounted device is an all-in-one unit, its limited battery capacity also affects its battery life.

[0032] To address the aforementioned technical problems, the inventors of this disclosure have discovered that displaying only a subset of UI objects in high definition can reduce graphics card resource consumption and alleviate GPU rendering pressure. Specifically, a judgment strategy is designed to select UI objects that the user is likely to be more interested in at the moment and display them in higher definition, while displaying the remaining UI objects in lower definition. Based on this, embodiments of this disclosure provide an image rendering method.

[0033] Figure 1 This is a schematic diagram illustrating an application scenario of the image rendering method provided in the embodiments of this disclosure. For example... Figure 1As shown, in head-mounted devices, such as VR all-in-one machines, multiple UI objects are displayed in the current image frame on the screen. Displaying all UI objects in high definition increases the rendering load on the GPU and the risk of screen tearing. Therefore, before rendering the image frame, multiple user graphical interface (UI) objects to be rendered can be obtained. For each UI object, if its feature data meets a first preset condition, it is rendered using a first processing method; if its feature data does not meet the first preset condition, it is rendered using a second processing method. The resource consumption of the first processing method is greater than that of the second processing method. The image rendering method provided in this disclosure, by dynamically filtering UI objects based on preset conditions, can process only those UI objects that meet the filtering conditions using the more energy-intensive first processing method, such as high-definition display. This reduces resource consumption, such as reducing GPU resource consumption, decreasing GPU rendering load, and reducing the risk of screen tearing. Simultaneously, it also reduces power consumption and improves the battery life of the all-in-one machine.

[0034] It should be noted that, Figure 1 The scene diagram shown is merely an example. The image rendering method and scene described in the embodiments of this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided in the embodiments of this application. As those skilled in the art will know, with the evolution of the system and the emergence of new business scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0035] The technical solutions of this application will be described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments.

[0036] refer to Figure 2 , Figure 2 Schematic diagram of the image rendering method provided in the embodiments of this disclosure Figure 1 The image rendering method includes:

[0037] 201. Obtain multiple UI objects to be rendered.

[0038] The execution subject of this disclosure embodiment is an XR device, such as... Figure 1 The VR all-in-one machine shown.

[0039] Specifically, virtual reality technology encompasses computer science, electronic information, and simulation technology. Its basic implementation involves computer-simulated virtual environments to create a sense of immersion. Users wear head-mounted devices, such as VR all-in-one headsets, and experience this immersion through a three-dimensional scene displayed on the screen—the VR space. The display of the three-dimensional scene is based on the rendering of image frames at a certain screen refresh rate. Each image frame can simultaneously include multiple UI objects (such as video playback interfaces, email interfaces, game interfaces, chat interfaces, settings interfaces, etc.). These multiple UI objects can originate from the image frames to be rendered, which are the image frames that will be rendered and displayed on the screen based on the screen refresh rate.

[0040] 202. For each of the plurality of UI objects, if the feature data of the UI object meets the first preset condition, the UI object is rendered and displayed based on the first processing method; if the feature data of the UI object does not meet the first preset condition, the UI object is rendered and displayed based on the second processing method; the resource consumption of the first processing method is greater than the resource consumption of the second processing method.

[0041] Specifically, multiple UI objects can be dynamically filtered based on a first preset condition, selecting some UI objects to be processed using a first processing method with higher resource consumption, and others using a second processing method with lower resource consumption. This achieves a balance between overall resource consumption and visual experience.

[0042] Resource consumption can include resources such as CPU and GPU, or electrical energy.

[0043] In one embodiment of this disclosure, rendering and displaying the UI object based on the first processing method may include: refreshing the window of the UI object at a first refresh rate; rendering and displaying the UI object based on the second processing method may include: refreshing the window of the UI object at a second refresh rate; the first refresh rate is greater than the second refresh rate.

[0044] In one embodiment of this disclosure, rendering and displaying the UI object based on the first processing method may include: rendering the window of the UI object at a first resolution; rendering and displaying the UI object based on the second processing method may include: rendering the window of the UI object at a second resolution; the first resolution is greater than the second resolution.

[0045] In one embodiment of this disclosure, rendering and displaying the UI object based on the second processing method includes at least one of the following: freezing the process corresponding to the UI object and displaying a first image rendered and displayed before the process was frozen in the window of the UI object; terminating the process corresponding to the UI object and displaying a second image rendered and displayed before the process was terminated in the window of the UI object. Rendering and displaying the UI object based on the first processing method may include: continuing to render and display the UI object based on the current rendering and display parameters. For example, when the second processing method is to freeze the process corresponding to the UI object, the first processing method may be to continue maintaining the process without processing it. When the second processing method is to terminate the process corresponding to the UI object, the first processing method may be to continue maintaining the process without terminating it.

[0046] In one embodiment of this disclosure, rendering and displaying the UI object based on a first processing method includes at least one of the following: refreshing the window of the UI object at a first refresh rate; rendering the window of the UI object at a first resolution; rendering and displaying the UI object based on a second processing method includes at least one of the following: Process 1: refreshing the window of the UI object at a second refresh rate; the first refresh rate is greater than the second refresh rate; Process 2: rendering the window of the UI object at a second resolution; the first resolution is greater than the second resolution; Process 3: freezing the process corresponding to the UI object, and displaying a first image rendered and displayed before the process was frozen in the window of the UI object; Process 4: terminating the process corresponding to the UI object, and displaying a second image rendered and displayed before the process was terminated in the window of the UI object.

[0047] In Process 1, the preset frequency can be less than or equal to 50Hz, such as 45Hz. In Process 2, the preset ratio can be less than or equal to 95%, such as 90%. In Process 3, freezing the process can stop the process from using CPU and memory resources, but it is still kept in memory, which can be understood as a "semi-active" state. At this time, the corresponding application cannot respond to user interaction. If the user switches the focus to the application, it will immediately return to the normal "active" state from the frozen state. In the frozen state, the application's resource allocation stops using CPU and memory resources, but the process is still kept in memory and can be quickly woken up and used. In the frozen state, the application interface: the page can display a snapshot of the last frame before freezing, and the window is not closed; no logic that requires real-time calls to system interfaces, such as video playback, audio playback, or location refresh, is performed; it is not interactive; when the user switches back to focus, there is no need to restart the process, only to wake up the interface before freezing after memory resource allocation. In Process 4, ending the process can kill the application interface. Specifically, the page displays a snapshot of the last frame before freezing, and the window is not closed; when the user switches back to focus, the process is restarted.

[0048] Specifically, processes 1-4 can be freely combined. In order to provide users with a high-level experience for as long as possible, the image rendering method provided in this embodiment can be implemented only when resources are consumed to a certain extent. That is, dynamic filtering is performed based on the first preset condition, and then different UI objects are rendered and displayed with different levels of resource consumption, thereby reducing the overall resource consumption and extending the usage time of the head-mounted device.

[0049] For example, multiple UI objects can be prioritized and scored based on their feature data (the score can be based on which conditions the UI object's feature data meets to give a corresponding score, and then obtain a total score, for example, if it is within the field of view, it can get 20 points, if it is the focus UI, it can get 50 points, and so on). Then, when the resource consumption reaches a preset value (for example, the power consumption of a certain performance module reaches a preset index), if the UI object's score meets the preset conditions (for example, it is the lowest score among multiple UI objects), the UI object can be processed by either process 1 (for example, the window refresh rate is reduced to 45 Hz) or process 2 (the window rendering resolution is reduced to 90% of the original resolution).

[0050] In one feasible approach, the optimization measures are executed with a cyclical logic. After execution, resource consumption is immediately re-monitored to see if it falls below a preset value (i.e., below the optimization threshold). If not, continuous optimization is required. Therefore, the execution sequence of optimization measures can be as follows: First, if the monitored metric crosses the threshold, reduce the window refresh rate of the process with the lowest priority score to 45Hz and mark it. Second, if the metric still crosses the threshold, reduce the rendering resolution of the window corresponding to the marked process (excluding those with reduced frame rate and resolution) by 10% and mark it. Third, if the metric still crosses the threshold, reduce the window refresh rate of the process with the lowest priority score (excluding those marked with reduced frame rate and resolution) to 45Hz and mark it. This process is repeated until the metric falls below the optimization threshold.

[0051] In one embodiment of this disclosure, rendering the UI object based on the first processing method may include: rendering the UI object at a first resolution; rendering the UI object based on the second processing method may include: rendering the UI object at a second resolution; the first resolution is greater than the second resolution. Specifically, for each of the plurality of UI objects, if the feature data of the UI object meets a first preset condition, then the UI object is rendered at the first resolution; if the feature data of the UI object does not meet the first preset condition, then the UI object is rendered at the second resolution; the first resolution is greater than the second resolution.

[0052] Specifically, rendering image frames consumes graphics card resources, increasing GPU load. When GPU load increases to a certain level, the graphics card's rendering speed cannot keep up with the screen's pixel-by-pixel refresh rate, leading to screen tearing. To reduce the risk of screen tearing, multiple UI objects can be prioritized. UI objects that meet preset conditions are designated as high-priority and can achieve high resolution using resource-intensive techniques like pass-through layers. UI objects that do not meet the preset conditions can be rendered using the standard rendering process (rendered by the left and right eye cameras, drawn onto the Eye Buffer, and then processed by the ATW thread for distortion and sampling before being rendered onto the VR screen), reducing graphics card resource consumption and displaying at a lower resolution. Optionally, UI objects that do not meet the preset conditions can be further prioritized after being identified as low-priority; that is, the second resolution can include multiple resolutions, but these multiple resolutions must be lower than the first resolution. Low-priority UI objects, after being reclassified, can be displayed differently using multiple resolutions included in the second resolution. This embodiment does not limit this approach.

[0053] In one embodiment of this disclosure, rendering the UI object using a first processing paradigm may include: based on pass-through layer technology, directly submitting the scene content corresponding to the UI object to the asynchronous time-warped (ATW) thread for processing without going through the rendering thread, and then rendering and displaying it on the screen of the head-mounted device with a first resolution; rendering the UI object using a second processing method may include: submitting the scene content corresponding to the UI object to the rendering thread for rendering with the left and right eye cameras, submitting the data rendered in the left and right eye rendering buffers to the asynchronous time-warped (ATW) thread for processing, and then rendering and displaying it on the screen of the head-mounted device with a second resolution.

[0054] Specifically, in head-mounted devices, such as VR all-in-one headsets, UI objects can have their clarity improved through pass-through layer technology. However, excessive pass-through content significantly increases the GPU rendering load, raising the risk of screen tearing. This embodiment employs a dynamic pass-through strategy to select UI objects for high-resolution rendering, balancing clarity and performance. Specifically, it ensures that high-priority UI objects are pass-through based on factors such as visibility and recent interaction intervals, while lower-priority UI objects are not. This guarantees optimal clarity for the user's interactive objects while improving the overall performance and battery life of the all-in-one headset. Furthermore, the performance balancing strategy allows the all-in-one headset to open more UI objects.

[0055] Higher resolution (first resolution) is achieved through pass-through layer technology, while lower resolution (second resolution) is achieved through the conventional Eyebuffer rendering process. Specifically, in the conventional VR rendering process, scene content is rendered by the left and right eye cameras and drawn onto the Eye Buffer. After drawing, the Eye Buffer undergoes distortion and sampling processing by the ATW thread before finally being rendered onto the VR screen. VR Compositor Layers technology offers a different rendering method than directly rendering scene content to the Eyebuffer. Instead of rendering the scene content directly to the Eye Buffer, it "passes through" it to the ATW thread for distortion, sampling, and compositing. The advantage of this is that it avoids an extra texture sampling, improving the clarity of textures and video.

[0056] In one embodiment of this disclosure, the feature data of the UI object may include at least one of the following: type, location, area of ​​the unobstructed portion, and interaction data.

[0057] The type of a UI object can be categorized based on its origin, purpose, etc. When a UI object is displayed in VR space, it may be obscured by adjacent UI objects. The area of ​​the unobscured portion of a UI object refers to the area of ​​the exposed portion. UI object interaction data refers to data related to user interaction with the UI object. For example, when a user injects interaction time into an interaction point using controller buttons (e.g., confirm, return, drag, etc.), the interacting UI object is recorded as the focus UI, and the recording time is the focus time. When using gestures, the position and orientation of the user's hand are obtained in real time to calculate the interaction point, and fixed gestures are used to trigger events such as confirm, return, and drag. After the trigger ends, the interacting UI object is recorded as the focus UI, and the recording time is the focus time.

[0058] In one embodiment of this disclosure, the feature data of the UI object satisfying the first preset condition includes: the feature data of the UI object belonging to a preset type.

[0059] The preset type can be a safety type. During VR all-in-one device use, there is a risk of collision with the surrounding environment. To prevent collisions, features such as safe zones can be set up, and a warning can be given when the user leaves the safe zone. UI objects related to this safety function can be categorized as safety type. To ensure user safety, all UI objects of the safety type can be processed using the first-level processing method, such as displaying them at the first resolution, i.e., displaying them at a higher resolution.

[0060] In one embodiment of this disclosure, the preset type may include a custom type; the method may further include: in response to a touch operation on a first control in the settings interface, determining the type of the selected UI object as a custom type.

[0061] Specifically, users typically have their own usage habits when using head-mounted devices, such as VR all-in-one headsets. To facilitate personalized settings and improve the user experience, custom UI object settings are provided. For example, if a user prioritizes the chat experience, the chat UI can be set to a custom type for higher resolution rendering.

[0062] In one embodiment of this disclosure, when the type of the UI object does not belong to a preset type, the feature data of the UI object satisfying the first preset condition may include satisfying at least one of the following conditions: Condition 1: The position of the UI object is within the field of view; Condition 2: The ratio between the area of ​​the unobstructed portion of the UI object and the total area of ​​the UI object is greater than a first preset ratio; Condition 3: The ratio between the area of ​​the unobstructed portion of the UI object and the total area of ​​the UI object ranks less than a first preset position in the ratio sorting; the ratio sorting is obtained by sorting the ratio between the area of ​​the unobstructed portion of the plurality of UI objects and the corresponding total area from largest to smallest; Condition 4: The focus time corresponding to the UI object ranks less than a second preset position in the focus sorting; the focus sorting is obtained by sorting the focus times corresponding to the plurality of UI objects in chronological order; the focus time is the time when the corresponding UI object is recorded as the focus UI after the interaction event injection is completed.

[0063] Specifically, when the preset type is used as a high-priority condition, for UI objects that do not meet the preset type, further judgment can be made based on conditions 1-4. Specifically, one or more conditions can be selected and combined to obtain the first preset condition. For example, condition 1 can be used as the first preset condition; conditions 1 and 2 can be used as the first preset condition; conditions 1-3 can be used as the first preset condition; conditions 1-4 can be used as the first preset condition; conditions 1, 2, and 4 can be used as the first preset condition; conditions 1, 3, and 4 can be used as the first preset condition; conditions 2 and 4 can be used as the first preset condition, and so on.

[0064] Regarding condition 1, the all-in-one machine's UI management system should be able to detect the visibility status of each UI object in real time. For UI objects that are fully or partially within the user's field of view, high-resolution passthrough should be prioritized, as they have the greatest impact on the user experience. For UI objects that are obscured by other UI objects or located at the edge of the screen, the passthrough priority can be reduced to decrease the rendering burden on the GPU.

[0065] Regarding condition 4, by recording the user's most recent interaction time with each UI object, we can determine which UI elements are currently most relevant or important to the user. Recently interacted UI elements should receive a higher passthrough priority, as they are more likely to become the user's focus again. UI elements that haven't been interacted with for a long time can have their clarity temporarily reduced to conserve GPU resources.

[0066] In one embodiment of this disclosure, the method further includes: acquiring the current pose of the head-mounted device; and determining the field of view of the head-mounted device based on the current pose.

[0067] In one possible implementation, the current pose of the head-mounted device can be obtained; the visual cone pose of the head-mounted device can be determined based on the current pose; the position of the UI object within the field of view can include: the position of the UI object within the visual cone.

[0068] For example, the visual cone pose can be determined based on the current pose of the head-mounted device, and then it can be determined whether each UI object is located within the visual cone. If it is within the visual cone, it indicates that the UI object is within the field of view. Figure 3 As shown, UI1 is outside the visual cone, so UI1 is outside the field of vision. UI2 is completely inside the visual cone, so it can be determined that it is within the field of vision. UI3 is partially inside the visual cone, so it can be determined that it is within the field of vision.

[0069] In another possible implementation, the current pose of the head-mounted device can be obtained; the current eye-tracking result can be obtained; and the field of view of the head-mounted device can be determined based on the current pose and the eye-tracking result. Specifically, the field of view determined by the eye-tracking function of the head-mounted device is relatively large. To further save power consumption and reduce resource consumption, the field of view can be further confirmed by combining the current pose of the head-mounted device.

[0070] In one embodiment of this disclosure, the feature data of the UI object satisfying the first preset condition may include satisfying at least one of the following conditions: Condition 1: The position of the UI object is within the field of view; Condition 2: The ratio between the area of ​​the unobstructed portion of the UI object and the total area of ​​the UI object is greater than a first preset ratio; Condition 3: The ratio between the area of ​​the unobstructed portion of the UI object and the total area of ​​the UI object ranks less than a first preset position in the ratio sorting; the ratio sorting is obtained by sorting the ratio between the area of ​​the unobstructed portion of each of the plurality of UI objects and the corresponding total area from largest to smallest; Condition 4: The focus time corresponding to the UI object ranks less than a second preset position in the focus sorting; the focus sorting is obtained by sorting the focus times corresponding to each of the plurality of UI objects in chronological order; the focus time is the time when the corresponding UI object is recorded as the focus UI after the interaction event injection is completed; Condition 5: The UI object belongs to a preset type.

[0071] Specifically, conditions 1-5 can be freely combined to obtain the first preset condition. The more conditions set, the more stringent the first preset condition becomes, resulting in fewer UI objects that meet the first preset condition and fewer UI objects displayed at higher resolution. This reduces the consumption of graphics card resources and the pressure on the GPU. However, the user's visual experience will inevitably decline. Therefore, the preset conditions can be set according to actual needs, and this embodiment does not impose any limitations on this.

[0072] As described above, by dynamically filtering UI objects based on preset conditions, only those UI objects that meet the filtering criteria can be processed using the most energy-intensive primary processing method, such as high-definition display. This reduces resource consumption, such as reducing graphics card resource consumption, decreasing GPU rendering pressure, and lowering the risk of screen tearing. Simultaneously, it also reduces power consumption and improves the all-in-one PC's battery life.

[0073] In one embodiment of this disclosure, based on the above embodiments, for example in Figure 2 Based on the embodiment shown, before step 201, the method may further include: responding to a touch operation on a second control in the settings interface, determining a target visual experience level; different visual experience levels correspond to different degrees of strictness of preset conditions for rendering and displaying the UI object; and determining the preset conditions corresponding to the target level as the first preset condition.

[0074] Considering the different preferences of various users, users can adjust the pass-through strategy according to their own preferences. For example, some users may prioritize performance and be willing to sacrifice some clarity, while others may focus more on visual experience and be willing to accept higher GPU load. By providing customizable options, all-in-one PCs can better meet the needs of different users. Based on this, users can be offered different tiers, each corresponding to a different level of stringency in preset conditions.

[0075] For example, the visual experience level can be divided into three levels. Level 1 offers the highest visual experience, and the preset condition for Level 1 can be set to satisfy condition 1 (the UI object is located within the field of view). Level 2 offers a relatively high visual experience, and the preset conditions for Level 2 can be set to satisfy conditions 1 and 4 (the focus time of the UI object is less than a second preset position in the focus ranking; the focus ranking is obtained by sorting the focus times of the multiple UI objects in chronological order; the focus time is the time when the corresponding UI object is recorded as the focus UI after the interaction event is injected). Level 3 offers the lowest visual experience, and the preset conditions for Level 3 can be set to satisfy conditions 1, 3 (the ratio between the area of ​​the non-obscured part of the UI object and the total area of ​​the UI object is less than a first preset position in the ratio ranking; the ratio ranking is obtained by sorting the ratio between the area of ​​the non-obscured part of the multiple UI objects and the corresponding total area from largest to smallest) and condition 4.

[0076] In one embodiment of this disclosure, the filtering strategy can be dynamically adjusted based on the current system state (such as CPU and GPU load, battery level, etc.). When system resources are scarce, the pass-through level of non-critical UIs can be further reduced to ensure that core applications and recently interacted UIs can obtain sufficient resources. When resources are plentiful, restrictions can be relaxed, allowing more UIs to pass through in high definition. By reducing unnecessary GPU rendering work, the dynamic resolution strategy helps reduce overall power consumption, thereby indirectly improving the all-in-one device's battery life. Especially when powered by battery, this strategy can significantly extend usage time without sacrificing user experience.

[0077] refer to Figure 4 , Figure 4 Schematic diagram of the image rendering method provided in the embodiments of this disclosure Figure 2 This embodiment exemplarily describes a method for determining the priority of UI objects based on conditions such as UI type, whether they are within the field of view, and their position in the focus sort. The first processing method is to render and display the image at a first resolution, and the second processing method is to render and display the image at a second resolution. This image rendering method includes:

[0078] 401. Obtain multiple UI objects to be rendered.

[0079] 402. For each of the plurality of UI objects, determine whether the UI object belongs to a preset type. If yes, proceed to step 403; otherwise, proceed to step 404.

[0080] 403. Render the UI object at the first resolution.

[0081] 404. Determine whether the position of the UI object is within the field of view. If yes, proceed to step 405; otherwise, proceed to step 406.

[0082] 405. Determine that the focus time corresponding to the UI object ranks less than a second preset rank in the focus sorting; the focus sorting is obtained by arranging the focus times corresponding to the multiple UI objects in chronological order; the focus time is the time when the corresponding UI object is recorded as the focus UI after the interaction event is injected. If yes, proceed to step 403. Otherwise, proceed to step 406.

[0083] 406. Render the UI object at a second resolution; the first resolution is greater than the second resolution.

[0084] Specifically, for multiple UI objects in an image frame to be rendered, dynamic filtering can be performed based on conditions such as the type, visibility, and recent interaction interval of each UI object. High-priority UI objects are selected and rendered at a higher resolution (first resolution). The remaining low-priority objects are rendered at a second resolution, which is lower than the first resolution.

[0085] For example, suppose the preset type is a non-third-party, non-resident type, that is, the UI of the all-in-one machine's built-in application. Figure 5 As shown, first determine whether the UI type is a third-party non-resident UI. Figure 5 In the multi-tasking large-screen window, non-third-party, non-resident application UIs are considered high-frequency UIs and are always passed through to ensure high definition. Secondly, the position and size of the UI object are determined based on the view frustum position and orientation of the all-in-one device to determine if the UI in the VR space is within the FOV (field of view) of the all-in-one device user. If any point of the UI object is within the view frustum, the UI object is considered within the FOV. For UI objects outside the user's field of view, non-passthrough is always used to reduce performance overhead and improve the all-in-one device's battery life. For UI objects within the FOV, the proportion of their exposed portion (i.e., appearing within the FOV and not obscured by other content) can be further calculated. If the unobscured proportion is higher than a threshold, or ranks among the top n in terms of unobscured proportion, then passthrough is used; otherwise, it is not. Finally, all-in-one device users can interact with the UI using a controller or gestures. When using a controller, the all-in-one device tracks the controller's position and orientation in real time, displaying a ray based on the controller's orientation and position. The point of collision between the ray and the UI plane is the interaction point. Users inject interactive events (such as confirmation, back, drag) into the interaction point using controller buttons. When using gestures, the user's hand position and orientation are acquired in real time to calculate the interaction point, and fixed gestures are used to trigger events such as confirmation, return, and dragging. Whenever the all-in-one machine user interacts with a UI element, injecting events such as confirmation, return, or dragging, that UI element is recorded as the focused UI, i.e., it gains focus. Panels filtered according to the above conditions are sorted according to the order in which they gained focus, with those gaining focus more recently appearing earlier. The first n UI elements (e.g., n=5) are passed through to ensure high clarity for frequently used UI objects, while other UI elements are not passed through to reduce performance overhead and improve the all-in-one machine's battery life.

[0086] As described above, during the use of the all-in-one device, UI objects in the center of the field of vision are high-resolution, while those at the edge or outside the field of vision are low-resolution. The UI elements that the user has recently interacted with (e.g., 5) are high-resolution, while the rest are low-resolution. The resolution of UI objects will dynamically change according to these conditions as the user interacts with different panels and moves freely while wearing the all-in-one device.

[0087] In one embodiment of this disclosure, the UI object includes multiple elements; each of the multiple elements corresponds to a preset condition; if the feature data of the UI object satisfies the first preset condition, the UI object is rendered and displayed using a first processing method; if the feature data of the UI object does not satisfy the first preset condition, the UI object is rendered and displayed using a second processing method; the resource consumption of the first processing method is greater than the resource consumption of the second processing method, which may include: for each element of the UI object, if the feature data of the UI object satisfies the preset condition corresponding to the element, the element is rendered and displayed using the first processing method; if the feature data of the UI object does not satisfy the preset condition corresponding to the element, the element is rendered and displayed using the second processing method, wherein the resource consumption of the first processing method is greater than the resource consumption of the second processing method.

[0088] For example, suppose a UI object includes element 1, element 2, and element 3. Each element has its own corresponding preset condition; for instance, element 1 corresponds to condition A, element 2 to condition B, and element 3 to condition C. Based on this, some elements may meet their corresponding preset conditions and be processed using the first processing method, while other elements may not meet their corresponding preset conditions and be processed using the second processing method. For example, when the UI object's feature data meets conditions A and B but not condition C, then elements 1 and 2 are processed using the first processing method, resulting in higher resource consumption, while element 3 is processed using the second processing method, resulting in lower resource consumption. This further reduces the overall resource consumption.

[0089] It should be noted that both the first and second processing methods can include multiple methods, as long as the resource consumption of each method belonging to the first processing method is greater than the resource consumption of each method belonging to the second processing method. Based on the above example, the first processing method can include method 1 and method 2, and the second processing method can include method 3 and method 4. Element 1 and element 2 are both processed using the first processing method, which can include: processing element 1 using method 1 of the first processing method and processing element 2 using method 2 of the second processing method, i.e., elements 1 and 2 use different first processing methods. Element 3 is processed using the second processing method, which can include: processing element 3 using method 3 of the second processing method, or processing element 4 using method 4 of the second processing method.

[0090] In one embodiment of this disclosure, the feature data of the UI object satisfying the preset conditions corresponding to the element may include: the type of the UI object belongs to a preset type.

[0091] In one embodiment of this disclosure, when the type of the UI object does not belong to a preset type, the feature data of the UI object satisfying the preset conditions corresponding to the element includes satisfying at least one of the following conditions: Condition a: The position of the UI object is within the field of view; Condition b: The ratio between the area of ​​the unobstructed part of the UI object and the total area of ​​the UI object is greater than a second preset ratio; Condition c: The ratio between the area of ​​the unobstructed part of the UI object and the total area of ​​the UI object ranks less than a third preset position in the ratio sorting; the ratio sorting is obtained by sorting the ratio between the area of ​​the unobstructed part of the multiple UI objects and the corresponding total area from largest to smallest; Condition d: The focus time corresponding to the UI object ranks less than a fourth preset position in the focus sorting; the focus sorting is obtained by sorting the focus times corresponding to the multiple UI objects in chronological order; the focus time is the time when the corresponding UI object is recorded as the focus UI after the interaction event injection is completed; Condition e: The element is being used.

[0092] For details, please refer to the descriptions of conditions 1-5 above; they will not be repeated here.

[0093] refer to Figure 6 , Figure 6 Schematic diagram of the image rendering method provided in the embodiments of this disclosure Figure 4 In this embodiment, the UI object includes multiple elements, such as a first element and a second element. The example describes how different dynamic filtering strategies are applied to different elements within the UI object. The image rendering method includes:

[0094] 601. Obtain multiple UI objects to be rendered.

[0095] 602. For each of the plurality of UI objects, determine whether the feature data of the UI object meets the first preset condition. If yes, proceed to step 603; otherwise, proceed to step 604.

[0096] 603. Render and display the first element of the UI object at a first resolution.

[0097] 604. Render the first element of the UI object at a second resolution; the first resolution is greater than the second resolution.

[0098] 605. Determine if the feature data of the UI object satisfies the second preset condition; if so, execute. 606. If not, proceed to step 607.

[0099] 606. Render and display the second element of the UI object at a first resolution.

[0100] 607. Render and display the second element of the UI object at a second resolution.

[0101] Specifically, a UI object can include multiple elements. For example, a UI object can be an application window, including the application's displayed content and the operation controls below the content (which can be a combination of bars or dots). The importance of the same UI object may vary, so different dynamic filtering strategies can be used for different elements of the UI object. This allows less important elements to be rendered at a lower resolution, minimizing their use of graphics card resources and freeing up more rendering load for the GPU.

[0102] In one embodiment of this disclosure, the feature data of the UI object satisfying the second preset condition includes: the type of the UI object belongs to a preset type.

[0103] In one embodiment of this disclosure, when the type of the UI object does not belong to a preset type, the feature data of the UI object satisfying the second preset condition includes satisfying at least one of the following conditions: Condition 6: The position of the first element of the UI object is within the field of view; Condition 7: The ratio between the area of ​​the non-occluded part of the first element of the UI object and the total area of ​​the UI object is greater than a second preset ratio; Condition 8: The ratio between the area of ​​the non-occluded part of the first element of the UI object and the total area of ​​the UI object ranks less than a third preset position in the ratio sorting; the ratio sorting is obtained by sorting the ratio between the area of ​​the non-occluded part of the plurality of UI objects and the corresponding total area from large to small; Condition 9: The focus time corresponding to the UI object ranks less than a fourth preset position in the focus sorting; the focus sorting is obtained by sorting the focus times corresponding to the plurality of UI objects in chronological order; the focus time is the time when the corresponding UI object is recorded as the focus UI after the interaction event injection is completed; Condition 10: The second element of the UI object is being used.

[0104] Specifically, when the preset type is used as a high-priority condition, for UI objects that do not meet the preset type, further judgment can be made based on conditions 6-10. Specifically, one or more conditions can be selected and combined to obtain the first preset condition. For example, condition 6 can be used as the first preset condition; conditions 6 and 7 can be used as the first preset condition; conditions 6-8 can be used as the first preset condition; conditions 6-10 can be used as the first preset condition; conditions 6, 7, and 9 can be used as the first preset condition; conditions 6, 8, and 9 can be used as the first preset condition; conditions 7 and 9 can be used as the first preset condition, and so on.

[0105] For example, such as Figure 7As shown, the host window, i.e., the UI object, undergoes the following process: First, it checks if the UI object's type is a preset type (i.e., not a third-party, non-resident application UI). If it is a preset type, meaning it's not a multi-tasking large-screen window, it's passed through and rendered at a higher resolution (first resolution). If it is a multi-tasking large-screen window, further checks are performed. In this further check, it checks if the UI object's window is within the field of view. If it's not within the field of view, it's not passed through and rendered at a lower resolution (second resolution). If it is within the field of view, it's further sorted by focus acquisition time, and the top n (e.g., 3) UI objects are selected for passing through, while the rest are not. Finally, it checks if the UI object is currently being interacted with, such as clicked or dragged. If it is, it's passed through.

[0106] As described above, by dynamically filtering different elements of a UI object using different strategies, the filtered elements are rendered and displayed with higher resolution. This adapts to situations where different elements of a UI object have varying degrees of importance, rendering important UI elements with higher resolution and less important elements with lower resolution, thus striking a balance between device performance and user visual experience. Simultaneously, it maximizes battery life.

[0107] Corresponding to the image rendering method in the above embodiments, Figure 8 This is a structural block diagram of an image rendering apparatus provided in an embodiment of the present disclosure. For ease of explanation, only elements relevant to embodiments of the present disclosure are shown. (Refer to...) Figure 8 The device includes an acquisition module 801 and a rendering module 802.

[0108] Among them, the acquisition module 801 is used to acquire multiple UI objects to be rendered;

[0109] The rendering module 802 is used to render and display each UI object among the plurality of UI objects according to a first processing method if the feature data of the UI object meets a first preset condition; and to render and display the UI object according to a second processing method if the feature data of the UI object does not meet the first preset condition; wherein the resource consumption of the first processing method is greater than the resource consumption of the second processing method.

[0110] In one embodiment of this disclosure, the rendering module 802 is specifically used to: render and display the UI object at a first resolution; render and display the UI object at a second resolution; wherein the first resolution is greater than the second resolution.

[0111] In one embodiment of this disclosure, the rendering module 802 is specifically configured to: refresh the window of the UI object at a first refresh rate; refresh the window of the UI object at a second refresh rate; wherein the first refresh rate is greater than the second refresh rate.

[0112] In one embodiment of this disclosure, the rendering module 802 is specifically used to: render the window of the UI object at a first resolution; render the window of the UI object at a second resolution; wherein the first resolution is greater than the second resolution.

[0113] In one embodiment of this disclosure, the rendering module 802 is specifically configured to: freeze the process corresponding to the UI object when rendering and displaying the UI object based on the second processing method, and display a first image rendered and displayed before the process was frozen in the window of the UI object; or,

[0114] The process corresponding to the UI object is terminated, and the second image rendered before the process termination is displayed in the window of the UI object.

[0115] In one embodiment of this disclosure, the feature data of the UI object includes at least one of the following: type, location, area of ​​the unobstructed portion, and interaction data.

[0116] In one embodiment of this disclosure, the feature data of the UI object satisfying the first preset condition includes: the feature data of the UI object belonging to a preset type.

[0117] In one embodiment of this disclosure, the preset type includes a security type.

[0118] In one embodiment of this disclosure, the preset type includes a custom type; the acquisition module 801 is further configured to: respond to a touch operation on the first control in the settings interface and determine the type of the selected UI object as a custom type.

[0119] In one embodiment of this disclosure, when the type of the UI object does not belong to a preset type; the feature data of the UI object satisfying the first preset condition includes satisfying at least one of the following conditions:

[0120] The UI object is located within the field of view;

[0121] The ratio between the area of ​​the unobstructed portion of the UI object and the total area of ​​the UI object is greater than a first preset ratio;

[0122] The ratio between the area of ​​the unobstructed portion of the UI object and the total area of ​​the UI object has a lower position in the ratio sorting than the first preset position; the ratio sorting is obtained by sorting the ratio between the area of ​​the unobstructed portion of each of the multiple UI objects and the corresponding total area from large to small.

[0123] The focus time corresponding to the UI object is ranked less than the second preset position in the focus sorting; the focus sorting is obtained by sorting the focus times corresponding to the multiple UI objects in chronological order; the focus time is the time when the corresponding UI object is recorded as the focus UI after the interaction event is injected.

[0124] In one embodiment of this disclosure, when the type of the UI object does not belong to a preset type, the feature data of the UI object satisfying the first preset condition includes: the position of the UI object is within the field of view, and the position of the focus time corresponding to the UI object in the focus sorting is less than the second preset position; or, the position of the UI object is within the field of view, and the position of the focus time corresponding to the UI object in the focus sorting is less than the second preset position, and the ratio between the area of ​​the non-obstructed part of the UI object and the total area of ​​the UI object in the ratio sorting is less than the first preset position.

[0125] In one embodiment of this disclosure, the acquisition module 801 is further configured to: acquire the current pose of the head-mounted device; and determine the field of view of the head-mounted device based on the current pose.

[0126] In one embodiment of this disclosure, the acquisition module 801 is specifically used to: acquire the current eye-tracking result; and determine the field of view of the head-mounted device based on the current pose and the eye-tracking result.

[0127] In one embodiment of this disclosure, the UI object includes multiple elements; each of the multiple elements corresponds to a preset condition; the rendering module 802 is specifically used to: for each element of the UI object, if the feature data of the UI object meets the preset condition corresponding to the element, then the element is rendered and displayed in a first processing mode; if the feature data of the UI object does not meet the preset condition corresponding to the element, then the element is rendered and displayed in a second processing mode.

[0128] In one embodiment of this disclosure, the feature data of the UI object satisfying the preset conditions corresponding to the element includes: the type of the UI object belongs to a preset type.

[0129] In one embodiment of this disclosure, when the type of the UI object does not belong to a preset type, the feature data of the UI object satisfying the preset conditions corresponding to the element includes satisfying at least one of the following conditions:

[0130] The UI object is located within the field of view;

[0131] The ratio between the area of ​​the unobstructed portion of the UI object and the total area of ​​the UI object is greater than the second preset ratio;

[0132] The ratio between the area of ​​the unobstructed portion of the UI object and the total area of ​​the UI object ranks less than the third preset position in the ratio sorting; the ratio sorting is obtained by sorting the ratio between the area of ​​the unobstructed portion of each of the multiple UI objects and the corresponding total area from largest to smallest.

[0133] The focus time corresponding to the UI object is less than the fourth preset position in the focus sorting; the focus sorting is obtained by sorting the focus times corresponding to the multiple UI objects in chronological order; the focus time is the time when the corresponding UI object is recorded as the focus UI after the interaction event is injected.

[0134] The element is being used.

[0135] In one embodiment of this disclosure, the rendering module 802 is specifically used to: based on pass-through layer technology, directly submit the scene content corresponding to the UI object to the asynchronous time-warped (ATW) thread for processing without going through the rendering thread, and then render and display it on the screen of the head-mounted device with a first resolution; submit the scene content corresponding to the UI object to the rendering thread for rendering by the left and right eye cameras, and submit the data rendered in the left and right eye rendering buffers EyeBuffer to the asynchronous time-warped (ATW) thread for processing, and then render and display it on the screen of the head-mounted device with a second resolution.

[0136] In one embodiment of this disclosure, the acquisition module 801 is further configured to: respond to a touch operation applied to a second control in the settings interface, determine a target visual experience level; different visual experience levels correspond to different degrees of strictness of the preset conditions for rendering the UI object with a first clarity; and determine the preset conditions corresponding to the target level as the first preset conditions.

[0137] The device provided in this embodiment can be used to execute the technical solutions of the above method embodiments. Its implementation principle and technical effect are similar, and will not be described again here.

[0138] To implement the above embodiments, this disclosure also provides an electronic device.

[0139] refer to Figure 9The diagram illustrates a structural schematic of an electronic device 900 suitable for implementing embodiments of the present disclosure. The electronic device 900 can be a terminal device or a server. The terminal device can include, but is not limited to, mobile terminals such as mobile phones, laptops, digital radio receivers, personal digital assistants (PDAs), portable Android devices (PADs), portable media players (PMPs), and in-vehicle terminals (e.g., in-vehicle navigation terminals), as well as fixed terminals such as digital TVs and desktop computers. Figure 9 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0140] like Figure 9 As shown, the electronic device 900 may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 901, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 902 or a program loaded from a storage device 908 into a random access memory (RAM) 903. The RAM 903 also stores various programs and data required for the operation of the electronic device 900. The processing unit 901, ROM 902, and RAM 903 are interconnected via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0141] Typically, the following devices can be connected to I / O interface 905: input devices 906 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 907 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 908 including, for example, magnetic tapes, hard disks, etc.; and communication devices 909. Communication device 909 allows electronic device 900 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 9 An electronic device 900 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0142] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 909, or installed from a storage device 908, or installed from a ROM 902. When the computer program is executed by a processing device 901, it performs the functions defined in the methods of embodiments of this disclosure.

[0143] It should be noted that the computer-readable medium described in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as a carrier element, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0144] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0145] The aforementioned computer-readable medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to perform the methods shown in the above embodiments.

[0146] Computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0147] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or element of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0148] The units described in the embodiments of this disclosure can be implemented in software or in hardware. The name of a unit does not necessarily limit the unit itself; for example, the first acquisition unit can also be described as "a unit that acquires at least two Internet Protocol addresses".

[0149] The functions described above in this document can be performed, at least elementally, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0150] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0151] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features disclosed in this disclosure that have similar functions.

[0152] Furthermore, while the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order. In certain environments, multitasking and parallel processing may be advantageous. Similarly, while several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this disclosure. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.

[0153] Although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative examples of implementing the claims.

Claims

1. An image rendering method, characterized in that, Applied to a head-mounted device, the method includes: Obtain multiple user graphical interface (UI) objects to be rendered; For each of the plurality of UI objects, if the feature data of the UI object meets the first preset condition, the UI object is rendered and displayed based on the first processing method; if the feature data of the UI object does not meet the first preset condition, the UI object is rendered and displayed based on the second processing method; the resource consumption of the first processing method is greater than the resource consumption of the second processing method.

2. The method according to claim 1, characterized in that, The rendering and display of the UI object based on the first processing method includes: The UI object is rendered and displayed at a first resolution; The rendering and display of the UI object based on the second processing method includes: The UI object is rendered and displayed at a second resolution; the first resolution is greater than the second resolution.

3. The method according to claim 1, characterized in that, The rendering and display of the UI object based on the first processing method includes: Refresh the window of the UI object at a first refresh rate; The rendering and display of the UI object based on the second processing method includes: The window of the UI object is refreshed at a second refresh rate; the first refresh rate is greater than the second refresh rate.

4. The method according to claim 1, characterized in that, The rendering and display of the UI object based on the first processing method includes: Render the window of the UI object at a first resolution; The rendering and display of the UI object based on the second processing method includes: The window of the UI object is rendered at a second resolution; the first resolution is greater than the second resolution.

5. The method according to claim 1, characterized in that, The rendering and display of the UI object based on the second processing method includes at least one of the following: Freeze the process corresponding to the UI object, and display the first image rendered before the process was frozen in the window of the UI object; The process corresponding to the UI object is terminated, and the second image rendered before the process termination is displayed in the window of the UI object.

6. The method according to claim 1, characterized in that, The characteristic data of the UI object includes at least one of the following: type, location, area of ​​the unobstructed part, and interaction data.

7. The method according to claim 1, characterized in that, The feature data of the UI object satisfying the first preset condition includes: The feature data of the UI object belongs to a preset type.

8. The method according to claim 7, characterized in that, The preset types include security types.

9. The method according to claim 7, characterized in that, If the type of the UI object does not belong to a preset type; the feature data of the UI object satisfies the first preset condition, including satisfying at least one of the following conditions: The UI object is located within the field of view; The ratio between the area of ​​the unobstructed portion of the UI object and the total area of ​​the UI object is greater than a first preset ratio; The ratio between the area of ​​the unobstructed portion of the UI object and the total area of ​​the UI object has a lower position in the ratio sorting than the first preset position; the ratio sorting is obtained by sorting the ratio between the area of ​​the unobstructed portion of each of the multiple UI objects and the corresponding total area from large to small. The focus time corresponding to the UI object has a lower position in the focus sorting than the second preset position; the focus sorting is obtained by sorting the focus times corresponding to the multiple UI objects in chronological order. The focus time is the time when the corresponding UI object is recorded as the focus UI after the interaction event injection is completed.

10. The method according to claim 7, characterized in that, If the type of the UI object does not belong to the preset type; The feature data of the UI object satisfying the first preset condition includes: The UI object is located within the field of view, and the focus time corresponding to the UI object is ranked less than the second preset position in the focus sorting. or, The UI object is located within the field of view, and the focus time corresponding to the UI object ranks less than the second preset rank in the focus sorting, and the ratio between the area of ​​the unobstructed part of the UI object and the total area of ​​the UI object ranks less than the first preset rank in the ratio sorting.

11. The method according to claim 9, characterized in that, The method further includes: Obtain the current pose of the head-mounted device; The field of view of the head-mounted device is determined based on the current pose.

12. The method according to claim 11, characterized in that, Determining the field of view of the head-mounted device based on the current pose includes: Obtain the current eye-tracking results; The field of view of the head-mounted device is determined based on the current pose and the eye-tracking results.

13. The method according to any one of claims 1-12, characterized in that, The UI object includes multiple elements; each of the multiple elements corresponds to its own preset condition; if the feature data of the UI object satisfies the first preset condition, the UI object is rendered and displayed based on a first processing method; if the feature data of the UI object does not satisfy the first preset condition, the UI object is rendered and displayed based on a second processing method, including: For each element of the UI object, if the feature data of the UI object meets the preset conditions corresponding to the element, the element is rendered and displayed using a first processing method; if the feature data of the UI object does not meet the preset conditions corresponding to the element, the element is rendered and displayed using a second processing method.

14. The method according to claim 13, characterized in that, The feature data of the UI object satisfies the preset conditions corresponding to the element, including: The type of the UI object belongs to the preset type.

15. The method according to claim 14, characterized in that, If the type of the UI object is not a preset type, the feature data of the UI object satisfies the preset conditions corresponding to the element, including satisfying at least one of the following conditions: The UI object is located within the field of view; The ratio between the area of ​​the unobstructed portion of the UI object and the total area of ​​the UI object is greater than the second preset ratio; The ratio between the area of ​​the unobstructed portion of the UI object and the total area of ​​the UI object ranks less than the third preset position in the ratio sorting; the ratio sorting is obtained by sorting the ratio between the area of ​​the unobstructed portion of each of the multiple UI objects and the corresponding total area from largest to smallest. The focus time corresponding to the UI object is less than the fourth preset position in the focus sorting; the focus sorting is obtained by sorting the focus times corresponding to the multiple UI objects in chronological order. The focus time is the time when the corresponding UI object is recorded as the focus UI after the interaction event injection is completed; The element is being used.

16. The method according to claim 1, characterized in that, The rendering and display of the UI object based on the first processing method includes: Based on the pass-through layer technology, the scene content corresponding to the UI object is directly submitted to the asynchronous time warp (ATW) thread for processing without going through the rendering thread, and then rendered and displayed on the screen of the head-mounted device in the first resolution. The rendering and display of the UI object based on the second processing method includes: The scene content corresponding to the UI object is submitted to the rendering thread for rendering by the left and right eye cameras. The data rendered in the left and right eye rendering buffers is submitted to the asynchronous time warp (ATW) thread for processing, and then rendered and displayed on the screen of the head-mounted device in second-resolution mode.

17. An image rendering device, characterized in that, include: The retrieval module is used to retrieve multiple UI objects to be rendered; The rendering module is used to render and display each UI object based on a first processing method if the feature data of the UI object meets a first preset condition. If the feature data of the UI object does not meet the first preset condition, the UI object is rendered and displayed based on the second processing method; the resource consumption of the first processing method is greater than the resource consumption of the second processing method.

18. An electronic device, characterized in that, include: Processor and memory; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory, causing the processor to perform the image rendering method as described in any one of claims 1 to 16.

19. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, implement the image rendering method as described in any one of claims 1 to 16.

20. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the image rendering method as described in any one of claims 1 to 16.

Images