Head-mounted device and data rendering method

Abstract

This invention discloses a head-mounted device and a data rendering method. The head-mounted device includes a storage component, a system-on-a-chip (SoC), and an output component. The storage component stores rendering resources. The SoC includes a first type of computing unit with a higher power consumption limit and a second type of computing unit with a lower power consumption limit. The first type of computing unit reads the corresponding rendering resources from the storage component according to the rendering task and transmits the rendering resources to the second type of computing unit. The second type of computing unit determines the rendering instance corresponding to the rendering task, generates rendering data and rendering instructions corresponding to the rendering instance, and performs rendering operations in a collaborative manner based on the rendering instructions, the rendering resources, and the rendering data to generate a binocular parallax image. The output component outputs the image. This invention implements rendering resource access in the high-power domain and data rendering computation in the low-power domain, thereby effectively improving the battery life of the head-mounted device.

Description

Technical Field

[0001] This invention relates to the field of smart wearable device technology, and more specifically, to a head-mounted device and a data rendering method. Background Technology

[0002] With the development of technology, more and more wearable smart devices are entering people's daily lives. Head-mounted devices are a common type of wearable smart device, which can include, for example, virtual reality (VR) glasses, augmented reality (AR) glasses, mixed reality (MR) glasses, or artificial intelligence (AI) glasses.

[0003] To ensure a good user experience, headsets typically include a variety of functions. However, headsets usually have limited power capacity, so performing various functions increases power consumption and significantly shortens battery life. Shortened battery life, in turn, reduces the user experience. Summary of the Invention

[0004] In view of this, embodiments of the present invention provide a head-mounted device and a data rendering method to improve the battery life of the head-mounted device by dividing the on-chip system into a high-power domain and a low-power domain, implementing the access to rendering resources in the high-power domain, and implementing data rendering calculations in the low-power domain.

[0005] In a first aspect, embodiments of the present invention provide a head-mounted device, the device comprising: The storage component is configured to store rendering resources; The system-on-a-chip includes a first type of computing unit and a second type of computing unit, wherein the power consumption limit of the first type of computing unit is greater than the power consumption limit of the second type of computing unit; and The output component is configured to output a binocular parallax image; The first type of computing unit is configured to read the target rendering resource from the storage component according to the resource loading instruction corresponding to the rendering task, and to transmit the target rendering resource to the second type of computing unit; The second type of computing unit is configured to determine the target rendering instance corresponding to the rendering task, generate the resource loading instruction, generate target rendering data and rendering instructions based on the target rendering instance, and perform rendering operations in a collaborative manner based on the rendering instructions, the target rendering resources and the target rendering data to generate the binocular parallax image.

[0006] Optionally, the head-mounted device further includes RAM; The first type of computing unit is further configured as follows: Each initial rendering resource is loaded into the runtime memory, and each initial rendering resource is decoded into corresponding runtime resource data in the runtime memory. Each runtime resource data is compressed in the runtime memory according to its data characteristics to obtain a compressed resource data block. The compressed resource data block is written into the storage component to store the rendering resources in a compressed form.

[0007] Optionally, the second type of computing unit includes a central processing unit and a graphics processing unit; The central processing unit is also configured to: The rendering instructions are distributed according to the task type of the rendering task, so that the central processing unit or the graphics processing unit can perform rendering calculations based on the rendering instructions, the target rendering resources and the target rendering data to generate the binocular parallax image.

[0008] Optionally, the central processing unit is further configured to: The rendering instructions are merged based on their instruction similarity and task type to determine a rendering instruction group, and the rendering instruction group is distributed according to the task type.

[0009] Optionally, the graphics processing unit is a two-dimensional graphics processing unit.

[0010] Optionally, the head-mounted device further includes a decompression component; The decompression component is configured as follows: The target rendering resource is decompressed to obtain the decompressed target rendering resource, so that the second type of computing unit can generate the binocular parallax image based on the decompressed target rendering resource.

[0011] Optionally, the binocular parallax image includes a first image and a second image, and the output component includes a first display component and a second display component; The first display component is configured to output the first image; The second display component is configured to output the second image.

[0012] Secondly, embodiments of the present invention provide a data rendering method applicable to head-mounted devices, wherein the head-mounted device includes a first type of computing unit and a second type of computing unit, wherein the power consumption limit of the first type of computing unit is greater than the power consumption limit of the second type of computing unit, and the method includes: Determine the target rendering instance corresponding to the rendering task; Load the target rendering resources according to the resource loading instructions corresponding to the rendering task; Generate target rendering data and rendering instructions based on the target rendering instance; Rendering calculations are performed using a multi-second type of computing unit in a collaborative manner based on the rendering instructions, the target rendering resources, and the target rendering data to generate a binocular parallax image; Output the binocular parallax image.

[0013] Thirdly, embodiments of the present invention provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described in the second aspect.

[0014] Fourthly, embodiments of the present invention provide a computer program product, the computer program product including a computer program / instruction, which, when executed by a processor, implements the method described in the second aspect.

[0015] The head-mounted device of this invention includes a storage component, a system-on-a-chip (SoC), and an output component. The storage component stores rendering resources. The SoC includes a first type of computing unit with a higher power consumption limit and a second type of computing unit with a lower power consumption limit. The first type of computing unit reads the corresponding rendering resources from the storage component according to the rendering task and transmits the rendering resources to the second type of computing unit. The second type of computing unit determines the rendering instance corresponding to the rendering task, generates rendering data and rendering instructions corresponding to the rendering instance, and performs rendering operations in a collaborative manner based on the rendering instructions, the rendering resources, and the rendering data to generate a binocular parallax image. The output component outputs the image. This invention implements rendering resource access in the high-power domain and data rendering computation in the low-power domain, thereby effectively improving the battery life of the head-mounted device. Attached Figure Description

[0016] The above and other objects, features and advantages of the present invention will become clearer from the following description of embodiments of the invention with reference to the accompanying drawings, in which: Figure 1 This is a schematic diagram of the hardware structure of the head-mounted device according to an embodiment of the present invention; Figure 2 This is a structural block diagram of a head-mounted device according to an embodiment of the present invention; Figure 3 This is a flowchart of the data rendering method according to an embodiment of the present invention; Figure 4 This is a flowchart of the data rendering method according to an embodiment of the present invention; Figure 5 This is a flowchart of the data rendering method according to an embodiment of the present invention; Figure 6This is a schematic diagram of the data rendering process in an embodiment of the present invention. Detailed Implementation

[0017] The present application is described below based on embodiments, but it is not limited to these embodiments. In the detailed description of the present application below, certain specific details are described in detail. Those skilled in the art can fully understand the present application without these details. To avoid obscuring the substance of the present application, well-known methods, processes, flows, elements, and circuits are not described in detail.

[0018] Furthermore, those skilled in the art should understand that the accompanying drawings provided herein are for illustrative purposes only and are not necessarily drawn to scale.

[0019] Unless the context explicitly requires it, words such as "including" or "contains" throughout the application should be interpreted as including rather than exclusive or exhaustive; that is, meaning "including but not limited to".

[0020] In the description of this application, it should be understood that the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance. Furthermore, in the description of this application, unless otherwise stated, "a plurality of" means two or more.

[0021] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0022] Figure 1 This is a schematic diagram of the hardware structure of a head-mounted device according to an embodiment of the present invention. Figure 2 This is a structural block diagram of a head-mounted device according to an embodiment of the present invention. Figure 1 and Figure 2 As shown, the head-mounted device 1 in this embodiment can be smart glasses, such as at least one of virtual reality glasses, augmented reality glasses, mixed reality glasses, artificial intelligence glasses, etc. The head-mounted device 1 includes a storage component 10, an output component 20, and a system-on-a-chip 30, and the storage component 10, the output component 20, and the system-on-a-chip 30 are respectively disposed at predetermined positions on the device body 40.

[0023] In this embodiment of the invention, the components in the head-mounted device 1, including the storage component 10, the output component 20, and the system-on-a-chip 30, are divided into a high-power domain and a low-power domain according to the upper limit of power consumption. The high-power domain and the low-power domain can be driven by two different power supply circuits, and the rated voltage of the power supply circuit used to drive the high-power domain is higher than the rated voltage of the power supply circuit used to drive the low-power domain.

[0024] Storage component 10 is used to store rendering resources. For example, the storage component can be at least one of an embedded multi-media card (EMMC), universal flash storage (UFS), or NOR-type flash memory. In some embodiments, storage component 10 can be located in a high-power domain.

[0025] The output component 20, also known as the display component, is used to output binocular parallax images. The output component 20 may include a waveguide and a projection mechanism. The waveguide is a flat optical element based on a transparent substrate (e.g., glass or polymer material), with diffraction gratings or geometric reflection structures integrated internally or on its surface. The on-chip system 30 controls the projection mechanism to couple emitted imaging light into the waveguide. The imaging light is replicated and expanded laterally or vertically within the waveguide and then coupled out to the human eye, forming a virtual image. Furthermore, the wearer can observe the environment and objects in front of their line of sight through the waveguide. The projection mechanism may include a display chip and a lens group. The display chip generates a high-resolution image through pixel-level modulation. The projection lens group may consist of multiple aspherical lenses or freeform lenses, used to relay the image plane of the display chip to the coupling area of ​​the waveguide. The display chip may include a liquid crystal on silicon, a digital micromirror, or a microdisplay. In some embodiments, the output component 20 may include two independent display components, each with its own independent projection mechanism. In some embodiments, the output component 20 can be a set of display components, and the on-chip system 30 controls a single projection mechanism to split the emitted imaging light into two independent image beams through a diffraction grating or a geometric beam splitting reflector, and couples the two image beams into corresponding waveguide sheets respectively. In some embodiments, the output component 20 can be set in the low-power domain.

[0026] The system-on-chip 30 is electrically connected to the storage component 10 and the output component 20, enabling control of the storage component 10 and the output component 20. The system-on-chip 30 may include a first type of computing unit 301 disposed in a high-power domain and a second type of computing unit 302 disposed in a low-power domain; that is, the power consumption limit of the first type of computing unit 301 is greater than the power consumption limit of the second type of computing unit 302. In some embodiments, the first type of computing unit 301 and the second type of computing unit 302 may be different computing cores of the same processor; for example, the first type of computing unit 301 and the second type of computing unit 302 may be different computing cores of a central processing unit (CPU). In some embodiments, the first type of computing unit 301 and the second type of computing unit 302 may be different processors; for example, the first type of computing unit 301 may be a central processing unit, and the second type of computing unit 302 may be a graphics processing unit (GPU). In some embodiments, the first type of computing unit 301 and the second type of computing unit 302 may be different computing cores of the same processor and different processors; for example, the first type of computing unit 301 may be some computing cores of the central processing unit, and the second type of computing unit 302 may be another part of the central processing unit and a GPU. The aforementioned processor may include at least one of a central processing unit (CPU), application processor (AP), modem processor, graphics processing unit (GPU), image signal processor (ISP), video processing unit (VPU), video encoder / decompressor, digital signal processor (DSP), baseband processor, and neural network processing unit (NPU). The system-on-a-chip 30 can generate operation control signals based on instruction opcodes and timing signals to control instruction fetching and execution. The processor may also include a memory for storing instructions and data. The processor executes various functional applications and data processing of the head-mounted device 1 by running the instructions stored in the memory. In some embodiments, the memory is a cache memory.

[0027] The first type of computing unit 301 is used to read the target rendering resources from the storage component 10 according to the rendering task, and transmit the target rendering resources to the second type of computing unit 302. The second type of computing unit 302 determines the target rendering corresponding to the rendering task, and performs rendering calculations in a cooperative manner according to the rendering instructions and target rendering data to generate a binocular parallax image.

[0028] In some embodiments, the head-mounted device 1 further includes running memory. The first type of computing unit 301 is further configured to load each initial rendering resource into the running memory, decode each initial rendering resource into corresponding runtime resource data in the running memory, compress each runtime resource data according to the data characteristics of each runtime resource data in the running memory to obtain compressed resource data blocks, and write the compressed resource data blocks into the storage component 10 to store the rendering resources in compressed form.

[0029] In some embodiments, the second type of computing unit 302 includes a central processing unit and a graphics processing unit. The central processing unit is further configured to distribute rendering instructions according to the task type of the rendering task, so as to perform rendering calculations by the central processing unit or the graphics processing unit according to the rendering instructions and target rendering data to generate a binocular parallax image.

[0030] In some embodiments, the central processing unit is further configured to merge the rendering instructions according to the instruction similarity and task type, determine the rendering instruction group, and distribute the rendering instruction group according to the task type.

[0031] In some embodiments, the graphics processing unit is a two-dimensional graphics processing unit.

[0032] In some embodiments, the head-mounted device 1 further includes a decompression component. The decompression component is used to decompress the target rendering resource to obtain the decompressed target rendering resource, so that the second type of computing unit 302 can generate a binocular parallax image based on the decompressed target rendering resource. Optionally, the decompression component can be located in the low-power domain of the head-mounted device.

[0033] In some embodiments, the output component 20 includes a first display component and a second display component, and the binocular parallax image includes a first image and a second image. The first display component is used to display the first image, and the second display component is used to display the second image.

[0034] Therefore, embodiments of the present invention can improve the battery life of head-mounted devices by dividing the on-chip system into a high-power domain and a low-power domain, implementing the access to rendering resources in the high-power domain, and implementing the rendering calculation process in the low-power domain.

[0035] The following describes the method through examples. Figure 3 This is a flowchart of a data rendering method according to an embodiment of the present invention. Figure 3 As shown, the method in this embodiment may include the following steps: Step S100: Determine the target rendering instance corresponding to the rendering task.

[0036] In practical applications, head-mounted devices can generate rendering tasks in response to user-triggered interactive commands or automatically triggered by the system (such as updates to the wearer's head posture). In this embodiment, to reduce the power consumption of the head-mounted device, the data rendering process is mainly implemented in the low-power domain. Therefore, in this step, the second type of computing unit of the head-mounted device can determine the target rendering instance corresponding to the rendering task.

[0037] In some embodiments, multiple rendering instances can be pre-defined, and a binding relationship can be established between each rendering instance and the resource identifier of the rendering resource. Therefore, the second type of computing unit can filter the pre-defined multiple rendering instances according to the filtering rules carried by the rendering task, such as the rendering viewport range (including camera clipping rectangle, screen visible area, etc.), rendering layer mask (including user interface layer, pop-up layer, hidden layer, etc.), rendering channel (including stereo rendering, main rendering, etc.), culling switch (including occlusion culling, distance culling, etc.), and determine the filtered rendering instance as the target rendering instance corresponding to the rendering task.

[0038] In some embodiments, the second type of computing unit can dynamically generate the corresponding target rendering resource according to the rendering task. Optionally, the second type of computing unit can load the corresponding target rendering resource from the storage component according to the resource location information (such as resource path, resource index, etc.) carried in the rendering task, and create the corresponding target rendering instance according to the valid resource identifier and layout parameters (such as interface size, control position, etc.) of the target rendering resource after loading.

[0039] Optionally, the wearer can trigger interaction commands in various ways. In one optional implementation, if the head-mounted device includes a voice interaction component, the wearer can issue interaction commands through the voice interaction component. The voice interaction component may include a microphone pickup unit, a voice recognition unit, and an intent recognition unit. The microphone pickup unit can be at least one microphone in a microphone array, used to collect voice information emitted by the wearer; the voice recognition unit is used to perform voice recognition on the voice information collected by the microphone pickup unit to determine the text content of the voice information; the intent recognition unit is used to perform intent recognition on the text content of the voice information to determine the intent information of the voice information. Furthermore, the voice interaction component can send the intent information to a second type of computing unit. The second type of computing unit can pre-store the mapping relationship between each intent information and the command, so after receiving the intent information of the voice information, it can determine whether the intent information is used to represent audio acquisition; if so, the second type of computing unit can determine that the wearer has issued an interaction command.

[0040] In one optional implementation, if the head-mounted device includes a command trigger button, the wearer can issue interactive commands through the command trigger button. The command trigger button can be a mechanical switch or a pressure sensing unit; this embodiment is not limited in this regard. The command trigger button can convert the wearer's trigger operation (such as a pressing operation) into a trigger signal and send the trigger signal to the second type of computing unit. The second type of computing unit can pre-store the mapping relationship between each trigger signal and command. Therefore, upon receiving a trigger signal, it can determine whether the trigger signal corresponds to an interactive command. If so, the second type of computing unit can determine that the wearer has issued an interactive command.

[0041] Depending on the hardware structure of the head-mounted device, the second type of computing unit can also determine whether an interaction command has been received in other ways. For example, if the head-mounted device has a touchpad, such as setting the temples as a touchpad, the wearer can perform specific gesture operations through the touchpad. If the second type of computing unit recognizes the specific gesture operation used to trigger magnetic calibration, it can determine that the wearer has issued an interaction command.

[0042] Step S200: Load the target rendering resource according to the resource loading instruction corresponding to the rendering task.

[0043] In this embodiment, both the storage component and the first type of computing unit can be located in the high-power domain. In some embodiments, rendering resources can be used to implement 3D rendering, and therefore, rendering resources can include types such as texture resources, 3D geometry resources, and lighting rendering resources. Among them, texture resources can include RGB color texture images (such as real images, real videos, and 3D model textures captured by a head-mounted device through a camera), alpha transparency maps, material rendering maps (such as normal maps, roughness / metal maps, specular maps, etc.); 3D geometry resources can include vertex resources (such as vertex coordinates, vertex colors, texture coordinates, etc.), model topology (such as facet index resources, triangle index resources, etc.), skeletal animation resources (such as bone transformation matrices, vertex weights, etc.), spatial interaction resources (such as bounding boxes, collider resources, etc.); lighting rendering resources can include lighting probes, LUT (Look-Up Table) color mapping, ambient light maps, static light source configuration maps, shadow maps, etc.

[0044] In some embodiments, rendering resources can be used for 2D rendering, and therefore, rendering resources may include texture resources, geometry buffer resources, shading and material resources, and lighting rendering resources. Texture resources may include RGB color texture images, alpha transparency maps, LUT color maps, user interface sprite atlases, font atlases, normal maps, specular maps, etc.; geometry buffer resources may include 2D vertex buffer resources, index buffer resources, etc.; shading and material resources may include 2D dedicated shader programs, sprite material parameters, user interface material configurations, etc.; lighting rendering resources may include 2D shadow maps, simple ambient light maps, etc.

[0045] Therefore, in this step, the second type of computing unit of the head-mounted device can load the target rendering resource corresponding to the rendering task into the low-power domain through communication methods such as inter-core communication with the first type of computing unit. Furthermore, the second type of computing unit can generate a resource loading instruction and send a rendering instruction to the first type of computing unit. The first type of computing unit can determine the target storage resource in the storage component based on the received resource loading instruction, read the target rendering resource from the target storage resource, and then transmit the target rendering resource to the second type of computing unit to realize the loading process of the target rendering resource.

[0046] In some embodiments, if the target rendering instance is pre-defined, the resource identifier of the target rendering resource will be permanently stored in the target rendering instance. Therefore, the second type of computing unit can generate a resource loading instruction based on the resource identifier of the target rendering resource carried by the target rendering instance. Therefore, in the above embodiments, step S200 can be executed after step S100.

[0047] In some embodiments, if the target rendering instance is dynamically generated, the second type of computing unit can generate resource loading instructions based on the resource location information carried in the rendering task. Therefore, in the above embodiments, step S200 can be executed before step S100.

[0048] To conserve power and data transmission overhead in the second type of computing unit, the bandwidth resources of the second type of computing unit are typically limited. Therefore, in some embodiments, rendering resources can be pre-compressed in the high-power domain to transmit rendering resources with lower data transmission latency. Figure 4 This is a flowchart of a data rendering method according to an embodiment of the present invention. Figure 4 As shown, in some embodiments, the target rendering resource can be stored in the following manner: Step S100A: Load each initial rendering resource into the runtime memory.

[0049] In some embodiments, in order to improve the efficiency of reading rendering resources and thus improve the data rendering efficiency of the head-mounted device, the first type of computing unit of the head-mounted device can load each initial rendering resource from other devices (such as servers, terminal devices, etc.) into its own running memory.

[0050] Step S200A: Decode each initial rendering resource into corresponding runtime resource data in the running memory.

[0051] In this step, each initial rendering resource can be decoded into corresponding runtime resource data based on its resource type using different decoding algorithms. For example, a rendering resource in JPEG format can be decoded using decoding algorithms such as Inverse Discrete Cosine Transform and Inverse Huffman Coding; a rendering resource in FBX format can be decompressed into a binary stream using the DEFLATE decompression algorithm, and the geometric parameters within the binary stream can be parsed using the Differential Integer Inverse Compression algorithm.

[0052] Step S300A: In the running memory, each runtime resource data is compressed according to its data characteristics to obtain compressed resource data blocks.

[0053] In this step, in order to reduce the resource transmission latency from the high-power domain to the low-power domain, the first type of computing unit can integrate runtime resource data with similar or consistent resource characteristics, and compress the integrated runtime resource data to obtain compressed resource data blocks.

[0054] Optionally, the first type of computing unit can integrate runtime resource data of the same resource type based on any one of the following: resource type, resource storage format (e.g., binary, vertex array, etc.), resource precision (e.g., 8-bit, 16-bit, etc.), image size, resolution, etc., and then compress the integrated runtime resource data. For example, the first type of computing unit can integrate runtime resource data of texture resources and 3D geometry resources, and then compress the integrated runtime resource data of the two types respectively.

[0055] Optionally, the first type of computing unit can calculate the feature similarity of each runtime resource data based on at least one of the image size, resolution, grayscale value, texture features, and semantic features (i.e., the content information carried by the image or text), and integrate the runtime resource data whose feature similarity satisfies the first similarity condition, and then compress the integrated runtime resource data.

[0056] The feature similarity between runtime resource data can be calculated in various ways, such as calculating the Euclidean distance, cosine value, or Jaccard similarity between runtime resource data. The first similarity condition can be determined according to the calculation method of feature similarity. For example, if the feature similarity is calculated by calculating the cosine value between rendering instructions, then the first similarity condition can be set to the cosine value between runtime resource data being greater than (or greater than or equal to) a preset similarity threshold, and the preset similarity threshold is a predetermined value greater than 0 and less than 1.

[0057] Optionally, depending on the actual settings, the second rendering resources can also be integrated and compressed in other ways based on the resource characteristics of each runtime resource data. This embodiment does not limit this.

[0058] Step S400A: Write the compressed resource data block to the storage component.

[0059] After obtaining the compressed resource data block, the first type of computing unit can write the compressed resource database into the storage component to store the rendering resources in a compressed form.

[0060] Step S300: Generate target rendering data and rendering instructions based on the target rendering instance.

[0061] In this step, the second type of computing unit can generate target rendering data based on the target rendering instance, and then generate corresponding rendering instructions based on the target rendering data. The process of generating target rendering data from the target rendering instance can be implemented in various ways. For example, it can traverse the target rendering instance, extract its attribute information such as mesh geometry, spatial transformation, and material texture, and then transform, package, and group this attribute information to obtain structured buffer data, which can then be used as rendering data. Similarly, the process of generating corresponding rendering instructions based on the target rendering data can also be implemented in various ways. For example, it can generate binding instructions based on the target rendering data, read the rendering state within the target rendering data, generate pipeline state instructions, and generate drawing instructions based on the number of indices (or instances) of the target rendering data.

[0062] In step S400, rendering calculations are performed in a collaborative manner using multiple second-type computing units based on rendering instructions, target rendering resources, and target rendering data to generate a binocular parallax image.

[0063] In some embodiments, the second type of computing unit may include multiple types of computing units, such as a central processing unit and a graphics processing unit. Therefore, to balance the performance and power consumption of the head-mounted device, different rendering instructions can be distributed to different types of computing units, enabling each type of second-type computing unit to perform corresponding rendering operations on the corresponding target rendering resources and target rendering data. To reduce the power consumption of the second-type computing unit, the graphics processing unit may be a two-dimensional graphics processor (2D GPU).

[0064] Step S400A: Distribute rendering instructions according to the task type of the rendering task.

[0065] Taking the central processing unit as the central processor and the graphics processing unit as the graphics processor as an example, in some embodiments, after the central processor generates rendering instructions, it can distribute the rendering instructions to its own computing cores or graphics processors according to the task type of the rendering task, so that the central processor and / or graphics processor can perform rendering calculations according to the received rendering instructions.

[0066] In some optional implementations, rendering tasks can be pre-classified into multiple categories based on their task types, such as geometric stage operation tasks (e.g., coordinate transformation, perspective projection), rasterization operation tasks (i.e., pixel interpolation data generation tasks, such as generating interpolation coefficients for normals, lighting, etc.), fragment shading operation tasks (e.g., texture color sampling, material calculation, etc.), and output merging operation tasks (i.e., testing and color mixing of the rendering results, and writing the rendering operation results into the target frame buffer to form a binocular parallax image), and establishing a mapping relationship with each type of second-class computing unit. Therefore, in this step, the central processing unit can distribute rendering instructions according to the mapping relationship between the task type of the rendering task and the second-class computing unit.

[0067] For example, if rendering task M1 is used to generate a control tree, the central processing unit can determine that the task type of rendering task M1 is a low-complexity task, and determine the second type of computing unit that executes rendering task M1 as the central processing unit according to the above mapping relationship. Therefore, the rendering instructions corresponding to rendering task M1 can be distributed to its own computing core.

[0068] It is easy to understand that step S400A can be performed before step S400.

[0069] In some alternative implementations, in order to improve the communication efficiency between different computing units and thus improve the data rendering efficiency, while reducing the communication overhead between different computing units, the central processing unit can also merge multiple rendering instructions and then distribute the merged rendering instructions to the corresponding computing units. Figure 5 This is a flowchart of a data rendering method according to an embodiment of the present invention. Figure 5 As shown, in an optional implementation of this embodiment, step S400A may include the following steps: Step S410A: Merge the rendering instructions according to their instruction similarity and task type to determine the rendering instruction group.

[0070] In this step, the instruction similarity between rendering instructions can be determined based on at least one of the following: operation type characteristics, operation object characteristics, parameter characteristics, and execution target characteristics. The instruction similarity of rendering instructions that satisfy the second similarity condition and whose task types match are then merged to determine the rendering instruction group. Task type matching can include having the same task type or the task type corresponding to the same type of second-class computing unit, etc.

[0071] Among them, the operation type feature is used to characterize the function type of the instruction, such as vertex drawing instructions, texture binding instructions, parallax calculation instructions, lighting calculation instructions, etc.; the operation object feature is used to characterize the resource characteristics of the target rendering resource, such as resource type, resource identifier, etc.; the parameter feature is used to characterize the variable numerical configuration parameters carried by the rendering instruction, such as global brightness, rendering distance, etc.; the execution target feature is used to characterize the carrier that carries the rendering calculation results, such as the first display component frame buffer, the second display component frame buffer, etc.

[0072] The calculation method for the similarity between rendering instructions can refer to the calculation method for the feature similarity of runtime resource data, and the setting method for the second similarity condition can refer to the setting method for the first similarity condition, which will not be elaborated here.

[0073] Step S420A: Distribute rendering instruction groups according to task type.

[0074] In this step, the central processing unit (CPU) can distribute rendering instruction sets to itself or to the graphics processing unit (GPU) based on the task type. Therefore, in step S400, both the CPU and the GPU can read the target rendering resource from a cache component (such as CPU memory) according to the resource identifier carried in the rendering instruction, and perform rendering calculations based on the target rendering resource, target rendering data, and rendering instructions, thereby generating a binocular parallax image in a collaborative manner.

[0075] In practical applications, graphics processing units (GPUs) integrate data rendering engines, so they can perform the above-mentioned rendering calculation process through these engines.

[0076] In some embodiments, if the target rendering resource has been compressed, the target rendering data can be decompressed by a decompression component to obtain the decompressed target rendering data, thereby performing rendering calculations in a collaborative manner with multiple second-type computing units.

[0077] In some embodiments, the data rendering engine can perform rendering computations through data rendering customization to reduce its memory footprint and thus lower the power consumption of the head-mounted device. Data rendering customization involves adjusting the rendering pipeline, shading logic, rendering resource scheduling, and rendering strategies according to business needs, thereby improving the adaptability of the rendering computation process to the head-mounted device.

[0078] For example, the rendering strategy of the general engine is to perform two complete rendering calculations for the left eye image and the right eye image respectively. However, in the data rendering customization mode, the data rendering engine can output the field of view of the left and right eyes simultaneously by sharing a set of vertex operations, thus effectively reducing the rendering overhead of the second type of computing unit.

[0079] Optionally, if the head-mounted device is AI glasses, an AI model can be run through the second type of computing unit to customize data rendering and optimize the rendering computation process. For example, if the AI ​​glasses include an eye-tracking component, the eye-tracking component can identify the wearer's gaze point. The AI ​​model can determine the central and edge regions based on the wearer's gaze point and render the central and edge regions with different rendering accuracies. Furthermore, the central region can be rendered with high accuracy, while the edge regions can be rendered with low accuracy, thereby saving computing power on the AI ​​glasses and improving their battery life.

[0080] Optionally, when the output component is a set of display components, the number of binocular parallax images is 1; when the output component is two independent sets of display components, the number of binocular parallax images is 2.

[0081] Step S500: Output binocular parallax image.

[0082] In this step, the second type of computing unit can transmit binocular parallax images to the output component, enabling the output component to output binocular parallax images to the wearer. Therefore, this embodiment effectively reduces the computing power overhead of the head-mounted device by dividing it into a high-power domain and a low-power domain, and performs the data rendering process in the low-power domain, thus improving the user experience.

[0083] Figure 6 This is a schematic diagram of the data rendering process according to an embodiment of the present invention, using the second type of computing unit including CPU and GPU as an example for illustration. Figure 6As shown, in step S61, the CPU in the low-power domain can load the target rendering resources corresponding to the rendering task from the storage component 10 in the high-power domain to the low-power domain by interacting with the first type of computing unit in the high-power domain. In step S62, the decompression component in the low-power domain can decompress the target rendering resources. In step S63, the CPU in the low-power domain can generate target rendering data and rendering instructions according to the target rendering instance corresponding to the rendering task. In step S64, the CPU in the low-power domain can distribute rendering instructions to itself or the GPU. In step S65, the CPU and GPU can collaboratively perform rendering calculations based on the target rendering resources, target rendering data, and rendering instructions to generate a binocular parallax image, namely a first image and a second image, and input the first image into the first display component and the second image into the second display component. In step S66, the binocular parallax image is output.

[0084] The head-mounted device of this invention includes a storage component, a system-on-a-chip (SoC), and an output component. The storage component stores rendering resources. The SoC includes a first type of computing unit with a higher power consumption limit and a second type of computing unit with a lower power consumption limit. The first type of computing unit reads the corresponding rendering resources from the storage component according to the rendering task and transmits the rendering resources to the second type of computing unit. The second type of computing unit determines the rendering instance corresponding to the rendering task, generates rendering data and rendering instructions corresponding to the rendering instance, and performs rendering operations in a collaborative manner based on the rendering instructions, the rendering resources, and the rendering data to generate a binocular parallax image. The output component outputs the image. This invention implements rendering resource access in the high-power domain and data rendering computation in the low-power domain, thereby effectively improving the battery life of the head-mounted device.

[0085] Those skilled in the art will understand that embodiments of this application can be provided as methods, apparatus (devices), or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-readable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0086] These computer program instructions may be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including an instruction means, the implementation process of which is described in the instruction means. Figure 1 The function specified in one or more processes.

[0087] These computer program instructions may also be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing device, produce instructions for implementing processes. Figure 1 A device for a function specified in one or more processes.

[0088] Another embodiment of the present invention relates to a non-volatile storage medium for storing a computer-readable program, which is used by a computer to execute some or all of the above-described method embodiments.

[0089] That is, those skilled in the art will understand that all or part of the steps in the methods of the above embodiments can be implemented by specifying the relevant hardware through a program. This program is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods described in the embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk. In one application scenario, the non-volatile storage medium storing the above-mentioned computer program product can be part of the system-on-a-chip of a head-mounted device.

[0090] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A head-mounted device, characterized in that, The device includes: The storage component is configured to store rendering resources; The system-on-a-chip includes a first type of computing unit and a second type of computing unit, wherein the power consumption limit of the first type of computing unit is greater than the power consumption limit of the second type of computing unit; and The output component is configured to output a binocular parallax image; The first type of computing unit is configured to read the target rendering resource from the storage component according to the resource loading instruction corresponding to the rendering task, and to transmit the target rendering resource to the second type of computing unit; The second type of computing unit is configured to determine the target rendering instance corresponding to the rendering task, generate the resource loading instruction, generate target rendering data and rendering instructions based on the target rendering instance, and perform rendering operations in a collaborative manner based on the rendering instructions, the target rendering resources and the target rendering data to generate the binocular parallax image.

2. The device according to claim 1, characterized in that, The head-mounted device also includes RAM; The first type of computing unit is further configured as follows: Each initial rendering resource is loaded into the runtime memory, and each initial rendering resource is decoded into corresponding runtime resource data in the runtime memory. Each runtime resource data is compressed in the runtime memory according to its data characteristics to obtain a compressed resource data block, and the compressed resource data block is written into the storage component to store the rendering resources in a compressed form.

3. The device according to claim 1, characterized in that, The second type of computing unit includes a central processing unit and a graphics processing unit; The central processing unit is also configured to: The rendering instructions are distributed according to the task type of the rendering task, so that the central processing unit or the graphics processing unit can perform rendering calculations based on the rendering instructions, the target rendering resources and the target rendering data to generate the binocular parallax image.

4. The device according to claim 3, characterized in that, The central processing unit is further configured as follows: The rendering instructions are merged based on their instruction similarity and task type to determine a rendering instruction group, and the rendering instruction group is distributed according to the task type.

5. The device according to claim 3, characterized in that, The graphics processing unit is a two-dimensional graphics processing unit.

6. The device according to claim 3, characterized in that, The head-mounted device also includes a decompression component; The decompression component is configured as follows: The target rendering resource is decompressed to obtain the decompressed target rendering resource, so that the second type of computing unit can generate the binocular parallax image based on the decompressed target rendering resource.

7. The device according to claim 1, characterized in that, The binocular parallax image includes a first image and a second image, and the output component includes a first display component and a second display component; The first display component is configured to output the first image; The second display component is configured to output the second image.

8. A data rendering method applicable to a head-mounted device, the head-mounted device comprising a first type of computing unit and a second type of computing unit, wherein the power consumption limit of the first type of computing unit is greater than the power consumption limit of the second type of computing unit, characterized in that, The method includes: Determine the target rendering instance corresponding to the rendering task; Load the target rendering resources according to the resource loading instructions corresponding to the rendering task; Generate target rendering data and rendering instructions based on the target rendering instance; Rendering calculations are performed using a multi-second type of computing unit in a collaborative manner based on the rendering instructions, the target rendering resources, and the target rendering data to generate a binocular parallax image; Output the binocular parallax image.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method as described in claim 8.

10. A computer program product, characterized in that, The computer program product includes a computer program / instruction that, when executed by a processor, implements the method as described in claim 8.

Images