Method for reducing calculation load during content generation and host

Abstract

The embodiment of the invention provides a method for reducing calculation load during content generation and a host. The method comprises the following steps: obtaining a texture map and a plurality of object texture coordinates, wherein the object texture coordinates correspond to at least one virtual object in the texture map; obtaining a first grid graph corresponding to the texture graph; determining a first mesh segment and a second mesh segment within the first mesh map based on the object texture coordinates, where the first mesh segment does not correspond to the object texture coordinates and the second mesh segment corresponds to the object texture coordinates; modifying the first grid chart into a second grid chart by reducing a first section area of the first grid section; and generating visual content by performing an asynchronous time warping operation on the texture map based on the second grid map.

Description

Technical Field

[0001] The present invention generally relates to a mechanism for generating content, and more particularly, to a method and a host for reducing a computational load when generating content. Background Art

[0002] In augmented reality (AR) technology, digital objects or virtual objects (ie, AR objects) are superimposed on the real world by using cameras and sensors, and it is expected that these objects will look like they are integrated into the physical environment.

[0003] However, since users wearing AR devices (e.g., AR glasses) can perform actions such as turning their heads, virtual objects in AR content cannot be well superimposed on the real world due to a problem called "pose delay."

[0004] “Pose latency” in the field of augmented reality (AR) generally refers to the lag or delay in tracking and updating the position and orientation (ie, pose) of a virtual object in an AR environment.

[0005] In the prior art, a graphics processing unit (GPU) in an AR device may implement an asynchronous time warp (ATW) operation to alleviate the problem of gesture delay.

[0006] See also Figure 1 , Figure 1 The mechanism of generating visual content is shown. Figure 1 In the embodiment of the present invention, the GPU may perform an ATW operation to generate visual content 13 (e.g., AR content) based on a texture map 11 (which involves AR objects 111 and 112) from an AR application and an associated mesh map 12. Since the computational load of the GPU is highly dependent on the size / content of the mesh map 12 associated with the texture map 11, if the size / content of the mesh map 12 is appropriately reduced, the computational load of the GPU during the ATW operation may be reduced. Summary of the invention

[0007] Therefore, the present invention relates to a method and a host for reducing calculation load when generating content, which can be used to solve the above technical problems.

[0008] An embodiment of the present invention provides a method for reducing computing load when generating content, which is applied to a host. The method includes: obtaining a texture map and a plurality of object texture coordinates by the host, wherein the plurality of object texture coordinates correspond to at least one virtual object in the texture map; obtaining a first mesh map corresponding to the texture map by the host; determining a first mesh segment and a second mesh segment within the first mesh map based on the plurality of object texture coordinates by the host, wherein the first mesh segment does not correspond to the plurality of object texture coordinates, and the second mesh segment corresponds to the plurality of object texture coordinates; modifying the first mesh map into a second mesh map by reducing the first segment area of ​​the first mesh segment by the host; and generating visual content by performing an asynchronous time warp operation on the texture map based on the second mesh map by the host.

[0009] An embodiment of the present invention provides a host including a storage circuit, a processor and a graphics processing unit. The storage circuit stores program code. The processor is coupled to the non-temporary storage circuit and accesses the program code to perform the following operations: obtain a texture map and a plurality of object texture coordinates, wherein the plurality of object texture coordinates correspond to at least one virtual object in the texture map; obtain a first mesh map corresponding to the texture map; determine a first mesh segment and a second mesh segment within the first mesh map based on the plurality of object texture coordinates, wherein the first mesh segment does not correspond to the plurality of object texture coordinates, and the second mesh segment corresponds to the plurality of object texture coordinates; modify the first mesh map to the second mesh map by reducing the first segment area of ​​the first mesh segment. The graphics processing unit is coupled to the processor and generates visual content by performing an asynchronous time warping operation on the texture map based on the second mesh map. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The accompanying drawings are included to provide a further understanding of the invention, and are incorporated in and constitute a part of this specification. The accompanying drawings illustrate embodiments of the invention and together with the description serve to explain the principles of the invention.

[0011] Figure 1 Shows the mechanism for generating visual content.

[0012] Figure 2 A schematic diagram showing a host according to an embodiment of the present invention.

[0013] Figure 3 A flow chart showing a method for reducing calculation load when generating content according to an embodiment of the present invention.

[0014] Figure 4 An application scenario according to an embodiment of the present invention is shown.

[0015] Figure 5 A schematic diagram showing how to modify a first grid map into a second grid map according to an embodiment of the present invention is shown.

[0016] [Explanation of Symbols]

[0017] 11, 40: Texture map

[0018] 12: Grid map

[0019] 13.43: Visual content

[0020] 41, 51: First grid map

[0021] 42, 52: Second grid map

[0022] 111, 112: AR objects

[0023] 200: Host

[0024] 202: Storage Circuit

[0025] 204: Processor

[0026] 206: GPU

[0027] 401, 402: Virtual object

[0028] 411, 511: First grid section

[0029] 412a, 412b, 512: second grid section

[0030] F1: First mesh face

[0031] F2: Second mesh face

[0032] F11, F12, F13, F14: First vertex

[0033] F21, F22: Second vertex

[0034] F23, F24: The third vertex

[0035] S310, S320, S330, S340, S350: Steps DETAILED DESCRIPTION

[0036] Reference will now be made in detail to the presently preferred embodiments of the present invention, examples of which are illustrated in the accompanying drawings. Wherever possible, the same reference numerals are used in the drawings and the description to refer to the same or like parts.

[0037] See also Figure 2 , Figure 2A schematic diagram of a host according to an embodiment of the present invention is shown. In various embodiments, the host 200 may be any smart device and / or computer device that can provide visual content of a reality service such as a virtual reality (VR) service, an augmented reality (AR) service, a mixed reality (MR) service, and / or an extended reality (XR) service, but the present invention is not limited thereto. In some embodiments, the host 200 may be a head-mounted display (HMD) that can show / provide visual content (e.g., AR / VR content) to a wearer / user. In order to better understand the concept of the present invention, it will be assumed that the host 200 is an AR device (e.g., a pair of AR glasses) for providing AR content (which may include virtual objects to be superimposed on the real world) to the user, but the present invention is not limited thereto. In some embodiments, the host 200 may be any device / system (e.g., an architecture between an application and real hardware in a machine) that can implement the method proposed in the present invention at runtime.

[0038] exist Figure 2 In the embodiment, the host 200 includes a storage circuit 202, a processor 204, and a GPU 206. The storage circuit 202 is a fixed or removable random access memory (RAM), a read-only memory (ROM), a flash memory, a hard disk, or any other similar device, or a combination thereof, and the storage circuit 202 records a plurality of modules and / or program codes that can be executed by the processor 204.

[0039] The processor 204 may be coupled to the memory circuit 202 and the GPU 206, and the processor 204 may be, for example, a general purpose processor, a special purpose processor, a conventional processor, a digital signal processor (DSP), a plurality of microprocessors, one or more microprocessors associated with a DSP core, a controller, a microcontroller, an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) circuit, any other type of integrated circuit (IC), a state machine, and the like.

[0040] In an embodiment of the present invention, the processor 204 may access the modules and / or program codes stored in the storage circuit 202 to implement the method for reducing the computational load when generating content provided in the present invention, which will be further discussed below.

[0041] See also Figure 3 , Figure 3 A flowchart of a method for reducing computing load when generating content according to an embodiment of the present invention is shown. The method of this embodiment can be Figure 2 The host 200 in the execution, and the following will be combined Figure 2 Components shown Figure 3 In order to better understand the concept of the present invention, the following will be used. Figure 4 As an example, Figure 4 An application scenario according to an embodiment of the present invention is shown.

[0042] In step S310 , the processor 204 obtains the texture map 40 and a plurality of object texture coordinates, wherein the plurality of object texture coordinates correspond to the virtual objects 401 and 402 in the texture map 40 .

[0043] In an embodiment of the present invention, the texture coordinates in the texture map 40 may be represented by corresponding UV coordinates. For example, the upper left corner of the texture map 40 may be defined as having UV coordinates (0, 0), the upper right corner of the texture map 40 may be defined as having UV coordinates (0, 1), the lower left corner of the texture map 40 may be defined as having UV coordinates (1, 0), and the lower right corner of the texture map 40 may be defined as having UV coordinates (1, 1), but the present invention is not limited thereto.

[0044] exist Figure 4 In the embodiment, the texture coordinates corresponding to the virtual objects 401 and 402 may be referred to as object texture coordinates, and the texture coordinates not corresponding to any virtual object may be referred to as non-object texture coordinates, but the present invention is not limited thereto.

[0045] In one embodiment, the processor 204 may receive the texture map 40 and the plurality of object texture coordinates from a first application (e.g., an AR application) that provides a reality service (e.g., an AR service). That is, the first application not only provides the texture map 40, but also provides the object texture coordinates corresponding to the virtual objects 401 and 402.

[0046] In step S320, the processor 204 obtains the first mesh map 41 corresponding to the texture map 40. In an embodiment of the present invention, the first mesh map 41 may be a conventional mesh map used for ATW operation.

[0047] exist Figure 4In the embodiment, the first mesh map 41 may include a plurality of mesh faces, which are shown as rectangular / square grids within the first mesh map 41. In other embodiments, the mesh faces may also be represented as triangles within the first mesh map, but the present invention is not limited thereto.

[0048] exist Figure 4 In the example, each mesh face includes four vertices, and adjacent mesh faces may have shared vertices. That is, two or more mesh faces may have the same vertices.

[0049] In step S330 , the processor 204 determines a first mesh segment and a second mesh segment in the first mesh map 41 based on the object texture coordinates, wherein the first mesh segment does not correspond to the object texture coordinates, and the second mesh segment corresponds to the object texture coordinates.

[0050] exist Figure 4 In , the processor 204 may determine a first mesh segment 411 that does not correspond to the plurality of object texture coordinates (ie, does not correspond to the virtual objects 401 and 402). Figure 4 In the illustrated scenario, the first mesh section 411 may include mesh surfaces illustrated using dashed lines.

[0051] Additionally, the processor 204 may determine second mesh segments 412 a and 412 b corresponding to the plurality of object texture coordinates (ie, corresponding to the virtual objects 401 and 402 ).

[0052] In step S340 , the processor 204 modifies the first grid map 41 into the second grid map 42 by reducing the first segment area of ​​the first grid segment 411 .

[0053] exist Figure 4 In the example, the processor 204 may adjust the first segment area of ​​the first mesh segment 411 to, for example, zero by modifying the positions of the vertices of the mesh faces within the first mesh segment 411. For example, the processor 204 may determine which vertices within the first mesh segment 411 are not shared by the second mesh segments 412a and 412b, and adjust the positions of these vertices to be located on the same line or point, so that the area of ​​each mesh face within the first mesh segment 411 is adjusted to zero. In this case, the first segment area of ​​the first mesh segment 411 will become zero accordingly, as shown in FIG. Figure 4 The second grid is shown in FIG42 .

[0054] In an embodiment of the present invention, the processor 204 may reduce the first segment area of ​​the first mesh segment 411 while maintaining the second segment area of ​​each of the second mesh segments 412a and 412b.

[0055] In step S350 , the GPU 206 generates the visual content 43 by performing an asynchronous time warping operation on the texture map 40 based on the second mesh map 42 .

[0056] exist Figure 4 In the example, since the GPU 206 only needs to determine the pixels corresponding to the second grid segments 412a and 412b within the second grid map 42 during the ATW operation, the computational load of the GPU 206 can be reduced, which improves the efficiency of the ATW operation. Therefore, the virtual objects 401 and 402 can be better superimposed on the real world in real time, and thus the user experience can be improved.

[0057] It should be noted that although Figure 4 The first segment area of ​​the first grid segment 411 is adjusted to zero, but as long as the first segment area of ​​the first grid segment 411 is reduced, the computational load of the GPU 206 can be reduced. In other words, in order to reduce the computational load of the GPU 206, the first segment area of ​​the first grid segment 411 does not necessarily need to be reduced to zero.

[0058] See also Figure 5 , Figure 5 A schematic diagram showing how to modify a first grid map into a second grid map according to an embodiment of the present invention.

[0059] exist Figure 5 , it is assumed that the processor 204 determines a first mesh segment 511 (not corresponding to any virtual object in the associated texture map) and a second mesh segment 512 (corresponding to at least one virtual object in the associated texture map) within the first mesh map 51 based on the object texture coordinates provided by the first application.

[0060] In this embodiment, the first mesh section 511 may include a plurality of first mesh surfaces that do not surround the second mesh section 512 and a plurality of second mesh surfaces that surround the second mesh section 512. Figure 5 In FIG. 51 , the first mesh surface is marked with a triangle, and the second mesh surface is marked with a star. However, it should be noted that the triangle and the star are only for visual aid and are not part of the first mesh diagram 51 .

[0061] In an embodiment of the present invention, the processor 204 may reduce the first segment area of ​​the first mesh segment 511 by reducing the first area of ​​at least a portion of the plurality of first mesh surfaces. In an embodiment, the processor 204 may reduce the first distance between each of the plurality of first vertices of each of the first mesh surfaces.

[0062] For example, for a first mesh surface F1 having first vertices F11 to F14, the processor 204 may reduce the first distance between the first vertices F11 to F14 by adjusting the first vertices F11 to F14 to be closer to each other, so that the first area of ​​the first mesh surface F1 may be reduced. In one embodiment, the processor 204 may adjust the first vertices F11 to F14 to be located on the same line or point, so that the first area of ​​the first mesh surface F1 may be reduced to zero, but the present invention is not limited thereto.

[0063] For other first mesh surfaces (ie, other mesh surfaces marked with triangles), the processor 204 may reduce the corresponding first areas based on the above teachings.

[0064] Additionally or alternatively, the processor 204 may reduce the first segment area of ​​the first mesh segment 511 by reducing the second area of ​​at least a portion of the plurality of second mesh surfaces.

[0065] In one embodiment, the second mesh section 512 includes a plurality of surrounding vertices (marked with solid circles to provide a visual aid). In one embodiment, each of the second mesh faces (marked with stars) includes at least one second vertex that does not correspond to any of the plurality of surrounding vertices and at least one third vertex that corresponds to at least one of the plurality of surrounding vertices.

[0066] exist Figure 5 In order to provide visual aid, the second vertex of each second mesh face is marked with a hollow circle, and the third vertex of each second mesh face can be understood as a surrounding vertex (marked with a solid circle).

[0067] In one embodiment, the processor 204 may reduce the second area of ​​at least a portion of the plurality of second mesh surfaces based on the same principle as reducing the first area of ​​the first mesh surface (e.g., shortening the distance between the second vertex and the third vertex), and details may refer to the above embodiments.

[0068] In one embodiment, the processor 204 may reduce the second area of ​​at least a portion of the plurality of second mesh surfaces by merging the second vertex of each second mesh surface into the third vertex of each second mesh surface.

[0069] For example, for the second mesh surface F2 having the second vertices F21 and F22 and the third vertices F23 and F24, the processor 204 may merge the second vertex F21 into the third vertex F23 and merge the second vertex F22 into the third vertex F24. In other words, the processor 204 may adjust the position of the second vertex F21 to be the same as the third vertex F23 and adjust the position of the second vertex F22 to be the same as the third vertex F24. In this case, the second area of ​​the second mesh surface F2 may be reduced to zero, but the present invention is not limited thereto.

[0070] exist Figure 5 In the example, the processor 204 may modify the first mesh map 51 to the second mesh map 52 by reducing the first segment area of ​​the first mesh segment 511 to, for example, zero, but the present invention is not limited thereto. In addition, the processor 204 may reduce the first segment area of ​​the first mesh segment 511 while maintaining the second segment area of ​​the second mesh segment 512 by maintaining the surrounding vertices (marked with solid circles). That is, during the modification of the first mesh map 51 to the second mesh map 52, the positions of the surrounding vertices of the second mesh segment do not change.

[0071] exist Figure 5 In the embodiment, since the GPU 206 only needs to determine the pixels corresponding to the second grid segment 512 within the second grid map 52 during the ATW operation, the computational load of the GPU 206 can be reduced, which improves the efficiency of the ATW operation.

[0072] In summary, the embodiments of the present invention provide a solution for reducing the content / size / information of a grid map by reducing the area of ​​a grid segment that does not correspond to a virtual object. Since the GPU can perform an ATW operation based on a grid map with reduced content / size / information, the efficiency of the ATW operation can be improved, and thus the user experience can be improved.

[0073] It will be apparent to those skilled in the art that various modifications and changes may be made to the structure of the present invention without departing from the scope or spirit of the present invention. In summary, the present invention is intended to cover various modifications and changes of the present invention as long as they fall within the scope of the appended claims and their equivalents.

Claims

1. A method for reducing computing load when generating content, the method being applied to a host, characterized in that: The method comprises: obtaining, by the host, a texture map and a plurality of object texture coordinates, wherein the plurality of object texture coordinates correspond to at least one virtual object in the texture map; The host obtains a first mesh image corresponding to the texture image; determining, by the host computer, a first mesh segment and a second mesh segment within the first mesh map based on the plurality of object texture coordinates, wherein the first mesh segment does not correspond to the plurality of object texture coordinates, and the second mesh segment corresponds to the plurality of object texture coordinates; modifying, by the host, the first grid map into a second grid map by reducing a first segment area of ​​the first grid segment; and Visual content is generated by the host by performing an asynchronous time warping operation on the texture map based on the second mesh map.

2. The method according to claim 1, wherein the step of obtaining the texture map and the plurality of object texture coordinates comprises: The texture map and the plurality of object texture coordinates are received from a first application providing a reality service.

3. The method of claim 2, wherein the reality service comprises an augmented reality service and the visual content comprises augmented reality content.

4. The method according to claim 1, wherein the first mesh segment includes a plurality of first mesh surfaces that do not surround the second mesh segment and a plurality of second mesh surfaces that surround the second mesh segment, and the step of modifying the first mesh map to the second mesh map by reducing the first segment area of ​​the first mesh segment comprises: A first area of ​​at least a portion of the plurality of first mesh surfaces is reduced.

5. The method according to claim 4, wherein the step of modifying the first grid map into the second grid map by reducing the first segment area of ​​the first grid segment further comprises: A second area of ​​at least a portion of the plurality of second mesh surfaces is reduced.

6. The method according to claim 4, wherein each of the plurality of first mesh surfaces comprises a plurality of first vertices, and the step of reducing the first area of ​​each of the plurality of first mesh surfaces comprises: A first distance between each of the plurality of first vertices of each of the plurality of first mesh faces is reduced.

7. The method according to claim 4, wherein each of the plurality of first mesh surfaces comprises a plurality of first vertices, and the step of reducing the first area of ​​each of the plurality of first mesh surfaces comprises: The plurality of first vertices of each of the plurality of first mesh faces are adjusted to be located on a line or a point.

8. The method of claim 5, wherein the second mesh section comprises a plurality of surrounding vertices, each of the plurality of second mesh faces comprises at least one second vertex that does not correspond to any of the plurality of surrounding vertices and at least one third vertex that corresponds to at least one of the plurality of surrounding vertices, and the step of reducing the second area of ​​the at least a portion of the plurality of second mesh faces comprises: The at least one second vertex is merged into the at least one third vertex.

9. The method according to claim 1, wherein the step of modifying the first grid map into the second grid map by reducing the first segment area of ​​the first grid segment comprises: The first grid pattern is modified to the second grid pattern by reducing the first segment area of ​​the first grid segment while maintaining the second segment area of ​​the second grid segment.

10. The method of claim 9, wherein the second mesh segment comprises a plurality of surrounding vertices, and the step of maintaining the second segment area of ​​the second mesh segment comprises: The plurality of surrounding vertices is maintained.

11. A host, characterized in that: include: A storage circuit for storing program codes; A processor is coupled to the memory circuit and accesses the program code to perform the following operations: Obtaining a texture map and a plurality of object texture coordinates, wherein the plurality of object texture coordinates correspond to at least one virtual object in the texture map; Obtaining a first mesh image corresponding to the texture image; determining a first mesh segment and a second mesh segment within the first mesh map based on the plurality of object texture coordinates, wherein the first mesh segment does not correspond to the plurality of object texture coordinates and the second mesh segment corresponds to the plurality of object texture coordinates; modifying the first grid pattern into a second grid pattern by reducing a first segment area of ​​the first grid segment; and A graphics processing unit is coupled to the processor and generates visual content by performing an asynchronous time warping operation on the texture map based on the second mesh map.

Images