Method, device and equipment for rendering three-dimensional content in multi-task system

Abstract

The invention relates to the technical field of computer systems, computer graphics and human-computer interaction, in particular to a method, device and equipment for rendering three-dimensional content in a multi-task system. Comprising the steps that an application is responsible for processing application logic, generating a rendering instruction of a rendering object and then sending the rendering instruction to a system, and the system renders the rendering object according to the rendering instruction. The quality of the display picture of the XR system and the user experience are improved, and the power consumption of the XR equipment is reduced.

Description

[0001] This application is a divisional application of the invention patent application with application number 202310659805.6, application date June 5, 2023, and invention name “Method, device and apparatus for rendering three-dimensional content in a multi-tasking system”. Technical Field

[0002] The present invention relates to the fields of computer systems, computer graphics and human-computer interaction technology, and in particular to a method, device and equipment for rendering three-dimensional content in a multi-tasking system. Background Art

[0003] In multi-task systems such as XR, multiple tasks (i.e., multiple processes) can be run simultaneously. Each process completes the logical processing and rendering of the object, and sends the image obtained by each frame of rendering to the system. The synthesizer in the system synthesizes the images sent by multiple processes to form the final picture displayed on the screen. In a two-dimensional multi-task system, this method can well distribute the rendering pressure to each process. The system can optimize the rendering through the plane occlusion relationship. For example, the part occluded by other processes is not rendered, and only the visible area is rendered and transmitted, thereby saving rendering resources and reducing the transmission bandwidth pressure. However, when the process and the system are both three-dimensional scenes, such as the three-dimensional multi-task effect in XR glasses, the content of each process is a three-dimensional scene, and the scene finally synthesized by the system is also three-dimensional. In the XR three-dimensional environment, the user's picture resolution requirements are high. Each process needs to render the rendering object according to the user's resolution requirements, generate a high-resolution image, and send the high-resolution image to the system, which causes huge transmission bandwidth pressure, increases transmission delay, and wastes some rendering resources. High transmission bandwidth pressure and transmission delay reduce the display quality and user experience of the XR system, and increase the power consumption of the XR device.

[0004] There is an urgent need for a method for rendering three-dimensional content in a multi-task system, so as to solve the problems of high transmission bandwidth pressure between applications and systems, high transmission delay, and tight rendering resources in the existing three-dimensional multi-task rendering technology. Summary of the invention

[0005] To solve the problems existing in the prior art, the embodiments of this article provide a method, device and equipment for rendering three-dimensional content in a multi-tasking system. The application is responsible for generating rendering instructions for rendering objects according to the application logic, and then sending the rendering instructions to the system, and the system renders the rendering objects according to the rendering instructions. This method can reduce the data transmission pressure between the application and the system, reduce transmission delays, and improve the quality of the binocular image and user experience of the XR system.

[0006] In order to solve the above technical problems, the specific technical solutions of this article are as follows:

[0007] On the one hand, an embodiment of the present invention provides a method for rendering three-dimensional content in a multi-task system, which is applied to a multi-task system of a device supporting display of three-dimensional content and is executed by an application in the multi-task system of a device supporting display of three-dimensional content. The method includes:

[0008] Generate rendering instructions for the rendering object according to the application logic, wherein the rendering instructions are used to guide the generation of a rendering image of the rendering object;

[0009] The rendering instruction is sent to the system, so that the system renders the rendering objects of multiple applications according to the received rendering instructions sent by the multiple applications, and obtains a rendering screen containing the rendering objects of the multiple applications.

[0010] Furthermore, generating a rendering instruction for a rendering object according to the application logic further includes:

[0011] Parsing the triggered rendering event to obtain metadata of the rendering object, wherein the metadata of the rendering object includes description information and attribute information of the rendering object;

[0012] The metadata of the rendering object is converted into a call applicable to the system to generate a rendering instruction for the rendering object.

[0013] Further, the rendering event includes: operation information of the operating user;

[0014] The system renders rendering objects of multiple applications according to the received rendering instructions sent by the multiple applications, specifically including:

[0015] The system renders rendering objects of multiple applications according to the received rendering instructions sent by the multiple applications and the observation angle information of the operating user.

[0016] Furthermore, the metadata of the rendering object also includes incremental information, and the rendering event includes an object initialization event and an object adjustment event;

[0017] The method further comprises:

[0018] Performing the application logic processing on the object initialization event to generate an initial rendering instruction for the rendering object, and sending the initial rendering instruction to the system, so that the system renders the rendering objects of multiple applications according to the received initial rendering instructions sent by the multiple applications;

[0019] The object adjustment event is subjected to the application logic processing to generate an incremental rendering instruction for the rendering object, and the incremental rendering instruction is sent to the system so that the system renders the rendering objects of multiple applications according to the incremental rendering instruction.

[0020] Furthermore, the metadata of the rendering object also includes a motion model of the rendering object;

[0021] The method further comprises:

[0022] The motion model of the rendering object is subjected to the application logic processing to generate a model rendering instruction of the rendering object, and the model rendering instruction is sent to the system so that the system renders the rendering objects of multiple applications according to the model rendering instruction.

[0023] On the other hand, an embodiment of the present invention further provides a method for rendering three-dimensional content in a multi-task system, characterized in that the method is applied to a multi-task system of a device supporting display of three-dimensional content and is executed by a system in the multi-task system of a device supporting display of three-dimensional content, and the method comprises:

[0024] Receiving rendering instructions of rendering objects sent by multiple applications, wherein the rendering instructions are generated by the applications according to application logic, and the rendering instructions are used to guide the generation of rendering images of the rendering objects;

[0025] Acquiring viewing angle information of an operating user according to a device that supports displaying three-dimensional content;

[0026] The rendering objects of the multiple applications are rendered according to the rendering instruction and the viewing angle information of the operating user to obtain a rendering screen including the rendering objects of the multiple applications.

[0027] Furthermore, the rendering instruction is generated by the application according to the application logic, and further includes:

[0028] The application parses the triggered rendering event and obtains metadata of the rendering object, wherein the rendering event includes operation information of the operating user, and the metadata includes description information and attribute information of the rendering object;

[0029] The metadata of the rendering object is converted into a call applicable to the system to generate a rendering instruction for the rendering object.

[0030] Further, the metadata of the rendering object also includes incremental information, the rendering event includes an object initialization event and an object adjustment event, the initial rendering instruction is generated by the application performing the application logic processing on the object initialization event, and the incremental rendering instruction is generated by the application performing the application logic processing on the object adjustment event;

[0031] The method further comprises:

[0032] Rendering the rendering objects of the multiple applications according to the received initial rendering instructions sent by the applications and the viewing angle information of the operating user;

[0033] The rendering objects of the multiple applications are rendered according to the received incremental rendering instructions sent by the applications and the viewing angle information of the operating user.

[0034] Furthermore, the metadata of the rendering object further includes a motion model of the rendering object, and the model rendering instruction is generated by the application performing the application logic processing on the motion model of the rendering object;

[0035] The method further comprises:

[0036] The rendering objects of multiple applications are rendered according to the received model rendering instructions sent by the applications and the viewing angle information of the operating user.

[0037] Furthermore, rendering the rendering objects of multiple applications according to the rendering instruction and the viewing angle information of the operating user further includes:

[0038] Obtaining a canvas range of the device supporting display of three-dimensional content;

[0039] Rendering objects of multiple applications are rendered within the canvas range according to the rendering instruction and the observation angle information.

[0040] Furthermore, rendering the rendering objects of multiple applications according to the rendering instruction and the viewing angle information of the operating user further includes:

[0041] Calculating the occlusion relationship between the metadata according to the viewing angle information of the operating user and all rendering instructions;

[0042] The metadata without any occlusion is rendered to obtain the rendered image.

[0043] On the other hand, an embodiment of the present invention further provides a device for rendering three-dimensional content in a multi-tasking system, characterized in that the device comprises:

[0044] A rendering instruction generating unit, used for generating a rendering instruction of the rendering object according to an application logic, wherein the rendering instruction is used for guiding the generation of a rendering image of the rendering object;

[0045] The rendering instruction sending unit is used to send the rendering instruction to the system, so that the system renders the rendering objects of multiple applications according to the received rendering instructions sent by the multiple applications to obtain a rendering picture including the rendering objects.

[0046] On the other hand, an embodiment of the present invention further provides a device for rendering three-dimensional content in a multi-tasking system, characterized in that the device comprises:

[0047] A rendering instruction receiving unit, used to receive rendering instructions of rendering objects sent by multiple applications, wherein the rendering instructions are generated by the applications according to application logic, and the rendering instructions are used to guide the generation of rendering images of the rendering objects;

[0048] An observation angle information acquisition unit, used to acquire the observation angle information of the operating user according to a device supporting the display of three-dimensional content;

[0049] The rendering unit is used to render the rendering objects of multiple applications according to the rendering instruction and the observation angle information of the operating user to obtain a rendering screen including the rendering objects.

[0050] On the other hand, the embodiments of the present invention further provide a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned method when executing the computer program.

[0051] Finally, an embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and the computer program implements the above-mentioned method when executed by a processor.

[0052] Using the embodiments of the present invention, the application of the multi-tasking system that supports the display of three-dimensional content devices generates rendering instructions for rendering objects according to the application logic, and then sends the rendering instructions to the system. Compared with the traditional method in which an application renders a rendering object and sends the rendered image to the system, the size of the rendering instructions sent by the application of the present invention is much smaller than the rendered image, so it can effectively reduce the data transmission pressure between the application and the system and reduce the transmission delay. Finally, the system renders the rendering objects of multiple applications based on the rendering instructions sent by the multiple applications and the user's observation angle information to obtain a rendered screen. The present invention still retains the execution of the application logic in the application, and the system performs rendering. Even if the processing logic of a certain application crashes, it will not affect the operation of the system. The rendering objects of other normally working applications can still be rendered normally, which improves the overall stability of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the technical solutions in the embodiments of this article or the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of this article. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0054] Figure 1 FIG. 1 is a schematic diagram of an implementation system of a method for rendering three-dimensional content in a multi-task system in an embodiment of this invention;

[0055] Figure 2 , Figure 3 It is a flow chart of a method for rendering three-dimensional content in a multi-tasking system in an embodiment of this invention;

[0056] Figure 4 It shows the steps of generating rendering instructions for rendering objects according to application logic in the embodiment of this article;

[0057] Figure 5 It shows the steps of rendering objects of multiple applications according to the rendering instructions sent by multiple applications received by the system in the embodiment of this article;

[0058] Figure 6 The figure shows the steps of rendering objects of multiple applications according to the rendering instruction and the viewing angle information of the operating user in the embodiment of this article.

[0059] Figure 7 The data flow diagram of generating rendering instructions for the rendering object according to the application logic in which the metadata of the rendering object also includes incremental information in the embodiment of the present invention is shown;

[0060] Figure 8 , Fig. 9 FIG. 1 is a schematic diagram of a structure of a device for rendering three-dimensional content in a multi-task system in an embodiment of this invention;

[0061] Fig.10 The figure is a schematic diagram of the structure of a rendering system for rendering three-dimensional content in a multi-tasking system;

[0062] Fig.11 The figure shows a schematic diagram of the structure of the computer device in the embodiment of this article;

[0063] Fig.12 Shown is a schematic diagram of implementing the interaction between the operating user and the rendered object in the embodiment of this article.

[0064]

Description of the accompanying drawings

[0065] 101. Support devices for displaying three-dimensional content;

[0066] 102. Application interaction module;

[0067] 103. System synthesis module;

[0068] 104. Displaying a three-dimensional interface module;

[0069] 801, rendering event receiving unit;

[0070] 802, rendering instruction generation unit;

[0071] 803, rendering instruction sending unit;

[0072] 901, rendering instruction receiving unit;

[0073] 902, observation angle information acquisition unit;

[0074] 903, rendering unit;

[0075] 1001. System;

[0076] 1002. Application;

[0077] 1102. Computer equipment;

[0078] 1104. Processing equipment;

[0079] 1106. Storage resources;

[0080] 1108, driving mechanism;

[0081] 1110, input / output module;

[0082] 1112. Input device;

[0083] 1114. Output device;

[0084] 1116. Presentation equipment;

[0085] 1118. Graphical user interface;

[0086] 1120. Network interface;

[0087] 1122. Communication link;

[0088] 1124. Communication bus. DETAILED DESCRIPTION

[0089] The following will be combined with the drawings in the embodiments of this article to clearly and completely describe the technical solutions in the embodiments of this article. Obviously, the described embodiments are only part of the embodiments of this article, not all of the embodiments. Based on the embodiments of this article, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this article.

[0090] It should be noted that the terms "first", "second", etc. in the specification and claims of this article and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of this article described here can be implemented in an order other than those illustrated or described here. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, device, product or equipment that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0091] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0092] like Figure 1 The figure shows a schematic diagram of an implementation system of a method for rendering three-dimensional content in a multi-task system according to an embodiment of the present invention, which is applied to a multi-task system supporting a device 101 for displaying three-dimensional content. The device supporting the display of three-dimensional content may be an XR device, a 3D projection device, etc. The multi-task system may include: a system synthesis module 103 and multiple application interaction modules 102. Each application interaction module 102 may interact with an operating user. The application in the application interaction module 102 may generate a rendering instruction for a rendering object according to the application logic. Specifically, the triggered rendering event is parsed, the metadata of the rendering object is obtained, the metadata of the rendering object is converted to be applicable to the system call, and a rendering instruction for the rendering object is generated. The application interaction module 102 and the system synthesis module 103 may communicate with each other through IPC and other methods, and the application sends the rendering instruction to the system synthesis module 103. The system synthesis module 103 may obtain the viewing angle information of the operating user, render the rendering object according to the received rendering instruction and the viewing angle information of the operating user, and obtain a rendering screen including the rendering objects of multiple applications. The rendering screen may be displayed to the operating user through the display three-dimensional interface module 104.

[0093] Subsequently, the operating user can also interact with the objects rendered in the display three-dimensional interface module 104 through the application interaction module 102. For example, the application interaction module 102 obtains the operating user's operation information, and the application interaction module 102 performs logical processing on the received rendering event (i.e., the operating user's operation information) to obtain a rendering instruction, and sends the rendering instruction to the system synthesis module 103. The system synthesis module 103 renders the rendering objects of multiple applications based on the received rendering instruction and the operating user's observation angle information to obtain a rendering screen containing the rendering objects of multiple applications, and displays the rendering screen to the operating user through the display three-dimensional interface module 104, thereby realizing the interactive effect between the operating user and the rendering object.

[0094] Of course, the rendering event triggered by the analysis of the application interaction module 102 may also be content generated inside the application. The application performs logical processing based on its own content to obtain rendering instructions, and sends the rendering instructions to the system synthesis module 103. The system synthesis module 103 renders the rendering objects of multiple applications based on the received rendering instructions and the observation angle information of the operating user to obtain a rendering screen containing the rendering objects of multiple applications.

[0095] In the embodiment of the present invention, the application interaction module 102 and the system synthesis module 103 may also transmit data through other data transmission methods, which are not limited in the embodiment of the present invention. Figure 1 What is shown is only one scenario provided by the present disclosure. In practice, other scenarios may also be included, and the present invention is not limited thereto.

[0096] In the prior art, each application renders its own rendering object to obtain a rendered image, and then sends the rendered image to the system process for synthesis. Due to the high-resolution requirements of XR, the resolution of the image rendered by the application is high, and the storage space occupied by the rendered image is large, which makes the transmission of high-resolution rendered images between processes subject to huge transmission pressure and increases transmission delay.

[0097] In order to solve the problems existing in the prior art, an embodiment of this article provides a method for rendering three-dimensional content in a multi-tasking system, where the application is responsible for generating rendering instructions for rendering objects according to the application logic, and then sending the rendering instructions to the system, which renders the rendering objects according to the rendering instructions. Figure 2The flowchart of a method for rendering three-dimensional content in a multi-tasking system in an embodiment of the present invention is shown. This figure describes the process of applying the system to perform rendering, but it may include more or fewer operation steps based on conventional or non-creative labor. The order of steps listed in the embodiment is only one way of executing the steps among many orders, and does not represent the only order of execution. When the system or device product is executed in practice, it can be executed in the order or in parallel according to the method shown in the embodiment or the accompanying drawings. Specifically, Figure 2 As shown, the method may be executed by an application and may include:

[0098] Step 201: Generate rendering instructions for a rendering object according to application logic, wherein the rendering instructions are used to guide the generation of a rendering image of the rendering object;

[0099] Step 202: Send rendering instructions to the system.

[0100] So that the system can render the rendering objects of multiple applications according to the received rendering instructions sent by the multiple applications, and obtain a rendering screen containing the rendering objects of the multiple applications.

[0101] Correspondingly, an embodiment of the present invention further provides a method for rendering three-dimensional content in a multi-task system, which is executed by the system, such as Figure 3 As shown, the method may include:

[0102] Step 301: receiving rendering instructions of rendering objects sent by multiple applications, where the rendering instructions are generated by the applications according to application logic, and the rendering instructions are used to guide the generation of rendering images of the rendering objects;

[0103] Step 302: Acquire viewing angle information of the operating user according to a device that supports displaying three-dimensional content;

[0104] Step 303: Rendering the rendering objects of multiple applications according to the rendering instruction and the viewing angle information of the operating user to obtain a rendering screen including the rendering objects of multiple applications.

[0105] Through the method of the embodiment of this article, the application generates a rendering instruction for the rendering object according to the application logic, and then the application sends the rendering instruction to the system. Compared with the traditional method in which the application renders the rendering object and sends the rendered image to the system, the size of the rendering instruction sent by the application of the present invention is much smaller than the rendered image, so it can effectively reduce the data transmission pressure between the application and the system and reduce the transmission delay. Finally, the system renders the rendering objects of multiple applications based on the rendering instructions sent by the received application and the observation angle information of the operating user to obtain a rendering screen. The present invention still retains the execution of the application logic in the application, and the system executes the rendering. Even if an application crashes due to the processing logic, it will not affect the operation of the system. The rendering objects of the remaining normally working applications can still be rendered normally, which improves the overall stability of the system.

[0106] The application generates rendering instructions according to the application logic. In the embodiment of the present invention, the rendering instruction is not an image obtained by rendering the rendering object, but is used to guide the generation of the rendering image of the rendering object. The application sends the rendering instruction to the system, which can be understood as the system generating the rendering image under the guidance of the rendering instruction. The rendering instruction is any instruction that can instruct the system to control the hardware to render the picture. Of course, the rendering instruction can also include two parts, one part is any content that can be converted into a hardware instruction, and the other part is a hardware instruction. The hardware instruction controls the hardware to render the picture, and the type of the hardware instruction can be a GPU instruction, such as the GPU API of OpenGL / Vulkan / DirectX, etc. The present invention does not limit the specific form of the rendering instruction.

[0107] Of course, the present invention does not limit one or some applications to directly send rendering images (for example, they may be relatively simple, have small memory or are more suitable for rendering images) to the system. The system receives rendering instructions and rendering images sent by multiple applications, and the system renders the rendering objects of the multiple applications based on the rendering instructions, and synthesizes the rendering images sent by the multiple applications, and finally obtains a rendering image containing the rendering objects of the multiple applications.

[0108] In the embodiments of this article, the granularity of the rendering instruction can be set according to actual needs, and the present invention does not limit it.

[0109] like Figure 4 As shown, generating a rendering instruction for a rendering object according to the application logic further includes:

[0110] Step 401: parse the triggered rendering event and obtain metadata of the rendering object;

[0111] Step 402: Perform call conversion for the rendering object's metadata to be applicable to the system, and generate rendering instructions for the rendering object.

[0112] This allows the system to render rendering objects of multiple applications according to the received rendering instructions sent by the multiple applications.

[0113] In the embodiment of the present invention, the rendering event may include the operation information of the operating user, and may also include the content generated by the application according to the application logic. The metadata of the rendering object includes the description information and attribute information of the rendering object. The category of the description information of the rendering object may be picture or video content, text content, model geometry information, material, texture, etc. The attribute information of the rendering object may be such as position, rotation, scaling, transparency, animation, and whether to display, etc.

[0114] For example, if the object to be rendered is a puppy, the application converts the metadata of the puppy's rendering content (such as current state (such as squatting), appearance, position, rotation, size, etc.) into corresponding rendering instructions and sends them to the system. The system renders the puppy at the specified position according to the received rendering instructions to obtain a rendered image. Multiple users can see a puppy squatting at a position in the environment.

[0115] The application can support the user to interact with the "puppy", such as lying down after the interaction. When the user interacts with the "puppy", such as touching it, the user's interactive operation is sent to the application as a rendering event. The application parses the rendering event, obtains the corresponding "puppy" metadata, and then determines that the "puppy" will lie down when touched based on the logic inside the application. Therefore, the application will convert the updated state of the "puppy" into a rendering instruction and pass it to the system. The system will draw the posture after the touch operation according to the rendering instruction. Animation can also be supported between these two actions. Animation-related content (such as motion models) can be notified to the system for rendering through model rendering instructions.

[0116] The rendering event triggered by the analysis in step 401 further includes:

[0117] The application analyzes the triggered rendering events and creates the 3D content that needs to be rendered. The 3D content can be generated directly by the application or analyzed by reading files;

[0118] 3D content can be described by traditional 3D model methods such as geometry and materials, read from 3D model file formats such as fbx and obj, or described by other methods such as NeRF. 3D content files can also be other custom file formats. The application parses the rendering event and converts it into metadata of the rendering object, that is, the model geometry information, text content, material and other description information of the rendering object, as well as attribute information such as position, rotation, and scale.

[0119] For example, the object to be rendered is a "puppy", and the "puppy" has its own animation effect A. When the user clicks or touches the "puppy", another animation effect B will be triggered. At this time, animation effect B is the rendering event.

[0120] The rendering event triggered by the application analysis is used to create the three-dimensional content of the animation effect B that needs to be rendered, and the animation effect B is converted and analyzed into metadata, including the model geometry information, material and other descriptive information of the "puppy", as well as the position, rotation, scale and other attribute information of the "puppy".

[0121] Step 402 converts the metadata of the rendering object into a call suitable for the system to generate a rendering instruction for the rendering object, which further includes:

[0122] Perform time serialization processing on the metadata of the rendering object to obtain the frame metadata of the rendering object;

[0123] Based on the rendering logic of the system, control instructions for frame metadata corresponding to the rendering object are generated.

[0124] The rendering instruction at this time includes both the frame metadata of the rendering object and the control instruction corresponding to the frame metadata. Both the frame metadata and the control instruction are transmitted to the system, and the system performs rendering to obtain a rendered image.

[0125] In the embodiment of the present invention, the rendering event includes operation information of the operating user, and the system renders the rendering objects of the multiple applications according to the received rendering instructions sent by the multiple applications, specifically including:

[0126] The system renders rendering objects of multiple applications according to the received rendering instructions sent by the multiple applications and the viewing angle information of the operating user.

[0127] The metadata may also be a motion model of a rendering object, and the metadata of the rendering object is converted into a call applicable to the system to generate a rendering instruction for the rendering object. The rendering instruction at this time may be a model rendering instruction, and the system may generate an image of the rendering object according to the model rendering instructions sent by multiple applications.

[0128] In an embodiment of the present invention, the motion model of the rendering object is subjected to application logic processing to generate a model rendering instruction for the rendering object, and the model rendering instruction is sent to the system so that the system renders the rendering objects of multiple applications according to the model rendering instruction.

[0129] Similarly, the system may render rendering objects of multiple applications according to the received model rendering instructions sent by the multiple applications and the viewing angle information of the operating user.

[0130] With the traditional method, each application has a corresponding canvas range, that is, the application can only render images within the specified canvas range, that is, the part of the rendered object within the specified canvas range can be rendered, and the part outside the canvas range cannot be rendered. Therefore, the interaction effect between the objects corresponding to each application cannot be achieved, the flexibility of expression and interaction of the XR system is low, and the user experience is relatively simple.

[0131] Compared with the traditional method of rendering images by applications, the present invention removes the limitation that the rendering objects of an application can only be rendered within the canvas range of the application by transmitting rendering instructions. Specifically, according to an embodiment of the present invention, Figure 5 As shown, the system renders the rendering objects of multiple applications according to the received rendering instructions sent by the multiple applications and the viewing angle information of the operating user, further comprising:

[0132] Step 501: Obtain a canvas range of a device that supports displaying three-dimensional content;

[0133] Step 502: Rendering objects of multiple applications within a canvas range according to the rendering instruction and the observation angle information.

[0134] In the embodiment of this article, the canvas range of the device supporting the display of three-dimensional content represents the range of the entire display interface. The system can obtain the range of the entire display interface by calling the system hardware interface, and then render the metadata when the operating user specifies the attributes (i.e., attribute information) of the rendering object in the entire display interface. Through the method of the present invention, the operating user can arbitrarily specify the attributes of the rendering object in the display interface, so that the interaction between multiple rendering objects can be realized, enriching the operating user experience.

[0135] In the prior art, an application renders a rendering object. When the system synthesizes the rendered images sent by multiple applications, it is necessary to calculate the occlusion relationship of the rendered images based on the properties of the rendered images and the viewing angle of the operating user, and remove the occluded parts of each image. The 2D multi-tasking system can know the occlusion situation because the viewing angle is fixed and the picture is a complete rectangular area. The 3D multi-tasking viewing angle changes every frame, and there will be subtle differences even if it does not move. In addition, the picture content of 3D applications is irregular and may have transparent areas, so the occlusion situation is more complicated. It is more expensive to calculate this information and transmit it to the application for each frame, and the performance loss may be greater than rendering without considering occlusion. Therefore, the application can only render the entire object, and for the part that the system needs to remove during synthesis, the application performs redundant rendering, resulting in a waste of application computing resources.

[0136] The application of the present invention no longer renders the rendering object, but sends the rendering instruction to the system, and the system renders the received rendering instruction. Because the system obtains the rendering instruction, the system can first calculate the occlusion relationship of the rendering object according to the rendering instruction, and then only render the metadata without occlusion. Specifically, according to an embodiment of the present invention, Figure 6 As shown, rendering objects of multiple applications are rendered according to the rendering instruction and the viewing angle information of the operating user, further comprising:

[0137] Step 601: Calculate the occlusion relationship between metadata according to the viewing angle information of the operating user and all rendering instructions;

[0138] Step 602: Render the metadata without occlusion to obtain a rendered image.

[0139] In the embodiment of the present invention, the step of calculating the occlusion relationship is a basic operation of graphics rendering, based on depth detection, and its process has been encapsulated in a common API such as OpenGL and completed at the GPU driver layer.

[0140] The transmission method of rendering object metadata in the embodiment of the present invention can provide the GPU with sufficient three-dimensional scene information to perform occlusion culling. Figure 6 In the method shown, the system first calculates the occlusion relationship of the metadata and then renders the metadata without occlusion, thereby reducing the amount of rendering calculations and saving computing resources.

[0141] In the prior art, because the system is only responsible for synthesizing the received rendered images, the system does not have the metadata of the rendered object. If the shape, color, and attributes of the rendered object change, the application needs to render the changed rendered object again and send the re-rendered image to the system for synthesis. If the rendered object is constantly moving and changing, the application needs to render the rendered object at each moment, generate a rendered image for each frame, and then send the rendered image for each frame to the system in sequence. This method causes an excessive amount of data to be transmitted between the application and the system, and a significant transmission delay.

[0142] To solve the above problem, according to one embodiment of the present invention, the metadata of the rendering object further includes incremental information, and the rendering event includes an object initialization event and an object adjustment event;

[0143] like Figure 7 As shown, the method also includes:

[0144] Step 701: the application performs application logic processing on the object initialization event to generate an initial rendering instruction for the rendering object;

[0145] Step 702: The application sends an initial rendering instruction to the system;

[0146] Step 703: the system renders rendering objects of multiple applications according to the received initial rendering instructions sent by the multiple applications;

[0147] Step 704: the application performs application logic processing on the object adjustment event to generate an incremental rendering instruction for the rendering object;

[0148] Step 705: The application sends the incremental rendering instruction to the system;

[0149] Step 706: The system renders rendering objects of multiple applications according to the incremental rendering instruction.

[0150] In an embodiment of the present invention, object initialization events and object adjustment events can be issued by an operating user or generated by an application. When issued by an operating user, the operating user first initializes the metadata of the rendering object, and the application processes the object initialization event with application logic to generate an initial rendering instruction for the rendering object; the system renders the rendering object initialized by the operating user, and then the operating user can interact with the rendered object through an input device such as a handheld handle, that is, the adjustment event of the rendered object is issued through an input device such as a handheld handle, and the metadata of the object (such as text content, material, etc. in the description information, or position, rotation, scaling, etc. in the attribute information) can be adjusted, and the application processes the object adjustment event with application logic to generate an incremental rendering instruction for the rendering object, and the system renders according to the incremental rendering instruction to achieve interaction between the user and the rendering object.

[0151] For example, Fig.12 As shown:

[0152] The user opens application A and performs the following object initialization operations, triggering an object initialization event. Initialize the metadata (such as color, size, etc.) of the rendering object "Little Dinosaur" corresponding to application A;

[0153] Open application B and perform the following object initialization operations to trigger an object initialization event: Initialize the metadata (such as size, etc.) of the rendering object "coconut tree" corresponding to application B;

[0154] After application A obtains the object initialization event, it parses it and obtains the metadata of the "little dinosaur" such as the color and size specified by the operating user, converts the metadata of the "little dinosaur" into a call applicable to the system, generates the initial rendering instruction of the "little dinosaur", and then application A sends the initial rendering instruction of the "little dinosaur" to the system;

[0155] After application B obtains the object initialization event, it parses it and obtains the metadata of the "coconut tree" such as the size specified by the operating user, performs a call conversion on the metadata of the "coconut tree" applicable to the system, generates the initial rendering instruction of the "coconut tree", and then application B sends the initial rendering instruction of the "coconut tree" to the system;

[0156] The system renders the “little dinosaur” and the “coconut tree” according to the initial rendering instruction of the “little dinosaur” and the initial rendering instruction of the “coconut tree”;

[0157] The user wants the "little dinosaur" to walk under the "coconut tree". The object adjustment event triggered by application A is parsed to obtain the metadata of the "little dinosaur". The metadata of the "little dinosaur" is converted to the applicable system call to generate the incremental rendering instructions of the "little dinosaur".

[0158] Application A sends the incremental rendering instructions of "Little Dinosaur" to the system;

[0159] The system recalculates the occlusion relationship between the "little dinosaur" and the "coconut tree" according to the incremental rendering instructions of the "little dinosaur", and re-renders the "little dinosaur" and the "coconut tree".

[0160] pass Figure 7 According to the method shown in the present invention, after the rendering object is adjusted or changed, the application only needs to send an incremental rendering instruction of the rendering object to the system, thereby reducing the pressure of data transmission.

[0161] In some other embodiments of the present invention, after the user initializes the rendering object, there may be no need to adjust the rendering object for a period of time. The rendering object only needs to be stationary or move regularly according to the logic set by the application. In this case, in order to further reduce the amount of data transmission between the application and the system, the metadata of the rendering object also includes a motion model of the rendering object. Figure 7 As shown, the method also includes:

[0162] Step 707: the application processes the motion model of the rendering object with the application logic, generates a model rendering instruction for the rendering object, and sends the model rendering instruction to the system;

[0163] Step 708: The system renders rendering objects of multiple applications according to the received model rendering instructions.

[0164] For example, the application logic of application A includes: rendering a "little dinosaur". The "little dinosaur" has its own animation effect. When receiving a rendering event in which the user clicks or touches the "little dinosaur", another animation effect will be triggered.

[0165] The application logic of application B includes: rendering a "coconut tree", and the "coconut tree" has its own animation effects, such as swaying in the wind.

[0166] When the user opens application A, the application generates rendering instructions corresponding to the "little dinosaur" to describe the appearance of the "little dinosaur" (that is, metadata, such as size, rotation, position, animation and other information) and sends it to the system.

[0167] When the user opens application B, the application generates rendering instructions corresponding to the "coconut tree" to describe the appearance of the "coconut tree" (i.e. metadata, such as size, rotation, position, animation, etc.) and sends it to the system.

[0168] The system performs unified rendering synthesis based on the rendering instructions of applications A and B.

[0169] When the user clicks on the "little dinosaur", application A receives the rendering event and triggers the corresponding animation effect of the little dinosaur according to the application logic. It also generates the corresponding incremental rendering instructions for this part of the change and sends them to the system.

[0170] The user moves the "little dinosaur" to under the "coconut tree", and the "little dinosaur" and the "coconut tree" are occluded. The system recalculates the occlusion relationship between the two based on the incremental information of the "little dinosaur"'s position change, and re-renders the "little dinosaur" and the "coconut tree". The occluded part is removed at this time and does not need to be rendered, saving rendering resources.

[0171] When the "little dinosaur" is between the two leaves of the "coconut tree", the correct occlusion relationship can also be achieved based on the metadata information. This situation cannot be handled for the previous application that sent the rendered image. It can only achieve the effect of the "little dinosaur" completely occluding the "coconut tree" or the "coconut tree" completely occluding the "little dinosaur".

[0172] In the embodiments of this article, the motion model can specify the motion law of the rendering object. After the application sends the model rendering instruction to the system, it is no longer necessary to parse the triggered rendering event and obtain the metadata of the rendering object. The system calculates the initial rendering instructions and incremental rendering instructions of the rendering object in real time according to the model rendering instruction, and dynamically renders the rendering object in real time, thereby reducing the amount of data transmission between the application and the system. Because the system needs to transmit data with multiple applications, and their data transmission is all through the same message transmission interface, such as IPC communication, so after reducing the amount of data transmission between the application and the system, the efficiency of the overall interactive communication is also significantly improved.

[0173] Based on the same inventive concept, an embodiment of the present invention further provides a device for rendering three-dimensional content in a multi-task system, such as Figure 8 As shown, including,

[0174] A rendering event receiving unit 801, used to receive a triggered rendering event;

[0175] A rendering instruction generating unit 802, used to generate a rendering instruction for a rendering object according to the application logic, wherein the rendering instruction is used to guide the generation of a rendering image of the rendering object;

[0176] The rendering instruction sending unit 803 is used to send the rendering instruction to the system, so that the system renders the rendering objects of multiple applications according to the received rendering instructions sent by the multiple applications to obtain a rendering screen including the rendering objects.

[0177] Correspondingly, an embodiment of the present invention further provides a device for rendering three-dimensional content in a multi-task system, such as Fig. 9 As shown, including,

[0178] The rendering instruction receiving unit 901 is used to receive rendering instructions of rendering objects sent by multiple applications. The rendering instructions are generated by the applications according to the application logic, and the rendering instructions are used to guide the generation of rendering images of the rendering objects.

[0179] An observation angle information acquisition unit 902 is used to acquire the observation angle information of the operating user according to the device supporting the display of the three-dimensional content;

[0180] The rendering unit 903 is used to render the rendering objects of multiple applications according to the rendering instruction and the viewing angle information of the operating user to obtain a rendering screen including the rendering objects.

[0181] Furthermore, an embodiment of the present invention also provides a rendering system for rendering three-dimensional content in a multi-task system, such as Fig.10 As shown, it includes a rendering system 1001 and multiple applications 1002;

[0182] When any application 1002 is rendering, it executes the method executed by the above application;

[0183] When rendering, the rendering system 1001 executes the method executed by the above system.

[0184] The beneficial effects obtained by the above-mentioned device and system are consistent with the beneficial effects obtained by the above-mentioned method, and are not described in detail in the embodiments of this specification.

[0185] like Fig.11The structure diagram of the computer device of the embodiment of the present invention is shown. The apparatus in the present invention can be the computer device in the embodiment, and executes the method of the present invention described above. The computer device 1102 may include one or more processing devices 1104, such as one or more central processing units (CPUs), and each processing unit may implement one or more hardware threads. The computer device 1102 may also include any storage resource 1106, which is used to store any kind of information such as code, settings, data, etc. Non-limitingly, for example, the storage resource 1106 may include any one or more combinations of the following: any type of RAM, any type of ROM, flash memory device, hard disk, optical disk, etc. More generally, any storage resource can use any technology to store information. Further, any storage resource can provide volatile or non-volatile retention of information. Further, any storage resource can represent a fixed or removable component of the computer device 1102. In one case, when the processing device 1104 executes an associated instruction stored in any storage resource or a combination of storage resources, the computer device 1102 can perform any operation of the associated instruction. The computer device 1102 also includes one or more drive mechanisms 1108 for interacting with any storage resources, such as a hard disk drive mechanism, an optical disk drive mechanism, and the like.

[0186] The computer device 1102 may also include an input / output module 1110 (I / O) for receiving various inputs (via input devices 1112) and for providing various outputs (via output devices 1114). A specific output mechanism may include a presentation device 1116 and an associated graphical user interface (GUI) 1118. In other embodiments, the input / output module 1110 (I / O), input device 1112, and output device 1114 may not be included, and the computer device 1102 may be used as a computer device in a network. The computer device 1102 may also include one or more network interfaces 1120 for exchanging data with other devices via one or more communication links 1122. One or more communication buses 1124 couple the components described above together.

[0187] The communication link 1122 may be implemented in any manner, for example, through a local area network, a wide area network (e.g., the Internet), a point-to-point connection, etc., or any combination thereof. The communication link 1122 may include any combination of hardwired links, wireless links, routers, gateway functions, name servers, etc. governed by any protocol or combination of protocols.

[0188] The embodiments of the present invention further provide a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and the computer program implements the above method when executed by a processor.

[0189] The embodiments of the present invention also provide a computer-readable instruction, wherein when a processor executes the instruction, the program therein causes the processor to execute the above method.

[0190] It should be understood that in the various embodiments of this document, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this document.

[0191] It should also be understood that in the embodiments of this article, the term "and / or" is only a description of the association relationship of the associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0192] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this article.

[0193] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0194] In the several embodiments provided herein, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of the units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, or can be electrical, mechanical or other forms of connection.

[0195] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the embodiments of this article.

[0196] In addition, each functional unit in each embodiment of this invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above integrated unit may be implemented in the form of hardware or in the form of software functional unit.

[0197] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this article is essentially or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of this article. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0198] Specific embodiments are used in this article to illustrate the principles and implementation methods of this article. The description of the above embodiments is only used to help understand the methods and core ideas of this article. At the same time, for general technicians in this field, according to the ideas of this article, there will be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as a limitation on this article.

Claims

1. A method for rendering three-dimensional content in a multi-tasking system, characterized in that: The method is applied to a multi-task system of a device supporting display of three-dimensional content and is executed by multiple applications in the multi-task system of a device supporting display of three-dimensional content. The method includes: Generate rendering instructions for the rendering objects according to the application logic and metadata of the rendering objects, wherein the rendering instructions are used to instruct the system to control hardware to generate rendering images of the rendering objects of multiple applications; The rendering instruction is sent to the system, so that the system renders the rendering objects of the multiple applications according to the received rendering instructions sent by the multiple applications, and obtains a rendering screen containing the rendering objects of the multiple applications.

2. The method according to claim 1, characterized in that Generating a rendering instruction for the rendering object according to the application logic and metadata of the rendering object further includes: Parsing the triggered rendering event to obtain metadata of the rendering object, wherein the metadata of the rendering object includes description information and attribute information of the rendering object; The metadata of the rendering object is converted into a call applicable to the system to generate a rendering instruction for the rendering object.

3. The method according to claim 2, characterized in that Rendering events include: operation information of the operating user; The system renders rendering objects of multiple applications according to the received rendering instructions sent by the multiple applications, specifically including: The system renders rendering objects of multiple applications according to the received rendering instructions sent by the multiple applications and the observation angle information of the operating user.

4. The method according to claim 2, characterized in that: The metadata of the rendering object also includes incremental information, and the rendering event includes an object initialization event and an object adjustment event; The method further comprises: Processing the object initialization event according to the application logic to generate an initial rendering instruction for the rendering object, and sending the initial rendering instruction to the system, so that the system renders the rendering objects of multiple applications according to the received initial rendering instructions sent by the multiple applications; The object adjustment event is processed by the application logic to generate an incremental rendering instruction for the rendering object, and the incremental rendering instruction is sent to the system so that the system renders the rendering objects of multiple applications according to the incremental rendering instruction.

5. The method according to claim 1, characterized in that The rendering instruction includes frame metadata of the rendering object and a control instruction corresponding to the frame metadata.

6. A method for rendering three-dimensional content in a multi-tasking system, characterized in that: The method is applied to a multi-task system of a device supporting display of three-dimensional content and is executed by a system in the multi-task system of a device supporting display of three-dimensional content, and includes: receiving rendering instructions of rendering objects sent by a plurality of applications, wherein the rendering instructions are generated by the applications according to application logic and metadata of the rendering objects, and the rendering instructions are used to guide generation of rendering images of the rendering objects; According to the device supporting the display of three-dimensional content, obtaining the viewing angle information of the operating user; According to the rendering instruction and the viewing angle information of the operating user, the hardware is controlled to render the rendering objects of multiple applications to obtain a rendering screen including the rendering objects of multiple applications.

7. The method according to claim 6, characterized in that The rendering instruction is generated by the application according to the application logic, and further includes: The application parses the triggered rendering event and obtains metadata of the rendering object, wherein the rendering event includes operation information of the operating user, and the metadata includes description information and attribute information of the rendering object; The metadata of the rendering object is converted into a call applicable to the system to generate a rendering instruction for the rendering object.

8. The method according to claim 7, characterized in that The metadata of the rendering object further includes incremental information, the rendering event includes an object initialization event and an object adjustment event, the initial rendering instruction is generated by the application performing the application logic processing on the object initialization event, and the incremental rendering instruction is generated by the application performing the application logic processing on the object adjustment event; The method further comprises: Rendering the rendering objects of the multiple applications according to the received initial rendering instructions sent by the applications and the viewing angle information of the operating user; The rendering objects of the multiple applications are rendered according to the received incremental rendering instructions sent by the applications and the viewing angle information of the operating user.

9. The method according to claim 6, characterized in that The rendering instruction includes frame metadata of the rendering object and a control instruction corresponding to the frame metadata.

10. The method according to claim 6, characterized in that Rendering the rendering objects of multiple applications according to the rendering instruction and the viewing angle information of the operating user further includes: Obtaining a canvas range of the device supporting display of three-dimensional content; Rendering objects of multiple applications are rendered within the canvas range according to the rendering instruction and the observation angle information.

11. The method according to claim 7, characterized in that Rendering the rendering objects of multiple applications according to the rendering instruction and the viewing angle information of the operating user further includes: Calculating the occlusion relationship between the metadata according to the viewing angle information of the operating user and all rendering instructions; The metadata without any occlusion is rendered to obtain the rendered image.

12. A device for rendering three-dimensional content in a multi-tasking system, characterized in that: The device comprises: A rendering instruction generating unit, used to generate a rendering instruction of a rendering object according to the application logic and metadata of the rendering object, wherein the rendering instruction is used to instruct the system control hardware to generate a rendering image of the rendering object; The rendering instruction sending unit is used to send the rendering instruction to the system, so that the system renders the rendering objects of multiple applications according to the received rendering instructions sent by the multiple applications to obtain a rendering picture including the rendering objects.

13. A device for rendering three-dimensional content in a multi-tasking system, characterized in that: The device comprises: A rendering instruction receiving unit, used to receive rendering instructions of rendering objects sent by multiple applications, wherein the rendering instructions are generated by the applications according to application logic and metadata of the rendering objects, and the rendering instructions are used to instruct the system control hardware to generate a rendered image of the rendering object; An observation angle information acquisition unit, used to acquire the observation angle information of the operating user according to a device supporting the display of three-dimensional content; The rendering unit is used to render the rendering objects of multiple applications according to the rendering instruction and the observation angle information of the operating user to obtain a rendering screen including the rendering objects.

14. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 11 is implemented.

15. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 11 is implemented.

Images