Hybrid rendering method, hybrid rendering system, rendering device and storage medium

Abstract

The invention relates to the technical field of computers, and discloses a hybrid rendering method, a hybrid rendering system, rendering equipment and a storage medium. The method is applied to the hybrid rendering system, the hybrid rendering system comprises a plurality of rendering devices, the plurality of rendering devices at least comprise a local device and a cloud rendering server, and the method comprises the following steps: aiming at a first rendering device and a second rendering device in the plurality of rendering devices, when the first rendering device renders a target task through a first rendering application, the second rendering device renders the target task through a second rendering application; monitoring the rendering performance of the first rendering application; under the condition that the rendering performance of the first rendering application does not meet the rendering requirement, the second rendering device loads a target rendering model from the cloud storage and performs rendering processing by using the target rendering model to obtain a rendered video stream, the target rendering model is a rendering model corresponding to the target task, and the rendering performance of the second rendering application does not meet the rendering requirement. The rendered video stream is used for being displayed by the local device through the second rendering application.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a hybrid rendering method, a hybrid rendering system, a rendering device, and a storage medium. Background Art

[0002] Browser rendering in the local device can be achieved through a local rendering application or a cloud rendering application.

[0003] A local rendering application is a rendering application that uses resources in a local device for rendering processing, and a cloud rendering application is a rendering application that uses resources in a cloud rendering server for rendering processing.

[0004] In the related art, generally, based on the capabilities and characteristics of the two rendering applications, a rendering application is autonomously selected according to the current scene. If the rendering performance of the currently running rendering application decreases, it cannot meet the rendering requirements of the current scene. Summary of the invention

[0005] In view of this, the present disclosure provides a hybrid rendering method, a hybrid rendering system, a rendering device, and a storage medium, which avoid the problem of poor rendering performance by dynamically switching rendering applications.

[0006] In a first aspect, the present disclosure provides a hybrid rendering method, which is applied to a hybrid rendering system, wherein the hybrid rendering system includes multiple rendering devices, and the multiple rendering devices include at least a local device and a cloud rendering server. The method includes:

[0007] For a first rendering device and a second rendering device among the multiple rendering devices, the first rendering device monitors the rendering performance of the first rendering application when rendering the target task through the first rendering application;

[0008] When the rendering performance of the first rendering application does not meet the rendering requirements, the second rendering device loads the target rendering model from the cloud storage, and uses the target rendering model for rendering processing to obtain a rendered video stream. The target rendering model is the rendering model corresponding to the target task, and the rendered video stream is used for display by the local device through the second rendering application.

[0009] Based on the above technical solution, a hybrid rendering system including multiple rendering devices is proposed, wherein the multiple rendering devices include at least a local device and a cloud rendering server. For a first rendering device and a second rendering device among the multiple rendering devices, the first rendering device monitors the rendering performance of a first rendering application that is rendering a target task. When the rendering performance of the first rendering application does not meet the rendering requirements, the second rendering device loads a target rendering model corresponding to the target task from the cloud storage, and the second rendering device transmits a video stream rendered by the target rendering model to a local device. The local device uses the second rendering application to display the rendered video stream, so as to switch from the first rendering application to the second rendering application, thereby ensuring the rendering performance through dynamic switching of rendering applications.

[0010] In some optional implementations, when the rendering performance of the first rendering application does not meet the rendering requirement, the second rendering device loads the target rendering model from the cloud storage, including:

[0011] When the rendering performance of the first rendering application does not meet the rendering requirement, the first rendering device instructs the first rendering application to start the second rendering application, and provides the cloud storage address of the target rendering model to the second rendering device;

[0012] The second rendering device loads the target rendering model from the cloud storage based on the cloud storage address.

[0013] Based on the above technical solution, by transferring the cloud storage address, the second rendering device can be helped to accurately load the target rendering model from the cloud storage.

[0014] In some optional implementations, the hybrid rendering system further includes: a management service, the management service being used to manage a plurality of cloud rendering servers, each cloud rendering server including at least one cloud rendering program;

[0015] In the case where the first rendering device is a local device and the second rendering device is a cloud rendering server, the method further includes:

[0016] The management device determines the required scale of the cloud rendering program corresponding to the cloud rendering application based on the size of the target rendering model;

[0017] The management device allocates cloud rendering programs corresponding to the demand scale to cloud rendering applications among multiple cloud rendering servers managed by the management device.

[0018] Based on the above technical solution, when switching from a local rendering application to a cloud rendering application, the management device determines the demand scale of the cloud rendering program corresponding to the cloud rendering application based on the size of the target rendering model, and then allocates the cloud rendering program corresponding to the demand scale to the cloud rendering application, thereby reasonably allocating rendering resources and ensuring the subsequent rendering effect of the cloud rendering application.

[0019] In some optional implementations, the management device allocates cloud rendering programs corresponding to the demand scale to the cloud rendering application among the multiple cloud rendering servers managed by the management device, including:

[0020] Management device to obtain the load status of each cloud rendering program;

[0021] The management device selects a cloud rendering program corresponding to the demand scale of the cloud rendering application from among the multiple cloud rendering servers managed by the management device in order of load from low to high.

[0022] Based on the above technical solution, the management device preferentially selects cloud rendering programs with low loads to allocate to cloud rendering applications based on the load conditions of each cloud rendering program, thereby achieving reasonable allocation of resources.

[0023] In some optional implementations, after the management device allocates the cloud rendering program corresponding to the demand scale to the cloud rendering application among the multiple cloud rendering servers managed by the management device, the method further includes:

[0024] The management device receives permission sharing indication information fed back by the cloud rendering application;

[0025] When the authority sharing indication information indicates that the cloud rendering application supports authority sharing, the management device distributes the cloud rendering program allocated to the cloud rendering application to other cloud rendering applications in other local devices.

[0026] Based on the above technical solution, when the cloud rendering application indicates through permission sharing indication information that the resources it applies for can be shared, the management device will allocate the cloud rendering program allocated to the cloud rendering application to other cloud rendering applications in other local devices, so that multiple local devices can use one cloud rendering program at the same time, thereby realizing resource sharing of the cloud rendering program.

[0027] In some optional implementations, the management device distributes the cloud rendering program allocated to the cloud rendering application to other cloud rendering applications in other local devices, including:

[0028] The management device allocates the cloud rendering program allocated to the cloud rendering application to other cloud rendering applications corresponding to the cloud rendering application identification information based on the cloud rendering application identification information indicated in the permission sharing indication information;

[0029] or,

[0030] The management device receives a resource usage request sent by other cloud rendering applications, where the cloud rendering program requested in the resource usage request is the cloud rendering program allocated to the cloud rendering application, and in response to the resource usage request, allocates the cloud rendering program allocated to the cloud rendering application to other cloud rendering applications.

[0031] Based on the above technical solution, the user sets the shared object through the cloud rendering application identification information in the cloud rendering program, and the management service allocates the shared rendering resources to the shared object corresponding to the cloud rendering application identification information, or, by default, supports sharing to all users. When other users request the shared rendering resources through other cloud rendering applications in other local devices, the shared resources are allocated to the requester, thereby effectively setting the shared objects for multi-user shared rendering resources.

[0032] In some optional implementations, after the local device displays the video stream rendered by the second rendering device through the second rendering application, the method further includes:

[0033] The first rendering device destroys rendering-related resources of the first rendering application.

[0034] Based on the above technical solution, after the switching of the rendering application is completed, the rendering-related resources of the first rendering application that were originally used are destroyed in time, thereby avoiding waste of resources.

[0035] In some optional implementations, the local device displays the video stream rendered by the second rendering device through the second rendering application, including:

[0036] The local device sets the play tag of the second rendering application to be visible, and sets the play tag of the first rendering application to be hidden;

[0037] The local device plays the rendered video stream corresponding to the second rendering application based on the play tag, and the rendered video stream is displayed over the display area of ​​the first rendering application.

[0038] Based on the above technical solution, after the local device receives the rendered video stream, it plays the rendered video stream corresponding to the second rendering application based on the attribute settings of the playback tag, and covers the display area of ​​the first rendering application, thereby achieving an imperceptible switching effect when switching between the two rendering applications.

[0039] In a second aspect, the present disclosure provides a hybrid rendering system, wherein the hybrid rendering system includes a plurality of rendering devices, wherein the plurality of rendering devices include at least a local device and a cloud rendering server;

[0040] For a first rendering device and a second rendering device among the plurality of rendering devices, the first rendering device is configured to monitor rendering performance of the first rendering application when rendering a target task through the first rendering application;

[0041] The second rendering device is used to load a target rendering model from cloud storage and use the target rendering model to perform rendering processing to obtain a rendered video stream when the rendering performance of the first rendering application does not meet the rendering requirements. The target rendering model is a rendering model corresponding to the target task, and the rendered video stream is used for display by the local device through the second rendering application.

[0042] In a third aspect, the present disclosure provides a rendering device, comprising: a memory and a processor, the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the hybrid rendering method of the first aspect or any corresponding embodiment thereof by executing the computer instructions.

[0043] In a fourth aspect, the present disclosure provides a computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to enable a computer to execute the hybrid rendering method of the first aspect or any corresponding embodiment thereof. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the specific embodiments of the present disclosure or the technical solutions in the prior art, the drawings required for use in the specific embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0045] Figure 1 is a flowchart of a hybrid rendering method according to an embodiment of the present disclosure;

[0046] Figure 2 is a flowchart of another hybrid rendering method according to an embodiment of the present disclosure;

[0047] Figure 3 is a flowchart of another hybrid rendering method according to an embodiment of the present disclosure;

[0048] Figure 4 is a schematic diagram of allocation of a cloud rendering program according to an embodiment of the present disclosure;

[0049] Figure 5 is a structural schematic diagram of a hybrid rendering system according to an embodiment of the present disclosure;

[0050] Figure 6 is a structural schematic diagram of another hybrid rendering system according to an embodiment of the present disclosure;

[0051] Figure 7 Schematic diagram of the hardware structure of a rendering device according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0052] In order to make the purpose, technical solution and advantages of the embodiments of the present disclosure clearer, the technical solution in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present disclosure.

[0053] First, some concepts involved in the embodiments of the present disclosure are briefly introduced:

[0054] Cloud rendering: Rendering is not done on local devices. The rendering results are obtained by transmitting videos over the network, and then displayed and interacted with in the form of a browser.

[0055] Cloud rendering server: a machine other than a local device, which can be a physical machine or a virtual machine; one cloud rendering server can have multiple cloud rendering programs.

[0056] Cloud rendering program: 3D software, the main body of the rendering program, responsible for rendering.

[0057] Management service: monitor the health status of the cloud rendering server (whether it is running normally, the usage of resources such as the central processing unit (CPU), memory, and graphics processing unit (GPU)), and dynamically adjust the start and stop of the cloud rendering program according to resource consumption to ensure maximum resource utilization.

[0058] Sentinel service: deployed on the cloud rendering server, only one process is started on one server; responsible for sending the health status of the current server to the management service, and responsible for the start and stop operations of the cloud rendering program. Network communication: data is transmitted through the Transmission Control Protocol (TCP) and the User Datagram Protocol (UDP), supporting IPv4 and IPv6.

[0059] Browser rendering can currently be achieved through local rendering applications or cloud rendering applications, but the advantages and disadvantages of these two implementation methods are quite obvious.

[0060] Local rendering applications are not yet fully mature, so it is inevitable that they will not be able to meet the needs of the scene during use, such as the obvious lack of processing of large scenes and low frame rates. Cloud rendering applications require network transmission, so the delay will be large and the real-time interactive experience is not good. Based on this, the embodiments of the present disclosure provide a hybrid rendering method that avoids the problem of poor rendering performance by automatically switching rendering applications dynamically.

[0061] According to an embodiment of the present disclosure, a hybrid rendering method embodiment is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0062] In this embodiment, a hybrid rendering method is provided, which can be used in a hybrid rendering system. The hybrid rendering system includes multiple rendering devices, and the multiple rendering devices include at least a local device and a cloud rendering server. Figure 1 is a flow chart of a hybrid rendering method according to an embodiment of the present disclosure. Figure 1 As shown, the process includes the following steps:

[0063] Step S101 : for a first rendering device and a second rendering device among a plurality of rendering devices, when the first rendering device renders a target task through the first rendering application, the first rendering device monitors the rendering performance of the first rendering application.

[0064] In this implementation, the rendering application of the currently running rendering target task is recorded as a first rendering application, the rendering device corresponding to the first rendering application is recorded as a first rendering device, and the first rendering device monitors the rendering performance of the currently running first rendering application.

[0065] Step S102, when the rendering performance of the first rendering application does not meet the rendering requirements, the second rendering device loads the target rendering model from the cloud storage, and uses the target rendering model to perform rendering processing to obtain a rendered video stream, the target rendering model is the rendering model corresponding to the target task, and the rendered video stream is used for display by the local device through the second rendering application.

[0066] In this embodiment, the rendering model is stored in the cloud storage, and the rendering models used by the rendering application are synchronized to the cloud storage. The rendering model used by the currently executing target task is recorded as the target rendering model. If it is monitored that the rendering performance of the first rendering application does not meet the rendering requirements, the second rendering device loads the target rendering model from the cloud storage, and the second rendering device uses the loaded target rendering model to execute the target task, and provides the rendered video stream to the local device. The local device uses the second rendering application to display the rendered video stream, and then switches from the first rendering application to the second rendering application to complete the dynamic switching of the rendering application.

[0067] The rendering model may be composed of at least one of the following information: scene geometry information, material information, camera, and lighting.

[0068] It is understandable that different rendering applications may monitor parameters in different aspects to determine rendering performance and define different rendering requirements.

[0069] Exemplarily, if the first rendering device is a local device and the first rendering application is a local rendering application, the rendering time of each frame is monitored to determine the rendering performance. If the rendering time of each frame exceeds a threshold and lasts for a period of time, it is considered that the rendering requirements are not met.

[0070] Exemplarily, if the first rendering device is a cloud rendering server, and the first rendering application is a cloud rendering application, the GPU usage, video memory usage, and frame rate are monitored to determine the rendering performance. If the GPU usage exceeds the threshold, the video memory usage exceeds the threshold, and the frame rate is lower than the threshold for a period of time, it is considered that the rendering requirements are not met. Among them, the threshold of GPU usage can be calculated as follows: the graphics card performance of the display device / the graphics card performance of the cloud rendering host*0.x, 0.x is to prevent the graphics card from being fully loaded and to tolerate calculation errors, and the graphics card performance is obtained by weighting parameters such as the number of stream processors, core frequency, video memory bit width, and video memory frequency; the threshold of GPU usage can be calculated as follows: the video memory of the display device*0.x, 0.x is to prevent the display device from overflowing the video memory and to tolerate calculation errors; the threshold of the frame rate can be a frame rate with better user experience, generally 30 frames per second.

[0071] Based on the above steps, the first rendering device may be a local device, the first rendering application may be a local rendering application, the second rendering device may be a cloud rendering server, and the second rendering application may be a cloud rendering application. That is, when the rendering performance of the local rendering application does not meet the rendering requirements, the local rendering application is switched to the cloud rendering application. The first rendering device may be a cloud rendering server, the first rendering application may be a cloud rendering application, the second rendering device may be a local device, and the second rendering application may be a local rendering application. That is, when the rendering performance of the cloud rendering application does not meet the rendering requirements, the cloud rendering application is switched to the local rendering application.

[0072] The hybrid rendering method provided in this embodiment proposes a hybrid rendering system including multiple rendering devices, wherein the multiple rendering devices include at least a local device and a cloud rendering server. For a first rendering device and a second rendering device among the multiple rendering devices, the first rendering device monitors the rendering performance of a first rendering application that is rendering a target task. When the rendering performance of the first rendering application does not meet the rendering requirements, the second rendering device loads a target rendering model corresponding to the target task from the cloud storage, and the second rendering device transmits a video stream rendered by the target rendering model to the local device. The local device uses the second rendering application to display the rendered video stream, so as to switch from the first rendering application to the second rendering application, thereby ensuring the rendering performance through dynamic switching of the rendering applications.

[0073] In this embodiment, a hybrid rendering method is provided, which can be used in a hybrid rendering system. The hybrid rendering system includes multiple rendering devices, and the multiple rendering devices include at least a local device and a cloud rendering server. Figure 2 is a flow chart of a hybrid rendering method according to an embodiment of the present disclosure. Figure 2 As shown, the process includes the following steps:

[0074] Step S201 : for a first rendering device and a second rendering device among a plurality of rendering devices, when the first rendering device renders a target task through the first rendering application, the first rendering device monitors the rendering performance of the first rendering application.

[0075] For details, please see Figure 1 Step S101 of the illustrated embodiment will not be described in detail here.

[0076] Step S202: When the rendering performance of the first rendering application does not meet the rendering requirement, the first rendering device instructs the first rendering application to start the second rendering application, and provides the cloud storage address of the target rendering model to the second rendering device.

[0077] In this embodiment, when the first rendering device monitors that the rendering performance of the first rendering application does not meet the rendering requirements, the first rendering device instructs the first rendering application to start the second rendering application, and the first rendering application provides the cloud storage address of the target rendering model to the second rendering application, and the second rendering application provides the cloud storage address of the target rendering model to the second rendering device.

[0078] The cloud storage address may be a Universal Resource Locator (URL).

[0079] Step S203: The second rendering device loads the target rendering model from the cloud storage based on the cloud storage address.

[0080] In this implementation, the second rendering device loads the target rendering model information from the cloud storage based on the acquired cloud storage address.

[0081] It is understandable that the cloud storage stores rendering models used by various rendering applications, and different types of data of rendering models can be stored separately. For example: when using a local rendering application, you can choose to load the rendering model of the corresponding user in the cloud storage, or load the local model. If you load the local model, the local rendering application will synchronize the model to the cloud storage. The next time the user uses the rendering model, he can directly choose to load the rendering model in his own cloud storage without loading the local model. The previously used model can be loaded anytime and anywhere; when the user makes changes to the rendering model, the local rendering application will record this information and store it in the cloud storage, so that the user can continue the previous changes when using the rendering model next time. When the user uses the cloud rendering application to modify the model, the cloud rendering application will also store the corresponding modification information in the cloud storage. The modification information includes geometric information, material information, camera, lighting, etc. Among them, the data of the rendering model in the cloud storage can be packaged in a binary file. In order to improve the read and write efficiency of cloud storage, geometric information, material information, camera, lighting, etc. can be stored separately. When the user modifies one of the data, it does not need to update other information, only the modified information.

[0082] Step S204: The second rendering device uses the target rendering model to perform rendering processing to obtain a rendered video stream, where the target rendering model is a rendering model corresponding to the target task, and the rendered video stream is used for display by the local device through the second rendering application.

[0083] For details, please see Figure 1 Step S102 of the illustrated embodiment will not be described in detail here.

[0084] In some optional embodiments, after step S204, the local device displays the video stream rendered and processed by the second rendering device through the second rendering application, including: the local device sets the play tag of the second rendering application to be visible, and sets the play tag of the first rendering application to be hidden; the local device plays the rendered video stream corresponding to the second rendering application based on the play tag, and the rendered video stream is displayed over the display area of ​​the first rendering application.

[0085] The rendered image can be transmitted in the form of a video stream (supporting H.264 encoding, and also supporting VP8 and VP9), and played using a multimedia player in the HTML5 standard. The playback source of the multimedia player is set using a playback tag (ie, a video tag).

[0086] In this embodiment, after the local device receives the rendered video stream, based on the attribute setting of the play tag, the rendered video stream corresponding to the second rendering application is played, and overlays the display area of ​​the first rendering application, thereby achieving an imperceptible switching effect when switching between the two rendering applications.

[0087] The play tag can be controlled by JavaScript. For example, JavaScript is used to set the play tag of the second rendering application to hidden=false to mark the second rendering application as visible, and JavaScript is used to set the play tag of the first rendering application to hidden=true to mark the first rendering application as hidden.

[0088] The display area covering the original first rendering application can be controlled by JavaScript. For example, JavaScript is used to set the canvas tag (ie, canvas tag) of the first rendering application to hidden=false to cover the display area of ​​the original first rendering application.

[0089] In some optional implementations, after step S204, the method further includes: the first rendering device destroying rendering-related resources of the first rendering application.

[0090] In this implementation, after the switching of the rendering application is completed, the rendering-related resources of the first rendering application that were originally used are destroyed in time, thereby avoiding waste of resources.

[0091] The rendering-related resources may include at least one of the following types of information: rendering model (geometric information, material information, camera, light), texture used for rendering, buffer, frame buffer, rendering buffer, shader, and assembly.

[0092] In the hybrid rendering method provided in this embodiment, when the rendering performance of the first rendering application does not meet the rendering requirements, the first rendering device instructs the first rendering application to start the second rendering application, and provides the cloud storage address of the target rendering model to the second rendering device. The second rendering device loads the information of the target rendering model from the cloud storage based on the acquired cloud storage address, thereby helping the second rendering device to accurately load the target rendering model.

[0093] In this embodiment, a hybrid rendering method is provided, which can be used in a hybrid rendering system. The hybrid rendering system includes multiple rendering devices and a management service. The multiple rendering devices include at least a local device and a cloud rendering server. The management service is used to manage the multiple cloud rendering servers. Each cloud rendering server includes at least one cloud rendering program. In the case where the first rendering device is a local device and the second rendering device is a cloud rendering server, Figure 3As shown, the process also includes the following steps:

[0094] Step S301: The management device determines the required scale of the cloud rendering program corresponding to the cloud rendering application based on the size of the target rendering model.

[0095] In this embodiment, after startup, the cloud rendering application communicates with the management service and applies for cloud rendering resources. The application carries indication information of the target rendering model. The management device determines the demand scale of the cloud rendering program corresponding to the cloud rendering application based on the size of the target rendering model.

[0096] Step S302: The management device allocates cloud rendering programs corresponding to the demand scale to cloud rendering applications among multiple cloud rendering servers managed by the management device.

[0097] In this embodiment, the management device allocates cloud rendering programs corresponding to the demand scale to the cloud rendering application among the multiple cloud rendering servers managed based on the allocation algorithm, and the cloud rendering application uses multiple cloud rendering programs for collaborative rendering.

[0098] For example, when multiple cloud rendering programs work together to render, the rendering target is to output an image with a resolution of W x H, and each cloud rendering program is responsible for outputting a portion of it. Figure 4 As shown in the figure, four cloud rendering programs simultaneously render at a resolution of 1920x 1080. No. 1 is responsible for rendering the upper left corner, No. 2 is responsible for rendering the upper right corner, No. 3 is responsible for rendering the lower left corner, and No. 4 is responsible for rendering the lower right corner. The cloud rendering programs also render at a resolution of 1920x 1080, but the pixels actually processed are those in the area they are responsible for, and pixels in other areas are not processed. The image is then cut into 960x 540 and encoded as a video stream to be transmitted to the local device. The local device plays the received video on the designated player according to the position, and splices the four designated players into one picture. The four cloud rendering programs synchronize the frame rate through network communication to ensure that the spliced ​​picture is smoother.

[0099] In some optional implementations, step S302 includes: the management device obtains the load status of each cloud rendering program; the management device selects, from multiple cloud rendering servers managed by the management device, a cloud rendering program corresponding to the demand scale of the cloud rendering application in order of load from low to high.

[0100] In this implementation, the management device preferentially selects cloud rendering programs with low loads to allocate to the cloud rendering application based on the load conditions of each cloud rendering program, thereby achieving reasonable allocation of resources.

[0101] Among them, the load status of each cloud rendering program can be calculated based on the weighted calculation of multiple data such as GPU utilization, video memory occupancy, memory utilization, disk utilization, network transmission utilization, etc.

[0102] Among them, the allocated cloud rendering program will become in use. Once the user has finished using it, the cloud rendering program will be marked as idle. The management service can allocate the program to other users. The user only occupies the cloud rendering program when using it. After use, the cloud rendering program will be recycled and the management service can allocate the cloud rendering program to other users.

[0103] In some optional implementations, after step S302, the method further includes: the management device receiving permission sharing indication information fed back by the cloud rendering application; when the permission sharing indication information indicates that the cloud rendering application supports permission sharing, the management device assigns the cloud rendering program assigned to the cloud rendering application to other cloud rendering applications in other local devices.

[0104] In this embodiment, when the cloud rendering application indicates through permission sharing indication information that the resources it applies for can be shared, the management device allocates the cloud rendering program allocated to the cloud rendering application to other cloud rendering applications in other local devices, so that multiple local devices can use one cloud rendering program at the same time, thereby realizing resource sharing of the cloud rendering program.

[0105] Among them, each cloud rendering program itself has corresponding cloud rendering program identification information. After the cloud rendering application applies for resources, the management service will return this identification information to the cloud rendering application. The cloud rendering application will mark this identification information on the local device side, and can generate and feedback permission sharing indication information.

[0106] Among them, permission sharing can be set to read-only, which means you can only see the rendering results, or set to read-write, which means you can see the rendering results, operate and modify them.

[0107] Among them, the cloud rendering program allocated to the user will become the user's resource during use, and other shared users can view the results of the cloud rendering program together. Once the user closes the cloud rendering application, the cloud rendering application will notify the management service to release the corresponding cloud rendering program. This cloud rendering program will be marked as unshareable. The management service will mark the cloud rendering program as idle and notify the cloud rendering program to stop rendering, releasing the occupied video memory, memory and other resources.

[0108] In some optional implementations, the management device allocates the cloud rendering program allocated to the cloud rendering application to other cloud rendering applications in other local devices corresponding to the cloud rendering application identification information based on the cloud rendering application identification information indicated in the permission sharing indication information; or, the management device receives a resource usage request sent by other cloud rendering applications in other local devices, the cloud rendering program requested in the resource usage request is the cloud rendering program allocated to the cloud rendering application, and in response to the resource usage request, allocates the cloud rendering program allocated to the cloud rendering application to other cloud rendering applications in other local devices.

[0109] In this embodiment, the user sets the shared object through the cloud rendering application identification information in the cloud rendering program, and the management service allocates the shared rendering resources to the shared object corresponding to the cloud rendering application identification information, or, by default, supports sharing to all users. When other users request the shared rendering resources through other cloud rendering applications in other local devices, the shared resources are allocated to the requester, thereby effectively setting the shared objects for multiple users to share the rendering resources.

[0110] In the hybrid rendering method provided in this embodiment, when switching from a local rendering application to a cloud rendering application, the management device determines the demand scale of the cloud rendering program corresponding to the cloud rendering application based on the size of the target rendering model, and the management device allocates the cloud rendering program corresponding to the demand scale to the cloud rendering application among multiple cloud rendering servers managed by the management device, thereby reasonably allocating rendering resources and ensuring the subsequent rendering effect of the cloud rendering application.

[0111] Next, combine Figure 5 The structural schematic diagram of the hybrid rendering system shown in FIG. 1 is an exemplary illustration of the above technical solution.

[0112] The "local rendering application" renders the model and responds to user operations based on WebGL, WebAssembly, or WebGPU; it counts the rendering time of each frame. When the rendering time exceeds the threshold, the bottleneck duration statistics are triggered. If the rendering time of a certain frame is lower than the threshold, the bottleneck duration statistics are cleared. Otherwise, the bottleneck duration is continuously counted. When the bottleneck duration reaches the threshold, it switches to cloud rendering and starts the "cloud rendering application" to continue rendering.

[0113] The "cloud rendering application" is based on cloud rendering. It obtains the rendered image output by the "cloud rendering program" through network communication. The image is transmitted in the form of a video stream. The video stream is played using the play tag on the web page. When the video can start playing, the play tag is displayed, the original "local rendering application" is blocked, and the "local rendering application" is destroyed; the user's keyboard and mouse operations are captured through JavaScript, the operations are encapsulated into message packets and transmitted to the "cloud rendering program" through network communication; there are relevant controls on the page that can support calling the application programming interface (Application Programming Interface, API) corresponding to the "cloud rendering program", such as obtaining camera properties, switching skyboxes, loading terrain, switching weather, adjusting resolution, etc. The API call is encapsulated into a JSON message and transmitted to the "cloud rendering program" through the network. There will be a unique identifier inside the API call message. After the "cloud rendering program" responds to the API call, the unique identifier is added to the response message, and then returned to the "cloud rendering application" through the network. The "cloud rendering application" can obtain the corresponding response result through the unique identifier.

[0114] The "Management Service" is responsible for managing the "Cloud Rendering Server". Multiple "Cloud Rendering Servers" can be built to form a cluster. The "Sentinel Service" will report the health status of the "Cloud Rendering Server" (CPU, memory, GPU, disk, network and other resource usage) to the "Management Service". The "Sentinel Service" will continue to send heartbeats to the "Management Service" to inform the "Sentinel Service" of its own health status. It is responsible for responding to the resource allocation of the "Cloud Rendering Application" and dynamically allocating resources based on the collected resource consumption, giving priority to allocating rendering tasks to "Cloud Rendering Servers" with lower CPU, memory, GPU and other resource usage. In case of the need to render a very large model, multiple "Cloud Rendering Servers" will be allocated according to the model level.

[0115] A “cloud rendering server” can be a virtualized machine or a bare metal machine as a rendering resource.

[0116] The "Sentinel Service" is responsible for monitoring the "Cloud Rendering Server" and "Cloud Rendering Program"; regularly reports the overall resource consumption of the "Cloud Rendering Server" and the resource consumption of a single Cloud Rendering Program (CPU, memory, GPU, disk, network, etc.) to the "Management Service"; is responsible for starting, shutting down, and restarting the "Cloud Rendering Program", and monitoring the process status of the "Cloud Rendering Program", and reporting to the "Management Service" if the process exits; when the "Cloud Rendering Program" exits abnormally, the process can be automatically started according to user requirements to ensure continuous use by users; continuously sends heartbeats to the "Management Service" to facilitate the "Management Service" to determine the health status of the "Sentinel Service".

[0117] As a rendering engine, the "cloud rendering program" is responsible for real-time rendering models, responding to various user operations, and encoding rendered images into video streams, which are transmitted to the "cloud rendering application" through network communication. Video transmission is based on UDP and various algorithms are introduced, such as congestion control (Google Congestion Control, GCC) and forward error correction (Forward Error Correction, FEC) to ensure that it can adapt to more complex network environments. Encoding supports CPU soft coding and GPU hard coding.

[0118] Based on the above hybrid rendering system, in one example, when the cloud rendering program detects that the rendering performance reaches a threshold, it instructs the cloud rendering application to switch to the local rendering application; the switch is initiated by the cloud rendering program, the cloud rendering application starts the local rendering application, and provides the information of the target rendering model currently being rendered to the local rendering application, the local rendering application loads the target rendering model, and after successful loading, marks its own play tag as visible, marks the play tag of the cloud rendering application as hidden, and destroys the cloud rendering application (calling the close or clear method of the resource in JavaScript, such as Websocket.close related to network communication). When the cloud rendering application is destroyed, the cloud rendering application notifies the management service to release the corresponding cloud rendering program resources, the management service marks the cloud rendering program as idle, and notifies the cloud rendering program to stop rendering, and releases the occupied video memory, memory and other resources.

[0119] Based on the above hybrid rendering system, in another example, when the local device detects that the rendering performance reaches a threshold, it instructs the local rendering application to switch to the cloud rendering application; the cloud rendering application communicates with the management service and applies for cloud rendering resources; after the cloud rendering application successfully obtains the resources, it provides the information of the currently rendered target rendering model to the cloud rendering program through network transmission: the cloud storage address URI of the target rendering model is provided to the cloud rendering program, and the cloud rendering program loads the corresponding model according to the URI; after the cloud rendering program loads the target rendering model and renders the image, it encodes the image into a video stream and transmits it to the local device through the network; after receiving the video stream, the local device plays it through the play tag, and displays the play tag, covering the original display area of ​​the local rendering application, and then destroys the rendering-related resources of the local rendering application.

[0120] In the above example, multiple local devices can simultaneously use the results of a cloud rendering program to achieve a many-to-one effect, with multiple people sharing rendering resources; when encountering a large rendering model, the management service allocates multiple cloud rendering servers to the cloud rendering application for rendering, achieving a one-to-many effect.

[0121] It can be understood that the above method embodiments can be implemented individually or in combination, and the present disclosure is not limited to this.

[0122] In this embodiment, a hybrid rendering system is also provided. The system is used to implement the above embodiments and preferred implementation modes, and the details that have been described will not be repeated here.

[0123] This embodiment provides a hybrid rendering system, which includes multiple rendering devices, and the multiple rendering devices include at least a local device and a cloud rendering server, such as Figure 6 As shown:

[0124] For a first rendering device and a second rendering device among the multiple rendering devices, the first rendering device 601 is configured to monitor the rendering performance of the first rendering application when rendering a target task through the first rendering application;

[0125] The second rendering device 602 is used to load the target rendering model from the cloud storage and use the target rendering model to perform rendering processing to obtain a rendered video stream when the rendering performance of the first rendering application does not meet the rendering requirements. The target rendering model is a rendering model corresponding to the target task, and the rendered video stream is used for display by the local device through the second rendering application.

[0126] In some optional implementations, the first rendering device 601 is configured to instruct the first rendering application to start the second rendering application when the rendering performance of the first rendering application does not meet the rendering requirement, and provide the cloud storage address of the target rendering model to the second rendering device;

[0127] The second rendering device 602 is used to load the target rendering model from the cloud storage based on the cloud storage address.

[0128] In some optional implementations, the hybrid rendering system further includes: a management service, the management service being used to manage a plurality of cloud rendering servers, each cloud rendering server including at least one cloud rendering program;

[0129] When the first rendering device is a local device and the second rendering device is a cloud rendering server, the management device is used to determine the demand scale of the cloud rendering program corresponding to the cloud rendering application based on the size of the target rendering model; and allocate the cloud rendering program corresponding to the demand scale to the cloud rendering application among the multiple cloud rendering servers managed.

[0130] In some optional implementations, the management device is used to obtain the load status of each cloud rendering program; among the multiple cloud rendering servers managed, the cloud rendering program corresponding to the demand scale of the cloud rendering application is selected in order of load from low to high.

[0131] In some optional implementations, the management device is configured to receive permission sharing indication information fed back by the cloud rendering application; when the permission sharing indication information indicates that the cloud rendering application supports permission sharing, the cloud rendering program allocated to the cloud rendering application is allocated to other cloud rendering applications in other local devices.

[0132] In some optional implementations, the management device is configured to allocate the cloud rendering program allocated to the cloud rendering application to other cloud rendering applications corresponding to the cloud rendering application identification information based on the cloud rendering application identification information indicated in the permission sharing indication information; receive a resource usage request sent by other cloud rendering applications, the cloud rendering program requested in the resource usage request being the cloud rendering program allocated to the cloud rendering application, and allocate the cloud rendering program allocated to the cloud rendering application to other cloud rendering applications in response to the resource usage request.

[0133] In some optional implementations, after the local device displays the video stream rendered and processed by the second rendering device through the second rendering application, the first rendering device 601 is configured to destroy rendering-related resources of the first rendering application.

[0134] In some optional embodiments, the local device is used to set the play tag of the second rendering application to be visible and the play tag of the first rendering application to be hidden; the rendered video stream corresponding to the second rendering application is played based on the play tag, and the rendered video stream is displayed over the display area of ​​the first rendering application.

[0135] The further functional description of each of the above modules and units is the same as that of the above corresponding embodiments and will not be repeated here.

[0136] See also Figure 7 , Figure 7 is a schematic diagram of a rendering device provided by an optional embodiment of the present disclosure, such as Figure 7 As shown, the rendering device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. The various components are connected to each other using different buses for communication, and can be installed on a common motherboard or installed in other ways as needed. The processor can process instructions executed in the rendering device, including instructions stored in or on the memory to display graphical information of the GUI on an external input / output device (such as a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories.

[0137] The processor 10 may be a central processing unit, a network processor or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic or any combination thereof.

[0138] The memory 20 stores instructions executable by at least one processor 10, so that the at least one processor 10 executes the method shown in the above embodiment.

[0139] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created by the use of a rendering device for the presentation of a small program landing page, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely arranged relative to the processor 10, and these remote memories may be connected to the rendering device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0140] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive; the memory 20 may also include a combination of the above types of memory.

[0141] The rendering device further comprises a communication interface 30 for the rendering device to communicate with other devices or a communication network.

[0142] The embodiments of the present disclosure also provide a computer-readable storage medium. The above-mentioned method according to the embodiments of the present disclosure can be implemented in hardware, firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium and downloaded through a network, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state drive, etc.; further, the storage medium can also include a combination of the above-mentioned types of memory. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor, or hardware, the method shown in the above embodiment is implemented.

[0143] Although the embodiments of the present disclosure have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present disclosure, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A hybrid rendering method, characterized in that: The method is applied to a hybrid rendering system, wherein the hybrid rendering system includes a plurality of rendering devices, wherein the plurality of rendering devices include at least a local device and a cloud rendering server, and the method includes: For a first rendering device and a second rendering device among the plurality of rendering devices, the first rendering device monitors rendering performance of the first rendering application when rendering a target task through the first rendering application; When the rendering performance of the first rendering application does not meet the rendering requirements, the second rendering device loads the target rendering model from the cloud storage, and uses the target rendering model to perform rendering processing to obtain a rendered video stream, where the target rendering model is the rendering model corresponding to the target task, and the rendered video stream is used for display by the local device through the second rendering application.

2. The method according to claim 1, characterized in that When the rendering performance of the first rendering application does not meet the rendering requirement, the second rendering device loads a target rendering model from the cloud storage, including: When the rendering performance of the first rendering application does not meet the rendering requirement, the first rendering device instructs the first rendering application to start the second rendering application, and provides the cloud storage address of the target rendering model to the second rendering device; The second rendering device loads the target rendering model from the cloud storage based on the cloud storage address.

3. The method according to claim 1, characterized in that The hybrid rendering system further includes: a management service, the management service being used to manage a plurality of cloud rendering servers, each cloud rendering server including at least one cloud rendering program; In a case where the first rendering device is the local device and the second rendering device is the cloud rendering server, the method further includes: The management device determines, based on the size of the target rendering model, a demand scale of a cloud rendering program corresponding to the cloud rendering application; The management device allocates the cloud rendering program corresponding to the demand scale to the cloud rendering application among the multiple cloud rendering servers managed by the management device.

4. The method according to claim 3, characterized in that The management device allocates the cloud rendering program corresponding to the demand scale to the cloud rendering application among the multiple cloud rendering servers managed by the management device, including: The management device obtains the load status of each cloud rendering program; The management device selects, from among the multiple cloud rendering servers managed by the management device, a cloud rendering program corresponding to the demand scale allocated to the cloud rendering application in order of load from low to high.

5. The method according to claim 3, characterized in that: After the management device allocates the cloud rendering program corresponding to the demand scale to the cloud rendering application among the multiple cloud rendering servers managed by the management device, the method further includes: The management device receives permission sharing indication information fed back by the cloud rendering application; In a case where the permission sharing indication information indicates that the cloud rendering application supports permission sharing, the management device distributes the cloud rendering program allocated to the cloud rendering application to other cloud rendering applications in other local devices.

6. The method according to claim 5, characterized in that The management device distributes the cloud rendering program allocated to the cloud rendering application to other cloud rendering applications in other local devices, including: The management device allocates the cloud rendering program allocated to the cloud rendering application to other cloud rendering applications corresponding to the cloud rendering application identification information based on the cloud rendering application identification information indicated in the permission sharing indication information; or, The management device receives a resource usage request sent by another cloud rendering application, wherein the cloud rendering program requested in the resource usage request is the cloud rendering program assigned to the cloud rendering application, and in response to the resource usage request, assigns the cloud rendering program assigned to the cloud rendering application to the other cloud rendering application.

7. The method according to claim 1, characterized in that After the local device displays the video stream rendered by the second rendering device through the second rendering application, the method further includes: The first rendering device destroys rendering-related resources of the first rendering application.

8. The method according to claim 1, characterized in that: The local device displays the video stream rendered by the second rendering device through the second rendering application, including: The local device sets the play tag of the second rendering application to be visible, and sets the play tag of the first rendering application to be hidden; The local device plays the rendered video stream corresponding to the second rendering application based on the play tag, and the rendered video stream is displayed over the display area of ​​the first rendering application.

9. A hybrid rendering system, characterized in that: The hybrid rendering system includes a plurality of rendering devices, and the plurality of rendering devices include at least a local device and a cloud rendering server; For a first rendering device and a second rendering device among the plurality of rendering devices, the first rendering device is configured to monitor rendering performance of the first rendering application when rendering a target task through the first rendering application; The second rendering device is used to load a target rendering model from cloud storage and use the target rendering model to perform rendering processing to obtain a rendered video stream when the rendering performance of the first rendering application does not meet the rendering requirements. The target rendering model is a rendering model corresponding to the target task, and the rendered video stream is used for display by the local device through the second rendering application.

10. A rendering device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the hybrid rendering method according to any one of claims 1 to 8 by executing the computer instructions.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the hybrid rendering method according to any one of claims 1 to 8.

Images