Digital twin application off-screen rendering method and device

Abstract

The invention relates to the crossing field of digital twinning and cloud native technologies, in particular to a digital twinning application off-screen rendering method and device, a front end submits a rendering request through a browser or an application, receives a rendering result through a pixel stream, distributes a task to an application instance, and displays the task on the application instance. After off-screen rendering is completed, a result is transmitted to a pixel stream, the pixel stream converts a rendering result into a video stream through a WebRTC technology, the video stream is transmitted to a front end in real time, a user checks the rendering result in a browser or application in real time, the user checks the rendering result at the front end and interacts through the pixel stream, and the interaction returns to a cloud end through the pixel stream and is further processed. Compared with the prior art, the parallel processing and scheduling of the off-screen rendering task can be efficiently carried out in the cloud environment, and flexible resource allocation and automatic management capabilities are provided, so that a complex digital twin application scene is supported.

Description

Technical Field

[0001] The present invention relates to the intersection of digital twin and cloud native technologies, and specifically provides a method and device for off-screen rendering of a digital twin application. Background Art

[0002] Cloud rendering technology: Current cloud rendering technology has been widely used in games, film and television production, architectural design and other fields. Through the powerful computing power of cloud computing, users can delegate rendering tasks to remote servers to avoid local hardware limitations. However, traditional cloud rendering solutions mainly face the following shortcomings:

[0003] Inflexible resource utilization: Traditional cloud rendering architectures are often based on static resource allocation and cannot dynamically adjust resources according to real-time needs. When the task load changes, it may cause resource waste or processing delays.

[0004] Difficulty in interaction: Many cloud rendering solutions are mainly for offline rendering. Users can only receive static images or video streams after rendering, but cannot perform real-time interactive operations during the rendering process, which limits its application in fields such as virtual reality and interactive design. Real-time cloud rendering solutions can provide powerful real-time rendering capabilities, but due to network transmission delays and the complexity of rendering content, users often experience high latency on terminal devices, especially in scenarios that require real-time interaction.

[0005] Off-screen rendering technology: Off-screen rendering is to store the rendering data in the background buffer without directly presenting the rendering results. This technology has been applied in many scenarios, such as image processing, rendering farms, and pre-calculation. However, traditional off-screen rendering has the following limitations:

[0006] Data transmission bottleneck: Off-screen rendering generates a large amount of data, and how to efficiently transmit these rendering results to the front end is a difficult problem. In most cases, off-screen rendering results require a large amount of data copying and transmission, which affects the overall efficiency.

[0007] Poor interactivity: Most of the current off-screen rendering solutions are usually used for static tasks or pre-calculated scenes and cannot meet the application scenarios that require real-time user interaction.

[0008] Single camera limitation: The digital twin engine usually uses a single main camera in the scene as the image rendering target. The engine's default offline rendering solution also collects the image capture results of the main camera. It is impossible to simultaneously collect images from multiple cameras in the scene for offline rendering.

[0009] Pixel streaming technology: Pixel streaming technology solves the problem of insufficient local rendering performance by transmitting cloud rendering results to the user end in the form of video streams. It reduces transmission delays and ensures high-quality rendering effects through the WebRTC protocol. However, the current pixel streaming technology also faces some problems:

[0010] High bandwidth requirements: Pixel streaming technology has high requirements for network bandwidth. In particular, when transmitting high-definition content, insufficient network bandwidth will cause problems such as blurred images and frame rate jitter, which will have a greater impact on users with poor network conditions.

[0011] Computing resource overhead: In large-scale concurrent scenarios, the pixel stream rendering end has a large computing overhead, which reduces the overall efficiency of the system and increases the maintenance cost of the rendering server. If a single machine is deployed, the single machine configuration is required to be high, with a large deployment cost, which indirectly increases the project cost.

[0012] Traditional rendering technology has performance bottlenecks, low resource utilization efficiency, and network latency in digital twin scenes. Traditional rendering technologies mostly rely on local computing resources. With the development of digital twin technology, the complexity and real-time requirements of scenes are gradually increasing. Traditional rendering modes are difficult to meet the needs of large-scale, high-concurrency, and complex scenes. Summary of the invention

[0013] In view of the deficiencies of the above-mentioned prior art, the present invention provides a highly practical off-screen rendering method for digital twin applications.

[0014] A further technical task of the present invention is to provide a reasonably designed, safe and applicable off-screen rendering device for digital twin applications.

[0015] The technical solution adopted by the present invention to solve its technical problem is:

[0016] A method for off-screen rendering of a digital twin application, wherein the front end submits a rendering request through a browser or application, receives the rendering result through a pixel stream, distributes the task to an application instance, and after the instance completes the off-screen rendering, transmits the result to the pixel stream. The pixel stream converts the rendering result into a video stream through WebRTC technology and transmits it to the front end in real time. The user views it in real time in the browser or application. The user views the rendering result on the front end and interacts through the pixel stream. The interaction is transmitted back to the cloud through the pixel stream for further processing.

[0017] Further, the steps are as follows:

[0018] S1. Check the rendering node status;

[0019] S2, select the optimal node;

[0020] S3, start the process in the node;

[0021] S4, find the process with the minimum load;

[0022] S5, receiving front-end operation;

[0023] S6. Operation feedback.

[0024] Further, in step S1, the user initiates a connection request through the client device, hoping to access the cloud service, and the cloud native system receives the user's connection request, starts processing, checks the status of the currently available rendering nodes, and determines whether there are idle or low-load nodes;

[0025] If there are online rendering nodes, the system will dynamically adjust resource allocation according to the current load. If there are no available rendering nodes or the existing node load is too high, the system will start a new rendering node to meet the demand;

[0026] In step S2, the cloud native system evaluates the load of all online nodes and selects a node with the lowest load to process the user's request.

[0027] Further, in step S3, on the selected node, the system starts the scene instance process, ready to receive and process the user's operation, the started process is registered in the system, and checks whether there is already a started process on the selected node. If one-to-one mapping is required, the system will start a new digital twin application instance;

[0028] In step S4, if a new application needs to be started, the system will look for a process with the smallest current load to start a new instance.

[0029] Furthermore, in step S5, the user performs operations on the front end, and the front end encapsulates these operations into requests and sends them to the application instance of the back end, and matches the operation request of the front end to the corresponding application instance. After receiving the operation request, the application instance performs the corresponding rendering and processing tasks. After the processing is completed, the application instance transmits the results back to the front end through the data stream;

[0030] In step S6, after receiving the data stream, the front end presents the operation results to the user. Depending on whether the user continues the operation, the system decides whether to continue processing new requests or release resources. If the application instance or process has no new operation requests within a certain period of time, the system will release these resources.

[0031] Furthermore, when performing off-screen rendering, the following steps are performed:

[0032] (1) The user initiates a screen request to access and start the application instance;

[0033] (2) Off-screen rendering is started to obtain GPU shared texture;

[0034] (3) Establish a P2P connection and receive the image.

[0035] Furthermore, in step (1), the user initiates a request for the screen content through a client or interface. After receiving the user request, the cloud native system first checks the current available resource status. If the current resources are insufficient to process the request, the system will perform resource scheduling, that is, add new rendering resources. After the resources are ready, the application instance is started to process the scene rendering task.

[0036] Furthermore, in step (2), the application instance starts to perform off-screen rendering operations, that is, rendering is performed without being displayed on the user's screen. The application instance loads the scene to be rendered and captures multiple rendering targets in real time through the configured camera. During the rendering process, the application instance obtains the texture resources shared by the GPU and copies the rendering targets to the GPU.

[0037] Furthermore, in step (3), after the rendering is completed, the cloud native system attempts to establish a point-to-point P2P connection. If the P2P connection fails to be established, the transit server intervenes, and the cloud native system schedules and pulls up the resources of the transit server. The image rendered by the application instance is finally sent to the user interface through the transit server.

[0038] The rendered image data is encoded and transmitted using WebRTC technology. After the image transmission connection is successfully established, the system starts sending the encoded image data. After the user end receives the image data, it decodes and displays it.

[0039] A digital twin application off-screen rendering device, comprising: at least one memory and at least one processor;

[0040] The at least one memory is used to store a machine-readable program;

[0041] The at least one processor is used to call the machine-readable program to execute a method for off-screen rendering of a digital twin application.

[0042] Compared with the prior art, the off-screen rendering method and device of a digital twin application of the present invention has the following outstanding beneficial effects:

[0043] (1) Improved rendering efficiency: Through parallel computing and flexible resource scheduling, the rendering speed of complex scenes is greatly improved, especially in multi-view and multi-task scenes, which can effectively reduce the rendering time.

[0044] (2) Improved resource utilization: The dynamic resource allocation mechanism under the cloud native architecture avoids the waste of traditional static resource allocation and ensures flexible adjustment when resource demand changes.

[0045] (3) Enhanced system stability: Through the cloud's automatic expansion and fault recovery mechanism, the system can maintain stable operation under high-load environments, ensuring the continuity and reliability of rendering tasks.

[0046] (4) Expansion of application scenarios: This technology can be widely used in digital twins, virtual simulation, industrial manufacturing, film and television rendering and other fields, and has broad applicability and prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0048] Attached Figure 1 is a flowchart of an off-screen rendering method for a digital twin application;

[0049] Attached Figure 2 It is a schematic diagram of a user access process in an off-screen rendering method of a digital twin application;

[0050] Attached Figure 3 It is a swimlane diagram of off-screen rendering in the off-screen rendering method of a digital twin application. DETAILED DESCRIPTION

[0051] In order to enable those skilled in the art to better understand the solution of the present invention, the present invention is further described in detail below in conjunction with specific implementation methods. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0052] A best embodiment is given below:

[0053] like Figure 1 As shown, in this embodiment, a method for off-screen rendering of a digital twin application is used. The front end submits a rendering request through a browser or application, and receives the rendering result through a pixel stream. The cloud scheduling module distributes the task to the application instance, and the instance transmits the result to the pixel stream module after completing the off-screen rendering. The pixel stream module converts the rendering result into a video stream through WebRTC technology, and transmits it to the front end in real time, and the user views it in real time in the browser or application. The user views the rendering results on the front end and interacts through the pixel stream (such as controlling the camera angle, triggering events, etc.), which are transmitted back to the cloud through the pixel stream for further processing.

[0054] like Figure 2 As shown, the steps are as follows:

[0055] S1. User connection: The user initiates a connection request through a client device and hopes to access the cloud service.

[0056] S2. Cloud native system response: The cloud native system receives the user's connection request and starts processing.

[0057] S3. Check the status of rendering nodes: The system checks the status of currently available rendering nodes to determine whether there are idle or low-load nodes.

[0058] S4. Dynamic scheduling of rendering resources: If there are online rendering nodes, the system will dynamically adjust resource allocation according to the current load. If there are no available rendering nodes or the existing node load is too high, the system will start a new rendering node to meet the demand.

[0059] S5. Select the optimal node: The system evaluates the load of all online nodes and selects a node with the lowest load to process user requests to ensure the response speed and stability of the service.

[0060] S6. Start the process within the node: On the selected node, the system starts the scene instance process and prepares to receive and process user operations.

[0061] S7. Register instance: The started process is registered with the system so that the system can manage and schedule it.

[0062] S8, Checking started processes: The system checks whether there is already a started process on the selected node. If a one-to-one mapping is required, the system will start a new digital twin application instance.

[0063] S9. Find the process with the smallest load: If a new application needs to be started, the system will look for the process with the smallest current load to start a new instance.

[0064] S10, receiving front-end operations: The user performs operations on the front-end, such as clicking, inputting, etc. The front-end encapsulates these operations into requests and sends them to the back-end application instance. After receiving the operation request, the application instance performs the corresponding rendering and processing tasks. After the processing is completed, the application instance transmits the results back to the front-end through the data stream.

[0065] S11, Operation feedback: After receiving the data stream, the front end presents the operation results to the user, such as rendered images or processed data. Depending on whether the user continues to operate, the system decides whether to continue to process new requests or release resources. If the application instance or process has no new operation requests within a certain period of time, the system will release these resources to optimize resource utilization and cost.

[0066] like Figure 3As shown, when performing off-screen rendering, there are the following steps:

[0067] (1) The user initiates a screen request to access and start the application instance;

[0068] The user initiates a request for the screen content through a client or interface. After receiving the user's request, the cloud native system first checks the current available resource status. If the current resources are insufficient to process the request, the system will schedule resources, which may include dynamic expansion, that is, adding new rendering resources. After the resources are ready, the application instance is started to process the scene rendering task.

[0069] (2) Off-screen rendering is started to obtain GPU shared texture;

[0070] The application instance starts to perform off-screen rendering operations, that is, rendering without displaying on the user's screen. Scene loading and real-time capture RenderTarget: The application instance loads the scene to be rendered and captures multiple render targets in real time through the configured camera. During the rendering process, the application instance obtains the texture resources shared by the GPU. The render target data is copied to the GPU without image file transfer, so that image processing is more efficient.

[0071] (3) Establish a P2P connection and receive images;

[0072] After rendering is completed, the system attempts to establish a peer-to-peer (P2P) connection to transmit data directly between the user device and the rendering server. There is a possibility that the P2P connection cannot be established due to the influence of the network environment deployment. Therefore, if the P2P connection fails, the relay server will intervene, and the cloud native system will schedule and pull up the resources of the relay server. The image rendered by the application instance is finally sent to the user interface through the relay server.

[0073] The rendered image data is encoded and transmitted using WebRTC technology. After the image transmission connection is successfully established, the system starts sending the encoded image data. After the client receives the image data, it decodes and displays it.

[0074] Through the cloud native platform, off-screen rendering instances are automatically managed and scheduled. This architecture design can achieve the following goals:

[0075] ① Elastic resource scheduling: According to the complexity of the task and real-time requirements, the cloud platform can dynamically allocate resources to ensure the effective use of computing resources during the rendering peak period, and automatically recycle excess resources after the task is completed to improve resource utilization.

[0076] ② Automatic expansion and fault recovery: The cloud platform provides automatic expansion and fault recovery capabilities to ensure that the system runs stably during peak load periods and automatically recovers tasks when a node fails.

[0077] The scene image data generated by the off-screen rendering task is directly acquired by the GPU through a shared texture, and finally packaged into WebRTC, and the user connects to the pixel stream at the front end to obtain real-time image data. The present invention uses GPU shared textures to ensure the efficiency of data transmission, reduce the number of times rendering data is transferred between the CPU and the GPU, directly capture the scene image through the rendering target camera and render it into the GPU, and the image data is synchronously stored in the shared memory of the GPU. Subsequent operations directly obtain the content of the shared memory for modification or compression, and finally simplify the image data processing pipeline, ensuring that the off-screen rendering results can be quickly delivered to the video frame of WebRTC and delivered to the client.

[0078] The management module of the system of the present invention provides management of functions such as resource allocation, task scheduling, and instance monitoring, so that rendering tasks can be flexibly configured and expanded, specifically including:

[0079] ① Task priority scheduling: By setting the priority of tasks, ensure that critical tasks receive priority resource scheduling and processing.

[0080] ②Instance monitoring and log management: By monitoring the status and resource usage of each instance, the system can make real-time adjustments and fault warnings to ensure the smooth completion of rendering tasks.

[0081] ③ Scalability and compatibility: The system supports multiple cloud platforms and can support more virtualization technologies and rendering tools through plug-in expansion.

[0082] Based on the above method, a digital twin application off-screen rendering device in this embodiment includes: at least one memory and at least one processor;

[0083] The at least one memory is used to store a machine-readable program;

[0084] The at least one processor is used to call the machine-readable program to execute a method for off-screen rendering of a digital twin application.

[0085] The above-mentioned specific implementations are only specific cases of the present invention. The patent protection scope of the present invention includes but is not limited to the above-mentioned specific implementations. Any technical solutions that conform to the above-mentioned specific implementations of the present invention and any appropriate changes or substitutions made by ordinary technicians in the relevant technical field shall fall within the patent protection scope of the present invention.

[0086] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A digital twin application off-screen rendering method, characterized in that: The front end submits a rendering request through a browser or application, receives rendering results through a pixel stream, distributes the task to an application instance, and after the instance completes off-screen rendering, it transmits the result to a pixel stream. The pixel stream converts the rendering result into a video stream through WebRTC technology and transmits it to the front end in real time. The user views it in real time in the browser or application. The user views the rendering results on the front end and interacts through the pixel stream. The interaction is transmitted back to the cloud through the pixel stream for further processing.

2. A digital twin application off-screen rendering method according to claim 1, characterized in that: The steps are as follows: S1. Check the rendering node status; S2, select the optimal node; S3, start the process in the node; S4, find the process with the minimum load; S5, receiving front-end operation; S6. Operation feedback.

3. A digital twin application off-screen rendering method according to claim 2, characterized in that: In step S1, the user initiates a connection request through a client device, hoping to access the cloud service. The cloud native system receives the user's connection request, starts processing, checks the status of the currently available rendering nodes, and determines whether there are idle or low-load nodes; If there are online rendering nodes, the system will dynamically adjust resource allocation according to the current load. If there are no available rendering nodes or the existing node load is too high, the system will start a new rendering node to meet the demand; In step S2, the cloud native system evaluates the load of all online nodes and selects a node with the lowest load to process the user's request.

4. A digital twin application off-screen rendering method according to claim 3, characterized in that: In step S3, on the selected node, the system starts the scene instance process, ready to receive and process user operations, the started process is registered in the system, and checks whether there is already a started process on the selected node. If one-to-one mapping is required, the system will start a new digital twin application instance; In step S4, if a new application needs to be started, the system will look for a process with the smallest current load to start a new instance.

5. A digital twin application off-screen rendering method according to claim 4, characterized in that: In step S5, the user performs operations on the front end, and the front end encapsulates these operations into requests and sends them to the application instance of the back end, matching the operation request of the front end to the corresponding application instance. After receiving the operation request, the application instance performs the corresponding rendering and processing tasks. After the processing is completed, the application instance transmits the results back to the front end through the data stream; In step S6, after receiving the data stream, the front end presents the operation results to the user. Depending on whether the user continues the operation, the system decides whether to continue processing new requests or release resources. If the application instance or process has no new operation requests within a certain period of time, the system will release these resources.

6. A digital twin application off-screen rendering method according to claim 5, characterized in that: When performing off-screen rendering, the following steps are involved: (1) The user initiates a screen request to access and start the application instance; (2) Off-screen rendering is started to obtain GPU shared texture; (3) Establish a P2P connection and receive images.

7. A digital twin application off-screen rendering method according to claim 6, characterized in that: In step (1), the user initiates a request for the screen content through a client or interface. After receiving the user request, the cloud native system first checks the current available resource status. If the current resources are insufficient to process the request, the system will perform resource scheduling, that is, add new rendering resources. After the resources are ready, the application instance is started to process the scene rendering task.

8. A digital twin application off-screen rendering method according to claim 7, characterized in that: In step (2), the application instance starts to perform off-screen rendering operations, that is, rendering is performed without displaying on the user's screen. The application instance loads the scene to be rendered and captures multiple rendering targets in real time through the configured camera. During the rendering process, the application instance obtains the texture resources shared by the GPU and copies the rendering targets to the GPU.

9. A digital twin application off-screen rendering method according to claim 8, characterized in that: In step (3), after rendering is completed, the cloud native system attempts to establish a point-to-point P2P connection. If the P2P connection fails, the transit server intervenes, and the cloud native system schedules and pulls up the resources of the transit server. The image rendered by the application instance is finally sent to the user interface through the transit server. The rendered image data is encoded and transmitted using WebRTC technology. After the image transmission connection is successfully established, the system starts sending the encoded image data. After the user end receives the image data, it decodes and displays it.

10. A digital twin application off-screen rendering device, characterized in that: include: at least one memory and at least one processor; The at least one memory is used to store a machine-readable program; The at least one processor is configured to call the machine-readable program to execute the method according to any one of claims 1 to 8.

Images