Lightweight three-dimensional cloud rendering method and system

Abstract

The invention relates to a lightweight three-dimensional cloud rendering method and system. The method comprises the following steps: decoupling a rendering engine and a video service; rendering the three-dimensional scene by a rendering engine at the server side to obtain a rendering result; the rendering result is output to a downstream WebSocket server; the client receives the output data from the WebSocket server; and the client displays the final result. By simplifying the three-dimensional rendering process of the server and adopting the lightweight Web server and the network protocol, the requirements on server resources and bandwidth are reduced and the display delay of the client is reduced on the premise of ensuring the rendering quality. As the server does not need to encode the picture stream into the video stream and then push the stream, the load of the server is greatly reduced; and meanwhile, the picture is segmented into tiles, so that concurrent transmission and loading of the picture stream are realized, the front-end rendering frame rate is correspondingly improved, and the picture quality is improved.

Description

Technical Field

[0001] The present invention belongs to the field of three-dimensional visualization rendering technology, and in particular to server-side rendering technology for three-dimensional visualization, namely three-dimensional cloud rendering technology, which can be used to realize lightweight three-dimensional cloud rendering of scenes including digital twin substations. Background Art

[0002] 3D visualization rendering technologies include client-side rendering and server-side rendering (i.e., cloud rendering).

[0003] Server-side rendering typically involves a rendering engine rendering a 3D scene on a high-performance server, pushing each rendered frame as a video stream to a video server. The browser then pulls the stream from the video server and displays the 3D scene by playing the video. Compared to client-side rendering, server-side rendering resolves the conflict between high-quality rendering and low-performance clients, enabling high-quality rendering on low-performance client devices.

[0004] However, during the implementation of the present invention, it was found that the following problems may exist due to the use of video encoding as the data communication method between the server and the client:

[0005] 1. In addition to rendering 3D scenes, servers also need to encode videos. Both require significant computing resources, resulting in very high server performance requirements. However, the number of clients that can be connected is relatively small. For example, Unity 3D and Unreal engines support a maximum of three clients per server, while a single Unreal server only supports dozens of clients.

[0006] 2. The use of video streaming requires a large amount of bandwidth. Since each frame is transmitted as a whole, when the network is not smooth, the playback will be stuck, which will seriously affect the rendering effect.

[0007] In summary, the current server-side rendering technology places high demands on both servers and bandwidth, and cannot meet the needs of more scenarios. Summary of the Invention

[0008] In order to solve the above problems, the present invention aims to provide a lightweight 3D cloud rendering method, which can reduce the requirements on server resources and bandwidth.

[0009] In order to achieve the above objectives, the main technical solutions provided by the present invention include:

[0010] A lightweight 3D cloud rendering method, characterized in that it comprises the following steps:

[0011] S1. Decouple the rendering engine and video service;

[0012] S2. On the server side, a rendering engine renders the three-dimensional scene to obtain a rendering result;

[0013] S3, outputs the rendering results to the downstream WebSocket server;

[0014] S4. The client receives output data from the WebSocket server.

[0015] S5. The client displays the final result.

[0016] With the help of the above technical solution, the lightweight 3D cloud rendering method of the present invention simplifies the server-side 3D rendering process and adopts a lightweight Web server and network protocol. Under the premise of ensuring rendering quality, it reduces the requirements for server resources and bandwidth and reduces the delay of client display. Since the server no longer encodes the image stream into a video stream and pushes the stream, the server load is greatly reduced; the technical solution of the present invention only involves a 3D rendering engine and a lightweight WebSocket server. Compared with the video streaming method, it is very lightweight and easy to quickly develop, deploy and implement.

[0017] In a preferred embodiment of the present invention, step S2 may further include:

[0018] Step S202: Slice the rendered image to obtain slice data. Slicing the image into tiles facilitates concurrent transmission and loading of image streams, thereby further improving the front-end rendering frame rate and image quality.

[0019] In a preferred embodiment of the present invention, the following steps may be further included before step S201:

[0020] Step S201: Determine whether the image size reaches a threshold. If the size of the rendered result is large, the image can be sliced. The sliced ​​image transmission volume is reduced and can be transmitted in parallel, greatly improving transmission efficiency and reducing network transmission delays.

[0021] In a preferred embodiment of the present invention, the following steps may be further included before step S201:

[0022] Step S201: Determine whether the network condition reaches a threshold. When the network condition is poor, slices are given priority, and the number and size of slices can be determined through actual measurement.

[0023] In a preferred embodiment of the present invention, in step S3, the sliced ​​image rendering results can be output in parallel to the downstream WebSocket server. By slicing the image into tiles and then concurrently transmitting and loading them through image streams, the front-end rendering frame rate is further increased, further enhancing image quality.

[0024] In a preferred embodiment of the present invention, in step S2, PNG image encoding is used to compress data in the color buffer to obtain an image rendering result. The image compression rate can reach 80%, thereby further reducing the size of the data transmission. Furthermore, because image compression requires very little computing resources, far less than video encoding, the method of the present invention is more lightweight, significantly reducing the server load, server resource and bandwidth requirements, and client display delays.

[0025] In one embodiment of the present invention, in step S4: the client receives the rendering result from the WebSocket server through the browser.

[0026] In one embodiment of the present invention, in step S5: the received rendering result includes compressed image data.

[0027] In one embodiment of the present invention, in step S5: the received rendering result includes compressed slice data.

[0028] One embodiment of the present invention further includes:

[0029] Step S0: Expanding the WebSocket server involves adding new servers and improving server performance based on the load of the WebSocket server. This allows, for example, when adding clients, to meet the required number of connections simply by adding new servers and improving server performance based on the load of the WebSocket server.

[0030] On the other hand, the present invention also provides a lightweight 3D cloud rendering system, which includes:

[0031] Decoupling module, used to decouple the rendering engine and video service;

[0032] The rendering module is used to render the 3D scene by the rendering engine on the server side to obtain the rendering result;

[0033] Output module, used to output rendering results to the downstream WebSocket server;

[0034] The data receiving module is used to receive output data from the WebSocket server on the client side;

[0035] The result display module is used to display the final rendering result on the client.

[0036] With the help of the above technical solution, the lightweight 3D cloud rendering system of the present invention simplifies the server-side 3D rendering process and adopts a lightweight Web server and network protocol. Under the premise of ensuring rendering quality, it reduces the requirements for server resources and bandwidth and reduces the delay of client display. Since the server no longer encodes the image stream into a video stream and pushes the stream, the server load is greatly reduced. The technical solution of the present invention only involves a 3D rendering engine and a lightweight WebSocket server. Compared with the video streaming method, it is very lightweight and easy to quickly develop, deploy and implement.

[0037] The beneficial effects of the embodiments of the present invention include:

[0038] The lightweight 3D cloud rendering method and system of the present invention simplifies the process of 3D rendering on the server side and adopts a lightweight Web server and network protocol. Under the premise of ensuring rendering quality, it reduces the requirements for server resources and bandwidth and reduces the delay of client display. Since the server side no longer encodes the image stream into a video stream before pushing the stream, the server load is greatly reduced; at the same time, the image is divided into tiles, which realizes the concurrent transmission and loading of the image stream, and the front-end rendering frame rate is correspondingly improved, and the picture quality is improved. The technical solution of the present invention only involves a 3D rendering engine and a lightweight WebSocket server. Compared with the video streaming method, it is more lightweight and easy to quickly develop, deploy and implement. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 This is a schematic diagram of the overall process of a lightweight 3D cloud rendering method according to an embodiment of the present invention;

[0040] Figure 2 This is a schematic diagram of the application process of a lightweight 3D cloud rendering method according to an embodiment of the present invention;

[0041] Figure 3 This is a schematic diagram of the overall structure of a lightweight 3D cloud rendering system that implements an embodiment of a camera-cloud-mirror linkage control method of the present invention;

[0042] Figure 4 This is a schematic diagram of the overall structure of a lightweight 3D cloud rendering system that implements an embodiment of a camera-cloud-mirror linkage control method of the present invention;

[0043] Figure 5 This is a specific implementation process of a lightweight 3D cloud rendering method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0044] In order to better explain the present invention and facilitate understanding, the present invention is described in detail below with reference to the accompanying drawings.

[0045] See also Figure 1 A lightweight 3D cloud rendering method according to an embodiment of the present invention includes the following steps:

[0046] S1. Decouple the rendering engine and video service;

[0047] S2. On the server side, a rendering engine renders the three-dimensional scene to obtain a rendering result;

[0048] S3, outputs the rendering results to the downstream WebSocket server;

[0049] S4. The client receives output data from the WebSocket server.

[0050] S5. The client displays the final result.

[0051] With the help of the above technical solution, the lightweight 3D cloud rendering method of the present invention simplifies the server-side 3D rendering process and adopts a lightweight Web server and network protocol. Under the premise of ensuring rendering quality, it reduces the requirements for server resources and bandwidth and reduces the delay of client display. Since the server no longer encodes the image stream into a video stream and pushes the stream, the server load is greatly reduced; the technical solution of the present invention only involves a 3D rendering engine and a lightweight WebSocket server. Compared with the video streaming method, it is very lightweight and easy to quickly develop, deploy and implement.

[0052] Any of the above camera cloud mirror linkage control methods can be Figure 3 The lightweight 3D cloud rendering system shown is implemented, and the system includes:

[0053] Point of interest capture module, used to capture the positioning parameters (3D coordinates) of predetermined points (points of interest) in the substation digital twin system;

[0054] A camera parameter acquisition module, configured to acquire predetermined parameters of at least one camera within a certain spatial distance from a predetermined point;

[0055] A through-view camera screening module is used to perform a through-view analysis on at least one acquired camera, and screen out through-view cameras that are unobstructed from the predetermined point;

[0056] A pointing parameter acquisition module is used to obtain the pointing parameters of the through-view camera to a predetermined point;

[0057] PTZ parameter conversion module, used to convert the acquired pointing parameters into PTZ parameters of the physical camera;

[0058] Camera adjustment module, used to adjust the physical camera according to PTZ parameters;

[0059] The status update module is used to update the status of the virtual camera in the substation digital twin system according to the PTZ parameters of the physical camera;

[0060] The monitoring display module is used to obtain monitoring information from physical cameras and display it in the substation digital twin system.

[0061] For specific implementation, see Figure 2 In one embodiment of the camera cloud mirror linkage control method of the present invention, the step S1 is preceded by the following steps:

[0062] S0. Build a digital twin scenario of the substation (including its equipment, buildings, and environment).

[0063] Specifically, step S0 includes the following steps:

[0064] S01. Build digital twin scenarios for equipment, buildings, and environments in substations;

[0065] S02. Construct an accurate mapping of each physical camera in the digital twin scene, including the following parameters: spatial position, upward direction, positive direction, pitch angle, and azimuth angle;

[0066] S03. Establish data mapping between each physical camera and the corresponding virtual camera to achieve virtual-real interaction.

[0067] In specific implementation, step S03 includes:

[0068] S031. The virtual camera adjusts its state in the digital twin scene by obtaining the PTZ (Pan / Tilt / Zoom) parameters of the physical camera. PTZ parameters refer to the control of the camera's pan / tilt (left / right / up / down) movement and lens zoom.

[0069] S032. The virtual camera sets the PTZ parameters of the physical camera through the feedback control service to change the pitch angle and azimuth angle of the physical camera.

[0070] Any of the above camera cloud mirror linkage control methods can be Figure 4 The lightweight 3D cloud rendering system shown in the figure is implemented. The substation digital twin scene construction modules of this system include:

[0071] The equipment digital twin construction submodule is used to build digital twin scenarios for equipment in substations;

[0072] The building digital twin construction submodule is used to construct the digital twin scene of the building in the substation;

[0073] The environmental digital twin construction submodule is used to build a digital twin scenario of the environment in the substation;

[0074] The camera mapping submodule is used to build an accurate mapping of each physical camera in the digital twin scene;

[0075] The interactive submodule is used to establish data mapping between each physical camera and the corresponding virtual camera to achieve virtual-reality interaction.

[0076] Preferably, the interactive submodule further includes:

[0077] The virtual camera state adjustment unit is used to adjust the state of the virtual camera in the digital twin scene by obtaining the PTZ parameters of the physical camera;

[0078] The camera feedback control service unit is used for the virtual camera to set the PTZ parameters of the physical camera through the feedback control service to change the pitch angle and azimuth angle of the physical camera.

[0079] With the help of the lightweight three-dimensional cloud rendering system in the above embodiment, substation operation and maintenance personnel only need to click on the point of interest in the twin scene. The digital twin system automatically obtains the associated camera, accurately controls the pan-tilt head, and makes the camera accurately point to the point of interest, displaying the real-time video image of the point of interest in all directions, truly achieving the camera control of "point and see".

[0080] See also Figure 5 , an embodiment of the present invention is a lightweight 3D cloud rendering method and system, and its specific implementation process is as follows:

[0081] 1. Build a digital twin scenario for a substation:

[0082] (1) Build digital twin scenarios for substation equipment, buildings, and environments;

[0083] (2) Construct an accurate mapping of the camera in the digital twin scene, especially the spatial position, upward direction, positive direction, pitch angle, and azimuth angle;

[0084] 2. Establish data mapping between physical cameras and virtual cameras to achieve virtual-reality interaction:

[0085] (1) The virtual camera adjusts its status in the digital twin scene by obtaining the PTZ (Pan / Tilt / Zoom in security monitoring applications, representing the full range of pan / tilt movement and lens zoom control in left / right / up / down) parameters of the physical camera;

[0086] (2) The virtual camera sets the PTZ parameters of the physical camera through the feedback control service to change the pitch and azimuth angles of the physical camera;

[0087] 3. Click on a point of interest in the digital twin scene and obtain its 3D coordinates:

[0088] 4. Calculate the spatial distance between the camera and the point of interest, and select the closest cameras for visibility analysis. This determines whether the point of interest is blocked by other objects when viewed from the camera. Only unobstructed cameras are selected for the next step of analysis.

[0089] 5. Through spatial calculation, obtain the angles between the line of sight connecting the camera focus and the point of interest and the upward and positive directions of the camera, namely the pitch angle and azimuth angle;

[0090] 6. Convert the pitch and azimuth angles into PTZ coordinates for the camera pan / tilt system, and send control commands to the cameras to direct them toward the point of interest.

[0091] 7. The virtual camera in the digital twin scene obtains the PTZ coordinates of the physical camera pan / tilt, updates its status, and points to the point of interest;

[0092] 8. Obtain real-time video images from associated cameras to fully display the real-time status of points of interest.

[0093] Through actual production verification, the effects presented by the embodiments of the present invention are highly compatible with actual business needs. When maintenance personnel select a point of interest in a 3D scene, the system automatically selects the relevant camera, quickly and accurately points it to the point of interest, and simultaneously displays real-time video footage. This greatly facilitates maintenance personnel, significantly reducing the tedious task of adjusting cameras and improving work efficiency. Furthermore, by linking multiple cameras and displaying real-time footage of points of interest from multiple angles, the system significantly reduces blind spots.

[0094] In any of the above embodiments, the physical camera is a camera with a pan / tilt head.

[0095] In summary, the lightweight 3D cloud rendering method and system, and its interactive scheme of camera linkage control, have changed the general video control method of directly controlling the camera to point to the point of interest. The scheme of the present invention selects the point of interest, calculates which cameras can see the point of interest, and calculates the control parameters of the camera to achieve collaborative control and precise control of multiple cameras, and then combines the visibility check (i.e., determining whether the line of sight between the camera and the point of interest is blocked, and if it is not blocked, it is considered to be an associated camera) and the pan-tilt parameter calculation (calculating the azimuth and pitch angle of the camera based on the line of sight between the camera and the point of interest, as well as the spatial position, positive direction, and upward direction of the camera), so as to achieve camera alignment. The digital twin system automatically obtains the relevant cameras and accurately controls the pan / tilt to point the cameras to the points of interest, and displays the real-time video images of the points of interest in all directions, truly achieving the "point where you see" control of the camera. Compared with the existing technology of controlling the camera "point where you see", the operation method is significantly simplified, which not only greatly reduces the tedious work of adjusting the camera and improves work efficiency, but also greatly reduces the occurrence of visual blind spots by displaying the real-time images of the points of interest from all directions through the linkage of multiple cameras, which brings great convenience to operation and maintenance.

[0096] For those skilled in the art, based on the technical solutions disclosed in the present invention, it is obvious to combine the above embodiments or one / some technical features therein into a new embodiment without contradiction.

Claims

1. A lightweight 3D cloud rendering method, characterized in that: It includes the following steps: S1. Decouple the rendering engine and video service; S2. On the server side, a rendering engine renders the three-dimensional scene to obtain a rendering result; S3, outputs the rendering results to the downstream WebSocket server; S4. The client receives output data from the WebSocket server. S5. The client displays the final result.

2. The lightweight 3D cloud rendering method according to claim 1, wherein: Step S2 also includes: Step S202: Slice the rendered image to obtain slice data.

3. The lightweight 3D cloud rendering method according to claim 2, wherein: Before step S201, the following steps are also included: Step S201: Determine whether the image size and / or network conditions have reached a threshold.

4. The lightweight 3D cloud rendering method according to claim 2, wherein: In step S3: the sliced ​​image rendering results are output in parallel to the downstream WebSocket server.

5. The lightweight 3D cloud rendering method according to claim 1, wherein: In step S2: PNG image encoding is used to compress data in the color buffer to obtain an image rendering result.

6. The lightweight 3D cloud rendering method according to claim 1, wherein: In step S4: the client receives the rendering result from the WebSocket server through the browser.

7. The lightweight 3D cloud rendering method according to claim 6, wherein: In step S5: the received rendering result includes compressed image or slice data.

8. The lightweight 3D cloud rendering method according to claim 1, wherein: Also includes: Step S0: Expand the WebSocket server, that is, according to the load of the WebSocket server, increase the number of servers and improve the server performance.

9. The lightweight 3D cloud rendering method according to claim 8, wherein: According to the load of the WebSocket server, step S0 is executed.

10. A lightweight 3D cloud rendering system, characterized in that: It includes: Decoupling module, used to decouple the rendering engine and video service; The rendering module is used to render the 3D scene by the rendering engine on the server side to obtain the rendering result; Output module, used to output rendering results to the downstream WebSocket server; The data receiving module is used to receive output data from the WebSocket server on the client side; The result display module is used to display the final rendering result on the client.

Images