A server-side rendering and real-time pixel stream service publishing method based on Cesium

Abstract

The present invention provides a server-side rendering and real-time pixel stream service publishing method based on Cesium, and takes Cesium, an open source component based on WebGL technology, as the entry point to study the method of three-dimensional real-time cloud rendering. This method can not only break the technical barriers of commercial game manufacturers, but also achieve the goal of three-dimensional real-time cloud rendering based on Cesium. The method opens a browser running in the process on the server side, uses this browser to access a three-dimensional scene page based on Cesium, and integrates the three-dimensional scene image capture and the script program called by the WebRTC class library in the page, thereby realizing the function of rendering and publishing real-time pixel stream services on the server side.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a server-side rendering and real-time pixel stream service publishing method based on Cesium. Background Art

[0002] With the rapid development of cloud computing, cloud rendering, as an emerging cutting-edge technology, has gradually become an important technical means in the field of 3D rendering, especially in the application of digital twin business. However, at present, the implementation of this technology mainly relies on commercial game engine manufacturers such as Unreal Engine and Unity. The current digital twin application is only based on the secondary development of these engines, which is contrary to the direction of scientific and technological development and the requirements of information innovation.

[0003] In the actual project construction process, it was found that the method of publishing pixel streams based on the server-side three-dimensional scene of the commercial game engine also has the following shortcomings: 1) The scene is fixed. After the production and deployment are completed, it is impossible to dynamically expand the new scene, and the new scene needs to be repackaged and deployed. 2) It is difficult to add new load services. In order to expand the load service, the game engine needs to transform its cloud rendering signaling program and perform cumbersome configuration. 3) The risk of non-independent research and development is uncontrollable. Enterprises must comply with the business rules and charging rules formulated by commercial game engine R&D companies, which brings technical risks and cost uncertainty risks to the company's independent research and development. Through research, many small and micro enterprises' digital twin applications often use lightweight WebGL technology for three-dimensional rendering. If they want to develop business based on cloud rendering, they will face the overthrow of the original technology accumulation, re-transformation of technology and reconstruction of the R&D team, which will lead to problems such as increased learning thresholds and increased development costs, and also means the loss of core technology.

[0004] Therefore, it is necessary to provide a server-side rendering and real-time pixel stream service publishing method based on Cesium to solve the above technical problems. Summary of the invention

[0005] The present invention provides a server-side rendering and real-time pixel stream service publishing method based on Cesium. The method uses Cesium, an open source component based on WebGL technology, as an entry point to perform three-dimensional real-time cloud rendering, opens a browser running in a process on the server side, uses the browser to access a three-dimensional scene page based on Cesium, and integrates a script program for three-dimensional scene image capture and WebRTC library calls in the page, thereby realizing the function of rendering on the server side and publishing real-time pixel stream services.

[0006] In order to solve the above technical problems, the present invention provides a server-side rendering and real-time pixel stream service publishing method based on Cesium, comprising the following steps:

[0007] S1. Start the rendering application; Start the rendering application built in Node.js language; During the startup process, the rendering application loads necessary Node.js modules, including Puppeteer library and fs library, to support subsequent operations;

[0008] S2. Start the headless mode browser; the rendering application calls the Puppeteer.launch() method loaded in S1, adds the server GPU available and the headless mode initialization configuration;

[0009] S3. Read the scene configuration file; the rendering application uses the fs library loaded in S1 to read the scene configuration file and load the file content into the program, and calculates the number of tabs that need to be opened next based on the number of scenes;

[0010] S4. Open the tab to access the URL of the 3D scene page; the rendering application uses the newPage() method of the Browser class object in S2 to open the corresponding number of tabs in S3 and obtain the Page class object that can control and operate the tabs; use the Page.goto() method to access the URL of the 3D scene page, the Page.setViewport() method to set the scene parameters in S3, and the Page.evaluate() method to call the DOM elements in the 3D scene page and execute JavaScript code;

[0011] S5. Capture 3D scene image; The 3D scene rendering site service page uses Cesium to render the 3D scene that meets the conditions on the canvas according to the received scene parameters, uses the canvas.captureStream() method to capture the canvas content, and creates a MediaStream object to store the captured pixel stream, where the MediaStream object is a standard object for pixel stream transmission in the WebRTC library;

[0012] S6. Send pixel stream; the server-side 3D scene rendering site service page uses the media channel connected in S4 and uses the RTCPeerConnection.addTrack() method to send the MediaStream object obtained in S5 to the client page, thereby realizing the server-side pixel stream publishing function;

[0013] S7. Receive and execute mouse operation instructions; the server-side 3D scene rendering site service page uses the data channel connected in S4 to receive the transmitted mouse operation instruction object DataView, parses the mouse coordinates, type and other related information in DataView, uses the MouseEvent event object to create a simulation event, and uses the dispatchEvent method of the DOM element to trigger the simulation event, thereby affecting the change of the server-side 3D scene and realizing the user's real-time remote control.

[0014] Preferably, when S2 starts the headless mode browser, a headless mode Chromium browser is opened in the process and the Browser class object returned by the launch() method is obtained, and the Chromium browser is controlled and operated through the Browser class object.

[0015] Preferably, when S3 reads a scene configuration file, the scene configuration file includes access URLs of multiple three-dimensional scene pages and scene configuration parameters.

[0016] Preferably, in S4, the Page.setViewport() method sets the scene parameters in S3, and the scene parameters include setting the resolution of the three-dimensional scene rendering and the frame rate during capture.

[0017] Preferably, S4 calls the init() method of the WebRTC class library when the page is loaded to initialize and obtain the RTCPeerConnection class object. After the initialization is completed, the media channel between the client page and the server-side 3D scene rendering site service page is connected, and the data channel is connected using the RTCPeerConnection.createDataChannel() method to facilitate the subsequent transmission of pixel streams and the reception of mouse operation instructions.

[0018] Preferably, the mouse operation types parsed in DataView in S7 include click, move and release.

[0019] Compared with the related art, the present invention has the following beneficial effects:

[0020] The present invention provides a server-side rendering and real-time pixel stream service publishing method based on Cesium. The method provides an efficient and economical solution for enterprises that hope to achieve server rendering with the help of Cesium, so that the system integrating GIS and digital twins using Cesium as the development engine can use cloud rendering technology at low cost.

[0021] The present invention provides a server-side rendering and real-time pixel stream service publishing method based on Cesium, which realizes the separate deployment of rendering application and three-dimensional scene rendering, associates them through scene configuration files, and reduces the access difficulty of three-dimensional projects that have adopted Cesium as a development engine.

[0022] The present invention provides a server-side rendering and real-time pixel stream service publishing method based on Cesium, which supports a single rendering application to be compatible with multiple three-dimensional scenes through configuration, greatly increases the flexibility of server-side pixel stream publishing, and enriches the service content of the rendering application.

[0023] The present invention provides a server-side rendering and real-time pixel stream service publishing method based on Cesium, which realizes cross-platform support and supports deployment and use in two server operating systems, Windows and Linux. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 The present invention is a flow chart of a method for publishing a real-time pixel stream service.

[0025] Figure 2 A schematic diagram of the production and deployment of an application program in the present invention. DETAILED DESCRIPTION

[0026] The following is a detailed description of the implementation scheme of the present invention in conjunction with the accompanying drawings. It should be emphasized that the embodiments described in the present invention are illustrative rather than restrictive. Therefore, the present invention is not limited to the embodiments described in the specific implementation scheme. Any other implementation schemes derived by those skilled in the art based on the technical scheme of the present invention also fall within the scope of protection of the present invention. It should be noted that the specific implementation of the WebRTC protocol and the production of the Cesium model involved in the present invention both use mature products and mature technical means in the prior art. These implementations are irrelevant to the method to be described in the present invention and will not be described in detail.

[0027] The present invention provides a server-side rendering and real-time pixel stream service publishing method based on Cesium, and takes Cesium, an open source component based on WebGL technology, as the entry point to study the method of three-dimensional real-time cloud rendering. According to research, there is currently no application of cloud rendering technology in the engine represented by Cesium in the field of three-dimensional GIS. Therefore, it is challenging to realize cloud rendering technology, but this also shows that this research has the characteristics of field innovation, which can not only break the technical barriers of commercial game manufacturers, but also realize the landing of three-dimensional real-time cloud rendering based on Cesium. The method opens a browser running in the process on the server side, uses this browser to access the three-dimensional scene page based on Cesium, integrates the three-dimensional scene image capture and the script program called by the WebRTC class library in the page, thereby realizing the function of rendering and publishing real-time pixel stream service on the server side.

[0028] The idea of ​​script program implementation is:

[0029] (1) Start the headless browser. Puppeteer is a Node.js library that is used to control and operate the Chromium browser through a program. Use the launch() method of puppeteer to add the server GPU availability and headless mode initialization configuration, start Chromium in the process and obtain the Browser class object. The Browser class object can be used to control and operate the Chromium browser. Headless mode means starting and running a browser without a visible user interface.

[0030] (2) Read the scene configuration file. fs is a core module in Node.js that provides a set of methods and functions for interacting with the file system. Use the readFileSync() method of fs to read the 3D scene URL and scene parameters (such as scene resolution, frame rate, etc.) in the scene configuration file and obtain the number of open tabs.

[0031] (3) Open the tab and access the 3D scene page. Use the newPage() method of the Browser class object to let the browser in the process open the tab and obtain the Page class object. Use the goto() method of the Page class object to access the URL of the 3D scene page. The setViewport() method sets the scene parameters mentioned in the previous step. The evaluate() method calls the DOM elements in the 3D scene page and executes the JavaScript method preset on the page. This method is used to start WebRTC communication.

[0032] (4) Capture 3D scene images. The browser opens the 3D scene page developed based on Cesium. While the page renders the 3D scene on the canvas, it uses the canvas captureStream method to capture pixel streams to obtain real-time pixel streams.

[0033] (5) Sending pixel stream objects. This method integrates the script called by the WebRTC class library in the 3D scene page, and uses the method of connecting media channels in the WebRTC class library to send the pixel stream captured in the previous step to the client, thereby realizing the server-side pixel stream publishing function.

[0034] (6) Receive mouse operation instructions. Use the method of creating a data stream channel in the WebRTC class library to establish a real-time data channel between the 3D scene page and the client page. After receiving the mouse operation instruction object (DataView) transmitted in the channel, parse the mouse coordinates, type (click, move, release, etc.) and other related information in the DataView, simulate and trigger mouse events in the 3D scene page, thereby affecting changes in the 3D scene on the server side, and finally realizing real-time remote control of users.

[0035] Package the rendering application source code. Use the pkg tool to package the rendering application source code into a program that can be executed by multiple operating systems.

[0036] In order to more clearly describe the specific implementation mode of the present invention, an embodiment is provided below:

[0037] like Figure 1 As shown, the present invention is a server-side rendering and real-time pixel stream service publishing method based on Cesium. The method mainly includes two core components: rendering application (RenderApplication) and 3D scene rendering site service (WebProviderServer). The workflow and implementation principle of this method are described in detail below:

[0038] (1) Start the rendering application. The first step of the present invention is to start the rendering application, which is implemented in the Node.js language. During the startup process, the rendering application loads the necessary Node.js modules, including the Puppeteer library and the fs library, to support subsequent operations.

[0039] (2) Start the headless mode browser. The rendering application calls the Puppeteer.launch() method loaded in the previous step, adds the server GPU available and the headless mode initialization configuration, starts a headless mode Chromium browser in the process and obtains the Browser class object returned by the launch() method. The Chromium browser can be controlled and operated through the Browser class object.

[0040] (3) Read the scene configuration file. The scene configuration file contains the access URLs of multiple 3D scene pages and scene configuration parameters. The rendering application uses the fs library loaded in (1) to read the scene configuration file and load the file content into the program, and calculates the number of tabs that need to be opened next based on the number of scenes.

[0041] (4) Open the tab to access the 3D scene page URL. The rendering application uses the newPage() method of the Browser class object in (2) to open the corresponding number of tabs in (3) and obtain the Page class object that can control and operate the tabs. Use the Page.goto() method to access the 3D scene page URL, the Page.setViewport() method to set the scene parameters mentioned in (3) (such as setting the resolution of 3D scene rendering and the frame rate during capture, etc.), and the Page.evaluate() method to call the DOM elements in the 3D scene page and execute JavaScript code. This method can be used to call the WebRTC class library init() method when the page is loaded to initialize and obtain the RTCPeerConnection class object. After the initialization is completed, the media channel between the client page and the server-side 3D scene rendering site service page will be connected. The RTCPeerConnection.createDataChannel() method can be used to connect the data channel, which lays the foundation for the subsequent transmission of pixel streams and the reception of mouse operation instructions.

[0042] (5) Capture 3D scene images. The 3D scene rendering site service page uses Cesium to render a 3D scene that meets the conditions on a canvas based on the received scene parameters. The canvas.captureStream() method can be used to capture the canvas content and create a MediaStream object to store the captured pixel stream. The MediaStream object is the standard object for pixel stream transmission in the WebRTC library.

[0043] (6) Send pixel stream. The server-side 3D scene rendering site service page uses the media channel connected in (4) and uses the RTCPeerConnection.addTrack() method to send the MediaStream object obtained in the previous step to the client page, thereby realizing the server-side pixel stream publishing function.

[0044] (7) Receive and execute mouse operation instructions. The server-side 3D scene rendering site service page uses the data channel connected in (4) to receive the transmitted mouse operation instruction object (DataView), parse the mouse coordinates, type (click, move, release, etc.) and other related information in DataView, use the MouseEvent event object to create a simulation event, and use the dispatchEvent method of the DOM element to trigger the simulation event, thereby affecting the change of the server-side 3D scene, and finally realizing the user's real-time remote control.

[0045] The above is a specific implementation method and principle of the present invention. Through the above method, Cesium technology is used to render and publish real-time pixel stream services on the server side.

[0046] Application development and deployment process:

[0047] (1) Use the pkg tool to package the rendering application source code. pkg can be used to package the rendering application source code into a program that can be executed on Windows or Linux, thereby supporting different operating systems.

[0048] (2) Prepare a scene configuration file. The scene configuration file contains the access URL of the 3D scene page and the scene configuration parameters. The number of configurations depends on the specific business scenario.

[0049] (3) Download and unzip Chromium. Download the compressed file of Chromium for the operating system suitable for the deployment environment and unzip it to the same directory as the RenderApplication running program. Chromium is an open source web browser project that provides many core functions of a web browser. It can run in headless mode and supports installation-free deployment.

[0050] (4) Deploy WebProviderServer to provide a site service for 3D scene rendering. The deployment of this service must ensure that it is interconnected with the network of the server where the rendering application is located, and that all 3D scene URLs can be correctly accessed through Chromium.

[0051] The above descriptions are merely embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A server-side rendering and real-time pixel stream service publishing method based on Cesium, characterized in that: The following steps are involved: S1. Start the rendering application; Start the rendering application built in Node.js language; During the startup process, the rendering application loads Node.js modules, including Puppeteer library and fs library, to support subsequent operations; S2. Start the headless mode browser; the rendering application calls the Puppeteer.launch() method loaded in S1, adds the server GPU available and the headless mode initialization configuration; S3. Read the scene configuration file; the rendering application uses the fs library loaded in S1 to read the scene configuration file and load the file content into the program, and calculates the number of tabs that need to be opened next based on the number of scenes; S4. Open the tab and visit the URL of the 3D scene page; The rendering application uses the newPage() method of the Browser class object in S2 to open the corresponding number of tabs in S3, and obtains the Page class object that can control and operate the tabs; uses the Page.goto() method to access the URL of the 3D scene page, the Page.setViewport() method to set the scene parameters in S3, and the Page.evaluate() method to call the DOM elements in the 3D scene page and execute JavaScript code; when the page is loaded, the S4 calls the init() method of the WebRTC class library to initialize and obtain the RTCPeerConnection class object, and after the initialization is completed, connects the media channel between the client page and the server-side 3D scene rendering site service page, and uses the RTCPeerConnection.createDataChannel() method to connect the data channel, so as to subsequently transmit pixel streams and receive mouse operation instructions; S5. Capture 3D scene image; The 3D scene rendering site service page uses Cesium to render the 3D scene that meets the conditions on the canvas according to the received scene parameters, uses the canvas.captureStream() method to capture the canvas content, and creates a MediaStream object to store the captured pixel stream, where the MediaStream object is a standard object for pixel stream transmission in the WebRTC library; S6. Send pixel stream; The server-side 3D scene rendering site service page uses the media channel connected in S4 and the RTCPeerConnection.addTrack() method to send the MediaStream object obtained in S5 to the client page, thereby realizing the server-side pixel stream publishing function; S7. Receive and execute mouse operation instructions; The server-side 3D scene rendering site service page uses the data channel connected in S4 to receive the mouse operation instruction object DataView, parse the mouse coordinates, type and other related information in DataView, use the MouseEvent event object to create a simulation event, and use the dispatchEvent method of the DOM element to trigger the simulation event, thereby affecting the change of the server-side 3D scene and realizing the user's real-time remote control.

2. According to a Cesium-based server-side rendering and real-time pixel stream service publishing method according to claim 1, it is characterized in that: When S2 starts the headless mode browser, a headless mode Chromium browser is opened in the process and the Browser class object returned by the launch() method is obtained, and the Chromium browser is controlled and operated through the Browser class object.

3. According to a Cesium-based server-side rendering and real-time pixel stream service publishing method according to claim 1, it is characterized in that: When the S3 reads the scene configuration file, the scene configuration file includes access URLs of multiple three-dimensional scene pages and scene configuration parameters.

4. According to a Cesium-based server-side rendering and real-time pixel stream service publishing method according to claim 1, it is characterized in that: In S4, the Page.setViewport() method sets the scene parameters in S3, and the scene parameters include setting the resolution of the three-dimensional scene rendering and the frame rate during capture.

5. According to a Cesium-based server-side rendering and real-time pixel stream service publishing method according to claim 1, it is characterized in that: The mouse operation types analyzed in the DataView in S7 include click, move and release.

Images