Methods and apparatus for fusing video streams with 3D GIS scenes

Abstract

The present application is applicable to the field of data processing technology, and provides a method and device for fusing video streams with three-dimensional GIS scenes. The method includes: sending a video stream request to a streaming media server, the video stream request includes a video stream address, and is used to request the streaming media server to obtain a target video stream according to the video stream address, and convert the target video stream into video stream data of the WebRTC protocol; receiving the video stream data sent by the streaming media server through the RTCPeerConnection data channel, and generating a Video tag of the video stream data; determining the camera parameters of the video fusion point in the video stream data, and determining the projection of the video fusion point in the three-dimensional GIS scene according to the camera parameters; constructing a projected three-dimensional sticker vector surface, and assigning the Video tag as a material object to the three-dimensional sticker vector surface. The present application can directly use video streams of multiple different protocols without additional plug-ins and hard decoding, and can also reduce video stream delays.

Description

Technical Field

[0001] The present application belongs to the field of data processing technology, and in particular, relates to a method and device for fusing video streams with three-dimensional GIS scenes. Background Art

[0002] The fusion visualization of video streams and three-dimensional scenes is widely used in various real-time digital twin projects. The traditional method of fusing video streams with three-dimensional scenes is to use the JSON file generated by the backend to align the video fusion projection surface, then access and parse the video stream, and finally fuse the registered three-dimensional scene with the parsed video stream for rendering.

[0003] At present, video fusion technology in the field of WebGIS uses different plug-ins and hard decoding methods for access and analysis of different video stream protocols. Moreover, using plug-ins and hard decoding to parse different video streams will cause the video stream to have a high delay of more than 1 second, which will seriously affect the performance when accessing multiple video streams. Summary of the invention

[0004] In order to overcome the problems existing in the related art, the embodiments of the present application provide a method and device for fusing a video stream with a three-dimensional GIS scene.

[0005] This application is implemented through the following technical solutions:

[0006] In a first aspect, an embodiment of the present application provides a method for fusing a video stream with a three-dimensional GIS scene, including:

[0007] The client sends a video stream request to the streaming media server, wherein the video stream request includes a video stream address, and the video stream request is used to request the streaming media server to obtain a target video stream according to the video stream address, and convert the target video stream into video stream data of the WebRTC protocol;

[0008] The client receives the video stream data sent by the streaming media server through the RTCPeerConnection data channel, and determines the Video tag of the video stream data;

[0009] The client determines the camera parameters of the video fusion point in the video stream data, and determines the projection of the video fusion point in the three-dimensional GIS scene according to the camera parameters;

[0010] The client constructs the projected three-dimensional tile vector surface, and assigns the Video tag as a material object to the three-dimensional tile vector surface.

[0011] In combination with the first aspect, in some embodiments, the method for fusing the video stream with the three-dimensional GIS scene further includes:

[0012] Construct a streaming media server, a signaling server and a client based on the WebRTC protocol; wherein the streaming media server supports unified access to multiple video streaming protocols, and sends out real-time video streams using the WebRTC protocol by parsing video streams of multiple protocols; the signaling server is used for signaling interaction with the client and sending business instructions to the streaming media server, and while ensuring a long connection with the streaming media server, by completing signaling interaction and forwarding with the client, a data stream channel is established between the client and the streaming media server, and the transmission of the WebRTC protocol video stream is completed; the client includes a WebRTC receiving end and a three-dimensional GIS engine client, and the WebRTC receiving end establishes a connection with the signaling server through a WebSocket, sends signaling and business instructions, and after the signaling exchange is completed, an RTCPeerConnection data channel is established with the streaming media server to transmit video stream data of the WebRTC protocol, and the video stream data is displayed as data using the Video tag; the three-dimensional GIS client is used to load a three-dimensional oblique photography model, and construct a real-life three-dimensional scene and two- and three-dimensional projection vector surfaces;

[0013] Build a network penetration service and set up the STUN / TURN network penetration service of the ICE framework to achieve end-to-end data channel connection of WebRTC.

[0014] In combination with the first aspect, in some embodiments, before the client sends a video stream request to the streaming media server, the method for fusing the video stream with the three-dimensional GIS scene further includes:

[0015] The client establishes a WebSocket connection with the signaling server;

[0016] The client completes signaling exchange with the signaling server and sends network penetration address information and user information;

[0017] The signaling server completes signaling forwarding and completes signaling exchange with the streaming media server;

[0018] The client establishes an RTCPeerConnection data channel with the streaming media server.

[0019] In combination with the first aspect, in some embodiments, the client determines the camera parameters of the video fusion point in the video stream data, including:

[0020] The client requests the signaling server for the camera parameters of the video fusion point in the video stream data, or the client parses the video stream data to determine the camera parameters of the video fusion point;

[0021] The camera parameters include one or more of the camera's spatial position, spatial posture, focal length, aspect ratio, viewing angle, and maximum viewing distance.

[0022] In combination with the first aspect, in some embodiments, determining the projection of the video fusion point in the three-dimensional GIS scene according to the camera parameters includes:

[0023] Parsing the camera parameters into world coordinates, camera focal length, camera near plane distance, far plane distance, camera aspect ratio, heading angle, pitch angle, and roll angle to construct a frustum object;

[0024] Obtaining the far surface information of the frustum object, where the far surface is composed of four vertex coordinates;

[0025] The world coordinates of the camera are used as the starting point of the ray, and four rays are constructed with the four vertices of the long-range surface respectively. When the camera is tilted downward, the four rays can intersect with the ground to obtain four key points. The area surrounded by the four key points is the projection.

[0026] In combination with the first aspect, in some embodiments, the client constructs the projected three-dimensional sticker vector surface, including:

[0027] A PolygonPrimitive surface is constructed in the GIS engine according to the four key points, and the ground-attaching properties of the PolygonPrimitive surface element are set to obtain a three-dimensional object vector surface in the three-dimensional scene; wherein the three-dimensional object vector is a surface element in which a two-dimensional vector surface completely fits on the three-dimensional element and has height information in the three-dimensional scene.

[0028] In combination with the first aspect, in some embodiments, assigning the Video tag as a material object to the three-dimensional sticker vector surface includes:

[0029] The material of the three-dimensional tile vector surface is set as the Video tag of the video stream, the three-dimensional tile vector surface is filled with the video stream, and the video stream is merged with the three-dimensional GIS scene.

[0030] In combination with the first aspect, in some embodiments, after assigning the Video tag as a material object to the three-dimensional tile vector surface, the method for fusing the video stream with the three-dimensional GIS scene further includes:

[0031] The spatial position and posture of the view cone in the 3D GIS scene are updated in real time.

[0032] In combination with the first aspect, in some embodiments, before the client determines the camera parameters of the video fusion point in the video stream data, the method for fusing the video stream with the three-dimensional GIS scene further includes:

[0033] Select a GIS engine;

[0034] Request the map tile data address, 3D model data address, terrain data address and other single model addresses around the video fusion point;

[0035] The data loading API interface provided by the GIS engine is used to render and visualize the obtained map tile data, three-dimensional model data, terrain data and other single models; in the process of loading the data in the three-dimensional GIS scene, the geographical location and orientation information of the GIS map and the coordinate system and direction information of the real-life model are obtained, and the real-life model is adjusted and aligned with the GIS map according to the geographical location and orientation information of the GIS map and the coordinate system and direction information of the real-life model.

[0036] In a second aspect, an embodiment of the present application provides a device for fusing a video stream with a three-dimensional GIS scene, including:

[0037] A request sending module is used to send a video stream request to a streaming media server, wherein the video stream request includes a video stream address, and the video stream request is used to request the streaming media server to obtain a target video stream according to the video stream address, and convert the target video stream into video stream data of the WebRTC protocol;

[0038] A data receiving module, used for receiving the video stream data sent by the streaming media server through the RTCPeerConnection data channel, and determining the Video tag of the video stream data;

[0039] A projection determination module, used to determine the camera parameters of the video fusion point in the video stream data, and determine the projection of the video fusion point in the three-dimensional GIS scene according to the camera parameters;

[0040] A fusion module is used to construct the projected three-dimensional patch vector surface and assign the Video tag as a material object to the three-dimensional patch vector surface.

[0041] Compared with the prior art, the embodiments of the present invention have the following beneficial effects:

[0042] In the embodiment of the present application, the client sends a video stream request to the streaming media server, and the video stream request is used to request the streaming media server to obtain the target video stream according to the video stream address, and convert the target video stream into video stream data of the WebRTC protocol; the client receives the video stream data sent by the streaming media server through the RTCPeerConnection data channel, and generates a Video tag for the video stream data. The client determines the camera parameters of the video fusion point in the video stream data, and determines the projection of the video fusion point in the three-dimensional GIS scene according to the camera parameters; the client constructs the three-dimensional sticker vector surface of the projection, and assigns the Video tag as a material object to the three-dimensional sticker vector surface to realize the fusion of the video stream in the three-dimensional GIS scene.

[0043] In the embodiment of the present application, a video stream is converted into video stream data of the WebRTC protocol. The WebRTC communication protocol is used to be compatible with video stream transmission of multiple video stream protocols, and video streams of various different protocols can be directly used without additional plug-ins and hard decoding. WebRTC is used as the communication protocol to perform video stream protocol conversion in a streaming media server, which reduces the time for conversion and decoding in the front-end browser, improves performance, and reduces video stream delay.

[0044] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present specification. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0046] Figure 1 is a schematic diagram of a system architecture provided by an embodiment of the present application;

[0047] Figure 2 It is a flowchart of a method for fusing a video stream with a three-dimensional GIS scene provided in one embodiment of the present application;

[0048] Figure 3 It is a structural diagram of a device for fusing video stream and three-dimensional GIS scene provided in an embodiment of the present application. DETAILED DESCRIPTION

[0049] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0050] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, wholes, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or combinations thereof.

[0051] It should also be understood that the term “and / or” used in the specification and appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0052] As used in the specification and appended claims of this application, the term "if" can be interpreted as "when" or "uponce" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "uponce it is determined" or "in response to determining" or "uponce [described condition or event] is detected" or "in response to detecting [described condition or event]", depending on the context.

[0053] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0054] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0055] First, the system architecture involved in the embodiment of the present application is described. Figure 1The system architecture of the embodiment of the present application includes a client 101, a signaling server 102 and a streaming media server 103.

[0056] Specifically, the streaming media server 103 supports unified access to mainstream video streaming protocols such as RTSP, FLV, HLS, etc. The streaming media server 103 parses video streams of various protocols such as RTSP, FLV, HLS, etc., converts the parsed video streams into real-time video streams of the WebRTC protocol and sends them out.

[0057] The signaling server 102 is used for signaling interaction with the client 101 and sending service instructions to the streaming server 103. While ensuring a long connection with the streaming server 103, the signaling server 102 completes signaling interaction and forwarding with the client 101, establishes a data stream channel between the client 101 and the streaming server 103, and completes the transmission of the WebRTC protocol video stream.

[0058] The client 101 may include a WebRTC receiver and a 3D GIS engine client. The client 101 establishes a connection with the signaling server 102 through WebSocket based on the WebRTC receiver, sends signaling and service instructions, and after the signaling exchange is completed, builds an RTCPeerConnection channel with the streaming media server 103 to transmit the video stream data of the WebRTC protocol and display the data with the Video tag. The 3D GIS client is built with current open source popular engines, such as Cesium and MapTalks, to load 3D oblique photography models, build real 3D scenes and 2D and 3D projection vector surfaces.

[0059] The following describes the process of a user requesting video stream data of multiple protocols from the streaming media server 103 through the client 101.

[0060] Step 1: The client 101 establishes a WebSocket connection with the signaling server 102.

[0061] Step 2: The client 101 completes the signaling exchange with the signaling server 102 and sends the address information and user information of the network penetration.

[0062] Step 3: The signaling server 102 completes signaling forwarding and completes signaling exchange with the streaming media server 103.

[0063] Step 4: The client 101 establishes an RTCPeerConnection data channel with the streaming server 103.

[0064] Step 5: The client 101 carries the video stream address such as RTSP, HLS, FLV, and the start playback command, and requests the WebRTC protocol video stream processed by the streaming media server 103.

[0065] Step 6: The streaming media server 103 accesses and parses the video streams such as RTSP, HLS, FLV, etc. according to the video stream address, converts the video stream into video stream data of the WebRTC protocol, and sends it to the client 101 through the established RTCPeerConnection channel.

[0066] Step 7: The client 101 parses the video stream data of the WebRTC protocol, and receives and displays it with a Video tag.

[0067] Step 8: The client 101 sends a service instruction to the signaling server 102 to complete the relevant control of the video stream playback, such as pause, stop, jump, debug, information acquisition, etc.

[0068] If the client 101 needs to request another video stream data, repeat the above steps 5 to 8, use the established RTCPeerConnection channel to complete the playback of other video streams, implement video stream multiplexing, and use SSRC (Synchronization Source) identifiers to distinguish different video streams.

[0069] The following is based on Figure 1 The method for fusing the video stream and the three-dimensional GIS scene in the embodiment of the present application is described in detail.

[0070] Reference Figure 2 The fusion method of the video stream and the three-dimensional GIS scene is described as follows:

[0071] Step 201: The client sends a video stream request to a streaming media server. The video stream request includes a video stream address. The video stream request is used to request the streaming media server to obtain a target video stream according to the video stream address and convert the target video stream into video stream data of the WebRTC protocol.

[0072] In some embodiments, before the client sends a video stream request to the streaming media server, the method for fusing the video stream with the three-dimensional GIS scene further includes:

[0073] The client establishes a WebSocket connection with the signaling server;

[0074] The client completes signaling exchange with the signaling server and sends network penetration address information and user information;

[0075] The signaling server completes signaling forwarding and completes signaling exchange with the streaming media server;

[0076] The client establishes an RTCPeerConnection data channel with the streaming media server.

[0077] In some embodiments, the method for fusing the video stream with the three-dimensional GIS scene may further include:

[0078] Construct a streaming media server, a signaling server and a client based on the WebRTC protocol; wherein the streaming media server supports unified access to multiple video streaming protocols, and sends out real-time video streams using the WebRTC protocol by parsing video streams of multiple protocols; the signaling server is used for signaling interaction with the client and sending business instructions to the streaming media server, and while ensuring a long connection with the streaming media server, by completing signaling interaction and forwarding with the client, a data stream channel is established between the client and the streaming media server, and the transmission of the WebRTC protocol video stream is completed; the client includes a WebRTC receiving end and a three-dimensional GIS engine client, and the WebRTC receiving end establishes a connection with the signaling server through a WebSocket, sends signaling and business instructions, and after the signaling exchange is completed, an RTCPeerConnection data channel is established with the streaming media server to transmit video stream data of the WebRTC protocol, and the video stream data is displayed as data using the Video tag; the three-dimensional GIS client is used to load a three-dimensional oblique photography model, and construct a real-life three-dimensional scene and two- and three-dimensional projection vector surfaces;

[0079] Build a network penetration service and set up the STUN / TURN network penetration service of the ICE framework to achieve end-to-end data channel connection of WebRTC.

[0080] Step 202: The client receives the video stream data sent by the streaming media server through the RTCPeerConnection data channel, and generates a Video tag for the video stream data.

[0081] Specifically, the client parses the video stream data of the WebRTC protocol, and receives and displays the parsed video stream data with a Video tag.

[0082] Step 203: The client determines the camera parameters of the video fusion point in the video stream data, and determines the projection of the video fusion point in the three-dimensional GIS scene according to the camera parameters.

[0083] Exemplarily, the client determines the camera parameters of the video fusion point in the video stream data, which may include: the client requests the camera parameters of the video fusion point in the video stream data from the signaling server, or the client parses the video stream data to determine the camera parameters of the video fusion point. The camera parameters include one or more of the camera's spatial position, spatial attitude (such as heading angle, pitch angle, roll angle), focal length, aspect ratio, viewing angle, and maximum viewing distance.

[0084] Exemplarily, the above-mentioned determination of the projection of the video fusion point in the three-dimensional GIS scene according to the camera parameters may include: parsing the camera parameters into world coordinates, camera focal length, camera near-plane distance, far-plane distance, camera aspect ratio, heading angle, pitch angle and roll angle, and constructing a viewing cone object according to the parsed world coordinates, camera focal length, camera near-plane distance, far-plane distance, camera aspect ratio, heading angle, pitch angle and roll angle; obtaining the far-plane information of the viewing cone object, the far-plane consisting of four vertex coordinates; using the world coordinates of the camera as the starting point of the ray, and constructing four rays with the four vertices of the far-plane respectively, when the camera is tilted downward, the four rays can intersect with the ground to obtain four key points, and the area surrounded by the four key points is the projection.

[0085] In some embodiments, before the client determines the camera parameters of the video fusion point in the video stream data, the method for fusing the video stream with the three-dimensional GIS scene further includes:

[0086] Select a GIS engine; for example, use the Cesium engine to build a WebGIS environment. The GIS engine can support the rendering of 3D scenes and the expansion of video fusion;

[0087] Request the map tile data address, 3D model data address, terrain data address and other single model addresses around the video fusion point;

[0088] Use the data loading API interface provided by the GIS engine to render and visualize the obtained map tile data, three-dimensional model data, terrain data and other single models; in the process of loading the data in the three-dimensional GIS scene, it is necessary to calibrate its position and orientation so that the model can be aligned with the GIS map. Specifically: obtain the geographic location and orientation information of the GIS map and the coordinate system and direction information of the real-life model, and adjust and align the real-life model with the GIS map according to the geographic location and orientation information of the GIS map and the coordinate system and direction information of the real-life model.

[0089] Step 204: The client constructs the projected three-dimensional tile vector surface and assigns the Video tag as a material object to the three-dimensional tile vector surface.

[0090] Exemplarily, the client constructs the projected three-dimensional object vector surface, which may include: constructing a PolygonPrimitive surface in the GIS engine according to the four key points, and setting the ground-fitting properties of the PolygonPrimitive surface element to obtain a three-dimensional object vector surface in the three-dimensional scene. The vector surface is a closed area composed of an xy coordinate array, that is, a surface obtained by connecting multiple coordinates in a closed loop; the three-dimensional object vector is a surface element in which a two-dimensional vector surface completely fits on a three-dimensional element and has height information in a three-dimensional scene.

[0091] Exemplarily, the above-mentioned assigning the Video tag as a material object to the three-dimensional tile vector surface may include: setting the material Matrial of the three-dimensional tile vector surface to the Video tag of the video stream, filling the three-dimensional tile vector surface with the video stream, and fusing the video stream with the three-dimensional GIS scene.

[0092] For example, if multiple video streams need to be merged with a 3D GIS model, the Vidio tag, projection and 3D tile vector surface corresponding to each video stream can be determined, and the Video tag can be assigned to the corresponding 3D tile vector surface as a material object.

[0093] In some embodiments, after assigning the Video tag as a material object to the three-dimensional tile vector surface, the method for fusing the video stream with the three-dimensional GIS scene further includes: updating the spatial position and posture of the viewing cone in the three-dimensional GIS scene in real time.

[0094] Specifically, after the video fusion is initially completed, due to camera parameter errors or real-time changes in the spatial posture of the camera (such as drone cameras) on the projection surface, the update method of the video fusion plug-in can be used to update the spatial position and posture of the view cone in the 3D GIS scene in real time, thereby ensuring that the video content and the 3D GIS scene are always in a highly fused state.

[0095] The following further describes the embodiments of the present application in combination with specific examples.

[0096] 1. Reference Figure 1 The system architecture runs the streaming media server and signaling server.

[0097] Specifically, the WebRTC streaming media server is first run to wait for the connection request and service request from the signaling server and the client. Then the signaling server (Java Server) is run, carrying the address of the streaming media server (for example, ws: / / 192.168.50.136:8888 / kurento) during the run, and a long connection between the signaling server and the streaming media server is established to wait for signaling and service instructions.

[0098] 2. Run the client and establish a connection with the signaling server.

[0099] Specifically, in the opened application client, a WebSocket connection request is initiated to the signaling server. The request format can be wss: / / 127.0.0.1:8443 / socket / player / dxs. After the connection is established, business messages can be sent and received to the signaling server.

[0100] 3. Connect the client to the streaming server.

[0101] 3.1 Carrying parameter information required for signaling exchange

[0102] Specifically, it includes audio and video transmission options, which are audio and video options; network penetration service parameters, including stun service address, the format can be stun:192.168.50.136:3478, turn

[0103]

[0104] urls:["turn:192.168.50.136:3478"],username:"kurento",credential:"kurento"; local candidate information.

[0105] 3.2 Complete the signaling exchange and establish a data channel between the client and the streaming media server.

[0106] Specifically, the client sends the above parameter information to the signaling server to exchange network addresses, media negotiation and other information between the local and streaming media servers to complete the signaling exchange; then establishes an RTCPeerConnection connection, enables the ICE process (Interactive Connectivity Establishment), selects the best transmission path between the client and the streaming media server, establishes a P2P connection, and after successfully connecting to the streaming media server, uses the connection to create a new data channel to receive video stream data.

[0107] 4. Client rendering of 3D GIS scene

[0108] 4.1 Front-end 2D and 3D rendering engine selection

[0109] Specifically, select a suitable GIS engine according to business and scenario requirements, such as Cesium.js or MapTalks.js, which can support the extension of 3D scene rendering and video fusion.

[0110] 4.2 Requesting 3D scene data

[0111] Specifically, according to the needs of the video fusion location, the map tile data address, 3D model data address, terrain data address and other single model addresses near the location may be requested.

[0112] 4.3 Constructing a 3D scene

[0113] Specifically, you can use the data loading API provided by Cesium to render and visualize the obtained map data services, 3D model data services, terrain data services, and other single model services, and set the camera of the 3D scene to be near the video fusion range.

[0114] 4.4 Dynamic Adjustment of 3D Scene

[0115] Specifically, when loading data in a 3D scene, its position and orientation need to be calibrated so that the model can be aligned with the GIS map. The calculation and conversion can be performed by obtaining the geographic location and orientation information on the GIS map, as well as the coordinate system and direction information of the real scene model.

[0116] 5. Request and parse video stream data of various protocols

[0117] 5.1 Request video stream address

[0118] Specifically, the client can request the data address of the current point video stream from the signaling server according to the current video fusion point information. The address can be in the protocol format of RTSP, HLS, or FLV. The format of RTSP can be rtsp: / / 192.168.50.135:8554 / cdutStream, the protocol format of HLS can be http: / / 192.168.50.135:8554 / cdutStream.m3u8, and the video format of the FLV protocol can be http: / / 192.168.50.135:8554 / cdutStream.flv.

[0119] 5.2 Requesting video stream data

[0120] Specifically, the video stream address obtained in the previous step is sent to the signaling server through WebSocket, and carries the start playback command. After receiving the video stream address and the start playback command, the signaling server forwards them to the streaming media server.

[0121] 5.3 Receive video stream data and display

[0122] Specifically, after the streaming media server receives the video stream address that needs to be processed and the start playback command, it pulls the video stream of the video stream address, processes the data after the stream is successfully pulled, completes the transmission protocol conversion, and transmits the video stream data from the previously established data stream channel to the client in the form of the WebRTC protocol. After the client receives the video stream data, it can directly display it with a video tag or canvas.

[0123] 6. Constructing the frustum projection fusion surface

[0124] 6.1 Request camera related parameters

[0125] Specifically, the client can parse the camera parameters carried in the video stream data transmitted by the streaming media server, or request the camera parameters from the signaling server. The camera parameters may include the camera's spatial position (latitude and longitude, altitude), spatial attitude (pitch angle, heading angle, roll angle), focal length, aspect ratio, viewing angle, maximum field of view distance and other parameters.

[0126] 6.2 Drawing the camera's viewing cone in a 3D scene

[0127] Specifically, according to the obtained camera parameters, the graphics drawing API interface provided by Cesium is used to correct the camera parameters in the three-dimensional scene and draw a frustum graphic that simulates the camera. The frustum graphic can represent the camera's field of view in the space.

[0128] 6.3 Drawing the video fusion projection vector surface

[0129] Specifically, based on the already drawn view frustum, the projection key points of the view frustum and the three-dimensional model or terrain are calculated, and the projected vector surface can be constructed based on the calculated projection key points; according to the fusion requirements, the fusion method of 2D plane and 3D sticker can be selected to construct different vector surfaces.

[0130] 7. Complete the integration of video stream and 3D GIS scene

[0131] 7.1 Fusion of video stream and 3D GIS scene

[0132] Specifically, the video stream received by the previous video or canvas is passed to the calculated vector surface as a material object. GIS engines such as Cesium support real-time updates with the content of the video tag, thereby realizing the dynamic fusion of the video stream and the three-dimensional GIS scene.

[0133] 7.2 Dynamic Interaction and Update of Video Fusion Surface

[0134] Specifically, after the video fusion is initially completed, due to camera parameter errors or real-time changes in the spatial posture of the camera on the projection surface (such as drone cameras), the update method of the video fusion plug-in can be used to update the spatial position and posture of the view cone in the three-dimensional scene in real time, thereby ensuring that the video content and the three-dimensional scene are always in a highly fused state.

[0135] 7.3 Multi-channel access for video fusion

[0136] Specifically, for the same 3D scene, there may be multiple video stream data fusion situations. The user can repeat the steps after establishing the data channel between the client and the streaming media server to complete the unified access of multiple video stream lines. Since the same data channel is used to transmit the video stream, the overall performance and delay can be guaranteed.

[0137] The above-mentioned method for fusing video streams with three-dimensional GIS scenes constructs a streaming media server, signaling server, client and network penetration service for end-to-end transmission for video fusion; uses the WebRTC communication protocol to be compatible with video stream transmission of multiple video stream protocols, and can directly use video streams of various protocols without additional plug-ins and hard decoding; one data channel transmits multiple video streams simultaneously to achieve high-performance multiplexing; uses WebRTC as the communication protocol, and performs video stream protocol conversion in the streaming media server, which reduces the time for conversion and decoding in the front-end browser and improves performance. The video stream delay can reach the millisecond level (within 1s); uses the cone simulation technology, and uses a small amount of camera information to complete the automatic calculation and alignment of the projection surface range, without the need to manually select the projection surface range, which improves the convenience and reliability of video fusion.

[0138] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0139] Corresponding to the method for fusing the video stream and the three-dimensional GIS scene described in the above embodiment, Figure 3 A structural block diagram of a device for fusing a video stream and a three-dimensional GIS scene provided in an embodiment of the present application is shown. For ease of explanation, only the parts related to the embodiment of the present application are shown.

[0140] See also Figure 3 The device for fusing a video stream with a three-dimensional GIS scene in an embodiment of the present application may include a request sending module 301 , a data receiving module 302 , a projection determination module 303 and a fusion module 304 .

[0141] Among them, the request sending module 301 is used to send a video stream request to the streaming media server, and the video stream request includes a video stream address. The video stream request is used to request the streaming media server to obtain the target video stream according to the video stream address and convert the target video stream into video stream data of the WebRTC protocol.

[0142] The data receiving module 302 is used to receive the video stream data sent by the streaming media server through the RTCPeerConnection data channel, and determine the Video tag of the video stream data.

[0143] The projection determination module 303 is used to determine the camera parameters of the video fusion point in the video stream data, and determine the projection of the video fusion point in the three-dimensional GIS scene according to the camera parameters.

[0144] The fusion module 304 is used to construct the projected three-dimensional tile vector surface and assign the Video tag as a material object to the three-dimensional tile vector surface.

[0145] Optionally, the above-mentioned video stream and three-dimensional GIS scene fusion device further includes a construction module, and the construction module is used to:

[0146] Construct a streaming media server, a signaling server and a client based on the WebRTC protocol; wherein the streaming media server supports unified access to multiple video streaming protocols, and sends out real-time video streams using the WebRTC protocol by parsing video streams of multiple protocols; the signaling server is used for signaling interaction with the client and sending business instructions to the streaming media server, and while ensuring a long connection with the streaming media server, by completing signaling interaction and forwarding with the client, a data stream channel is established between the client and the streaming media server, and the transmission of the WebRTC protocol video stream is completed; the client includes a WebRTC receiving end and a three-dimensional GIS engine client, and the WebRTC receiving end establishes a connection with the signaling server through a WebSocket, sends signaling and business instructions, and after the signaling exchange is completed, an RTCPeerConnection data channel is established with the streaming media server to transmit video stream data of the WebRTC protocol, and the video stream data is displayed as data using the Video tag; the three-dimensional GIS client is used to load a three-dimensional oblique photography model, and construct a real-life three-dimensional scene and two- and three-dimensional projection vector surfaces;

[0147] Build a network penetration service and set up the STUN / TURN network penetration service of the ICE framework to achieve end-to-end data channel connection of WebRTC.

[0148] Optionally, the above-mentioned video stream and three-dimensional GIS scene fusion device may further include a connection module, and the connection module is used to:

[0149] Establishing a WebSocket connection with the signaling server;

[0150] Complete signaling exchange with the signaling server, and send network penetration address information and user information; wherein the signaling server completes signaling forwarding and completes signaling exchange with the streaming media server;

[0151] Establish an RTCPeerConnection data channel with the streaming media server.

[0152] Optionally, the projection determination module 303 is specifically configured to:

[0153] Requesting the signaling server for camera parameters of the video fusion point in the video stream data, or the client parsing the video stream data to determine the camera parameters of the video fusion point;

[0154] The camera parameters include one or more of the camera's spatial position, spatial posture, focal length, aspect ratio, viewing angle, and maximum viewing distance.

[0155] Optionally, the projection determination module 303 is specifically configured to:

[0156] Parsing the camera parameters into world coordinates, camera focal length, camera near plane distance, far plane distance, camera aspect ratio, heading angle, pitch angle, and roll angle to construct a frustum object;

[0157] Obtaining the far surface information of the frustum object, where the far surface is composed of four vertex coordinates;

[0158] The world coordinates of the camera are used as the starting point of the ray, and four rays are constructed with the four vertices of the long-range surface respectively. When the camera is tilted downward, the four rays can intersect with the ground to obtain four key points. The area surrounded by the four key points is the projection.

[0159] Optionally, the fusion module 304 is specifically used for:

[0160] A PolygonPrimitive surface is constructed in the GIS engine according to the four key points, and the ground-attaching properties of the PolygonPrimitive surface element are set to obtain a three-dimensional object vector surface in the three-dimensional scene; wherein the three-dimensional object vector is a surface element in which a two-dimensional vector surface completely fits on the three-dimensional element and has height information in the three-dimensional scene.

[0161] Optionally, the fusion module 304 is specifically used for:

[0162] The material of the three-dimensional tile vector surface is set as the Video tag of the video stream, the three-dimensional tile vector surface is filled with the video stream, and the video stream is merged with the three-dimensional GIS scene.

[0163] Optionally, the above-mentioned video stream and three-dimensional GIS scene fusion device also includes an update module, which is used to update the spatial position and posture of the viewing cone in the three-dimensional GIS scene in real time.

[0164] Optionally, the above-mentioned video stream and three-dimensional GIS scene fusion device further includes a three-dimensional model rendering module, and the three-dimensional model rendering module is used to:

[0165] Select a GIS engine;

[0166] Request the map tile data address, 3D model data address, terrain data address and other single model addresses around the video fusion point;

[0167] The data loading API interface provided by the GIS engine is used to render and visualize the obtained map tile data, three-dimensional model data, terrain data and other single models; in the process of loading the data in the three-dimensional GIS scene, the geographical location and orientation information of the GIS map and the coordinate system and direction information of the real-life model are obtained, and the real-life model is adjusted and aligned with the GIS map according to the geographical location and orientation information of the GIS map and the coordinate system and direction information of the real-life model.

[0168] It should be noted that the information interaction, execution process, etc. between the above-mentioned devices / units are based on the same concept as the method embodiment of the present application. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0169] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0170] The present application also provides a terminal device, which may include: at least one processor and a memory, wherein the memory stores a computer program that can be run on the at least one processor, and when the processor executes the computer program, the steps in any of the above method embodiments are implemented, such as Figure 2 Steps 201 to 204 in the illustrated embodiment. Alternatively, when the processor executes the computer program, the functions of each module / unit in the above-mentioned device embodiments are implemented, for example Figure 3 Functions of modules 301 to 304 are shown.

[0171] Exemplarily, the computer program may be divided into one or more modules / units, one or more modules / units are stored in a memory and executed by a processor to complete the present application. The one or more modules / units may be a series of computer program segments capable of completing specific functions, and the program segments are used to describe the execution process of the computer program in a terminal device.

[0172] The processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0173] The memory may be an internal storage unit of the terminal device, or an external storage device of the terminal device, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. The memory is used to store the computer program and other programs and data required by the terminal device. The memory may also be used to temporarily store data that has been output or is to be output.

[0174] The bus may be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the bus in the drawings of the present application is not limited to only one bus or one type of bus.

[0175] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium can at least include: any entity or device that can carry the computer program code to the camera / terminal device, recording medium, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, RandomAccess Memory), electric carrier signal, telecommunication signal and software distribution medium. For example, a USB flash drive, a mobile hard disk, a magnetic disk or an optical disk. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electric carrier signals and telecommunication signals.

[0176] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0177] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0178] In the embodiments provided in the present application, it should be understood that the disclosed devices / network equipment and methods can be implemented in other ways. For example, the device / network equipment embodiments described above are merely schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0179] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0180] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A method for fusing a video stream with a three-dimensional GIS scene, characterized in that: include: The client sends a video stream request to the streaming media server, wherein the video stream request includes a video stream address, and the video stream request is used to request the streaming media server to obtain a target video stream according to the video stream address, and convert the target video stream into video stream data of the WebRTC protocol; wherein the streaming media server supports unified access to multiple video stream protocols, and sends out real-time video streams in the WebRTC protocol by parsing video streams of multiple protocols; The client receives the video stream data sent by the streaming media server through the RTCPeerConnection data channel, and generates a Video tag for the video stream data; The client determines the camera parameters of the video fusion point in the video stream data, and determines the projection of the video fusion point in the three-dimensional GIS scene according to the camera parameters; The client constructs the projected three-dimensional tile vector surface, and assigns the Video tag as a material object to the three-dimensional tile vector surface; The method for fusing the video stream with the three-dimensional GIS scene also includes: Construct a streaming media server, a signaling server and a client based on the WebRTC protocol; wherein the signaling server is used for signaling interaction with the client and sending business instructions to the streaming media server. Under the condition of ensuring a long connection with the streaming media server, the signaling interaction and forwarding are completed with the client, a data stream channel is established between the client and the streaming media server, and the transmission of the WebRTC protocol video stream is completed; the client includes a WebRTC receiver and a 3D GIS engine client. The WebRTC receiver establishes a connection with the signaling server through WebSocket, sends signaling and business instructions, and after the signaling exchange is completed, an RTCPeerConnection data channel is established with the streaming media server to transmit the video stream data of the WebRTC protocol, and the video stream data is displayed as data with the Video tag; the 3D GIS engine client is used to load a 3D oblique photography model, and construct a real-life 3D scene and 2D and 3D projection vector surfaces; Build network penetration services and STUN / TURN network penetration services of the ICE framework to achieve end-to-end data channel connection for WebRTC; The client determines the camera parameters of the video fusion point in the video stream data, including: The client requests the signaling server for the camera parameters of the video fusion point in the video stream data, or the client parses the video stream data to determine the camera parameters of the video fusion point; The camera parameters include one or more of the camera's spatial position, spatial posture, focal length, aspect ratio, viewing angle, and maximum viewing distance; Determining the projection of the video fusion point in the three-dimensional GIS scene according to the camera parameters includes: Parsing the camera parameters into world coordinates, camera focal length, camera near plane distance, far plane distance, camera aspect ratio, heading angle, pitch angle, and roll angle to construct a frustum object; Obtaining the far surface information of the frustum object, where the far surface is composed of four vertex coordinates; Using the world coordinates of the camera as the starting point of the ray, four rays are constructed with the four vertices of the long-range surface respectively. When the camera is tilted downward, the four rays can intersect with the ground to obtain four key points. The area surrounded by the four key points is the projection. The client constructs the projected three-dimensional object vector surface, including: A PolygonPrimitive surface is constructed in the GIS engine according to the four key points, and the ground-attaching properties of the PolygonPrimitive surface element are set to obtain a three-dimensional object vector surface in the three-dimensional scene; wherein the three-dimensional object vector is a surface element in which a two-dimensional vector surface completely fits on the three-dimensional element and has height information in the three-dimensional scene.

2. The method for fusing a video stream with a three-dimensional GIS scene as claimed in claim 1, characterized in that: Before the client sends a video stream request to the streaming media server, the method for fusing the video stream with the three-dimensional GIS scene further includes: The client establishes a WebSocket connection with the signaling server; The client completes signaling exchange with the signaling server and sends network penetration address information and user information; The signaling server completes signaling forwarding and completes signaling exchange with the streaming media server; The client establishes an RTCPeerConnection data channel with the streaming media server.

3. The method for fusing a video stream with a three-dimensional GIS scene according to claim 1, characterized in that: The step of assigning the Video tag as a material object to the three-dimensional object vector surface includes: The material of the three-dimensional tile vector surface is set as the Video tag of the video stream, the three-dimensional tile vector surface is filled with the video stream, and the video stream is merged with the three-dimensional GIS scene.

4. The method for fusing a video stream with a three-dimensional GIS scene as claimed in claim 1, characterized in that: After assigning the Video tag as a material object to the three-dimensional object vector surface, the method for fusing the video stream with the three-dimensional GIS scene further includes: The spatial position and posture of the view cone in the 3D GIS scene are updated in real time.

5. The method for fusing a video stream with a three-dimensional GIS scene as claimed in claim 1, characterized in that: Before the client determines the camera parameters of the video fusion point in the video stream data, the method for fusing the video stream with the three-dimensional GIS scene further includes: Select a GIS engine; Request the map tile data address, 3D model data address, terrain data address and other single model addresses around the video fusion point; The data loading API interface provided by the GIS engine is used to render and visualize the obtained map tile data, three-dimensional model data, terrain data and other single models; in the process of loading the data in the three-dimensional GIS scene, the geographical location and orientation information of the GIS map and the coordinate system and direction information of the real-life model are obtained, and the real-life model is adjusted and aligned with the GIS map according to the geographical location and orientation information of the GIS map and the coordinate system and direction information of the real-life model.

6. A device for fusing video streams with three-dimensional GIS scenes, characterized in that: include: A request sending module, used for sending a video stream request to a streaming media server, wherein the video stream request includes a video stream address, and the video stream request is used for requesting the streaming media server to obtain a target video stream according to the video stream address, and convert the target video stream into video stream data of the WebRTC protocol; wherein the streaming media server supports unified access to multiple video stream protocols, and sends out real-time video streams in the WebRTC protocol by parsing video streams of multiple protocols; A data receiving module, used for receiving the video stream data sent by the streaming media server through the RTCPeerConnection data channel, and determining the Video tag of the video stream data; A projection determination module, used to determine the camera parameters of the video fusion point in the video stream data, and determine the projection of the video fusion point in the three-dimensional GIS scene according to the camera parameters; A fusion module, used for constructing the projected three-dimensional patch vector surface, and assigning the Video tag as a material object to the three-dimensional patch vector surface; A construction module is used to construct a streaming media server, a signaling server and a client based on the WebRTC protocol; wherein the signaling server is used for signaling interaction with the client and sending business instructions to the streaming media server. Under the condition of ensuring a long connection with the streaming media server, the signaling interaction and forwarding are completed with the client, a data stream channel is established between the client and the streaming media server, and the transmission of the WebRTC protocol video stream is completed; the client includes a WebRTC receiving end and a 3D GIS engine client. The WebRTC receiving end establishes a connection with the signaling server through WebSocket, sends signaling and business instructions, and after the signaling exchange is completed, an RTCPeerConnection data channel is established with the streaming media server to transmit the video stream data of the WebRTC protocol, and the video stream data is displayed as data with the Video tag; the 3D GIS engine client is used to load a 3D oblique photography model, and construct a real-life 3D scene and 2D and 3D projection vector surfaces; Build network penetration services and STUN / TURN network penetration services of the ICE framework to achieve end-to-end data channel connection for WebRTC; The projection determination module is specifically used for: The client requests the signaling server for the camera parameters of the video fusion point in the video stream data, or the client parses the video stream data to determine the camera parameters of the video fusion point; The camera parameters include one or more of the camera's spatial position, spatial posture, focal length, aspect ratio, viewing angle, and maximum viewing distance; The projection determination module is specifically used for: Parsing the camera parameters into world coordinates, camera focal length, camera near plane distance, far plane distance, camera aspect ratio, heading angle, pitch angle, and roll angle to construct a frustum object; Obtaining the far surface information of the frustum object, where the far surface is composed of four vertex coordinates; Using the world coordinates of the camera as the starting point of the ray, four rays are constructed with the four vertices of the long-range surface respectively. When the camera is tilted downward, the four rays can intersect with the ground to obtain four key points. The area surrounded by the four key points is the projection. The fusion module is specifically used for: A PolygonPrimitive surface is constructed in the GIS engine according to the four key points, and the ground-attaching properties of the PolygonPrimitive surface element are set to obtain a three-dimensional object vector surface in the three-dimensional scene; wherein the three-dimensional object vector is a surface element in which a two-dimensional vector surface completely fits on the three-dimensional element and has height information in the three-dimensional scene.

Images