Sluice and video camera virtual-real fusion method based on digital twin technology

Abstract

The invention discloses a sluice and video camera virtual-real fusion method based on a digital twinborn technology, and the method comprises the steps: carrying out the vectorization of monitoring data, building (structure) attribute data and other information hung on a BIM model through the BIM technology and the digital twinborn technology, and mapping the vectorized information to a corresponding position in real video data; video resources and digital twinborn scenes are fused, so that real images are reflected by virtual images; the gate state is automatically recognized through a video recognition algorithm and a gate dynamic simulation program, a gate model of a digital twinning scene is driven through state data, interactive animation rendering of the gate according to parameters is achieved, virtual images are reflected through real, and therefore the virtual and real data interaction depth and the virtual and real data interaction degree are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of virtual - reality combination, and is particularly applicable to a method for virtual - reality fusion of a sluice and a video camera based on digital twin technology. Background Art

[0002] With the proposal of the digital twin concept, virtual - reality fusion technology has emerged and has gradually been widely applied in actual scenarios. The virtual - reality fusion technology combines virtual information with the real environment to achieve two - way interaction and mapping linkage between the virtual world and the physical world. In the field of water conservancy projects, by combining the actual images captured by cameras with virtual digital twin scenarios, the image data is matched with real - scene data such as laser - scanned point clouds and oblique photography to reflect the operation status of water conservancy projects in real - time, complete scenario pre - rehearsals and optimizations, and provide data support for the refined management of water conservancy projects.

[0003] However, the following problems still exist: 1. The depth of virtual - reality data interaction is insufficient. In practical applications, the BIM model in the digital twin scenario usually only serves as a carrier of static data and lacks the ability to dynamically interact with real - time data. As a result, monitoring data and building (structure) data are difficult to accurately display in the video, causing the separation of virtual and real and unable to reflect the true operation status of physical objects in real - time.

[0004] 2. The difficulty of virtual - reality data interoperability and fusion is high. Due to the huge amount of video - stream data generated by video cameras, powerful computing capabilities and efficient data - processing algorithms are required to analyze these data in real - time to complete image recognition and parsing and extract useful information. In addition, the video - stream data also needs to be spatially aligned with the virtual model to achieve real - time synchronization of dynamic data. This requires extremely large computing resources and efficient data - processing algorithms.

[0005] In view of the above problems, the present invention proposes a method for virtual - reality fusion of a sluice and a video camera based on digital twin technology, which combines BIM technology and video fusion technology to improve the depth of virtual - reality combination and the degree of data interoperability, and ultimately provides strong support for the operation management decision - making of water conservancy projects or other industries. Summary of the Invention

[0006] The object of the present invention is to provide a method for virtual - reality fusion of a sluice and a video camera based on digital twin technology, which is used to solve the problems of insufficient depth of virtual - reality data interaction and high difficulty of virtual - reality data interoperability and fusion in digital twin technology.

[0007] To achieve the above object, the present invention adopts the following technical solutions: The method for virtual - reality fusion of a sluice and a video camera based on digital twin technology according to the present invention includes the following steps: S1, build the BIM model of the sluice gate; merge the BIM models of the sluice gate components according to the structure, and perform data encoding and associated attributes for the gate model code after each model is merged; S2, build a digital twin scene of the sluice gate based on the GIS data of the sluice gate location, and use digital twin technology to achieve dynamic simulation of the sluice gate opening and closing status; S3, obtaining video data of the sluice gate and camera parameters used for shooting, and identifying the gate opening and closing state parameters in the video data through an image recognition algorithm; S4, assigning the camera parameters to the digital twin scene of the sluice gate through the data interface, and driving the digital twin scene to simulate the real scene; S5, assigning the gate opening and closing state parameters to the sluice gate BIM model in the sluice gate digital twin scene through the data interface, and driving the sluice gate BIM model to simulate the real state of the sluice gate.

[0008] Furthermore, in step S1, the sluice gate BIM model includes a gate chamber, a gate, a bottom plate, and a side wall; the gate is used as a single sluice gate component to perform BIM model merging.

[0009] Furthermore, in step S2, a corresponding dynamic simulation program is developed for each opening and closing mode of the gate, which can be linked and controlled according to the gate opening and closing state parameters. Through advanced timeline editing technology, the dynamic simulation timeline and key frames are finely adjusted.

[0010] Furthermore, the opening and closing methods include vertical opening and closing, horizontal opening and closing, arc opening and closing, and lifting and rotating opening and closing.

[0011] Furthermore, in step S3, the camera parameters include the camera number, position, focal length, and viewing angle; the gate opening and closing status parameters include the gate number, opening method, number of opening holes, opening position, and opening value.

[0012] Furthermore, in step S4, the camera parameters are loaded for the first time, and the camera parameters are vectorized and fused and aligned with the digital twin scene. The pixel shader and mask texture technology are used to filter out the occluded pixels and perform constraints and feathering.

[0013] Furthermore, in step S4, the digital twin scene of the sluice gate is positioned according to the camera parameters so that the digital twin scene of the sluice gate is consistent with the video data of the sluice gate taken by the camera.

[0014] Furthermore, in step S5, the sluice gate BIM model is determined according to the gate number in the gate opening and closing state parameters, the corresponding dynamic simulation program is called according to the opening method, and the dynamic simulation program is driven to simulate the real-time state of the sluice gate according to the number of opening holes, opening position, and opening value.

[0015] Further, it also includes regularly obtaining the water level monitoring information before and after the gate, and transmitting it to the water surface models in front of and behind the gate in the digital twin scenario of the sluice through a data interface method to adjust the height of the water surface models.

[0016] The advantages of the present invention are as follows: By using BIM technology and digital twin technology, the monitoring data and building (structure) attribute data etc. attached to the BIM model are vectorized and then mapped to the corresponding positions in the real video data, and the video resources and the digital twin scenario are integrated to achieve virtual reflecting reality; then through video recognition algorithms and the gate dynamic simulation program, the gate state is automatically recognized, and the sluice model in the digital twin scenario is driven by the state data to realize the interactive animation rendering of the gate according to the parameters, achieving reality reflecting virtuality, thereby improving the depth of virtual-real data interaction and the degree of virtual-real data interoperability. Description of the Drawings

[0017] Figure 1 It is a flowchart of the method for virtual-real fusion of a sluice and a video camera based on digital twin technology of the present invention.

[0018] Figure 2 It is a flowchart of establishing the sluice BIM model in the method of the present invention.

[0019] Figure 3 It is a flowchart of constructing the digital twin scenario of the sluice in the method of the present invention.

[0020] Figure 4 It is a flowchart of driving the fusion of the digital twin scenario and the real scenario in the method of the present invention.

[0021] Figure 5 It is a flowchart of driving the sluice BIM model to simulate the real state of the sluice in the method of the present invention. Detailed Embodiments

[0022] Next, the technical solutions in the embodiments of the present invention will be described clearly and completely. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0023] As Figure 1 shown, the method for virtual-real fusion of a sluice and a video camera based on digital twin technology of the present invention includes the following steps: S1, building a sluice BIM model; merging the BIM models of the sluice components according to the structure, encoding the data for the merged gate model of each model and associating the attributes. Specifically as Figure 2 shown: Based on the digital design factory technology, the sluice BIM model is quickly established according to the real geographic coordinates, including the lock chamber, gate, bottom plate, side wall and other structures, and the BIM model of the sluice components is merged according to the structure. Among them, the gate structure is the core of the sluice, and the flow can be accurately adjusted through the opening and closing operation, thereby controlling the upstream and downstream water level difference. Each gate body model needs to be merged according to the single sluice component. The gate model after each model is merged needs to be data encoded and associated with attributes.

[0024] S2, the established sluice BIM model is combined with the GIS data of the sluice location to construct a sluice digital twin scene, and the digital twin technology is used to realize the dynamic simulation of the sluice opening and closing status. Figure 3 As shown in the figure, a corresponding dynamic simulation program is developed for each gate opening and closing mode, which can be linked and controlled according to the gate opening and closing state parameters. The dynamic simulation timeline and keyframes are finely adjusted through advanced timeline editing technology. The gate opening and closing modes include vertical opening and closing, horizontal opening and closing, arc opening and closing, and lifting and rotating opening and closing.

[0025] According to the gate position, the gate number, gate ID, gate attributes, the relationship between the gate and the gate opening and closing state parameters of the dynamic simulation program, and the relationship between the gate and the camera number and position are established for each gate. The gate attributes mainly include the gate length, width, height and other parameters, and the gate opening and closing state parameters include the gate number, opening method, number of opening holes, opening position, opening value, etc. The above data is managed through the database to realize the parameter drive of the gate dynamic simulation program.

[0026] S3, obtaining video data of the sluice gate and camera parameters used for shooting, and identifying the gate opening and closing state parameters in the video data through an image recognition algorithm. The camera parameters include the camera number, position, focal length, and viewing angle.

[0027] Based on the real-time video data of the sluice gate captured by the camera, the trained video recognition model is used to identify the opening and closing status parameters of the sluice gate, including the opening method, number of opening holes, opening position and gate opening value.

[0028] The video recognition model is obtained by learning a large amount of manually calibrated real-time video data of sluice gates using a deep learning algorithm.

[0029] S4, assign the camera parameters and the captured video data to the digital twin scene of the sluice gate through the data interface, and drive the fusion of the digital twin scene with the real scene. When the video data captured by the camera is loaded for the first time, the camera parameters and the digital twin scene need to be calibrated, and the pixel shader and mask texture technology are used to filter out the occluded pixels and constrain and feather the video images captured by the camera.

[0030] Specifically, it includes the following steps, such as Figure 4 as shown below: S4.1, regularly obtain the camera position parameters, locate the twin scene according to the camera position, and ensure that the object contours in the camera image and the twin scene are basically the same.

[0031] S4.2, perform pixel-by-pixel depth judgment through the Pixel Shader, filter out occluded pixels, ensure that the video image captured by the camera will not be projected onto the occluded area, and use the mask texture to constrain and feather the range of the video image captured by the camera.

[0032] S4.3, according to the association relationship between the camera and the gate, determine the corresponding BIM model of the gate through the camera number, attach the BIM attribute data of the gate to the video image captured by the camera, and position each attribute box to the corresponding building structure.

[0033] S5, assign the gate opening and closing state parameters to the BIM model of the sluice in the digital twin scene of the sluice through the data interface, and drive the BIM model of the sluice to simulate the real state of the sluice. Determine the BIM model of the sluice according to the gate number in the gate opening and closing state parameters, call the corresponding dynamic simulation program according to the opening method, and drive the dynamic simulation program to simulate the real-time state of the sluice according to the number of opening holes, opening position, and opening degree value. While simulating the state of the sluice in real time, regularly obtain the water level monitoring information before and after the gate, and transmit it to the water surface models in front of and behind the gate in the twin scene through the data interface method, and adjust the height of the water surface model in the scene according to the real water level height.

Claims

1. A virtual-real fusion method of a sluice and a video camera based on digital twin technology, characterized in that, The following steps are involved: S1, build the BIM model of the sluice gate; merge the BIM models of the sluice gate components according to the structure, and perform data encoding and associated attributes for the gate model code after each model is merged; S2, build a digital twin scene of the sluice gate based on the GIS data of the sluice gate location, and use digital twin technology to achieve dynamic simulation of the sluice gate opening and closing status; S3, obtaining video data of the sluice gate and camera parameters used for shooting, and identifying the gate opening and closing state parameters in the video data through an image recognition algorithm; S4, assigning the camera parameters to the digital twin scene of the sluice gate through the data interface, and driving the digital twin scene to simulate the real scene; S5, assigning the gate opening and closing state parameters to the sluice gate BIM model in the sluice gate digital twin scene through the data interface, and driving the sluice gate BIM model to simulate the real state of the sluice gate.

2. The virtual-real fusion method of the sluice and video camera based on digital twin technology according to claim 1, wherein: In step S1, the sluice gate BIM model includes a gate chamber, a gate, a bottom plate, and a side wall; the gate is used as a single sluice gate component to merge the BIM model.

3. The virtual-real fusion method of the sluice and the video camera based on the digital twin technology according to claim 1, wherein: In step S2, a corresponding dynamic simulation program is developed for each opening and closing mode of the gate, which can be linked and controlled according to the gate opening and closing state parameters. Through advanced timeline editing technology, the dynamic simulation timeline and key frames are finely adjusted.

4. The virtual-real fusion method of the sluice and video camera based on digital twin technology according to claim 3, wherein: The opening and closing methods include vertical opening and closing, horizontal opening and closing, arc opening and closing, and lifting and rotating opening and closing.

5. The virtual-real fusion method of the sluice and the video camera based on the digital twin technology according to claim 1, characterized in that: In step S3, the camera parameters include the camera number, position, focal length, and viewing angle; the gate opening and closing status parameters include the gate number, opening method, number of opening holes, opening position, and opening value.

6. The virtual-real fusion method of the sluice and the video camera based on the digital twin technology according to claim 1, characterized in that: In step S4, the camera parameters are loaded for the first time, and the camera parameters are vectorized and fused and aligned with the digital twin scene. The occluded pixels are filtered out and constrained and feathered using pixel shader and mask texture technology.

7. The virtual-real fusion method of the sluice and video camera based on the digital twin technology according to claim 1, wherein: In step S4, the digital twin scene of the sluice gate is positioned according to the camera parameters so that the digital twin scene of the sluice gate is consistent with the video data of the sluice gate taken by the camera.

8. The virtual-real fusion method of the sluice and the video camera based on the digital twin technology according to claim 1, characterized in that: In step S5, the sluice gate BIM model is determined according to the gate number in the gate opening and closing state parameters, the corresponding dynamic simulation program is called according to the opening method, and the dynamic simulation program is driven to simulate the real-time state of the sluice gate according to the number of opening holes, opening position, and opening value.

9. The virtual-real fusion method of the sluice and video camera based on digital twin technology according to claim 1, characterized in that: It also includes regularly obtaining water level monitoring information before and after the gate, transmitting it to the water surface models in front of and behind the gate in the digital twin scene of the sluice gate through a data interface, and adjusting the height of the water surface model.

Images