Digital twin pumped storage power station three-dimensional real-time modeling fusion method

Abstract

The invention discloses a digital twin pumped storage power station three-dimensional real-time modeling fusion method which comprises the following steps: (1) obtaining three-dimensional modeling basic image data through a video monitoring system, and constructing a three-dimensional modeling basic image database; (2) performing real-time three-dimensional modeling and model fusion; and (3) system integration and real-time updating. According to the invention, deep learning is utilized to improve the image quality of a video monitoring system, and the precision of a three-dimensional model in a digital twin system can be improved; and new equipment in the image is identified by combining deep learning, real-time modeling of single equipment is completed on the premise of not increasing the model acquisition cost, and low cost, simplification and intelligentization of refined three-dimensional real-time modeling are realized.

Description

Technical Field

[0001] The invention relates to a three-dimensional real-time modeling fusion method for a digital twin pumped-storage power station, and belongs to the technical field of image processing and digital twin. Background Art

[0002] The emergence of digital twin technology provides a means of virtual mapping entities for the operation and maintenance management of pumped storage power stations to achieve efficient and accurate management needs. Traditional 3D modeling includes forward modeling based on 2D drawings and reverse modeling through oblique photography and other methods. However, the workload of 2D drawing modeling is huge and cannot be updated in real time later. Modeling by oblique photography and other methods is simple and fast, but the model accuracy is low and submodules cannot be distinguished in the model. How to effectively integrate real-time video streams with 3D models to achieve virtual-real synchronization still faces technical challenges.

[0003] Patent CN 114280955 A discloses a smart hydropower plant system and architecture method based on domain information model and digital twin, including a control center and a data twin server, which performs digital twin simulation on energy storage and water transmission, system power generation and energy discharge tail water to improve the power plant repair capability and early warning function. However, the three-dimensional model in the invention is fixed, and data can only be updated through sensors in the physical system during real-time operation, but the model cannot be updated in real time, which will lead to poor applicability of the later three-dimensional model.

[0004] Patent CN 112506156 B discloses a visualized digital twin factory server embedded in a distributed control system, which includes at least a two-dimensional module and a three-dimensional module. The two-dimensional module is used to display the controlled object in a two-dimensional manner so that the operator can control its state; the three-dimensional module is used to display the controlled object in a three-dimensional manner so that the operator can display the controlled object currently controlled or viewed. By setting the two-dimensional module and the three-dimensional module in parallel, the workload of the three-dimensional module is reduced, thereby achieving accurate, fast and real-time control and viewing of two-dimensional and three-dimensional graphics. However, the system data acquisition and model update rely on smart wearable devices, which require smart devices to transmit location and operation information in real time. The practicality needs to be improved and the degree of intelligence is poor.

[0005] Patent CN 116664780 A discloses a three-dimensional dynamic modeling method for digital twins of auxiliary equipment in coal-fired power plants. By emitting ultrasonic waves to penetrate and scan the auxiliary equipment, virtual three-dimensional comprehensive modeling of the auxiliary equipment is achieved. At the same time, according to the actual operation process of the auxiliary equipment, tracer particles are adsorbed on the internal dynamic structure of the auxiliary equipment to build a corresponding sub-real-time model to achieve dynamic and refined three-dimensional modeling. However, the dynamic modeling process system is complex, the modeling cost is high, the rate is low, and the practicality is poor. Summary of the invention

[0006] In order to overcome the above-mentioned shortcomings, the present invention proposes a fusion method of three-dimensional real-time modeling of a digital twin pumped-storage power station combined with video surveillance. The basic image data of three-dimensional modeling is obtained through video surveillance, and the corresponding equipment is identified by machine learning. The image quality is improved, and high-quality real-time modeling of a single device is completed, thereby realizing low cost and simplification of refined three-dimensional real-time modeling.

[0007] The technical solution of the present invention is as follows: A three-dimensional real-time modeling fusion method for a digital twin pumped storage power station, characterized by comprising the following steps: (1) Obtain basic image data for 3D modeling through a video surveillance system and build a basic image database for 3D modeling; The video surveillance system is configured as follows: Deploy multi-angle, high-resolution camera arrays, drone aerial photography equipment, and laser point cloud scanning equipment. The camera array is used for routine monitoring, the drone aerial photography equipment is used for supplementary data collection under special working conditions, and the laser point cloud scanning equipment is used for detailed 3D modeling of indoor equipment. (2) Real-time 3D modeling and model fusion First, the basic image data for 3D modeling is preprocessed through the image preprocessing module, and the preprocessed image is sent to the feature extraction module to extract key features; the 3D reconstruction module receives the feature information and generates the corresponding 3D model; the newly generated 3D model is registered and fused with the existing digital twin model through the model fusion module; Cyclic training: In an unsupervised environment, the image quality and 3D reconstruction accuracy are continuously optimized through adversarial learning between the generator and the discriminator until satisfactory performance is achieved; Ultimately, the images in the video stream are corrected in real time, and the digital twin model is updated to achieve accurate mapping and monitoring of the real world; (3) System integration and real-time update All modules are integrated into a unified digital twin platform. By receiving video streams from each camera in real time, the image processing and 3D modeling processes are automatically triggered, and the digital twin model is updated in real time. At the same time, when a camera is damaged, images obtained by cameras at other nearby angles are used for alternative imaging. The system has self-diagnosis and optimization functions, and can automatically adjust camera parameters according to environmental changes to optimize image quality and the accuracy of the 3D model.

[0008] Preferably, the image preprocessing module in the above step (2) is specifically as follows: Perform preprocessing operations such as denoising, enhancement, and distortion correction on the original video images to prepare high-quality images for subsequent processing, build an unsupervised image-to-image algorithm, extract video image features from the database through the encoder, and then use the generator to generate high-quality images. Use the discriminator to judge the generated images until the discriminator considers the generated images to be true, complete the network training, and then correct the real-time video images.

[0009] Preferably, the feature extraction module in the above step (2) is specifically as follows: Offset prediction: First, the offset of each convolution kernel is predicted through an additional branch, usually another convolution layer; the output of this branch has the same size as the feature map, but the number of channels is twice the number of convolution kernels, which is used for offsets in the x and y directions; Offset addition: Add the predicted offset to the default sampling position to get the new sampling point position; Bilinear interpolation: Since the offset may produce coordinates with non-integer values, bilinear interpolation is used to calculate the feature values ​​at these locations; Standard convolution: Finally, use these adjusted sampling points to perform convolution operations to obtain the final feature map; Feature fusion: Use two parallel convolutional layers to extract information from the channel dimension and spatial dimension of the feature map respectively, and then fuse the two outputs; Generate an attention map: Generate a weight map of the same size as the feature map through convolution, pooling and activation functions. Each value of the map represents the importance of the corresponding position. Feature weighting: multiply the generated attention weight map with the original feature map to weight the features of different regions; Use deformable convolutions in the early or middle layers of the network to handle the pose and deformation of objects. In the later stages of the network, when the resolution of the feature map is reduced, apply spatial attention mechanisms to focus on the most relevant information and reduce background interference.

[0010] Preferably, the three-dimensional reconstruction module in the above step (2) includes the following steps: Data collection and preprocessing: RGB images: RGB images of the target scene taken from multiple angles; Infrared image: Obtaining the corresponding infrared image can provide temperature distribution information; Depth information: Depth maps are captured by devices such as depth cameras or LiDAR to provide distance information; Multimodal feature extraction: Use convolutional neural networks to extract visual features from RGB images and infrared images respectively, preprocess the depth map, and then convert it into a dense 3D point cloud or voxel grid form; Modal Fusion: Design a multimodal fusion network, which can be a simple concatenation, gating mechanism or a more complex interactive attention module to fuse features from RGB, infrared and depth information; Spatial consistency correction: Ensure that data from different modalities are spatially aligned, which needs to be done through point cloud registration or projection of depth maps to RGB images; NeRF optimization: The fused features are used as additional inputs to the NeRF network or embedded into the NeRF volume rendering equation; During the training process, in addition to the loss of RGB images, the loss of depth and infrared images can also be introduced to guide NeRF to better fit the 3D structure and thermal characteristics; High-precision reconstruction: Using the fused multimodal information, NeRF can learn a more accurate representation of the 3D scene, including the shape, texture, and physical properties of objects. To achieve high fidelity, it is necessary to increase the resolution of NeRF or use upsampling and refinement techniques to improve the details. Post-processing and evaluation: Post-process the reconstructed 3D scene, including removing noise, optimizing surface texture and color, and evaluating the accuracy of the reconstruction, using standard 3D matching and comparison techniques, including average distance error or texture similarity metrics between point clouds; New view synthesis: Once the model is trained, NeRF is used to synthesize new RGB images from arbitrary viewpoints and simultaneously generate corresponding infrared images and depth maps.

[0011] Preferably, in the three-dimensional model fusion step in the above step (2), a model registration technology based on the ICP algorithm is used to accurately align and fuse the newly generated three-dimensional model with the existing digital twin model to maintain the consistency of the model.

[0012] A three-dimensional real-time modeling system for a digital twin pumped-storage power station includes a video monitoring system, an image processing and equipment identification module, a reverse modeling and three-dimensional model fusion module, a voxel point repair module, and a digital twin system update module. The modules perform corresponding operations according to the above method.

[0013] The beneficial effects of the present invention are: Using deep learning to improve the image quality of video surveillance systems can improve the accuracy of three-dimensional models in digital twin systems; By combining deep learning to identify new devices in images, real-time modeling of a single device can be completed without increasing the cost of model acquisition, achieving low-cost, simple and intelligent refined three-dimensional real-time modeling. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 This is the system architecture diagram; Figure 2 This is a schematic diagram of camera layout. DETAILED DESCRIPTION

[0015] The present invention is further described below in conjunction with the examples. The following examples are only used to more clearly illustrate the technical solutions of the present invention, and are not intended to limit the scope of protection of the present invention. Example

[0016] like Figure 1 , 2 As shown, a three-dimensional real-time modeling fusion method for a digital twin pumped storage power station includes the following steps: (1) Obtain basic image data for 3D modeling through a video surveillance system and build a basic image database for 3D modeling; The video surveillance system is configured as follows: Deploy multi-angle, high-resolution camera arrays, drone aerial photography equipment, and laser point cloud scanning equipment. The camera array is used for routine monitoring, the drone aerial photography equipment is used for supplementary data collection under special working conditions, and the laser point cloud scanning equipment is used for detailed 3D modeling of indoor equipment. 1. Real-time monitoring of reservoirs and mountain environments: Camera arrangement: HD panoramic cameras are deployed around the reservoir and inlet and outlet areas to cover the entire water area and surrounding mountains to monitor water surface fluctuations, water level changes and mountain stability. At the same time, remote cameras with pan / tilt cameras are installed high up in the mountains to overlook the entire reservoir and capture large-scale environmental changes.

[0017] Pixel enhancement method: Use image super-resolution algorithms, such as SRGAN (SuperResolution Generative Adversarial Networks) or ESPCN (Efficient Sub-Pixel Convolutional Neural Network) based on deep learning, to convert low-resolution images into high-resolution images, increase the number of pixels, and improve image details.

[0018] 2. Real-time monitoring of pumping stations and power plants: Camera placement: High-resolution cameras are installed at key locations inside the pump station and power plant, such as pump units, generator units, and control rooms, for 24-hour uninterrupted monitoring. At the same time, cameras are also installed around the outside to monitor personnel activities and equipment transportation.

[0019] Pixel enhancement method: Use image enhancement technologies, such as contrast stretching and histogram equalization, to improve image clarity. For surveillance videos under low-light conditions, use night vision enhancement algorithms, such as optical flow and background subtraction, to ensure nighttime surveillance effects.

[0020] (2) Real-time 3D modeling and model fusion: The real-time video stream enters the system and is first preprocessed by the image preprocessing module. The preprocessed image is sent to the feature extraction module to extract key features. The 3D reconstruction module receives the feature information and generates the corresponding 3D model. The newly generated 3D model is registered and fused with the existing digital twin model through the model fusion module. Cyclic training: In an unsupervised environment, the adversarial learning of the generator and the discriminator is used to continuously optimize the image quality and the accuracy of 3D reconstruction until satisfactory performance is achieved. Finally, the image in the video stream is corrected in real time, and the digital twin model is updated to achieve accurate mapping and monitoring of the real world.

[0021] Specific structure of the algorithm: Image preprocessing module: Perform preprocessing operations such as denoising, enhancement, and distortion correction on the original video image to prepare high-quality images for subsequent processing. Construct an unsupervised image-to-image algorithm, extract the video image features in the database through the encoder, then use the generator to generate high-quality images, and use the discriminator to discriminate the generated images until the discriminator considers the generated images to be true, completing the network training, and then correcting the real-time video images.

[0022] Feature extraction module: Convolutional neural network (CNN) is used to extract image features and identify the contours and textures of key equipment and environmental elements. Offset prediction: First, an additional branch (usually another convolution layer) is used to predict the offset of each convolution kernel. The output of this branch has the same size as the feature map, but the number of channels is twice the number of convolution kernels (for offsets in the x and y directions). Offset addition: The predicted offset is added to the default sampling position to obtain the new sampling point position. Bilinear interpolation: Since the offset may produce coordinates with non-integer values, bilinear interpolation is used to calculate the feature values ​​at these positions. Standard convolution: Finally, these adjusted sampling points are used for convolution operations to obtain the final feature map. Feature fusion: Usually two parallel convolution layers are used to extract information from the channel dimension and spatial dimension of the feature map respectively, and then the two outputs are fused. Generate attention map: A weight map of the same size as the feature map is generated through convolution, pooling and activation functions (such as sigmoid). Each value of the map represents the importance of the corresponding position. Feature weighting: Multiply the generated attention weight map with the original feature map to weight the features of different regions. Use deformable convolutions in the early or middle layers of the network to handle the pose and deformation of objects. In the later stages of the network, when the resolution of the feature map is reduced, apply spatial attention mechanisms to focus on the most relevant information and reduce background interference.

[0023] 3D reconstruction module: Image to 3D model conversion algorithm based on deep learning, data collection and preprocessing: RGB image: Take RGB images of the target scene from multiple angles. Infrared image: Obtain the corresponding infrared image, which can provide temperature distribution information. Depth information: Capture the depth map through a depth camera or LiDAR and other devices to provide distance information. Multimodal feature extraction: Use convolutional neural network (CNN) to extract visual features from RGB images and infrared images respectively. Preprocess the depth map, such as filling holes, smoothing, and then convert it into a dense 3D point cloud or voxel grid form. Modal fusion: Design a multimodal fusion network, which can be a simple splicing, gating mechanism or a more complex interactive attention module to fuse features from RGB, infrared and depth information. Spatial consistency correction: Ensure that data from different modalities are spatially aligned, which may require point cloud registration or projection of depth map to RGB image. NeRF optimization: Use the fused features as additional input to the NeRF network or embed them into NeRF's volume rendering equation. During the training process, in addition to the loss of RGB images, losses of depth and infrared images can also be introduced to guide NeRF to better fit the 3D structure and thermal properties. High-precision reconstruction: Using the fused multimodal information, NeRF can learn a more accurate representation of the 3D scene, including the shape, texture, and physical properties of objects. To achieve high fidelity, it may be necessary to increase the resolution of NeRF or use upsampling and refinement techniques to improve the details. Post-processing and evaluation: Post-process the reconstructed 3D scene, such as removing noise and optimizing surface texture and color. To evaluate the accuracy of the reconstruction, standard 3D matching and comparison techniques can be used, such as the average distance error between point clouds or texture similarity indicators. New perspective synthesis: Once the model is trained, NeRF can be used to synthesize new RGB images from any perspective, and corresponding infrared images and depth maps can also be generated.

[0024] Model fusion module: Uses model registration technology based on the ICP (Iterative Closest Point) algorithm to accurately align and fuse the newly generated 3D model with the existing digital twin model to maintain model consistency.

[0025] (3) System integration and real-time update: All the above modules are integrated into a unified digital twin platform, which automatically triggers the image processing and 3D modeling process by receiving video streams from various cameras in real time, and updates the digital twin model in real time. At the same time, the system has self-diagnosis and optimization functions, which can automatically adjust camera parameters according to environmental changes, optimize image quality and 3D model accuracy.

[0026] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A three-dimensional real-time modeling fusion method for a digital twin pumped storage power station, characterized in that The steps include: (1) Obtain basic image data for 3D modeling through a video surveillance system and build a basic image database for 3D modeling; The video surveillance system is configured as follows: Deploy multi-angle, high-resolution camera arrays, drone aerial photography equipment, and laser point cloud scanning equipment. The camera array is used for routine monitoring, the drone aerial photography equipment is used for supplementary data collection under special working conditions, and the laser point cloud scanning equipment is used for detailed 3D modeling of indoor equipment. (2) Real-time 3D modeling and model fusion Firstly, the basic image data of 3D modeling is preprocessed by the image preprocessing module, and the preprocessed image is sent to the feature extraction module to extract key features; The 3D reconstruction module receives the feature information and generates the corresponding 3D model; the newly generated 3D model is registered and fused with the existing digital twin model through the model fusion module; Cyclic training: In an unsupervised environment, the image quality and 3D reconstruction accuracy are continuously optimized through adversarial learning between the generator and the discriminator until satisfactory performance is achieved; Ultimately, the images in the video stream are corrected in real time, and the digital twin model is updated to achieve accurate mapping and monitoring of the real world; (3) System integration and real-time update All modules are integrated into a unified digital twin platform. By receiving video streams from each camera in real time, the image processing and 3D modeling processes are automatically triggered, and the digital twin model is updated in real time. At the same time, the system has self-diagnosis and optimization functions, which can automatically adjust camera parameters according to environmental changes, optimize image quality and the accuracy of the 3D model.

2. A three-dimensional real-time modeling fusion method for a digital twin pumped storage power station according to claim 1, characterized in that The image preprocessing module in step (2) is specifically as follows: Perform preprocessing operations such as denoising, enhancement, and distortion correction on the original video image to prepare high-quality images for subsequent processing, build an unsupervised image-to-image algorithm, extract video image features from the database through the encoder, and then use the generator to generate high-quality images. Use the discriminator to judge the generated image until the discriminator considers the generated image to be true, complete the network training, and then correct the real-time video image.

3. A three-dimensional real-time modeling fusion method for a digital twin pumped storage power station according to claim 1, characterized in that The feature extraction module in step (2) is as follows: Offset prediction: First, the offset of each convolution kernel is predicted through an additional branch, usually another convolution layer; the output of this branch has the same size as the feature map, but the number of channels is twice the number of convolution kernels, which is used for offsets in the x and y directions; Offset addition: Add the predicted offset to the default sampling position to get the new sampling point position; Bilinear interpolation: Since the offset may produce coordinates with non-integer values, bilinear interpolation is used to calculate the feature values ​​at these locations; Standard convolution: Finally, use these adjusted sampling points to perform convolution operations to obtain the final feature map; Feature fusion: Use two parallel convolutional layers to extract information from the channel dimension and spatial dimension of the feature map respectively, and then fuse the two outputs; Generate an attention map: Generate a weight map of the same size as the feature map through convolution, pooling and activation functions. Each value of the map represents the importance of the corresponding position. Feature weighting: multiply the generated attention weight map with the original feature map to weight the features of different regions; Use deformable convolutions in the early or middle layers of the network to handle the pose and deformation of objects. In the later stages of the network, when the resolution of the feature map is reduced, apply spatial attention mechanisms to focus on the most relevant information and reduce background interference.

4. A three-dimensional real-time modeling fusion method for a digital twin pumped storage power station according to claim 1, characterized in that: The three-dimensional reconstruction module in step (2) includes the following steps: Data collection and preprocessing: RGB images: RGB images of the target scene taken from multiple angles; Infrared image: Obtaining the corresponding infrared image can provide temperature distribution information; Depth information: Depth maps are captured by devices such as depth cameras or LiDAR to provide distance information; Multimodal feature extraction: Use convolutional neural networks to extract visual features from RGB images and infrared images respectively, preprocess the depth map, and then convert it into a dense 3D point cloud or voxel grid form; Modal Fusion: Design a multimodal fusion network, which can be a simple concatenation, gating mechanism or a more complex interactive attention module to fuse features from RGB, infrared and depth information; Spatial consistency correction: Ensure that data from different modalities are spatially aligned, which needs to be done through point cloud registration or projection of depth maps to RGB images; NeRF optimization: The fused features are used as additional inputs to the NeRF network or embedded into the NeRF volume rendering equation; During the training process, in addition to the loss of RGB images, the loss of depth and infrared images can also be introduced to guide NeRF to better fit the 3D structure and thermal characteristics; High-precision reconstruction: Using the fused multimodal information, NeRF can learn a more accurate representation of the 3D scene, including the shape, texture, and physical properties of objects. To achieve high fidelity, it is necessary to increase the resolution of NeRF or use upsampling and refinement techniques to improve the details. Post-processing and evaluation: Post-process the reconstructed 3D scene, including removing noise, optimizing surface texture and color, and evaluating the accuracy of the reconstruction, using standard 3D matching and comparison techniques, including average distance error or texture similarity metrics between point clouds; New view synthesis: Once the model is trained, NeRF is used to synthesize new RGB images from arbitrary viewpoints and simultaneously generate corresponding infrared images and depth maps.

5. A three-dimensional real-time modeling fusion method for a digital twin pumped storage power station according to claim 1, characterized in that: In the three-dimensional model fusion step in step (2), a model registration technology based on the ICP algorithm is used to accurately align and fuse the newly generated three-dimensional model with the existing digital twin model to maintain the consistency of the model.

6. A three-dimensional real-time modeling system for a digital twin pumped-storage power station, comprising a video monitoring system, an image processing and equipment identification module, a reverse modeling and three-dimensional model fusion module, a voxel point repair module and a digital twin system update module, wherein the modules perform corresponding operations according to the methods described in claims 1 to 5.

Images