Power plant digital twin method and system based on real scene panoramic image and storage medium

Abstract

This application constructs a method, system, and storage medium for creating a digital twin of a power plant based on real-scene panoramic imagery. The method includes the following steps: Step S1: Acquire panoramic image data of the power plant site and perform panoramic image digitization processing to generate panoramic data; Step S2: Use a deep learning model to detect and identify equipment targets in the panoramic image data, extract equipment identification information, calculate the spatial location information of the equipment in the panoramic image data, and establish an association mapping based on the identification information and spatial location information. This application achieves low-cost, high-fidelity real-scene restoration through panoramic image digitization processing, achieves automatic equipment identification and annotation through a deep learning model, and achieves the fusion of visual data and ledger data through spatial location association mapping, thus realizing the rapid construction of a real-scene twin of a power plant.

Description

Technical Field

[0001] This invention relates to the field of industrial internet and digital twin technology, and more specifically, to a method, system and storage medium for digital twin of power plants based on real-scene panoramic images. Background Technology

[0002] With the deepening of digital transformation in the power industry, the construction of "smart power plants" has become an industry trend. Power plants have a large number of equipment, complex pipelines, and operate in high-temperature and high-pressure environments, placing higher demands on visualized operation and maintenance management. However, existing power plant visualization and management technologies mainly suffer from the following three types of deficiencies: First, there are two-dimensional configuration software programs that display equipment status using two-dimensional topology diagrams and process flow diagrams. Lacking spatial dimension information, these programs cannot intuitively reflect the physical appearance of the equipment and the surrounding environment. When alarms occur, maintenance personnel must physically visit the site to confirm, prolonging troubleshooting time. Second, while three-dimensional geometric modeling platforms can build 3D models, their construction costs are high (hundreds of thousands to millions of yuan per unit), and the cycle is lengthy (6-12 months). Furthermore, after on-site technical upgrades, model maintenance is difficult, texture details are distorted, and rendering performance bottlenecks are prominent, making widespread adoption on mobile devices difficult. Often, they are "affordable to build, but unaffordable to maintain." Third, traditional panoramic technology, while providing 360-degree real-world images, suffers from data silos, failing to integrate with production management systems and real-time databases. Equipment hotspot labeling relies on manual entry, resulting in a massive workload and high error rate for over 5000 points per unit, and it lacks intelligent diagnostic capabilities. Summary of the Invention

[0003] The technical problem to be solved by this invention is to provide a method, system and storage medium for digital twin of power plants based on real-scene panoramic images, which addresses the shortcomings of existing technologies such as lack of spatial awareness in two-dimensional configuration and high cost and difficulty in maintenance of three-dimensional modeling.

[0004] The technical solution adopted by this application to solve its technical problem is: constructing a digital twin method for power plants based on real-scene panoramic images, including the following steps: Step S1: Acquire panoramic image data of the power plant site, and perform panoramic image digitization processing on the panoramic image data to generate panoramic data; Step S2: The improved YOLOv8 target detection network is used to detect and identify the device targets in the panoramic image data, extract the identification information of the device, calculate the spatial location information of the device in the panoramic image data, and establish an association mapping based on the identification information and the spatial location information; The improved YOLOv8 target detection network has been optimized for power plant equipment detection as follows: A high-resolution detection head is added to the end of the feature pyramid structure to detect small target devices in the input image; A CBAM attention mechanism module is embedded in the backbone network to focus on key structural features of the device during feature extraction; The VarifocalLoss function is used as the classification loss function to address the class imbalance problem between foreground and background in object detection.

[0005] Furthermore, the panoramic image data includes: a sequence of power plant site images from multiple angles and exposures; In step S1, the panoramic image digitization processing of the panoramic image data includes: The multi-angle, multi-exposure image sequences are synthesized with high dynamic range and spherical stitching to generate panoramic images in equidistant columnar projection format; The generated panoramic image is subjected to multi-resolution pyramid slicing to obtain a hierarchical slice file.

[0006] Further, in step S2, the detection and identification of device targets in the panoramic image data using the improved YOLOv8 target detection network, and the extraction of the device's identification information, includes: Determine whether the resolution of the panoramic image data exceeds a preset threshold: If the number of images exceeds the limit, a sliding window is used to segment the panoramic image data to generate several sub-images, and the sub-images are then input into the improved YOLOv8 object detection network. If the value does not exceed the limit, the panoramic image data is directly input into the improved YOLOv8 target detection network as a whole image. The device target in the input image is identified, and after perspective correction is performed on the region of the device target, it is input into the CRNN character recognition model to extract the functional location code on the nameplate of the device target as the identification information of the device.

[0007] Furthermore, the improved YOLOv8 target detection network has been optimized for power plant equipment detection as follows: A high-resolution detection head is added to the end of the feature pyramid structure to detect small target devices in the input image; A CBAM attention mechanism module is embedded in the backbone network to focus on key structural features of the device during feature extraction; The VarifocalLoss function is used as the classification loss function to address the class imbalance problem between foreground and background in object detection.

[0008] Furthermore, after perspective correction is performed on the region of the device target, it is input into a CRNN character recognition model to extract the functional location code on the nameplate of the device target, which serves as the device's identification information, including: The detected region of the device target is subjected to perspective correction processing, and the perspective-corrected region of the device target is input into the CRNN text recognition model. The CRNN text recognition model consists of three parts: CNN convolutional neural network, RNN recurrent neural network and CTC connectionist temporal classification decoder. The CRNN character recognition model predicts the corresponding character sequence end-to-end from the perspective-corrected region image of the device target, and extracts the functional location code on the nameplate of the device target, the functional location code including KKS code; The extracted functional location code is used as the device's identification information.

[0009] Furthermore, the improved YOLOv8 object detection network also outputs the type and bounding box coordinates of each detected device; the calculation of the spatial location information of the device in the panoramic image data includes: Based on the bounding box coordinates and the coordinates of the sub-image in the panoramic image data, the planar pixel coordinates of the device in the panoramic image data are calculated. Based on the planar pixel coordinates, the spherical hotspot coordinates of the device in the panoramic image data are calculated using a coordinate inverse calculation algorithm, and used as the spatial location information.

[0010] Furthermore, the bounding box coordinates include the minimum and maximum x-coordinates, minimum and maximum y-coordinates of the bounding box in the input image; The step of calculating the planar pixel coordinates of the device in the panoramic image data based on the bounding box coordinates and the coordinates of the sub-image in the panoramic image data includes: In the formula, , These are the x-coordinate and y-coordinate of the center of the bounding box in the sub-image, respectively; The horizontal coordinate of the device in the panoramic image data is the planar pixel coordinate. The vertical coordinate of the device in the panoramic image data is the planar pixel coordinate. The x-coordinate of the sub-image at the reference point in the panoramic image data. The ordinate of the sub-image is the reference point in the panoramic image data.

[0011] Further, calculating the spatial position information of the device in the panoramic image data based on the planar pixel coordinates includes: According to the planar pixel coordinates The coordinates are mapped to panoramic spherical hotspot coordinates through a coordinate inverse calculation algorithm; The panoramic image data adopts an equidistant cylindrical projection format, with a width of W and a height of H; The panoramic spherical hotspot coordinates include longitude. and latitude The calculation formula is as follows: The longitude The latitude The device's identification information and device type are associated to establish a mapping between the device's spatial location information in the panoramic image data and the identification information.

[0012] Furthermore, the method also includes: Receive user-input identification information; Based on the identity information, obtain the panoramic spherical hotspot coordinates of the device and switch the viewpoint to the panoramic view location of the device. Based on the device's identification information, obtain the device's real-time operating data; The device's identification information and real-time operating data are displayed in a visual floating window.

[0013] This application also provides a power plant digital twin system based on real-scene panoramic images, including a processor and a memory storing a computer program. When the processor executes the computer program, it implements the steps of any of the above-described power plant digital twin methods based on real-scene panoramic images.

[0014] This application also provides a storage medium storing a computer program that, when executed, implements the steps of any of the above-described methods for creating a digital twin of a power plant based on real-scene panoramic images.

[0015] The beneficial effects of this invention are as follows: This application constructs a method for digital twinning power plants based on real-scene panoramic images, including the following steps: Step S1: Acquire panoramic image data of the power plant site, and perform panoramic image digitization processing on the panoramic image data to generate panoramic data; Step S2: Use a deep learning model to detect and identify equipment targets in the panoramic image data, extract the identification information of the equipment, calculate the spatial location information of the equipment in the panoramic image data, and establish an association mapping based on the identification information and the spatial location information. This application achieves low-cost, high-fidelity real-scene restoration through panoramic image digitization processing, achieves automatic equipment identification and annotation through a deep learning model, and achieves the fusion of visual data and ledger data through spatial location association mapping, thereby solving the problems of high cost of 3D modeling, low annotation efficiency, and data silos in the prior art, and realizing the rapid construction and intelligent management of real-scene twinning of power plants. Attached Figure Description

[0016] The present invention will be further described below with reference to the accompanying drawings and embodiments. In the accompanying drawings: Figure 1 This is a logic flowchart of a power plant digital twin method based on real-scene panoramic images, as an example. Figure 2 This is a schematic diagram of the front-end and back-end interaction process of the power plant virtual reality twin platform; Figure 3 This is a diagram showing the system architecture and subsystem interaction relationships of a power plant virtual twin platform. Detailed Implementation

[0017] To provide a clearer understanding of the technical features, objectives, and effects of the present invention, specific embodiments of the invention are now described in detail with reference to the accompanying drawings. In the following description, specific details such as particular structures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of the invention. However, those skilled in the art will understand that the invention can also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known devices, circuits, and methods are omitted to avoid unnecessary detail.

[0018] like Figure 1 As shown, Figure 1 This is a logic flowchart of a power plant digital twin method based on real-scene panoramic imagery, as implemented in one embodiment. The provided power plant digital twin method based on real-scene panoramic imagery includes the following steps: Step S1: Acquire panoramic image data of the power plant site, and perform panoramic image digitization processing on the panoramic image data to generate panoramic data; It's important to note in this step that the quality of the panoramic image data directly impacts the accuracy of subsequent device recognition and spatial mapping. The core of this step lies in converting the raw, multi-source images into panoramic data of a unified format with spatial reference through digital processing, providing a standardized data foundation for subsequent intelligent annotation.

[0019] Specifically, the panoramic image data is obtained by capturing images at key locations on the power plant site using professional-grade 8K panoramic cameras. Each shooting point yields a sequence of images taken from multiple angles and with multiple exposures. During shooting, a tripod is used to stabilize the camera, and GIS positioning equipment is used to record the spatial coordinates of each shooting point to ensure coordinate consistency across multiple scenes. After acquisition, the image sequences are transmitted to an image processing server for subsequent digital processing.

[0020] Step S2: A deep learning model is used to detect and identify equipment targets in the panoramic image data, extract the equipment identification information, and calculate the spatial location information of the equipment in the panoramic image data. An association mapping is established based on the identification information and spatial location information, including: Among them, the improved YOLOv8 target detection network has been optimized for power plant equipment detection as follows: a high-resolution detection head is added to the end of the feature pyramid structure to detect small target equipment in the input image; a CBAM attention mechanism module is embedded in the backbone network to focus on the key structural features of the equipment during feature extraction; and the VarifocalLoss function is used as the classification loss function to deal with the problem of foreground and background class imbalance in target detection.

[0021] It is important to note that the key to this step lies in accurately identifying minute equipment targets (such as nameplates, small valves, etc.) from ultra-high resolution panoramic images and establishing a precise mapping relationship between equipment identification information and spatial location. This invention addresses the characteristics of power plant scenarios, such as large differences in equipment target size and dense concentration of small targets. It optimizes the YOLOv8 network structure and designs a combined algorithm integrating sliding window segmentation, YOLOv8 detection, OCR recognition, and coordinate inverse calculation.

[0022] Specifically, the system first determines whether the resolution of the panoramic image data exceeds a preset threshold. If it does, a sliding window segmentation strategy is employed, using a set window size and overlap step to perform sliding segmentation, generating a series of sub-images to ensure that any small target can appear completely within at least one window. Then, the image blocks are input into an improved YOLOv8 object detection network for device detection. After perspective correction of the detected nameplate area, it is input into a CRNN text recognition model to extract the functional location code. Finally, a coordinate inverse calculation algorithm converts the detection box coordinates into panoramic spherical hotspot coordinates, which are then associated with and stored with the device identification information. An auditing interface is also provided, displaying the AI-automated annotation results and allowing annotators to quickly verify, correct, or supplement the recognition results.

[0023] Furthermore, the panoramic image data includes: a sequence of power plant site images from multiple angles and exposures; in step S1, the panoramic image data is processed by panoramic image digitization, which includes: performing high dynamic range synthesis and spherical stitching on the multi-angle, multi-exposure image sequence to generate a panoramic image in equidistant columnar projection format; and performing multi-resolution pyramid slicing on the generated panoramic image to obtain a hierarchical slice file.

[0024] Specifically, by utilizing high dynamic range static panoramic imaging and pyramid slicing technology, high-fidelity reconstruction of the power plant's on-site environment can be achieved at an extremely low cost, approximately one-tenth that of 3D modeling, thus solving the problems of high cost and severe texture distortion associated with traditional 3D modeling. In detail, VR panoramic data production and display is responsible for high-definition acquisition of panoramic images, spherical stitching, high dynamic range light and shadow fusion, multi-resolution pyramid slicing processing, and high-performance streaming rendering on the web.

[0025] In one embodiment, the VR panoramic image acquisition and multi-resolution tiling process is divided into two core stages: First, the VR panoramic image acquisition stage, which involves planning and confirming acquisition points, preparing professional panoramic cameras and equipment, shooting from multiple angles and exposures on site, data transmission and integrity verification, image stitching and denoising, and HDR synthesis, ultimately generating a 360° spherical panoramic image; then, the multi-resolution tiling process, which involves first establishing an image pyramid structure, then dividing the image into regular tiles, performing tile encoding and compression optimization (such as WebP / JPEG format), generating tile indexes and metadata, storing them in a hierarchical database, and deploying them in the cloud, ultimately enabling the platform to load on demand and provide streaming services. This process fully covers the entire process from on-site shooting to web-browsable panoramic images. The panoramic image data consists of multi-angle, multi-exposure power plant on-site image sequences. First, equidistant cylindrical projection panoramic images are generated through HDR synthesis and spherical stitching. Then, multi-resolution pyramid slicing processing is performed to obtain hierarchical slice files. This not only ensures the acquisition quality and restoration effect of the panoramic images, but also improves loading efficiency and interactive smoothness through layered slicing technology, providing efficient and stable technical support for panoramic visualization applications in scenarios such as power plant real-world twins.

[0026] Further, in step S2, an improved YOLOv8 target detection network is used to detect and identify device targets in the panoramic image data, and the device identification information is extracted by: determining whether the resolution of the panoramic image data exceeds a preset threshold; if it does, a sliding window is used to cut the panoramic image data to generate several sub-images, and the sub-images are input into the improved YOLOv8 target detection network; if it does not exceed the threshold, the panoramic image data is directly input into the improved YOLOv8 target detection network as a whole image; the device targets in the input image are identified, and after perspective correction of the area of ​​the device targets, the functional location code on the nameplate of the device targets is extracted as the device identification information.

[0027] Specifically, the preset threshold is preferably 1920×1080 pixels. When the panoramic image resolution exceeds this threshold, a 640×640 pixel sliding window with a step size of 320 pixels (50% overlap) is used to cut the panoramic image into several sub-image blocks. It should be noted that this sliding window cutting and detection process is suitable for offline batch processing scenarios and does not have high real-time requirements. Therefore, even if hundreds of sub-image blocks are generated, the system can still complete the full image processing within a few minutes. Each sub-image block is sequentially input into the improved YOLOv8 object detection network for inference. After perspective correction, the detected equipment target area is input into the CRNN text recognition model to extract the functional location code (i.e., KKS code) on the nameplate. This functional location code serves as the unique identification information of the equipment and is used to associate it with the equipment ledger in the production management system. The model training data comes from historical inspection photos, equipment manuals, and on-site specially collected images, covering equipment under different lighting, angles, and dirt conditions to ensure the model's generalization ability. The detection logic adopts a coarse screening and fine judgment process: first, the improved YOLOv8 network quickly locates the potential device area to complete the coarse screening; then, the detection area is subjected to perspective correction and fed into the CRNN model to finely recognize the nameplate text to complete the fine judgment, thus balancing detection efficiency and recognition accuracy. First, YOLOv8 locates the potential device area, and then the area image is subsequently recognized.

[0028] Furthermore, the improved YOLOv8 target detection network has been optimized for power plant equipment detection as follows: a high-resolution detection head is added at the end of the feature pyramid structure to detect small target devices in the input image; A CBAM attention mechanism module is embedded in the backbone network to focus on key structural features of the device during feature extraction; the VarifocalLoss function is used as the classification loss function to handle the class imbalance problem between foreground and background in object detection.

[0029] Specifically, a high-resolution detection head is added at the end of the feature pyramid structure. This head has a higher resolution than the original head, preserving more detailed features of small targets and effectively improving the recall rate of small target devices such as small instruments, nameplates, and valve handles. The embedded CBAM attention mechanism module sequentially uses a channel attention submodule and a spatial attention submodule. The channel attention submodule learns the weights of different feature channels to highlight key structural features of the equipment, while the spatial attention submodule learns the weights of different positions in the feature map to suppress interference from complex industrial backgrounds. The two submodules are used in series, allowing the model to focus more on key structural features of the equipment, such as the nameplate text area and instrument dials, while suppressing interference from the complex industrial background of the power plant. A variable focus loss function (Varifocal Loss) is adopted instead of the standard Focal Loss function for classification. Varifocal Loss dynamically scales the loss contribution of positive and negative samples. It weights the loss of negative samples based on the predicted class confidence, while scaling the loss of positive samples according to the actual intersection-over-union ratio (IoU). This better handles the extreme foreground-background class imbalance problem in object detection, improving the model's classification accuracy in sparse equipment target scenes. Testing shows that the improved YOLOv8 model achieves an average accuracy of 92% on power plant equipment detection tasks.

[0030] Furthermore, after perspective correction of the detected equipment target area, the data is input into a CRNN character recognition model to extract the functional location code on the equipment target's nameplate, which serves as the equipment's identification information, including: The detected equipment target area is subjected to perspective correction processing. The perspective-corrected equipment target area is then processed by the CRNN character recognition model, which consists of three parts: CNN convolutional neural network, RNN recurrent neural network, and CTC connectionist temporal classification decoder. The CRNN character recognition model predicts the corresponding character sequence end-to-end from the perspective-corrected equipment target area image and extracts the functional location code on the nameplate of the equipment target. The functional location code includes the KKS code. The extracted functional location code is used as the identification information of the equipment.

[0031] Specifically, perspective correction is performed on the detected equipment bounding box area to eliminate text distortion caused by the shooting angle. Perspective correction corrects the tilted nameplate image into a frontal view image by detecting the four corner points of the nameplate and calculating the perspective transformation matrix. The corrected image is then input into the CRNN character recognition model, which consists of three parts: a CNN convolutional neural network, an RNN recurrent neural network, and a CTC connectionist temporal classification decoder. The CNN part uses a ResNet34 architecture to extract the image's feature sequences, the RNN part uses a bidirectional LSTM to perform sequence modeling on the feature sequences to capture the contextual dependencies between characters, and the CTC decoder aligns the predicted sequences output by the RNN to the final character sequences, thus predicting the corresponding character sequences end-to-end and extracting the functional location code on the equipment nameplate. The functional location code includes a KKS code, which serves as the unique identification information for the equipment and is used to associate it with the equipment ledger in the production management system.

[0032] Furthermore, the improved YOLOv8 object detection network also outputs the type of each detected device and the coordinates of its bounding box; the spatial location information of the device in the panoramic image data is calculated as follows: based on the bounding box coordinates and combined with the coordinates of the sub-images in the panoramic image data, the planar pixel coordinates of the device in the panoramic image data are calculated; based on the planar pixel coordinates, the spherical hotspot coordinates of the device in the panoramic image data are calculated using a coordinate inverse calculation algorithm as the spatial location information.

[0033] Specifically, the improved YOLOv8 object detection network outputs the bounding box coordinates of each detected device along with the device type. When calculating the spatial location information of the device in the panoramic image data, it first converts the bounding box coordinates into planar pixel coordinates of the device in the panoramic image data, and then calculates the spatial location information of the device in the panoramic image data based on the planar pixel coordinates.

[0034] Furthermore, the bounding box coordinates include the minimum and maximum x-coordinates, minimum and maximum y-coordinates of the bounding box in the input image; Based on the bounding box coordinates and the coordinates of the sub-image in the panoramic image data, the planar pixel coordinates of the device in the panoramic image data are calculated as follows: In the formula, , These are the x and y coordinates of the bounding box center in the sub-image, respectively; This represents the horizontal pixel coordinate of the device within the panoramic image data. This represents the vertical pixel coordinates of the device within the panoramic image data. The x-coordinate of the sub-image at the reference point in the panoramic image data. The ordinate of the sub-image is the reference point in the panoramic image data.

[0035] Specifically, the bounding box coordinates include the minimum and maximum x-coordinates, minimum and maximum y-coordinates of the bounding box in the input image. The center point of the bounding box is used as the representative position of the device, and the planar pixel coordinates of the device in the panoramic image data are the center coordinates of the bounding box. Let the coordinates of the top-left corner of the sub-image in the panoramic image be... The center coordinates of the detection box in the subgraph are Then through the formula and The coordinates of the point in the panoramic plane pixel coordinate system were calculated. When panoramic image data is input directly as a whole image without being cropped by a sliding window, and All are zero, that is equal , equal .

[0036] Furthermore, the spatial location information of the device in the panoramic image data calculated based on the planar pixel coordinates includes: Based on planar pixel coordinates The coordinates are mapped to panoramic spherical hotspot coordinates through a coordinate inverse calculation algorithm; The panoramic image data uses an equidistant cylindrical projection format, with a width of W and a height of H; Panoramic spherical hotspot coordinates include longitude and latitude The calculation formula is as follows: Longitude ,latitude The device's identification information and device type are associated to establish a mapping between the device's spatial location information and identification information in the panoramic image data.

[0037] Specifically, based on the planar pixel coordinates, a coordinate inverse calculation algorithm is used to map them to panoramic spherical hotspot coordinates. The panoramic image in equidistant cylindrical projection format uses a longitude-latitude coordinate system, where the horizontal direction corresponds to longitude and the vertical direction corresponds to latitude. Let the resolution of the panoramic image be... Then longitude and latitude The calculation formula is: , The calculated longitude and latitude This refers to the hotspot location of the device on the panoramic sphere. This hotspot is automatically associated with the device's functional location code identified by OCR and the device type identified by YOLO, and written into the database to establish a mapping between the device and its spatial location information, thus completing automated annotation.

[0038] Furthermore, the method also includes: receiving user-inputted identification information; obtaining the panoramic spherical hotspot coordinates of the device based on the identification information, and switching the viewpoint to the panoramic position of the device; obtaining the device's real-time operating data based on the device's identification information; and displaying the device's identification information and real-time operating data in a visual floating window. The real-time operating data includes temperature, pressure, flow rate, valve opening, current, and voltage.

[0039] Specifically, the front-end interface provides a search input box, allowing users to enter the device name or functional location code for fuzzy searching. The system uses an inverted index from Elasticsearch to match the input identification information in the database and retrieve the device's panoramic spherical hotspot coordinates (longitude). and latitude The system drives the panoramic player to smoothly rotate to the specified coordinate position and highlights the device in the image. Simultaneously, the system retrieves real-time operational data from the time-series database based on the device's identification information, including temperature, pressure, and valve opening. This data is then displayed as a visual floating window next to the device, allowing maintenance personnel to quickly understand its status.

[0040] In one specific embodiment, in the power plant reality twin platform, the mapping between the spherical coordinates and planar pixel coordinates of the panoramic image is the core foundation for realizing the bidirectional conversion between the three-dimensional scene and the two-dimensional image: the platform constructs a virtual sphere centered on the panoramic camera, mapping the three-dimensional reality of the power plant onto the sphere. Any point on the sphere is determined by its longitude λ (horizontal direction, with a value range of [-π,π]) and latitude. (Vertical direction, value range [-π / 2, π / 2]) is defined as .

[0041] It should be noted that the positive Y-axis of the panoramic image coordinate system points vertically downwards, opposite to the positive latitude direction (upwards) of the spherical image. Therefore, a corresponding transformation is required in the latitude calculation formula. Specifically: Orthographic projection (spherical to planar): Mapping spherical latitude and longitude coordinates to pixel coordinates of a two-dimensional panoramic image. Since the coordinate systems are oriented in opposite directions, the transformation formula is: Or equivalently, when the Y-axis of the image coordinate system is downward, y = ×H Negative values ​​are required.

[0042] Back projection (planar to spherical): This method restores the pixel coordinates of an image to spherical latitude and longitude, taking into account differences in coordinate system orientation. The calculation formula is as follows: Where x and y are planar pixel coordinates, and W and H are the width and height of the panoramic image.

[0043] This two-way mapping mechanism provides key technical support for panoramic visualization, scene roaming, and precise interaction of power plant real-world scenarios.

[0044] Explanation of formula symbols: λ: Spherical longitude, representing the horizontal viewing angle, with a value range of [-π,π] (radians), corresponding to the horizontal orientation of the scene; : Spherical latitude, representing the vertical pitch angle, with a value range of [-π / 2, π / 2] (radians), corresponding to the vertical viewpoint of the scene; W: Width of the 2D panoramic image (unit: pixels); H: Height of the 2D panoramic image (unit: pixels); x: The horizontal component of the planar pixel coordinates, with a value range of [0, W), representing the horizontal position of the pixel in the image; y: The vertical component of the planar pixel coordinates, with a value range of [0, H), representing the vertical position of the pixel in the image; π: Pi constant, used for normalization calculations of angles and radians.

[0045] The significant advantages of this application are: construction costs reduced by approximately 90%, requiring only 50,000-100,000 RMB per unit and eliminating the need for expensive modeling labor; construction cycle shortened by over 90%, from 6-12 months to 2-3 weeks for rapid deployment; visual fidelity achieving 100% real-scene reproduction, clearly showing details such as rust, nameplates, and oil levels, and supporting remote visual inspection; extremely low barrier to update and maintenance, requiring only frontline staff to retake and replace images without the need for professional modification of the model; equipment annotation efficiency increased by more than 10 times, achieving an accuracy rate of 95% through AI automatic recognition and batch association; fault diagnosis capabilities with secondary analysis logic, filtering out 80% of instrument false alarms and reducing unnecessary attendance; and superior client performance, opening instantly in a regular browser (less than 1 second), requiring no high-end graphics card, no plugins, and compatible with mobile devices.

[0046] This application also provides a power plant digital twin system based on real-scene panoramic images, including a processor and a memory storing a computer program. When the processor executes the computer program, it implements the steps of any of the above-mentioned power plant digital twin methods based on real-scene panoramic images.

[0047] like Figure 2 As shown, the system of this invention also includes a real-scene twin management backend and a comprehensive display frontend. The real-scene twin management backend provides functions such as panoramic map editing, resource version management, annotation data review workflow, and user access control. Its interaction sequence with the display frontend is as follows: Figure 2 As shown, the integrated front-end provides immersive interactive roaming, fuzzy search based on Elasticsearch, floating window display of device details, visual rendering of alarms, and shortest path navigation.

[0048] The overall network topology of the system of this invention is as follows: Figure 3 As shown, strictly following the power industry's cybersecurity zoning principles, it mainly consists of the following key areas: The on-site data acquisition layer is located at the bottom layer and includes panoramic acquisition equipment (professional-grade 8K panoramic cameras and GIS positioning equipment) to acquire high-definition static image data and spatial coordinates of the site. It also includes an industrial control network that connects the transmitters, actuators, PLC controllers and DCS / SIS servers in the field. These devices continuously generate real-time operating data such as temperature, pressure, flow rate and valve opening.

[0049] The data transmission layer uses a high-bandwidth industrial fiber optic ring network as the backbone to ensure high-speed transmission of massive image data and real-time measurement point data. A one-way physical isolation gateway is deployed between the production control area (Zone III) and the management information area (Zone IV). This gateway uses a "ferry" mechanism to ensure that data can only flow unidirectionally from the high-density area to the low-density area, physically cutting off the attack path of the external network to the production control system.

[0050] The data processing and storage layer serves as the core backend, comprising an image processing cluster (deploying high-performance GPU servers, running frameworks such as OpenCV and PyTorch, and executing image stitching, HDR synthesis, and tiling algorithms), an AI inference cluster (deploying trained YOLOv8 and CRNN models for batch offline inference of panoramic images), and a hybrid database cluster. The relational database stores device ledgers, device function location codes, user permissions, and scene metadata; the time-series database stores DCS historical trend data with high compression ratio and supports millisecond-level queries; the object storage contains massive amounts of panoramic image tile files and model files; and the search engine provides high-performance inverted index retrieval of device information.

[0051] The application service layer is based on the Spring Boot microservice architecture and includes user authentication service (OAuth2), panoramic roaming service, search service, etc.

[0052] The presentation layer includes a web client and a mobile client. The web client is based on HTML5 / WebGL technology and supports mainstream browsers, while the mobile client is adapted to iOS / Android tablets and industrial PDAs and supports on-site offline inspection.

[0053] This application also provides a storage medium storing a computer program that, when executed, implements the steps of any of the above-described methods for creating a digital twin of a power plant based on real-scene panoramic images.

[0054] It is understood that the above embodiments only illustrate preferred embodiments of the present invention, and their descriptions are relatively specific and detailed, but they should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can freely combine the above technical features without departing from the concept of the present invention, and can also make several modifications and improvements, all of which fall within the protection scope of the present invention. Therefore, all equivalent transformations and modifications made with respect to the scope of the claims of the present invention should fall within the scope of the claims of the present invention.

Claims

1. A method for creating a digital twin of a power plant based on real-scene panoramic images, characterized in that, Includes the following steps: Step S1: Acquire panoramic image data of the power plant site, and perform panoramic image digitization processing on the panoramic image data to generate panoramic data; Step S2: The improved YOLOv8 target detection network is used to detect and identify the device targets in the panoramic image data, extract the identification information of the device, calculate the spatial location information of the device in the panoramic image data, and establish an association mapping based on the identification information and the spatial location information; The improved YOLOv8 target detection network has been optimized for power plant equipment detection as follows: A high-resolution detection head is added to the end of the feature pyramid structure to detect small target devices in the input image; A CBAM attention mechanism module is embedded in the backbone network to focus on key structural features of the device during feature extraction; The VarifocalLoss function is used as the classification loss function to address the class imbalance problem between foreground and background in object detection.

2. The power plant digital twin method based on real-scene panoramic images according to claim 1, characterized in that, The panoramic image data includes: a sequence of power plant site images from multiple angles and exposures; In step S1, the panoramic image digitization processing of the panoramic image data includes: The multi-angle, multi-exposure image sequences are synthesized with high dynamic range and spherical stitching to generate panoramic images in equidistant columnar projection format; The generated panoramic image is subjected to multi-resolution pyramid slicing to obtain a hierarchical slice file.

3. The power plant digital twin method based on real-scene panoramic images according to claim 1, characterized in that, In step S2, the improved YOLOv8 target detection network is used to detect and identify device targets in the panoramic image data, and the identification information of the device is extracted, including: Determine whether the resolution of the panoramic image data exceeds a preset threshold: If the number of images exceeds the limit, a sliding window is used to segment the panoramic image data to generate several sub-images, and the sub-images are then input into the improved YOLOv8 object detection network. If the value does not exceed the limit, the panoramic image data is directly input into the improved YOLOv8 target detection network as a whole image. The device target in the input image is identified, and after perspective correction is performed on the region of the device target, the functional location code on the nameplate of the device target is extracted as the identification information of the device.

4. The power plant digital twin method based on real-scene panoramic images according to claim 3, characterized in that, After perspective correction of the target area, the data is input into a CRNN character recognition model to extract the functional location code on the target's nameplate, which serves as the device's identification information, including: The detected region of the device target is subjected to perspective correction processing, and the perspective-corrected region of the device target is input into the CRNN text recognition model. The CRNN text recognition model consists of three parts: CNN convolutional neural network, RNN recurrent neural network and CTC connectionist temporal classification decoder. The CRNN character recognition model predicts the corresponding character sequence end-to-end from the perspective-corrected region image of the device target, and extracts the functional location code on the nameplate of the device target, the functional location code including KKS code; The extracted functional location code is used as the device's identification information.

5. The power plant digital twin method based on real-scene panoramic images according to claim 3, characterized in that, The improved YOLOv8 object detection network also outputs the type and bounding box coordinates of each detected device; the calculation of the spatial location information of the device in the panoramic image data includes: Based on the bounding box coordinates and the coordinates of the sub-image in the panoramic image data, the planar pixel coordinates of the device in the panoramic image data are calculated. Based on the planar pixel coordinates, the spherical hotspot coordinates of the device in the panoramic image data are calculated using a coordinate inverse calculation algorithm, and used as the spatial location information.

6. The method for digital twin of a power plant based on real-scene panoramic images according to claim 5, characterized in that, The bounding box coordinates include the minimum and maximum x-coordinates, minimum and maximum y-coordinates of the bounding box in the input image; The step of calculating the planar pixel coordinates of the device in the panoramic image data based on the bounding box coordinates and the coordinates of the sub-image in the panoramic image data includes: In the formula, , These are the x-coordinate and y-coordinate of the center of the bounding box in the sub-image, respectively; The horizontal coordinate of the device in the panoramic image data is the planar pixel coordinate. The vertical coordinate of the device in the panoramic image data is the planar pixel coordinate. The x-coordinate of the sub-image at the reference point in the panoramic image data. The ordinate of the sub-image is the reference point in the panoramic image data.

7. The method for digital twinning power plants based on real-scene panoramic images according to claim 5, characterized in that, The step of calculating the spatial position information of the device in the panoramic image data based on the planar pixel coordinates includes: Based on the planar pixel coordinates, a coordinate inverse calculation algorithm is used to map them to panoramic spherical hotspot coordinates; The panoramic image data adopts an equidistant cylindrical projection format, with a width of W and a height of H; The panoramic spherical hotspot coordinates include longitude. and latitude The calculation formula is as follows: The longitude The latitude The device's identification information and device type are associated to establish a mapping between the device's spatial location information in the panoramic image data and the identification information.

8. The method for digital twinning power plants based on real-scene panoramic images according to claim 1, characterized in that, The power plant digital twin method based on real-scene panoramic images also includes: Receive user-input identification information; Based on the identity information, obtain the panoramic spherical hotspot coordinates of the device and switch the viewpoint to the panoramic view location of the device. Based on the device's identification information, obtain the device's real-time operating data; The device's identification information and real-time operating data are displayed in a visual floating window.

9. A power plant digital twin system based on real-scene panoramic images, comprising a processor and a memory storing a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the power plant digital twin method based on real-scene panoramic images as described in any one of claims 1-8.

10. A storage medium, characterized in that, The storage medium stores a computer program, which, when executed, implements the steps of the power plant digital twin method based on real-scene panoramic images as described in any one of claims 1 to 8.

Images