Power station camera pose automatic calibration method and device based on digital twinning

Abstract

The application provides a power station camera pose automatic calibration method and device based on digital twinning, which comprises the following steps: obtaining a three-dimensional digital twinning model of a power station, and establishing a calibration reference digital twinning model; arranging a plurality of reference two-dimensional code targets on a fixed structure, and binding each reference two-dimensional code target with a global three-dimensional coordinate in the calibration reference digital twinning model; controlling a camera to be calibrated to collect a real-time image on site, identifying the reference two-dimensional code targets in the real-time image on site, and extracting pixel coordinates of the target centers of the reference two-dimensional code targets; selecting effective images; based on the mapping relationship between the three-dimensional space points and the two-dimensional image points formed by the global three-dimensional coordinates of the reference two-dimensional code targets and the corresponding pixel coordinates in the effective images, the pose external parameters of the camera to be calibrated in the global coordinate system are solved. In this way, full-automatic calibration is realized, the calibration efficiency and the calibration accuracy are improved, and the problem of incompatible coordinate systems of the inspection system can be solved.

Description

Technical Field

[0001] This invention relates to the field of pose calibration technology, and in particular to a method and apparatus for automatic pose calibration of a power station camera based on digital twins. Background Technology

[0002] As pumped-storage power stations play an increasingly important role in new power systems, their operation and maintenance models are rapidly evolving towards unmanned and minimally manned operations. Video inspection, with its significant advantage of reusing existing industrial television systems within the power station, has become a core technological path for unmanned inspection. Related technologies typically achieve automatic analysis of spatial visibility between the camera and the inspection point, as well as automatic calculation of preset points. However, the efficient and accurate implementation of this method highly depends on the accuracy of the camera pose parameters in the digital twin model.

[0003] In actual engineering projects, industrial television cameras generally have significant installation deviations after installation. These deviations directly lead to problems such as failure of visibility analysis, deviations in preset point calculations, and inaccurate execution of inspection strategies. They have become a key technical bottleneck restricting the implementation of large-scale power plant video inspection projects. Summary of the Invention

[0004] In view of this, the purpose of this invention is to provide a method and device for automatic calibration of the pose of a power station camera based on digital twins, which realizes fully automatic calibration, improves calibration efficiency and accuracy, and can solve the problem of incompatibility of coordinate systems in inspection systems.

[0005] In a first aspect, embodiments of the present invention provide an automatic pose calibration method for a power station camera based on digital twins. The method includes: acquiring a three-dimensional digital twin model of the power station; extracting a fixed-structure three-dimensional entity model and the precise three-dimensional coordinates of the three-dimensional entity model in a global coordinate system from the three-dimensional digital twin model of the power station to establish a calibration reference digital twin model; deploying multiple reference QR code targets on the fixed structure and binding each reference QR code target to the global three-dimensional coordinates in the calibration reference digital twin model to form a global positioning reference; controlling the camera to be calibrated to acquire real-time images of the site, identifying the reference QR code targets in the real-time images of the site and extracting the pixel coordinates of the target center of the reference QR code targets; selecting valid images from the real-time images of the site based on pre-set filtering rules; and calculating the pose extrinsic parameters of the camera to be calibrated in the global coordinate system based on the mapping relationship between the global three-dimensional coordinates and the corresponding pixel coordinates of the reference QR code targets in the valid images, thereby completing the pose calibration of the camera to be calibrated.

[0006] In a preferred embodiment of the present invention, the above-mentioned arrangement of multiple reference QR code targets on a fixed structure includes: arranging multiple reference QR code targets on a fixed structure in a non-coplanar manner, such that the positions of the reference QR code targets cover the entire field of view of the camera to be calibrated.

[0007] In a preferred embodiment of the present invention, the above-mentioned selection of valid images from real-time images based on pre-set filtering rules includes: determining the number of non-collinear reference QR code targets in each frame of real-time images; selecting the frame with the most targets as a valid image; if there are multiple frames with the same number of targets, then selecting the frame with no local occlusion of the reference QR code target and the highest image clarity of the target center as a valid image.

[0008] In a preferred embodiment of the present invention, after selecting valid images from real-time images based on pre-set filtering rules, the method further includes: for a fixed-focal-length bullet camera, performing camera intrinsic parameter calibration to obtain the intrinsic parameter matrix and distortion coefficients of the camera to be calibrated; for a rotatable zoom PTZ camera, selecting multiple focal lengths at equal intervals within the zoom range, performing camera intrinsic parameter calibration for each focal length, and obtaining the intrinsic parameter matrix and distortion coefficients of the camera to be calibrated; establishing a focal length-intrinsic parameter mapping table based on the intrinsic parameter matrix and distortion coefficients; after performing distortion correction on the valid image based on the intrinsic parameter matrix and distortion coefficients obtained from the focal length-intrinsic parameter mapping table with the current focal length, extracting the pixel coordinates of the target center again to obtain the target pixel coordinates, so as to provide a mapping relationship between three-dimensional spatial points and two-dimensional image points based on the global three-dimensional coordinates of the reference QR code target in the valid image and the corresponding target pixel coordinates.

[0009] In a preferred embodiment of the present invention, the above-mentioned calculation of the pose extrinsic parameters of the camera to be calibrated in the global coordinate system includes: taking the mapping relationship between three-dimensional spatial points and two-dimensional image points as input, and using a pose calculation algorithm to calculate the rotation matrix and translation vector of the camera to be calibrated in the global coordinate system; transforming based on the rotation matrix to obtain the initial horizontal angle and initial pitch angle corresponding to the initial normal vector of the camera to be calibrated; and transforming based on the translation vector to obtain the three-dimensional spatial coordinates of the optical center of the camera to be calibrated in the global coordinate system.

[0010] In a preferred embodiment of the present invention, for a rotatable zoom PTZ camera, the method further includes: performing zero-position reference calibration on the camera to be calibrated to obtain the zero-position reference for the rotation of the gimbal; performing horizontal rotation range calibration on the camera to be calibrated to obtain the actual horizontal rotation angle range of the gimbal; performing pitch rotation range calibration on the camera to be calibrated to obtain the actual pitch rotation angle range of the gimbal; and binding the actual horizontal rotation angle range and the actual pitch rotation angle range of the gimbal to the zero-position reference respectively.

[0011] In a preferred embodiment of the present invention, after completing the pose calibration of the camera to be calibrated, the method further includes: selecting at least one reference QR code target that has not participated in the pose extrinsic parameter calculation as a verification target; calculating the verification pixel coordinates of the target center of the verification target based on the pose extrinsic parameter matrix obtained from the calibration; determining the reprojection error between the verification pixel coordinates and the actual pixel coordinates corresponding to the verification target; and determining that the calibration is valid if the reprojection error is less than or equal to a preset pixel threshold.

[0012] In a preferred embodiment of the present invention, the method further includes: dividing the power plant area into multiple calibration units according to functional areas, and deploying multiple common reference QR code targets at the boundaries of adjacent calibration units; the common reference QR code targets are simultaneously bound to the global three-dimensional coordinates of two adjacent units, serving as the positioning references for the two adjacent units; each calibration unit is equipped with an independent set of reference QR code targets, which, combined with the common reference QR code targets, form the global positioning reference for the calibration unit; image acquisition, effective image selection, and pose extrinsic parameter calculation are sequentially performed on all cameras to be calibrated within each calibration unit; the coordinate system consistency verification of adjacent calibration units is completed based on the common reference QR code targets to correct coordinate deviations between calibration units; after all calibration units are completed, the pose parameters of all cameras are synchronously updated to the three-dimensional digital twin model of the power plant.

[0013] Secondly, embodiments of the present invention also provide an automatic pose calibration device for a power station camera based on digital twins. The device includes: a model acquisition module, used to acquire a three-dimensional digital twin model of the power station, extract a three-dimensional solid model of a fixed structure and the precise three-dimensional coordinates of the three-dimensional solid model in the global coordinate system from the three-dimensional digital twin model of the power station, and establish a calibration reference digital twin model; a coordinate binding module, used to deploy multiple reference QR code targets on the fixed structure and bind each reference QR code target to the global three-dimensional coordinates in the calibration reference digital twin model to form a global positioning reference; a pixel coordinate extraction module, used to control the camera to be calibrated to acquire real-time images of the site, identify the reference QR code targets in the real-time images of the site and extract the pixel coordinates of the target center of the reference QR code targets; an effective image selection module, used to select effective images from the real-time images of the site based on pre-set filtering rules; and a pose calibration module, used to calculate the pose extrinsic parameters of the camera to be calibrated in the global coordinate system based on the mapping relationship between the global three-dimensional coordinates and the corresponding pixel coordinates of the reference QR code targets in the effective images, thereby completing the pose calibration of the camera to be calibrated.

[0014] Thirdly, embodiments of the present invention also provide an electronic device, including a processor and a memory, wherein the memory stores computer-executable instructions that can be executed by the processor, and the processor executes the computer-executable instructions to implement the above-described method for automatic pose calibration of a power station camera based on digital twins.

[0015] The embodiments of the present invention bring the following beneficial effects: This invention provides a method and apparatus for automatic pose calibration of a power station camera based on digital twins. The method involves acquiring a 3D digital twin model of the power station, extracting a fixed-structure 3D entity model and its precise 3D coordinates in a global coordinate system from the 3D digital twin model, and establishing a calibration reference digital twin model. Multiple reference QR code targets are deployed on the fixed structure, and each reference QR code target is bound to the global 3D coordinates in the calibration reference digital twin model to form a global positioning reference. The camera to be calibrated is controlled to acquire real-time images from the site, and the reference QR code targets in the real-time images are identified, with the pixel coordinates of the target center extracted. Valid images are selected from the real-time images based on pre-set filtering rules. Based on the mapping relationship between 3D spatial points and 2D image points formed by the global 3D coordinates and corresponding pixel coordinates of the reference QR code targets in the valid images, the pose extrinsic parameters of the camera to be calibrated in the global coordinate system are calculated, completing the pose calibration of the camera. This method achieves fully automatic calibration, improves calibration efficiency and accuracy, and solves the problem of coordinate system incompatibility in inspection systems.

[0016] Other features and advantages of this disclosure will be set forth in the following description, or some features and advantages may be inferred from the description or determined without doubt, or may be learned by practicing the techniques described above.

[0017] To make the above-mentioned objects, features and advantages of this disclosure more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0018] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0019] Figure 1 A flowchart of an automatic pose calibration method for a power station camera based on digital twin, provided for an embodiment of the present invention; Figure 2 A flowchart of another automatic pose calibration method for a power station camera based on digital twin provided in an embodiment of the present invention; Figure 3 A flowchart illustrating another method for automatic pose calibration of a power station camera based on digital twin, provided as an embodiment of the present invention; Figure 4 A schematic diagram of a power station camera pose automatic calibration device based on digital twin provided in an embodiment of the present invention; Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0021] Pumped storage power stations, as key energy storage facilities in new power systems, play a vital role in ensuring the absorption of new energy sources and the stability of the power system through their safe and stable operation. With the development of power station operation and maintenance towards unmanned and minimally manned methods, video inspection, with its advantages of utilizing existing industrial television systems and requiring no large-scale new equipment, has become a core component of unmanned inspection.

[0022] The power plant video inspection strategy generation method based on related technologies enables automatic analysis of many-to-many spatial visibility between industrial TV cameras and inspection points, as well as automatic calculation of camera preset points and automatic generation of inspection strategies. This significantly improves the efficiency of inspection strategy configuration and solves problems such as omissions and misconfigurations that exist in manual configuration, providing core technical support for the automated implementation of power plant video inspection. However, the accurate implementation of this method highly depends on the accuracy of the pose parameters of the industrial TV cameras in the digital twin model, including core parameters such as the three-dimensional spatial coordinates of the camera's optical center, the initial horizontal and pitch angles corresponding to the initial normal vector, and the range of gimbal rotation angles.

[0023] In practical engineering applications, power plant industrial television cameras, especially rotatable cameras like PTZ cameras, generally suffer from installation deviations after installation. The spatial deviation between design parameters and actual pose can range from millimeters to centimeters, with angular deviations reaching several degrees. This directly leads to errors in subsequent spatial visibility analysis, inaccurate calculations of preset points, and the inability of the generated inspection strategies to accurately capture inspection point images. It can even result in misjudgments of visibility, leading to blind spots in inspections. In related technologies, camera pose calibration mainly relies on manual on-site measurement, requiring specialized equipment such as total stations and laser rangefinders. For complex scenarios such as underground power plants and areas with significant elevation differences, this method is difficult and inefficient. Calibrating a single PTZ camera takes 1-2 hours, and calibrating hundreds of cameras within a power plant requires substantial manpower and resources. Furthermore, the accuracy of manual measurements is affected by the operator's skill level, resulting in poor consistency and failing to meet the millimeter-level accuracy requirements for generating inspection strategies.

[0024] Among other related camera calibration technologies, the traditional checkerboard calibration method can only complete the intrinsic parameter calibration of the lens and cannot obtain the pose parameters of the camera in the global coordinate system, making it unsuitable for the large-scale global calibration needs of power plants. LiDAR-based calibration methods are costly and complex to operate, and cannot achieve rapid batch calibration. Existing vision-based global calibration methods mostly require on-site 3D scanning to reconstruct the scene, making it impossible to reuse existing digital twin models of the power plant. This not only increases implementation costs but also results in coordinate system deviations between the reconstructed model and the design model, preventing a closed loop with subsequent inspection strategy generation technologies. Related multi-camera calibration technologies are mainly applied to small device scenarios such as VR / AR all-in-one machines. They can only calculate the relative poses between multiple cameras, unable to obtain the absolute pose in the global coordinate system, and do not reuse existing high-precision digital twin models of the power plant, failing to solve the problem of inspection strategy failure caused by inconsistencies in coordinate systems in power plant scenarios. Furthermore, power plant digital twin-related technologies only focus on camera deployment planning and do not address the pose calibration stage after camera installation.

[0025] Therefore, the relevant technologies have not yet formed a set of camera pose calibration methods that can reuse existing digital twin models of power plants, adapt to rotatable cameras such as PTZ cameras, are easy to operate, have high precision, can be implemented in batches, and are fully compatible with existing automatic generation technologies for inspection strategies. This has become a key prerequisite issue restricting the large-scale implementation of unmanned video inspection of power plants.

[0026] Based on this, the present invention provides a method and apparatus for automatic pose calibration of a power station camera based on digital twins. This method involves acquiring a three-dimensional digital twin model of the power station, extracting a fixed-structure three-dimensional entity model and its precise three-dimensional coordinates in the global coordinate system from the model, and establishing a calibration reference digital twin model. Multiple reference QR code targets are deployed on the fixed structure, and each target is bound to its global three-dimensional coordinates in the calibration reference digital twin model, forming a global positioning reference. The camera to be calibrated is controlled to acquire real-time images, identify the reference QR code targets in the images, and extract the pixel coordinates of the target center. Valid images are selected from the real-time images based on pre-set filtering rules. Based on the mapping relationship between the global three-dimensional coordinates and corresponding pixel coordinates of the reference QR code targets in the valid images, the pose extrinsic parameters of the camera to be calibrated in the global coordinate system are calculated, thus completing the pose calibration of the camera. This method achieves fully automatic calibration, improves calibration efficiency and accuracy, and solves the problem of coordinate system incompatibility in inspection systems.

[0027] To facilitate understanding of the embodiments of this application, a method for automatic pose calibration of a power station camera based on digital twin disclosed in the embodiments of this invention will first be described in detail.

[0028] Example 1 This invention provides an automatic pose calibration method for power station cameras based on digital twins. Figure 1 This is a flowchart illustrating an automatic pose calibration method for a power station camera based on digital twin technology, provided as an embodiment of the present invention. Figure 1 As shown, the automatic pose calibration method for power plant cameras based on digital twins may include the following steps: Step S101: Obtain the three-dimensional digital twin model of the power station, extract the three-dimensional solid model of the fixed structure and the precise three-dimensional coordinates of the three-dimensional solid model in the global coordinate system from the three-dimensional digital twin model of the power station, and establish the calibration benchmark digital twin model.

[0029] Among them, the power station's 3D digital twin model is a high-precision as-built BIM (Building Information Modeling) digital twin model that matches the actual site at a 1:1 scale. This model is established based on the power station's 3D forward design or as-built survey, and contains accurate 3D geometric information and spatial coordinates of all building structures, equipment and facilities of the power station. The model's global coordinate system is completely unified with the on-site engineering survey coordinate system, and the model's spatial accuracy can reach ±5mm.

[0030] The fixed structure may include: power plant columns, trolley tracks, equipment foundations, walls, gate slots, and fixed unit equipment that has been installed and positioned. The fixed structure does not change position after the power plant is put into operation, and its three-dimensional coordinates have millimeter-level accuracy.

[0031] Among them, the calibration reference digital twin model is a sub-model extracted from the complete power plant digital twin model according to the scope of influence of this calibration operation (such as a certain area or a certain calibration unit). It only contains the fixed structural information required for calibration and serves as the spatial reference for subsequent target deployment and coordinate binding.

[0032] Step S102: Multiple reference QR code targets are deployed on the fixed structure, and each reference QR code target is bound to the global three-dimensional coordinates in the calibration reference digital twin model to form a global positioning reference.

[0033] The global coordinate system is a three-dimensional spatial coordinate system that is completely unified with the power plant engineering measurement coordinate system and the digital twin model coordinate system. It usually takes a fixed reference point of the power plant (such as the center point of the southwest corner column of the 0-meter layer of the underground auxiliary powerhouse) as the origin.

[0034] Among them, multiple reference QR code targets can be deployed non-coplanarly on a fixed structure, so that the location of the reference QR code targets covers the entire field of view of the camera to be calibrated.

[0035] The subsequent calibration can be completed by deploying no fewer than three baseline QR code targets.

[0036] Furthermore, in order to improve the calibration accuracy, calibration precision verification can also be performed. Therefore, it is necessary to add one reference QR code target for back calculation. At this time, the number of reference QR code targets should not be less than four.

[0037] The reference QR code target is positioned within all possible rotational angles of the camera to be calibrated, including all horizontal and vertical angles. For rotatable zoom PTZ cameras, the reference QR code target needs to be positioned around the camera in a 360° space, covering the entire vertical range from the lowest to the highest pitch.

[0038] The baseline QR code target uses industrial-grade waterproof, scratch-resistant, and high-contrast QR code targets. Each QR code has a unique ID to distinguish different targets.

[0039] Specifically, a one-to-one correspondence can be established between each benchmark QR code target and its corresponding global 3D coordinates in the calibration benchmark digital twin model, and stored in a database. After the benchmark QR code targets are deployed, their physical locations correspond to those coordinate points in the digital twin model.

[0040] Step S103: Control the camera to be calibrated to acquire real-time images of the scene, identify the benchmark QR code target in the real-time image of the scene, and extract the pixel coordinates of the target center of the benchmark QR code target.

[0041] Among them, the design and installation parameters of the camera to be calibrated include at least the design and installation coordinates, the range of horizontal rotation angle of the gimbal, the range of vertical rotation angle of the gimbal, and the preset values ​​of the lens intrinsic parameters.

[0042] The cameras to be calibrated may include fixed-focus bullet cameras and rotatable zoom PTZ cameras.

[0043] When the camera to be calibrated is a rotatable zoom PTZ camera, the camera's pan-tilt head is controlled to rotate within its designed horizontal and pitch rotation range, continuously acquiring multiple frames of on-site images and identifying all reference QR code targets within the field of view.

[0044] Specifically, for a rotatable zoom PTZ camera, a zero-position reference calibration can be performed on the camera to be calibrated to obtain the zero-position reference for the gimbal rotation; the horizontal rotation range calibration can be performed on the camera to be calibrated to obtain the actual horizontal rotation angle range of the gimbal; the pitch rotation range calibration can be performed on the camera to be calibrated to obtain the actual pitch rotation angle range of the gimbal; and the actual horizontal rotation angle range and the actual pitch rotation angle range of the gimbal can be bound to the zero-position reference respectively.

[0045] Among them, the real-time images are real-time video frames or still photos captured by the camera to be calibrated in its current pose.

[0046] Among them, the target center pixel coordinates are the position of the target center point of the reference QR code target in the image pixel coordinate system.

[0047] Step S104: Select valid images from the real-time images on site based on pre-set filtering rules.

[0048] Specifically, selecting valid images from real-time images based on pre-set filtering rules may include: determining the number of non-collinear reference QR code targets in each frame of real-time images; selecting the frame with the most targets as a valid image; if there are multiple frames with the same number of targets, selecting the frame with no local occlusion of the reference QR code target and the highest image clarity of the target center as a valid image.

[0049] The method further includes, after selecting valid images from real-time images based on pre-set filtering rules, the following steps: For a fixed-focal-length bullet camera, perform camera intrinsic parameter calibration to obtain the intrinsic parameter matrix and distortion coefficients of the camera to be calibrated; for a rotatable zoom PTZ camera, select multiple focal lengths at equal intervals within the zoom range, perform camera intrinsic parameter calibration for each focal length, and obtain the intrinsic parameter matrix and distortion coefficients of the camera to be calibrated; establish a focal length-intrinsic parameter mapping table based on the intrinsic parameter matrix and distortion coefficients; perform distortion correction on the valid images based on the intrinsic parameter matrix and distortion coefficients obtained from the focal length-intrinsic parameter mapping table at the current focal length, and then extract the pixel coordinates of the target center to obtain the target pixel coordinates, so as to establish a mapping relationship between three-dimensional spatial points and two-dimensional image points based on the global three-dimensional coordinates of the baseline QR code target in the valid image and the corresponding target pixel coordinates.

[0050] Specifically, the camera intrinsic parameter calibration adopts the checkerboard calibration method (Zhang Zhengyou calibration method). This method uses a checkerboard calibration board of known size to acquire multiple images in different poses, extracts the pixel coordinates of the checkerboard corner points, calculates the initial values ​​of the camera intrinsic and extrinsic parameters through the homography matrix, estimates the distortion coefficients using the nonlinear least squares method, and finally optimizes the parameters using the maximum likelihood estimation method.

[0051] Among them, five focal lengths can be selected at equal intervals within the zoom range, and the intrinsic parameters can be calibrated for each focal length using the checkerboard calibration method to establish a focal length-intrinsic parameter mapping table.

[0052] The intrinsic parameter matrix is ​​a 3×3 matrix describing the internal optical characteristics of the camera to be calibrated, including the focal length and principal point coordinates. It is a core parameter of the perspective projection model, used to map 3D points in the camera coordinate system to the pixel coordinate system.

[0053] The distortion coefficients are parameters describing the optical distortion of the lens, including radial distortion coefficients (k1, k2, k3) and tangential distortion coefficients (p1, p2). Radial distortion causes a straight line to be imaged as a curve, while tangential distortion is caused by the lens not being parallel to the image sensor.

[0054] Distortion correction is used to perform inverse distortion transformation on the original image using the calibrated intrinsic parameter matrix and distortion coefficients, thereby eliminating the influence of lens distortion on pixel coordinate accuracy.

[0055] Step S105: Based on the mapping relationship between the three-dimensional space points and two-dimensional image points formed by the global three-dimensional coordinates and corresponding pixel coordinates of the reference QR code target in the effective image, calculate the pose extrinsic parameters of the camera to be calibrated in the global coordinate system, and complete the pose calibration of the camera to be calibrated.

[0056] Among them, the pose extrinsic parameters may include at least: the three-dimensional spatial coordinates of the camera optical center, the initial horizontal angle and the initial pitch angle corresponding to the initial normal vector.

[0057] Among them, the reference QR code targets that meet the requirements for the number of pose calculations are at least 3 non-collinear reference QR code targets.

[0058] Specifically, calculating the pose extrinsic parameters of the camera to be calibrated in the global coordinate system can include: taking the mapping relationship between three-dimensional spatial points and two-dimensional image points as input, and using a pose calculation algorithm to calculate the rotation matrix and translation vector of the camera to be calibrated in the global coordinate system; transforming the rotation matrix to obtain the initial horizontal angle and initial pitch angle corresponding to the initial normal vector of the camera to be calibrated; and transforming the translation vector to obtain the three-dimensional spatial coordinates of the optical center of the camera to be calibrated in the global coordinate system.

[0059] The pose extrinsic parameters of the camera to be calibrated in the global coordinate system can be calculated using the PnP algorithm.

[0060] Among them, the PnP algorithm preferably adopts the EPnP algorithm, but similar pose calculation algorithms such as UPnP and DLS can also be used. The rotation matrix R and translation vector T of the camera in the global coordinate system are calculated. Based on the rotation matrix R, the initial horizontal angle and initial pitch angle corresponding to the initial normal vector of the camera are obtained. Based on the translation vector T, the three-dimensional spatial coordinates of the optical center of the camera in the global coordinate system are obtained.

[0061] The rotation matrix is ​​a 3×3 orthogonal matrix used to describe the orientation of the camera coordinate system relative to the global coordinate system (i.e., the orientation of the camera to be calibrated). Each column of the rotation matrix represents the direction vector of each axis of the camera coordinate system to be calibrated in the global coordinate system.

[0062] The translation vector is a 3×1 vector used to describe the position of the origin (optical center) of the camera coordinate system to be calibrated in the global coordinate system.

[0063] The initial normal vector is the unit vector of the optical axis direction of the camera to be calibrated, i.e., the direction the camera is facing. In the rotation matrix, it usually corresponds to the third column (Z-axis direction).

[0064] The initial horizontal angle (yaw angle) is the angle between the projection of the optical axis direction onto the horizontal plane (XY plane) and the positive X-axis direction, reflecting the left and right orientation of the camera to be calibrated.

[0065] The initial pitch angle is the angle between the optical axis and the horizontal plane, reflecting the up-down orientation of the camera to be calibrated.

[0066] This can be achieved using the following formula based on the rotation matrix. R The initial horizontal angle corresponding to the initial normal vector of the camera to be calibrated is obtained by conversion. θ With the initial pitch angle β : β =arcsin( r 31 ) θ =arctan2( r 21 / cos β , r 11 / cos β ) in, r 31 Rotation matrix R The element in the 3rd row and 1st column. r 21 Rotation matrix R The element in the 2nd row and 1st column. r 11 Rotation matrix R The element in the first row and first column.

[0067] For rotatable zoom PTZ cameras, the calibration of the gimbal rotation angle range can also be included. By controlling the gimbal to rotate to the mechanical limit pose and acquiring images, the angle parameters of the corresponding pose are calculated, the actual values ​​of the gimbal's horizontal and pitch rotation angle ranges are calibrated, and the installation deviations of the design parameters are corrected.

[0068] The automatic pose calibration method for power plant cameras based on digital twins provided in this invention can be achieved by acquiring a three-dimensional digital twin model of the power plant, extracting a fixed-structure three-dimensional entity model and its precise three-dimensional coordinates in the global coordinate system from the model, and establishing a calibration reference digital twin model. Multiple reference QR code targets are deployed on the fixed structure, and each target is bound to the global three-dimensional coordinates in the calibration reference digital twin model to form a global positioning reference. The camera to be calibrated is controlled to acquire real-time images, identify the reference QR code targets in the images, and extract the pixel coordinates of the target centers. Valid images are selected from the real-time images based on pre-set filtering rules. Based on the mapping relationship between the global three-dimensional coordinates and corresponding pixel coordinates of the reference QR code targets in the valid images, the pose extrinsic parameters of the camera to be calibrated in the global coordinate system are calculated, thus completing the pose calibration of the camera. This method achieves fully automatic calibration, improves calibration efficiency and accuracy, and solves the problem of coordinate system incompatibility in inspection systems.

[0069] Example 2 This invention also provides another method for automatic pose calibration of power station cameras based on digital twins; this method is implemented based on the method in the above embodiments.

[0070] Figure 2 A flowchart of another automatic pose calibration method for power station cameras based on digital twins provided in this embodiment of the invention is shown below. Figure 2 As shown, the method may also include the following steps: Step S201: Select at least one reference QR code target that has not participated in the pose extrinsic parameter calculation as a verification target.

[0071] Among them, the reference QR code targets that did not participate in the pose extrinsic parameter calculation are those targets identified in the valid image that were not used as input to the PnP algorithm. They are independent of the calculation process and can be used to verify the accuracy of the calculation results.

[0072] Step S202: Based on the pose extrinsic and intrinsic parameter matrix obtained from the calibration, calculate the verification pixel coordinates of the target center of the verification target.

[0073] The process involves substituting the global three-dimensional coordinates of the target into the calibrated camera imaging model (including intrinsic and extrinsic parameters) to calculate the pixel coordinates that it should theoretically be imaged in the image.

[0074] Step S203: Determine the reprojection error between the verification pixel coordinates and the actual pixel coordinates corresponding to the verification target.

[0075] The reprojection error is the Euclidean distance between the calculated pixel coordinates and the actual identified pixel coordinates, expressed in pixels. A smaller error indicates a more accurate calibration result.

[0076] In step S204, if the reprojection error is less than or equal to the preset pixel threshold, then the calibration is determined to be valid.

[0077] The pixel threshold can be 0.5 pixels, and there is a corresponding relationship between the reprojection error and the global spatial coordinate calibration error.

[0078] If the reprojection error is ≤0.5 pixels, it corresponds to a global spatial coordinate calibration error of ≤5mm.

[0079] If the reprojection error exceeds the pixel threshold, it is necessary to re-control the camera to be calibrated to acquire real-time images of the scene. That is, steps S103-S105 must be executed again.

[0080] Example 3 This invention also provides another method for automatic pose calibration of a power station camera based on digital twins; this method is implemented based on the method described in the above embodiments.

[0081] Figure 3 A flowchart illustrating another method for automatic pose calibration of a power station camera based on digital twins, as provided in this embodiment of the invention, is shown below. Figure 3 As shown, the method may also include the following steps: Step S301: Divide the power plant area into multiple calibration units according to the functional areas of the plant area, and set up multiple common benchmark QR code targets at the junction of adjacent calibration units.

[0082] Among them, the public reference QR code target is bound to global three-dimensional coordinates and serves as the positioning reference for two adjacent units.

[0083] The calibration unit is an independent calibration work area divided according to the functional areas of the power plant (such as underground auxiliary powerhouse, main transformer tunnel, GIS room, surface switchyard, water conveyance corridor, etc.). Each calibration unit is spatially continuous and has an independent target layout scheme.

[0084] Among them, the public benchmark QR code target is simultaneously bound to the global three-dimensional coordinates of two adjacent units.

[0085] Among them, the public reference QR code target is a target deployed in the overlapping area at the boundary of adjacent units. It belongs to two calibration units at the same time and is bound to a unique global three-dimensional coordinate, serving as a bridge for cross-unit coordinate transmission.

[0086] Step S302: An independent set of reference QR code targets is set up in each calibration unit, and the global positioning reference of the calibration unit is formed by combining the common reference QR code targets.

[0087] Step S303: For all cameras to be calibrated in each calibration unit, perform image acquisition, effective image selection and pose extrinsic parameter calculation in sequence.

[0088] Step S304: Based on the common reference QR code target, the coordinate system consistency verification of adjacent calibration units is completed to correct the coordinate deviation between calibration units.

[0089] This allows for verification of whether the common target coordinates calibrated by two adjacent calibration units are consistent. If there is a deviation, coordinate registration correction is performed.

[0090] Step S305: After all calibration units are completed, the pose parameters of all cameras are synchronously updated to the three-dimensional digital twin model of the power station.

[0091] Among these features, all calibrated camera pose parameters can be written into the device attribute database of the digital twin model, replacing the original design parameters.

[0092] Furthermore, the batch calibration process can be integrated with the power plant operation and maintenance management system to achieve fully automated assembly line operations: 1. The system automatically traverses the list of all camera IDs; 2. For each camera, it automatically controls its traversal rotation and image acquisition; 3. It automatically completes target recognition, effective image filtering, and pose calculation; 4. It automatically performs accuracy verification; 5. It automatically writes qualified results into the database, and automatically retryes and logs unqualified results. The entire process requires no manual intervention and can calibrate the entire plant with one click. After calibration, it can also automatically trigger the regeneration of inspection strategies, re-perform spatial visibility analysis based on the updated pose parameters, and update the triplet of inspection point-camera-preset point, realizing a fully automated closed loop of calibration-analysis-strategy generation.

[0093] As a specific embodiment of this application: This application focuses on the batch calibration of an industrial television system covering the underground auxiliary powerhouse, GIS switchyard, main transformer tunnel, and water conveyance corridor of a pumped storage power station. After commissioning, the power station has established an industrial television system covering the entire plant area, including 198 rotatable zoom PTZ cameras and 92 fixed bullet cameras. A total of 2847 equipment inspection points have been identified, covering the entire area including the auxiliary powerhouse control room, relay protection room, 35kV / 10kV switchgear room, GIS room, main transformer tunnel, ground switchyard, and water inlet / outlet of the water conveyance corridor. The power station has a high-precision as-built BIM digital twin model that matches the actual site at a 1:1 scale. The model's global coordinate system is completely unified with the on-site engineering measurement coordinate system, and the model's spatial accuracy is ±5mm, meeting the requirements for industrial-grade spatial analysis and calibration accuracy. All steps in this application embodiment are implemented in the power plant digital twin operation and maintenance platform. QR code target recognition, pose calculation, and distortion correction are all completed through the platform's built-in algorithm module. The ONVIF general standard protocol is preferentially used to directly connect to the industrial television system to realize remote camera control, image acquisition, and parameter feedback.

[0094] The four types of coordinate systems involved in the embodiments of this application are explicitly defined as follows: The global coordinate system is completely unified with the power plant engineering survey coordinate system and the digital twin model coordinate system. The origin is the center point of the southwest corner column of the 0-meter level of the underground auxiliary powerhouse. The X-axis runs along the east-west direction of the powerhouse (due east is positive), the Y-axis runs along the north-south direction of the powerhouse (due north is positive), and the Z-axis is perpendicular to the ground and upwards.

[0095] Camera coordinate system: with the optical center of the camera as the origin, the X-axis is parallel to the image sensor and points horizontally to the right, the Y-axis is parallel to the image sensor and points vertically downward, and the Z-axis is positive and points forward along the optical axis of the camera.

[0096] Image coordinate system: The origin is the center of the image plane, the X-axis is horizontal to the right, and the Y-axis is vertical downward. The unit is mm.

[0097] Pixel coordinate system: with the top left corner of the image as the origin, the u-axis is horizontal to the right and the v-axis is vertical to the bottom, with the unit being pixels.

[0098] The PnP algorithm in this application includes, but is not limited to, similar pose calculation algorithms such as EPnP, UPnP, and DLS. In this application, EPnP is the preferred solution.

[0099] The specific implementation steps of this application are as follows: Step A1: Preprocessing of the benchmark digital twin model.

[0100] From the existing 3D digital twin model of the power plant, the 3D solid model of the fixed structures within the scope of this calibration operation and the precise 3D coordinates in the global coordinate system are extracted to establish a calibration benchmark digital twin model. The fixed structures extracted in this embodiment include: auxiliary plant building load-bearing columns, high-voltage switchgear cabinets, GIS room equipment foundations, cable tray embedded parts, main transformer hole firewalls, switchyard structure columns, installed fixed relay protection cabinets, gate slots, and trolley tracks. The above structures have no positional changes after the power plant is put into operation, and the 3D coordinates of their key feature points have a millimeter-level accuracy of ±5mm in the digital twin model, eliminating the need for secondary on-site measurements.

[0101] For the core area identified in this application, the global three-dimensional coordinates of the key fixed structural reference points extracted from the digital twin model are as follows:

[0102] Step A2: Deploy and bind the baseline QR code target to its coordinates.

[0103] On a fixed structure with known precise 3D coordinates extracted in step A1, reference QR code targets are deployed. Each reference QR code target is bound to its global 3D coordinates in the calibration reference digital twin model, serving as a global positioning reference. In this embodiment, 10 industrial-grade reference QR code targets are deployed within a single calibration unit. The targets use 8×8 version QR codes, with a physical size of 120mm×120mm, and are made of waterproof, scratch-resistant, matte, high-contrast material to avoid strong light reflection affecting recognition accuracy, adapting to the low-light, dusty, and humid working environment of underground power plant buildings. All reference QR code targets are non-coplanar, and their deployment positions completely cover the entire working field of view of the camera to be calibrated. The number of targets deployed is no less than 4, ensuring that a single camera can capture at least 3 non-collinear reference targets after traversing and rotating. In this embodiment, the deployment positions and bound global 3D coordinates of the 10 reference QR code targets are as follows:

[0104] For batch calibration scenarios across the entire plant, two additional non-collinear common reference QR code targets are deployed at the boundary of adjacent calibration units. The common reference targets are bound to unique three-dimensional coordinates in the global world coordinate system and serve as positioning references for the two adjacent units, which are used for subsequent cross-unit coordinate system consistency verification.

[0105] Step A3: Image acquisition and target recognition from the camera to be calibrated.

[0106] The industrial television camera to be calibrated is controlled to adjust its pose, acquire real-time images of the site, identify the reference QR code targets within the images and extract the center pixel coordinates of the targets, and filter out valid images containing at least three non-collinear reference QR code targets. For a single 35kV switchgear room 35kV variable zoom PTZ camera (equipment code: CX-SQ-042) to be calibrated in this embodiment of the application, the specific implementation process is as follows: Remote control: Using the standard SDK protocol of industrial television systems (preferably the ONVIF general standard protocol, but also compatible with the proprietary SDK protocols of various brands of industrial television systems), the PTZ camera can be remotely controlled to rotate at a constant speed within its designed horizontal rotation angle range of 0-360° and pitch rotation angle range of -90° to 90°; the rotation speed is set to 5° / s, continuously acquiring 25 frames of real-time high-definition images from the site, with an image resolution of 2592×1944.

[0107] Image preprocessing: Each frame of the acquired image is preprocessed sequentially by grayscale conversion, Gaussian filtering for noise reduction, and histogram equalization to improve the target recognition rate in the low-light environment of the underground factory.

[0108] Target recognition: Using a QR code recognition algorithm, the preprocessed image is used to identify the baseline QR code target and extract the sub-pixel level coordinates of the center of each identified target.

[0109] Valid image selection: The 25 frames are sorted from high to low according to the number of non-collinear reference QR code targets identified, and the frame with the most targets is selected as the valid image. If there are multiple frames with the same number of targets, the frame with no local occlusion of the target and the highest image clarity of the target center is selected as the valid image.

[0110] In this embodiment, the final selected valid images clearly identify four non-collinear reference QR code targets, J1, J2, J5, and J9. The corresponding extracted target center sub-pixel coordinates are (426, 368), (2174, 364), (1304, 182), and (1306, 1032), which serve as input data for subsequent pose calculation.

[0111] For fixed guns, there is no need to perform pan-tilt rotation operations. Instead, a single high-definition image can be directly acquired, and valid images containing at least three non-collinear reference QR code targets can be selected according to the above rules.

[0112] Step A4: Automatic calculation and calibration of camera pose.

[0113] Based on the correspondence between the global three-dimensional coordinates of the reference QR code target within the effective image and the corresponding image pixel coordinates (3D-2D correspondence between three-dimensional spatial points and two-dimensional image points), the PnP algorithm is used to calculate the pose extrinsic parameters of the camera to be calibrated in the global coordinate system. The pose extrinsic parameters include at least the three-dimensional spatial coordinates of the camera's optical center, the initial horizontal angle corresponding to the initial normal vector, and the initial pitch angle, thus completing the camera pose calibration. In this embodiment, before the formal pose calculation, the camera intrinsic parameter pre-calibration is first completed, specifically implemented in two scenarios: fixed bullet camera and zoom PTZ camera. (1) Fixed bolt internal parameter calibration.

[0114] For fixed-focus cameras, a standard checkerboard calibration plate with a grid size of 20mm and a grid of 12×9 is used. The intrinsic parameters are calibrated using the Zhang Zhengyou calibration method. Twenty checkerboard images are collected in different poses. The camera intrinsic parameter matrix, radial distortion coefficients (k1, k2, k3), and tangential distortion coefficients (p1, p2) are obtained. After distortion correction is performed on the collected valid images, the target center pixel coordinates are extracted.

[0115] (2) Calibration of internal parameters of zoom PTZ camera.

[0116] For the zoom PTZ camera in this embodiment, its optical zoom range is 4mm-72mm. The specific calibration steps are as follows: Focal length settings: Within the full zoom range of the PTZ camera, six focal length settings are selected at equal intervals: 4mm, 16mm, 30mm, 44mm, 58mm, and 72mm, covering wide-angle, medium telephoto, and telephoto scenarios.

[0117] Single-position intrinsic parameter calibration: For each focal length position, with the focal length of the PTZ camera fixed, at least 20 chessboard images are collected at different distances, angles, and pitch poses within the camera's field of view. The intrinsic parameter matrix, radial distortion coefficient, and tangential distortion coefficient at that focal length are solved using the Zhang Zhengyou calibration method.

[0118] Mapping table creation: Each focal length setting is bound to its corresponding intrinsic parameters and distortion coefficients to create a "Focal Length-Intrinsic Parameter Mapping Table" and stored in the camera calibration database.

[0119] On-site calibration and adaptation: When acquiring valid images on-site, the current real-time focal length of the camera is obtained synchronously through the SDK. The corresponding intrinsic parameters and distortion coefficients are matched from the mapping table. After distortion correction is performed on the valid images, target recognition and pixel coordinate extraction are then performed.

[0120] In this embodiment of the application, the real-time focal length of the PTZ camera when acquiring effective images is 16mm, and the corresponding intrinsic parameter matrix obtained by matching the mapping table is as follows: radial distortion coefficient k 1 = -0.118 k2 = 0.096 k 3 = 0.012, tangential distortion coefficient p 1 = 0.002 p 2 = -0.001, based on the above parameters, complete the distortion correction of the effective image and eliminate the influence of wide-angle lens distortion on the pixel coordinate extraction accuracy.

[0121] (3) Core process of pose calculation.

[0122] Based on the distortion-corrected target center pixel coordinates and the corresponding global 3D coordinates bound to the target, a 3D-2D point-to-point correspondence is constructed, and the EPnP algorithm is used to calculate the rotation matrix of the PTZ camera in the global coordinate system. R With translation vector T The core solution process is as follows: 1) Input parameters: global 3D coordinates of 4 reference targets (known quantities), corresponding corrected sub-pixel coordinates of the image (known quantities), and camera intrinsic parameter matrix. K (Known quantity).

[0123] 2) Solution Output: The perspective projection equation is constructed using the EPNP algorithm, and the rotation matrix of the PTZ camera in the global coordinate system is obtained by solving it. R With translation vector T : Rotation matrix

[0124] Translation vector: , unit mm.

[0125] 3) Euler angle transformation: based on rotation matrix R This is converted to obtain the initial horizontal angle (yaw angle) corresponding to the initial normal vector of the camera. θ Initial pitch angle β The conversion formula is: β =arcsin( r 31 ) θ =arctan2( r 21 / cos β , r 11 / cos β ) in, r 31 Rotation matrix R The element in the 3rd row and 1st column. r 21 Rotation matrix R The element in the 2nd row and 1st column.r 11 Rotation matrix R The element in the first row and first column.

[0126] In this embodiment of the application, the initial horizontal angle of the PTZ camera is calculated. θ =0.04°, initial pitch angle β =-0.03°, all within the gimbal's designed rotation angle range; based on translation vector T The three-dimensional spatial coordinates of the optical center of the PTZ camera in the global coordinate system are obtained as (3186, 4502, 10220) mm, thus completing the calibration of the core pose parameters of the PTZ camera.

[0127] (4) Calibration of the rotation angle range of the PTZ camera.

[0128] For rotatable zoom PTZ cameras, further calibration of the gimbal rotation angle range is performed to correct design parameter errors caused by installation deviations. The specific steps are as follows: 1) Zero-position reference calibration: Control the PTZ camera to return to the factory zero position, acquire valid images in this pose, and perform the pose calculation steps described above to obtain the horizontal angle corresponding to the zero position. θ 0. Pitch angle β 0 serves as the reference zero point for gimbal rotation.

[0129] 2) Horizontal rotation range calibration: Control the gimbal to rotate horizontally to the left limit mechanical stop, and after it stabilizes, acquire a valid image and calculate the horizontal angle of this pose. θ left Then control the gimbal to rotate horizontally to the right limit mechanical stop, and after it stabilizes, acquire a valid image and calculate the horizontal angle of that pose. θ right The final calibration yielded the actual horizontal rotation angle range of the gimbal as [ θ left , θ right In this embodiment of the application, the actual horizontal rotation angle range of the PTZ camera after calibration is 0-359.7°.

[0130] 3) Pitch rotation range calibration: Control the gimbal to rotate to the upper limit mechanical limit, hold it stable, acquire valid images, and calculate the pitch angle of that pose. β up Then control the gimbal to tilt and rotate to the lower limit mechanical stop, and after it stabilizes, acquire a valid image and calculate the tilt angle of that pose. β down The final calibration yielded the actual pitch rotation angle range of the gimbal as […]. β down , β upIn this embodiment of the application, the actual pitch rotation angle range of the PTZ camera after calibration is -89.6° to 90°.

[0131] Deviation correction: The calibrated gimbal rotation angle range is bound to the zero-position reference and stored in the camera parameter library as the reference parameter for preset position calculation in subsequent inspection strategy generation.

[0132] (5) Calibration accuracy verification.

[0133] After completing the core pose calculation, perform calibration accuracy verification. The specific steps are as follows: 1) Select target J10, which did not participate in this pose calculation, as the verification target. Its known global three-dimensional coordinates are (8900, 6800, 2600) mm.

[0134] 2) Based on the external and internal parameter matrix of the camera pose obtained in this calibration, the image pixel coordinates of the J10 target center are calculated as (2186, 1030) using the perspective projection formula.

[0135] 3) Compare the reprojection error with the actual target center pixel coordinates (2186.4, 1029.7) and calculate that the reprojection error is 0.42 pixels, which is ≤ the preset threshold of 0.5 pixels. The corresponding global spatial coordinate calibration error is ≤ 4 mm, and the calibration result is determined to be valid. If the reprojection error exceeds the threshold, a valid image is re-acquired and the calibration steps are performed.

[0136] Steps for batch calibration of cameras throughout the factory area.

[0137] The specific implementation steps for the batch calibration of 290 cameras throughout the power plant area are as follows: 1) Calibration unit division: The power station is divided into 5 independent calibration units according to the underground auxiliary powerhouse, main transformer tunnel, GIS room, ground switch station and water conveyance corridor. Each unit covers a continuous spatial area, and overlapping areas are set at the boundary of adjacent units.

[0138] 2) Deployment of common reference targets: In the overlapping area of ​​adjacent units, two non-collinear common reference QR code targets are deployed. The targets are bound to unique three-dimensional coordinates in the global world coordinate system and serve as positioning references for the two adjacent units.

[0139] 3) Independent calibration within a unit: An independent set of reference QR code targets is set up in each unit. Combined with the common reference targets, a global positioning reference for the unit is formed. Steps A3-A4 above are executed sequentially for all cameras to be calibrated in the unit to complete the automatic calibration of a single camera.

[0140] 4) Coordinate system consistency verification: For two adjacent units, based on the common reference target, calculate the coordinate values ​​of the common target in the calibration results of the two units and calculate the coordinate deviation; if the deviation is ≤5mm, the coordinate systems of the two units are determined to be consistent; if the deviation is >5mm, the global coordinates of the common reference target are used as the reference to perform coordinate registration correction on the calibration results of the two units to eliminate the deviation between units. 5) Unified coordinates across the entire plant: After all units have completed calibration and consistency verification, the coordinates of all camera pose parameters are unified based on the global coordinate system of the power plant, ensuring that all calibration results are fully matched with the coordinate system of the digital twin model.

[0141] Implementation steps for handling abnormal engineering scenarios.

[0142] In response to common anomalies during power plant on-site calibration, this application provides a complete handling solution, as detailed below: 1) Scenario with insufficient number of targets in the field of view: If, after a single camera has traversed and rotated, at most only 2 non-collinear reference targets can be identified in all the acquired images, which cannot meet the calculation requirement of 3 targets, then 2 reference QR code targets bound to global coordinates are temporarily added on the fixed structure accessible by the camera's field of view, and the calibration is re-executed after supplementing the positioning references; if the number of targets required for pose calculation changes due to the adjustment of the PnP-like algorithm used, reference QR code targets that meet the algorithm requirements need to be added accordingly, and the calibration is re-executed after supplementing the positioning references.

[0143] 2) Target partial occlusion scenario: If the identified baseline QR code target is partially occluded and the complete target center coordinates cannot be extracted, the occluded area is corrected and restored using the QR code error correction algorithm; if the accurate coordinates still cannot be extracted after error correction, the target is removed and at least 3 non-collinear valid targets are selected from the remaining targets for pose calculation.

[0144] 3) Reprojection error exceeding the threshold scenario: If the reprojection error in the accuracy verification is >0.5 pixels, the calibration result is deemed invalid. At least 3 valid images are re-acquired, and pose calculation and accuracy verification are performed respectively. The set of results with the smallest reprojection error is selected as the final calibration result. If the accuracy requirements still cannot be met after multiple acquisitions, the accuracy of the coordinate binding and the compliance of the deployment of the reference target are re-checked. After eliminating the problem, the calibration is re-performed.

[0145] 4) Short-term occlusion dynamic tracking scenario: For temporary occlusion by personnel or equipment during the calibration process of the PTZ camera, five frames of images are continuously acquired, invalid frames with occlusion are removed, and valid frames without occlusion are selected to complete the calibration without interrupting the calibration process.

[0146] The calibration results are linked with the inspection strategy implementation steps.

[0147] After completing the camera calibration, the calibration results can be combined with the power station video inspection strategy to form a closed-loop optimization. The specific implementation steps are as follows: 1) Parameter synchronization: The pose parameters of the cameras that have been calibrated individually or in batches, including the three-dimensional spatial coordinates of the optical center, the initial horizontal angle, the initial pitch angle, the actual rotation angle range of the gimbal, and the focal length-internal parameter mapping table, are synchronously updated to the digital twin model of the power station, replacing the original design parameters and ensuring the pose consistency between the virtual model and the on-site entity.

[0148] 2) Spatial visibility analysis update: Based on the updated camera pose parameters, the spatial visibility analysis of all 2847 inspection points is carried out again using the common methods in existing technologies, and the effective inspection point-camera tuple is updated.

[0149] 3) Inspection strategy update: Based on the updated binary pair, recalculate and generate camera preset points, update the inspection strategy of the full inspection point-camera-preset point triplet, and complete the closed-loop optimization of calibration and inspection strategy.

[0150] Example 4 Corresponding to the above method embodiments, this invention provides an automatic pose calibration device for power station cameras based on digital twins. Figure 4 This is a schematic diagram of the structure of an automatic pose calibration device for a power station camera based on digital twin, provided in an embodiment of the present invention. Figure 4 As shown, the digital twin-based automatic pose calibration device for power plant cameras may include: The model acquisition module 401 is used to acquire the three-dimensional digital twin model of the power station, extract the three-dimensional solid model of the fixed structure and the accurate three-dimensional coordinates of the three-dimensional solid model in the global coordinate system from the three-dimensional digital twin model of the power station, and establish a calibration benchmark digital twin model.

[0151] The coordinate binding module 402 is used to deploy multiple reference QR code targets on a fixed structure and bind each reference QR code target to the global three-dimensional coordinates in the calibration reference digital twin model to form a global positioning reference.

[0152] The pixel coordinate extraction module 403 is used to control the camera to be calibrated to acquire real-time images of the scene, identify the reference QR code target in the real-time image, and extract the pixel coordinates of the target center of the reference QR code target.

[0153] The effective image selection module 404 is used to select effective images from real-time images on site based on pre-set filtering rules.

[0154] The pose calibration module 405 is used to calculate the pose extrinsic parameters of the camera to be calibrated in the global coordinate system based on the mapping relationship between the global three-dimensional coordinates and the corresponding pixel coordinates of the reference QR code target in the effective image, and to complete the pose calibration of the camera to be calibrated.

[0155] The automatic pose calibration device for power plant cameras based on digital twins provided in this invention can acquire a three-dimensional digital twin model of the power plant, extract a fixed-structure three-dimensional entity model and its precise three-dimensional coordinates in the global coordinate system from the three-dimensional digital twin model, and establish a calibration reference digital twin model. Multiple reference QR code targets are deployed on the fixed structure, and each reference QR code target is bound to the global three-dimensional coordinates in the calibration reference digital twin model to form a global positioning reference. The camera to be calibrated is controlled to acquire real-time images from the site, identify the reference QR code targets in the real-time images, and extract the pixel coordinates of the target center of the reference QR code targets. Valid images are selected from the real-time images based on pre-set filtering rules. Based on the mapping relationship between the three-dimensional spatial points and two-dimensional image points formed by the global three-dimensional coordinates and corresponding pixel coordinates of the reference QR code targets in the valid images, the pose extrinsic parameters of the camera to be calibrated in the global coordinate system are calculated, completing the pose calibration of the camera to be calibrated. This method achieves fully automatic calibration, improves calibration efficiency and accuracy, and solves the problem of coordinate system incompatibility in inspection systems.

[0156] In some embodiments, the coordinate binding module is further configured to deploy multiple reference QR code targets that are not coplanar on a fixed structure, such that the positions of the deployed reference QR code targets cover the entire field of view of the camera to be calibrated.

[0157] In some embodiments, the effective image selection module is further configured to determine the number of non-collinear reference QR code targets in each frame of real-time on-site image; select the frame with the largest number of targets as the effective image; if there are multiple frames with the same number of targets, select the frame with no local occlusion of the reference QR code target and the highest image clarity of the target center as the effective image.

[0158] In some embodiments, the effective image selection module is further configured to: perform camera intrinsic parameter calibration for a fixed-focal-length bullet camera to obtain the intrinsic parameter matrix and distortion coefficients of the camera to be calibrated; select multiple focal lengths at equal intervals within the zoom range for a rotatable zoom PTZ camera, perform camera intrinsic parameter calibration for each focal length to obtain the intrinsic parameter matrix and distortion coefficients of the camera to be calibrated; establish a focal length-intrinsic parameter mapping table based on the intrinsic parameter matrix and distortion coefficients; perform distortion correction on the effective image based on the intrinsic parameter matrix and distortion coefficients obtained from the focal length-intrinsic parameter mapping table at the current focal length; and then extract the pixel coordinates of the target center to obtain the target pixel coordinates, so as to provide a mapping relationship between three-dimensional space points and two-dimensional image points based on the global three-dimensional coordinates of the reference QR code target in the effective image and the corresponding target pixel coordinates.

[0159] In some embodiments, the pose calibration module is further configured to take the mapping relationship between three-dimensional spatial points and two-dimensional image points as input, and use a pose calculation algorithm to calculate the rotation matrix and translation vector of the camera to be calibrated in the global coordinate system; based on the rotation matrix transformation, obtain the initial horizontal angle and initial pitch angle corresponding to the initial normal vector of the camera to be calibrated; and based on the translation vector transformation, obtain the three-dimensional spatial coordinates of the optical center of the camera to be calibrated in the global coordinate system.

[0160] In some embodiments, the pose calibration module is further configured to perform zero-position reference calibration on the camera to be calibrated to obtain the zero-position reference for gimbal rotation; perform horizontal rotation range calibration on the camera to be calibrated to obtain the actual horizontal rotation angle range of the gimbal; perform pitch rotation range calibration on the camera to be calibrated to obtain the actual pitch rotation angle range of the gimbal; and bind the actual horizontal rotation angle range and the actual pitch rotation angle range of the gimbal to the zero-position reference respectively.

[0161] In some embodiments, the pose calibration module is further configured to select at least one reference QR code target that has not participated in the pose extrinsic parameter calculation as a verification target; calculate the verification pixel coordinates of the target center of the verification target based on the pose extrinsic parameter matrix obtained by calibration; determine the reprojection error between the verification pixel coordinates and the actual pixel coordinates corresponding to the verification target; and determine that the calibration is valid if the reprojection error is less than or equal to a preset pixel threshold.

[0162] In some embodiments, the pose calibration module is further configured to divide the power plant area into multiple calibration units according to the functional areas of the power plant, and to deploy multiple common reference QR code targets at the boundaries of adjacent calibration units; the common reference QR code targets are simultaneously bound to the global three-dimensional coordinates of two adjacent units, serving as the positioning references for the two adjacent units; each calibration unit is configured with an independent set of reference QR code targets, which, combined with the common reference QR code targets, form the global positioning reference of the calibration unit; image acquisition, effective image selection, and pose extrinsic parameter calculation are sequentially performed on all cameras to be calibrated in each calibration unit; the coordinate system consistency verification of adjacent calibration units is completed based on the common reference QR code targets to correct the coordinate deviations between calibration units; after all calibration units are completed, the pose parameters of all cameras are synchronously updated to the three-dimensional digital twin model of the power plant.

[0163] The device provided in this embodiment of the invention has the same implementation principle and technical effect as the aforementioned method embodiment. For the sake of brevity, any parts not mentioned in the device embodiment can be referred to the corresponding content in the aforementioned method embodiment.

[0164] Example 5 This invention also provides an electronic device for running the above-described automatic pose calibration method for power station cameras based on digital twins; see also Figure 5 The diagram shows the structure of an electronic device, which includes a memory 500 and a processor 501. The memory 500 stores one or more computer instructions, which are executed by the processor 501 to implement the above-mentioned automatic pose calibration method for power station cameras based on digital twins.

[0165] Furthermore, Figure 5 The electronic device shown also includes a bus 502 and a communication interface 503. The processor 501, the communication interface 503, and the memory 500 are connected via the bus 502.

[0166] The memory 500 may include high-speed random access memory (RAM) and may also include non-volatile memory, such as at least one disk storage device. Communication between this system network element and at least one other network element is achieved through at least one communication interface 503 (which can be wired or wireless), such as the Internet, wide area network, local area network, metropolitan area network, etc. The bus 502 can be an ISA bus, PCI bus, or EISA bus, etc. The bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 5 The symbol is represented by a single double-headed arrow, but this does not mean that there is only one bus or one type of bus.

[0167] Processor 501 may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method can be completed by the integrated logic circuitry in the hardware of processor 501 or by instructions in software form. Processor 501 can be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it can also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this invention. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this invention can be directly manifested as execution by a hardware decoding processor, or execution by a combination of hardware and software modules in the decoding processor. The software module can reside in a readily available storage medium in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in memory 500, and processor 501 reads information from memory 500 and, in conjunction with its hardware, completes the steps of the method described in the foregoing embodiments.

[0168] This invention also provides a computer-readable storage medium storing computer-executable instructions. When these computer-executable instructions are called and executed by a processor, they cause the processor to implement the above-described automatic pose calibration method for a power station camera based on digital twins. For specific implementation details, please refer to the method embodiments, which will not be repeated here.

[0169] The computer program product for the automatic pose calibration method of power station camera based on digital twin provided in this embodiment of the invention includes a computer-readable storage medium storing non-volatile program code executable by a processor. The instructions included in the program code can be used to execute the methods described in the preceding method embodiments. For specific implementation, please refer to the method embodiments, which will not be repeated here.

[0170] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0171] In the several embodiments provided by this invention, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Additionally, the coupling or direct coupling or communication connection shown or discussed may be through some communication interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0172] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of the embodiments of this application, depending on actual needs.

[0173] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0174] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a processor-executable, non-volatile, computer-readable storage medium. Based on this understanding, the technical solution of this invention, essentially, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0175] Finally, it should be noted that the above-described embodiments are merely specific implementations of the present invention, used to illustrate the technical solutions of the present invention, and not to limit it. The scope of protection of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments within the technical scope disclosed in the present invention, or make equivalent substitutions for some of the technical features; and these modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for automatic pose calibration of a power station camera based on digital twin, characterized in that, The method includes: A three-dimensional digital twin model of the power plant is obtained, and a three-dimensional solid model with a fixed structure and the precise three-dimensional coordinates of the three-dimensional solid model in the global coordinate system are extracted from the three-dimensional digital twin model of the power plant to establish a calibration reference digital twin model. Multiple reference QR code targets are deployed on the fixed structure, and each reference QR code target is bound to the global three-dimensional coordinates in the calibration reference digital twin model to form a global positioning reference. Control the camera to be calibrated to acquire real-time images of the scene, identify the reference QR code target in the real-time image of the scene, and extract the pixel coordinates of the target center of the reference QR code target; Valid images are selected from the real-time images based on pre-set filtering rules; Based on the mapping relationship between the three-dimensional spatial points and two-dimensional image points formed by the global three-dimensional coordinates and corresponding pixel coordinates of the reference QR code target in the effective image, the pose extrinsic parameters of the camera to be calibrated in the global coordinate system are calculated to complete the pose calibration of the camera to be calibrated.

2. The method according to claim 1, characterized in that, The method of deploying multiple reference QR code targets on the fixed structure includes: Multiple reference QR code targets are non-coplanarly arranged on the fixed structure, so that the positions of the reference QR code targets cover the entire field of view of the camera to be calibrated.

3. The method according to claim 1, characterized in that, The selection of valid images from the real-time images based on pre-set filtering rules includes: Determine the number of non-collinear reference QR code targets in each frame of the real-time image of the scene; The frame with the largest number of targets is selected as the valid image. If multiple frames of images with the same number of targets exist, the frame with the highest image clarity at the center of the target and no local occlusion of the baseline QR code target is selected as the valid image.

4. The method according to claim 1, characterized in that, After selecting valid images from the real-time images based on pre-set filtering rules, the method further includes: For fixed-focus bullet cameras, perform camera intrinsic parameter calibration to obtain the intrinsic parameter matrix and distortion coefficients of the camera to be calibrated; For a rotatable zoom PTZ camera, multiple focal lengths are selected at equal intervals within the zoom range, and camera intrinsic parameter calibration is performed for each focal length to obtain the intrinsic parameter matrix and distortion coefficient of the camera to be calibrated. Based on the intrinsic parameter matrix and the distortion coefficients, a focal length-intrinsic parameter mapping table is established; Based on the intrinsic parameter matrix and distortion coefficients obtained by matching the focal length-intrinsic parameter mapping table with the current focal length, the effective image is distorted and then the pixel coordinates of the target center are extracted again to obtain the target pixel coordinates, so as to provide a mapping relationship between the three-dimensional space points and two-dimensional image points based on the global three-dimensional coordinates of the reference QR code target in the effective image and the corresponding target pixel coordinates.

5. The method according to claim 1, characterized in that, The calculation of the pose extrinsic parameters of the camera to be calibrated in the global coordinate system includes: The mapping relationship between the three-dimensional spatial points and the two-dimensional image points is used as input, and the rotation matrix and translation vector of the camera to be calibrated in the global coordinate system are calculated by the pose calculation algorithm. Based on the rotation matrix transformation, the initial horizontal angle and initial pitch angle corresponding to the initial normal vector of the camera to be calibrated are obtained; The optical center of the camera to be calibrated is obtained in three-dimensional space in the global coordinate system based on the translation vector transformation.

6. The method according to claim 1, characterized in that, For a rotatable zoom PTZ camera, the method further includes: The zero-position reference of the camera to be calibrated is obtained by performing zero-position reference calibration on the camera to be calibrated. The horizontal rotation range of the camera to be calibrated is calibrated to obtain the actual horizontal rotation angle range of the gimbal. The camera to be calibrated is subjected to pitch rotation range calibration to obtain the actual pitch rotation angle range of the gimbal; The actual horizontal rotation angle range and the actual pitch rotation angle range of the gimbal are respectively bound to the zero-position reference.

7. The method according to claim 1, characterized in that, After completing the pose calibration of the camera to be calibrated, the method further includes: Select at least one baseline QR code target that is not involved in the pose extrinsic parameter calculation as the verification target; The calibration pixel coordinates of the target center of the calibration target are calculated based on the extrinsic and intrinsic parameter matrix of the pose obtained from the calibration. Determine the reprojection error between the verification pixel coordinates and the actual pixel coordinates corresponding to the verification target; If the reprojection error is less than or equal to a preset pixel threshold, the calibration is deemed valid.

8. The method according to claim 1, characterized in that, The method further includes: The power plant area is divided into multiple calibration units according to its functional areas. Multiple common reference QR code targets are set at the boundaries of adjacent calibration units. The common reference QR code targets are bound to global three-dimensional coordinates and serve as the positioning reference for two adjacent units. Each calibration unit is equipped with an independent set of reference QR code targets, which, together with the common reference QR code targets, form the global positioning reference for the calibration unit. For all cameras to be calibrated within each calibration unit, image acquisition, valid image selection, and pose extrinsic parameter calculation are performed sequentially. Based on the common reference QR code target, the coordinate system consistency of adjacent calibration units is verified to correct the coordinate deviation between calibration units; After all calibration units are completed, the pose parameters of all cameras will be synchronously updated to the three-dimensional digital twin model of the power station.

9. An automatic pose calibration device for power station cameras based on digital twins, characterized in that, The device includes: The model acquisition module is used to acquire a three-dimensional digital twin model of the power station, extract a three-dimensional solid model with a fixed structure and the precise three-dimensional coordinates of the three-dimensional solid model in the global coordinate system from the three-dimensional digital twin model of the power station, and establish a calibration benchmark digital twin model. The coordinate binding module is used to deploy multiple reference QR code targets on the fixed structure and bind each reference QR code target to the global three-dimensional coordinates in the calibration reference digital twin model to form a global positioning reference. The pixel coordinate extraction module is used to control the camera to be calibrated to acquire real-time images of the scene, identify the reference QR code target in the real-time image of the scene, and extract the pixel coordinates of the target center of the reference QR code target. The effective image selection module is used to select effective images from the real-time images on site based on pre-set filtering rules; The pose calibration module is used to calculate the pose extrinsic parameters of the camera to be calibrated in the global coordinate system based on the mapping relationship between the global three-dimensional coordinates and the corresponding pixel coordinates of the reference QR code target in the effective image, and to complete the pose calibration of the camera to be calibrated.

10. An electronic device, characterized in that, The device includes a processor and a memory, the memory storing computer-executable instructions that can be executed by the processor, the processor executing the computer-executable instructions to implement the automatic pose calibration method for a power station camera based on digital twin as described in any one of claims 1 to 7.

Images