A transformer multi-view three-dimensional reconstruction method and system for a substation scene

Abstract

The application discloses a transformer multi-view three-dimensional reconstruction method and system for a substation scene, and the method comprises the following steps: acquiring multi-view image data of a transformer; pre-processing the image data to obtain pre-processed image data; performing a labeling operation on the pre-processed image data to make a transformer scene data set; constructing a transformer multi-view three-dimensional reconstruction model for a substation scene; inputting the transformer scene data set into the transformer multi-view three-dimensional reconstruction model for a substation scene, setting a loss function according to the model, updating parameters of the model through back propagation, and finally obtaining a trained transformer three-dimensional reconstruction model; and inputting the multi-view image data of the transformer into the trained transformer multi-view three-dimensional reconstruction model to obtain a transformer three-dimensional reconstruction point cloud result. The application has the advantages of high reconstruction precision.

Description

TECHNICAL FIELD

[0001] The present application mainly relates to the technical field of power equipment, and particularly relates to a transformer multi-view three-dimensional reconstruction method and system for a substation scene. BACKGROUND

[0002] In the current field of power systems, with the development of smart grids and the improvement of the automation level of power equipment, efficient monitoring and management of substation equipment is particularly important. As a key node in the power system, substations bear the important tasks of power conversion, distribution and regulation, and among them, the transformer, as one of the core devices, directly affects the safety and stability of the power grid. In traditional substation management, the monitoring of transformers often relies on manual inspection and single-angle monitoring equipment. This approach has the problems of narrow monitoring range, low efficiency, and incomplete understanding of the equipment condition. In addition, the current monitoring technology still has room for improvement in efficiency and accuracy, and cannot meet the high requirements for monitoring the complex internal structure and details of the transformer.

[0003] To address the above problems, introducing advanced perception technology and data processing technology to achieve comprehensive monitoring and intelligent management of transformers and their surrounding environment is the only way in the field of power systems. In the field of power systems, research directions that have been proposed include the application of sensor networks, optimization of data fusion and analysis algorithms, and research on the prediction of equipment operating status. However, current research still faces some challenges and problems. For example, existing monitoring equipment is mostly single-angle or single-sensing mode, which cannot achieve all-around, high-precision three-dimensional reconstruction and real-time monitoring of transformers. In addition, traditional image processing and data analysis methods lack efficient three-dimensional reconstruction technology support when dealing with complex power equipment structures, resulting in limited accuracy and reliability of monitoring results.

[0004] Three-dimensional reconstruction technology uses multiple view data acquisition, combined with advanced computer vision and image processing algorithms, to achieve high-precision three-dimensional model reconstruction of transformers and their surrounding environment. This method not only can achieve comprehensive display of equipment structure and details, but also can provide effective support for remote monitoring, operating status analysis and maintenance decision-making of equipment. Therefore, it is particularly necessary and important to introduce a three-dimensional reconstruction method based on multi-view image fusion.

[0005] The transformer multi-view three-dimensional reconstruction method for a substation scene has been increasingly concerned in recent years. Compared with traditional methods based on photogrammetry or laser scanning, multi-view three-dimensional reconstruction technology is more challenging, and the following difficulties may exist in the specific implementation process:

[0006] 1、In the substation scene, the transformer is usually located in a complex environment, and there may be buildings, equipment and other obstacles around. These elements may interfere with or block the transformer reconstruction process. It is difficult to reasonably control multiple cameras to obtain key information of the transformer.

[0007] 2、Different cameras have their own internal coordinate systems, which leads to differences in coordinate systems between pictures taken by different cameras; at the same time, there may be deviations in time synchronization, which leads to inaccurate alignment of collected images or errors in time.

[0008] 3、When doing scene reconstruction, the MVS method is usually used, but this method is difficult to handle weak texture areas and non-Lambertian surfaces because these areas are difficult to match, resulting in incomplete reconstruction. SUMMARY

[0009] To solve the technical problems existing in the prior art, the present application provides a transformer multi-view three-dimensional reconstruction method and system for substation scenes with high reconstruction accuracy.

[0010] To solve the above technical problems, the technical solution provided by the present application is:

[0011] A transformer multi-view three-dimensional reconstruction method for a substation scene, comprising the steps of:

[0012] Obtaining multi-view image data of the transformer;

[0013] Pretreating the image data to obtain pretreated image data;

[0014] Labeling the pretreated image data to make a transformer scene dataset;

[0015] Constructing a transformer multi-view three-dimensional reconstruction model for a substation scene;

[0016] Inputting the transformer scene dataset into the transformer multi-view three-dimensional reconstruction model for a substation scene, setting a loss function according to the model, updating the parameters of the model by back propagation, and finally obtaining a trained transformer three-dimensional reconstruction model;

[0017] Inputting the multi-view image data of the transformer into the trained transformer multi-view three-dimensional reconstruction model to obtain a transformer three-dimensional reconstruction point cloud result.

[0018] Preferably, constructing the transformer multi-view three-dimensional reconstruction model for a substation scene further comprises feature extraction, differentiable homography transformation, cost volume construction and depth map acquisition; wherein the feature extraction comprises the steps of:

[0019] Convolution of the original image data to obtain non-overlapping 2D feature maps;

[0020] After extracting the 2D feature map, for a given input feature F i-2 , it is processed by a depthwise separable convolution to capture the local spatial information of the feature map, followed by batch normalization to obtain an intermediate state F i-1 ;

[0021] The intermediate state F i-1 mixes channel information through convolution and preserves the distribution of information through the activation function, and the processed feature F i-1 is fused with the original input feature F i-2 using a residual connection to obtain the fused feature.

[0022] Preferably, the differentiable homographic transformation comprises the steps of:

[0023] A 2D-to-3D conversion method based on homographic transformation is applied, and the geometric relationship of the camera is combined into the neural network; the features on other images are warped to 256 depth planes in the camera frustum of the reference image, and there is always a depth corresponding to it;

[0024] The homographic matrix H i of the i-th feature warped to the depth d plane of the camera frustum of the reference image

[0025]

[0026] where K i , R i , t i represent the intrinsic matrix, rotation matrix, and translation matrix of the camera, respectively; K l , R l , t l represent the intrinsic matrix and rotation and translation matrix of a selected reference image, and the final obtained depth map is the depth map of the selected reference image.

[0027] Preferably, the cost volume construction comprises the steps of:

[0028] After the differentiable homographic transformation, N feature volumes are obtained, and these feature volumes are combined into a cost volume. Specifically, the cost volume is calculated pixel by pixel using the variance method, and the process is shown in the following formula:

[0029]

[0030] where V represents the feature volume generated after the differentiable homographic transformation.

[0031] Preferably, the depth map acquisition comprises the following steps:

[0032] The cost volume is acquired and is generated into a possibility size of each pixel at each depth through a four-level U-Net structure;

[0033] After the probability distribution on the depth is acquired, an initial depth map is calculated;

[0034] The depth map and the original image are concatenated into a four-channel input, and a depth residual is obtained through a neural network, and then is added to the previous depth map to obtain a final depth map.

[0035] Preferably, the pre-processing of the image comprises: enhancing the image by using a frequency domain method.

[0036] Preferably, the pre-processing of the image comprises calibrating the camera, and acquiring internal and external parameters of the camera, and the specific steps are as follows:

[0037] A chessboard calibration plate with a known size is prepared, and a plurality of groups of images of the calibration plate are captured by using the camera at different angles and positions, so that the calibration plate can be seen in the whole scene and the field of view of the camera is covered as much as possible;

[0038] The acquired calibration images are imported into a MATLAB environment, and the side length of the chessboard is set according to the actual size of the calibration plate;

[0039] The camera corner detection result is displayed, and the calibration is performed after removing the images with poor effects, and the internal and external parameters of the camera are exported and recorded for standby use.

[0040] Preferably, the loss function L is specifically represented as:

[0041]

[0042] Wherein d represents a ground truth depth map, represents an initial depth map, represents a final depth map after correction, and p represents a region with a value in the ground truth.

[0043] The application further discloses a transformer multi-view three-dimensional reconstruction system for a substation scene, comprising:

[0044] An image acquisition module is configured to acquire multi-view image data of the transformer.

[0045] An image pre-processing module is configured to pre-process the image data to obtain pre-processed image data.

[0046] A data set making module is configured to perform annotation work on the pre-processed image data to make a transformer scene data set.

[0047] The model construction module is configured to construct a transformer multi-view three-dimensional reconstruction model for a substation scene.

[0048] The model training module is configured to input a transformer scene dataset into the transformer multi-view three-dimensional reconstruction model for a substation scene constructed by the model construction module, set a loss function according to the model, update parameters of the model through back propagation, and finally obtain a trained transformer three-dimensional reconstruction model.

[0049] The three-dimensional reconstruction module is configured to input multi-view image data of a transformer into the trained transformer multi-view three-dimensional reconstruction model to obtain a transformer three-dimensional reconstruction point cloud result.

[0050] Preferably, in the model construction module, constructing the transformer multi-view three-dimensional reconstruction model for a substation scene includes a feature extraction unit, a differentiable homography transformation, a cost volume construction, and a depth map acquisition.

[0051] The feature extraction unit includes:

[0052] The convolution module is configured to perform convolution on the original image data to obtain non-overlapping 2D feature maps.

[0053] The processing module is configured to, after extracting the 2D feature maps, process the given input feature through a depth separable convolution to capture local spatial information of the feature maps, and then use batch normalization to obtain an intermediate state F i -1 ;

[0054] The fusion module is configured to fuse the intermediate state F i-1 through a convolution to mix channel information and retain a distribution of information through an activation function, and use a residual connection to fuse the processed feature F i-1 with the original input feature F i-2 to obtain a fused feature.

[0055] Preferably, the differentiable homography transformation includes:

[0056] A 2D-to-3D conversion method based on a homography transformation is applied, and the geometric relationship of the camera is combined into the neural network; features on other images are warped to 256 depth planes in the camera frustum of the reference image, and there is always a corresponding depth;

[0057] The homography matrix H i (d) is calculated by the following formula:

[0058]

[0059] where Ki , R i , t i respectively represent the intrinsic matrix, rotation matrix and translation matrix of the camera; K l , R l , t l represent the intrinsic matrix and rotation and translation matrix of the selected reference image, and the finally obtained depth map is the depth map of the selected reference image.

[0060] Preferably, in the model construction module, the cost volume construction comprises:

[0061] After the differentiable homography transformation, N feature volumes are obtained, and the feature volumes are combined into a cost volume. Specifically, the cost volume is calculated pixel by pixel using the variance method, and the process is shown in the following formula:

[0062]

[0063] wherein V represents the feature volume generated after the differentiable homography transformation.

[0064] Preferably, in the model construction module, the depth map acquisition comprises:

[0065] The generation module is configured to generate the possibility size of each pixel at each depth from the obtained cost volume through a four-level U-Net structure;

[0066] The initial depth map acquisition module is configured to calculate the initial depth map after obtaining the probability distribution on the depth.

[0067] The final depth map acquisition module is configured to concatenate the depth map and the original image into a four-channel input, obtain a depth residual through a neural network, and then add the depth residual to the previous depth map to obtain the final depth map.

[0068] Preferably, in the image preprocessing module, the image is enhanced by using the frequency domain method.

[0069] Preferably, in the image preprocessing module, the camera is calibrated to obtain the intrinsic and extrinsic parameters of the camera, specifically:

[0070] A chessboard calibration plate with a known size is prepared, and multiple groups of images of the calibration plate are captured by using the camera at different angles and positions, so that the calibration plate can be seen in the entire scene and the entire field of view of the camera is covered as much as possible.

[0071] The collected calibration images are imported into the MATLAB environment, and the side length of the chessboard is set according to the actual size of the calibration plate.

[0072] The camera corner detection result is displayed, and the calibration is performed after removing the images with poor effects. The intrinsic and extrinsic parameters of the camera are exported and recorded for future use.

[0073] Preferably, in the image preprocessing module, the loss function L is specifically represented as:

[0074]

[0075] wherein d represents a depth map of a ground truth, represents an initial depth map, represents a final depth map after correction, and p represents a region with a value in the ground truth.

[0076] The application further discloses a computer program product comprising a computer program which, when executed by a processor, performs the steps of the method as described above.

[0077] The application further discloses a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, performs the steps of the method as described above.

[0078] Compared with the prior art, the application has the advantages that:

[0079] The multi-view three-dimensional reconstruction model for the substation scene of the application not only saves the workload of traditional manual inspection and liberates human resources, but also, compared with single sensor data, uses multiple view image acquisition devices to comprehensively capture each angle and detail of the transformer in the substation, so that the monitoring of the transformer is more fine and accurate. The application introduces image processing technology to perform denoising, distortion correction and other operations on the collected images, optimizes the image quality to make a data set, and provides high-quality input image data for subsequent three-dimensional reconstruction.

[0080] In the three-dimensional reconstruction model proposed by the application, for the feature extraction module, the traditional CNN module is discarded, the advantages of Transformer and CNN are combined, a mamba module similar to the mamba module is introduced, the capture of local features is enhanced, the mamba has a global receptive field while retaining the linear complexity calculation of the CNN. In the process of obtaining the feature volume through homography transformation, a differentiable homography transformation is adopted, which implicitly encodes the camera geometry in the network to construct a 3D cost volume according to the 2D image features, and realizes end-to-end training. In order to adapt to any number of source images in the input, a variance-based metric is proposed to map multiple features to a cost feature in the volume.

[0081] The application uses a method based on depth map reconstruction in the whole three-dimensional reconstruction process, decomposes MVS reconstruction into small problems of each view depth map estimation, and can better adapt to large-scale reconstruction. The method aims to use multi-view image data, realizes accurate three-dimensional reconstruction of the transformer in the transformer substation through comprehensive analysis and processing, and improves the position monitoring accuracy and the effect of structure detail acquisition. BRIEF DESCRIPTION OF DRAWINGS

[0082] Figure 1 The transformer multi-view three-dimensional reconstruction method of the application is in the flowchart of the embodiment.

[0083] Figure 2 The multi-view three-dimensional reconstruction model of the application is in the construction diagram of the embodiment.

[0084] Figure 3 The frequency domain method in the application is in the processing flowchart of the embodiment. DETAILED DESCRIPTION

[0085] The application will be further described below in combination with the drawings and specific embodiments.

[0086] As shown in the drawings, Figure 1 The transformer multi-view three-dimensional reconstruction method for the transformer substation scene provided by the embodiment of the application includes the following steps:

[0087] S1: Constructing a transformer multi-view platform for the transformer substation scene, collecting multi-view high-resolution image data of the transformer;

[0088] S2: Preprocessing and correcting the image, calibrating the camera, obtaining the intrinsic matrix, rotation matrix and translation matrix of the camera;

[0089] S3: According to the needs, performing annotation work and making a transformer scene dataset; dividing the annotated image data into a training set and a validation set;

[0090] S4: Constructing a multi-view three-dimensional reconstruction model for the transformer substation scene, specifically including feature extraction, differentiable homography transformation, cost volume construction and depth map acquisition;

[0091] S5: Inputting the training set into the transformer multi-view three-dimensional reconstruction model for the transformer substation scene constructed in step S4, setting a loss function according to the model, updating the parameters of the model through back propagation, and finally obtaining a trained transformer three-dimensional reconstruction model; then inputting the validation set into the trained transformer three-dimensional reconstruction model, obtaining a three-dimensional reconstruction point cloud result, evaluating the result, and then obtaining a final transformer three-dimensional reconstruction model;

[0092] S6, input the multi-view image data of the transformer into the trained transformer multi-view three-dimensional reconstruction model, and obtain a transformer three-dimensional reconstruction point cloud result.

[0093] In step S1, the transformer multi-view platform of the substation scene is constructed, including the following steps:

[0094] S101: First, install four high-resolution, low-distortion cameras on the ceiling beams of the substation scene. The four cameras cover all key angles and positions of the transformer, ensuring that all key details of the transformer are accurately recorded.

[0095] S102: Capture image data of the transformer from multiple angles and positions, ensuring sufficient overlap and field of view for each view.

[0096] In step S2, image preprocessing includes enhancing the image using a frequency domain method. The purpose is to enhance the local features of interest (transformer structure) and suppress the local features that are not of interest (background, other obstacles, etc.), in order to achieve more accurate three-dimensional reconstruction results. The specific process of enhancing the image using the frequency domain method is shown in Figure 3

[0097] wherein the filter uses a Gaussian high-pass filter to enhance the edges, making the picture clearer, and the specific representation is shown in equation (1):

[0098]

[0099] wherein D0 represents the passband radius, and D(x, y) is the distance to the spectrum center, and the specific representation is shown in equation (2):

[0100]

[0101] wherein X and Y represent the size of the spectrum image, and the spectrum center.

[0102] In step S2, the acquisition of camera internal and external parameters includes the following steps:

[0103] S201: Prepare a chessboard calibration plate of known size. Take 30 groups of images of the calibration plate at different angles and positions using the camera; ensure that the calibration plate can be seen throughout the scene and cover the entire field of view of the camera as much as possible;

[0104] S202: Import the collected calibration images into the MATLAB environment, and set the chessboard edge length according to the actual size of the calibration plate;

[0105] ​S203: display the camera corner point detection result, remove the images with poor effect, calibrate, export the camera internal and external parameters and record and save for standby; wherein, after obtaining the distortion parameters k1, k2, k3, p1, p2, the de-distortion operation is performed, such as formulas (3)-(4), wherein (3) is to remove radial distortion, and (4) is to remove tangential distortion;

[0106]

[0107]

[0108] wherein, (x, y) is the coordinate before de-distortion; (x corrected ,y corrected ) is the coordinate after de-distortion; r=x 2 +y 2 .

[0109] In step S3, the transformer scene dataset is made, including the following steps:

[0110] S301: screening and cleaning the pre-processed and corrected images, and retaining the image data with high quality and various viewing angles; according to the need, the screened data is labeled, and the position information of the transformer and other important structures is labeled;

[0111] S302: divide the labeled data set into a training set and a validation set, and preset the training set to account for 80% of the total data and the validation set to account for 20%.

[0112] In step S4, the feature extraction part includes the following steps:

[0113] S401: the original image adopts a kernel size of 4 and a step size of 4 to obtain non-overlapping 2D feature maps;

[0114] S402: after extracting the feature map, a module is added to enhance the capture of local features according to the mamba architecture; specifically, for a given input feature , it is processed by a depth separable convolution. This process operates independently on each input channel. This method effectively captures the local spatial information of the input feature map while reducing the computational cost and the number of parameters.

[0115] Then, batch normalization is used, which not only effectively reduces the risk of overfitting, but also improves the generalization ability of the model. The specific conversion process obtains the intermediate state F i-1 , as formula (5):

[0116] F i-1 =BN(DWConv 3×3 (F i-2 )) (5)

[0117] S403: Intermediate state F i-1 The channel information is mixed by 1x1 convolution, and the distribution of information is better preserved by the activation function, so that the model can learn more complex feature representations and extract rich multi-scale context information from the input feature map. The activation function uses GELU, which changes the number of channels of the features without changing the spatial size, thereby enhancing the feature representation;

[0118] Finally, the processed features F i-1 are fused with the original input features F i-2 , and the specific process is as formula (6):

[0119]

[0120] where F i is the output feature, and Φ represents the activation function.

[0121] In step S4, the specific process of the differentiable homography transformation part is as follows:

[0122] A 2D-to-3D conversion method based on homographic transformation is applied, and the geometric relationship of the camera is combined into the neural network; the features on other images are warped to 256 depth planes in the camera frustum of the reference image, and there is always a corresponding depth.

[0123] The homography matrix H i that warps the i-th feature to the depth d plane in the camera frustum of the reference image is calculated as formula (7):

[0124]

[0125] where K i , R i , and t i represent the parameters obtained after step S2 for other images, respectively representing the intrinsic matrix, rotation matrix, and translation matrix of the camera; K l , R l , and t l represent the intrinsic matrix and rotation and translation matrices of the selected reference image, and the final obtained depth map is the depth map of the selected reference image.

[0126] In step S4, the specific process of the cost volume construction part is as follows:

[0127] After the differentiable homographic transformation, N feature volumes are obtained, and then these feature volumes are combined into a cost volume. Specifically, the cost volume is calculated pixel by pixel using the variance method, and the process is as formula (8):

[0128]

[0129] wherein V represents the feature volume generated after the differentiable homographic transformation.

[0130] In step S4, the depth map acquisition part includes the following steps:

[0131] S411: The cost volume obtained is passed through a four-stage U-Net structure to generate the possibility size of each pixel at each depth;

[0132] S412: After obtaining the probability distribution on the depth, the initial depth map is calculated, specifically as formula (9):

[0133]

[0134] wherein P(d) represents the possibility size of each pixel at a specific depth obtained in step S451;

[0135] Step S453: Since the initial depth map obtained is still not fine enough at the edges of the object, the original image information is used for refinement; here the depth map and the original image are concatenated into a four-channel input, and the depth residual is obtained through the neural network, and then added to the previous depth map to obtain the final depth map.

[0136] In step S5, the loss function is specifically represented as:

[0137]

[0138] The loss function includes two parts: the L1 distance of the initial depth map to the ground truth and the L1 distance of the refined depth map to the ground truth. Wherein d represents the depth map of the ground truth, represents the initial depth map, represents the final depth map after correction, and p e P represents the region with a value in the ground truth.

[0139] The training set of the data set is input into the above model, and the parameters of the model are updated through the loss function. After the training is completed, the performance evaluation of the hyperparameters is performed through the validation set, and the hyperparameters including network learning rate, batch size, feature dimension, etc. are adjusted according to the results to find the local optimal solution of the hyperparameters and obtain the optimal network model under the current data set.

[0140] In step S6, the newly collected data is input into the three-dimensional model constructed to realize real-time monitoring of the state of the transformer. By comparing the real-time image data with the modeled standard state, any abnormalities or potential problems on the surface of the transformer, such as cracks, oil leaks, etc. can be detected.

[0141] The real-time monitoring can early discover problems, reduce maintenance costs and avoid potential risks caused by equipment failure to power supply; meanwhile, the monitoring results enable the operation and maintenance team to quickly respond to problems, improve the availability of equipment and the overall stability of the system.

[0142] The application acquires image data from different perspectives through multiple cameras, and pre-processes and corrects to optimize image quality and make data sets; a three-dimensional reconstruction model for multi-perspective transformer is constructed, which can be used for relatively fine reconstruction of transformer surface; the model includes feature extraction, differentiable homography transformation, cost volume construction, probability volume prediction, depth map acquisition, depth fusion acquisition three-dimensional point cloud and other modules; this method provides comprehensive and accurate equipment state monitoring capability, and provides an innovative technical solution for operation and maintenance management of power systems.

[0143] The multi-perspective three-dimensional reconstruction model for substation scene of the application not only saves the workload of traditional manual inspection and liberates human resources; compared with single sensor data, the use of multiple perspective image acquisition devices can comprehensively capture the details of the transformer from all angles in the substation, achieving more fine and accurate monitoring of the transformer.

[0144] The application introduces image processing technology to denoise and correct distortion of the collected images, optimizes image quality to make data sets, and provides high-quality input image data for subsequent three-dimensional reconstruction.

[0145] In the proposed three-dimensional reconstruction model, the feature extraction module discards the traditional CNN module, combines the advantages of Transformer and CNN, introduces a mamba module similar to mamba, enhances the capture of local features, and mamba has a global receptive field while retaining the linear complexity calculation of CNN.

[0146] In the process of obtaining the feature volume through homography transformation, the application adopts a differentiable homography transformation, which implicitly encodes the camera geometry in the network to construct a 3D cost volume according to the 2D image features, and realizes end-to-end training. To adapt to any number of source images in the input, a variance-based metric is proposed to map multiple features to a cost feature in the convolution.

[0147] In the whole three-dimensional reconstruction process, the application uses a method based on depth map reconstruction, decomposes MVS reconstruction into a small problem of each view depth map estimation, and can better adapt to large-scale reconstruction.

[0148] The embodiment of the application also provides a transformer multi-perspective three-dimensional reconstruction system for a substation scene, comprising:

[0149] An image acquisition module is configured to acquire multi-view image data of a transformer;

[0150] An image preprocessing module is configured to preprocess the image data to obtain preprocessed image data;

[0151] A dataset production module is configured to perform annotation work on the preprocessed image data to produce a transformer scene dataset;

[0152] A model construction module is configured to construct a transformer multi-view three-dimensional reconstruction model for a substation scene, and includes a feature extraction unit, which specifically includes:

[0153] A convolution module is configured to perform convolution on the original image data to obtain non-overlapping 2D feature maps;

[0154] A processing module is configured to, after extracting the 2D feature maps, process the given input features by a depth separable convolution to capture local spatial information of the feature maps, and then use batch normalization to obtain an intermediate state F i -1 ;

[0155] A fusion module is configured to fuse the intermediate state F i-1 by mixing channel information through convolution and preserving the distribution of information through an activation function, and use a residual connection to fuse the processed features F i-1 with the original input features F i-2 to obtain fused features;

[0156] A model training module is configured to input the transformer scene dataset into the transformer multi-view three-dimensional reconstruction model for a substation scene constructed by the model construction module, set a loss function according to the model, update parameters of the model by back propagation, and finally obtain a trained transformer three-dimensional reconstruction model;

[0157] A three-dimensional reconstruction module is configured to input the multi-view image data of the transformer into the trained transformer multi-view three-dimensional reconstruction model to obtain a transformer three-dimensional reconstruction point cloud result.

[0158] As shown in Figure 2 , in the model construction module, constructing the transformer multi-view three-dimensional reconstruction model for a substation scene further includes a differentiable homography transformation, a cost volume construction, and a depth map acquisition; the differentiable homography transformation includes:

[0159] A 2D-to-3D conversion method based on homography transformation is applied, and the geometric relationship of the camera is combined into the neural network; features on other images are warped onto 256 depth planes in the camera frustum of the reference image, and there is always a corresponding depth;

[0160] Homography H of the ith feature warping to the depth d-plane of the reference image camera frustum i (d) is calculated by the following formula:

[0161]

[0162] wherein K i , R i , t i respectively represent the intrinsic matrix, rotation matrix, translation matrix of the camera; K l , R l , t l represent the intrinsic matrix and rotation and translation matrix of the selected one as a reference image, and the finally obtained depth map is the depth map of the selected reference image.

[0163] Specifically, in the model construction module, the cost volume construction includes:

[0164] After the differentiable homography transformation, N feature volumes are obtained, and the feature volumes are combined into a cost volume. Specifically, the cost volume is calculated pixel by pixel by using the variance method, and the process is shown in the following formula:

[0165]

[0166] wherein V represents the feature volume generated after the differentiable homography transformation.

[0167] Specifically, in the model construction module, the depth map acquisition includes:

[0168] The generation module is configured to generate the possibility size of each pixel at each depth from the obtained cost volume through a four-level U-Net structure;

[0169] The initial depth map acquisition module is configured to calculate the initial depth map after obtaining the probability distribution on the depth.

[0170] The final depth map acquisition module is configured to concatenate the depth map and the original image into a four-channel input, obtain a depth residual through a neural network, and then add the depth residual to the previous depth map to obtain the final depth map.

[0171] Specifically, in the image preprocessing module, the frequency domain method is used to enhance the image.

[0172] Specifically, in the image preprocessing module, the camera is calibrated to obtain the intrinsic and extrinsic parameters of the camera, specifically:

[0173] A checkerboard calibration board with a known size is prepared, and multiple groups of images of the calibration board are captured by using the camera at different angles and positions, so that the calibration board can be seen in the entire scene and the entire field of view of the camera is covered as much as possible.

[0174] The collected calibration image is imported into the MATLAB environment, and the chessboard edge length is set according to the actual size of the calibration board;

[0175] The camera corner detection result is displayed, the images with poor effect are removed, calibration is carried out, and the camera internal and external parameters are exported and recorded for standby.

[0176] Specifically, in the image preprocessing module, the loss function L is specifically represented as:

[0177]

[0178] Wherein d represents the depth map of the ground truth, represents the initial depth map, represents the final depth map after correction, and p represents the region with a value in the ground truth.

[0179] The application also discloses a computer program product, comprising a computer program which executes the steps of the method when run by a processor. The application further discloses a computer readable storage medium having stored thereon a computer program which executes the steps of the method when run by a processor. The product, medium and system of the application correspond to the above method and also have the advantages of the above method.

[0180] The present application can realize all or part of the processes in the above-mentioned embodiment methods, and can also be completed by computer program instruction related hardware. The computer program can be stored in a computer readable storage medium. When the computer program is executed by a processor, the steps of the above-mentioned method embodiment can be realized. The computer program includes computer program code, which can be in the form of source code, object code, executable files or some intermediate forms, etc. The computer readable storage medium includes any entity or device capable of carrying computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal and software distribution medium, etc. The memory is used to store computer programs and / or modules. The processor realizes various functions by running or executing the computer programs and / or modules stored in the memory, and calling the data stored in the memory. The memory can include high-speed random access memory, and can also include non-volatile memory, such as hard disk, memory, plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, at least one magnetic disk storage device, flash memory device, or other volatile solid-state storage device, etc.

[0181] The above is only the preferred embodiment of the present application, and the protection scope of the present application is not limited to the above-mentioned embodiments. Any technical solution falling within the concept of the present application shall fall within the protection scope of the present application. It should be noted that some improvements and refinements made by ordinary skilled persons in the art without departing from the principles of the present application shall be considered as falling within the protection scope of the present application.

Claims

1. A transformer multi-view three-dimensional reconstruction method for a substation scene, characterized in that, The method comprises the steps of: obtaining multi-view image data of a transformer; preprocessing the image data to obtain preprocessed image data; performing annotation work on the preprocessed image data to produce a transformer scene dataset; constructing a transformer multi-view three-dimensional reconstruction model facing a transformer substation scene; inputting the transformer scene dataset into the transformer multi-view three-dimensional reconstruction model facing the transformer substation scene, setting a loss function according to the model, updating parameters of the model through back propagation, and finally obtaining a trained transformer three-dimensional reconstruction model; inputting the multi-view image data of the transformer into the trained transformer multi-view three-dimensional reconstruction model to obtain a transformer three-dimensional reconstruction point cloud result; The transformer multi-view three-dimensional reconstruction model facing the transformer substation scene comprises feature extraction, differentiable homography transformation, cost volume construction, and depth map acquisition; wherein the feature extraction comprises the steps of: performing convolution on the original image data to obtain non-overlapping 2D feature maps; After extracting the 2D feature map, for a given input feature , it is processed by a depthwise separable convolution to capture the local spatial information of the feature map, followed by batch normalization to obtain an intermediate state ; Intermediate state The channel information is mixed by convolution, and the distribution of the information is reserved by the activation function, and the processed features are connected by residual connection The features of the original input are fused to obtain fused features, specifically: wherein, is an output feature, denotes an activation function; The preprocessing of the image comprises calibrating the camera to obtain the internal and external parameters of the camera, and the specific steps are as follows: Prepare a calibration board with a known size of a checkerboard, take multiple images of the calibration board at different angles and positions using the camera, ensure that the calibration board can be seen in the entire scene, and try to cover the entire field of view of the camera; Import the collected calibration images into the MATLAB environment, and set the length of the checkerboard according to the actual size of the calibration board; Display the camera corner detection result, remove the images with poor effects, and then calibrate, export and record the camera internal and external parameters for future use.

2. The transformer multi-view three-dimensional reconstruction method for a substation scene according to claim 1, characterized in that, The differentiable homography transformation comprises the steps of: applying a 2D to 3D conversion method based on homography transformation, and combining the geometric relationship of the camera into the neural network; warp the features on other images to 256 depth planes in the camera frustum of the reference image, and there is always a corresponding depth; The first Depth of a feature warp to a reference image camera frustum Homography of a plane is computed by the following equation: wherein, , , respectively represent the intrinsic matrix, rotation matrix, and translation matrix of the camera; , , represent the intrinsic matrix and rotation and translation matrix of a selected one as a reference image, and the last obtained depth map is the depth map of the selected reference image.

3. The transformer multi-view three-dimensional reconstruction method for a substation scene according to claim 2, characterized in that, The cost volume construction comprises the steps of: After the differentiable homography transformation, we get feature volumes, and merge these feature volumes into a cost volume. Specifically, we use the variance method to obtain the cost volume pixel by pixel, as shown in the following formula: wherein, represent the feature volumes generated after the differentiable homography transformation.

4. The transformer multi-view three-dimensional reconstruction method for a substation scene according to claim 3, characterized in that, The depth map acquisition comprises the following steps: The obtained cost volume is generated through a four-level U-Net structure to obtain the possibility size of each pixel at each depth; After obtaining the probability distribution on the depth, the initial depth map is calculated; Concatenate the depth map and the original image into a four-channel input, pass it through the neural network to obtain the depth residual, and then add it to the previous depth map to obtain the final depth map.

5. The transformer multi-view 3D reconstruction method for substation scene according to any one of claims 1-4, characterized in that, The preprocessing of the image comprises enhancing the image in the frequency domain.

6. The transformer multi-view three-dimensional reconstruction method for a substation scene according to claim 1, characterized in that, The loss function L Specifically represented as: wherein represents a depth map of ground truth, represents an initial depth map, represents a final depth map after correction, represents a region where ground truth has a value.

7. A transformer multi-view three-dimensional reconstruction system for a substation scene, characterized in that, The method comprises the steps of: an image acquisition module for obtaining multi-view image data of a transformer; an image preprocessing module for preprocessing the image data to obtain preprocessed image data; a dataset production module for performing annotation work on the preprocessed image data to produce a transformer scene dataset; a model construction module for constructing a transformer multi-view three-dimensional reconstruction model facing a transformer substation scene; a model training module for inputting the transformer scene dataset into the transformer multi-view three-dimensional reconstruction model facing the transformer substation scene constructed by the model construction module, setting a loss function according to the model, updating parameters of the model through back propagation, and finally obtaining a trained transformer three-dimensional reconstruction model; A three-dimensional reconstruction module is configured to input multi-view image data of the transformer into a trained transformer multi-view three-dimensional reconstruction model to obtain a transformer three-dimensional reconstruction point cloud result. In the model construction module, the transformer multi-view three-dimensional reconstruction model for the substation scene includes a feature extraction unit, a differentiable homography transformation, a cost volume construction, and a depth map acquisition. The feature extraction unit includes: A convolution module is configured to perform convolution on the original image data to obtain non-overlapping 2D feature maps. The processing module is used to extract 2D feature maps and then, for a given input feature map... The feature map is processed using depthwise separable convolution to capture local spatial information, followed by batch normalization to obtain intermediate states. ; A fusion module is configured to fuse the intermediate states The processed features are mixed by convolution, and the distribution of the information is reserved by an activation function, and the processed features are connected by a residual connection The features of the original input are fused to obtain fused features, specifically: wherein, is an output feature, denotes an activation function; In the image preprocessing module, the camera is calibrated to obtain the internal and external parameters of the camera, specifically: Prepare a chessboard calibration board with a known size, use the camera to shoot multiple images of the calibration board at different angles and positions, ensure that the calibration board can be seen in the entire scene, and try to cover the entire field of view of the camera. Import the collected calibration images into the MATLAB environment, and set the chessboard side length according to the actual calibration board size. Display the camera corner detection result, remove the images with poor effect, and calibrate, export the camera internal and external parameters, and record and save them for future use.

8. The transformer multi-view three-dimensional reconstruction system for substation scene according to claim 7, characterized in that, The differentiable homography transformation includes: A 2D to 3D conversion method based on homography transformation is applied, and the geometric relationship of the camera is combined into the neural network; the features on other images are warped to 256 depth planes in the camera frustum of the reference image, and there is always a corresponding depth; The first Depth of a feature warp to a reference image camera frustum Homography of a plane is computed by the following equation: wherein, , , respectively represent the intrinsic matrix, the rotation matrix, and the translation matrix of the camera; , , represent the intrinsic matrix and the rotation and translation matrix of the selected one as the reference image, and the finally obtained depth map is the depth map of the selected reference image.

9. The transformer multi-view three-dimensional reconstruction system for substation scene of claim 8, wherein, In the model construction module, the cost volume construction includes: After the differentiable homography transformation, we get feature volumes, and merge these feature volumes into a cost volume. Specifically, we use the variance method to obtain the cost volume pixel by pixel, as shown in the following formula: wherein, represent the feature volumes generated after the differentiable homography transformation.

10. The transformer multi-view three-dimensional reconstruction system for substation scene of claim 9, wherein, In the model construction module, the depth map acquisition includes: A generation module is configured to generate the possibility size of each pixel at each depth from the obtained cost volume through a four-level U-Net structure; An initial depth map acquisition module is configured to calculate the initial depth map after obtaining the probability distribution on the depth. A final depth map acquisition module is configured to concatenate the depth map and the original image into a four-channel input, pass it through the neural network to obtain the depth residual, and then add it to the previous depth map to obtain the final depth map.

11. The transformer multi-view three-dimensional reconstruction system for substation scene according to any one of claims 7-9, characterized in that, In the image preprocessing module, the image is enhanced using the frequency domain method.

12. The transformer multi-view three-dimensional reconstruction system for substation scene according to any one of claims 7-9, characterized in that, In the image preprocessing module, the loss function L Specifically represented as: wherein represents a depth map of ground truth, represents an initial depth map, represents a final depth map after correction, represents a region where ground truth has a value.

13. A computer program product comprising a computer program, characterized in that, The computer program, when executed by the processor, performs the steps of the method of any one of claims 1-6.

14. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program, when executed by the processor, performs the steps of the method of any one of claims 1-6.

Images