Transformer substation scene-oriented transformer multi-view three-dimensional reconstruction method and system

Abstract

The invention discloses a transformer substation scene-oriented transformer multi-view three-dimensional reconstruction method and system. The method comprises the steps of obtaining multi-view image data of a transformer; preprocessing the image data to obtain preprocessed image data; marking the preprocessed image data, and making a transformer scene data set; constructing a transformer multi-view three-dimensional reconstruction model oriented to the transformer substation scene; inputting the transformer scene data set into a transformer multi-view three-dimensional reconstruction model oriented to the transformer substation scene, setting a loss function according to the model, performing back propagation on the model to update parameters, and finally obtaining a trained transformer three-dimensional reconstruction model; and inputting the multi-view image data of the transformer into the trained multi-view three-dimensional reconstruction model of the transformer to obtain a three-dimensional reconstruction point cloud result of the transformer. The method has the advantages of high reconstruction precision and the like.

Description

Technical Field

[0001] The present invention mainly relates to the technical field of electric power equipment, and in particular to a transformer multi-view 3D reconstruction method and system for substation scenes. Background Art

[0002] In the current power system field, with the development of smart grids and the improvement of the automation level of power equipment, it is particularly important to efficiently monitor and manage substation equipment. As a key node in the power system, the substation undertakes the important tasks of power conversion, distribution and regulation. Among them, the transformer is one of the core equipment, and its operating status directly affects the safety and stability of the power grid. In traditional substation management, the monitoring of transformers often relies on manual inspections and single-angle monitoring equipment. This method has the problems of narrow monitoring range, low efficiency and incomplete grasp of equipment status. In addition, the current monitoring technology still has room for improvement in efficiency and accuracy, and cannot meet the high requirements for monitoring the complex internal structure and details of the transformer.

[0003] In view of the above problems, the introduction of advanced sensing technology and data processing technology to achieve comprehensive monitoring and intelligent management of transformers and their surrounding environment is the only way forward in the field of power systems. In the field of power systems, research directions that have been proposed include the application of sensor networks, optimization of data fusion and analysis algorithms, and research on the prediction of equipment operating status. However, current research still faces some challenges and problems. For example, most of the existing monitoring equipment is single-angle or single-sensing mode, which cannot achieve all-round, high-precision three-dimensional reconstruction and real-time monitoring of transformers. In addition, traditional image processing and data analysis methods lack efficient three-dimensional reconstruction technology support when dealing with complex power equipment structures, resulting in limited accuracy and reliability of monitoring results.

[0004] 3D reconstruction technology uses data collection from multiple perspectives, combined with advanced computer vision and image processing algorithms, to achieve high-precision 3D model reconstruction of the transformer and its surroundings. This method can not only fully display the structure and details of the equipment, but also provide effective support for remote monitoring, operation status analysis and maintenance decisions of the equipment. Therefore, it is particularly necessary and important to introduce a 3D reconstruction method based on multi-perspective image fusion.

[0005] The multi-view 3D reconstruction method for transformers in substation scenarios has attracted more and more attention in recent years. Compared with traditional methods based on photogrammetry or laser scanning, the multi-view 3D reconstruction technology is more challenging, and the following difficulties may exist in the specific implementation process:

[0006] 1. In substation scenarios, transformers are usually located in complex environments, surrounded by buildings, equipment, and other obstacles. These elements may interfere with or block the transformer reconstruction process. It is difficult to reasonably deploy multiple cameras to obtain key information about the transformer.

[0007] 2. Different cameras have their own internally constructed coordinate systems, which leads to differences in coordinate systems for images taken by different cameras. At the same time, there may be deviations in time synchronization, resulting in the inability to accurately align the collected images or errors in time.

[0008] 3. The MVS method is usually used for scene reconstruction, but this method has difficulty in handling weak texture areas and non-Lambertian surfaces because these areas are difficult to match, resulting in incomplete reconstruction. Summary of the invention

[0009] In view of the technical problems existing in the prior art, the present invention provides a transformer multi-view 3D reconstruction method and system for substation scenes with high reconstruction accuracy.

[0010] In order to solve the above technical problems, the technical solution proposed by the present invention is:

[0011] A transformer multi-view 3D reconstruction method for a substation scene, comprising the following steps:

[0012] Obtain multi-view image data of the transformer;

[0013] Preprocessing the image data to obtain preprocessed image data;

[0014] Label the preprocessed image data to create a transformer scene dataset;

[0015] Construct a multi-view 3D reconstruction model of transformer for substation scenarios;

[0016] The transformer scene dataset is input into the transformer multi-view 3D reconstruction model for the substation scene. The loss function is set according to the model, and the model is back-propagated to update the parameters, and finally a trained transformer 3D reconstruction model is obtained.

[0017] The transformer's multi-view image data is input into the trained transformer's multi-view 3D reconstruction model to obtain the transformer's 3D reconstruction point cloud result.

[0018] Preferably, constructing a transformer multi-view 3D reconstruction model for a substation scene also includes feature extraction, differentiable homography transformation, cost volume construction and depth map acquisition; wherein feature extraction includes the steps of:

[0019] Convolve the original image data to obtain non-overlapping 2D feature maps;

[0020] After extracting the 2D feature map, for a given input feature F i-2 , processed by depthwise separable convolution to capture the local spatial information of the feature map, and then batch normalization is used to obtain the intermediate state F i-1 ;

[0021] Intermediate state F i-1 The channel information is mixed by convolution, and the distribution of information is preserved by activation function. The processed features F are connected by residual connection. i-1 The feature F of the original input i-2 The fused features are obtained.

[0022] Preferably, the differentiable homography transform comprises the steps of:

[0023] Apply a 2D to 3D conversion method based on homography transformation and incorporate the geometric relationship of the camera into the neural network; warp the features on other images to the 256 depth planes in the camera cone of the reference image, and there is always a corresponding depth;

[0024] The homography matrix H of the i-th feature warp to the depth d plane of the camera frustum of the reference image i (d) is calculated by the following formula:

[0025]

[0026] Among them, K i , R i ,t i They are respectively represented as the camera's intrinsic parameter matrix, rotation matrix, and translation matrix; K l , R l ,t l It means that an intrinsic parameter matrix and rotation and translation matrix are selected as the reference image, and the depth map finally obtained is the depth map of the selected reference image.

[0027] Preferably, the cost volume construction comprises the steps of:

[0028] After the differentiable homography transformation, N feature volumes are obtained, which are merged into the cost volume. Specifically, the variance method is used to calculate the cost volume pixel by pixel. The process is shown in the following formula:

[0029]

[0030] Among them, V represents the feature volume generated after the differentiable homography transformation.

[0031] Preferably, the depth map acquisition comprises the following steps:

[0032] The obtained cost volume passes through a four-level U-Net structure to generate the likelihood size of each pixel at each depth;

[0033] After obtaining the probability distribution of depth, calculate the initial depth map;

[0034] The depth map and the original image are concatenated into a four-channel input, the depth residual is obtained through the neural network, and then added to the previous depth map to obtain the final depth map.

[0035] Preferably, preprocessing the image includes: enhancing the image using a frequency domain method.

[0036] Preferably, preprocessing the image includes calibrating the camera to obtain the internal and external parameters of the camera, and the specific steps are:

[0037] Prepare a checkerboard calibration plate of known size, and use the camera to take multiple sets of images of the calibration plate at different angles and positions to ensure that the calibration plate can be seen in the entire scene and covers the entire camera field of view as much as possible;

[0038] Import the collected calibration image into the MATLAB environment and set the chessboard side length according to the actual calibration board size;

[0039] Display the camera corner detection results, remove images with poor results and perform calibration, export the camera internal and external parameters and record them for future use.

[0040] Preferably, the loss function L is specifically expressed as:

[0041]

[0042] Where d represents the depth map of the ground truth, represents the initial depth map, represents the final depth map after correction, and p∈P represents the area with value in ground truth.

[0043] The present invention also discloses a transformer multi-view 3D reconstruction system for substation scenes, including:

[0044] An image acquisition module, used to acquire multi-view image data of the transformer;

[0045] An image preprocessing module is used to preprocess image data to obtain preprocessed image data;

[0046] The data set making module is used to label the pre-processed image data and make transformer scene data sets;

[0047] Model building module, used to build a multi-view 3D reconstruction model of the transformer for substation scenarios;

[0048] The model training module is used to input the transformer scene data set into the transformer multi-view 3D reconstruction model for the substation scene constructed in the model construction module, set the loss function according to the model, perform back propagation to update the parameters of the model, and finally obtain the trained transformer 3D reconstruction model;

[0049] The 3D reconstruction module is used to input the multi-view image data of the transformer into the trained multi-view 3D reconstruction model of the transformer to obtain the 3D reconstruction point cloud result of the transformer.

[0050] Preferably, in the model building module, building a transformer multi-view 3D reconstruction model for a substation scene includes a feature extraction unit, a differentiable homography transformation, a cost volume construction, and a depth map acquisition;

[0051] The feature extraction unit comprises:

[0052] The convolution module is used to: convolve the original image data to obtain non-overlapping 2D feature maps;

[0053] Processing module, used to extract 2D feature maps and then Processed by depthwise separable convolution, capturing the local spatial information of the feature map, and then using batch normalization to obtain the intermediate state F i -1 ;

[0054] The fusion module is used to transform the intermediate state F i-1 The channel information is mixed by convolution, and the distribution of information is preserved by activation function. The processed features F are connected by residual connection. i-1 The feature F of the original input i-2 The fused features are obtained.

[0055] Preferably, the differentiable homography transform comprises:

[0056] Apply a 2D to 3D conversion method based on homography transformation and incorporate the geometric relationship of the camera into the neural network; warp the features on other images to the 256 depth planes in the camera cone of the reference image, and there is always a corresponding depth;

[0057] The homography matrix H of the i-th feature warp to the depth d plane of the camera frustum of the reference image i (d) is calculated by the following formula:

[0058]

[0059] Among them, Ki , R i ,t i They are respectively represented as the camera's intrinsic parameter matrix, rotation matrix, and translation matrix; K l , R l ,t l It means that an intrinsic parameter matrix and rotation and translation matrix are selected as the reference image, and the depth map finally obtained is the depth map of the selected reference image.

[0060] Preferably, in the model building module, the cost volume building includes:

[0061] After the differentiable homography transformation, N feature volumes are obtained, which are merged into the cost volume. Specifically, the variance method is used to calculate the cost volume pixel by pixel. The process is shown in the following formula:

[0062]

[0063] Among them, V represents the feature volume generated after the differentiable homography transformation.

[0064] Preferably, in the model building module, the depth map acquisition includes:

[0065] A generation module is used to generate the probability size of each pixel at each depth by passing the acquired cost volume through a four-level U-Net structure;

[0066] An initial depth map acquisition module is used to obtain the probability distribution of depth and then calculate the initial depth map;

[0067] The final depth map acquisition module is used to concatenate the depth map and the original image into a four-channel input, obtain the depth residual through the neural network, and then add it to the previous depth map to obtain the final depth map.

[0068] Preferably, in the image preprocessing module, the image is enhanced using a frequency domain method.

[0069] Preferably, in the image preprocessing module, the camera is calibrated to obtain the internal and external parameters of the camera, specifically:

[0070] Prepare a checkerboard calibration plate of known size, and use the camera to take multiple sets of images of the calibration plate at different angles and positions to ensure that the calibration plate can be seen in the entire scene and covers the entire camera field of view as much as possible;

[0071] Import the collected calibration image into the MATLAB environment and set the chessboard side length according to the actual calibration board size;

[0072] Display the camera corner detection results, remove images with poor results and perform calibration, export the camera internal and external parameters and record them for future use.

[0073] Preferably, in the image preprocessing module, the loss function L is specifically expressed as:

[0074]

[0075] Where d represents the depth map of the ground truth, represents the initial depth map, represents the final depth map after correction, and p∈P represents the area with value in ground truth.

[0076] The present invention also discloses a computer program product, comprising a computer program, wherein the computer program executes the steps of the method described above when executed by a processor.

[0077] The present invention further discloses a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the method described above are executed.

[0078] Compared with the prior art, the advantages of the present invention are:

[0079] The multi-view 3D reconstruction model for substation scenes of the present invention not only saves the workload of traditional manual inspection and liberates human resources; at the same time, compared with single sensor data, it uses image acquisition equipment with multiple viewpoints to fully capture all angles and details of the transformer in the substation, achieving a more precise and accurate monitoring effect on the transformer. The present invention introduces image processing technology to perform operations such as denoising and distortion correction on the collected images, optimizes the image quality to produce a data set, and provides high-quality input image data for subsequent 3D reconstruction.

[0080] In the proposed 3D reconstruction model, the present invention abandons the traditional CNN module for the feature extraction module, combines the advantages of Transformer and CNN, introduces a module similar to Mamba, and enhances the capture of local features. Mamba has a global receptive field while retaining the linear complexity calculation of CNN. In the process of obtaining feature volumes through homography transformation, the present invention adopts a differentiable homography transformation, which implicitly encodes the camera geometry in the network to construct a 3D cost volume based on 2D image features and realizes end-to-end training. In order to adapt to any number of source images in the input, a variance-based metric is proposed to map multiple features to one cost feature in the volume.

[0081] In the whole 3D reconstruction process, the present invention uses a method based on depth map reconstruction to decompose MVS reconstruction into small problems of depth map estimation for each view, which can better adapt to large-scale reconstruction. This method aims to use multi-view image data, through comprehensive analysis and processing, to achieve accurate 3D reconstruction of transformers in substations, so as to improve the accuracy of position monitoring and the effect of obtaining structural details. BRIEF DESCRIPTION OF THE DRAWINGS

[0082] Figure 1 The figure is a flow chart of an embodiment of the transformer multi-view 3D reconstruction method of the present invention.

[0083] Figure 2 This is a construction diagram of a multi-view 3D reconstruction model in an embodiment of the present invention.

[0084] Figure 3 It is a processing flow chart of the frequency domain method in an embodiment of the present invention. DETAILED DESCRIPTION

[0085] The present invention is further described below in conjunction with the accompanying drawings and specific embodiments.

[0086] like Figure 1 As shown, the transformer multi-view 3D reconstruction method for a substation scene provided by an embodiment of the present invention includes the following steps:

[0087] S1: Build a transformer multi-view platform for substation scenes to collect high-resolution multi-view image data of transformers;

[0088] S2: Preprocess and correct the image. The camera needs to be calibrated to obtain the camera's intrinsic matrix, rotation matrix, and translation matrix.

[0089] S3: Perform labeling work as needed to create a transformer scene dataset; divide the labeled image data into a training set and a validation set;

[0090] S4: Construct a multi-view 3D reconstruction model for substation scenes, including feature extraction, differentiable homography transformation, cost volume construction, and depth map acquisition;

[0091] S5: input the training set into the transformer multi-view 3D reconstruction model for the substation scene constructed in step S4, set the loss function according to the model, back-propagate the model to update the parameters, and finally obtain the trained 3D reconstruction model of the transformer; then input the validation set into the trained 3D reconstruction model of the transformer to obtain the 3D reconstruction point cloud result, and evaluate the result to obtain the final 3D reconstruction model of the transformer;

[0092] S6. Input the collected multi-view image data of the transformer into the trained multi-view 3D reconstruction model of the transformer to obtain the 3D reconstruction point cloud result of the transformer.

[0093] In step S1, building a transformer multi-view platform for substation scenarios includes the following steps:

[0094] S101: First, four high-resolution, low-distortion cameras were installed on the ceiling beams of the substation scene. The four cameras covered all key angles and positions of the transformer to ensure that all key details of the transformer could be accurately recorded.

[0095] S102: Capture image data of the transformer from multiple angles and positions simultaneously, ensuring that each viewing angle has sufficient overlapping area and field of view coverage.

[0096] In step S2, image preprocessing includes: enhancing the image using the frequency domain method, the purpose is to enhance the local features of interest (transformer structure) and suppress the local features of no interest (background, other obstacles, etc.), so as to achieve a more accurate 3D reconstruction effect. The specific process of frequency domain method to enhance the image is as follows: Figure 3 As shown:

[0097] The filter uses a Gaussian high-pass filter to enhance the edge, making the image clearer. The specific expression is shown in formula (1):

[0098]

[0099] Where D0 represents the passband radius, and D(x,y) is the distance to the center of the spectrum, as shown in formula (2):

[0100]

[0101] Where X and Y represent the size of the spectrum image, The center of the spectrum.

[0102] In step S2, obtaining the internal and external parameters of the camera includes the following steps:

[0103] S201: prepare a checkerboard calibration plate of known size, and use a camera to take 30 sets of images of the calibration plate at different angles and positions; ensure that the calibration plate can be seen in the entire scene and cover the entire camera field of view as much as possible;

[0104] S202: Import the collected calibration image into the MATLAB environment, and set the chessboard side length according to the actual calibration board size;

[0105] S203: Display the camera corner detection results, remove the images with poor results and perform calibration, derive the camera internal and external parameters and record them for future use; after obtaining the distortion parameters k1, k2, k3, p1, p2, perform the de-distortion operation as shown in formula (3)-(4), where (3) is to remove radial distortion and (4) is to remove tangential distortion;

[0106]

[0107]

[0108] Among them, (x, y) is the coordinate before distortion removal; (x corrected ,y corrected ) is the coordinate after dedistortion; r = x 2 +y 2 .

[0109] In step S3, making a transformer scene dataset includes the following steps:

[0110] S301: Screening and cleaning the preprocessed and corrected images to retain high-quality image data with diverse viewing angles; annotating the screened data as needed to mark the location information of transformers and other important structures;

[0111] S302: Divide the labeled data set into a training set and a validation set, where the training set accounts for 80% of the total data and the validation set accounts for 20%.

[0112] In step S4, the feature extraction part includes the following steps:

[0113] S401: The original image is convolved with a kernel size of 4 and a stride of 4 to obtain a non-overlapping 2D feature map;

[0114] S402: After extracting the feature map, a module is added to enhance the capture of local features with reference to the Mamba architecture; specifically, for a given input feature Processed via depthwise separable convolutions, this process operates on each input channel independently. This approach effectively captures the local spatial information of the input feature map while reducing the computational cost and number of parameters.

[0115] Next, batch normalization is used to effectively reduce the risk of overfitting and improve the generalization ability of the model. The specific conversion process obtains the intermediate state F i-1 As shown in formula (5):

[0116] F i-1 =BN(DWConv 3×3 (F i-2 )) (5)

[0117] S403: Intermediate state F i-1 By mixing channel information through 1x1 convolution and better preserving the distribution of information through activation function, the model can learn more complex feature representations and extract rich multi-scale contextual information from the input feature map. The activation function uses GELU, which changes the number of channels of the feature without changing the spatial size, thereby enhancing the feature representation;

[0118] Finally, the processed feature F is connected using residual connections i-1 The feature F of the original input i-2 The specific process is as follows:

[0119]

[0120] Among them, F i is the output feature, and Φ represents the activation function.

[0121] In step S4, the specific process of the differentiable homography transformation is as follows:

[0122] A 2D to 3D conversion method based on homography transformation is applied, and the geometric relationship of the camera is incorporated into the neural network; the features on other images are warped to the 256 depth planes in the camera cone of the reference image, and there is always a corresponding depth.

[0123] The homography matrix H of the i-th feature warp to the depth d plane of the camera frustum of the reference image i (d) Calculated by formula (7):

[0124]

[0125] Among them, K i , R i ,t i represents the parameters of other images obtained after step S2, which are respectively represented as the camera's intrinsic parameter matrix, rotation matrix, and translation matrix; K l , R l ,t l It means that an intrinsic parameter matrix and rotation and translation matrix are selected as the reference image, and the depth map finally obtained is the depth map of the selected reference image.

[0126] In step S4, the specific process of cost volume construction is as follows:

[0127] After the differentiable homography transformation, N feature volumes are obtained. The next step is to merge these feature volumes into a cost volume. Specifically, the variance method is used to calculate the cost volume pixel by pixel. The process is shown in formula (8):

[0128]

[0129] Among them, V represents the feature volume generated after the differentiable homography transformation.

[0130] In step S4, the depth map acquisition part includes the following steps:

[0131] S411: The obtained cost volume is passed through a four-level U-Net structure to generate the probability size of each pixel at each depth;

[0132] S412: After obtaining the probability distribution of depth, the initial depth map is calculated, specifically as shown in formula (9):

[0133]

[0134] Wherein, P(d) represents the probability size of each pixel at the specific depth obtained in step S451;

[0135] Step S453: Since the initial depth map is not fine enough at the edge of the object, the original image information is used for refinement; here, the depth map and the original image are concatenated into a four-channel input, the depth residual is obtained through the neural network, and then added to the previous depth map to obtain the final depth map.

[0136] In step S5, the loss function is specifically expressed as:

[0137]

[0138] The loss has two parts, the L1 distance from the initial depth map to the ground truth and the L1 distance from the refined depth map to the ground truth. Where d represents the depth map of the ground truth, represents the initial depth map, represents the final depth map after correction, and p∈P represents the area with value in ground truth.

[0139] The training set of the data set is input into the above model, and the parameters of the model are updated through the loss function. After the training is completed, the performance evaluation of the hyperparameters is performed through the validation set. According to the results, the hyperparameters including the network learning rate, batch size, feature dimension, etc. are adjusted to find the local optimal solution of the hyperparameters and obtain the optimal network model under the current data set.

[0140] In step S6, the newly acquired data is input into the constructed 3D model to achieve real-time monitoring of the transformer status. By comparing the real-time image data with the modeled standard status, any abnormalities or potential problems on the transformer surface, such as cracks, oil leakage, etc., can be detected.

[0141] The above real-time monitoring can detect problems early, reduce maintenance costs and avoid potential risks to power supply caused by equipment failure; at the same time, the monitoring results enable the operation and maintenance team to respond to problems quickly, improve equipment availability and the overall stability of the system.

[0142] The present invention uses multiple cameras to capture image data from different perspectives, and performs preprocessing and correction to optimize image quality and produce a data set; constructs a three-dimensional reconstruction model for a multi-perspective transformer, which can be used for relatively fine reconstruction tasks of the transformer surface; the model includes modules such as feature extraction, differentiable homography transformation, cost volume construction, probabilistic volume prediction, depth map acquisition, and deep fusion acquisition of three-dimensional point cloud; this method provides comprehensive and accurate equipment status monitoring capabilities, and provides innovative technical solutions for the operation and maintenance management of power systems.

[0143] The multi-perspective 3D reconstruction model for substation scenarios of the present invention not only saves the workload of traditional manual inspections and liberates human resources; at the same time, compared with single sensor data, it uses image acquisition devices with multiple perspectives to comprehensively capture all angles and details of the transformer in the substation, thereby achieving more precise and accurate monitoring of the transformer.

[0144] The present invention introduces image processing technology to perform operations such as denoising and distortion correction on the collected images, optimizes the image quality to produce a data set, and provides high-quality input image data for subsequent three-dimensional reconstruction.

[0145] In the 3D reconstruction model proposed in the present invention, for the feature extraction module, the traditional CNN module is abandoned, and the advantages of Transformer and CNN are combined to introduce a module similar to Mamba to enhance the capture of local features. Mamba has a global receptive field while retaining the linear complexity calculation of CNN.

[0146] In the process of obtaining feature volumes through homography transformation, the present invention adopts a differentiable homography transformation, which implicitly encodes the camera geometry in the network to construct a 3D cost volume based on 2D image features and achieves end-to-end training. To adapt to any number of source images in the input, a variance-based metric is proposed to map multiple features to one cost feature in the volume.

[0147] During the entire 3D reconstruction process, the present invention uses a reconstruction method based on a depth map, decomposing the MVS reconstruction into small problems of estimating the depth map of each view, which can better adapt to large-scale reconstruction.

[0148] The embodiment of the present invention further provides a transformer multi-view 3D reconstruction system for substation scenes, including:

[0149] An image acquisition module, used to acquire multi-view image data of the transformer;

[0150] An image preprocessing module is used to preprocess image data to obtain preprocessed image data;

[0151] The data set making module is used to label the pre-processed image data and make transformer scene data sets;

[0152] The model building module is used to build a multi-view 3D reconstruction model of a transformer for a substation scenario, including a feature extraction unit, specifically including:

[0153] The convolution module is used to: convolve the original image data to obtain non-overlapping 2D feature maps;

[0154] Processing module, used to extract 2D feature maps and then Processed by depthwise separable convolution, capturing the local spatial information of the feature map, and then using batch normalization to obtain the intermediate state F i -1 ;

[0155] The fusion module is used to transform the intermediate state F i-1 The channel information is mixed by convolution, and the distribution of information is preserved by activation function. The processed features F are connected by residual connection. i-1 The feature F of the original input i-2 Perform fusion to obtain fused features;

[0156] The model training module is used to input the transformer scene data set into the transformer multi-view 3D reconstruction model for the substation scene constructed in the model construction module, set the loss function according to the model, perform back propagation to update the parameters of the model, and finally obtain the trained transformer 3D reconstruction model;

[0157] The 3D reconstruction module is used to input the multi-view image data of the transformer into the trained multi-view 3D reconstruction model of the transformer to obtain the 3D reconstruction point cloud result of the transformer.

[0158] like Figure 2 As shown, in the model building module, building a transformer multi-view 3D reconstruction model for the substation scene also includes differentiable homography transformation, cost volume construction and depth map acquisition; the differentiable homography transformation includes:

[0159] Apply a 2D to 3D conversion method based on homography transformation and incorporate the geometric relationship of the camera into the neural network; warp the features on other images to the 256 depth planes in the camera cone of the reference image, and there is always a corresponding depth;

[0160] The homography matrix H of the i-th feature warp to the depth d plane of the camera frustum of the reference image i (d) is calculated by the following formula:

[0161]

[0162] Among them, K i , R i ,t i They are respectively represented as the camera's intrinsic parameter matrix, rotation matrix, and translation matrix; K l , R l ,t l It means that an intrinsic parameter matrix and rotation and translation matrix are selected as the reference image, and the depth map finally obtained is the depth map of the selected reference image.

[0163] Specifically, in the model building module, the cost volume construction includes:

[0164] After the differentiable homography transformation, N feature volumes are obtained, which are merged into the cost volume. Specifically, the variance method is used to calculate the cost volume pixel by pixel. The process is shown in the following formula:

[0165]

[0166] Among them, V represents the feature volume generated after the differentiable homography transformation.

[0167] Specifically, in the model building module, depth map acquisition includes:

[0168] A generation module is used to generate the probability size of each pixel at each depth by passing the acquired cost volume through a four-level U-Net structure;

[0169] An initial depth map acquisition module is used to obtain the probability distribution of depth and then calculate the initial depth map;

[0170] The final depth map acquisition module is used to concatenate the depth map and the original image into a four-channel input, obtain the depth residual through the neural network, and then add it to the previous depth map to obtain the final depth map.

[0171] Specifically, in the image preprocessing module, the frequency domain method is used to enhance the image.

[0172] Specifically, in the image preprocessing module, the camera is calibrated to obtain the internal and external parameters of the camera, specifically:

[0173] Prepare a checkerboard calibration plate of known size, and use the camera to take multiple sets of images of the calibration plate at different angles and positions to ensure that the calibration plate can be seen in the entire scene and covers the entire camera field of view as much as possible;

[0174] Import the collected calibration image into the MATLAB environment and set the chessboard side length according to the actual calibration board size;

[0175] Display the camera corner detection results, remove images with poor results and perform calibration, export the camera internal and external parameters and record them for future use.

[0176] Specifically, in the image preprocessing module, the loss function L is specifically expressed as:

[0177]

[0178] Where d represents the depth map of the ground truth, represents the initial depth map, represents the final depth map after correction, and p∈P represents the area with value in ground truth.

[0179] The present invention also discloses a computer program product, including a computer program, which executes the steps of the above method when executed by a processor. The present invention further discloses a computer-readable storage medium, which stores a computer program, which executes the steps of the above method when executed by a processor. The product, medium and system of the present invention correspond to the above method and also have the advantages described in the above method.

[0180] The present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by hardware related to computer program instructions. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned method embodiment when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. Computer-readable storage media include: any entity or device that can carry computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium. The memory is used to store computer programs and / or modules, and the processor implements various functions by running or executing computer programs and / or modules stored in the memory, and calling data stored in the memory. The memory may include high-speed random access memory and may also include non-volatile memory, such as a hard disk, internal memory, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0181] The above are only preferred embodiments of the present invention, and the protection scope of the present invention is not limited to the above embodiments. All technical solutions under the concept of the present invention belong to the protection scope of the present invention. It should be pointed out that for ordinary technicians in this technical field, some improvements and modifications without departing from the principle of the present invention should be regarded as the protection scope of the present invention.

Claims

1. A transformer multi-view 3D reconstruction method for substation scenes, characterized in that: Includes steps: Obtain multi-view image data of the transformer; Preprocessing the image data to obtain preprocessed image data; Label the preprocessed image data to create a transformer scene dataset; Construct a multi-view 3D reconstruction model of transformer for substation scenarios; The transformer scene dataset is input into the transformer multi-view 3D reconstruction model for the substation scene. The loss function is set according to the model, and the model is back-propagated to update the parameters, and finally a trained transformer 3D reconstruction model is obtained. The transformer's multi-view image data is input into the trained transformer's multi-view 3D reconstruction model to obtain the transformer's 3D reconstruction point cloud result.

2. The transformer multi-view 3D reconstruction method for substation scenes according to claim 1 is characterized in that: Constructing a transformer multi-view 3D reconstruction model for substation scenes includes feature extraction, differentiable homography transformation, cost volume construction and depth map acquisition; the feature extraction includes the following steps: Convolve the original image data to obtain non-overlapping 2D feature maps; After extracting the 2D feature map, for a given input feature F i-2 , processed by depthwise separable convolution to capture the local spatial information of the feature map, and then batch normalization is used to obtain the intermediate state F i-1 ; Intermediate state F i-1 The channel information is mixed by convolution, and the distribution of information is preserved by activation function. The processed features F are connected by residual connection. i-1 The feature F of the original input i-2 The fused features are obtained.

3. The transformer multi-view 3D reconstruction method for substation scenes according to claim 2 is characterized in that: The differentiable homography transformation comprises the steps of: Apply a 2D to 3D conversion method based on homography transformation and incorporate the geometric relationship of the camera into the neural network; warp the features on other images to the 256 depth planes in the camera cone of the reference image, and there is always a corresponding depth; The homography matrix H of the i-th feature warp to the depth d plane of the camera frustum of the reference image i (d) is calculated by the following formula: Among them, K i , R i ,t i They are respectively represented as the camera's intrinsic parameter matrix, rotation matrix, and translation matrix; K l , R l ,t l It means that an intrinsic parameter matrix and rotation and translation matrix are selected as the reference image, and the depth map finally obtained is the depth map of the selected reference image.

4. The transformer multi-view 3D reconstruction method for substation scenes according to claim 3 is characterized in that: The cost volume construction includes the steps of: After the differentiable homography transformation, N feature volumes are obtained, which are merged into the cost volume. Specifically, the variance method is used to calculate the cost volume pixel by pixel. The process is shown in the following formula: Among them, V represents the feature volume generated after the differentiable homography transformation.

5. The transformer multi-view 3D reconstruction method for substation scenes according to claim 4 is characterized in that: The depth map acquisition comprises the following steps: The obtained cost volume passes through a four-level U-Net structure to generate the likelihood size of each pixel at each depth; After obtaining the probability distribution of depth, calculate the initial depth map; The depth map and the original image are concatenated into a four-channel input, the depth residual is obtained through the neural network, and then added to the previous depth map to obtain the final depth map.

6. The transformer multi-view 3D reconstruction method for substation scenes according to any one of claims 1 to 5, characterized in that: The image preprocessing includes: enhancing the image using the frequency domain method.

7. The transformer multi-view 3D reconstruction method for substation scenes according to any one of claims 1 to 5, characterized in that: Image preprocessing includes calibrating the camera and obtaining the internal and external parameters of the camera. The specific steps are as follows: Prepare a checkerboard calibration plate of known size, and use the camera to take multiple sets of images of the calibration plate at different angles and positions to ensure that the calibration plate can be seen in the entire scene and covers the entire camera field of view as much as possible; Import the collected calibration image into the MATLAB environment and set the chessboard side length according to the actual calibration board size; Display the camera corner detection results, remove images with poor results and perform calibration, export the camera internal and external parameters and record them for future use.

8. The transformer multi-view 3D reconstruction method for substation scenes according to any one of claims 1 to 5, characterized in that: The loss function L is specifically expressed as: Where d represents the depth map of the ground truth, represents the initial depth map, represents the final depth map after correction, and p∈P represents the area with value in ground truth.

9. A transformer multi-view 3D reconstruction system for substation scenes, characterized in that: include: An image acquisition module, used to acquire multi-view image data of the transformer; An image preprocessing module is used to preprocess image data to obtain preprocessed image data; The data set making module is used to label the pre-processed image data and make transformer scene data sets; Model building module, used to build a multi-view 3D reconstruction model of the transformer for substation scenarios; The model training module is used to input the transformer scene data set into the transformer multi-view 3D reconstruction model for the substation scene constructed in the model construction module, set the loss function according to the model, perform back propagation to update the parameters of the model, and finally obtain the trained transformer 3D reconstruction model; The 3D reconstruction module is used to input the multi-view image data of the transformer into the trained multi-view 3D reconstruction model of the transformer to obtain the 3D reconstruction point cloud result of the transformer.

10. The transformer multi-view 3D reconstruction system for substation scenes according to claim 9, characterized in that: In the model building module, the construction of a transformer multi-view 3D reconstruction model for substation scenarios includes feature extraction unit, differentiable homography transformation, cost volume construction and depth map acquisition; The feature extraction unit comprises: The convolution module is used to: convolve the original image data to obtain non-overlapping 2D feature maps; Processing module, used to extract 2D feature maps and then Processed by depthwise separable convolution, capturing the local spatial information of the feature map, and then using batch normalization to obtain the intermediate state F i-1 ; The fusion module is used to transform the intermediate state F i-1 The channel information is mixed by convolution, and the distribution of information is preserved by activation function. The processed features F are connected by residual connection. i-1 The feature F of the original input i-2 The fused features are obtained.

11. The transformer multi-view 3D reconstruction system for substation scenes according to claim 10, characterized in that: The differentiable homography transformation includes: Apply a 2D to 3D conversion method based on homography transformation and incorporate the geometric relationship of the camera into the neural network; warp the features on other images to the 256 depth planes in the camera cone of the reference image, and there is always a corresponding depth; The homography matrix H of the i-th feature warp to the depth d plane of the camera frustum of the reference image i (d) is calculated by the following formula: Among them, K i , R i ,t i They are respectively represented as the camera's intrinsic parameter matrix, rotation matrix, and translation matrix; K l , R l ,t l It means that an intrinsic parameter matrix and rotation and translation matrix are selected as the reference image, and the depth map finally obtained is the depth map of the selected reference image.

12. The transformer multi-view 3D reconstruction system for substation scenes according to claim 11, characterized in that: In the model building module, cost volume construction includes: After the differentiable homography transformation, N feature volumes are obtained, which are merged into the cost volume. Specifically, the variance method is used to calculate the cost volume pixel by pixel. The process is shown in the following formula: Among them, V represents the feature volume generated after the differentiable homography transformation.

13. The transformer multi-view 3D reconstruction system for substation scenes according to claim 12, characterized in that: In the model building module, depth map acquisition includes: A generation module is used to generate the probability size of each pixel at each depth by passing the acquired cost volume through a four-level U-Net structure; An initial depth map acquisition module is used to obtain the probability distribution of depth and then calculate the initial depth map; The final depth map acquisition module is used to concatenate the depth map and the original image into a four-channel input, obtain the depth residual through the neural network, and then add it to the previous depth map to obtain the final depth map.

14. The transformer multi-view 3D reconstruction system for substation scenes according to any one of claims 10 to 12, characterized in that: In the image preprocessing module, the frequency domain method is used to enhance the image.

15. The transformer multi-view 3D reconstruction system for substation scenes according to any one of claims 10 to 12, characterized in that: In the image preprocessing module, the camera is calibrated to obtain the internal and external parameters of the camera, specifically: Prepare a checkerboard calibration plate of known size, and use the camera to take multiple sets of images of the calibration plate at different angles and positions to ensure that the calibration plate can be seen in the entire scene and covers the entire camera field of view as much as possible; Import the collected calibration image into the MATLAB environment and set the chessboard side length according to the actual calibration board size; Display the camera corner detection results, perform calibration after removing images with poor results, export the camera internal and external parameters and record them for future use.

16. The transformer multi-view 3D reconstruction system for substation scenes according to any one of claims 10 to 12, characterized in that: In the image preprocessing module, the loss function L is specifically expressed as: Where d represents the depth map of the ground truth, represents the initial depth map, represents the final depth map after correction, and p∈P represents the area with value in ground truth.

17. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are performed.

18. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the computer program performs the steps of the method according to any one of claims 1 to 8.

Images