Infrared power distribution machine patrol three-dimensional Gaussian scene generation method and device based on image fusion, terminal equipment and storage medium

Abstract

The invention discloses an infrared power distribution machine patrol three-dimensional Gaussian scene generation method and device based on image fusion, terminal equipment and a storage medium, and belongs to the field of image processing, and the method comprises the steps: obtaining an infrared image and a visible light image of a power distribution scene; performing hyper-parameter processing on the infrared image to obtain a hyper-parameter infrared image; registering the hyper-parameter infrared image and the visible light image based on preset calibration data to obtain a hyper-parameter infrared registered image and a visible light registered image; inputting the hyper-parameter infrared registration image and the visible light registration image into an image fusion model to obtain a fused image; and based on the fused image, determining the three-dimensional Gaussian model, performing scene rendering on the three-dimensional Gaussian model, and obtaining the three-dimensional model of the power distribution scene, so that the problem of low accuracy of constructing the three-dimensional model by using the infrared image in the prior art can be solved.

Description

Technical Field

[0001] The present invention relates to the field of image processing, and in particular to a method, device, terminal equipment and storage medium for generating a three-dimensional Gaussian scene for infrared distribution motor patrol based on image fusion. Background Art

[0002] Thermal imagers are often used to collect temperature information for imaging during power distribution inspections. Infrared images have unique all-weather imaging capabilities and are not restricted by optical lighting and weather, enabling clear perception of objects in harsh environments.

[0003] However, in thermal infrared imaging, atmospheric transmission effects vary with viewing angle, causing the characteristics of the same object to differ from viewpoint to viewpoint. Synthesized images from new viewpoints may exhibit color differences or floating artifacts. Furthermore, heat conduction between objects reduces boundary information, leading to blurred or vanishing edge features in the new viewpoint images. Therefore, directly using infrared images for 3D modeling can lead to technical issues such as inaccurate 3D models. Summary of the Invention

[0004] The present invention provides a method, device, terminal equipment and storage medium for generating a three-dimensional Gaussian scene of an infrared distribution motor patrol based on image fusion, which can solve the problem of low accuracy in constructing a three-dimensional model using infrared images in the prior art.

[0005] The present invention provides a method for generating a three-dimensional Gaussian scene for infrared distribution motor patrol based on image fusion, comprising:

[0006] Acquire infrared and visible light images of power distribution scenes;

[0007] Performing hyperparameter processing on the infrared image to obtain a hyperparameter infrared image;

[0008] Based on the preset calibration data, the hyperparameter infrared image and the visible light image are registered to obtain the hyperparameter infrared registered image and the visible light registered image;

[0009] Inputting the hyperparameter infrared registration image and the visible light registration image into an image fusion model to obtain a fused image; wherein the image fusion model includes: an encoding layer, a fusion layer, and a decoding layer; the encoding layer is used to extract features of the hyperparameter infrared registration image and the visible light registration image to obtain infrared image features corresponding to the hyperparameter infrared registration image and visible light image features corresponding to the visible light registration image; the fusion layer is used to calculate visible light image feature gradients based on the visible light registration image, and calculate fused image features based on the visible light image feature gradients and infrared image features; the decoding layer is used to restore the fused image features to obtain a fused image;

[0010] Based on the fused image, a three-dimensional Gaussian model is determined, and scene rendering is performed on the three-dimensional Gaussian model to obtain a three-dimensional model of the power distribution scene.

[0011] Furthermore, the training of the image fusion model includes:

[0012] Obtain the hyperparameter infrared registration training image and the visible light registration training image after registration;

[0013] Based on the hyperparameter infrared registration training image, the visible light registration training image, the DenseFuse reconstruction model and the DenseFuse fusion model, the training operation is repeatedly performed. When the total loss function value converges, the training operation is stopped, and the weight data when the total loss function value is the smallest is selected as the model parameter of the DenseFuse model, and an image fusion model is obtained; wherein the weight data includes: encoding layer weights and decoding layer weights; in each training operation, the encoder weights and decoder weights corresponding to the DenseFuse reconstruction model and the DenseFuse fusion model are the same; the DenseFuse reconstruction model includes an encoding layer and a decoding layer, and the DenseFuse fusion model includes an encoding layer, a fusion layer and a decoding layer.

[0014] Furthermore, the training operation is specifically as follows:

[0015] Randomly extract an image from the hyperparameter infrared registration training image or visible light registration training image as a training sample and input it into the DenseFuse reconstruction model corresponding to the current weight to generate reconstructed data. Based on the reconstructed data and the training sample, calculate the pixel loss function value and the structural similarity loss function value.

[0016] A set of matching hyperparameter infrared registration training images and visible light registration training images is selected as training samples and input into the DenseFuse fusion model corresponding to the current weight to generate fused data. Based on the fused data and the hyperparameter infrared registration training images in the training samples, the infrared image structure loss value and edge loss value are calculated.

[0017] Calculate the total loss function value based on the pixel loss function value, the structural similarity loss function value, the infrared image structure loss value and the edge loss value;

[0018] Determine whether the total loss function value of the current training operation is the same as the total loss function value of the previous training operation;

[0019] If so, it is determined that the total loss function value has converged and the training operation is stopped;

[0020] If not, the current weight data is updated, and the updated weight data is used as the weight data of the DenseFuse fusion model and the DenseFuse reconstruction model in the next training operation.

[0021] Furthermore, the infrared image is generated by an infrared imager, and the visible light image is generated by a visible light camera;

[0022] The acquisition of the calibration data includes:

[0023] Obtain a calibration plate that has reached a preset temperature;

[0024] Receiving an infrared calibration image captured by the infrared imager on the calibration plate, and a visible light calibration image captured by the visible light camera on the calibration plate;

[0025] The infrared calibration image and the visible light calibration image are processed according to a preset calibration method to obtain calibration data; wherein the calibration data includes: the intrinsic parameters of the infrared imager, the intrinsic parameters of the visible light camera, the rotation matrix from the infrared imager to the visible light camera, and the translation vector from the infrared imager to the visible light camera.

[0026] Furthermore, determining the three-dimensional Gaussian model based on the fused image includes:

[0027] Extracting and matching feature points of each fused image to obtain a two-dimensional feature point correspondence relationship;

[0028] The two-dimensional feature point correspondence is converted according to the SFM algorithm to obtain an initial three-dimensional Gaussian volume;

[0029] The initial three-dimensional Gaussian body is used as the input of the first model update operation, and the model update operation is repeatedly performed. After reaching a preset number of iterations, the model update operation is stopped, and the three-dimensional Gaussian body with the smallest loss function value is obtained as a three-dimensional Gaussian model; wherein the parameters of the three-dimensional Gaussian body input in each model update operation are inconsistent.

[0030] Furthermore, the model updating operation includes:

[0031] Select a target fusion image, project the input onto a two-dimensional plane with the same viewing angle as the target fusion image, perform raster rendering on the projection result on the two-dimensional plane, and obtain a rendered scene image;

[0032] Calculate the loss function value between the rendered scene image and the target fusion image;

[0033] According to the loss function value and the back propagation operation, the input parameters are updated to obtain an updated three-dimensional Gaussian body, and the updated three-dimensional Gaussian body is used as the input of the next model update operation to execute the next model update operation.

[0034] Furthermore, the parameters of the three-dimensional Gaussian body include: position information, covariance matrix, opacity, and spherical harmonics for controlling color.

[0035] Another embodiment of the present invention further provides a device for generating a three-dimensional Gaussian scene for infrared power distribution patrol based on image fusion, comprising: a data acquisition module, a data processing module, a registration module, a fusion module, and a result generation module;

[0036] The data acquisition module is used to acquire infrared images and visible light images of the power distribution scene;

[0037] The data processing module is used to perform hyperparameter processing on the infrared image to obtain a hyperparameter infrared image;

[0038] The registration module is used to register the hyperparameter infrared image and the visible light image based on preset calibration data to obtain a hyperparameter infrared registered image and a visible light registered image;

[0039] The fusion module is configured to input the hyperparameter infrared registration image and the visible light registration image into an image fusion model to obtain a fused image; wherein the image fusion model includes: an encoding layer, a fusion layer, and a decoding layer; the encoding layer is configured to perform feature extraction on the hyperparameter infrared registration image and the visible light registration image to obtain infrared image features corresponding to the hyperparameter infrared registration image and visible light image features corresponding to the visible light registration image; the fusion layer is configured to calculate visible light image feature gradients based on the visible light registration image, and calculate fused image features based on the visible light image feature gradients and infrared image features; the decoding layer is configured to restore the fused image features to obtain a fused image;

[0040] The result generation module is used to determine a three-dimensional Gaussian model based on the fused image, perform scene rendering on the three-dimensional Gaussian model, and obtain a three-dimensional model of the power distribution scene.

[0041] Another embodiment of the present invention also provides a terminal device, including: a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor. When the processor executes the computer program, it implements the steps of the method for generating a three-dimensional Gaussian scene for infrared distribution motor patrol based on image fusion as provided in the present invention.

[0042] Another embodiment of the present invention also provides a computer-readable storage medium item, including: a stored computer program, which, when the computer program is running, controls the device where the computer-readable storage medium is located to execute the steps of the method for generating a three-dimensional Gaussian scene for infrared distribution motor patrol based on image fusion as provided in the present invention.

[0043] The following beneficial effects are achieved by implementing the present invention:

[0044] The present invention acquires infrared and visible light images of a power distribution scene, performs hyperparameter processing on the infrared image, and obtains a hyperparameter infrared image, thereby enhancing the features of the infrared image. The hyperparameter infrared image is then registered with the visible light image to obtain a correlated hyperparameter infrared registered image and a visible light registered image. The hyperparameter infrared and visible light registered images are then input into an image fusion model. Feature extraction is performed on the hyperparameter infrared and visible light registered images via the encoding layer, and the extracted features are input into the fusion layer for fusion. Finally, the obtained fused image features are input into the decoding layer for restoration to obtain a fused image. This fusion combines the features of the visible light image with those of the infrared image, imparting more features to the infrared image. A three-dimensional Gaussian model is determined from the fused image with more features, and scene rendering is then performed based on the three-dimensional Gaussian model to obtain a three-dimensional model of the power distribution scene. By constructing a three-dimensional model based on the fused image that possesses both infrared and visible light image features, the accuracy of the three-dimensional model is significantly improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the technical solution of the present application, the following is a brief introduction to the drawings required for use in the implementation. Obviously, the drawings described below are only some implementation methods of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0046] Figure 1 This is a flow chart of a method for generating a three-dimensional Gaussian scene for infrared distribution motor patrol based on image fusion provided by one embodiment of the present invention;

[0047] Figure 2 1 is a schematic structural diagram of a three-dimensional Gaussian scene generation device for infrared distribution motor patrol based on image fusion provided by an embodiment of the present invention;

[0048] Figure 3 It is a flow chart of a method for generating a three-dimensional Gaussian scene for infrared distribution motor patrol based on image fusion provided by another embodiment of the present invention. DETAILED DESCRIPTION

[0049] To make the objectives, technical solutions, and advantages of this application more clear, the technical solutions in this application will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of this application. Obviously, the described embodiments are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making any creative efforts are within the scope of protection of this application.

[0050] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this application belongs; the terms used herein are only for the purpose of describing specific embodiments and are not intended to limit this application; the terms "including" and "having" and any variations thereof in the specification and claims of this application and the above-mentioned figure descriptions are intended to cover non-exclusive inclusions.

[0051] In the description of the embodiments of this application, the technical terms "first" and "second" are used only to distinguish different objects and should not be understood to indicate or imply relative importance or implicitly specify the quantity, specific order, or primary and secondary relationship of the indicated technical features. In the description of the embodiments of this application, the meaning of "plurality" is more than two, unless otherwise clearly and specifically defined.

[0052] References herein to "embodiments" mean that a particular feature, structure, or characteristic described in connection with the embodiments may be included in at least one embodiment of the present application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor does it constitute an independent or alternative embodiment that is mutually exclusive of other embodiments. It is understood, both explicitly and implicitly, by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0053] In the description of the embodiments of this application, the term "and / or" is simply a description of the association relationship between associated objects, indicating that three relationships can exist. For example, A and / or B can represent the following three situations: A exists alone, A and B exist simultaneously, and B exists alone. In addition, the character " / " in this document generally indicates that the associated objects are in an "or" relationship.

[0054] In the description of the embodiments of the present application, the term "multiple" refers to more than two (including two). Similarly, "multiple groups" refers to more than two groups (including two groups), and "multiple pieces" refers to more than two pieces (including two pieces).

[0055] In the description of the embodiments of the present application, unless otherwise expressly specified or limited, technical terms such as "installed," "connected," "connected," and "fixed" should be understood in a broad sense. For example, they can refer to fixed connections, detachable connections, or integration; mechanical connections or electrical connections; direct connections or indirect connections through an intermediate medium; internal connections between two components or interactions between two components. Those skilled in the art can understand the specific meanings of the above terms in the embodiments of the present application based on specific circumstances.

[0056] See also Figure 1To solve the problem of low accuracy in constructing three-dimensional models using infrared images in the prior art, an embodiment of the present invention provides a method for generating a three-dimensional Gaussian scene for an infrared distribution motor patrol based on image fusion, comprising:

[0057] 101. Acquire infrared images and visible light images of the power distribution scene.

[0058] In a specific embodiment, a drone patrol is used to capture multiple image pairs consisting of infrared images and visible light images of the same power distribution scene.

[0059] 102. Perform hyperparameter processing on the infrared image to obtain a hyperparameter infrared image.

[0060] In a specific embodiment, the infrared image I th Input ChaSNet super-resolution module to infrared image I th Processing to obtain hyperparameter infrared image

[0061] In a specific embodiment, the ChaSNet network is divided into three modules: SFE, DFE, and IRec. A low-resolution thermal infrared image with a resolution of 512*640 is first input into the SFE module. The SFE module consists of two convolutional layers, one with a 3*3 kernel and one with a 5*5 kernel, which respectively extract local and global features from the shallow layer of the image. The DFE module consists of multiple stacked CB modules and an LA module. The CB module extracts multi-scale features. The feature maps of each CB module are input into the LA module for merging. The result of the last CB module is fused with the output of the LA module and the output of the DFE module is obtained through a convolutional layer with a 1*1 kernel. IRec includes an upsampling module and two convolutional layers with kernels of 1*1 and 3*3, respectively, outputting a super-resolution residual grayscale image of 2048*2560. Finally, the input 512*640 low-resolution thermal infrared image is upsampled by bicubic interpolation and added to the 2048*2560 super-resolution residual grayscale image to obtain a 2048*2560 super-resolution thermal infrared image.

[0062] 103. Based on the preset calibration data, the hyperparameter infrared image and the visible light image are registered to obtain a hyperparameter infrared registered image and a visible light registered image.

[0063] Furthermore, the infrared image is generated by an infrared imager, and the visible light image is generated by a visible light camera;

[0064] The acquisition of the calibration data includes:

[0065] Obtain a calibration plate that has reached a preset temperature;

[0066] Receiving an infrared calibration image captured by the infrared imager on the calibration plate, and a visible light calibration image captured by the visible light camera on the calibration plate;

[0067] The infrared calibration image and the visible light calibration image are processed according to a preset calibration method to obtain calibration data; wherein the calibration data includes: the intrinsic parameters of the infrared imager, the intrinsic parameters of the visible light camera, the rotation matrix from the infrared imager to the visible light camera, and the translation vector from the infrared imager to the visible light camera.

[0068] In a specific embodiment, a black and white checkerboard calibration plate is heated using a ceramic heating pad or an infrared heater. After the calibration plate is heated evenly and reaches a preset temperature, an infrared imager and a visible light camera are simultaneously captured using a drone.

[0069] In a specific embodiment, the calibration method can be Zhang Zhengyou calibration method, and the intrinsic parameter K of the infrared camera is obtained according to Zhang Zhengyou calibration method. t , the internal parameter K of the visible light camera r And the rotation matrix R and translation vector T from the infrared camera to the visible light camera.

[0070] 104. Input the hyperparameter infrared registration image and the visible light registration image into an image fusion model to obtain a fused image; wherein the image fusion model includes: a coding layer, a fusion layer and a decoding layer; the coding layer is used to extract features of the hyperparameter infrared registration image and the visible light registration image to obtain infrared image features corresponding to the hyperparameter infrared registration image and visible light image features corresponding to the visible light registration image; the fusion layer is used to calculate visible light image feature gradients based on the visible light registration image, and calculate fused image features based on the visible light image feature gradients and infrared image features; the decoding layer is used to restore the fused image features to obtain a fused image.

[0071] In a specific embodiment, the calculation of the fused image features is specifically as follows:

[0072]

[0073] Where, F fusion To fuse image features, F th is the infrared image feature, is the characteristic gradient of the visible light image, α is the weight parameter. In order to make the fused image mainly based on infrared information and at the same time incorporate the edge texture information of the visible light image, α can be set to 0.2-0.5.

[0074] In a specific embodiment, the visible light image feature gradient is obtained by chain rule and automatic differentiation.

[0075] In one specific implementation, the hyperparameter infrared image Visible light image I rgb Perform weak data enhancement processing such as random cropping and random flipping. Visible light image I rgb ′, the intrinsic parameter K of the infrared camera t , the internal parameter K of the visible light camera r The rotation matrix R and translation vector T from the infrared camera to the visible light camera are input into the improved DenseFuse image fusion model (i.e., image fusion model) for image fusion.

[0076] In a specific embodiment, the improved DenseFuse image fusion model consists of three parts: an encoder, a fusion layer, and a decoder. The registered image pairs are fed into the encoder, which includes two modules, C1 and DenseBlock, to extract coarse and deep features, respectively. The encoder output is fed into the fusion layer, where it is fused using an improved fusion strategy. Finally, the decoder, consisting of four convolutional layers with 3x3 kernels, gradually restores image details.

[0077] Furthermore, the training of the image fusion model includes:

[0078] Obtain the hyperparameter infrared registration training image and the visible light registration training image after registration;

[0079] Based on the hyperparameter infrared registration training image, the visible light registration training image, the DenseFuse reconstruction model and the DenseFuse fusion model, the training operation is repeatedly performed. When the total loss function value converges, the training operation is stopped, and the weight data when the total loss function value is the smallest is selected as the model parameter of the DenseFuse model, and an image fusion model is obtained; wherein the weight data includes: encoding layer weights and decoding layer weights; in each training operation, the encoder weights and decoder weights corresponding to the DenseFuse reconstruction model and the DenseFuse fusion model are the same; the DenseFuse reconstruction model includes an encoding layer and a decoding layer, and the DenseFuse fusion model includes an encoding layer, a fusion layer and a decoding layer.

[0080] Furthermore, the training operation is specifically as follows:

[0081] Randomly extract an image from the hyperparameter infrared registration training image or visible light registration training image as a training sample and input it into the DenseFuse reconstruction model corresponding to the current weight to generate reconstructed data. Based on the reconstructed data and the training sample, calculate the pixel loss function value and the structural similarity loss function value.

[0082] A set of matching hyperparameter infrared registration training images and visible light registration training images is selected as training samples and input into the DenseFuse fusion model corresponding to the current weight to generate fused data. Based on the fused data and the hyperparameter infrared registration training images in the training samples, the infrared image structure loss value and edge loss value are calculated.

[0083] Calculate the total loss function value based on the pixel loss function value, the structural similarity loss function value, the infrared image structure loss value and the edge loss value;

[0084] Determine whether the total loss function value of the current training operation is the same as the total loss function value of the previous training operation;

[0085] If so, it is determined that the total loss function value has converged and the training operation is stopped;

[0086] If not, the current weight data is updated, and the updated weight data is used as the weight data of the DenseFuse fusion model and the DenseFuse reconstruction model in the next training operation.

[0087] During the training phase, two DenseFuse models are trained simultaneously. One model removes the fusion layer (i.e., the DenseFuse reconstruction model), retains the encoding layer and decoding layer for self-training, and inputs a visible light or infrared light image. The model reconstructs the original image and uses the input image as supervision to calculate the pixel loss function value and the structural similarity loss function value:

[0088] L p =||OI|| 2 ;

[0089] L ssim =1-SSIM(O,I);

[0090] Where, L p is the pixel loss function value, O and I represent the output image and input image respectively, L ssim is the structural similarity loss function value, and SSIM(·) represents the structural similarity operation. The training goal is to enable the autoencoder to accurately reconstruct the original image when a single image is input, thereby ensuring that the extracted features have good representational capabilities.

[0091] The other model is the complete DenseFuse model (i.e., DenseFuse fusion model), which inputs the registered infrared and visible light image pairs, outputs the fused image, and calculates the infrared image structure loss and edge loss values ​​respectively:

[0092] L th =||F fusion -F th ||1;

[0093] L edge =||Sobel(F fusion )-Sobel(F rgb )||1;

[0094] Where, L th is the infrared image structure loss value, L edge is the edge loss value, F rgb is the visible light image feature, F fusion To fuse image features, F th is the infrared image feature, and Sobel(·) is the Sobel edge detection algorithm. This fused image is primarily infrared information, while also incorporating edge texture information from the visible light image. Both models are trained simultaneously, with encoder and decoder weights shared.

[0095] 105. Determine a three-dimensional Gaussian model based on the fused image, perform scene rendering on the three-dimensional Gaussian model, and obtain a three-dimensional model of the power distribution scene.

[0096] Furthermore, determining the three-dimensional Gaussian model based on the fused image includes:

[0097] Extracting and matching feature points of each fused image to obtain a two-dimensional feature point correspondence relationship;

[0098] The two-dimensional feature point correspondence is converted according to the SFM algorithm to obtain an initial three-dimensional Gaussian volume;

[0099] The initial three-dimensional Gaussian body is used as the input of the first model update operation, and the model update operation is repeatedly performed. After reaching a preset number of iterations, the model update operation is stopped, and the three-dimensional Gaussian body with the smallest loss function value is obtained as a three-dimensional Gaussian model; wherein the parameters of the three-dimensional Gaussian body input in each model update operation are inconsistent.

[0100] In one specific embodiment, the colmap 3D reconstruction algorithm is used to extract and match feature points from the fused image, obtaining 2D feature point correspondences for the fused image of the same power distribution scene. Based on these 2D feature point correspondences, the SFM algorithm is used to generate a 3D sparse point cloud of the power distribution scene. The 3D coordinates of these point clouds serve as the initial 3D Gaussian volume of the point cloud.

[0101] Furthermore, the model updating operation includes:

[0102] Select a target fusion image, project the input onto a two-dimensional plane with the same viewing angle as the target fusion image, perform raster rendering on the projection result on the two-dimensional plane, and obtain a rendered scene image;

[0103] Calculate the loss function value between the rendered scene image and the target fusion image;

[0104] According to the loss function value and the back propagation operation, the input parameters are updated to obtain an updated three-dimensional Gaussian body, and the updated three-dimensional Gaussian body is used as the input of the next model update operation to execute the next model update operation.

[0105] In a specific embodiment, an initialized 3D Gaussian volume is generated from a point cloud using SfM. Each 3D Gaussian volume is initialized with parameters such as position information x, y, z, covariance matrix Σ, opacity α, and spherical harmonics SH, which controls color. These parameters are optimized through a model update operation to minimize the loss function between the image rendered by the 3D Gaussian volume and images from the same viewpoint. The model update operation requires a fused image dataset of a scene. For example, 20 images are selected every five images as a validation set, and images from four viewpoints are determined. The 3D Gaussian model predicts and renders images corresponding to the four viewpoints based on the 16 training images. The loss function is then calculated against the images in the validation set, and the aforementioned parameters are optimized based on the loss function. Minimizing the loss function is used as the iteration target, and the number of iterations is set (e.g., 10,000, which can be customized by the user based on the test scenario). Finally, the 3D Gaussian volume with the lowest loss function is selected as the 3D Gaussian model. The 3D Gaussian model needs to be retrained for each scene to obtain the 3D model that best represents the scene.

[0106] In a specific embodiment, the loss function is:

[0107]

[0108]

[0109]

[0110]

[0111]

[0112]

[0113] Among them, L total is the loss function value, is the mean absolute error, is the structural similarity loss, is the modal consistency loss, is the smoothing loss, λ1, λ2, λ3, and λ4 are weight coefficients, and N is the total number of pixels. and are the j-th pixel values ​​of the input image and the generated image, μ i and μ o are the means of the input image and the generated image respectively, (σ i ) 2 and (σ o ) 2 are the variances of the input image and the generated image, σ io is the covariance of the input image and the generated image, C1 and C2 are constants, F(λ) is the feature extraction function, is the normalized pixel value at the jth pixel, β is the edge sharpness parameter, I max and I min are the maximum and minimum pixel values ​​in the output image.

[0114] Furthermore, the parameters of the three-dimensional Gaussian body include: position information, covariance matrix, opacity, and spherical harmonics for controlling color.

[0115] For a better explanation, see Figure 3 , infrared images and visible light images of the power distribution scene are obtained by using a drone equipped with an infrared imager and a visible light camera, and the infrared images are preprocessed; the infrared imager and visible light camera are calibrated; the preprocessed infrared image, visible light image and calibration results are input into the improved DenseFuse image fusion algorithm module to obtain a fused image; the fused image is input into a three-dimensional Gaussian model, and a new perspective image of the scene is synthesized through three-dimensional Gaussian sputtering rendering, and the model is iteratively trained.

[0116] It's important to note that intelligent drone inspections of power distribution equipment, using drones equipped with infrared thermal imagers and visible light cameras, enable regular, large-scale inspections of distribution facilities. This is particularly useful in areas with complex terrain and limited access, saving significant manpower and time costs and significantly reducing safety risks for inspectors. Real-world images of power equipment captured during inspections can be visualized through 3D reconstruction. The resulting 3D model serves as a digital archive of power facilities, providing essential data support for equipment status monitoring, fault diagnosis, and maintenance decision-making.

[0117] Real-life 3D reconstruction of power equipment uses LiDAR (Lidar) scanning to acquire a large amount of point cloud data from the target object's surface. This point cloud data is then processed and modeled to generate a 3D point cloud model of the target object. LiDAR provides highly accurate spatial position information, resulting in a highly precise 3D model that accurately reflects the shape, location, and other details of the power facility. Vision-based 3D reconstruction utilizes multi-view image data and computer vision algorithms to calculate the object's 3D structure, generating a more realistic and intuitive 3D model that includes rich texture information. Furthermore, multimodal fusion 3D reconstruction, which fuses data from multiple sensors such as LiDAR, optical cameras, and infrared cameras, and leverages the strengths of each data type, can effectively improve the accuracy and reliability of 3D reconstruction.

[0118] like Figure 2 As shown, based on the above method embodiment, a corresponding device embodiment is provided;

[0119] An embodiment of the present invention provides a three-dimensional Gaussian scene generation device for infrared power distribution patrol based on image fusion, comprising: a data acquisition module 201, a data processing module 202, a registration module 203, a fusion module 204 and a result generation module 205;

[0120] The data acquisition module is used to acquire infrared images and visible light images of the power distribution scene;

[0121] The data processing module is used to perform hyperparameter processing on the infrared image to obtain a hyperparameter infrared image;

[0122] The registration module is used to register the hyperparameter infrared image and the visible light image based on preset calibration data to obtain a hyperparameter infrared registered image and a visible light registered image;

[0123] The fusion module is configured to input the hyperparameter infrared registration image and the visible light registration image into an image fusion model to obtain a fused image; wherein the image fusion model includes: an encoding layer, a fusion layer, and a decoding layer; the encoding layer is configured to perform feature extraction on the hyperparameter infrared registration image and the visible light registration image to obtain infrared image features corresponding to the hyperparameter infrared registration image and visible light image features corresponding to the visible light registration image; the fusion layer is configured to calculate visible light image feature gradients based on the visible light registration image, and calculate fused image features based on the visible light image feature gradients and infrared image features; the decoding layer is configured to restore the fused image features to obtain a fused image;

[0124] The result generation module is used to determine a three-dimensional Gaussian model based on the fused image, perform scene rendering on the three-dimensional Gaussian model, and obtain a three-dimensional model of the power distribution scene.

[0125] It can be understood that the above-mentioned device embodiment corresponds to the method embodiment of the present invention, which can implement any of the above-mentioned method embodiments of the present invention to provide a three-dimensional Gaussian scene generation method for infrared distribution motor patrol based on image fusion.

[0126] This embodiment has the following effects:

[0127] By acquiring infrared and visible light images of a power distribution scene and performing hyperparameter processing on the infrared image to obtain a hyperparameter infrared image, the features of the infrared image can be enhanced. The hyperparameter infrared image is then registered with the visible light image to obtain a correlated hyperparameter infrared registered image and a visible light registered image. These hyperparameter infrared and visible light registered images are then fed into an image fusion model. Feature extraction is performed on these images at the encoding layer, and the extracted features are fed into the fusion layer for fusion. Finally, the features of the fused image are fed into the decoding layer for restoration to obtain a fused image. This fusion of visible light and infrared image features enhances the infrared image's features. A three-dimensional Gaussian model is determined from the fused image, which is then used to render the scene, resulting in a three-dimensional model of the power distribution scene. By constructing a three-dimensional model based on this fused image, which combines features from both the infrared and visible light images, the accuracy of the model is significantly improved.

[0128] It should be noted that the device embodiments described above are merely illustrative, and some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment. Furthermore, in the drawings of the device embodiments provided by the present invention, the connection relationship between modules indicates that they have a communication connection, which may be implemented as one or more communication buses or signal lines. Those skilled in the art can understand and implement the present invention without inventive effort.

[0129] Based on the above-mentioned embodiment of the method for generating a three-dimensional Gaussian scene for an infrared distribution network based on image fusion, another embodiment of the present invention provides a terminal device, which includes a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor. When the processor executes the computer program, the method for generating a three-dimensional Gaussian scene for an infrared distribution network based on image fusion according to any embodiment of the present invention is implemented.

[0130] For example, in this embodiment, the computer program may be divided into one or more modules, which are stored in the memory and executed by the processor to implement the present invention. The one or more module elements may be a series of computer program instruction segments capable of performing specific functions, and the instruction segments are used to describe the execution process of the computer program in the terminal device.

[0131] The terminal device may be a computing device such as a desktop computer, a notebook computer, a PDA, a cloud server, etc. The terminal device may include, but is not limited to, a processor and a memory.

[0132] The processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc. The processor is the control center of the terminal device, connecting various parts of the entire terminal device using various interfaces and lines.

[0133] On the basis of the above-mentioned method embodiments, another embodiment of the present invention provides a computer-readable storage medium, comprising a stored computer program, wherein when the computer program is running, the device where the computer-readable storage medium is located is controlled to execute the three-dimensional Gaussian scene generation method for infrared distribution motor patrol based on image fusion as described in any one of the above-mentioned method embodiments of the present invention.

[0134] Wherein, the module / unit integrated in the device / terminal equipment, if implemented in the form of a software functional unit and sold or used as an independent product, can be stored in a computer-readable storage medium. Based on this understanding, the present invention implements all or part of the process in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and when the computer program is executed by the processor, it can implement the steps of the above-mentioned various method embodiments. Wherein, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device that can carry the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc.

[0135] The above is a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. A method for generating three-dimensional Gaussian scenes of infrared distribution motor patrol based on image fusion, characterized in that: include: Acquire infrared and visible light images of power distribution scenes; Performing hyperparameter processing on the infrared image to obtain a hyperparameter infrared image; Based on the preset calibration data, the hyperparameter infrared image and the visible light image are registered to obtain the hyperparameter infrared registered image and the visible light registered image; Inputting the hyperparameter infrared registration image and the visible light registration image into an image fusion model to obtain a fused image; wherein the image fusion model includes: an encoding layer, a fusion layer, and a decoding layer; the encoding layer is used to extract features of the hyperparameter infrared registration image and the visible light registration image to obtain infrared image features corresponding to the hyperparameter infrared registration image and visible light image features corresponding to the visible light registration image; the fusion layer is used to calculate visible light image feature gradients based on the visible light registration image, and calculate fused image features based on the visible light image feature gradients and infrared image features; the decoding layer is used to restore the fused image features to obtain a fused image; Based on the fused image, a three-dimensional Gaussian model is determined, and scene rendering is performed on the three-dimensional Gaussian model to obtain a three-dimensional model of the power distribution scene.

2. The method for generating a three-dimensional Gaussian scene based on image fusion for infrared distribution network patrol according to claim 1, characterized in that: The training of the image fusion model includes: Obtain the hyperparameter infrared registration training image and the visible light registration training image after registration; Based on the hyperparameter infrared registration training image, the visible light registration training image, the DenseFuse reconstruction model and the DenseFuse fusion model, the training operation is repeatedly performed. When the total loss function value converges, the training operation is stopped, and the weight data when the total loss function value is the smallest is selected as the model parameter of the DenseFuse model, and an image fusion model is obtained; wherein the weight data includes: encoding layer weights and decoding layer weights; in each training operation, the encoder weights and decoder weights corresponding to the DenseFuse reconstruction model and the DenseFuse fusion model are the same; the DenseFuse reconstruction model includes an encoding layer and a decoding layer, and the DenseFuse fusion model includes an encoding layer, a fusion layer and a decoding layer.

3. The method for generating a three-dimensional Gaussian scene based on image fusion for infrared distribution network patrol according to claim 2, characterized in that: The training operation is specifically as follows: Randomly extract an image from the hyperparameter infrared registration training image or visible light registration training image as a training sample and input it into the DenseFuse reconstruction model corresponding to the current weight to generate reconstructed data. Based on the reconstructed data and the training sample, calculate the pixel loss function value and the structural similarity loss function value. A set of matching hyperparameter infrared registration training images and visible light registration training images is selected as training samples and input into the DenseFuse fusion model corresponding to the current weight to generate fused data. Based on the fused data and the hyperparameter infrared registration training images in the training samples, the infrared image structure loss value and edge loss value are calculated. Calculate the total loss function value based on the pixel loss function value, the structural similarity loss function value, the infrared image structure loss value and the edge loss value; Determine whether the total loss function value of the current training operation is the same as the total loss function value of the previous training operation; If so, it is determined that the total loss function value has converged and the training operation is stopped; If not, the current weight data is updated, and the updated weight data is used as the weight data of the DenseFuse fusion model and the DenseFuse reconstruction model in the next training operation.

4. The method for generating a three-dimensional Gaussian scene based on image fusion for infrared distribution network patrol according to claim 3, characterized in that: The infrared image is generated by an infrared imager, and the visible light image is generated by a visible light camera; The acquisition of the calibration data includes: Obtain a calibration plate that has reached a preset temperature; Receiving an infrared calibration image captured by the infrared imager on the calibration plate, and a visible light calibration image captured by the visible light camera on the calibration plate; The infrared calibration image and the visible light calibration image are processed according to a preset calibration method to obtain calibration data; wherein the calibration data includes: the intrinsic parameters of the infrared imager, the intrinsic parameters of the visible light camera, the rotation matrix from the infrared imager to the visible light camera, and the translation vector from the infrared imager to the visible light camera.

5. The method for generating a three-dimensional Gaussian scene based on image fusion for infrared distribution network patrol according to claim 4, characterized in that: The determining of the three-dimensional Gaussian model based on the fused image includes: Extracting and matching feature points of each fused image to obtain a two-dimensional feature point correspondence relationship; The two-dimensional feature point correspondence is converted according to the SFM algorithm to obtain an initial three-dimensional Gaussian volume; The initial three-dimensional Gaussian body is used as the input of the first model update operation, and the model update operation is repeatedly performed. After reaching a preset number of iterations, the model update operation is stopped, and the three-dimensional Gaussian body with the smallest loss function value is obtained as a three-dimensional Gaussian model; wherein the parameters of the three-dimensional Gaussian body input in each model update operation are inconsistent.

6. The method for generating a three-dimensional Gaussian scene based on image fusion for infrared distribution network patrol according to claim 5, characterized in that: The model update operation includes: Select a target fusion image, project the input onto a two-dimensional plane with the same viewing angle as the target fusion image, perform raster rendering on the projection result on the two-dimensional plane, and obtain a rendered scene image; Calculate the loss function value between the rendered scene image and the target fusion image; According to the loss function value and the back propagation operation, the input parameters are updated to obtain an updated three-dimensional Gaussian body, and the updated three-dimensional Gaussian body is used as the input of the next model update operation to execute the next model update operation.

7. The method for generating a three-dimensional Gaussian scene based on image fusion for infrared distribution network patrol according to claim 6, characterized in that: The parameters of the three-dimensional Gaussian body include: position information, covariance matrix, opacity, and spherical harmonic functions for controlling color.

8. A three-dimensional Gaussian scene generation device for infrared distribution motor patrol based on image fusion, characterized in that: include: Data acquisition module, data processing module, registration module, fusion module and result generation module; The data acquisition module is used to acquire infrared images and visible light images of the power distribution scene; The data processing module is used to perform hyperparameter processing on the infrared image to obtain a hyperparameter infrared image; The registration module is used to register the hyperparameter infrared image and the visible light image based on preset calibration data to obtain a hyperparameter infrared registered image and a visible light registered image; The fusion module is configured to input the hyperparameter infrared registration image and the visible light registration image into an image fusion model to obtain a fused image; wherein the image fusion model includes: an encoding layer, a fusion layer, and a decoding layer; the encoding layer is configured to perform feature extraction on the hyperparameter infrared registration image and the visible light registration image to obtain infrared image features corresponding to the hyperparameter infrared registration image and visible light image features corresponding to the visible light registration image; the fusion layer is configured to calculate visible light image feature gradients based on the visible light registration image, and calculate fused image features based on the visible light image feature gradients and infrared image features; the decoding layer is configured to restore the fused image features to obtain a fused image; The result generation module is used to determine a three-dimensional Gaussian model based on the fused image, perform scene rendering on the three-dimensional Gaussian model, and obtain a three-dimensional model of the power distribution scene.

9. A terminal device, characterized in that: The method comprises a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor. When the processor executes the computer program, the method for generating a three-dimensional Gaussian scene for infrared distribution motor patrol based on image fusion according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that include: A stored computer program, wherein when the computer program is running, the device where the computer-readable storage medium is located is controlled to execute the three-dimensional Gaussian scene generation method for infrared distribution motor patrol based on image fusion according to any one of claims 1 to 7.

Images