Porcelain insulator surface reflection removal method and system based on deep learning

Abstract

The present application relates to the technical field of image processing, and more particularly to a porcelain insulator surface reflection removal method and system based on deep learning, comprising obtaining images and depth images of porcelain insulators; preprocessing the depth images of the porcelain insulators with reflections; constructing a DIAN subnetwork and an IDN subnetwork, the IDN subnetwork being used for insulator image restoration; the DIAN subnetwork using depth information to guide the IDN subnetwork to remove reflections; introducing gradient loss on the basis of combining structural similarity index and mean square error loss function to construct a GL-SSIM loss function. The present application solves the problem that existing networks cannot effectively preprocess the reflections on the surface of porcelain insulators.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a method and system for removing surface reflections from ceramic insulators based on deep learning. Background Technology

[0002] Deep learning is a concept based on research into artificial neural networks. The multilayer perceptron, a deep learning structure with multiple hidden layers, reflects the characteristics of deep learning technology. By interacting with low-level features and continuously generating more abstract high-level concepts, it determines the attributes, categories, or features of the data under study. This technique has been widely used in image processing, including removing reflections from ceramic insulating surfaces.

[0003] Porcelain insulators, as a common electrical insulation material, exhibit reflectivity under light due to their smooth surface. This makes the reflective areas very similar in color to the product itself, which existing DAN+IRN networks cannot accurately and effectively distinguish between. To address the issue of poor reflection removal due to low contrast between the reflective areas and the product, this invention proposes a Mem-Residual block module based on a memory network and a GL-SSIM loss function. This improves the low insulator recognition rate when reflection is present in porcelain insulator images, enhances the rendering of surface defects in reflective areas, and provides assurance for subsequent detection of surface defects in porcelain insulators. Summary of the Invention

[0004] To address the shortcomings of existing algorithms, this invention solves the problem that existing networks cannot effectively preprocess the surface reflection of porcelain insulators.

[0005] The technical solution adopted in this invention is: a method for removing surface reflections of ceramic insulators based on deep learning, comprising the following steps:

[0006] Step 1: Obtain the color and depth images of the porcelain insulator;

[0007] Furthermore, step one specifically includes:

[0008] The system captures depth and color information of the environment using a depth camera; corrects lens distortion; and synchronizes color and depth images.

[0009] Furthermore, a depth image is constructed by calculating the depth value of an object in the camera coordinate system based on similar triangle relationships and parallax, and then mapping the depth value to pixel values.

[0010] Step 2: Preprocess the depth image of the reflective porcelain insulator;

[0011] Further preprocessing includes: using a mean filter to remove noise from the depth image, preserving edge information, and performing normalization.

[0012] Furthermore, it also includes applying opening and closing operations to eliminate holes and protrusions on porcelain insulator images.

[0013] Step 3: Construct the DIAN subnetwork and the IDN subnetwork. The IDN subnetwork is used for insulator image restoration; the DIAN subnetwork uses depth information to guide the IDN subnetwork in reflection removal.

[0014] Furthermore, the DIAN sub-network includes: five residual blocks, one convolutional module, and two deconvolutional layers; wherein, the first residual block adopts three convolutional layers with kernel sizes of 5×5, 1×1, and 3×3 respectively, and the input is residually connected to the third convolutional layer.

[0015] The IDN subnetwork includes: five residual blocks, two Mem-Residual block modules, two deconvolutional layers, three add layer modules, and three convolutional layers; a first Mem-Residual block is inserted between the first and second residual blocks; a second Mem-Residual block is inserted between the first and second add layer modules.

[0016] Furthermore, the Mem-Residual block module includes a memory layer and two 3×3 convolutions. The input features are concatenated with the residuals of the second 3×3 convolution. The memory layer sums the values ​​of all spatial locations of the i-th channel in the feature map and calculates the average value to obtain the global average pooling feature, pooling_feature. Let the weight parameters W and bias b of the fully connected layer be set. The global average pooling feature is passed through the fully connected layer, and matrix multiplication and addition operations are applied to obtain the feature vector p = W·pooling_feature + b. The ReLU activation function is used to perform a non-linear mapping on p to obtain the encoded context feature vector h = ReLU(p). h is then expanded into a feature map of the same size as the input feature map through a broadcast operation.

[0017] Step 4: Based on the combination of structural similarity index and mean squared error loss function, gradient loss is introduced to construct the GL-SSIM loss function;

[0018] Furthermore, the formula for the GL-SSIM loss function is as follows:

[0019]

[0020] in, These represent half and a quarter of the original image size, respectively, β i It is a weighting parameter; I grad The MSE loss is the gradient of the output image and the background image.

[0021]

[0022] Where α represents the weight, It is the loss function of mean squared error (MSE); L SSIM (y,y′) represents the minimum prediction loss.

[0023] Furthermore, the deep learning-based surface reflection removal system for ceramic insulators includes: a memory for storing instructions executable by a processor; and a processor for executing the instructions to implement the deep learning-based surface reflection removal method for ceramic insulators.

[0024] The beneficial effects of this invention are:

[0025] 1. The residual block designed using this invention can capture a wider range of spatial information, which is more effective for removing large reflective areas in an image and can better capture the contextual information around the reflective area;

[0026] 2. The 5*5 convolutional layer used in this invention helps to extract more complex and advanced features. In the task of removing reflections, reflections often lead to the loss or distortion of image details. Larger convolutional kernels can better capture the relationship between reflections and surrounding textures, thereby extracting the original details and structure of the image more accurately.

[0027] 3. Improve the existing cascaded sub-network structure by proposing a Mem-Residual block module based on a memory network. This module associates the previous state with the current state, makes full use of contextual information, effectively restores the details in the background image in the strongly reflective area, reduces the complexity of the network model, and improves the generation quality of de-reflective images.

[0028] 4. The GL-SSIM loss function is adopted as the objective function of the deep learning network. Based on the existing loss function that combines the structural similarity index (SSIM) and the mean squared error loss function, a gradient loss function is introduced to establish a loss function based on multi-scale representation to punish the dissimilarity between multiple scales. The output is obtained from the hidden layers of different scales, which to a certain extent ensures the quality of the reconstructed de-reflected image. Attached Figure Description

[0029] Figure 1 This is a model diagram of the DIAN subnetwork and IDN subnetwork of the present invention;

[0030] Figure 2This is a schematic diagram of the placement of the binocular vision positioning device of the present invention;

[0031] Figure 3 This is a plan view of the binocular vision positioning device of the present invention;

[0032] Figure 4 (a) and (b) are structural diagrams of the Mem-Residual block and the Residual block, respectively.

[0033] Figure 5 (a) and (b) are comparisons of the anti-reflection effects of the existing DAN+IRN network and the network of the present invention, respectively. Detailed Implementation

[0034] The present invention will be further described below with reference to the accompanying drawings and embodiments. The drawings are simplified schematic diagrams, which only illustrate the basic structure of the present invention in a schematic manner, and therefore only show the components related to the present invention.

[0035] like Figure 1 As shown, the deep learning-based method for removing surface reflections from ceramic insulators includes the following steps:

[0036] Step 1: Acquire depth images of the porcelain insulator using an Intel RealSense D435 binocular camera. The Intel RealSense D435 camera first captures 3D information of the environment, such as depth and color images. The camera stores the depth and visual information as separate frames in a buffer. The acquired data is then processed through parsing, decalibration, lens distortion correction, and synchronization of color and depth images. Figure 2 As shown, O cl and O cr Let X be the optical center, and set the origin at the optical center. Define the coordinate systems of the two cameras using their respective lens major axes, minor axes, and optical axes; the major axis of the left camera is X. cl The minor axis is Y cl The optical axis is Z. cl The major axis, minor axis, and optical axis of the camera on the right are X, X, and X, respectively. cr Y cr and Z cr Point P represents the object to be measured. The image points of point P on the left and right lenses are respectively P1 and P2. l P r ;like Figure 3 The left and right cameras are positioned on the same horizontal line, Z l Z is the optical axis of the left camera. r Let be the optical axis of the right camera, b represent the center distance between the left and right cameras, and f represent the focal length; the image points of P on the left and right sides of the camera are respectively P0 and P1. l P r;P l and P r The coordinate on the x-axis is X. l X r According to the theory of similar triangles, we can obtain:

[0037]

[0038] From equation (1), we can derive:

[0039]

[0040] During the localization process, based on the relationship between similar triangles and in combination with disparity, we can conclude that:

[0041]

[0042] Among them, X l X r The coordinates of the two lens imaging points on the x-axis are represented by X, Y, and Z. X represents the horizontal position of the object in the camera coordinate system, Y represents the vertical position of the object in the camera coordinate system, and Z represents the depth of the object in the camera coordinate system. d represents the distance between the object being measured and the optical center of the camera.

[0043] In the process of solving for the depth information Z, the camera's inherent parameters B and f are required. These parameters can be obtained through camera calibration and measurement. The depth distance Z is inversely correlated with the imaging parallax D of the left and right cameras. After the depth information is calculated, the required depth image can be constructed by mapping the depth values ​​to pixel values.

[0044] Step 2: Preprocess the depth image with reflective porcelain insulators by using a mean filter to remove noise while preserving edge information. The formula for the mean filter is as follows:

[0045]

[0046] Among them, f (xy) f(x+i,y+j) represents the gray value of a pixel after filtering, m represents the height of the filter, n represents the width of the filter, a represents the radius of the filter in the x-direction, b1 represents the radius of the filter in the y-direction, i represents the offset of the filter in the x-direction, j represents the offset of the filter in the y-direction, and f(x+i,y+j) represents the gray value of a pixel in the depth image.

[0047] The depth image is transformed to a standard range so that each pixel value is on the same scale, avoiding unnecessary interference to the neural network caused by differences between different pixel values; the standardization formula is as follows:

[0048]

[0049] Where, N norm This represents the standardized pixel value, where N is the original depth value, and n max and n min These are the maximum and minimum values ​​of the depth image, respectively.

[0050] Opening and closing operations are used to eliminate holes and protrusions in small porcelain insulator images. The formula for the opening operation is as follows:

[0051] opening(N)=dilation(erosion(N)) (6)

[0052] Where N is the original depth value. After depth image preprocessing, the processed depth image is scaled to 640×640 so that it can be input into the neural network for training.

[0053] Step 3: Construct a deep learning network model;

[0054] The present invention includes an IDN image reconstruction subnetwork and a DIAN subnetwork; wherein, the IDN subnetwork is used for insulator image restoration; and the DIAN subnetwork uses depth information to guide the IDN subnetwork to remove reflections.

[0055] The DIAN subnetwork consists of five residual blocks, one convolutional module, and two deconvolutional layers.

[0056] When an input depth image of size 640×640×3 passes through the first convolutional layer of the DIAN network, 64 3×3 convolutional kernels are used, each with 3 channels, to perform convolution operations with the 3 channels of the input image. The convolutional kernels slide across the input image with a stride of 1. At each position, the convolutional kernel performs a dot product operation with the overlapping 3×3×3 pixels, and then the results are added together. The input shape is enlarged by adding 0s around the image boundaries to keep the output shape the same as the input image shape. For each convolutional kernel, the convolution operation is performed on the entire input image, resulting in a single-channel feature map. There are a total of 64 convolutional kernels, so 64 feature maps are obtained. After the convolution operation, the feature maps are processed using the ReLU (Revised Linear Unit) activation function. The ReLU function sets all negative values ​​to 0 and keeps positive values ​​unchanged, generating and outputting feature map D1 of size 640×640×64.

[0057] The feature map D1 is input into the first residual block, and the first convolution operation is performed using a 5×5 convolution kernel. The convolution operation is performed on the input feature map with a stride of 1, generating an intermediate feature map A. 11 ; For intermediate feature map A 11Apply the ReLU activation function; then perform a second convolution operation: using a 1×1 convolution kernel on the intermediate feature map A. 11 Perform convolution operation on the top to obtain intermediate feature map A. 12 For intermediate feature map A 12 The ReLU activation function is also applied; then the final convolution operation is performed: using a 3×3 convolution kernel on the intermediate feature map A. 12 The intermediate feature map A is obtained by performing convolution operation on the top. 13 Increase the number of channels from 64 to 128, and connect the input feature map D1 with the intermediate feature map A. 13 The features are summed to obtain a residual connection; the nonlinear activation function ReLU is applied to the feature map after the residual connection; an output feature map D2 of size 640×640×128 is obtained as the output of the first residual block.

[0058] The feature map D2 is input into the second residual block, and the first convolution operation is performed using a 3×3×128 convolution kernel. The convolution operation is performed on the input feature map D2 with a stride of 1, generating an intermediate feature map A. 21 ; For intermediate feature map A 21 After applying the ReLU activation function, a second convolution operation is performed using a 3×3 convolution kernel on the intermediate feature map A. 21 The convolution operation is performed with a stride of 2 to obtain the intermediate feature map A. 22 For intermediate feature map A 22 After applying a non-linear activation function, the input feature map D2 and the intermediate feature map A are compared. 22 The input and output feature maps are added together to obtain a residual connection. This step adds the input and output feature maps to help the model learn the residual between the input and output. The ReLU activation function is applied to the feature map after the residual connection to obtain an output feature map D3 of size 320×320×256 as the output of the second residual block.

[0059] Feature map D3 is input into the third residual block, where the first convolution operation is performed using a 3×3×256 convolution kernel. The convolution operation is performed on the input feature map with a stride of 1, generating the intermediate feature map A. 31 ; For intermediate feature map A 31 After applying the ReLU activation function, a second convolution operation is performed using a 3×3 convolution kernel on the intermediate feature map A. 31 The convolution operation is performed with a stride of 2 to obtain the intermediate feature map A. 32 For intermediate feature map A 32 After applying a non-linear activation function, the input feature map D3 is combined with the intermediate feature map A obtained from the last convolution operation. 32The input and output feature maps are added together to obtain a residual connection. This step adds the input and output feature maps to help the model learn the residual between the input and output. The ReLU activation function is applied to the feature map after the residual connection to obtain an output feature map D4 of size 160×160×256 as the output of the third residual block.

[0060] Feature map D4 is input into the first deconvolution module. The deconvolution layer, also known as the transposed convolution layer or upsampling layer, can enlarge the size of the input feature map. In the deconvolution operation, the feature map obtained from the convolution operation is enlarged by inter-pixel interpolation. After interpolation, the size of the output feature map becomes 320×320×256. Through this deconvolution operation, the size of the input feature map is doubled while the number of channels remains unchanged, resulting in a 320×320×256 feature map D5 as the output.

[0061] Feature map D5 is input into the fourth residual block. First, a 3×3 convolution operation is performed on the input feature map D5 with 256 kernels, a stride of 1, and padding of "same", resulting in the intermediate feature map A. 51 For intermediate feature map A 51 Apply the ReLU activation function to the intermediate feature map A. 51 Perform another 3×3 convolution operation, with the same number of kernels (256), stride of 1, and padding of "same", to obtain the intermediate feature map A. 52 For intermediate feature map A 52 The ReLU activation function is also applied to the intermediate feature map A. 52 Adding it to feature map D5 and applying the ReLU activation function as the output of the residual block, the size of the output feature map D6 remains 320×320×256.

[0062] Feature map D6 is input into the second deconvolution module, where it undergoes a convolution operation. The deconvolution layer has 3×3×128×256 parameters; where 3×3 is the size of the deconvolution kernel, 128 is the number of channels in the output feature map, and 256 is the number of channels in the input feature map. The convolution operation weights and sums each pixel in the input feature map D6 with the deconvolution kernel to obtain the value of a pixel in the output feature map. Upsampling is then performed on the input feature map, inserting a certain number of blank pixels between each pixel to increase the size of the feature map. This upsampling operation can be implemented using a deconvolution layer, which applies the parameters of the deconvolution kernel to each pixel in the input feature map during the convolution operation, generating the corresponding pixel in the output feature map. The output feature map D7 is 640×640×128 in size.

[0063] The feature map D7 is input into the 5th residual block, and the first convolution operation is performed using a 3×3×128 convolution kernel. The convolution operation is performed on the input feature map D7 with a stride of 1, generating an intermediate feature map A. 71 After applying the ReLU activation function to the intermediate feature map, a second convolution operation is performed using a 3×3 convolution kernel on the intermediate feature map A. 71 The convolution operation is performed with a stride of 1 to obtain the intermediate feature map A. 72 For intermediate feature map A 72 After applying a nonlinear activation function, and comparing it with the intermediate feature map A 71 The features are added together to obtain a residual connection; the ReLU activation function is applied to the feature map after the residual connection to obtain an output feature map D8 with a size of 640×640×128.

[0064] The IDN network consists of five residual blocks, two Mem-Residual block modules, three add layer modules, two deconvolutional layers, and three convolutional layers. The residual blocks after F1 and F4 in the existing IDN sub-network are replaced with Mem-Residual block modules. The specific steps for inputting a 640×640×3 reflective ceramic insulator image into the IDN network are as follows:

[0065] The input image is 640×640×3. In the first convolutional layer, 64 convolutional kernels are used to extract features. The kernels slide across the image with a step of 1, moving one pixel at a time. The input shape is expanded by adding zeros around the image boundaries, so that the output shape generated after convolution is the same as the input image shape. The "VALID" padding mode is selected, which means that zeros are not padded at the boundaries. After the convolution operation is performed, the ReLU activation function is used to process the feature map, increasing the non-linear expressive power of the network, better mining features, and generating a 640×640×64 feature map F0 as the output.

[0066] The feature map F0, with a size of 640×640×64, is input into the first residual block of the IDN network. The feature map is then processed through the first convolutional layer, using a 5×5 convolutional kernel to extract specific features. This kernel is convolved across the entire image using a sliding window approach, resulting in an intermediate feature map B of constant size. 11This is a 640×640×64 feature map. A ReLU activation function is applied to the intermediate feature map for non-linear feature transformation. The activation function forwards information from neurons and keeps values ​​greater than 0 unchanged, achieving a non-linear transformation. In the subsequent second convolutional operation, a 1x1 convolutional kernel is used to reduce the number of channels in the feature tensor. This reduction in computational cost improves computational efficiency and network performance. Convolution operations are performed between each channel, generating a new 640×640×64 intermediate feature map B. 12 Again, examine the intermediate feature map B. 12 The ReLU activation function is applied for nonlinear feature transformation; the third convolution operation uses a 3×3 convolution kernel to extract higher-level features. This convolution kernel is also convolved on the entire image using a sliding window to obtain an output feature map F1 with a size of 640×640×128.

[0067] Input the feature map F1 Figure 4 (a) shows the Mem-Residual block module. Feature map F1 first passes through the Memory Layer. For each channel i in feature map F1, the following operations are performed:

[0068] Add up the values ​​of all spatial positions in the i-th channel of the feature map to get the sum. a i,jk This represents the pixel value of the i-th channel at position (j,k) in the input image;

[0069] Calculate the average: We obtain a global average pooling feature of size 1×1×128: ​​pooling_feature=[avg1,avg2,...,avg 128 Let the weight parameter W of the fully connected layer be 128×128 and the bias b be 128×1. The global average pooling features are passed through the fully connected layer, and matrix multiplication and addition operations are applied to obtain a new feature vector: p = W·pooling_feature + b. The ReLU activation function is used to perform a non-linear mapping on the feature vector to obtain the encoded context feature vector: h = ReLU(p). The context feature vector h is then expanded into a feature map of the same size as the input feature map through a broadcasting operation. Let the expanded feature map be H, with a size of 640×640×128, where the feature value at each position is equal to h:H. ijk =h kWhere i represents the row index, j represents the column index, and k represents the channel index; the output feature map after the memory layer operation is the context-expanded feature map G, with a size of 640×640×128; the feature map after the memory layer processing is output to... Figure 4 In the residual block shown in (b), the feature map G undergoes a first convolution operation, using a 3×3 kernel for sliding window convolution, resulting in an output feature map of size 640×640×128, which is the transformed result of the input feature map G. In the second convolution, a 3×3 kernel is used again to convolve the output feature map of the first layer, also using a sliding window, ultimately producing an output feature map of the same size, i.e., 640×640×128. After the above operations, a skip connection is applied to the output results of the two convolutions, that is, the input feature map G is directly added to the output result of the second convolution layer, completing the entire Mem-Residual block module operation process from input to output, and the output feature map F2 is still 640×640×128.

[0070] In the feature map F2 input residual block, the first convolutional layer of the residual block consists of 256 3x3 convolutional kernels. It performs convolution on the input image using a moving window with a stride of 2, and adds the result to the bias. Because of the stride of 2, this layer reduces the original input by a factor of 2, and the output size is 320×320×256. The feature map contains 256 channels, so it is equivalent to a slight expansion. Then, a batch normalization operation is performed to normalize the output of this layer to avoid randomness and gradient vanishing problems. The normalization operation uses the ReLU activation function for non-linear transformation. The second convolutional layer adopts the same strategy. Since the input feature map size has not changed, the output feature map F3 is still 320×320×256.

[0071] The F4 feature map is obtained by adding layers to the F3 and D3 feature maps. The addition layer operation on the D3 and F3 feature maps is represented as follows:

[0072] F4(i,j,k)=F3(i,j,k)+D3(i,j,k) (7)

[0073] Where i, j, and k represent the position indices in the three dimensions, respectively.

[0074] F4, as a new feature map, is passed to the next layer for processing. Feature map F4 passes through the second Mem-Residual block module, which has the same structure as the first Mem-Residual block module, to obtain feature map F5, which has a size of 160×160×256.

[0075] The output F5 feature map and the D4 feature map of the DIAN sub-network are combined with an add layer operation to obtain feature map F6 with a size of 160×160×256.

[0076] Feature map F6 is input into the third residual block. The input feature map has a size of 160×160×256. Batch normalization is performed on the input feature map to reduce internal covariate bias and improve the stability of network training. The normalized feature map is then convolved through a 3x3 convolutional layer with 256 kernels, a stride of 1, and "same" padding (keeping the input and output feature map sizes consistent). Batch normalization is performed on the convolution result, which is then non-linearly transformed using a ReLU activation function. It is then convolved again through a 3x3 convolutional layer with 256 kernels, a stride of 1, and "same" padding. Batch normalization is performed on the result of the second convolution, which is then added to the input feature map F6 to obtain a residual connection. The result of the residual connection is then non-linearly transformed again using a ReLU activation function. The output feature map F7 is obtained, with a size of 160×160×256.

[0077] The feature map F7 is input into the deconvolution module. The weights of the deconvolution layer are a 3×3×256×128 tensor, where 3×3 is the kernel size, the stride should be set to 2, and the padding should be set to 1. The input feature map is convolved with the deconvolution kernel with the stride and padding settings. This will produce a temporary feature map of size 320×320×128. Then, the final output feature map F8 of size 320×320×128 is obtained by applying the ReLU activation function to the temporary feature map.

[0078] The feature map F8 is input into the fourth residual block. The size of the input feature map is 320×320×128. Batch normalization is performed on the input feature map to reduce internal covariate shift and improve the stability of network training. The normalized feature map is then convolved through a 3x3 convolutional layer. The number of convolutional kernels is set to 128, the stride is 1, and the padding is "same" to keep the input and output feature map sizes consistent. Batch normalization is performed on the convolution result, and the normalized result is then non-linearly transformed using a ReLU activation function. The resulting feature map is then convolved again through a 3x3 convolutional layer with 128 kernels, a stride of 1, and "same" padding. Batch normalization is performed on the result of the second convolution, and the result of the second normalization is added to the input feature map to obtain a residual connection. The result of the residual connection is then non-linearly transformed again using a ReLU activation function. The output feature map F9 is obtained, with a size of 320×320×128.

[0079] The feature map F9 is input into the deconvolution module using a 3×3×128 deconvolution kernel. This kernel will be used to perform a convolution operation on the input feature map to enlarge the image size. Here, 3×3 is the kernel size, the stride should be set to 2, and the padding should be set to 1. The input feature map is convolved with the deconvolution kernel and the stride and padding settings. This will produce a temporary feature map of size 640×640×128. Then, the ReLU activation function is applied to the temporary feature map to obtain the final output feature map F10 of size 640×640×128.

[0080] The output F10 feature map and the D8 feature map of the DIAN sub-network are combined with an add layer operation to obtain the feature map F11, which has a size of 640×640×128.

[0081] The feature map F11 is input into the fifth residual block. The size of the input feature map is 640×640×128. Batch normalization is then performed on the input feature map. Normalization is performed to reduce internal covariate bias and improve network training stability. The normalized feature map is then convolved through a 3x3 convolutional layer with 128 kernels, a stride of 1, and "same" padding to ensure consistent input and output feature map sizes. Batch normalization is applied to the convolution results, followed by a ReLU activation function for non-linear transformation. The resulting feature map is then convolved through another 3x3 convolutional layer with 128 kernels, a stride of 1, and "same" padding, resulting in an intermediate feature map of size 640×640×64. This intermediate feature map is then normalized through a batch normalization layer, and the ReLU activation function is applied to the normalized feature map to obtain feature map F12, also of size 640×640×64.

[0082] The feature map F12 is input into two convolutional modules. First, a 3×3×64 convolution kernel is used to convolve the input feature map, which produces an output feature map of the same size (640×640×64). During the convolution, padding is used to maintain the size of the output feature map. Then, a 3×3×3 convolution kernel is used to convolve the output feature map obtained in the previous step, producing a new output feature map of the same size (640×640×3). In this step, the number of channels in the convolution kernel must match the number of channels in the input feature map to ensure that the output feature map has 3 channels. After the convolution operation, an appropriate activation function (such as ReLU) is used to perform non-linear activation on the output feature map, ultimately generating and outputting a non-reflective ceramic insulator image.

[0083] Step 4: Construct the GL-SSIM loss function. This involves introducing gradient loss based on the combination of the Structural Similarity Index (SSIM) and the Mean Squared Error (MSE) loss function. The gradient loss function primarily focuses on the edges and texture details of the image to help remove reflections. The MSE loss of the gradients of the network output image and the background image is calculated using the following formula:

[0084]

[0085] in, Let represent the gradient of image T in the x-direction, and represent the rate of change of the image in the horizontal direction. This represents the gradient of T in the y-direction, and represents the rate of change of the image in the vertical direction; Represents background image The gradient in the x-direction, Represents background image The gradient in the y-direction.

[0086] The SSIM (Structural Similarity Index) function is a commonly used method for evaluating the structural similarity between two images. Its formula is as follows:

[0087]

[0088] Where y and y′ represent the two images to be compared, μ y μ y′ y and y′ represent the mean values ​​of y and y′, respectively. σ represents the variance of y and y′ respectively. yy′ This represents the covariance between y and y′; parameters C1 and C2 are constants used for smoothing coefficients to ensure that the denominator is not zero; the value of SSIM(y,y′) is in the range of [-1,1]. The larger the value in the range, the higher the similarity between the two images.

[0089] Use L SSIM To minimize the prediction loss, it is expressed as follows:

[0090] L SSIM (y,y′)=1-SSIM(y,y′) (10)

[0091] Therefore, the loss is obtained by combining the Structural Similarity Index (SSIM) and the Mean Squared Error Loss Function:

[0092]

[0093] Where α represents the weight, The loss function is the mean squared error (MSE), specifically the L2 norm. Furthermore, multi-scale representation is highly effective in extracting image details. Therefore, a loss function based on multi-scale representation is established to penalize the dissimilarity between multiple scales. Outputs are obtained from hidden layers at different scales, and the loss value between the output and its corresponding ground truth is calculated. The proposed reflection removal loss function L2... GL (y,y′), combined with formulas (11) and (8), can be expressed as follows:

[0094]

[0095] in, These represent half and a quarter of the original image size, respectively, β i (i∈1,2,3,4) are weight parameters.

[0096] Use a trained neural network model to remove reflections from the test image;

[0097] The network was trained end-to-end and optimized using the Adam optimizer with an initial learning rate of 10. -4 The learning rate decays to 10 after 10 iterations. -5 The proposed generative loss function, structural similarity index (SSIM), and gradient function are combined; α in formula (11) is set to 2, and β1, β2, β3, and β4 in formula (12) are set to 0.8, 0.3, 0.6, and 0.2 respectively to improve the quality of the de-reflected image and obtain a non-reflective porcelain insulator photo that meets the requirements for output; Figure 5 (a) and Figure 5 (b) is a comparison chart of the de-reflection effects of existing DAN+IRN networks and the method of this invention. It can be seen that... Figure 5 (b) The anti-reflective effect is obvious.

[0098] Based on the above-described preferred embodiments of the present invention, and through the foregoing description, those skilled in the art can make various changes and modifications without departing from the inventive concept. The technical scope of this invention is not limited to the contents of the specification, but must be determined according to the scope of the claims.

Claims

1. A method for removing surface reflections from porcelain insulators based on deep learning, characterized in that, Includes the following steps: Step 1: Obtain the color and depth images of the porcelain insulator; Step 2: Preprocess the depth image of the reflective porcelain insulator; Step 3: Construct the DIAN subnetwork and the IDN subnetwork. The IDN subnetwork is used for insulator image recovery. The DIAN subnetwork utilizes depth information to guide the IDN subnetwork in removing reflections; The DIAN sub-network consists of five residual blocks, one convolutional module, and two deconvolutional layers. The first residual block uses three convolutional layers with kernel sizes of 5×5, 1×1, and 3×3, respectively, and performs residual connections between the input features and the output of the third convolutional layer. The IDN subnetwork includes: five residual blocks, two Mem-Residual block modules, two deconvolutional layers, three addlayer modules, and three convolutional layers; a first Mem-Residual block is inserted between the first and second residual blocks; a second Mem-Residual block is inserted between the first and second addlayer modules. The Mem-Residual block module includes a memory layer and two 3×3 convolutions. The input features are concatenated with the output residual of the second 3×3 convolution. The memory layer stores the first feature map in the memory layer. i The values ​​of all spatial locations of each channel are summed to obtain the total, and the average value is calculated to obtain the global average pooling feature. Let the weight parameters of the fully connected layer be... and bias The global average pooling features are passed through a fully connected layer, and matrix multiplication and addition operations are applied to obtain the feature vector. Using the ReLU activation function to Perform a nonlinear mapping to obtain the encoded context feature vector. ,Will h Through a broadcast operation, it is expanded into a feature map of the same size as the input feature map; Step 4: Based on the combination of structural similarity index and mean squared error loss function, gradient loss is introduced to construct the GL-SSIM loss function; The formula for the GL-SSIM loss function is: （12） in, , These represent half and a quarter of the original image size, respectively. These are weight parameters; The MSE loss is the gradient of the output image and the background image. （11） in, Represents weight, It is the loss function of mean squared error (MSE); To minimize the prediction loss.

2. The method for removing surface reflections of porcelain insulators based on deep learning according to claim 1, characterized in that, Step one specifically includes: The system captures depth and color information of the environment using a depth camera; corrects lens distortion; and synchronizes color and depth images.

3. The method for removing surface reflections of porcelain insulators based on deep learning according to claim 2, characterized in that, Depth images are created by calculating the depth of an object in the camera coordinate system based on similar triangles and parallax, and then mapping the depth values ​​to pixel values.

4. The method for removing surface reflections of porcelain insulators based on deep learning according to claim 1, characterized in that, Preprocessing includes: The mean filter is used to remove noise from the depth image, preserve edge information, and perform normalization.

5. The method for removing surface reflections of porcelain insulators based on deep learning according to claim 4, characterized in that, Also includes: Opening and closing operations are used to eliminate holes and protrusions in images of porcelain insulators.

6. A deep learning-based system for removing surface reflections from ceramic insulators, characterized in that, include: Memory is used to store instructions that can be executed by the processor; A processor for executing instructions to implement the deep learning-based method for removing surface reflections from ceramic insulators as described in any one of claims 1-5.

Images