Instrument image super-resolution reconstruction method and device based on NLSA and SWL

Abstract

The invention relates to the technical field of image processing, and relates to an instrument image super-resolution reconstruction method and device based on NLSA and SWL. The method comprises the following steps: acquiring a low-resolution instrument image XLR; performing image enhancement preprocessing on the XLR to obtain an enhanced image XEm; and processing the XEm through a trained super-resolution reconstruction model to obtain a reconstructed high-resolution instrument image XHR. According to the invention, a self-designed super-resolution reconstruction model is adopted, on one hand, multiple attention mechanisms are fused to realize multi-scale and all-around feature extraction, and on the other hand, a loss function including static wavelet loss and L1 loss of an RGB domain is adopted to ensure a model training effect; therefore, real-time reconstruction is achieved under the condition that the model reconstruction effect is guaranteed, and the problem that an existing super-resolution reconstruction method based on the CNN or Transform model is not suitable for processing cabinet instrument images is solved.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and more specifically to: 1. A super-resolution reconstruction method for instrument images based on NLSA and SWL; 2. A super-resolution reconstruction device for instrument images based on NLSA and SWL. Background Art

[0002] Super-resolution reconstruction (i.e., super-resolution reconstruction) can reconstruct high-resolution images from low-resolution images and significantly improve the spatial resolution of the image. Therefore, it has received widespread attention in the fields of image processing and computer vision.

[0003] In the relay room of the thermal power plant where the inventor works, high-definition photography of instruments and meters is usually required during the autonomous robot inspection process for subsequent analysis and maintenance. However, traditional high-resolution cameras are expensive and not flexible enough when used in a small space. Although low-resolution cameras have the advantages of being lightweight and low-cost, they have obvious deficiencies in capturing high-quality images, resulting in the inability to accurately present or identify key details of the image, such as readings, scale lines, and edges. Therefore, how to improve the quality of images taken by low-resolution cameras, especially in special environments such as relay rooms, has become a problem that needs to be solved urgently. Therefore, the inventor thought of combining super-resolution reconstruction algorithms to improve the clarity and details of images taken by low-resolution cameras, improve image quality and detail performance, and achieve efficient image acquisition in resource-constrained environments, which is of great significance to promoting the advancement of autonomous inspection image processing technology!

[0004] After conducting research, the inventor found that:

[0005] 1. Although the current super-resolution reconstruction algorithms based on deep learning (such as convolutional neural networks (CNNs)) perform well in the reconstruction of natural images, CNNs have limitations in processing long-distance dependencies and high-frequency details in images. Especially in cabinet instrument images, CNNs have difficulty in effectively restoring complex edges, scale lines, and digital details.

[0006] 2. Currently, the super-resolution reconstruction algorithm based on the Transformer model has been widely used in computer vision due to its powerful long-distance dependency modeling ability. However, the Transformer model may have certain limitations in restoring high-frequency details in cabinet instrument images.

[0007] Therefore, it is necessary to design an effective super-resolution reconstruction scheme based on the requirements of autonomous inspection of the relay room of a thermal power plant. Combined with efficient feature extraction and noise reduction technology, it can speed up the reconstruction speed to achieve real-time reconstruction while ensuring the quality of the reconstructed image. Summary of the invention

[0008] Based on this, it is necessary to address the problem that the existing super-resolution reconstruction methods based on CNN or Transformer models are not suitable for processing cabinet instrument images, and provide an instrument image super-resolution reconstruction method and device based on NLSA and SWL.

[0009] The present invention is implemented by the following technical solutions:

[0010] In a first aspect, the present invention discloses a super-resolution reconstruction method for an instrument image based on NLSA and SWL, comprising:

[0011] Step 1: Get low-resolution instrument image X LR ;

[0012] Step 2: X LR Perform image enhancement preprocessing to obtain the enhanced image X Em ;

[0013] Step 3: X Em The trained super-resolution reconstruction model is used to process the reconstructed high-resolution instrument image X HR .

[0014] Among them, the super-resolution reconstruction model includes: 3 convolution parts Conv1~Conv3, 4 non-local sparse attention parts NLSA1~NLSA4, 1 residual mixed attention group part RHAGs, and 1 pixel rearrangement part PS.

[0015] Conv1 is used to Em Perform preliminary feature extraction to obtain the intermediate feature map z1;

[0016] NLSA1 is used to perform the first round of global information extraction on z1 to obtain the intermediate feature map z2;

[0017] NLSA2 is used to perform a second round of global information extraction on z2 to obtain the intermediate feature map z3;

[0018] RHAGs is used to perform multi-scale learning to expand the receptive field of z3 to obtain the intermediate feature map z4;

[0019] NLSA3 is used to perform the third round of global information extraction on z4 to obtain the intermediate feature map z5;

[0020] NLSA4 is used to perform the fourth round of global information extraction on z5 to obtain the intermediate feature map z6;

[0021] Conv2 is used to extract deeper features from z6 and adjust the number of feature channels to obtain the intermediate feature map z7 adapted to PS;

[0022] PS is used to rearrange pixels of z7 to increase the image resolution and obtain the intermediate feature map z8;

[0023] Conv3 is used to smooth z8 to get X HR .

[0024] Among them, the loss function L used in the training of the super-resolution reconstruction model is total for:

[0025] L Total =L SWT +L RGB ;

[0026] Where, L SWT represents the static wavelet loss, L RGB represents the L1 loss in RGB domain.

[0027] This instrument image super-resolution reconstruction method based on NLSA and SWL implements the method or process according to the embodiment of the present disclosure.

[0028] In a second aspect, the present invention discloses an instrument image super-resolution reconstruction device based on NLSA and SWL, which uses the instrument image super-resolution reconstruction method based on NLSA and SWL disclosed in the first aspect.

[0029] The instrument image super-resolution reconstruction device based on NLSA and SWL includes: an image acquisition module, an image preprocessing module, and an image super-resolution reconstruction module.

[0030] The image acquisition module is used to acquire low-resolution instrument images X LR .

[0031] The image preprocessing module is used to LR Perform image enhancement preprocessing to obtain the enhanced image X Em .

[0032] The image super-resolution reconstruction module is used to transform X Em The trained super-resolution reconstruction model is used to process the reconstructed high-resolution instrument image X HR .

[0033] The instrument image super-resolution reconstruction device based on NLSA and SWL implements the method or process according to the embodiment of the present disclosure.

[0034] Compared with the prior art, the present invention has the following beneficial effects:

[0035] 1. The instrument image super-resolution reconstruction method based on NLSA and SWL provided by the present invention adopts a self-designed super-resolution reconstruction model. On the one hand, it integrates multiple attention mechanisms to realize multi-scale and comprehensive feature extraction. On the other hand, it adopts a loss function including static wavelet loss and L1 loss in RGB domain to ensure the effect of model training, thereby realizing real-time reconstruction while ensuring the model reconstruction effect.

[0036] 2. The instrument image super-resolution reconstruction method based on NLSA and SWL provided by the present invention can generate high-resolution instrument images using existing low-quality instrument images without increasing the cost of imaging equipment, thereby significantly improving the clarity and detail of the image.

[0037] 3. The instrument image super-resolution reconstruction method based on NLSA and SWL provided by the present invention is particularly suitable for super-resolution reconstruction of low-resolution images in special scenarios such as the relay room of a thermal power plant. It can effectively restore details such as numbers and scales under various environmental conditions such as high noise and large temperature changes, ensuring the robustness and stability of image processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0039] Figure 1 A data flow diagram of the instrument image super-resolution reconstruction method based on NLSA and SWL provided in Example 1 of the present invention;

[0040] Figure 2 for Figure 1 The data flow graph of any non-local sparse attention unit (NLSA);

[0041] Figure 3 for Figure 1 The data flow diagram of any residual hybrid attention unit (RHAG);

[0042] Figure 4 for Figure 3 The data flow diagram of any hybrid attention block (HAB);

[0043] Figure 5 A diagram showing the construction process of the static wavelet loss provided in Example 1 of the present invention;

[0044] Figure 6 A schematic diagram of the static wavelet transform (SWT) provided in Example 1 of the present invention. DETAILED DESCRIPTION

[0045] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0046] It should be noted that when a component is referred to as being "mounted on" another component, it may be directly on the other component or there may be a central component. When a component is considered to be "set on" another component, it may be directly set on the other component or there may be a central component at the same time. When a component is considered to be "fixed to" another component, it may be directly fixed on the other component or there may be a central component at the same time.

[0047] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which the present invention belongs. The terms used herein in the specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The term "or / and" used herein includes any and all combinations of one or more of the related listed items.

[0048] Example 1

[0049] See also Figure 1 , Figure 1 The data flow diagram of the instrument image super-resolution reconstruction method based on NLSA and SWL is shown, which actually also shows the process of the instrument image super-resolution reconstruction method based on NLSA and SWL.

[0050] like Figure 1 As shown, the instrument image super-resolution reconstruction method based on NLSA and SWL includes the following steps:

[0051] Step 1: Get low-resolution instrument image X LR .

[0052] Among them, X LR The specifications are (H, W, C); H represents the height; W represents the width; C represents the number of channels.

[0053] If the instrument is a cabinet instrument in the relay room of a thermal power plant, this step can be implemented by using a robot in the relay room of the thermal power plant to directly photograph the cabinet instrument during an autonomous inspection; of course, other photographing equipment can also be used to directly photograph the cabinet instrument.

[0054] Step 2: XLR Perform image enhancement preprocessing to obtain the enhanced image X Em .

[0055] Among them, X Em The specifications are still (H, W, C).

[0056] X LR The details and edge information in the image will be blurred due to the sparse arrangement of pixels. Directly performing step three processing on it will increase the burden of model processing. Therefore, it is necessary to perform image enhancement preprocessing (including basic image enhancement steps such as denoising and normalization) before performing step three processing.

[0057] It should be noted that although step 2 adds a processing step, the processing load of step 2 itself is very low and can significantly reduce the processing load of step 3, so it is very meaningful.

[0058] Step 3: X Em The trained super-resolution reconstruction model is used to process the reconstructed high-resolution instrument image X HR .

[0059] It should be noted that the present invention requires the use of a trained super-resolution reconstruction model whose network parameters are in an optimal state.

[0060] The core of the present invention is the specific structure of the super-resolution reconstruction model, and the core is the loss function L used in the training of the super-resolution reconstruction model. total ; Only by combining the two can we balance the reconstruction effect and computing resource consumption.

[0061] 1. First, let’s look at the specific structure of the super-resolution reconstruction model:

[0062] See also Figure 1 The super-resolution reconstruction model includes: 3 convolution parts Conv1~Conv3, 4 non-local sparse attention parts NLSA1~NLSA4, 1 residual hybrid attention group part RHAGs, and 1 pixel rearrangement part PS.

[0063] From the data flow point of view, the functions of the above parts are as follows:

[0064] Conv1 is used to Em Perform preliminary feature extraction to obtain the intermediate feature map z1;

[0065] NLSA1 is used to perform the first round of global information extraction on z1 to obtain the intermediate feature map z2;

[0066] NLSA2 is used to perform a second round of global information extraction on z2 to obtain the intermediate feature map z3;

[0067] RHAGs is used to perform multi-scale learning to expand the receptive field of z3 to obtain the intermediate feature map z4;

[0068] NLSA3 is used to perform the third round of global information extraction on z4 to obtain the intermediate feature map z5;

[0069] NLSA4 is used to perform the fourth round of global information extraction on z5 to obtain the intermediate feature map z6;

[0070] Conv2 is used to extract deeper features from z6 and adjust the number of feature channels to obtain the intermediate feature map z7 adapted to PS;

[0071] PS is used to rearrange pixels of z7 to increase the image resolution and obtain the intermediate feature map z8;

[0072] Conv3 is used to smooth z8 to get X HR .

[0073] Among them, the specifications of z1~z6 are all (H, W, C); the specifications of z7 are (H, W, C×r 2 ); z8, X HR The specifications are (H×r, W×r, C). r represents the magnification factor.

[0074] The following is a detailed description:

[0075] (i) Conv1 is intended to perform preliminary feature extraction, which is mainly to extract basic features of the image, including edges, textures, details, etc. In this embodiment 1, Conv1 uses a two-dimensional convolution with a convolution kernel size of 5×5.

[0076] Conv2 is intended to extract deeper features and change the number of feature channels for subsequent pixel rearrangement. In this embodiment 1, Conv1 uses a two-dimensional convolution with a convolution kernel size of 3×3.

[0077] Conv3 aims to spatially smooth the feature map after pixel rearrangement and eliminate discontinuities or artifacts that may be caused by pixel rearrangement - it can learn the contextual information around each pixel position to generate a clearer high-resolution image. In this embodiment 1, Conv3 uses a two-dimensional convolution with a convolution kernel size of 3×3.

[0078] (ii) NLSA1~NLSA4 as non-local sparse attention (NLSA), which aims to reduce redundant information through sparse processing and capture long-range dependencies in the image through non-local mechanisms.

[0079] Unlike ordinary convolution that only processes local features, NLSA can use local sensitive hashing operations to divide image features into sparse groups and calculate sparse self-attention to enhance the global feature extraction capability of the model.

[0080] See also Figure 2 NLSA1~NLSA4 have the same structure, including: 1 local sensitive hashing layer LSH, 1 sparse self-attention layer SA, and 1 remapping layer Concat.

[0081] In any non-local sparse attention unit:

[0082] 201, LSH is used to perform multiple rounds of local sensitive hashing operations on the input Input of the non-local sparse attention part to obtain I hash buckets B1~B i Among them, B i Contains j i Features.

[0083] LSH divides the global features into multiple subsets through a hash function, so that similar feature points have a greater probability of being assigned to the same hash bucket.

[0084] It should be noted that the hash function ensures that similar features will be assigned to the same bucket, and feature points between different buckets will not calculate attention to each other, which will greatly reduce the amount of global attention calculation. However, after a single round of local sensitive hashing operation, some related features always have a small probability of being incorrectly hashed into different hash buckets. Therefore, LSH will perform multiple rounds of local sensitive hashing operations to make the hashing process more robust - generally, it is controlled to at least 3 rounds. In addition, this model designs NLSA1~NLSA4 and increases the overall LSH processing rounds to further optimize the hashing process.

[0085] It should be noted that B1~B i The number of features included is not necessarily the same.

[0086] 202, SA is used for the i-th hash bucket B i Perform self-attention calculation to obtain the feature output y in the bucket i .

[0087] The core idea of ​​SA is to calculate the mutual dependence of feature points in the bucket, so as to capture the correlation of features in the bucket.

[0088] Specifically, Figure 2 As shown, SA includes: feature conversion layer, dot product layer, Softmax function layer, and weighted summation layer.

[0089] Ⅰ. Feature conversion layer is used to transform B iAny feature b is transformed to obtain the corresponding query point Q b , key K b , value matrix V b Of course, the processing of the feature conversion layer can be formulated as:

[0090]

[0091] Where b∈B i ; W Q is the query point transformation matrix, W K is the key point transformation matrix, W V is the value matrix change matrix.

[0092] II. The dot product layer is used to calculate Q through dot product operations b , K b Similarity X b Of course, the processing of the dot product layer can be formulated as:

[0093]

[0094] In the formula, K b T K b Bias of K b Dimension.

[0095] III. Softmax function layer is used to b Normalize through the softmax function to get the attention weight weight b Of course, the processing of the Softmax function layer can be formulated as:

[0096] weight b =softmax(X b )·V b ;

[0097] Where, softmax(.) represents the softmax function.

[0098] It should be noted that the sparse attention mechanism ensures that each query point only interacts with other key points in the bucket to which it belongs.

[0099] IV. The weighted summation layer is used to weight b 、V b Perform a weighted sum to get y i Of course, the processing of the weighted sum layer can be formulated as:

[0100]

[0101] 203, Concat is used to output the features in I buckets y1~y i Remap it back to the original position to aggregate into a feature map with the same size as the Input and serve as the output of the non-local sparse attention unit.

[0102] Of course, the Concat process can be formulated as:

[0103] Output = concat(y1,y2,...,y i );

[0104] In the formula, concat(.) indicates Concat processing.

[0105] Based on the above non-local sparse attention structure:

[0106] The input of NLSA1 is z1, and the output is z2;

[0107] The input of NLSA2 is z2, and the output is z3;

[0108] The input of NLSA3 is z4, and the output is z5;

[0109] The input of NLSA4 is z5, and the output is z6.

[0110] (III) RHAGs is built based on four residual mixed attention units, which further enhances the feature expression ability of the image by introducing the mixed attention mechanism and alleviates the gradient vanishing problem through residual connections while maintaining image details.

[0111] See also Figure 1 , RHAGs includes: 4 residual mixed attention parts RHAG1~RHAG4 and 1 stacking layer.

[0112] In RHAGs:

[0113] 301, RHAG m Used to introduce a hybrid attention mechanism to the intermediate feature map z 3,m-1 Further feature extraction is performed to obtain the intermediate feature map z 3,m ; m∈[1,4]; where z 3,0 is z3.

[0114] 302, the overlay is used to connect z3 with z 3,4 Superposition is performed to obtain z4.

[0115] In general, the RHAGs process can be formulated as follows:

[0116] z4=z3+RHAG(RHAG(RHAG(RHAG(z3))));

[0117] Wherein, RHAG(.) represents RHAG treatment.

[0118] For details, see Figure 3 RHAG1~RHAG4 have the same structure, including: 4 hybrid attention parts HAB1~HAB4, 1 overlapping cross attention layer OCA, 1 sub-convolutional layer conv, and 1 sub-overlay layer.

[0119] In any residual mixed attention part:

[0120] Ⅰ.HAB m For the intermediate feature map g based on the hybrid attention mechanism m-1 Perform feature extraction to obtain the intermediate feature map g m ; m∈[1,4]; where g0 is the input IN of the residual mixed attention unit.

[0121] See also Figure 4 ,Specifically, the structures of HAB1 to HAB4 are the same, and all include: 2 normalization layers Norm1 to Norm2, 1 channel attention layer CA, 1 window attention layer WA, 2 grandson stacking layers, and 1 multilayer perceptron MLP.

[0122] In any mixed attention unit,

[0123] Norm1 is used to normalize the input in of the mixed attention part to obtain the intermediate feature map t1;

[0124] CA is used to process t1 based on the channel attention mechanism to obtain the intermediate feature map t2;

[0125] WA is used to process t1 based on the spatial attention mechanism to obtain the intermediate feature map t3;

[0126] The first superposition layer is used to superimpose t2 and t3 to obtain the intermediate feature map t4;

[0127] Norm2 is used to normalize t4 to obtain the intermediate feature map t5;

[0128] MLP is used to further extract and combine global features from t5 to obtain the intermediate feature map t6;

[0129] The second grand superposition layer is used to superimpose t6 and in to obtain the output out of the mixed attention part.

[0130] That is to say, the hybrid attention unit first normalizes in, and then inputs two parallel branches - the channel attention perception branch and the window attention perception branch (the channel attention perception branch is responsible for extracting the color of the image, using point-by-point convolution to capture color and brightness features; the window attention perception branch focuses on restoring the spatial details of the image, such as edge and contour information). The outputs of the two branches are then fused to ensure that the color and spatial information remain consistent during the super-resolution reconstruction process, and the fusion result is normalized. After that, the multi-layer perceptron is connected to further extract and combine global features. Finally, in is connected through the residual to ensure that the features flow smoothly between modules and reduce feature loss.

[0131] Therefore, the processing of the mixed attention part can be formulated as:

[0132] out=HAB(in)=in+MLP(Norm(F channel (Norm(in))+F windows (Norm(in))));

[0133] F channel (Norm(in))=σ(W channel *Norm(in)+b′);

[0134] F windows (Norm(in))=σ(W windows *Norm(in)+b′);

[0135] Where, HAB(.) represents HAB processing, MLP(.) represents MLP processing, Norm(.) represents normalization processing, and F channel (.) indicates CA processing, F windows (.) represents WA processing, σ represents the LeakyReLU activation function, and b' represents the bias term used to adjust the attention weight.

[0136] II. OCA is used to process g4 based on the overlapping cross attention mechanism to obtain the intermediate feature map g 5 .

[0137] OCA is an attention mechanism used to enhance the performance of the model when processing sequence data. Its core is to consider the overlapping parts of multiple sequence fragments when calculating attention.

[0138] III. conv is used to perform convolution on g5 to obtain the intermediate feature map g6;

[0139] IV. The sub-overlay layer is used to superimpose g6 and IN to obtain the input OUT of the residual mixed attention part.

[0140] Therefore, the processing of the residual mixed attention part can be formulated as:

[0141] OUT=IN+conv(OCA(HAB(HAB(HAB(HAB(IN))))));

[0142] Wherein, HAB(.) represents HAB processing, and conv(.) represents convolution processing.

[0143] (iv) PS aims to rearrange the pixels of the low-resolution feature map according to specific rules to expand it to a high-resolution output.

[0144] For PS, the input z7 specification is (H, W, C×r 2 ), the output z8 specifications are (H×r, W×r, C).

[0145] PS distributes low-resolution pixels in multiple channels to high-resolution space, so that the spatial resolution is increased while the number of channels is reduced, thus increasing the height and width of the image.

[0146] Specifically, z7 is located at (h, w, c×r 2 +i×r+j) will be rearranged to the position of (h×r+i, w×r+j, c) in z8; where h represents the row index, w represents the column index, c represents the channel index, i represents the row offset, and j represents the column offset.

[0147] 2. Next, let’s look at the loss function L used in the training of the super-resolution reconstruction model. total :

[0148] It should be noted that the training set used for super-resolution reconstruction model training can adopt data instrument images of the relay room of a thermal power plant with different resolutions, which can be taken by a low-resolution camera and a high-resolution camera respectively.

[0149] Among them, multiple rounds of training are performed based on real low-resolution images (i.e. images taken by low-resolution cameras). Each round of training will generate reconstructed high-resolution images (i.e. images output by the super-resolution reconstruction model), and the corresponding real high-resolution images (i.e. images taken by high-resolution cameras) are used as label data.

[0150] Of course, the real low-resolution images used for training are also expanded by spatially downsampling the real high-resolution images, using Gaussian blur kernel filtering and bilinear interpolation downsampling. In addition, Gaussian white noise can be added to the real low-resolution images and real high-resolution images to simulate noise interference in real scenes.

[0151] L total It consists of two parts: static wavelet loss LSWT , L1 loss L in RGB domain RGB That is, L total The expression is

[0152] L Total =L SWT +L RGB .

[0153] I.L SWT Based on the static wavelet transform (SWT):

[0154] See also Figure 5 , the reconstructed high-resolution image generated by a round of training and the corresponding real high-resolution image are mapped to the luminance and chrominance space by YCbCr conversion, and then decomposed into several sub-bands based on static wavelet transform, and then the L1 loss of each corresponding sub-band is calculated and summed to obtain L SWT .

[0155] Among them, see Figure 6 ,Static wavelet transform can decompose a certain image Pic (i.e., the image after YCbCr conversion) into low-frequency and high-frequency sub-bands, retaining edge detail information at different scales.

[0156] Specifically, Pic is first decomposed along the row into a low-frequency part h that retains the general outline of the image low and the high-frequency part h that retains the edge information in the image high ; Then h low 、h high Apply static wavelet transform along the column direction: ① Change h low ② Decompose each column of the high-frequency part into HL sub-band and HH sub-band.

[0157] Among them, the LL subband is the low-frequency (horizontal and vertical) component, representing the smooth area of ​​the image;

[0158] The LH subband is the low-frequency (horizontal) and high-frequency (vertical) components, representing the horizontal smoothness and vertical edge details in the image;

[0159] The HH subband is the high-frequency (horizontal and vertical) components, representing the details in the image (such as noise or rapidly changing parts);

[0160] The HL subbands are high-frequency (horizontal) and low-frequency (vertical) components, representing the details of horizontal edges and vertical smooth areas in the image.

[0161] Therefore, if Figure 5As shown, first calculate the L1 loss of the LL subband corresponding to the reconstructed high-resolution image and the LL subband corresponding to the true high-resolution image, the L1 loss of the LH subband corresponding to the reconstructed high-resolution image and the LH subband corresponding to the true high-resolution image, the L1 loss of the HL subband corresponding to the reconstructed high-resolution image and the HL subband corresponding to the true high-resolution image, and the L1 loss of the HH subband corresponding to the reconstructed high-resolution image and the HH subband corresponding to the true high-resolution image; then add the four L1 losses to get L SWT .

[0162] So, L SWT The expression is:

[0163]

[0164] In the formula, ||.||1 represents L1 loss; SWT(.) represents static wavelet transform, λ j represents the weight corresponding to the j-th subband generated by the static wavelet transform; G represents the reconstructed high-resolution instrument image generated during model training; y represents the real high-resolution instrument image used during model training.

[0165] II.L RGB Built on the RGB domain:

[0166] The reconstructed high-resolution image generated by a round of training and the corresponding real high-resolution image are directly processed in the RGB domain, and the RGB values ​​of the same pixels are compared. The sum and average are first calculated to obtain L RGB .

[0167] So, L RGB The expression is:

[0168]

[0169] Where N represents the total number of pixels; G(i) R 、G(i) G , G(i) B Respectively represent the RGB value of the i-th pixel in G; y i,R ,y i,G ,y i,B Represent the RGB value of the i-th pixel in y respectively.

[0170] In summary, based on L total Back propagation determines the gradient vector to update the parameters of the super-resolution reconstruction model. total It can ensure accurate reconstruction of high-frequency details in the image while maintaining the overall image quality.

[0171] It should be noted that the above-mentioned method is not only applicable to the above-mentioned thermal power plant cabinet instrument image, but can also be extended to other instrument images that also contain details such as readings and scale lines.

[0172] Simulation Verification

[0173] This embodiment 1 verifies the effect of the above-mentioned method on the cabinet instrument of the relay room of a thermal power plant:

[0174] The reconstructed high-resolution image obtained by the above-mentioned method is compared with the real high-resolution image, and relevant indicators are introduced for quality evaluation. The results are shown in Table 1.

[0175] Table 1 Index results

[0176] index SSIM PSNR Reconstruction takes time Index value 0.96 33.86 0.32ms

[0177] in,

[0178] (1) Structural Similarity Index Measure (SSIM) is an indicator to measure image quality. SSIM takes into account aspects such as image brightness, contrast, and structure, and evaluates the similarity between two images by comparing their structural similarities. The dynamic range of SSIM is [-1, 1]. The closer the value is to 1, the better the fusion quality.

[0179] (2) Peak Signal-to-Noise Ratio (PSNR) is an indicator to measure the quality of an image or video. PSNR evaluates the quality by comparing the ratio of effective signal to noise between images. The larger the value, the better.

[0180] (3) Reconstruction time is the total time taken by the model to generate and reconstruct a high-resolution image. The smaller the value, the better.

[0181] As can be seen from Table 1, the values ​​of SSIM, PSNR, and reconstruction time indicate that the method of the present invention has a good use effect, meets the requirements of autonomous inspection of relays in thermal power plants, and can achieve real-time reconstruction while ensuring the model reconstruction effect.

[0182] Example 2

[0183] This embodiment 2 provides an instrument image super-resolution reconstruction device based on NLSA and SWL, which uses the instrument image super-resolution reconstruction method based on NLSA and SWL disclosed in embodiment 1.

[0184] The instrument image super-resolution reconstruction device based on NLSA and SWL includes: an image acquisition module, an image preprocessing module, and an image super-resolution reconstruction module.

[0185] The image acquisition module is used to acquire low-resolution instrument images X LR .

[0186] The image preprocessing module is used to LR Perform image enhancement preprocessing to obtain the enhanced image X Em .

[0187] The image super-resolution reconstruction module is used to transform X Em The trained super-resolution reconstruction model is used to process the reconstructed high-resolution instrument image X HR .

[0188] Since the present device uses the instrument image super-resolution reconstruction method based on NLSA and SWL in Example 1, it also has the same effect, which will not be repeated here.

[0189] Example 3

[0190] This embodiment 3 discloses a computer device, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the steps of the instrument image super-resolution reconstruction method based on NLSA and SWL disclosed in embodiment 1 are implemented.

[0191] This embodiment 3 also discloses a readable storage medium, in which computer program instructions are stored. When the computer program instructions are read and executed by a processor, the steps of the instrument image super-resolution reconstruction method based on NLSA and SWL disclosed in embodiment 1 are executed.

[0192] This embodiment 3 also discloses a computer program product, including a computer program. When the computer program is executed by a processor, the steps of the instrument image super-resolution reconstruction method based on NLSA and SWL disclosed in embodiment 1 are implemented.

[0193] The above-mentioned embodiments only express several implementation methods of the present invention, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, and these all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.

Claims

1. A super-resolution reconstruction method for instrument images based on NLSA and SWL, characterized in that: include: Step 1: Get low-resolution instrument image X LR ; Step 2: X LR Perform image enhancement preprocessing to obtain the enhanced image X Em ; Step 3: X Em The trained super-resolution reconstruction model is used to process the reconstructed high-resolution instrument image X HR ; Among them, the super-resolution reconstruction model includes: 3 convolution parts Conv1~Conv3, 4 non-local sparse attention parts NLSA1~NLSA4, 1 residual hybrid attention group part RHAGs, and 1 pixel rearrangement part PS; Conv1 is used to Em Perform preliminary feature extraction to obtain the intermediate feature map z1; NLSA1 is used to perform the first round of global information extraction on z1 to obtain the intermediate feature map z2; NLSA2 is used to perform a second round of global information extraction on z2 to obtain the intermediate feature map z3; RHAGs is used to perform multi-scale learning to expand the receptive field of z3 to obtain the intermediate feature map z4; NLSA3 is used to perform the third round of global information extraction on z4 to obtain the intermediate feature map z5; NLSA4 is used to perform the fourth round of global information extraction on z5 to obtain the intermediate feature map z6; Conv2 is used to extract deeper features from z6 and adjust the number of feature channels to obtain the intermediate feature map z7 adapted to PS; PS is used to rearrange pixels of z7 to increase the image resolution and obtain the intermediate feature map z8; Conv3 is used to smooth z8 to get X HR ; Among them, the loss function L used in the training of the super-resolution reconstruction model is total for: L Total =L SWT +L RGB ; Where, L SWT represents the static wavelet loss, L RGB represents the L1 loss in RGB domain.

2. The instrument image super-resolution reconstruction method based on NLSA and SWL according to claim 1 is characterized in that: The convolution kernel size of Conv1 is 5×5; Or / and, the convolution kernel size of Conv2 is 3×3; Or / and, the convolution kernel size of Conv3 is 3×3.

3. The instrument image super-resolution reconstruction method based on NLSA and SWL according to claim 1 is characterized in that: NLSA1~NLSA4 have the same structure, including: 1 local sensitive hashing layer LSH, 1 sparse self-attention layer SA, and 1 remapping layer Concat; In any non-local sparse attention unit: LSH is used to perform multiple rounds of local sensitive hashing operations on the input of the non-local sparse attention part to obtain I hash buckets B1~B i Among them, B i Contains j i Features SA is used to calculate the i-th hash bucket B i Perform self-attention calculation to obtain the feature output y in the bucket i ; Concat is used to output the features in buckets y1~y i Remap it back to the original position to aggregate into a feature map with the same size as the Input and serve as the output of the non-local sparse attention unit.

4. The instrument image super-resolution reconstruction method based on NLSA and SWL according to claim 3 is characterized in that: SA includes: The feature conversion layer is used to transform B i Any feature b is transformed to obtain the corresponding query point Q b , key K b , value matrix V b ; The dot product layer is used to calculate Q through the dot product operation b , K b Similarity X b ; Softmax function layer, which is used to b Normalize through the softmax function to get the attention weight weight b ; as well as The weighted summation layer is used to weight b 、V b Perform a weighted sum to get y i .

5. The instrument image super-resolution reconstruction method based on NLSA and SWL according to claim 1, characterized in that: RHAGs includes: 4 residual mixed attention parts RHAG1~RHAG4 and 1 stacking layer; In RHAGs: RHAG m Used to introduce a hybrid attention mechanism to the intermediate feature map z 3,m-1 Further feature extraction is performed to obtain the intermediate feature map z 3,m ; m∈[1,4]; where z 3,0 is z3; Overlays are used to connect z3 with z 3,4 Superposition is performed to obtain z4.

6. The instrument image super-resolution reconstruction method based on NLSA and SWL according to claim 5, characterized in that: RHAG1~RHAG4 have the same structure. Includes: 4 hybrid attention parts HAB1~HAB4, 1 overlapping cross attention layer OCA, 1 sub-convolutional layer conv, and 1 sub-overlay layer; In any residual mixed attention part: HAB m For the intermediate feature map g based on the hybrid attention mechanism m-1 Perform feature extraction to obtain the intermediate feature map g m ; m∈[1,4]; where g0 is the input IN of the residual mixed attention unit; OCA is used to process g4 based on the overlapping cross attention mechanism to obtain the intermediate feature map g5; Conv is used to perform convolution on g5 to obtain the intermediate feature map g6; The sub-overlay layer is used to superimpose g6 with IN to obtain the input OUT of the residual mixed attention part.

7. The instrument image super-resolution reconstruction method based on NLSA and SWL according to claim 6 is characterized in that: HAB1~HAB4 have the same structure, including: 2 normalization layers Norm1~Norm2, 1 channel attention layer CA, 1 window attention layer WA, 2 grandson stacking layers, and 1 multi-layer perceptron MLP; In any mixed attention unit, Norm1 is used to normalize the input in of the mixed attention part to obtain the intermediate feature map t1; CA is used to process t1 based on the channel attention mechanism to obtain the intermediate feature map t2; WA is used to process t1 based on the spatial attention mechanism to obtain the intermediate feature map t3; The first superposition layer is used to superimpose t2 and t3 to obtain the intermediate feature map t4; Norm2 is used to normalize t4 to obtain the intermediate feature map t5; MLP is used to further extract and combine global features from t5 to obtain the intermediate feature map t6; The second grand superposition layer is used to superimpose t6 and in to obtain the output out of the mixed attention part.

8. The instrument image super-resolution reconstruction method based on NLSA and SWL according to claim 1, characterized in that: X LR , X Em 、The specifications of z1~z6 are all (H, W, C); The specifications of z7 are (H, W, C×r 2 ); z8、X HR The specifications are (H×r, W×r, C); Among them, H represents height; W represents width; C represents the number of channels; and r represents the magnification factor.

9. The instrument image super-resolution reconstruction method based on NLSA and SWL according to claim 1, characterized in that: L SWT The expression is: In the formula, ||.||1 represents L1 loss; SWT(.) represents static wavelet transform, λ j represents the weight corresponding to the j-th subband generated by the static wavelet transform; G represents the reconstructed high-resolution instrument image generated during model training; y represents the real high-resolution instrument image used during model training; L RGB The expression is: Where N represents the total number of pixels; G(i) R 、G(i) G 、G(i) B Respectively represent the RGB value of the i-th pixel in G; y i,R ,y i,G ,y i,B Represent the RGB value of the i-th pixel in y respectively.

10. An instrument image super-resolution reconstruction device based on NLSA and SWL, characterized in that: It uses the instrument image super-resolution reconstruction method based on NLSA and SWL as described in any one of claims 1 to 9; The instrument image super-resolution reconstruction device based on NLSA and SWL includes: Image acquisition module, which is used to acquire low-resolution instrument image X LR ; Image preprocessing module, which is used to LR Perform image enhancement preprocessing to obtain the enhanced image X Em ;as well as Image super-resolution reconstruction module, which is used to transform X Em The trained super-resolution reconstruction model is used to process the reconstructed high-resolution instrument image X HR .

Images