Leaf reflective coding image stripe repair system and method combined with multi-exposure fusion

Abstract

The present invention designs a leaf reflective coding image stripe repair system and method combined with multi-exposure fusion. The repair system includes: an image acquisition module, an image fusion module, a mask image production module, an image annotation module, a generator module and a discriminator module; the image acquisition module collects leaf stripe images of different exposures, and the image fusion module fuses leaf stripe images of different exposures under the same posture to obtain an exposure-fused leaf stripe image; the mask image production module produces an irregular mask image; the image annotation module annotates the reflection and stripe confusion and missing parts in the leaf stripe image to obtain an annotated leaf stripe image, namely a label image; the generator module uses the leaf stripe image fused by the multi-exposure algorithm and the irregular mask image as input to perform feature extraction and reconstruction to generate a finely repaired leaf stripe image; the discriminator module distinguishes the leaf stripe image generated by the generator from the label image.

Description

Technical Field

[0001] The present invention belongs to the field of three-dimensional measurement of aero-engine blades, and in particular relates to a blade reflective coding image stripe repair system and method combined with multi-exposure fusion. Background Art

[0002] Structured light 3D reconstruction is one of the active 3D reconstruction methods used to reconstruct the three-dimensional topography data of the surface of the object to be measured. Because it is a non-contact measurement, it can effectively avoid the inefficiency and instability of contact measurement and adapt to complex measurement environments. In addition, 3D scanning measurement is fast and can obtain a large amount of information, making it widely used in modern industrial inspection. However, this method has certain problems when reconstructing the topography of free-form surface objects. The free-form surface will produce severe reflections, obscuring the coded stripes projected onto the surface. This results in a lack of sufficient spatial position information in the reflective area during 3D reconstruction, which seriously affects the final reconstruction accuracy.

[0003] Currently, industrial approaches to addressing the reflection problem caused by free-form surfaces are primarily divided into manual and algorithmic approaches. Manual approaches typically involve manually applying dots and powder coating to reflective areas of the surface to suppress reflections. This approach not only significantly reduces measurement efficiency, but also increases uncertainty and error in the 3D reconstruction process due to manual intervention, thereby reducing measurement accuracy. Algorithmic approaches include exposure image fusion, adaptive fringe projection, and multi-viewpoint methods. These methods typically require complex theoretical support and are generally less time-efficient. Furthermore, direct 3D reconstruction using image intensity is not accurate or reliable. In recent years, deep learning-based exposure image fusion algorithms have emerged to address the reflection problem in structured light coded fringe images. These methods fuse multiple fringe images with different exposure times to produce a high-quality, reflection-free image. However, exposure image fusion methods still cannot effectively handle large, heavily reflective areas or dark backgrounds that obscure the coded fringe. Currently, no method exists to repair fringe patterns in leaf coded images.

[0004] Chinese patent "CN116205843A A method for acquiring three-dimensional point clouds of high-reflective aircraft engine blades based on adaptive stripe iteration" provides a method for acquiring three-dimensional point clouds of high-reflective aircraft engine blades based on adaptive stripe iteration. By binarizing the image, the part greater than a threshold of 250 is regarded as an overexposed area, and the boundary information of the overexposed area is extracted to obtain the coordinate index and corresponding grayscale value of the outer edge pixel point set of the overexposed area; by establishing a nonlinear least squares model to iteratively solve the optimal grayscale value of the overexposed area, three-dimensional reconstruction is performed based on the collected adaptive stripes and the joint calibration results of the camera and projector.

[0005] However, its definition of high-reflection areas is determined by a set grayscale threshold, which has limitations. Although the final result can alleviate the impact of high reflection to a certain extent, it cannot handle complex exposure situations. Summary of the Invention

[0006] To address the shortcomings of existing technologies, the present invention designs a system and method for restoring blade reflective coded image stripes by combining multi-exposure fusion. This method is used to suppress surface reflections on curved objects and repair structured light coded stripes. This method addresses the issue of reflective occlusion during structured light reconstruction. Once this issue is resolved, the method can be applied to measure a variety of complex curved reflective objects, not just engine blades. This significantly improves the performance and accuracy of 3D reconstruction and has broad application in measuring complex machined parts.

[0007] The leaf reflective coding image stripe restoration system combined with multi-exposure fusion includes: image acquisition module, image fusion module, mask image production module, image annotation module, generator module and discriminator module;

[0008] Among them, the image acquisition module includes a left camera and a right camera of the optical system, which are used to collect leaf stripe images with different exposures. The leaf stripe images include a sequence of leaf stripe images collected by the left camera of the structured light system and a sequence of leaf stripe images collected by the right camera; the image fusion module fuses the leaf stripe images with different exposures under the same posture through the existing exposure image fusion algorithm to obtain the leaf stripe image after exposure fusion; the mask image production module produces an irregular mask image; the image annotation module annotates the reflections, stripe confusion, and missing parts in the leaf stripe image to obtain the annotated leaf stripe image, that is, the label image; the generator module uses the leaf stripe image and the irregular mask image fused by the multi-exposure algorithm as input to perform feature extraction and reconstruction to generate a precisely repaired leaf stripe image; the discriminator module distinguishes the true from the false between the leaf stripe image generated by the generator and the produced label image;

[0009] The generator module includes: a coarse stage repair module and a fine stage repair module;

[0010] The coarse stage restoration module includes: a coarse stage encoder module and a coarse stage decoder module; used for extracting features and reconstructing the leaf stripe image and the mask image to obtain a coarse restoration leaf stripe image;

[0011] The fine stage restoration module includes: a fine stage texture encoder module, a fine stage structure encoder module, and a fine stage decoder module; and is used to extract features and reconstruct the coarse restoration leaf stripe image obtained by the coarse stage restoration module to obtain a high-quality leaf stripe image;

[0012] The coarse stage encoder module is used to perform a multi-stage downsampling operation on the input leaf stripe image after adding a mask, thereby reducing the resolution of the input image while extracting feature information; the coarse stage decoder module is used to restore the output of the coarse stage encoder module to the original resolution to complete image decoding and reconstruction, thereby obtaining a coarse repaired leaf stripe image;

[0013] The fine-stage texture encoder module is used to perform a multi-stage downsampling operation on the coarse-stage repaired leaf stripe image obtained by the coarse-stage repair module, thereby reducing the resolution of the input image while extracting feature information, and performing feature matching through the CA module; the fine-stage structure encoder module is used to perform a multi-stage downsampling operation on the coarse-stage repaired leaf stripe image obtained by the coarse-stage repair module, thereby reducing the resolution of the input image while extracting feature information, and reasonably predicting the structural information contained in the covered area through void convolution; the fine-stage decoder module is used to cascade the feature information obtained by the fine-stage texture encoder module and the fine-stage structure encoder module, and restore the cascaded features to the original resolution to complete image decoding and reconstruction, thereby obtaining a fine-repaired leaf stripe image;

[0014] The CA module is used to match the features / foreground in the missing pixels covered by the mask with the features / background of the valid pixels outside the mask area;

[0015] The discriminator module includes: an edge detection module, a discriminator structure branch module, and a discriminator texture branch module;

[0016] The edge detection module is used to perform edge detection on the finely repaired leaf stripe image obtained by the generator to obtain edge information of the finely repaired leaf stripe image;

[0017] The discriminator structure branch module is used to take the edge information obtained by the edge detection module as input and perform a multi-stage downsampling operation, thereby reducing the resolution of the input image while extracting feature information;

[0018] The discriminator texture branch module is used to extract texture features from the finely repaired leaf stripe image obtained by the generator.

[0019] The leaf reflective coded image stripe restoration method combined with multi-exposure fusion is based on the above-mentioned leaf reflective coded image stripe restoration system combined with multi-exposure fusion, and specifically includes the following steps:

[0020] Step 1: Using the image acquisition module, a sequence of leaf stripe images with different exposures is acquired as a pre-processed leaf stripe image;

[0021] Step 2: Use the existing multi-exposure image fusion algorithm in the image fusion module to fuse the leaf stripe images with different exposures at the same pose to obtain the fused leaf stripe image;

[0022] Step 3: Create irregular mask images with different mask ratios using a mask image creation module;

[0023] Step 4: Use the image annotation module to annotate the collected leaf stripe image for parts with reflections, stripe confusion, or missing parts, and obtain an annotated leaf stripe image.

[0024] Step 5: The fused leaf stripe image and irregular mask image are sent to the generator for coarse-stage stripe repair and fine-stage stripe repair to obtain a fine-repaired leaf stripe image;

[0025] The step 5 comprises:

[0026] Step 5.1: Multiply the fused leaf stripe image and the irregular mask image to obtain the leaf stripe image with mask, and use it as the input of the coarse stage restoration of the generator with a resolution of S*S;

[0027] Step 5.2: The input is fed into the coarse-stage encoder module in the coarse-stage restoration module, and feature A0 is obtained through a gated convolution with a kernel size of 5*5*64 and a dilation factor of 1.

[0028] Step 5.3: Feature A0 is convolved through a gated convolution with a kernel size of 3*3*128, a dilation factor of 1, and a stride of 2*2, and a gated convolution with a kernel size of 3*3*128, a dilation factor of 1, and a stride of 1*1 to obtain feature A1. The resolution of A1 is (S / 2)*(S / 2).

[0029] Step 5.4: Feature A1 is convolved through a gated convolution with a kernel size of 3*3*256, a dilation factor of 1, and a stride of 2*2, and a gated convolution with a kernel size of 3*3*256, a dilation factor of 1, and a stride of 1*1 to obtain feature A2. The resolution of A2 is (S / 4)*(S / 4).

[0030] Step 5.5: Obtain feature A3 by performing gated convolution on feature A2 with kernel size of 3*3*256, stride of 1*1, and dilation factors of 2, 4, 8, and 16 respectively. The resolution of A3 is (S / 4)*(S / 4).

[0031] Step 5.6: Feature A3 is passed through two gated convolutions with kernel sizes of 3*3*256, dilation factor of 1, and stride of 1*1 to obtain feature A4. The resolution of A4 is (S / 4)*(S / 4).

[0032] Step 5.7: Deconvolve feature A4 with a convolution kernel of 4*4*128, a dilation factor of 1, and a stride of 1 / 2*1 / 2, and a convolution kernel of 4*4*64, a dilation factor of 1, and a stride of 1 / 2*1 / 2 to obtain feature A5. The resolution of A5 is S*S.

[0033] Step 5.8: Apply gated deconvolution to feature A5 with a convolution kernel size of 3*3*3, a dilation factor of 1, and a stride of 1*1 to obtain a coarsely restored leaf stripe image.

[0034] Step 5.9: Send the coarse restored leaf stripe image to the fine stage restoration as the input of the fine stage texture encoder module and the fine stage structure encoder module;

[0035] Step 5.10: The input enters the fine-stage texture encoder module and the fine-stage structure encoder module respectively, and is respectively subjected to a gated convolution with a convolution kernel size of 5*5*64 and a dilation factor of 1 to obtain features B0 and C0, both with a resolution of S*S.

[0036] Step 5.11: Convolve features B0 and C0 through a gated convolution with a kernel size of 3*3*128, a dilation factor of 1, and a stride of 2*2, and a gated convolution with a kernel size of 3*3*128, a dilation factor of 1, and a stride of 1*1, respectively, to obtain features B1 and C1. The resolution of the two is (S / 2)*(S / 2);

[0037] Step 5.12: Feature B1 is subjected to contextual attention matching and a gated convolution with a kernel size of 3*3*256, a dilation factor of 1, and a stride of 1*1 to obtain feature B2. Feature C1 is subjected to two gated convolutions with kernel sizes of 3*3*256, a stride of 1*1, and dilation factors of 2 and 4 respectively to obtain feature C2. The resolution of B2 and C2 is (S / 4)*(S / 4);

[0038] The context attention matching includes:

[0039] Step S1.1: Use the feature B1 obtained in step 5.11 as the input of the CA module, and use a 3×3 convolution kernel to extract the foreground features and background features in the input feature map to obtain the foreground feature Q0 and background feature H0;

[0040] Step S1.2: reshape the background feature H0 into a one-dimensional feature to obtain the reshaped background feature H1;

[0041] Step S1.3: Use cosine similarity to measure the feature similarity matching between the foreground feature Q0 and the background feature H1 after feature reshaping;

[0042] Step S1.4: Use the Softmax function to calculate the attention weight W corresponding to each background area on the background feature H1;

[0043] Step S1.5: Use the background area block with the highest attention weight W as the deconvolution kernel to reconstruct the foreground feature pixel information covered by the mask.

[0044] Step 5.13: Use the concatenation of features B2 and C2 to obtain feature D0 as the input of the fine-stage decoder module. D0 is then passed through two gated convolutions with kernel size 3*3*256, dilation factor 1, and stride 1*1 to obtain feature D1. The resolution of D1 is (S / 2)*(S / 2).

[0045] Step 5.14: Deconvolve feature D1 with a convolution kernel of size 4*4*128, dilation factor 1, stride 1 / 2*1 / 2 and a gated deconvolution kernel of size 4*4*64, dilation factor 1, stride 1 / 2*1 / 2 to obtain feature D2. The resolution of D2 is S*S.

[0046] Step 5.15: The feature D2 is subjected to gated deconvolution with a convolution kernel size of 3*3*3, a dilation factor of 1, and a step size of 1*1 to obtain a finely restored leaf stripe image.

[0047] Step 6: The finely restored leaf stripe image generated by the generator is sent to the discriminator for feature extraction and image reconstruction, and the generated image is judged as true or false;

[0048] The step 6 comprises:

[0049] Step 6.1: The finely restored leaf stripe image obtained by the generator module is subjected to edge detection by the edge detection module in the discriminator module to obtain a leaf stripe edge image;

[0050] The edge detection is specifically described as follows:

[0051] S2.1: The finely restored leaf stripe image obtained by the generator module is passed through a convolution layer with a kernel size of 3*3*16 and a stride of 1 to obtain feature H0;

[0052] S2.2: Pass feature H0 through two convolutional layers with kernel size of 3*3*16 and stride of 1 to obtain feature H1;

[0053] S2.3: Add features H0 and H1 to obtain feature H2;

[0054] S2.4: Pass feature H2 through a convolution layer with a kernel size of 1*1*1 and a stride of 1 to obtain feature H3, and activate H3 through the Sigmoid function to obtain the leaf stripe edge image.

[0055] Step 6.2: The finely restored leaf stripe image obtained by the generator module and the leaf stripe edge image obtained by the edge detection module are used as the input of the discriminator texture branch module and the discriminator structure branch module respectively. The features E0 and F0 are obtained by passing them through three convolutional layers with a kernel size of 4*4 and a stride of 2 respectively.

[0056] Step 6.3: Pass features E0 and F0 through two convolutional layers with a kernel size of 4*4 and a stride of 1 to obtain features E1 and F1;

[0057] Step 6.4: Normalize the mapping interval of features E1 and F1 to [0, 1] using the sigmoid function, perform a cascade operation, and then determine whether they are true or false.

[0058] Beneficial technical effects of the present invention:

[0059] (1) This invention uses a deep learning-based image processing method to reduce the impact of curved surface reflections on the accuracy of structured light 3D reconstruction of blades. Compared with traditional reflection processing methods, this invention proposes a two-stage method of first fusion and then restoration, which provides a new approach to solving the problem of reflection interference in structured light 3D reconstruction.

[0060] (2) The present invention uses a two-stage coarse-to-fine generative network architecture to establish connections between relevant feature information at distant spatial locations in leaf stripe images;

[0061] (3) This invention overcomes the limitation of convolutional neural networks that cannot effectively draw on distant spatial position features of images due to the use of local convolution kernels for image feature extraction by designing a texture generation branch based on the contextual attention module (CA Module). BRIEF DESCRIPTION OF THE DRAWINGS

[0062] Figure 1 Flowchart of a method for restoring leaf reflective coded image stripes using multi-exposure fusion in an embodiment of the present invention;

[0063] Figure 2 Schematic diagram of qualitative comparison results between the embodiment of the present invention and other methods;

[0064] Figure 3 An example diagram of mask image restoration based on user guidance in an embodiment of the present invention;

[0065] Figure 4 The rendering of the three-dimensional reconstruction of a blade according to an embodiment of the present invention. DETAILED DESCRIPTION

[0066] The present invention will be further described below with reference to the accompanying drawings and embodiments;

[0067] The leaf reflective coding image stripe restoration system combined with multi-exposure fusion includes: image acquisition module, image fusion module, mask image production module, image annotation module, generator module and discriminator module;

[0068] Among them, the image acquisition module includes a left camera and a right camera of the optical system, which are used to collect leaf stripe images with different exposures. The leaf stripe images include a sequence of leaf stripe images collected by the left camera of the structured light system and a sequence of leaf stripe images collected by the right camera; the image fusion module fuses the leaf stripe images with different exposures under the same posture through the existing exposure image fusion algorithm to obtain the leaf stripe image after exposure fusion; the mask image production module produces an irregular mask image; the image annotation module annotates the reflections, stripe confusion, and missing parts in the leaf stripe image to obtain the annotated leaf stripe image, that is, the label image; the generator module uses the leaf stripe image and the irregular mask image fused by the multi-exposure algorithm as input to perform feature extraction and reconstruction to generate a precisely repaired leaf stripe image; the discriminator module distinguishes the true from the false between the leaf stripe image generated by the generator and the produced label image;

[0069] The generator module includes: a coarse stage repair module and a fine stage repair module;

[0070] The coarse stage restoration module includes: a coarse stage encoder module and a coarse stage decoder module; used for extracting features and reconstructing the leaf stripe image and the mask image to obtain a coarse restoration leaf stripe image;

[0071] The fine stage restoration module includes: a fine stage texture encoder module, a fine stage structure encoder module, and a fine stage decoder module; and is used to extract features and reconstruct the coarse restoration leaf stripe image obtained by the coarse stage restoration module to obtain a high-quality leaf stripe image;

[0072] The coarse stage encoder module is used to perform a multi-stage downsampling operation on the input leaf stripe image after adding a mask, thereby reducing the resolution of the input image while extracting feature information; the coarse stage decoder module is used to restore the output of the coarse stage encoder module to the original resolution to complete image decoding and reconstruction, thereby obtaining a coarse repaired leaf stripe image;

[0073] The fine-stage texture encoder module is used to perform a multi-stage downsampling operation on the coarse-stage repaired leaf stripe image obtained by the coarse-stage repair module, thereby reducing the resolution of the input image while extracting feature information, and performing feature matching through the CA module; the fine-stage structure encoder module is used to perform a multi-stage downsampling operation on the coarse-stage repaired leaf stripe image obtained by the coarse-stage repair module, thereby reducing the resolution of the input image while extracting feature information, and reasonably predicting the structural information contained in the covered area through void convolution; the fine-stage decoder module is used to cascade the feature information obtained by the fine-stage texture encoder module and the fine-stage structure encoder module, and restore the cascaded features to the original resolution to complete image decoding and reconstruction, thereby obtaining a fine-repaired leaf stripe image;

[0074] The CA module is used to match the features / foreground in the missing pixels covered by the mask with the features / background of the valid pixels outside the mask area;

[0075] The discriminator module includes: an edge detection module, a discriminator structure branch module, and a discriminator texture branch module;

[0076] The edge detection module is used to perform edge detection on the finely repaired leaf stripe image obtained by the generator to obtain edge information of the finely repaired leaf stripe image;

[0077] The discriminator structure branch module is used to take the edge information obtained by the edge detection module as input and perform a multi-stage downsampling operation, thereby reducing the resolution of the input image while extracting feature information;

[0078] The discriminator texture branch module is used to extract texture features from the finely repaired leaf stripe image obtained by the generator.

[0079] The leaf reflective coded image stripe restoration method combined with multi-exposure fusion is based on the leaf reflective coded image stripe restoration system combined with multi-exposure fusion, as shown in the attached figure. Figure 1 As shown, the specific steps include:

[0080] Step 1: Using the image acquisition module, a sequence of leaf stripe images with different exposures is acquired as a pre-processed leaf stripe image;

[0081] Step 2: Use the existing multi-exposure image fusion algorithm in the image fusion module to fuse the leaf stripe images with different exposures at the same pose to obtain the fused leaf stripe image;

[0082] Step 3: Create irregular mask images with different mask ratios using a mask image creation module;

[0083] Similar mask images were produced by referring to the existing paper Image Inpainting for Irregular Holes Using PartialConvolutions. Specifically, masks of random stripes and holes of arbitrary shapes were collected, and the size, i.e., the proportion, of the masks was classified into several proportion intervals.

[0084] Step 4: Use the image annotation module to annotate the collected leaf stripe image for parts with reflections, stripe confusion, or missing parts, and obtain an annotated leaf stripe image.

[0085] Step 5: The fused leaf stripe image and irregular mask image are sent to the generator for coarse-stage stripe repair and fine-stage stripe repair to obtain a fine-repaired leaf stripe image;

[0086] The step 5 comprises:

[0087] Step 5.1: Multiply the fused leaf stripe image and the irregular mask image to obtain the leaf stripe image with mask, and use it as the input of the coarse stage restoration of the generator with a resolution of 256*256;

[0088] Step 5.2: The input is fed into the coarse-stage encoder module in the coarse-stage restoration module, and feature A0 is obtained through a gated convolution with a kernel size of 5*5*64 and a dilation factor of 1.

[0089] Step 5.3: Feature A0 is convolved through a gated convolution with a kernel size of 3*3*128, a dilation factor of 1, and a stride of 2*2, and a gated convolution with a kernel size of 3*3*128, a dilation factor of 1, and a stride of 1*1 to obtain feature A1. The resolution of A1 is 128*128.

[0090] Step 5.4: Feature A1 is convolved through a gated convolution with a kernel size of 3*3*256, a dilation factor of 1, and a stride of 2*2, and a gated convolution with a kernel size of 3*3*256, a dilation factor of 1, and a stride of 1*1 to obtain feature A2. The resolution of A2 is 64*64.

[0091] Step 5.5: Feature A2 is convolved through four gated convolutions with kernel sizes of 3*3*256, stride of 1*1, and dilation factors of 2, 4, 8, and 16 to obtain feature A3. The resolution of A3 is 64*64.

[0092] Step 5.6: Feature A3 is passed through two gated convolutions with kernel sizes of 3*3*256, dilation factor of 1, and stride of 1*1 to obtain feature A4. The resolution of A4 is 64*64.

[0093] Step 5.7: Feature A4 is deconvolved with a convolution kernel of 4*4*128, a dilation factor of 1, and a stride of 1 / 2*1 / 2, and a convolution kernel of 4*4*64, a dilation factor of 1, and a stride of 1 / 2*1 / 2 to obtain feature A5. The resolution of A5 is 256*256.

[0094] Step 5.8: Apply gated deconvolution to feature A5 with a convolution kernel size of 3*3*3, a dilation factor of 1, and a stride of 1*1 to obtain a coarsely restored leaf stripe image.

[0095] Step 5.9: Send the coarse restored leaf stripe image to the fine stage restoration as the input of the fine stage texture encoder module and the fine stage structure encoder module;

[0096] Step 5.10: The input enters the fine-stage texture encoder module and the fine-stage structure encoder module respectively, and is respectively subjected to a gated convolution with a convolution kernel size of 5*5*64 and a dilation factor of 1 to obtain features B0 and C0, both with a resolution of 256*256;

[0097] Step 5.11: Convolve features B0 and C0 through a gated convolution with a kernel size of 3*3*128, a dilation factor of 1, and a stride of 2*2, and a gated convolution with a kernel size of 3*3*128, a dilation factor of 1, and a stride of 1*1, respectively, to obtain features B1 and C1, both with a resolution of 128*128.

[0098] Step 5.12: Feature B1 is subjected to contextual attention matching and a gated convolution with a kernel size of 3*3*256, a dilation factor of 1, and a stride of 1*1 to obtain feature B2. Feature C1 is subjected to two gated convolutions with kernel sizes of 3*3*256, a stride of 1*1, and dilation factors of 2 and 4 respectively to obtain feature C2. The resolution of B2 and C2 is 64*64.

[0099] The context attention matching includes:

[0100] Step S1.1: Use the feature B1 obtained in step 5.11 as the input of the CA module, and use a 3×3 convolution kernel to extract the foreground features and background features in the input feature map to obtain the foreground feature Q0 and background feature H0;

[0101] Step S1.2: reshape the background feature H0 into a one-dimensional feature to obtain the reshaped background feature H1;

[0102] Step S1.3: Use cosine similarity to measure the feature similarity matching between the foreground feature Q0 and the background feature H1 after feature reshaping;

[0103] Step S1.4: Use the Softmax function to calculate the attention weight W corresponding to each background area on the background feature H1;

[0104] Step S1.5: Use the background area block with the highest attention weight W as the deconvolution kernel to reconstruct the foreground feature pixel information covered by the mask.

[0105] Step 5.13: Use the concatenation of features B2 and C2 to obtain feature D0 as the input of the fine-stage decoder module. D0 is then passed through two gated convolutions with kernel sizes of 3*3*256, dilation factor of 1, and stride of 1*1 to obtain feature D1. The resolution of D1 is 128*128.

[0106] Step 5.14: Deconvolve feature D1 with a convolution kernel of 4*4*128, a dilation factor of 1, and a stride of 1 / 2*1 / 2, and a convolution kernel of 4*4*64, a dilation factor of 1, and a stride of 1 / 2*1 / 2 to obtain feature D2. The resolution of D2 is 256*256.

[0107] Step 5.15: The feature D2 is subjected to gated deconvolution with a convolution kernel size of 3*3*3, a dilation factor of 1, and a step size of 1*1 to obtain a finely restored leaf stripe image.

[0108] Step 6: The finely restored leaf stripe image generated by the generator is sent to the discriminator for feature extraction and image reconstruction, and the generated image is judged as true or false;

[0109] The step 6 comprises:

[0110] Step 6.1: The finely restored leaf stripe image obtained by the generator module is subjected to edge detection by the edge detection module in the discriminator module to obtain a leaf stripe edge image;

[0111] The edge detection is specifically described as follows:

[0112] S2.1: The finely restored leaf stripe image obtained by the generator module is passed through a convolution layer with a kernel size of 3*3*16 and a stride of 1 to obtain feature H0;

[0113] S2.2: Pass feature H0 through two convolutional layers with kernel size of 3*3*16 and stride of 1 to obtain feature H1;

[0114] S2.3: Add features H0 and H1 to obtain feature H2;

[0115] S2.4: Pass feature H2 through a convolution layer with a kernel size of 1*1*1 and a stride of 1 to obtain feature H3, and activate H3 through the Sigmoid function to obtain the leaf stripe edge image.

[0116] Step 6.2: The finely restored leaf stripe image obtained by the generator module and the leaf stripe edge image obtained by the edge detection module are used as the input of the discriminator texture branch module and the discriminator structure branch module respectively. The features E0 and F0 are obtained by passing them through three convolutional layers with a kernel size of 4*4 and a stride of 2 respectively.

[0117] Step 6.3: Pass features E0 and F0 through two convolutional layers with a kernel size of 4*4 and a stride of 1 to obtain features E1 and F1;

[0118] Step 6.4: Normalize the mapping interval of features E1 and F1 to [0, 1] using the sigmoid function, perform a cascade operation, and then determine whether they are true or false.

[0119] In order to verify the performance of the blade reflective coding image stripe repair of the system and method of the present invention, the system and method proposed in the present invention are compared with the existing image repair methods. All methods are uniformly trained and tested on the same device, and the data set uses the independently produced EBS-II data set. The test results are shown in Table 1. The quantitative performance index comparison results of the method of the present invention and the existing image repair methods are intuitively demonstrated. The three indicators of SSIM, PSNR and LPIPS are used to verify the performance of the system and method of the present invention. In the table, the upward arrow at the evaluation index represents that the larger the value, the better, and the downward arrow represents that the smaller the value, the better. From the quantitative index comparison results shown in Table 1, it can be seen that the system and method of the present invention have advantages in extraction accuracy compared with other existing methods, and can obtain better stripe repair performance.

[0120] Table 1 Quantitative index results of the method of the present invention and other image restoration methods;

[0121]

[0122] from Figure 2 It can be seen from the figure that the method proposed in the present invention restores a more reasonable edge structure and clearer texture details, and its control of edge artifacts and semantic continuity of stripe structure are also better than other comparative methods.

[0123] The stripe repair method of the present invention was trained with the EBS-II dataset and then tested for reflection and stripe repair based on user guidance. Figure 3The results of restoration of leaf-encoded stripe images from the same scan sequence at different time domains are shown. The ellipses mark the areas of partial reflection and missing stripes in the image to be restored. Furthermore, a user-guided image restoration interface is used to draw user-guided masks corresponding to the areas to be restored. The restoration results clearly show that the proposed stripe restoration method successfully restores the reflective areas in the image to be restored and generates pixel predictions of grating stripes with clear texture, continuous structure, and reasonable semantics in the missing stripes.

[0124] The present invention also demonstrates the results of three-dimensional reconstruction in different postures after processing the stripe image using the invented method. Figure 4 As shown, in the first posture, due to the influence of the reflection on the blade surface, a large area of ​​point cloud is missing in the point cloud model (a) generated by direct three-dimensional reconstruction, especially in the entire left half of the blade surface and the root of the blade. After multi-exposure fusion processing of the original scan sequence images, the point cloud model (b) successfully completes the large area of ​​missing point cloud on the left surface of the blade in (a), but there are still a small amount of missing point cloud areas in the upper right corner and root of the blade. Point cloud model (c) uses the stripe repair method proposed in the invention to carry out targeted repair of these missing areas. It can be seen from the figure that the accuracy of the point cloud in the ellipse and square marked areas is significantly improved. The stripe repair method proposed in the present invention effectively completes the missing stripes at the root of the blade and generates a more accurate three-dimensional point cloud model. For the second posture, the point cloud model (f) generated after image repair has a higher point cloud density than models (d) and (e), and contains more blade surface information.

Claims

1. The leaf reflective coding image stripe restoration system combined with multi-exposure fusion is characterized by: include: Image acquisition module, image fusion module, mask image production module, image annotation module, generator module and discriminator module; Among them, the image acquisition module includes a left camera and a right camera of the optical system, which are used to collect leaf stripe images with different exposures. The leaf stripe images include a sequence of leaf stripe images collected by the left camera of the structured light system and a sequence of leaf stripe images collected by the right camera; the image fusion module fuses the leaf stripe images with different exposures under the same posture through the existing exposure image fusion algorithm to obtain the leaf stripe image after exposure fusion; the mask image production module produces an irregular mask image; the image annotation module annotates the reflections, stripe confusion, and missing parts in the leaf stripe image to obtain the annotated leaf stripe image, that is, the label image; the generator module uses the leaf stripe image and the irregular mask image fused by the multi-exposure algorithm as input to perform feature extraction and reconstruction to generate a precisely repaired leaf stripe image; the discriminator module distinguishes the true from the false between the leaf stripe image generated by the generator and the produced label image; Wherein, the generator module includes: a coarse stage repair module and a fine stage repair module; The coarse stage restoration module includes: a coarse stage encoder module and a coarse stage decoder module; used for extracting features and reconstructing the leaf stripe image and the mask image to obtain a coarse restoration leaf stripe image; The fine stage restoration module includes: a fine stage texture encoder module, a fine stage structure encoder module, and a fine stage decoder module; and is used to extract features and reconstruct the coarse restoration leaf stripe image obtained by the coarse stage restoration module to obtain a high-quality leaf stripe image.

2. The blade reflective coding image stripe restoration system combined with multi-exposure fusion according to claim 1 is characterized in that: The discriminator module includes: an edge detection module, a discriminator structure branch module, and a discriminator texture branch module; The edge detection module is used to perform edge detection on the finely repaired leaf stripe image obtained by the generator to obtain edge information of the finely repaired leaf stripe image; The discriminator structure branch module is used to take the edge information obtained by the edge detection module as input and perform a multi-stage downsampling operation, thereby reducing the resolution of the input image while extracting feature information; The discriminator texture branch module is used to extract texture features from the finely repaired leaf stripe image obtained by the generator.

3. The blade reflective coding image stripe restoration system combined with multi-exposure fusion according to claim 1, characterized in that: The coarse stage encoder module is used to perform a multi-stage downsampling operation on the input leaf stripe image after adding a mask, thereby reducing the resolution of the input image while extracting feature information; the coarse stage decoder module is used to restore the output of the coarse stage encoder module to the original resolution to complete image decoding and reconstruction, thereby obtaining a coarse repaired leaf stripe image; The fine-stage texture encoder module is used to perform a multi-stage downsampling operation on the coarse-stage repaired leaf stripe image obtained by the coarse-stage repair module, thereby reducing the resolution of the input image while extracting feature information, and performing feature matching through the CA module; the fine-stage structure encoder module is used to perform a multi-stage downsampling operation on the coarse-stage repaired leaf stripe image obtained by the coarse-stage repair module, thereby reducing the resolution of the input image while extracting feature information, and reasonably predicting the structural information contained in the covered area through void convolution; the fine-stage decoder module is used to cascade the feature information obtained by the fine-stage texture encoder module and the fine-stage structure encoder module, and restore the cascaded features to the original resolution to complete image decoding and reconstruction, thereby obtaining a fine-repaired leaf stripe image; The CA module is used to match the features\foreground in the missing pixels covered by the mask with the features\background of the valid pixels outside the mask area.

4. A method for restoring leaf reflective coded image stripes in combination with multi-exposure fusion is implemented based on the system for restoring leaf reflective coded image stripes in combination with multi-exposure fusion according to claim 1, characterized in that: The specific steps include: Step 1: Using the image acquisition module, a sequence of leaf stripe images with different exposures is acquired as a pre-processed leaf stripe image; Step 2: Use the existing multi-exposure image fusion algorithm in the image fusion module to fuse the leaf stripe images with different exposures at the same pose to obtain the fused leaf stripe image; Step 3: Create irregular mask images with different mask ratios using a mask image creation module; Step 4: Use the image annotation module to annotate the collected leaf stripe image for parts with reflections, stripe confusion, or missing parts, and obtain an annotated leaf stripe image. Step 5: The fused leaf stripe image and irregular mask image are sent to the generator for coarse-stage stripe repair and fine-stage stripe repair to obtain a fine-repaired leaf stripe image; Step 6: The finely restored leaf stripe image generated by the generator is sent to the discriminator for feature extraction and image reconstruction, and the generated image is judged as true or false.

5. The method for restoring leaf reflective coded image stripes combined with multi-exposure fusion according to claim 4, characterized in that: The step 5 comprises: Step 5.1: Multiply the fused leaf stripe image and the irregular mask image to obtain the leaf stripe image with mask, and use it as the input of the coarse stage restoration of the generator with a resolution of S*S; Step 5.2: The input is fed into the coarse-stage encoder module in the coarse-stage restoration module, and feature A0 is obtained through a gated convolution with a kernel size of 5*5*64 and a dilation factor of 1. Step 5.3: Feature A0 is convolved through a gated convolution with a kernel size of 3*3*128, a dilation factor of 1, and a stride of 2*2, and a gated convolution with a kernel size of 3*3*128, a dilation factor of 1, and a stride of 1*1 to obtain feature A1. The resolution of A1 is (S / 2)*(S / 2). Step 5.4: Feature A1 is convolved through a gated convolution with a kernel size of 3*3*256, a dilation factor of 1, and a stride of 2*2, and a gated convolution with a kernel size of 3*3*256, a dilation factor of 1, and a stride of 1*1 to obtain feature A2. The resolution of A2 is (S / 4)*(S / 4). Step 5.5: Obtain feature A3 by performing gated convolution on feature A2 with kernel size of 3*3*256, stride of 1*1, and dilation factors of 2, 4, 8, and 16 respectively. The resolution of A3 is (S / 4)*(S / 4). Step 5.6: Feature A3 is passed through two gated convolutions with kernel sizes of 3*3*256, dilation factor of 1, and stride of 1*1 to obtain feature A4. The resolution of A4 is (S / 4)*(S / 4). Step 5.7: Deconvolve feature A4 with a convolution kernel of 4*4*128, a dilation factor of 1, and a stride of 1 / 2*1 / 2, and a convolution kernel of 4*4*64, a dilation factor of 1, and a stride of 1 / 2*1 / 2 to obtain feature A5. The resolution of A5 is S*S. Step 5.8: Apply gated deconvolution to feature A5 with a convolution kernel size of 3*3*3, a dilation factor of 1, and a stride of 1*1 to obtain a coarsely restored leaf stripe image. Step 5.9: Send the coarse restored leaf stripe image to the fine stage restoration as the input of the fine stage texture encoder module and the fine stage structure encoder module; Step 5.10: The input enters the fine-stage texture encoder module and the fine-stage structure encoder module respectively, and is respectively subjected to a gated convolution with a convolution kernel size of 5*5*64 and a dilation factor of 1 to obtain features B0 and C0, both with a resolution of S*S. Step 5.11: Convolve features B0 and C0 through a gated convolution with a kernel size of 3*3*128, a dilation factor of 1, and a stride of 2*2, and a gated convolution with a kernel size of 3*3*128, a dilation factor of 1, and a stride of 1*1, respectively, to obtain features B1 and C1. The resolution of the two is (S / 2)*(S / 2); Step 5.12: Feature B1 is subjected to contextual attention matching and a gated convolution with a kernel size of 3*3*256, a dilation factor of 1, and a stride of 1*1 to obtain feature B2. Feature C1 is subjected to two gated convolutions with kernel sizes of 3*3*256, a stride of 1*1, and dilation factors of 2 and 4 respectively to obtain feature C2. The resolution of B2 and C2 is (S / 4)*(S / 4); Step 5.13: Use the concatenation of features B2 and C2 to obtain feature D0 as the input of the fine-stage decoder module. D0 is then passed through two gated convolutions with kernel size 3*3*256, dilation factor 1, and stride 1*1 to obtain feature D1. The resolution of D1 is (S / 2)*(S / 2). Step 5.14: Deconvolve feature D1 with a convolution kernel of size 4*4*128, dilation factor 1, stride 1 / 2*1 / 2 and a gated deconvolution kernel of size 4*4*64, dilation factor 1, stride 1 / 2*1 / 2 to obtain feature D2. The resolution of D2 is S*S. Step 5.15: The feature D2 is subjected to gated deconvolution with a convolution kernel size of 3*3*3, a dilation factor of 1, and a step size of 1*1 to obtain a finely restored leaf stripe image.

6. The method for restoring leaf reflective coded image stripes combined with multi-exposure fusion according to claim 5, characterized in that: The context attention matching includes: Step S1.1: Use the feature B1 obtained in step 5.11 as the input of the CA module, and use a 3×3 convolution kernel to extract the foreground features and background features in the input feature map to obtain the foreground feature Q0 and background feature H0; Step S1.2: reshape the background feature H0 into a one-dimensional feature to obtain the reshaped background feature H1; Step S1.3: Use cosine similarity to measure the feature similarity matching between the foreground feature Q0 and the background feature H1 after feature reshaping; Step S1.4: Use the Softmax function to calculate the attention weight W corresponding to each background area on the background feature H1; Step S1.5: Use the background area block with the highest attention weight W as the deconvolution kernel to reconstruct the foreground feature pixel information covered by the mask.

7. The method for restoring leaf reflective coded image stripes combined with multi-exposure fusion according to claim 4, characterized in that: The step 6 comprises: Step 6.1: The finely restored leaf stripe image obtained by the generator module is subjected to edge detection by the edge detection module in the discriminator module to obtain a leaf stripe edge image; Step 6.2: The finely restored leaf stripe image obtained by the generator module and the leaf stripe edge image obtained by the edge detection module are used as the input of the discriminator texture branch module and the discriminator structure branch module respectively. The features E0 and F0 are obtained by passing them through three convolutional layers with a kernel size of 4*4 and a stride of 2 respectively. Step 6.3: Pass features E0 and F0 through two convolutional layers with a kernel size of 4*4 and a stride of 1 to obtain features E1 and F1; Step 6.4: Normalize the mapping interval of features E1 and F1 to [0, 1] using the sigmoid function, perform a cascade operation, and then determine whether they are true or false.

8. The method for restoring leaf reflective coded image stripes combined with multi-exposure fusion according to claim 7, characterized in that: The edge detection is specifically described as follows: S2.1: The finely restored leaf stripe image obtained by the generator module is passed through a convolution layer with a kernel size of 3*3*16 and a stride of 1 to obtain feature H0; S2.2: Pass feature H0 through two convolutional layers with kernel size of 3*3*16 and stride of 1 to obtain feature H1; S2.3: Add features H0 and H1 to obtain feature H2; S2.4: Pass feature H2 through a convolution layer with a kernel size of 1*1*1 and a stride of 1 to obtain feature H3, and activate H3 through the Sigmoid function to obtain the leaf stripe edge image.

Images