A method for optimizing low-quality images of cabin doors based on deep learning

Abstract

A method for optimizing low-quality images of a cabin door based on deep learning, comprising the following steps: collecting cabin door docking images and making a high-quality - low-quality image dataset; constructing a deep learning network with a three-stage architecture based on the encoder-decoder of the U-net architecture and the single-scale channel structure; introducing an attention mechanism between each stage of the deep learning network; introducing image refinement modules between the encoder-decoder, and between the encoder-decoder and the single-scale channel structure respectively; training the deep learning network. The method of the present invention optimizes the low-quality images collected during the cabin door docking process based on deep learning, making the image features clearer, so that the entire docking process between the cabin door and the air bridge is smoother and more accurate.

Description

Technical Field

[0001] The present invention relates to the technical field of computer vision, and in particular to an optimization method for low-quality images of cabin doors based on deep learning. Background Art

[0002] During the docking process of the aircraft cabin door and the jet bridge, visual methods are used to collect cabin door images. However, the collected cabin door images are not clear enough, which will affect the extraction of image information. Therefore, it is necessary to find an algorithm for optimizing low-quality images with good effect and high efficiency.

[0003] Some current cabin door image optimization methods obtain the movement path of the camera by adding auxiliary devices, and use non-blind restoration methods to optimize blurred images on the premise of determining the blur kernel. This method obviously increases the cost and calculation difficulty. Moreover, for the docking process of the cabin door and the jet bridge, the movement of the visual camera is often relatively slight and frequent, and sometimes it is impossible to quantify in bad weather such as heavy rain and thick fog. Therefore, there are certain limitations. Summary of the Invention

[0004] In order to solve the deficiencies of the existing technology, the purpose of the present invention is to provide an optimization method for low-quality images of cabin doors based on deep learning, which optimizes the low-quality images collected during the docking process of the cabin door through deep learning based on a multi-stage convolutional neural network and an attention mechanism, making the image features clearer and facilitating the smooth progress of the docking work.

[0005] To achieve the above purpose, the optimization method for low-quality images of cabin doors based on deep learning provided by the present invention includes the following steps:

[0006] Collect cabin door docking images and make a high-quality - low-quality image dataset;

[0007] Based on the U-net architecture encoder-decoder and single-scale channel structure, construct a deep learning network with a three-stage architecture;

[0008] Introduce an attention mechanism between each stage of the deep learning network;

[0009] Introduce image refinement modules between the encoder-decoder and between the encoder-decoder and the single-scale channel structure respectively;

[0010] Train the deep learning network.

[0011] Further, the step of collecting cabin door docking images and making a high-quality - low-quality image dataset further includes: extracting low-quality frame images with pixel values lower than a preset value from the cabin door docking video;

[0012] Pair high-quality and low-quality images, where the pixel values of the high-quality images are higher than those of the low-quality images;

[0013] Use MATLAB to process the augmented data.

[0014] Furthermore, the step of constructing a deep learning network with a three-stage architecture and introducing the encoder-decoder of the U-net architecture and a single-scale channel structure includes: constructing a deep learning network with an early, middle, and late three-stage architecture; adding the encoder-decoder of the U-net architecture to the early stage and the middle stage, and using a single-scale channel structure to complete the late stage.

[0015] Furthermore, for the single-scale channel structure, an original resolution sub-network is adopted.

[0016] Furthermore, for the attention mechanism, a supervised attention module is adopted.

[0017] Furthermore, the step of training the deep learning network includes:

[0018] Input the image into the network and segment the image into multiple hierarchical structures;

[0019] Set an optimized end-to-end loss function;

[0020] Train the entire network by sub-tasks.

[0021] Furthermore, the formula for the optimized end-to-end loss function is:

[0022]

[0023] where X S is the restored image, Y is the ground truth image, the parameter λ is set to 0.05, and Ψ char and Ψ edge are two loss functions, and their expressions are as follows:

[0024]

[0025]

[0026] where the constant ε is set to 10^(-3) and Δ is the Laplacian operator.

[0027] Even further, the step of training the entire network by sub-tasks includes:

[0028] In the early and middle stages of the deep learning network, use multiple channel attention blocks at each scale, and select the 2×2 maximum mode for the pooling layer;

[0029] In a later stage of the deep learning network, the original resolution sub-network includes a plurality of original resolution sub-blocks, each of which uses a plurality of channel attention blocks;

[0030] Set the number of iterations.

[0031] By providing a cabin door low-quality image optimization method based on deep learning provided by the present invention, low-quality images collected during the cabin door docking process are optimized based on a deep learning network, so that the image features will be clearer, thereby making the entire docking process of the cabin door and the corridor bridge smoother and more accurate.

[0032] Other features and advantages of the present invention will be set forth in the description which follows, and in part will be obvious from the description, or may be learned by practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0034] Figure 1 A flow chart of a method for optimizing a low-quality image of a cabin door based on deep learning according to the present invention;

[0035] Figure 2 A schematic diagram of a characteristic channel according to the present invention;

[0036] Figure 3 is a schematic diagram of the original resolution sub-network structure according to the present invention;

[0037] Figure 4 A schematic diagram of a supervised attention module according to the present invention;

[0038] Figure 5 is a schematic diagram of a CFSS module according to the present invention;

[0039] Figure 6 A schematic diagram of a deep learning network structure according to the present invention;

[0040] Figure 7 Schematic diagram of comparison between a restored image according to the present invention and its corresponding high-quality image. DETAILED DESCRIPTION

[0041] The preferred embodiments of the present invention are described below in conjunction with the accompanying drawings. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention.

[0042] It should be understood that terms such as "high quality" and "low quality" may be used in this text to describe images in a schematic rather than restrictive manner. These terms are used only to distinguish images in terms of clarity or blurriness. Without departing from the scope of the exemplary embodiments, the quality level of the images can be changed.

[0043] To make the objectives, technical solutions and advantages of the present invention clearer, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings.

[0044] Figure 1 For the flowchart of the method for optimizing low-quality hatch images based on deep learning according to the present invention, reference will be made below to Figure 1 to describe in detail the method for optimizing low-quality hatch images based on deep learning of the present invention.

[0045] In step 101, collect hatch docking images and create a high-quality - low-quality image dataset.

[0046] The preprocessing of data has a great impact on the final effect of deep learning. Good preprocessing of the dataset can optimize the network processing effect, reduce the number of network iterations, and lower the computing cost.

[0047] In the embodiments of the present invention, the step of collecting hatch docking images and creating a high-quality - low-quality image dataset further includes: extracting low-quality frame images with pixel values lower than a preset value from the hatch docking video; pairing high-quality - low-quality images; and using MATLAB to process and expand the data. The preset pixel value can be set according to actual needs.

[0048] In the embodiments of the present invention, the low-quality frame images include blurred and distorted images. In some embodiments, it is possible to manually determine whether an image is a low-quality image. If it is a low-quality image, a relatively clear high-quality image in the adjacent frames of this image needs to be found as the paired image of this low-quality image, and a dataset is made.

[0049] To extract low-quality frames from the hatch docking video, the dataset used is obtained by processing the captured video stream. The video frame extraction code written in Python is set to extract 1 picture per 1 frame, with a total of 10,763 pictures, and the image annotation tool (labelimg) is used for unified annotation.

[0050] In addition to collecting low-quality images, clear images are also needed for effect comparison. Therefore, it is first selected to locate the low-quality images and collect drone picture data, and then find the high-quality images in the adjacent frames. A total of 262 pairs of high-low quality images are made.

[0051] Obviously, 262 pairs of images are not enough for the deep learning CNN network. The relatively small sample size is also likely to cause overfitting of the network, greatly reducing its robustness. Therefore, it is necessary to expand the sample set through certain means: in MATLAB, use the command: PSF = fspecial('motion', lenth, theta), and artificially blur high-quality clear images by manually setting the pixel horizontal displacement and offset angle, and finally complete the production of the data set.

[0052] In step 102, based on the encoder-decoder of the U-net architecture and the single-scale channel structure, a deep learning network with a three-stage architecture is constructed.

[0053] In the embodiments of the present invention, since the single-stage network model has an unsatisfactory solution effect on advanced vision problems, the design advantages of multi-stage networks have gradually emerged in recent years. Therefore, a three-stage network structure is selected to process low-quality hatch images, and different stages of the network are designed according to the advantages and disadvantages of different types of networks.

[0054] The encoder-decoder structure based on the U-net architecture is widely used in deep learning. It is a fully convolutional structure and can achieve better results with a relatively small data set.

[0055] On the contraction path, it uses two 3×3 convolutions, sets the Relu (activation function) and 2×2 max pooling operation with a stride of 2 after convolution. In each downsampling process, the size of the feature map is reduced to 1 / 2 of the original, and the number of feature channels is increased to 2 times the original.

[0056] After 5 times of sampling, a feature channel of 32×32 is finally obtained as Figure 2 shown.

[0057] On the expansion path, each step includes upsampling and 2×2 up-convolution, and the feature channels and feature sizes also perform corresponding operations when contracting.

[0058] The U-net structure has a very strong receptive field, but due to frequent downsampling, spatial information is lost. Therefore, the U-net architecture encoder-decoder is placed in the early and middle stages of the three-stage architecture deep learning network; in addition, TC is no longer used at the decoder end to increase spatial resolution, and bilinear sampling is set and then convolution is performed.

[0059] In the last stage of the deep learning network, in order to seek better spatial information and retain the details from the input image to the output image, the original resolution subnetwork (ORSNet) is introduced. This network does not include downsampling operations, so it can well retain the original image information.Figure 3 Schematic diagram of the original resolution sub-network structure according to the present invention. As Figure 3 shown, multiple channel attention blocks (CABs) constitute the original resolution block (ORB), and multiple ORBs constitute the entire ORSNet network.

[0060] In step 103, an attention mechanism is introduced between each stage of the deep learning network.

[0061] In the embodiments of the present invention, each layer of the convolutional neural network (CNN) outputs a feature map of C×H×W, and the planar weights in the same channel C are consistent. The attention mechanism is a method that mimics the human vision's focus on local regions. During the iterative process, the weights of the key regions are increased, and the weights of the non-key regions are decreased to form a new weight distribution. This weight distribution is given to the subsequent tasks, enabling the tasks to focus on the key regions and improving the effect of optimizing low-quality images.

[0062] An attention mechanism is introduced between each stage of the three-stage architecture, that is, a supervised attention module (SAM) is added to pay more attention to the contribution of the entity regions in the feature map and achieve progressive learning.

[0063] Figure 4 Schematic diagram of the supervised attention module according to the present invention. As Figure 4 shown, the supervised attention module of the present invention is specifically:

[0064] Process ①: Use the reshape and transpose functions to convert the feature matrix A into the transpose matrix of the C×H×W matrix. Among them, reshape is a function that can re-adjust the number of rows, columns, and dimensions of the matrix, and the transpose matrix of the matrix can be obtained using the transpose function;

[0065] Process ②: Multiply the above transpose matrix by the matrix A of C×H×W;

[0066] Process ③: Perform a softmax operation on the result obtained in Process ② to obtain the attention model X. Among them, the meaning of softmax is that instead of uniquely determining a certain maximum value, a probability value is assigned to the result of each output classification, indicating the possibility of belonging to each category;

[0067] Process ④: Multiply X T by A to obtain a matrix of dimension C*R;

[0068] Process ⑤: Convert the matrix of dimension C*R into C×H×W through the reshape function.

[0069] Process ⑥: Multiply the result obtained in Process ⑤ by the parameter Y, and then add it to A to obtain the output matrix E.

[0070] Introduce a supervised attention module (SAM) between different stages. SAM obtains the input feature map F of the previous stage in ∈R H×W×C , first generate a residual image R through a 1×1 convolution S ∈R H×W×3 , then add it to the degraded image I to obtain the restored image X S ∈R H×W×3 , and use the ground-truth image to provide explicit supervision information.

[0071] In step 104, introduce a CFSS module between the encoder-decoder and between the encoder-decoder and the single-scale channel structure respectively.

[0072] In the embodiment of the present invention, to effectively connect the three stages of the deep learning network, an image refinement module (CFSS module) is introduced between the encoder-decoder and between the encoder-decoder and the single-scale channel structure respectively, that is, before the features of the previous stage are input into the next stage, they are refined with a 1×1 convolution kernel, which helps to propagate the multi-scale context features from the early stage to the later stage.

[0073] Figure 5 As shown in the schematic diagram of the CFSS module according to the present invention, as Figure 5 shown, in the CFSS module of the present invention, when propagating image features at different stages, a 1×1 convolution kernel is used for refinement, which can retain more information and help to enrich the features of the next stage.

[0074] In step 105, train the deep learning network.

[0075] Figure 6 As shown in the schematic diagram of the deep learning network structure according to the present invention, as Figure 6 shown, the overall process of training the deep learning network of the present invention is as follows:

[0076] Input the image into the network, divide the image into a multi-block hierarchical structure, each stage of the network can access the input image, and each stage does not directly output the predicted image, but outputs the predicted residual image R s ;

[0077] Add the degraded input image I to it to get X S = I + R S ;

[0078] The end-to-end function for optimizing the architecture network is as follows:

[0079]

[0080] Among them, X S is the restored image, Y is the ground-truth image, and the parameter λ controls the relative importance of the two loss terms in the above formula. Therefore, λ is set to 0.05, and Ψ char and Ψ edge are two loss functions, where

[0081]

[0082]

[0083] Among them, the constant ε is set to 10^(-3), and Δ is the Laplacian operator.

[0084] The entire network is divided into three different tasks for training. At each scale in the early stage and the middle stage, 2 CABs are used, and the pooling layer selects the 2×2 maximum mode; in the ORSNet of the final stage, 3 ORBs are included, and each ORB uses 8 CABs; the number of iterations is set to 4×10 5 .

[0085] The effects of the deep learning network of the present invention on each dataset are shown in the following table:

[0086]

[0087] Among them, PSRN is the Peak Signal to Noise Ratio, and SSIM is the Structural Similarity Index, which is an index for measuring the similarity between two images; it can be seen from the table that the multi-stage progressive image restoration (MPRNet) method is used to optimize the image, and its peak signal-to-noise ratio is larger and the structural similarity index is better.

[0088] Figure 7 It is a schematic diagram for comparing the restored image according to the present invention with its corresponding high-quality image. As Figure 7 shown, the quality effect of the restored image obtained by the hatch low-quality image optimization method of the present invention is very good.

[0089] The low-quality image optimization method for the cabin door based on deep learning of the present invention applies a multi-stage CNN network to the optimization of the low-quality images of the aircraft cabin door. To improve the effect, an attention mechanism is introduced and a CFSS module is added; through verification among various data sets, it is closer to the real scene; generally speaking, the performance of the network can effectively optimize the low-quality images of the cabin door at the airport, so as to make the entire docking process between the cabin door and the air bridge smoother and more accurate.

[0090] Those of ordinary skill in the art can understand that the above are only the preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, for those skilled in the art, they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements for some of the technical features. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims

1. An optimization method for low-quality images of cabin doors based on deep learning, characterized in that, It includes the following steps: Collect hatch docking images and create a high-quality - low-quality image dataset; Based on the encoder-decoder of the U-net architecture and the single-scale channel structure, construct a deep learning network with a three-stage architecture; Introduce an attention mechanism between each stage of the deep learning network; Introduce an image refinement module between the encoder-decoder and between the encoder-decoder and the single-scale channel structure respectively; Train the deep learning network; The step of constructing a deep learning network with a three-stage architecture based on the encoder-decoder of the U-net architecture and the single-scale channel structure includes: constructing a deep learning network with early, middle, and late three-stage architectures; adding the U-net architecture encoder-decoder to the early stage and the middle stage, and adding the single-scale channel structure to the late stage; The step of training the deep learning network includes: Input the image into the network and segment the image into multiple hierarchical structures; Set an optimized end-to-end loss function; Train the entire network by tasks.

2. The method for optimizing low-quality images of the hatch based on deep learning according to claim 1, characterized in that, The step of collecting hatch docking images and creating a high-quality - low-quality image dataset further includes: extracting low-quality frame images with pixel values lower than a preset value from the hatch docking video; Pair high-quality - low-quality images, where the pixel value of the high-quality image is higher than that of the low-quality image; Use MATLAB to process and augment the data.

3. The method for optimizing low-quality images of the hatch based on deep learning according to claim 1, characterized in that The single-scale channel structure adopts an original resolution sub-network.

4. The method for optimizing low-quality images of the hatch based on deep learning according to claim 1, characterized in that, The attention mechanism adopts a channel attention model.

5. The method for optimizing low-quality images of the hatch based on deep learning according to claim 1, characterized in that The formula for the optimized end-to-end loss function is: Among them, X S is the restored image, Y is the real image, the parameter λ is set to 0.05, and Ψ char , Ψ edge are two loss functions, and their expressions are as follows: where the constant ε is set to 10^(-3) and Δ is the Laplacian operator.

6. The method for optimizing low-quality images of the hatch based on deep learning according to claim 1, characterized in that, The step of training the entire network by tasks includes: In the early and middle stages of the deep learning network, use multiple channel attention blocks at each scale, and select the 2×2 maximum mode for the pooling layer; In the late stage of the deep learning network, the original resolution sub-network includes multiple original resolution sub-blocks, and each original resolution sub-block uses multiple channel attention blocks; Set the number of iterations.

Images