A method and system for removing motion blur from airborne video images in a mine

Abstract

A method and system for removing motion blur from airborne video images in a mine, steps: constructing a blind image quality evaluation module based on a convolutional neural network for real scenarios, and extracting quality features related to quality in the image by means of the module; constructing a feature prediction module based on a codebook, pre-training a feature prediction network containing a high-quality feature codebook by means of vector coding, and performing prediction coding based on the quality features of the blurred image, so as to obtain high-quality prior knowledge of the image; embedding the obtained high-quality prior knowledge into the encoded features of the deblurring module after dimension alignment, and obtaining the restored clear image after decoding by the decoder. The system includes an image acquisition module, a blind image quality evaluation module, a feature prediction module and a deblurring module. The present invention can quickly process image and video loss of fidelity, effectively adapt to the underground environment of the mine, improve the accuracy of subsequent image processing and analysis, and improve the safety and efficiency of mining operations.

Description

Technical Field

[0001] The present invention relates to a method and system for removing motion blur from airborne video images in mines, belonging to the technical field of image deblurring. Background Art

[0002] In underground coal mine transportation, airborne videos of facilities such as battery locomotives, single rail hoists, and rubber-tyred vehicles without rails often suffer from picture jitter due to equipment vibration and movement, resulting in motion blur. This blurring phenomenon makes it difficult to clearly present video images, affecting subsequent image processing and analysis. Therefore, deblurring technology becomes crucial, aiming to restore the clarity of images through various algorithms. By eliminating motion blur, the video quality can be improved, providing more accurate visual data for subsequent detection, recognition, and monitoring. This not only helps improve the safety and efficiency of mining operations in the mining area but also optimizes the subsequent data processing process. Traditional deblurring methods abstract the blurring problem as a process of image convolution with a blur kernel, estimating the blur kernel by manually designing feature extraction. Therefore, they are only applicable to blurs in uniform situations. However, most blurs in the real world are non-uniform, so the effect of traditional deblurring methods in dealing with real image blurs is poor.

[0003] In recent years, with the continuous improvement of deep learning theory and the wide application of convolutional neural networks, deep learning-based methods have become the mainstream. Deep learning-based methods can automatically learn the deep features in the image blurring process through the convolution kernel by using blurred-clear image pairs to train the model, avoiding the biases generated in the blur kernel estimation process. However, the process of image deblurring is an ill-posed process. The deep learning-based method of end-to-end mapping lacks the guidance of prior knowledge, highly depends on the training of a large amount of data for the perception of blurred parts, cannot select key parts for processing, and has a poor processing effect on severely blurred parts.

[0004] Image deblurring aims to restore a clear image from a blurred image, and the quality evaluation algorithm aims to evaluate the quality of an image. The quality evaluation model can extract the features most relevant to the quality in the image. However, when actually deblurring, only the features of the blurred image can be obtained, and the features of the corresponding high-quality image need to be further obtained. Therefore, how to predict clear features from blurred deep features and effectively process severely blurred parts has become an urgent problem to be solved in the deblurring task. Summary of the Invention

[0005] The object of the present invention is to provide a method and system for removing motion blur from airborne video images in mines. This method can quickly process the distortion of image videos, effectively adapt to the underground mine environment, improve the accuracy of subsequent image processing and analysis, and enhance the safety and efficiency of mining operations in the mining area.

[0006] To achieve the above object, the present invention provides a method for removing motion blur from airborne video images in a mine, comprising the following steps:

[0007] S1. Construct a blind image quality evaluation module based on a convolutional neural network for real scenes, and extract quality features related to quality in the image by means of the module;

[0008] S2. Construct a feature prediction module based on a codebook, pre-train a feature prediction network containing a high-quality feature codebook by means of vector coding, and perform prediction coding based on the quality features of the blurred image, so as to obtain high-quality prior knowledge of the image;

[0009] S3. Embed the high-quality prior knowledge obtained in S2 into the encoded features of the deblurring module after dimension alignment, and obtain the restored clear image after decoding by the decoder of the module.

[0010] Further, the specific process of S1 is as follows:

[0011] S1.1. Fine-tune the image quality evaluation model pre-trained using the ImageNet database with a real distortion image quality evaluation dataset: Use Resnet50 as the basic architecture, and use the image quality evaluation dataset Koniq to train and fine-tune the image quality evaluation model. The formula is:

[0012]

[0013] In the formula, the input image is denoted as I, the quality score predicted by the image quality evaluation model is denoted as S p , and the dataset label score is denoted as S l . Update the parameters in a supervised learning manner and through the L1 loss to obtain the blind image quality evaluation model, and remove the last two convolutional layers to obtain the blind image quality evaluation module M BIQA ;

[0014] S1.2. Use the blind image quality evaluation module M BIQA to obtain the image quality features. Denote the input low-quality image as I LQ , and the extracted quality feature f LP is expressed as:

[0015] f LP = M BIQA (I LQ )(2).

[0016] Further, the specific process of S2 is as follows:

[0017] S2.1. Before sending the obtained quality feature f LP into the feature prediction network based on the codebook, for the quality feature fLP Refinement and alignment of dimensions are performed. A pre-fusion module consisting of three convolutional layers is adopted and denoted as Fblock1, which hierarchically refines the quality feature f LP for refinement, dimensionality reduction, and the quality feature f after dimensionality alignment L ' P is as follows:

[0018] f L ' P = Fblock1(f LP )(3);

[0019] S2.2. (1) A feature prediction network is composed of an Encoder, a Decoder, a Discriminator, a codebook, and a codebook entry prediction network Latentformer. First, a high-quality codebook is created by training using the ImageNet database. In a self-supervised learning manner, the Encoder and Decoder of the feature prediction network are adapted to the search and reconstruction of deep features, and the reconstruction image task in self-supervised learning is used to guide the construction of the codebook and the training of the Encoder and Decoder. Then, the codebook prediction and search ability of the codebook entry prediction network Latentformer is trained using image pairs. Denote the input image as x and the Encoder of the feature prediction network as E. Then the encoded feature of the input image x after being encoded by the Encoder is as shown in the formula:

[0020]

[0021] where h, w, and n z respectively represent the height, width, and number of channels of the encoded feature ;

[0022] (2) The corresponding code element is found from the codebook by means of nearest neighbor search, and the quantized feature z q is obtained, where K represents the number of code elements in the codebook, represents the per-pixel feature in the encoded feature :

[0023]

[0024] Then the reconstructed image is as follows:

[0025]

[0026] (3) To enhance the encoding and decoding of the codebook entry features by the feature prediction network Encoder and Decoder, a discriminator Discriminator, denoted as D, is designed. This discriminator consists of 4 convolutional layers, 3 ReLU layers, and two BN layers. It aggregates and compresses the feature information under the action of the convolutional layers, and eliminates the linear bias and reduces the data shift under the combined action of the ReLU layer and the BN layer. At the same time, the adversarial loss L is introduced

[0027] and GAN :

[0028]

[0029] (4) The consistency loss in VQ-VAE is introduced to constrain the update of the codebook and the Encoder and Decoder of the feature prediction network. Here, sg represents freezing, and β is an adjustable parameter set manually. Then the overall function in the first

[0030] stage is as follows:

[0031]

[0032] S2.3. Train a codebook entry prediction network Latentformer based on a nine-layer multi-head attention module using the training results of the first stage. Utilize the information of the encoded features under the action of the multi-head attention module MSAM to predict the corresponding code elements of high-quality prior knowledge and use it to predict the corresponding code elements of high-quality prior knowledge Denote the codebook entries in the codebook as s i ∈R K . Then the loss function in the second stage is:

[0033]

[0034] The codebook-based feature prediction network obtained after training is denoted as P c . Then the high-quality prior knowledge f HP in the inference stage is:

[0035]

[0036] Furthermore, the specific process of S3 is as follows:

[0037] S3.1. Use a post-fusion module Fblock2 containing three convolutional layers to further refine and align the dimensions of the high-quality prior knowledge, facilitating the embedding of the high-quality prior knowledge into the deblurring module and guiding the deblurring process. The refined and aligned high-quality prior knowledge f H ' P is:

[0038] fH ' P =Fblock2(f HP )(11);

[0039] S3.2, embed high-quality prior knowledge into the features extracted by the deblur module encoder, and pass the fused features through the deblur module decoder D d Decode a clear image I HQ , and use the MSE loss as a constraint to bring the deblurred image closer to the clear image in the pixel space domain The distance between them, the deblurring module encoder is E d , the input blurred image is I LQ The embedding recovery and loss function are as follows:

[0040]

[0041] A motion blur removal system for mine-mounted video images includes an image acquisition module, a blind image quality evaluation module, a feature prediction network module and a deblurring module:

[0042] The image acquisition module is composed of a camera and a storage card, which collects and saves the video stream of the airborne device in the mine and transmits it to the terminal for processing;

[0043] The blind image quality assessment module mainly includes a pre-trained image quality assessment model, which is used to receive the video and image transmitted by the image acquisition module and extract quality features from them;

[0044] The feature prediction network module includes a feature prediction network and a pre-fusion module, which is used to receive the quality features extracted by the blind image quality assessment module and predict high-quality prior knowledge;

[0045] The deblurring module includes a deblurring network composed of an encoder and a decoder and a post-fusion module, which is used to receive the high-quality prior knowledge obtained by the feature prediction network module and the video image transmitted by the image acquisition module, integrate and process them and output the deblurred video image, thereby completing the overall construction of the de-motion blurring system.

[0046] The present invention extracts deep features related to image quality in an image by using a blind image quality evaluation module trained for real distortion, and at the same time adopts a codebook-based prediction strategy to predict high-quality prior knowledge. In this process, by reducing the dimension of features, encoding, and searching for the codebook, the prediction difficulty is greatly reduced and the prediction accuracy is improved. Finally, the high-quality prior knowledge is embedded into the deblurring module to guide the deblurring process. By introducing high-quality prior knowledge, the present invention greatly improves the deblurring performance of the model, and as a constraint in the training process of the model, improves the generalization ability of the model, effectively copes with the harsh and complex conditions underground, efficiently processes the jitter blur generated by the camera of the airborne equipment, improves the accuracy of subsequent image processing and analysis, and meets the needs of daily safety production supervision and advanced vision tasks in the mine. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 is the flowchart of the method of the present invention;

[0048] Figure 2 is the structural diagram of the deblurring algorithm of the present invention;

[0049] Figure 3 is the relationship diagram between image quality and image sharpness in the embodiment of the present invention;

[0050] Figure 4 is the comparison diagram of the ability of the basic model and the image quality evaluation model in the embodiment of the present invention to capture blur distortion;

[0051] Figure 5 is the structural diagram of the discriminator of the de-motion blur algorithm for mine airborne video images in the embodiment of the present invention;

[0052] Figure 6 is the Latentformer structural diagram of the de-motion blur algorithm for mine airborne video images in the embodiment of the present invention;

[0053] Figure 7 is the deblurring effect diagram on the GORPO dataset in the embodiment of the present invention;

[0054] Figure 8 is the deblurring effect diagram on the RealJ dataset in the embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0055] The present invention will be further described below with reference to the accompanying drawings.

[0056] As Figure 1 shown, a method for removing motion blur from mine airborne video images includes the following steps:

[0057] S1. Construct a blind reference quality evaluation module based on a convolutional neural network for real scenarios, and extract quality features related to quality in the image by means of the module;

[0058] S2. Construct a feature prediction network module based on a codebook, pre-train a feature prediction network containing a high-quality feature codebook by means of vector coding, and perform prediction coding based on the quality features of the blurred image, so as to obtain high-quality prior knowledge of the image;

[0059] S3. Embed the high-quality prior knowledge obtained in S2 into the encoded features of the deblurring module after dimension alignment, and obtain the restored clear image after decoding by the decoder of the module.

[0060] A de-motion-blur system for mine airborne video images includes an image acquisition module, a blind image quality evaluation module, a feature prediction network module, and a deblurring module:

[0061] The image acquisition module consists of a camera and a storage card, acquires and saves the video stream of the airborne device underground and transmits it to the terminal for processing;

[0062] The blind image quality evaluation module mainly includes a pre-trained image quality evaluation model, which is used to receive the videos and images transmitted by the image acquisition module and extract quality features therefrom;

[0063] The feature prediction network module includes a feature prediction network and a pre-fusion module, which are used to receive the quality features extracted by the blind image quality evaluation module and predict high-quality prior knowledge;

[0064] The deblurring module includes a deblurring network composed of an encoder and a decoder and a post-fusion module, which are used to receive the high-quality prior knowledge obtained by the feature prediction network module and the video images transmitted by the image acquisition module, integrate and process them, and output the de-blurred video images, thereby completing the overall construction of the de-motion-blur system.

[0065] As Figure 2 shown, the deblurring algorithm proposed by the present invention aims to extract information related to image quality and predict high-quality prior information based on a codebook to guide the process of image deblurring. First, use the blind image quality evaluation module to extract the quality features related to image quality in the image; then, adopt a feature prediction strategy based on a codebook to predict high-quality prior knowledge according to the quality features; finally, embed the high-quality prior knowledge into the features encoded by the deblurring model, and decode and reconstruct a clear image.

[0066] (1) Extract high-quality prior knowledge of the image: As Figure 3The figure shows an in-library image taken from the image quality evaluation dataset Koniq. It can be seen that the blurring distortion of the image is an important dimension affecting image quality; for example, Figure 4 As shown, the blind image quality evaluation module captures the feature of the part related to image quality in the image and fits it to a score that conforms to the subjective perception of human vision, that is, MOS, the mean opinion score; therefore, we can capture the feature information related to blurring in the image through the image quality evaluation model as the quality prior.

[0067] Traditional quality evaluation models perform well in dealing with simulated distortions, but are not applicable to real-scene distortions. Moreover, traditional quality evaluation models use manually designed feature extraction, and the quality scores they fit using distortions do not conform to human visual perception. The motion blurring distortion we are targeting is a real-scene distortion, and using a deep learning-based quality evaluation model is more in line with the requirements. It is difficult to find a clear image as a reference for blurred distortion images in the real scene. Therefore, a deep learning-based blind image quality evaluation method needs to be adopted, and it is trained using a real distortion image quality evaluation dataset to enable it to have the ability to extract image quality features. First, a blind image quality evaluation model was built and trained using the Koniq dataset, and the last two layers were removed to obtain the blind image quality evaluation module denoted as M BIQA , which is fine-tuned based on the Resnet50 architecture. The fine-tuning formula is shown below. Here, I represents the input image, and S p represents the quality score predicted by the image quality evaluation model, and S l represents the dataset label score:

[0068]

[0069] The input low-quality image is denoted as I LQ , then the extracted quality feature f LP is:

[0070] f LP = M BIQA (I LQ );

[0071] (2) Predict high-quality prior knowledge based on the codebook: Before predicting high-quality prior knowledge for the features, first refine and reduce the dimension of the quality feature f LP to make it more suitable for the space of feature prediction. Here, we use a fusion module Fblock1 composed of three fully connected layers to complete this operation. The dimension-reduced quality feature f' LP is:

[0072] f L ' P = Fblock1(f LP );

[0073] Adopt a feature prediction strategy based on the quality feature f L ' P Predict high-quality prior knowledge f HP , since the quality evaluation model focuses on the features of severely distorted parts, and the traditional deblurring model has poor restoration effect on severely distorted parts, so we adopt a codebook prediction-based strategy to focus on processing these parts and predict high-quality prior features. The predictor consists of several parts: Encoder, Decoder, codebook, Discriminator, and Latentformer. The advantages of this feature prediction network constructed based on codebook prediction are as follows: (1) Reducing the prediction difficulty and computational complexity by reducing the dimension and predicting the features; (2) Improving the prediction accuracy by constructing a high-quality codebook for lookup and prediction; (3) Adopting a multi-task learning method for training to reduce the training complexity; (4) The training of the predictor and the construction of the codebook only exist in the training stage, and almost no additional computing power consumption occurs in the testing stage.

[0074] During the training process, a two-stage training strategy is adopted: The first stage is the process of constructing the codebook. In this process, Latentformer in the prediction does not participate in the operation. The reconstruction task in self-supervised learning is adopted, and the Encoder, Decoder of the feature prediction network and the codebook are trained based on ImageNet. The nearest neighbor search method is used to utilize the codebook to obtain the quantized feature z after the codebook during the training process q :

[0075]

[0076] Meanwhile, in order to enhance the encoding and decoding of the Encoder and Decoder of the feature prediction network for the codebook entry features, a discriminator Discriminator as shown in Figure 5 is designed and denoted as D, introducing an adversarial loss for the training of the first stage:

[0077]

[0078] Here, x represents the input training image, represents the reconstructed image;

[0079] The loss function of the first stage of training the feature prediction network is as follows:

[0080]

[0081] The idea of VQVAE is to perform feature prediction to find the codebook through nearest neighbor search. However, the problem that this method is prone to is that since the codebook features are discrete features searched according to entries, and the codebook of the prediction network module is established based on high-quality image features, in the face of high-quality prior knowledge, nearest neighbor search can be classified into the correct codebook entries. When facing low-quality image features, due to the fact that the features have undergone various distortions and degradations, it is difficult to classify them into the correct codebook entries through nearest neighbor search. Here we construct a codebook entry prediction module Latentformer based on the multi-head attention module, as Figure 6 shown, and use the deblurring dataset for training to accurately find the codebook entries at a long distance and more accurately predict high-quality prior knowledge. Denote the codebook entries in the codebook as s i ∈R K , and the predicted codebook entry is The image features encoded by Encoder are Then the loss function for the second-stage training of the feature prediction network is:

[0082]

[0083] During the inference process, use the feature prediction network P c Based on the quality feature f' obtained by dimensionality reduction in the previous step LP Predict the high-quality prior feature f HP :

[0084] f HP = P c (f' LP );

[0085] (3) Quality prior knowledge-guided deblurring: There is still a problem of dimensional misalignment in the predicted high-quality prior knowledge. Therefore, we design a post-fusion module Fblock2 to perform dimensional alignment and feature refinement on the high-quality prior knowledge f HP To obtain the high-quality prior knowledge f H ': P :

[0086] f H ' P = Fblock2(f HP );

[0087] After a series of operations, high-quality prior knowledge is obtained. In this process, through the multi-task learning method, the deblurring model also obtains the features of the input image in parallel. Embed the high-quality prior knowledge into the encoded features, and decode and reconstruct through the deblurring model to obtain a clear image, so as to guide the deblurring process. The loss function for embedding reconstruction and training to update parameters is:

[0088]

[0089] In the present invention, the proposed algorithm is respectively used on the GoPro and RealJ datasets to introduce high-quality prior guidance based on codebook prediction for deblurring, and the results are as Figure 7 and Figure 8 shown. It can be seen from Figure 7 and Figure 8 that the algorithm proposed in the present invention significantly improves the deblurring effect.

Claims

1. A method for removing motion blur from airborne video images in a mine, characterized in that, It includes the following steps: S1. Construct a blind image quality evaluation module based on a convolutional neural network for real scenarios, and extract quality-related features in the image by means of the module; S2. Construct a feature prediction module based on a codebook, pre-train a feature prediction network containing a high-quality feature codebook by means of vector coding, and perform prediction coding based on the quality features of the blurred image, so as to obtain high-quality prior knowledge of the image; the process is as follows: Introduce the consistency loss in VQ-VAE to constrain the update of the codebook and the Encoder and Decoder of the feature prediction network, where x is the input image, sg represents freezing, E(x) represents the encoding of the input image x by the Encoder, and z q represents the feature after code element quantization, and L GAN represents the adversarial loss, β is an adjustable parameter set manually, and the overall function in the first stage is as follows: Train a codebook entry prediction network Latentformer based on the training results of the first stage. Under the action of the multi-head attention module MSAM, utilize the information of the encoded features and use it to predict the corresponding code elements of high-quality prior knowledge Let the codebook entry in the codebook be s i ∈R K , then the loss function in the second stage is: The codebook-based feature prediction network obtained after training is denoted as P c , then the high-quality prior knowledge f HP in the inference stage is as follows: f HP = P c (f' LP )(10); S3. Embed the high-quality prior knowledge obtained in S2 into the encoded features of the deblurring module after dimension alignment, and obtain the restored clear image after decoding by the decoder of the module.

2. The method for removing motion blur from airborne video images in a mine according to claim 1, characterized in that, The specific process of S1 is as follows: S1.

1. Fine-tune the image quality evaluation model pre-trained using the ImageNet database with a real distortion image quality evaluation dataset: use Resnet50 as the basic architecture, and use the image quality evaluation dataset Koniq to train and fine-tune the image quality evaluation model. The formula is: In the formula, the input image is denoted as I, and the quality score predicted by the image quality evaluation model is denoted as S p , the dataset label score is denoted as S l , a blind reference quality evaluation model is obtained by updating the parameters in a supervised learning manner and through the L1 loss, and the last two convolutional layers are removed to obtain the blind image quality evaluation module M BIQA ; S1.2 Use the blind image quality evaluation module M BIQA to obtain image quality features. Denote the input low-quality image as I LQ , and the extracted quality feature f LP is expressed as: f LP = M BIQA (I LQ ) (2).

3. The method for removing motion blur of airborne video images in a mine according to claim 2, characterized in that, The specific process of S2 further includes: S2.

1. Send the obtained quality feature f LP Before feeding it into the codebook-based feature prediction network, refine and align the dimensions of the quality feature f LP A pre-fusion module denoted as Fblock1 containing three convolutional layers is adopted to refine and reduce the dimension of the quality feature f LP level by level. The dimension-aligned quality feature f' LP is as follows: f’ LP = Fblock1(f LP ) (3); S2.2, (1) A feature prediction network is composed of an Encoder, a Decoder, a Discriminator, a codebook, and a codebook entry prediction network Latentformer. First, a high-quality codebook is trained and created using the ImageNet database. In a self-supervised learning manner, the Encoder and Decoder of the feature prediction network are adapted to the search and reconstruction of deep features. The reconstruction image task in self-supervised learning is used to guide the construction of the codebook and the training of the Encoder and Decoder. Then, the image pair is used to train the codebook prediction and search ability of the codebook entry prediction network Latentformer. Denote the input image as x and the Encoder of the feature prediction network as E. Then, the encoded feature after the input image x is encoded by the Encoder As shown in the formula: Among them, h, w, and n z respectively represent the height, width, and number of channels of the encoded feature ; (2) Use the nearest neighbor search method to find the corresponding code element from the codebook and obtain the quantized feature z q , where K represents the number of code elements in the codebook, represents the per-pixel feature in the encoded feature : The reconstructed image is as follows: (3) Design the discriminator Discriminator, denoted as D. This discriminator consists of 4 convolutional layers, 3 ReLU layers, and two BN layers. It aggregates and compresses feature information under the action of convolutional layers, eliminates linear biases and reduces data drift under the combined action of ReLU layers and BN layers. At the same time, an adversarial loss L is introduced during training. GAN :

4. The method for removing motion blur of airborne video images in a mine according to claim 1, characterized in that, The specific process of S3 is as follows: S3.

1. Further refine and align the dimensions of the high-quality prior knowledge using a post-fusion module Fblock2 with three convolutional layers, facilitating the embedding of the high-quality prior knowledge into the encoded features of the deblurring module and guiding the deblurring process. The refined high-quality prior knowledge f' after aligning the dimensions HP is as follows: f’ HP = Fblock2(f HP ) (11); S3.2, embed high-quality prior knowledge into the features extracted by the encoder of the deblurring module, and pass the fused features through the decoder D of the deblurring module d Decode a clear image I HQ , and use the MSE loss as a constraint to bring the deblurred image closer to the clear image in the pixel space domain The distance between them, the deblurring module encoder is E d , the input blurred image is I LQ The embedding recovery and loss function are as follows:

Images