Mine airborne video image motion blur removing method and system

Abstract

The invention discloses a motion blur removing method and system for a mine airborne video image, and the method comprises the steps: constructing a blind image quality evaluation module which is based on a convolutional neural network and faces a real scene, and extracting quality features related to the quality in the image through the module; the method comprises the following steps: constructing a codebook-based feature prediction module, pre-training a feature prediction network containing a high-quality feature codebook through a vector coding mode, and carrying out predictive coding based on the quality features of a blurred image so as to obtain high-quality priori knowledge of the image; obtained high-quality priori knowledge is embedded into coding features of the deblurring module after being subjected to dimension alignment, and a restored clear image is obtained after decoding is conducted through a decoder. The system comprises an image acquisition module, a blind image quality evaluation module, a feature prediction module and a deblurring module. According to the method, image video loss can be quickly processed, the method effectively adapts to the environment under the mine, the subsequent image processing and analysis accuracy is improved, and the safety and efficiency of mining area operation are improved.

Description

Technical Field

[0001] The invention relates to a method and system for de-motion blurring of a mine machine-borne video image, belonging to the technical field of image de-blurring. Background Art

[0002] In underground transportation in coal mines, the onboard video of facilities such as electric locomotives, monorail cranes and trackless rubber-wheeled vehicles often causes screen shaking due to equipment vibration and movement, resulting in motion blur. This blurring phenomenon makes it difficult to present video images clearly, affecting subsequent image processing and analysis. Deblurring technology has therefore become crucial, and it aims to restore the clarity of the image through various algorithms. By eliminating motion blur, the video quality can be improved, providing more accurate visual data for subsequent detection, recognition and monitoring. This not only helps to improve the safety and efficiency of mining operations, but also optimizes the subsequent data processing process. Traditional deblurring methods abstract the blur problem into a process of convolution between an image and a blur kernel, and estimate the blur kernel by manually designing and extracting features, so it is only applicable to uniform blur. However, most blurs in the real world are non-uniform, so traditional deblurring methods are less effective in dealing with real image blurs.

[0003] In recent years, with the continuous improvement of deep learning theory and the widespread application of convolutional neural networks, deep learning-based methods have become mainstream. By using blurry-clear image pairs to train models, deep learning-based methods can automatically learn deep features in the image blurring process through convolution kernels, avoiding the deviations generated in the blur kernel estimation process. However, the image deblurring process is an ill-posed process. End-to-end mapping deep learning-based methods lack the guidance of prior knowledge, and the perception of blurred parts is heavily dependent on the training of a large amount of data. It is impossible to select key parts for processing, and the processing effect is not good for severely blurred parts.

[0004] Image deblurring aims to restore a clear image from a blurred image, and quality assessment algorithms aim to evaluate the quality of an image. The quality assessment model can extract the features in the image that are most relevant to quality. However, in actual deblurring, only the features of the blurred image can be obtained, and the corresponding high-quality image features need to be further obtained. Therefore, how to predict clear features from blurred depth features and effectively handle severely blurred areas has become an urgent problem to be solved in the deblurring task. Summary of the invention

[0005] The object of the present invention is to provide a method and system for de-motion blurring of mine-mounted video images, which can quickly process image and video stream distortion, effectively adapt to the underground mine environment, improve the accuracy of subsequent image processing and analysis, and improve the safety and efficiency of mining operations.

[0006] In order to achieve the above object, the present invention provides a method for removing motion blur of a mine machine-borne video image, comprising the following steps:

[0007] S1. Construct a blind image quality assessment module for real scenes based on convolutional neural networks, and extract quality features related to quality in images with the help of the module;

[0008] S2. Construct a feature prediction module based on a codebook, pre-train a feature prediction network containing a high-quality feature codebook by means of vector coding, and perform predictive coding based on the quality features of the blurred image, thereby obtaining high-quality prior knowledge of the image;

[0009] S3, embed the high-quality prior knowledge obtained in S2 into the deblurring module encoding features after dimension alignment, and obtain the restored clear image after decoding by the module decoder.

[0010] Furthermore, the specific process of S1 is as follows:

[0011] S1.1. Fine-tune the image quality assessment model pre-trained with the ImageNet database using a real distorted image quality assessment dataset: Resnet50 is used as the basic architecture, and the image quality assessment dataset Koniq is used to train and fine-tune the image quality assessment model. The formula is:

[0012]

[0013] In the formula, the input image is denoted as I, and the quality score predicted by the image quality assessment model is denoted as S p , the dataset label score is recorded as S l , the blind image quality assessment model is obtained by updating the parameters through L1 loss in a supervised learning manner. After removing the last two convolutional layers, the blind image quality assessment module M is obtained. BIQA ;

[0014] S1.2 Using the blind image quality assessment module M BIQA Get the image quality features and record the input low-quality image as I LQ , the extracted quality feature f LP It is expressed as:

[0015] f LP =M BIQA (I LQ )(2).

[0016] Furthermore, the specific process of S2 is as follows:

[0017] S2.1、The quality characteristics f to be obtained LP Before being sent to the codebook-based feature prediction network, the quality feature fLP To refine and align the dimensions, a front fusion module consisting of three convolutional layers is used, denoted as Fblock1, and the quality features f are fused layer by layer. LP The quality feature f after refinement, dimensionality reduction and dimension alignment L ' P for:

[0018] f L ' P =Fblock1(f LP )(3);

[0019] S2.2, (1) The feature prediction network is composed of Encoder, Decoder, Discriminator, codebook and codebook entry prediction network Latentformer. First, the ImageNet database is used to train and create a high-quality codebook. The Encoder and Decoder of the feature prediction network are adapted to the search and reconstruction of deep features by self-supervised learning. The image reconstruction task in self-supervised learning is used to guide the construction of the codebook, the training of the Encoder and Decoder, and then the image is used to train the codebook entry prediction network Latentformer's codebook prediction search ability. Let the input image be x, and the Encoder of the feature prediction network be E. Then the encoded feature of the input image x after being encoded by the Encoder is As shown in the formula:

[0020]

[0021] Among them, h, w, n z Represents the encoding features The height, width and number of channels;

[0022] (2) Using the nearest neighbor search method to find the codebook Find the corresponding code element and get the quantized feature z q , where K represents the number of code elements in the codebook, Represents the encoding feature Pixel-by-pixel features in:

[0023]

[0024] The reconstructed image for:

[0025]

[0026] (3) In order to enhance the encoding and decoding of the codebook entry features by the feature prediction network Encoder and Decoder, a discriminator is designed, denoted as D. The discriminator consists of 4 convolutional layers, 3 ReLU layers and two BN layers. The convolutional layers aggregate and compress feature information. The ReLU layers and BN layers work together to eliminate linear deviations and reduce data offsets.

[0027] The adversarial loss L is introduced GAN :

[0028]

[0029] (4) The consistency loss in VQ-VAE is introduced to constrain the update of the codebook and the Encoder and Decoder of the feature prediction network, where sg represents freezing and β is an adjustable parameter set manually.

[0030] The overall function of the stage is as follows:

[0031]

[0032] S2.3. Based on the training results of the first stage, a codebook entry prediction network Latentformer based on a 9-layer multi-head attention module is trained. The encoded features are used under the action of the multi-head attention module MSAM. information and used to predict the corresponding codewords of high-quality prior knowledge The codebook entry in the codebook is s i ∈R K , then the loss function of the second stage is:

[0033]

[0034] The codebook-based feature prediction network obtained after training is denoted as P c , then the high-quality prior knowledge f in the reasoning stage HP for:

[0035]

[0036] Furthermore, the specific process of S3 is as follows:

[0037] S3.1, a post-fusion module Fblock2 containing three convolutional layers is used to further refine and align the dimensions of high-quality prior knowledge, so as to facilitate the embedding of high-quality prior knowledge into the deblurring module and guide the deblurring process, and refine the high-quality prior knowledge after the alignment dimension f H ' P for:

[0038] fH ' P =Fblock2(f HP )(11);

[0039] S3.2, embed high-quality prior knowledge into the features extracted by the deblurring module encoder, and pass the fused features through the deblurring module decoder D d Decode a clear image I HQ , and use the MSE loss as a constraint to bring the deblurred image closer to the clear image in the pixel space domain The distance between them, the deblurring module encoder is E d , the input blurred image is I LQ The embedding recovery and loss function are as follows:

[0040]

[0041] A motion blur removal system for mine-mounted video images includes an image acquisition module, a blind image quality evaluation module, a feature prediction network module and a deblurring module:

[0042] The image acquisition module is composed of a camera and a storage card, which collects and saves the video stream of the airborne device in the mine and transmits it to the terminal for processing;

[0043] The blind image quality assessment module mainly includes a pre-trained image quality assessment model, which is used to receive the video and image transmitted by the image acquisition module and extract quality features from them;

[0044] The feature prediction network module includes a feature prediction network and a pre-fusion module, which is used to receive the quality features extracted by the blind image quality assessment module and predict high-quality prior knowledge;

[0045] The deblurring module includes a deblurring network composed of an encoder and a decoder and a post-fusion module, which is used to receive the high-quality prior knowledge obtained by the feature prediction network module and the video image transmitted by the image acquisition module, integrate and process them and output the deblurred video image, thereby completing the overall construction of the de-motion blurring system.

[0046] The present invention extracts quality-related deep features in the image by utilizing a blind image quality evaluation module trained for real distortion, and adopts a codebook-based prediction strategy to predict high-quality prior knowledge. In this process, the prediction difficulty is greatly reduced and the prediction accuracy is improved by reducing the dimension of the features, encoding, and searching the codebook. Finally, the high-quality prior knowledge is embedded in the deblurring module to guide the deblurring process. The present invention greatly improves the deblurring performance of the model by introducing high-quality prior knowledge, and uses it as a constraint in the training process of the model to improve the generalization ability of the model, effectively cope with the harsh and complex conditions in the mine, efficiently handle the jitter blur generated by the camera of the airborne equipment, improve the accuracy of subsequent image processing and analysis, and meet the needs of daily safety production supervision in mines and advanced visual tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 It is a work flow chart of the method of the present invention;

[0048] Figure 2 is a structural diagram of the deblurring algorithm of the present invention;

[0049] Figure 3 is a graph showing the relationship between image quality and image clarity in an embodiment of the present invention;

[0050] Figure 4 is a comparison diagram of the blur and distortion capturing capabilities of the basic model and the image quality assessment model in an embodiment of the present invention;

[0051] Figure 5 It is a structural diagram of a de-motion blur algorithm discriminator for a mine machine-borne video image in an embodiment of the present invention;

[0052] Figure 6 It is a structure diagram of the Latentformer algorithm for de-motion blurring of mine machine-borne video images in an embodiment of the present invention;

[0053] Figure 7 is a deblurring effect diagram on the GORPO dataset in an embodiment of the present invention;

[0054] Figure 8 It is a deblurring effect diagram on the RealJ dataset in an embodiment of the present invention. DETAILED DESCRIPTION

[0055] The present invention will be further described below in conjunction with the accompanying drawings.

[0056] like Figure 1 As shown, a method for removing motion blur of a mine machine-borne video image comprises the following steps:

[0057] S1. Construct a blind reference quality evaluation module for real scenes based on convolutional neural networks, and extract quality features related to quality in images with the help of the module;

[0058] S2. Construct a feature prediction network module based on a codebook, pre-train a feature prediction network containing a high-quality feature codebook by means of vector coding, and perform predictive coding based on the quality features of the blurred image, thereby obtaining high-quality prior knowledge of the image;

[0059] S3, embed the high-quality prior knowledge obtained in S2 into the deblurring module encoding features after dimension alignment, and obtain the restored clear image after decoding by the module decoder.

[0060] A motion blur removal system for mine-mounted video images includes an image acquisition module, a blind image quality evaluation module, a feature prediction network module and a deblurring module:

[0061] The image acquisition module is composed of a camera and a storage card, which collects and saves the video stream of the airborne device in the mine and transmits it to the terminal for processing;

[0062] The blind image quality assessment module mainly includes a pre-trained image quality assessment model, which is used to receive the video and image transmitted by the image acquisition module and extract quality features from them;

[0063] The feature prediction network module includes a feature prediction network and a pre-fusion module, which is used to receive the quality features extracted by the blind image quality assessment module and predict high-quality prior knowledge;

[0064] The deblurring module includes a deblurring network composed of an encoder and a decoder and a post-fusion module, which is used to receive the high-quality prior knowledge obtained by the feature prediction network module and the video image transmitted by the image acquisition module, integrate and process them and output the deblurred video image, thereby completing the overall construction of the de-motion blurring system.

[0065] like Figure 2 As shown in the figure, the deblurring algorithm proposed in the present invention aims to extract information related to image quality and obtain high-quality prior information based on codebook prediction, so as to guide the image deblurring process. First, the blind image quality evaluation module is used to extract quality features related to image quality in the image; then, the feature prediction strategy based on the codebook is used to predict high-quality prior knowledge based on the quality features; finally, the high-quality prior knowledge is embedded in the features encoded by the deblurring model, and a clear image is reconstructed by decoding.

[0066] (1) Extracting prior knowledge of image quality: Figure 3The following are images from the Koniq image quality assessment dataset. It can be seen that image blur and distortion are an important dimension affecting image quality. Figure 4 The blind image quality assessment module shown captures the image quality-related part features in the image and fits them to a score that conforms to the subjective visual perception of human beings, namely MOS, the mean subjective opinion score; therefore, we can capture the feature information related to blur in the image as a quality prior through the image quality assessment model.

[0067] Traditional quality assessment models work well for simulated distortion, but are not suitable for real scene distortion. In addition, traditional quality assessment models use manually designed features to extract features, and the quality scores based on distortion fitting do not conform to human visual perception. The motion blur distortion we are targeting is real scene distortion, and the quality assessment model based on deep learning is more in line with the needs. It is difficult to find a clear image as a reference for blurred and distorted images in real scenes. Therefore, it is necessary to adopt a blind image quality assessment method based on deep learning, and use the real distorted image quality assessment dataset for training so that it has the ability to extract image quality features. First, a blind image quality assessment model was built and trained using the Koniq dataset. The blind image quality assessment module obtained by removing the last two layers is denoted as M. BIQA This module is fine-tuned based on the Resnet50 architecture. The fine-tuning formula is as follows: Here, I represents the input image, S p represents the quality score predicted by the image quality assessment model, S l Represents the dataset label score:

[0068]

[0069] The input low-quality image is denoted as I LQ , then the extracted quality feature f LP for:

[0070] f LP =M BIQA (I LQ );

[0071] (2) Predicting high-quality prior knowledge based on the codebook: Before predicting high-quality prior knowledge based on features, first predict the quality feature f LP Refine and reduce the dimension to make it more suitable for feature prediction space. Here we use three layers of fully connected layers to form a fusion module Fblock1 to complete this operation. The quality feature f' after dimensionality reduction LP for:

[0072] f L ' P =Fblock1(f LP );

[0073] Adopt feature prediction strategy based on quality feature f L ' P Predict high-quality prior knowledge f HP , since the quality evaluation model focuses on the features of severely distorted parts, and the traditional deblurring model has poor recovery effect on severely distorted parts, we adopted a codebook prediction-based strategy to focus on these parts and predict high-quality prior features. The predictor consists of Encoder, Decoder, codebook, Discriminator, and Latentformer. The advantages of this feature prediction network based on codebook prediction are: (1) It reduces the difficulty and amount of calculation by reducing the dimension and predicting the features; (2) It improves the prediction accuracy by building a high-quality codebook for search and prediction; (3) It uses multi-task learning for training, which reduces the complexity of training; (4) The training of the predictor and the construction of the codebook only exist in the training phase, and almost no additional computing power is consumed in the test phase.

[0074] During the training process, a two-stage training strategy is adopted: the first stage is the process of building a codebook. In this process, the Latentformer in the prediction does not participate in the calculation. The reconstruction task in self-supervised learning is used to train the Encoder and Decoder of the feature prediction network based on ImageNet, and the codebook is constructed. The codebook is used in the nearest neighbor search method to obtain the quantized feature z after the codebook in the training process. q :

[0075]

[0076] At the same time, in order to enhance the encoding and decoding of codebook entry features by the feature prediction network Encoder and Decoder, the following Figure 5 The discriminator shown, denoted as D, introduces an adversarial loss for the first stage of training:

[0077]

[0078] Here x represents the input training image, represents the reconstructed image;

[0079] The loss function for the first stage of training the feature prediction network is as follows:

[0080]

[0081] The idea of ​​VQVAE is to perform feature prediction and codebook search through nearest neighbor search. However, the problem that this method is prone to is that since the codebook feature is a discrete feature that is searched based on the entry, the codebook of the prediction network module is based on high-quality image features. Therefore, in the face of high-quality prior knowledge, the nearest neighbor search can be classified into the correct codebook entry. When faced with low-quality image features, since the features have been degraded through various distortions, it is difficult to classify them into the correct codebook entry through the nearest neighbor search. Here we construct a codebook entry prediction module Latentformer based on a multi-head attention module, such as Figure 6 As shown in the figure, the deblurred dataset is used for training to accurately find the codebook entries at a long distance and more accurately predict the high-quality prior knowledge. The codebook entry in the codebook is s i ∈R K , the predicted codebook entry is The image features encoded by Encoder are Then the loss function of the second stage training of the feature prediction network is:

[0082]

[0083] During the inference process, the feature prediction network P is used c Based on the quality feature f' obtained in the previous step of dimensionality reduction LP Predict high-quality prior features f HP :

[0084] f HP =P c (f' LP );

[0085] (3) Quality prior knowledge guided deblurring: The predicted high-quality prior knowledge still has the problem of dimensional misalignment, so we designed a post-fusion module Fblock2 to fuzzify the high-quality prior knowledge f. HP Perform dimension alignment and feature refinement to obtain high-quality prior knowledge f H ' P :

[0086] f H ' P =Fblock2(f HP );

[0087] After a series of operations, high-quality prior knowledge is obtained. In this process, the deblurring model also obtains the features of the input image in parallel through multi-task learning. The high-quality prior knowledge is embedded into the encoded features, and a clear image is obtained by decoding and reconstructing the deblurring model to guide the deblurring process. The loss function of embedding, reconstruction, training and updating parameters is:

[0088]

[0089] The proposed algorithm is used on two datasets, GoPro and RealJ, to introduce high-quality prior-guided deblurring based on codebook prediction. The results are shown in Figure 2. Figure 7 and Figure 8 As shown. Figure 7 and Figure 8 It can be seen that the algorithm proposed in the present invention significantly improves the deblurring effect.

Claims

1. A method for de-motion blurring of mine-mounted video images, characterized in that: The steps include: S1. Construct a blind image quality assessment module for real scenes based on convolutional neural networks, and extract quality features related to quality in images with the help of the module; S2. Construct a feature prediction module based on a codebook, pre-train a feature prediction network containing a high-quality feature codebook by means of vector coding, and perform predictive coding based on the quality features of the blurred image, thereby obtaining high-quality prior knowledge of the image; S3, embed the high-quality prior knowledge obtained in S2 into the deblurring module encoding features after dimension alignment, After being decoded by the module's decoder, a restored clear image is obtained.

2. The method for de-motion blurring of mine machine-borne video images according to claim 1, characterized in that: The specific process of S1 is as follows: S1.

1. Fine-tune the image quality assessment model pre-trained with the ImageNet database using a real distorted image quality assessment dataset: Resnet50 is used as the basic architecture, and the image quality assessment dataset Koniq is used to train and fine-tune the image quality assessment model. The formula is: In the formula, the input image is denoted as I, and the quality score predicted by the image quality assessment model is denoted as S p , the dataset label score is recorded as S l , and update the parameters through L1 loss in a supervised learning manner to obtain a blind reference quality assessment model. The last two convolutional layers are removed to obtain a blind image quality assessment module M BIQA ; S1.2 Using the blind image quality assessment module M BIQA Get the image quality features and record the input low-quality image as I LQ , the extracted quality feature f LP It is expressed as: f LP =M BIQA (I LQ )(2)。 3. The method for de-motion blurring of mine machine-borne video images according to claim 2, characterized in that: The specific process of S2 is as follows: S2.1、The quality characteristics f to be obtained LP Before being sent to the codebook-based feature prediction network, the quality feature f LP To refine and align the dimensions, a front fusion module consisting of three convolutional layers is used, denoted as Fblock1, and the quality features f are fused layer by layer. LP The quality feature f after refinement, dimensionality reduction and dimension alignment L ' P for: f L ' P =Fblock1(f LP )(3); S2.2, (1) The feature prediction network is composed of Encoder, Decoder, Discriminator, codebook and codebook entry prediction network Latentformer. First, the ImageNet database is used to train and create a high-quality codebook. The Encoder and Decoder of the feature prediction network are adapted to the search and reconstruction of deep features by self-supervised learning. The image reconstruction task in self-supervised learning is used to guide the construction of the codebook, the training of the Encoder and Decoder, and then the image is used to train the codebook entry prediction network Latentformer's codebook prediction search ability. Let the input image be x, and the Encoder of the feature prediction network be E. Then the encoded feature of the input image x after being encoded by the Encoder is As shown in the formula: Among them, h, w, n z Represents the encoding features The height, width and number of channels; (2) Using the nearest neighbor search method to find the codebook Find the corresponding code element and get the quantized feature z q , where K represents the number of code elements in the codebook, Represents the encoding feature Pixel-by-pixel features in: The reconstructed image for: (3) Design the discriminator, denoted as D. The discriminator consists of 4 convolutional layers, 3 ReLU layers, and two BN layers. The convolutional layers aggregate and compress feature information. The ReLU and BN layers work together to eliminate linear deviations and reduce data offsets. At the same time, the adversarial loss L is introduced during training. GAN : (4) The consistency loss in VQ-VAE is introduced to constrain the update of the codebook and the Encoder and Decoder of the feature prediction network, where sg represents freezing and β is an adjustable parameter set manually. The overall function of the first stage is as follows: S2.

3. Based on the training results of the first stage, a codebook entry prediction network Latentformer based on a 9-layer multi-head attention module is trained. Under the action of the multi-head attention module MSAM, the encoding features are used information and used to predict the corresponding codewords of high-quality prior knowledge The codebook entry in the codebook is s i ∈R K , then the loss function of the second stage is: The codebook-based feature prediction network obtained after training is denoted as P c , then the high-quality prior knowledge f in the reasoning stage HP for: f HP =P c (f' LP )(10)。 4. The method for de-motion blurring of mine machine-borne video images according to claim 3, characterized in that: The specific process of S3 is as follows: S3.1, a post-fusion module Fblock2 containing three convolutional layers is used to further refine and align the dimensions of high-quality prior knowledge, so as to facilitate the embedding of high-quality prior knowledge into the encoding features of the deblurring module and guide the deblurring process, and refine the high-quality prior knowledge after the alignment dimension f H ' P for: f H ' P =Fblock2(f HP )(11); S3.2, embed high-quality prior knowledge into the features extracted by the encoder of the deblurring module, and pass the fused features through the decoder D of the deblurring module d Decode a clear image I HQ , and use the MSE loss as a constraint to bring the deblurred image closer to the clear image in the pixel space domain The distance between them, the deblurring module encoder is E d , the input blurred image is I LQ The embedding recovery and loss function are as follows:

5. A system for removing motion blur of mine-mounted video images, characterized in that: It includes image acquisition module, blind image quality assessment module, feature prediction network module and deblurring module: The image acquisition module is composed of a camera and a storage card, which collects and saves the video stream of the airborne device in the mine and transmits it to the terminal for processing; The blind image quality assessment module mainly includes a pre-trained image quality assessment model, which is used to receive the video and image transmitted by the image acquisition module and extract quality features from them; The feature prediction network module includes a feature prediction network and a pre-fusion module, which is used to receive the quality features extracted by the blind image quality assessment module and predict high-quality prior knowledge; The deblurring module includes a deblurring network composed of an encoder and a decoder and a post-fusion module, which is used to receive the high-quality prior knowledge obtained by the feature prediction network module and the video image transmitted by the image acquisition module, integrate and process them and output the deblurred video image, thereby completing the overall construction of the de-motion blurring system.

Images