Downhole image enhancement method based on space-time reconstruction

Abstract

The invention provides an underground image enhancement method based on space-time reconstruction, and relates to the technical field of image processing, and the underground image enhancement method based on space-time reconstruction comprises the steps: obtaining underground to-be-processed image data, and carrying out the coding remodeling of the to-be-processed image data based on an encoder, and obtaining first coding data; obtaining a first parameter matrix and a second parameter matrix, and determining an adaptive position coding matrix according to the first parameter matrix and the second parameter matrix; based on the adaptive position coding matrix and the first coding data, reasoning to obtain second coding data; and reasoning an enhanced target image according to the first coded data and the second coded data, thereby solving the technical problem of poor definition of the monitoring image in the coal mine in the prior art.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a method for downhole image enhancement based on spatiotemporal reconstruction. Background Art

[0002] In mine safety production, the monitoring system has become a core tool to ensure the smooth progress of production and the safety of personnel. The clarity of underground monitoring video directly affects the identification of safety hazards and the effectiveness of equipment monitoring; however, the complex environmental conditions inside the mine, such as coal dust, muddy water, fog, and strong direct light underground, often cause dirt on the surface of the camera lens, which greatly reduces the quality of the monitoring video. Problems such as blurred video, occlusion, and reflection occur frequently, seriously hindering the normal operation of the monitoring system; key information such as the operating status of underground equipment and personnel activities may be obscured or distorted, and cannot provide effective decision-making support for production management. Therefore, how to improve video clarity and remove the impact of water vapor, dust and dirt on the monitoring screen in harsh mine environments has become one of the key issues in mine safety management.

[0003] With the development of digital and intelligent technologies in mines, the clarity requirements of surveillance videos are constantly increasing. Although traditional image enhancement methods, such as brightness adjustment, contrast adjustment, and denoising filtering, can improve video quality to a certain extent, the effects of these methods are usually limited to static image processing. It is difficult for decontamination algorithms to distinguish between pollutants and actual targets, and the decontamination effect is poor. Summary of the invention

[0004] The present application aims to solve one of the technical problems in the related art at least to some extent.

[0005] To this end, the first purpose of the present application is to propose a downhole image enhancement method based on spatiotemporal reconstruction to achieve better image enhancement.

[0006] The second objective of the present application is to propose a downhole image enhancement device based on spatiotemporal reconstruction.

[0007] The third objective of the present application is to provide an electronic device.

[0008] A fourth objective of the present application is to provide a computer-readable storage medium.

[0009] A fifth object of the present application is to provide a computer program product.

[0010] To achieve the above-mentioned purpose, the first embodiment of the present application proposes a downhole image enhancement method based on spatiotemporal reconstruction, comprising:

[0011] Acquire downhole image data to be processed, and encode and reshape the image data to be processed based on an encoder to obtain first encoded data;

[0012] Acquire a first parameter matrix and a second parameter matrix, and determine an adaptive position coding matrix according to the first parameter matrix and the second parameter matrix;

[0013] Inferring second coded data based on the adaptive position coding matrix and the first coded data;

[0014] An enhanced target image is inferred based on the first encoded data and the second encoded data.

[0015] To achieve the above-mentioned purpose, the second embodiment of the present application proposes a downhole image enhancement device based on spatiotemporal reconstruction, comprising:

[0016] A first acquisition module is used to acquire downhole image data to be processed, and encode and reshape the image data to be processed based on an encoder to obtain first encoded data;

[0017] A second acquisition module, used to acquire a first parameter matrix and a second parameter matrix, and determine an adaptive position coding matrix according to the first parameter matrix and the second parameter matrix;

[0018] A third acquisition module, configured to infer second coded data based on the adaptive position coding matrix and the first coded data;

[0019] An image enhancement module is used to infer an enhanced target image based on the first encoded data and the second encoded data.

[0020] To achieve the above-mentioned purpose, the third aspect of the present application provides an electronic device, including: a processor, and a memory communicatively connected to the processor;

[0021] The memory stores computer-executable instructions;

[0022] The processor executes the computer-executable instructions stored in the memory to implement the method described in the embodiment of the first aspect.

[0023] To achieve the above-mentioned purpose, the fourth aspect embodiment of the present application proposes a computer-readable storage medium, in which computer-readable storage medium is stored computer execution instructions, and when the computer execution instructions are executed by a processor, they are used to implement the method described in the first aspect embodiment.

[0024] To achieve the above-mentioned purpose, the fifth aspect of the present application proposes a computer program product, including a computer program, which implements the method described in the first aspect when executed by a processor.

[0025] The present application provides an underground image enhancement method based on spatiotemporal reconstruction, which encodes and reshapes the underground image data to be processed to obtain first encoded data, obtains first parameter matrices and second parameter matrices, and further determines adaptive position encoding based on the first parameter matrix and the second parameter matrix, thereby learning the feature information of each position in the image data, and obtains second encoded data based on the adaptive position encoding matrix and the first encoded data. The second encoded data incorporates the feature information of each position, thereby smoothly distinguishing the edge position of the image and reducing the information loss caused by the edge effect. Fusion decoding is performed based on the first encoded data and the second encoded data, thereby obtaining an enhanced target image, which solves the problem of poor clarity of underground monitoring images in coal mines and ensures safe management of mine production.

[0026] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0028] Figure 1 A schematic diagram of a flow chart of a downhole image enhancement method based on spatiotemporal reconstruction provided in an embodiment of the present application;

[0029] Figure 2 A schematic flow chart of another downhole image enhancement method based on spatiotemporal reconstruction provided in an embodiment of the present application;

[0030] Figure 3 A schematic diagram of a process of spatiotemporal attention based on long-view convolution provided in an embodiment of the present application;

[0031] Figure 4 A schematic diagram of a process of channel dilation convolution provided in an embodiment of the present application;

[0032] Figure 5 A logic flow chart of a downhole image enhancement method based on spatiotemporal reconstruction provided in an embodiment of the present application;

[0033] Figure 5A A comparison diagram of image enhancement effects provided by an embodiment of the present application;

[0034] Figure 5B Another comparison diagram of image enhancement effects provided by an embodiment of the present application;

[0035] Figure 6A schematic structural diagram of a downhole image enhancement device based on spatiotemporal reconstruction provided in an embodiment of the present application. DETAILED DESCRIPTION

[0036] Embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.

[0037] This application is applicable to the scene where the camera is used for a long time in a mine environment and is affected by environmental factors such as dust, smoke, and lighting conditions, resulting in blurred edges of target objects in the video and reduced image quality, so as to improve the clarity of video images and remove the influence of water vapor, dust and dirt on the monitoring screen in a harsh mine environment. The following describes an underground image enhancement method based on spatiotemporal reconstruction in an embodiment of the present application with reference to the accompanying drawings.

[0038] Figure 1 A schematic flow chart of a downhole image enhancement method based on spatiotemporal reconstruction provided in an embodiment of the present application.

[0039] like Figure 1 As shown, the method comprises the following steps:

[0040] S101, obtaining underground image data to be processed, and encoding and reshaping the image data to be processed based on an encoder to obtain first encoded data.

[0041] In some implementations, a monitoring system will be deployed underground for real-time monitoring during the mine's safe production process to ensure smooth production and personnel safety. The monitoring system can obtain underground image data to be processed in real time. The image data to be processed can be video stream data, which is image data with poor clarity that requires image enhancement.

[0042] The encoder can be used to capture the spatial features of the image data to be processed and expand the time dimension; in this embodiment, the encoder can separate the unobstructed static background information in the surveillance video frame to obtain the first encoded data; optionally, the process of the encoder encoding and reshaping the image data to be processed to obtain the first encoded data can be expressed as:

[0043]

[0044] in, is the first encoded data after encoding, k is the convolution kernel size; t in is the number of input images; b is the number of batches; h and w are the image dimensions; c in is the input image channel; c hidis the channel of the hidden layer; stride is the convolution kernel step size, which is 2 in this embodiment, and can play a role in reducing the resolution; Conv2d is a two-dimensional convolution operation.

[0045] S102, obtaining a first parameter matrix and a second parameter matrix, and determining an adaptive position coding matrix according to the first parameter matrix and the second parameter matrix.

[0046] In some implementations, the first parameter matrix and the second parameter matrix may be parameter matrices generated during the model training process. For example, the first parameter matrix is ​​a matrix for learning features at different positions, and the second parameter matrix is ​​a matrix for learning the effects of positions on different features. That is, in the process of training the model based on the training samples, the evolutionary features of different positions of the training samples and the effects of positions on different features are learned, so that the corresponding first parameter matrix and second parameter matrix are obtained after the model training is completed.

[0047] Optionally, the product of the first parameter matrix and the second parameter matrix may be calculated, and the multiplication result may be used as an adaptive position coding matrix, thereby fusing the position features learned by the first parameter matrix and the second parameter matrix.

[0048] S103, inferring and obtaining second coded data based on the adaptive position coding matrix and the first coded data.

[0049] Optionally, the first encoded data can be encoded again based on the adaptive position coding matrix, for example, the adaptive position coding matrix is ​​added to the first encoded data to fuse the position coding information in the first encoded data to obtain the second encoded data; or the result of adding the first encoded data to the adaptive position coding matrix is ​​again subjected to attention mechanism analysis or convolution processing to obtain the second encoded data.

[0050] S104: Inferring an enhanced target image according to the first coded data and the second coded data.

[0051] In some implementations, the first encoded data and the second encoded data are fused, for example, the first encoded data and the second encoded data are added to obtain fused encoded data; the fused encoded data is decoded to obtain an enhanced target image.

[0052] Optionally, the decoded image may be subjected to image enhancement processing, such as denoising and filtering processing, to obtain a target image with better quality and clarity.

[0053] In this embodiment, the first coded data is obtained by encoding and reshaping the underground image data to be processed, the first parameter matrix and the second parameter matrix are obtained, and the adaptive position coding is further determined according to the first parameter matrix and the second parameter matrix, so as to learn the characteristic information of each position in the image data, and obtain the second coded data according to the adaptive position coding matrix and the first coded data. The second coded data integrates the characteristic information of each position, so as to smoothly distinguish the edge position of the image and reduce the information loss caused by the edge effect. Fusion decoding is performed based on the first coded data and the second coded data to obtain an enhanced target image, thereby improving the output quality of the underground monitoring image, optimizing the decontamination effect of the image, and performing underground environmental monitoring and analysis based on the monitoring image with better decontamination effect to ensure the safe management of mine production.

[0054] Figure 2 The flowchart of another downhole image enhancement method based on spatiotemporal reconstruction provided by the embodiment of the present application is shown in FIG. Figure 2 As shown, the method comprises the following steps:

[0055] S201, obtaining underground image data to be processed, and encoding and reshaping the image data to be processed based on an encoder to obtain first encoded data.

[0056] In some implementations, first video data underground can be obtained, and the first video data can be collected by a monitoring system; the first video data can be further cropped and reduced in resolution to obtain second video data, so as to improve the processing efficiency of the first video data; the second video data can be normalized and converted into a floating point format to obtain image data to be processed, so as to improve the processing effect of the image data to be processed.

[0057] Optionally, a pre-trained image processing model can be determined, which image processing model includes at least a coding layer, a processing layer and a decoding layer; the processing layer may include one or more spatiotemporal compilers, and after the image data to be processed is input into the image processing model, the encoder based on the coding layer encodes and reshapes the image data to be processed, thereby obtaining first encoded data.

[0058] In the embodiment of the present application, the implementation method of step S201 can be implemented by any of the methods in the embodiments of the present disclosure, which is not limited here and will not be repeated.

[0059] S202, obtaining a first parameter matrix and a second parameter matrix.

[0060] Optionally, a pre-trained image processing model is obtained; a first parameter matrix is ​​generated according to the features of different pixel positions in the two-dimensional plane in the pre-trained image processing model; a second parameter matrix is ​​generated according to the differences in the influence of absolute positions on different features in the pre-trained image processing model. That is, in the process of training the image processing model, the evolution features of different positions in the two-dimensional plane are learned from the training samples to generate the first parameter matrix; at the same time, the second parameter matrix is ​​generated by learning the differences in the influence of absolute positions on different features. After the image processing model training is completed, the first parameter matrix and the second parameter matrix are obtained from the image processing model.

[0061] S203, multiplying the second parameter matrix by the first parameter matrix to obtain an adaptive position coding matrix.

[0062] In some implementations, the size of the first parameter matrix can be 1*N, the size of the second parameter matrix can be M*1, and the dimension of the adaptive position coding matrix after multiplying the second parameter matrix by the first parameter matrix is ​​M*N, which is the same as the size of the image data to be processed.

[0063] S204, encoding the first encoded data according to the adaptive position encoding matrix to obtain third encoded data.

[0064] Optionally, the sum of the adaptive position coding matrix and the first coding data can be calculated to obtain third coding data; the third coding data can be calculated and acquired in the spatiotemporal compiler of the processing layer, and the adaptive position coding matrix can enhance the adaptability to spatial heterogeneity such as edge effects after being added to the first coding data.

[0065] It can be understood that the adaptive position coding matrix has more advantages than the existing spatiotemporal reasoning, and the existing spatiotemporal reasoning process can be expressed as:

[0066]

[0067] Wherein, x is the input monitoring video stream, that is, multiple frames of image data to be processed in this embodiment, and the data dimension is (t in ,h,w); the output is a clear decontamination image y, the data dimension is (t out =1,h,w).

[0068] The compiler of the processing layer in this embodiment learns the translation invariance in spatiotemporal data through a 1×1 convolutional layer, which can be expressed as:

[0069]

[0070] Among them, y i,j is a single pixel that makes up the output decontamination image, x i.jIt is the data within the relative space-time range with the coordinate (i, j) as the origin, and its range and shape are determined by the model receptive field; the space-time compiler of the 1×1 convolutional layer proposed in this embodiment utilizes the translation invariance in the space-time system to greatly reduce the parameter scale of the training model, accelerate the convergence speed of model training, and enhance the robustness of model learning.

[0071] The convolution or visual self-attention model with a shared weight mechanism in the prior art fails to take into account the edge effect of the convolution. An adaptive position encoding matrix is ​​introduced in the compiler of the processing layer to address the potential spatial heterogeneity problem caused by the edge effect and enhance the image processing effect of the image processing model.

[0072] S205, perform spatiotemporal attention mechanism processing on the third encoded data to obtain attention encoded data.

[0073] In some implementations, the third encoded data can be subjected to one or more channel convolutions, and fourth encoded data can be obtained based on the results of the channel convolutions; point convolution can be performed on the fourth encoded data, and the result of the point convolution can be split to obtain first split data and second split data; attention encoded data can be obtained based on the first split data and the second split data.

[0074] In some implementations, the attention mechanism analysis can be performed based on the spatiotemporal attention unit in the spatiotemporal compiler of the processing layer, and the local self-attention mechanism can be performed on the obtained spatial information. Compared with the existing attention unit processing, the pooling operation is deleted in this embodiment to better retain the local features and improve the accuracy of the image processing model. The processing flow is as follows: Figure 3 shown.

[0075] Optionally, Figure 3 The channel convolutions in are expressed as:

[0076] x1 (b,c,h,w) =Conv2d c,c (x 3,(b,c,h,w) ,k=1×w,groups=c)

[0077] x2 (b,c,h,w) =Conv2d c,c (x1 3,(b,c,h,w) ,k=h×1,groups=c)

[0078] Among them, x 3,(b,c,h,w) Represents the third encoded data, and performs a channel convolution on the third encoded data to obtain x1 (b ,c,h,w) , for x1 (b,c,h,w) Perform channel convolution again to get x2 (b,c,h,w) , for x2 (b,c,h,w)Batch normalization is performed to obtain the fourth encoded data. The above channel convolution has a longer receptive field than the existing convolution, and the core attention mechanism can be expressed as:

[0079] x3 (b,2c,h,w) =Conv2d c,2c (BN(x2 (b,c,h,w) ),k=1)

[0080] x4 (b,c,h,w) ,x4` (b,c,h,w) =split(x3 (b,2c,h,w) )

[0081]

[0082] Among them, BN(x2 (b,c,h,w) ) is the value of x2 (b,c,h,w) The fourth encoded data is obtained after batch normalization, BN() is a batch normalization operation, and point convolution is performed on the fourth encoded data to obtain x3 (b,2c,h,w) ; Use the split function to split x3 (b ,2c,h,w) Split, that is, split after point convolution to get the first split data x4 (b,c,h,w) and second split data x4` (b,c,h,w) ; sigmoid is the activation function, for x4 (b,c,h,w) After sigmoid operation, x4` (b,c,h,w) Multiply to get the attention encoding data

[0083] S206, perform channel dilation convolution on the attention encoded data to obtain second encoded data.

[0084] Optionally, the attention coded data can be dilated and convolved to obtain dilated coded data; the dilated coded data can be divided according to the channel dimension to obtain coded data of different dimensions; spatial convolutions of different scales can be performed on the coded data of different dimensions, and the results of the spatial convolution are fused to obtain the coded data to be shrunk; and the shrunk coded data can be shrunk and convolved to obtain second coded data.

[0085] Optionally, channel dilation convolution is performed on the attention encoding data, which can be used to expand the receptive field to reduce edge effects and ensure that sufficient spatial details are retained during the reconstruction of spatiotemporal information. The process of channel dilation and contraction can be expressed as:

[0086]

[0087] x′ (b,c,h,w) =Conv2d 4c,c (x6(b,4c,h,w) ,k=1,groups=1)

[0088] in, is the output data of the core attention mechanism, that is, the attention encoding data, k is the convolution kernel size, groups is the number of convolution groups, the smaller the value of groups, the more complex the convolution operation, but the information processing capability will also be enhanced.

[0089] In some implementations, the expanded coded data x5 after channel expansion (b,4c,h,w) It is divided into four parts according to the channel dimension, each part is denoted by x (b,c,h,w) , the four parts are processed with spatial convolution of different scales respectively and then the channels are contracted for subsequent processing. The four-channel convolution can be expressed as:

[0090] x1 (b,c,h,w) =Conv2d c,c (x (b,c,h,w) ,k=3,groups=c)

[0091] x2 (b,c,h,w) =Conv2d c,c (x (b,c,h,w) ,k=5,groups=c)

[0092] x3 (b,c,h,w) =Conv2d c,c (x (b,4,h,w) ,k=5,groups=c,dilation=2)

[0093] x4 (b,c,h,w) =Conv2d c,c (x (b,c,h,w) ,k=5,groups=c,dilation=3)

[0094] Among them, x1 (b,c,h,w) , x2 (b,c,h,w) , x3 (b,c,h,w) and x4 (b,c,h,w) They are the results of four-channel convolution respectively; dilation is the convolution kernel expansion rate, which can expand the convolution receptive field but reduce the convolution kernel density; the four convolution parameters are to decompose the original large convolution kernel into four convolution kernels of different scales. The parameters of each convolution kernel ensure that different convolution kernels at the same position have as little overlap as possible. Compared with the existing large-kernel convolution operation at the full-channel scale, it reduces the number of parameters and calculations and improves processing efficiency.

[0095] Furthermore, batch normalization and GELU activation function are performed after spatial convolution processing of different scales to obtain the shrunk coded data x6 (b,4c,h,w) , treat the compressed coded data x6(b,4c,h,w) Perform channel contraction, that is, point convolution processing to obtain the second encoded data x′ (b,c,h,w) , 3×3 convolution is used to quadruple the number of channels. At this time, convolution only plays a role in dimensional expansion, so the number of convolution groups is c, reducing the overall complexity; when the channel shrinks, the number of convolution groups is 1, so that the model can process the spatiotemporal information of different feature channels; the process of channel expansion, convolution and contraction is as follows Figure 4 As shown, four-channel convolution is used to reduce the edge effect of image processing, and finally the processed second encoded data is obtained.

[0096] In some implementations, the image processing model may include one or more spatiotemporal compilers, where the first spatiotemporal compiler processes the input first encoded data and passes it to the second spatiotemporal compiler for the same processing, and so on, until the Nth spatiotemporal compiler completes the processing and outputs it; the spatiotemporal compiler can implement adaptive absolute position encoding, spatiotemporal attention, and channel dilation convolution processes to reduce information loss caused by edge effects, enhance the memory ability of details and spatiotemporal information in the video, and thus improve the decontamination effect of moving targets.

[0097] S207: Infer an enhanced target image according to the first coded data and the second coded data.

[0098] In some implementations, the first coded data and the second coded data may be fused to obtain fused coded data; the fused coded data may be decoded based on a decoder to obtain a decontaminated image; that is, the first coded data and the second coded data may be decoded at a decoding layer, thereby combining the first coded data of the locally pollution-free original information processed by the encoder and the moving target feature information reconstructed by the compiler, and the obtained decontaminated image may be expressed as:

[0099]

[0100] in, represents the decontamination image; x f is the first coded data; The second coded data.

[0101] Furthermore, the resolution of the decontamination image can be upsampled to obtain a target image, which is a high-resolution monitoring image, to ensure that the output target image can meet the requirements of observation and analysis, more realistically reflect the monitoring scene, reduce the possibility of misjudgment of moving targets such as personnel and machinery and information omission, and enhance the interpretability and recognition accuracy of the decontamination and de-occlusion results.

[0102] In this embodiment, underground video image data is obtained, and the video image data is preprocessed to obtain image data to be processed that is easy to process. A first parameter matrix and a second parameter matrix are determined according to a preprocessed image processing model, and the multiplication result of the first parameter matrix and the second parameter matrix is ​​used as an adaptive position coding matrix. The first coded data is processed according to the adaptive position coding matrix to reduce the edge effect in the image data to obtain third coded data. The third coded data is further subjected to an attention mechanism based on long-view convolution and a channel expansion convolution, and the global maximum pooling and average pooling in the existing spatiotemporal attention are cancelled. The purpose of global attention is indirectly achieved through long-view convolution, and the advantages of local feature information are better retained, the output accuracy of the model is improved, and the decontamination effect is optimized. Compared with the existing large-core convolution and multi-layer convolution, the channel expansion convolution realizes more efficient receptive field expansion by disassembling a single large-core convolution into multi-layer convolution, solves the problem of long-distance dependence in the image, and finally obtains the second coded data. The first coded data and the second coded data are fused and decoded in the decoder to obtain a decontaminated image, and then the decontaminated image is upsampled to obtain a high-resolution target image, thereby improving the monitoring effect of the underground environment of the coal mine and ensuring safe production.

[0103] Based on the above embodiments, Figure 5 : is a logic flow chart of an underground image enhancement method based on spatiotemporal reconstruction, wherein multiple frames of underground video image data are taken as input for preprocessing, that is, downsampling, resolution reduction or cropping to obtain t low-resolution input images, and first coded data is obtained through a spatiotemporal encoder of a pre-trained image processing model, and the first coded data is copied and input into a spatiotemporal compiler, and processed by N spatiotemporal compilers, and second coded data is obtained through adaptive position coding, spatiotemporal attention and channel expansion convolution in sequence, and the second coded data is fused and decoded with the first coded data in a spatiotemporal decoder to obtain a decontaminated low-resolution image, and the low-resolution image is upsampled to obtain a high-resolution target image, that is, decontaminated image frame data; Figure 5A and Figure 5B As shown, the comparison diagrams of the effect of image enhancement using this embodiment are respectively shown, which significantly improves the image clarity and ensures the monitoring effect of the underground environment of the coal mine.

[0104] In order to implement the above embodiment, the present application also proposes an image enhancement device.

[0105] Figure 6 The schematic diagram of the structure of a downhole image enhancement device based on spatiotemporal reconstruction provided in an embodiment of the present application is shown in FIG. Figure 6 As shown, the downhole image enhancement device based on spatiotemporal reconstruction includes:

[0106] A first acquisition module 601 is used to acquire downhole image data to be processed, and encode and reshape the image data to be processed based on an encoder to obtain first encoded data;

[0107] A second acquisition module 602 is used to acquire a first parameter matrix and a second parameter matrix, and determine an adaptive position coding matrix according to the first parameter matrix and the second parameter matrix;

[0108] A third acquisition module 603 is used to infer the second coded data based on the adaptive position coding matrix and the first coded data;

[0109] The image enhancement module 604 is used to infer an enhanced target image according to the first coded data and the second coded data.

[0110] Furthermore, in a possible implementation of the embodiment of the present application, the second acquisition module 602 includes:

[0111] Get pre-trained image processing models;

[0112] Generate a first parameter matrix according to features of different pixel positions in a two-dimensional plane in a pre-trained image processing model;

[0113] A second parameter matrix is ​​generated according to the differences in how different features in the pre-trained image processing model are affected by absolute positions.

[0114] Furthermore, in a possible implementation of the embodiment of the present application, the second acquisition module 602 includes:

[0115] The second parameter matrix is ​​multiplied by the first parameter matrix to obtain an adaptive position encoding matrix.

[0116] Further, in a possible implementation of the embodiment of the present application, the third acquisition module 603 includes:

[0117] Performing encoding processing on the first encoded data according to the adaptive position encoding matrix to obtain third encoded data;

[0118] Performing spatiotemporal attention mechanism processing on the third encoded data to obtain attention encoded data;

[0119] Perform channel dilation convolution on the attention encoded data to obtain second encoded data.

[0120] Further, in a possible implementation of the embodiment of the present application, the third acquisition module 603 includes:

[0121] A sum of the adaptive position coding matrix and the first coded data is calculated to obtain third coded data.

[0122] Further, in a possible implementation of the embodiment of the present application, the third acquisition module 603 includes:

[0123] Perform one or more channel convolutions on the third encoded data, and obtain fourth encoded data according to the results of the channel convolutions;

[0124] Performing point convolution on the fourth encoded data, and splitting the result after the point convolution to obtain first split data and second split data;

[0125] Attention encoding data is obtained based on the first split data and the second split data.

[0126] Further, in a possible implementation of the embodiment of the present application, the third acquisition module 603 includes:

[0127] Performing dilated convolution on the attention encoding data to obtain dilated encoding data;

[0128] Dividing the expanded coded data according to the channel dimension to obtain coded data of different dimensions;

[0129] Performing spatial convolution of different scales on the coded data of different dimensions, and fusing the results of the spatial convolution to obtain the coded data to be shrunk;

[0130] Performing contraction convolution on the to-be-contracted coded data to obtain second coded data.

[0131] Furthermore, in a possible implementation of the embodiment of the present application, the first acquisition module 601 includes:

[0132] Acquire first video data underground;

[0133] Performing cropping and resolution reduction processing on the first video data to obtain second video data;

[0134] The second video data is normalized and converted into a floating point format to obtain image data to be processed.

[0135] Furthermore, in a possible implementation of the embodiment of the present application, the image enhancement module 604 includes:

[0136] Merging the first coded data and the second coded data to obtain fused coded data;

[0137] Decoding the fused coded data based on a decoder to obtain a decontaminated image;

[0138] The resolution of the decontaminated image is upsampled to obtain the target image.

[0139] It should be noted that the aforementioned explanation of the embodiment of the downhole image enhancement method based on spatiotemporal reconstruction is also applicable to the downhole image enhancement device based on spatiotemporal reconstruction of this embodiment, and will not be repeated here.

[0140] In an embodiment of the present application, underground video image data is obtained, and the video image data is preprocessed to obtain image data to be processed that is easy to process. A first parameter matrix and a second parameter matrix are determined according to a preprocessed image processing model, and the multiplication result of the first parameter matrix and the second parameter matrix is ​​used as an adaptive position coding matrix. The first coded data is processed according to the adaptive position coding matrix to reduce the edge effect in the image data to obtain third coded data. The third coded data is further subjected to long-view convolution and channel expansion convolution, and the global maximum pooling and average pooling in the existing spatiotemporal attention are cancelled. The purpose of global attention is indirectly achieved through long-view convolution, and the advantages of local feature information are better retained, the model output accuracy is improved, and the decontamination effect is optimized. Compared with the existing large-core convolution and multi-layer convolution, the channel expansion convolution realizes more efficient receptive field expansion by disassembling a single large-core convolution into multi-layer convolution, solves the problem of long-distance dependence in the image, and finally obtains the second coded data. The first coded data and the second coded data are fused and decoded in the decoder to obtain a decontaminated image, and then the decontaminated image is upsampled to obtain a high-resolution target image, thereby improving the monitoring effect of the underground environment of the coal mine and ensuring safe production.

[0141] In order to implement the above embodiments, the present application also proposes an electronic device, comprising: a processor, and a memory communicatively connected to the processor; the memory stores computer-executable instructions; the processor executes the computer-executable instructions stored in the memory to implement the method provided by the above embodiments.

[0142] In order to implement the above embodiments, the present application also proposes a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to implement the methods provided by the above embodiments.

[0143] In order to implement the above embodiments, the present application also proposes a computer program product, including a computer program, which implements the methods provided by the above embodiments when executed by a processor.

[0144] The collection, storage, use, processing, transmission, provision and disclosure of user personal information involved in this application are in compliance with relevant laws and regulations and do not violate public order and good morals.

[0145] It should be noted that personal information from users should be collected for legitimate and reasonable purposes and should not be shared or sold outside of these legitimate uses. In addition, such collection / sharing should be carried out after receiving the user's informed consent, including but not limited to notifying the user to read the user agreement / user notice and sign the agreement / authorization including authorization of relevant user information before the user uses the function. In addition, any necessary steps should be taken to protect and safeguard access to such personal information data and ensure that others who have access to personal information data comply with its privacy policy and procedures.

[0146] The present application is expected to provide an implementation scheme for users to selectively block the use or access of personal information data. That is, the present disclosure is expected to provide hardware and / or software to prevent or block access to such personal information data. Once the personal information data is no longer needed, the risk can be minimized by limiting data collection and deleting the data. In addition, when applicable, such personal information is de-identified to protect the privacy of the user.

[0147] In the description of the aforementioned embodiments, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0148] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of this application, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0149] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or more executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present application belong.

[0150] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute the instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purpose of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or processing in other suitable ways if necessary, and then stored in a computer memory.

[0151] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0152] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0153] In addition, each functional unit in each embodiment of the present application may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0154] The storage medium mentioned above may be a read-only memory, a magnetic disk or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limiting the present application. A person of ordinary skill in the art may change, modify, replace and modify the above embodiments within the scope of the present application.

Claims

1. A downhole image enhancement method based on spatiotemporal reconstruction, characterized in that: include: Acquire downhole image data to be processed, and encode and reshape the image data to be processed based on an encoder to obtain first encoded data; Acquire a first parameter matrix and a second parameter matrix, and determine an adaptive position coding matrix according to the first parameter matrix and the second parameter matrix; Inferring second coded data based on the adaptive position coding matrix and the first coded data; An enhanced target image is inferred based on the first encoded data and the second encoded data.

2. The method according to claim 1, characterized in that The obtaining of the first parameter matrix and the second parameter matrix comprises: Get pre-trained image processing models; generating a first parameter matrix according to features of different pixel positions in a two-dimensional plane in the pre-trained image processing model; A second parameter matrix is ​​generated according to the differences in how the absolute positions affect different features in the pre-trained image processing model.

3. The method according to claim 2, characterized in that The determining the adaptive position coding matrix according to the first parameter matrix and the second parameter matrix comprises: The second parameter matrix is ​​multiplied by the first parameter matrix to obtain the adaptive position coding matrix.

4. The method according to any one of claims 1 to 3, characterized in that: The inferring second coded data based on the adaptive position coding matrix and the first coded data comprises: Performing encoding processing on the first encoded data according to the adaptive position encoding matrix to obtain third encoded data; Performing spatiotemporal attention mechanism processing on the third encoded data to obtain attention encoded data; Perform channel dilation convolution on the attention encoded data to obtain second encoded data.

5. The method according to claim 4, characterized in that The encoding process of the first encoded data according to the adaptive position encoding matrix to obtain third encoded data includes: A sum of the adaptive position coding matrix and the first coded data is calculated to obtain the third coded data.

6. The method according to claim 4, characterized in that The processing of the third encoded data by a spatiotemporal attention mechanism to obtain attention encoded data includes: Perform one or more channel convolutions on the third encoded data, and obtain fourth encoded data according to the results of the channel convolutions; Performing point convolution on the fourth encoded data, and splitting the result after the point convolution to obtain first split data and second split data; Attention encoding data is obtained based on the first split data and the second split data.

7. The method according to claim 4, characterized in that The performing channel dilation convolution on the attention encoding data to obtain second encoding data includes: Performing dilated convolution on the attention encoding data to obtain dilated encoding data; Dividing the expanded coded data according to channel dimensions to obtain coded data of different dimensions; Performing spatial convolution of different scales on the coded data of different dimensions, and fusing the results of the spatial convolution to obtain the coded data to be shrunk; Performing contraction convolution on the to-be-contracted coded data to obtain second coded data.

8. The method according to claim 1, characterized in that: Before obtaining the image data to be processed, the method includes: Acquire first video data underground; Performing cropping and resolution reduction processing on the first video data to obtain second video data; The second video data is normalized and converted into a floating point format to obtain image data to be processed.

9. The method according to claim 8, characterized in that The inferring an enhanced target image according to the first coded data and the second coded data comprises: Merging the first coded data and the second coded data to obtain fused coded data; Decoding the fused coded data based on a decoder to obtain a decontaminated image; The decontamination image is upsampled in resolution to obtain a target image.

10. A downhole image enhancement device based on spatiotemporal reconstruction, characterized in that: include: A first acquisition module is used to acquire downhole image data to be processed, and encode and reshape the image data to be processed based on an encoder to obtain first encoded data; A second acquisition module, used to acquire a first parameter matrix and a second parameter matrix, and determine an adaptive position coding matrix according to the first parameter matrix and the second parameter matrix; A third acquisition module, configured to infer second coded data based on the adaptive position coding matrix and the first coded data; An image enhancement module is used to infer an enhanced target image based on the first encoded data and the second encoded data.

Images