A method for downhole image enhancement based on spatiotemporal reconstruction
By using an image enhancement method based on spatiotemporal reconstruction, the problem of image quality degradation in mine monitoring videos was solved, achieving efficient image enhancement and safety management.
Patent Information
- Application Number
- CN202411690057.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-22
- Publication Date
- 2025-10-28
- Estimated Expiration
- 2044-11-22
AI Technical Summary
Environmental factors have led to a decline in the image quality of mine monitoring videos, and existing image enhancement methods are unable to effectively distinguish between pollutants and targets, which affects safety management.
An image enhancement method based on spatiotemporal reconstruction is adopted. The image data is encoded and reshaped by an encoder to obtain an adaptive positional coding matrix and a parameter matrix. The image is then enhanced by combining the adaptive positional coding matrix and the coded data to reduce information loss caused by edge effects.
It improved the clarity of underground monitoring images, optimized the decontamination effect, and ensured the safe management of mine production.
Smart Images
Figure CN119762725B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of image processing technology, and in particular to a method for enhancing downhole images based on spatiotemporal reconstruction. Background Technology
[0002] In mine safety production, monitoring systems have become a core tool for ensuring smooth production and personnel safety. The clarity of underground monitoring videos directly affects the identification of safety hazards and the effectiveness of equipment monitoring. However, the complex environmental conditions inside mines, such as coal dust, mud, fog, and direct sunlight, often cause dirt to accumulate on the surface of camera lenses, resulting in a significant decrease in video quality. Problems such as blurred video images, obstructions, and reflections occur frequently, seriously hindering the normal operation of the monitoring system. Key information such as the operating status of underground equipment and personnel activities may be obscured or distorted, failing to provide effective decision support for production management. Therefore, how to improve video clarity and remove the impact of water vapor, dust, and dirt on monitoring images in harsh mine environments has become one of the key issues in mine safety management.
[0003] With the development of digital and intelligent technologies in mines, the requirements for the clarity of surveillance videos are constantly increasing. Traditional image enhancement methods, such as brightness adjustment, contrast adjustment, and noise reduction filtering, can improve video quality to some extent, but the effects of these methods are usually limited to static image processing. Decontamination algorithms have difficulty distinguishing between contaminants and actual targets, resulting in poor decontamination effects. Summary of the Invention
[0004] This application aims to at least partially address one of the technical problems in the related art.
[0005] Therefore, the first objective of this application is to propose a downhole image enhancement method based on spatiotemporal reconstruction to achieve better image enhancement results.
[0006] The second objective of this application is to propose a downhole image enhancement device based on spatiotemporal reconstruction.
[0007] The third objective of this application is to propose an electronic device.
[0008] The fourth objective of this application is to provide a computer-readable storage medium.
[0009] The fifth objective of this application is to provide a computer program product.
[0010] To achieve the above objectives, a first aspect of this application proposes a downhole image enhancement method based on spatiotemporal reconstruction, comprising:
[0011] The downhole image data to be processed is acquired, and the image data to be processed is encoded and reshaped based on the encoder to obtain the first encoded data;
[0012] Obtain the first parameter matrix and the second parameter matrix, and determine the adaptive position encoding matrix based on the first parameter matrix and the second parameter matrix;
[0013] Based on the adaptive position coding matrix and the first coding data, the second coding data is inferred.
[0014] The enhanced target image is inferred based on the first encoded data and the second encoded data.
[0015] To achieve the above objectives, a second aspect of this application provides a downhole image enhancement device based on spatiotemporal reconstruction, comprising:
[0016] The first acquisition module is used to acquire downhole image data to be processed, and to encode and reshape the image data to be processed based on the encoder to obtain first encoded data;
[0017] The second acquisition module is used to acquire the first parameter matrix and the second parameter matrix, and determine the adaptive position encoding matrix based on the first parameter matrix and the second parameter matrix;
[0018] The third acquisition module is used to infer the second encoded data based on the adaptive position encoding matrix and the first encoded data;
[0019] The image enhancement module is used to infer the enhanced target image based on the first encoded data and the second encoded data.
[0020] To achieve the above objectives, a third aspect of this application provides an electronic device, including: a processor, and a memory communicatively connected to the processor;
[0021] The memory stores computer-executed instructions;
[0022] The processor executes computer execution instructions stored in the memory to implement the method described in the first aspect embodiment.
[0023] To achieve the above objectives, a fourth aspect of this application provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the method described in the first aspect embodiment.
[0024] To achieve the above objectives, a fifth aspect of this application provides a computer program product, including a computer program that, when executed by a processor, implements the method described in the first aspect embodiment.
[0025] This application provides a spatiotemporal reconstruction-based method for enhancing underground images. It involves encoding and reshaping the underground image data to obtain first encoded data, acquiring first and second parameter matrices, and further determining adaptive positional encoding based on these matrices to learn the feature information of each position in the image data. Second encoded data is then obtained based on the adaptive positional encoding matrix and the first encoded data, incorporating the feature information of each position to successfully distinguish image edge positions and reduce information loss caused by edge effects. Finally, the first and second encoded data are fused and decoded to obtain the enhanced target image. This method solves the problem of poor image clarity in underground coal mine monitoring and ensures safe management of mine production.
[0026] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description
[0027] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein:
[0028] Figure 1 A schematic flowchart illustrating a downhole image enhancement method based on spatiotemporal reconstruction provided in this application embodiment;
[0029] Figure 2 A schematic flowchart illustrating another downhole image enhancement method based on spatiotemporal reconstruction provided in this application embodiment;
[0030] Figure 3 A schematic diagram illustrating a spatiotemporal attention process based on long-viewpoint convolution provided in an embodiment of this application;
[0031] Figure 4 A schematic diagram of a channel-dilated convolution provided in an embodiment of this application;
[0032] Figure 5 A flowchart illustrating a downhole image enhancement method based on spatiotemporal reconstruction provided in this application embodiment;
[0033] Figure 5A This is a comparison image of the effect of image enhancement provided in an embodiment of this application;
[0034] Figure 5B This is a comparison image of another image enhancement effect provided in an embodiment of this application;
[0035] Figure 6This is a schematic diagram of the structure of a downhole image enhancement device based on spatiotemporal reconstruction, provided in an embodiment of this application. Detailed Implementation
[0036] The embodiments of this application are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application.
[0037] This application is applicable to scenarios where cameras used for extended periods in mining environments are affected by environmental factors such as dust, smoke, and lighting conditions, leading to blurred edges of target objects and degraded image quality in the video. It aims to improve video image clarity and remove the impact of moisture, dust, and dirt on monitoring images in harsh mining environments. An embodiment of this application based on spatiotemporal reconstruction is described below with reference to the accompanying drawings.
[0038] Figure 1 This is a schematic flowchart of a downhole image enhancement method based on spatiotemporal reconstruction provided in an embodiment of this application.
[0039] like Figure 1 As shown, the method includes the following steps:
[0040] S101: Acquire downhole image data to be processed, and re-encode and reshape the image data to be processed based on the encoder to obtain the first encoded data.
[0041] In some implementations, a monitoring system is deployed underground for real-time monitoring during the mine's safe production process to ensure smooth production and personnel safety. The monitoring system can acquire image data to be processed underground in real time. This image data can be video stream data, which is image data with poor clarity that needs to be enhanced.
[0042] The encoder can be used to capture the spatial features of the image data to be processed and extend the temporal dimension; in this embodiment, the encoder can separate the uncluttered static background information in the monitoring video frame to obtain the first encoded data; optionally, the process of the encoder encoding and reshaping the image data to be processed to obtain the first encoded data can be represented as:
[0043]
[0044] in, The first encoded data is denoted as k, where k is the kernel size; t in b is the number of input images; b is the batch size; h and w are the image dimensions; c in c is the input image channel; hidis the channel of the hidden layer; stride is the kernel stride, which is 2 in this embodiment, and can reduce the resolution; Conv2d is the two-dimensional convolution operation.
[0045] S102, obtain the first parameter matrix and the second parameter matrix, and determine the adaptive position coding matrix based on the first parameter matrix and the second parameter matrix.
[0046] In some implementations, the first parameter matrix and the second parameter matrix can be parameter matrices generated during model training. For example, the first parameter matrix is a matrix for learning features at different positions, and the second parameter matrix is a matrix for how different features are affected by position. That is, during the model training process based on training samples, the evolution features of different positions of the training samples and the positional influences on different features are learned, so that the corresponding first parameter matrix and second parameter matrix are obtained after the model training is completed.
[0047] Optionally, the product of the first parameter matrix and the second parameter matrix can be calculated, and the result of the multiplication can be used as an adaptive position encoding matrix, thereby fusing the position features learned from the first parameter matrix and the second parameter matrix.
[0048] S103, based on the adaptive position coding matrix and the first coding data, the second coding data is inferred.
[0049] Optionally, the first encoded data can be re-encoded based on the adaptive positional encoding matrix. For example, the adaptive positional encoding matrix can be added to the first encoded data to fuse positional encoding information in the first encoded data, thereby obtaining the second encoded data; or the result of adding the first encoded data and the adaptive positional encoding matrix can be subjected to attention mechanism analysis or convolution processing to obtain the second encoded data.
[0050] S104, based on the first encoded data and the second encoded data, infer the enhanced target image.
[0051] In some implementations, the first coded data and the second coded data are fused, for example, by adding the first coded data and the second coded data together to obtain the fused coded data; the fused coded data is then decoded to obtain the enhanced target image.
[0052] Optionally, the decoded image can also be enhanced, such as denoising and filtering, to obtain a higher quality and clearer target image.
[0053] In this embodiment, the underground image data to be processed is encoded and reshaped to obtain first encoded data, and first and second parameter matrices are obtained. Adaptive position coding is further determined based on the first and second parameter matrices to learn the feature information of each position in the image data. Second encoded data is obtained based on the adaptive position coding matrix and the first encoded data. The second encoded data incorporates the feature information of each position, thereby successfully distinguishing the image edge position and reducing information loss caused by edge effects. Based on the first and second encoded data, fusion decoding is performed to obtain an enhanced target image, improving the output quality of underground monitoring images, optimizing the image decontamination effect, and conducting underground environmental monitoring and analysis based on the monitoring image with better decontamination effect to ensure the safe management of mine production.
[0054] Figure 2 This is a schematic flowchart illustrating another downhole image enhancement method based on spatiotemporal reconstruction provided in an embodiment of this application. Figure 2 As shown, the method includes the following steps:
[0055] S201: Acquire downhole image data to be processed, and re-encode and reshape the image data to be processed based on the encoder to obtain the first encoded data.
[0056] In some implementations, the first video data from underground can be acquired, which can be collected by a monitoring system. Further, the first video data can be cropped and down-resolution to obtain the second video data, thereby improving the processing efficiency of the first video data. The second video data is then normalized and converted to a floating-point format to obtain the image data to be processed, thus improving the processing effect of the image data to be processed.
[0057] Optionally, a pre-trained image processing model can be determined, which includes at least an encoding layer, a processing layer, and a decoding layer; the processing layer may include one or more spatiotemporal compilers, which encode and reshape the image data to be processed based on the encoder of the encoding layer after inputting the image data to be processed into the image processing model, thereby obtaining the first encoded data.
[0058] In this application embodiment, the implementation method of step S201 can be implemented in any of the various embodiments of this disclosure, and no limitation is made here, nor will it be described in detail.
[0059] S202, obtain the first parameter matrix and the second parameter matrix.
[0060] Optionally, a pre-trained image processing model is obtained; a first parameter matrix is generated based on the features of different pixel positions in the two-dimensional plane within the pre-trained image processing model; and a second parameter matrix is generated based on the differences in how different features are affected by absolute position within the pre-trained image processing model. In other words, during the training of the image processing model, the evolutionary features of different positions in the two-dimensional plane are learned from the training samples to generate the first parameter matrix; simultaneously, the differences in how different features are affected by absolute position are learned to generate the second parameter matrix. After the image processing model is trained, the first and second parameter matrices are obtained from the image processing model.
[0061] S203, multiply the second parameter matrix by the first parameter matrix to obtain the adaptive position coding matrix.
[0062] In some implementations, the size of the first parameter matrix can be 1*N, the size of the second parameter matrix is M*1, and the adaptive position encoding matrix after multiplying the second parameter matrix and the first parameter matrix has a dimension of M*N, which is the same as the size of the image data to be processed.
[0063] S204, The first encoded data is encoded according to the adaptive position encoding matrix to obtain the third encoded data.
[0064] Optionally, the third encoded data can be obtained by summing the adaptive positional encoding matrix and the first encoded data. The third encoded data can be obtained by calculating the spatiotemporal compiler in the processing layer. The addition of the adaptive positional encoding matrix and the first encoded data can enhance the adaptability to spatial heterogeneity such as edge effects.
[0065] It is understandable that adaptive positional encoding matrices have advantages over existing spatiotemporal reasoning, which can be represented as:
[0066]
[0067] Where x is the input monitoring video stream, which is the multi-frame image data to be processed in this embodiment, and the data dimension is (t in The output is a clear, uncontaminated image y, with data dimensions (t, h, w). out =1,h,w).
[0068] In this embodiment, the compiler of the processing layer learns the translation invariance in spatiotemporal data through a 1×1 convolutional layer, which can be represented as:
[0069]
[0070] Among them, y i,j For each individual pixel that makes up the output desmudged image, x i.jThe data is within a relative spatiotemporal range with coordinates (i,j) as the origin, and its range and shape are determined by the receptive field of the model. The spatiotemporal compiler of the 1×1 convolutional layer proposed in this embodiment utilizes the translation invariance in the spatiotemporal system to significantly reduce the parameter size of the training model, accelerate the model training convergence speed, and enhance the robustness of model learning.
[0071] Existing convolutional or visual self-attention models with shared weight mechanisms fail to consider the edge effects of convolution. To address the potential spatial heterogeneity issues caused by edge effects, an adaptive positional encoding matrix is introduced into the compiler of the processing layer, thereby enhancing the image processing model's image processing performance.
[0072] S205, the third encoded data is processed by a spatiotemporal attention mechanism to obtain attention-encoded data.
[0073] In some implementations, the third encoded data can be subjected to one or more channel convolutions, and the fourth encoded data can be obtained based on the result of the channel convolutions; the fourth encoded data can be subjected to point convolutions, and the result of the point convolutions can be split to obtain the first split data and the second split data; attention encoded data can be obtained based on the first split data and the second split data.
[0074] In some implementations, attention mechanism analysis can be performed based on the spatiotemporal attention unit in the spatiotemporal compiler of the processing layer. Local self-attention mechanisms are then applied to the obtained spatial information. Compared to existing attention unit processing, this embodiment removes the pooling operation to better preserve local features and improve the accuracy of the image processing model. The processing flow is as follows: Figure 3 As shown.
[0075] Optionally, Figure 3 The channel convolutions in the diagram are represented as follows:
[0076] x1 (b,c,h,w) =Conv2d c,c (x 3,(b,c,h,w) k = 1 × w, groups = c)
[0077] x2 (b,c,h,w) =Conv2d c,c (x1 3,(b,c,h,w) k = h × 1, groups = c)
[0078] Where, x 3,(b,c,h,w) This represents the third encoded data. Performing a single channel convolution on the third encoded data yields x1. (b ,c,h,w) For x1 (b,c,h,w) Perform channel convolution again to obtain x2 (b,c,h,w) For x2 (b,c,h,w)Batch normalization is performed to obtain the fourth encoded data. The above channel convolution has a longer receptive field compared to existing convolutions, achieving both high complexity and low complexity. The core attention mechanism can be represented as:
[0079] x3 (b,2c,h,w) =Conv2d c,2c (BN(x2 (b,c,h,w) ),k=1)
[0080] x4 (b,c,h,w) ,x4` (b,c,h,w) =split(x3) (b,2c,h,w) )
[0081]
[0082] Wherein, BN(x2) (b,c,h,w) ) is for x2 (b,c,h,w) The fourth encoded data is obtained after batch normalization. BN() is the batch normalization operation. A point convolution is performed on the fourth encoded data to obtain x3. (b,2c,h,w) ; using the split function to split x3 (b ,2c,h,w) The data is split, that is, after point convolution, it is split to obtain the first split data x4. (b,c,h,w) Second split data x4` (b,c,h,w) ; sigmoid is the activation function, for x4 (b,c,h,w) After performing the sigmoid operation, it is combined with x4` (b,c,h,w) Multiplication yields attention-encoded data
[0083] S206, perform channel dilation convolution on the attention-encoded data to obtain the second encoded data.
[0084] Optionally, the attention-encoded data can be dilated by convolution to obtain dilated encoded data; the dilated encoded data can be divided according to the channel dimension to obtain encoded data of different dimensions; spatial convolutions of different dimensions can be performed on the encoded data of different dimensions, and the results of spatial convolutions can be fused to obtain encoded data to be shrunk; and shrinking convolutions can be performed on the encoded data to be shrunk to obtain second encoded data.
[0085] Optionally, channel-expanding convolutions can be applied to the attention-encoded data to expand the receptive field, reduce edge effects, and ensure sufficient spatial details are preserved during spatiotemporal information reconstruction. The process of channel expansion and contraction can be represented as follows:
[0086]
[0087] x′ (b,c,h,w) =Conv2d 4c,c (x6(b,4c,h,w) (k=1,groups=1)
[0088] in, The output data of the core attention mechanism is the attention-encoded data. k is the kernel size and groups is the number of convolution groups. The smaller the groups value, the more complex the convolution operation, but the information processing capability will also be enhanced.
[0089] In some implementations, the expanded encoded data after channel expansion is x5 (b,4c,h,w) It will be divided into four parts according to the channel dimension, each part denoted as x. (b,c,h,w) The four parts are spatially convolved at different scales and then channel shrinking is performed for subsequent processing. The four-channel convolution can be represented as:
[0090] x1 (b,c,h,w) =Conv2d c,c (x (b,c,h,w) (k=3,groups=c)
[0091] x2 (b,c,h,w) =Conv2d c,c (x (b,c,h,w) (k=5,groups=c)
[0092] x3 (b,c,h,w) =Conv2d c,c (x (b,4,h,w) ,k=5,groups=c,dilation=2)
[0093] x4 (b,c,h,w) =Conv2d c,c (x (b,c,h,w) ,k=5,groups=c,dilation=3)
[0094] Where x1 (b,c,h,w) x2 (b,c,h,w) x3 (b,c,h,w) and x4 (b,c,h,w) These are the results of four-channel convolution; dilation is the kernel expansion rate, which can expand the receptive field of convolution but will reduce the kernel density; the four convolution parameters decompose the original large convolution kernel into four convolution kernels of different scales. Each convolution kernel parameter ensures that different convolution kernels at the same position overlap as little as possible. Compared with the existing large kernel convolution operation at the full channel scale, this reduces the number of parameters and computation, and improves processing efficiency.
[0095] Furthermore, after performing spatial convolution processing at different scales, batch normalization and the GELU activation function are applied to obtain the 6 x encoded data to be shrunk. (b,4c,h,w) To shrink the encoded data x6(b,4c,h,w) Channel shrinking, also known as point convolution, is used to obtain the second encoded data x′. (b,c,h,w) Using 3×3 convolutions quadruples the number of channels. In this case, the convolution only expands the dimension, so the number of convolution groups is c, reducing overall complexity. When shrinking channels, the number of convolution groups is 1, allowing the model to process spatiotemporal information from different feature channels. The process of channel expansion and contraction is as follows: Figure 4 As shown, edge effects in image processing are reduced by using four-channel convolution, ultimately yielding the processed second encoded data.
[0096] In some implementations, the image processing model may include one or more spatiotemporal compilers. The first spatiotemporal compiler processes the first encoded input data, and the processed data is passed to the second spatiotemporal compiler for the same processing, and so on, until the Nth spatiotemporal compiler completes the processing and outputs the data. The spatiotemporal compiler can implement adaptive absolute position encoding, spatiotemporal attention, and channel dilation convolution processes to reduce information loss caused by edge effects, enhance the ability to remember details and spatiotemporal information in the video, and thus improve the desmearing effect of moving targets.
[0097] S207, based on the first encoded data and the second encoded data, infer the enhanced target image.
[0098] In some implementations, the first and second encoded data can be fused to obtain fused encoded data; the fused encoded data is then decoded by the decoder to obtain a decontamination image; that is, the first and second encoded data are decoded at the decoding layer, thereby combining the locally uncontaminated original information of the first encoded data processed by the encoder with the moving target feature information reconstructed by the compiler. The resulting decontamination image can be represented as:
[0099]
[0100] in, Represents a decontamination image; x f This is the first encoded data; This is the second encoded data.
[0101] Furthermore, the decontamination image can be upsampled to obtain a target image, which is a high-resolution monitoring image. This ensures that the output target image meets the requirements of observation and analysis, more realistically reflects the monitoring scene, reduces the possibility of misjudgment and information omission of moving targets such as personnel and machinery, and enhances the interpretability and recognition accuracy of the decontamination and de-occlusion results.
[0102] In this embodiment, video image data from underground is acquired and preprocessed to obtain image data that is easy to process. A first parameter matrix and a second parameter matrix are determined based on the preprocessed image processing model. The product of the first and second parameter matrices is used as an adaptive positional encoding matrix. The first encoded data is processed based on the adaptive positional encoding matrix to reduce edge effects in the image data, resulting in third encoded data. Further, a long-field-of-view convolution-based attention mechanism and channel-dilation convolution are applied to the third encoded data. This eliminates the global max pooling and average pooling in existing spatiotemporal attention methods, indirectly achieving global attention through long-field-of-view convolution. This better preserves local feature information, improves model output accuracy, and optimizes the decontamination effect. Compared to existing large-kernel convolution and multi-layer convolution, channel-dilation convolution decomposes a single large-kernel convolution into multiple convolutions, achieving more efficient receptive field expansion and solving long-distance dependency problems in images. Finally, second encoded data is obtained. The first and second encoded data are fused and decoded in the decoder to obtain a decontaminated image. The decontaminated image is then upsampled to obtain a high-resolution target image, improving the monitoring effect of the underground coal mine environment and ensuring safe production.
[0103] Based on the above embodiments, Figure 5 The flowchart illustrates the logic of a spatiotemporal reconstruction-based downhole image enhancement method. Multi-frame downhole video image data is used as input for preprocessing, specifically downsampling or cropping to obtain t low-resolution input images. The first encoded data is obtained through a pre-trained spatiotemporal encoder of an image processing model. This first encoded data is copied and input into a spatiotemporal compiler, where it is processed by N spatiotemporal compilers. The second encoded data is obtained through adaptive positional encoding, spatiotemporal attention, and channel dilation convolution. The second encoded data and the first encoded data are then fused and decoded in a spatiotemporal decoder to obtain a decontaminated low-resolution image. This low-resolution image is then upsampled to obtain a high-resolution target image, i.e., the decontaminated image frame data. Figure 5A and Figure 5B As shown, the image enhancement effect using this embodiment is compared, which significantly improves image clarity and ensures the monitoring effect of the underground coal mine environment.
[0104] To achieve the above embodiments, this application also proposes an image enhancement device.
[0105] Figure 6 This is a schematic diagram of a downhole image enhancement device based on spatiotemporal reconstruction, provided as an embodiment of this application. Figure 6 As shown, the downhole image enhancement device based on spatiotemporal reconstruction includes:
[0106] The first acquisition module 601 is used to acquire downhole image data to be processed, and to encode and reshape the image data to be processed based on the encoder to obtain the first encoded data;
[0107] The second acquisition module 602 is used to acquire the first parameter matrix and the second parameter matrix, and determine the adaptive position encoding matrix based on the first parameter matrix and the second parameter matrix;
[0108] The third acquisition module 603 is used to infer the second encoded data based on the adaptive position encoding matrix and the first encoded data;
[0109] The image enhancement module 604 is used to infer the enhanced target image based on the first encoded data and the second encoded data.
[0110] Furthermore, in one possible implementation of this application embodiment, the second acquisition module 602 includes:
[0111] Obtain a pre-trained image processing model;
[0112] Based on the features of different pixel positions in the two-dimensional plane in the pre-trained image processing model, the first parameter matrix is generated.
[0113] The second parameter matrix is generated based on the differences in how different features are affected by absolute position in the pre-trained image processing model.
[0114] Furthermore, in one possible implementation of this application embodiment, the second acquisition module 602 includes:
[0115] Multiplying the second parameter matrix by the first parameter matrix yields the adaptive position coding matrix.
[0116] Furthermore, in one possible implementation of this application embodiment, the third acquisition module 603 includes:
[0117] The first encoded data is encoded according to the adaptive position encoding matrix to obtain the third encoded data;
[0118] The third encoded data is processed using a spatiotemporal attention mechanism to obtain attention-encoded data;
[0119] The attention-encoded data is subjected to channel-dilated convolution to obtain the second encoded data.
[0120] Furthermore, in one possible implementation of this application embodiment, the third acquisition module 603 includes:
[0121] The third coded data is obtained by summing the adaptive positional coding matrix with the first coded data.
[0122] Furthermore, in one possible implementation of this application embodiment, the third acquisition module 603 includes:
[0123] Perform one or more channel convolutions on the third encoded data, and obtain the fourth encoded data based on the results of the channel convolutions;
[0124] Perform point convolution on the fourth encoded data, and split the result of point convolution to obtain the first split data and the second split data;
[0125] Attention-encoded data is obtained based on the first split data and the second split data.
[0126] Furthermore, in one possible implementation of this application embodiment, the third acquisition module 603 includes:
[0127] The attention-encoded data is subjected to dilated convolution to obtain dilated encoded data;
[0128] The extended coded data is divided according to the channel dimension to obtain coded data of different dimensions;
[0129] The encoded data of different dimensions are subjected to spatial convolution at different scales, and the results of spatial convolution are fused to obtain the encoded data to be shrunk.
[0130] The data to be shrunk is subjected to a shrinking convolution to obtain the second coded data.
[0131] Furthermore, in one possible implementation of this application embodiment, the first acquisition module 601 includes:
[0132] Acquire the first video data from underground;
[0133] The first video data is cropped and down-resolution to obtain the second video data;
[0134] The second video data is normalized and converted into floating-point format to obtain the image data to be processed.
[0135] Furthermore, in one possible implementation of this application embodiment, the image enhancement module 604 includes:
[0136] The first coded data and the second coded data are merged to obtain the merged coded data;
[0137] The fused encoded data is decoded using a decoder to obtain a desmeared image;
[0138] The target image is obtained by upsampling the resolution of the decontamination image.
[0139] It should be noted that the foregoing explanation of the embodiment of the downhole image enhancement method based on spatiotemporal reconstruction also applies to the downhole image enhancement device based on spatiotemporal reconstruction in this embodiment, and will not be repeated here.
[0140] In this embodiment, video image data from underground is acquired and preprocessed to obtain image data that is easy to process. A first parameter matrix and a second parameter matrix are determined based on the preprocessed image processing model. The product of the first and second parameter matrices is used as an adaptive positional encoding matrix. The first encoded data is processed according to the adaptive positional encoding matrix to reduce edge effects in the image data, resulting in third encoded data. Further, long-field convolution and channel-expanding convolution are performed on the third encoded data, eliminating the global max pooling and average pooling in existing spatiotemporal attention methods. Long-field convolution indirectly achieves global attention, better preserving local feature information, improving model output accuracy, and optimizing the decontamination effect. Compared to existing large-kernel convolution and multi-layer convolution, channel-expanding convolution decomposes a single large-kernel convolution into multiple layers, achieving more efficient receptive field expansion and solving long-distance dependency problems in images, ultimately obtaining second encoded data. The first and second encoded data are fused and decoded in the decoder to obtain a decontaminated image. The decontaminated image is then upsampled to obtain a high-resolution target image, improving the monitoring effect of the underground coal mine environment and ensuring safe production.
[0141] To implement the above embodiments, this application also proposes an electronic device, including: a processor and a memory communicatively connected to the processor; the memory stores computer execution instructions; the processor executes the computer execution instructions stored in the memory to implement the method provided in the foregoing embodiments.
[0142] To implement the above embodiments, this application also proposes a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the methods provided in the foregoing embodiments.
[0143] To implement the above embodiments, this application also proposes a computer program product, including a computer program that, when executed by a processor, implements the methods provided in the foregoing embodiments.
[0144] The collection, storage, use, processing, transmission, provision, and disclosure of user personal information involved in this application comply with relevant laws and regulations and do not violate public order and good morals.
[0145] It should be noted that personal information collected from users should be used for legitimate and reasonable purposes and should not be shared or sold outside of these legitimate uses. Furthermore, such collection / sharing should only be conducted after receiving the user's informed consent, including but not limited to notifying the user to read the user agreement / user notice and sign an agreement / authorization that includes authorization of relevant user information before the user uses the function. In addition, any necessary steps must be taken to protect and safeguard access to such personal information data and ensure that others with access to personal information data comply with their privacy policies and procedures.
[0146] This application is intended to provide an implementation scheme for users to selectively prevent the use or access to their personal information data. Specifically, this disclosure is intended to provide hardware and / or software to prevent or block access to such personal information data. Once personal information data is no longer needed, risks can be minimized by restricting data collection and deleting data. Furthermore, where applicable, such personal information is de-identified to protect user privacy.
[0147] In the foregoing descriptions of the embodiments, the terms "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.
[0148] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of the technical features being referred to. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of such features. Throughout the description of this application, "plurality" means at least two, for example, two, three, etc., unless otherwise specifically defined.
[0149] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing custom logic functions or processes, and the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as should be understood by those skilled in the art to which embodiments of this application pertain.
[0150] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or processing it in another suitable manner if necessary, and then storing it in a computer memory.
[0151] It should be understood that various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.
[0152] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.
[0153] Furthermore, the functional units in the various embodiments of this application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.
[0154] The storage medium mentioned above can be a read-only memory, a disk, or an optical disk, etc. Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of this application.
Claims
1. A method for enhancing downhole images based on spatiotemporal reconstruction, characterized in that, include: The downhole image data to be processed is acquired, and the image data to be processed is encoded and reshaped based on the encoder to obtain the first encoded data; Obtain the first parameter matrix and the second parameter matrix, and determine the adaptive position encoding matrix based on the first parameter matrix and the second parameter matrix; Based on the adaptive position coding matrix and the first coding data, the second coding data is inferred. Based on the first encoded data and the second encoded data, the enhanced target image is inferred; The step of inferring the second encoded data based on the adaptive position encoding matrix and the first encoded data includes: The first encoded data is encoded according to the adaptive position encoding matrix to obtain the third encoded data; The third encoded data is processed using a spatiotemporal attention mechanism to obtain attention-encoded data; The attention-encoded data is subjected to channel-dilated convolution to obtain the second encoded data; The process of performing spatiotemporal attention mechanism processing on the third encoded data to obtain attention encoded data includes: Perform one or more channel convolutions on the third encoded data, and obtain the fourth encoded data based on the results of the channel convolutions; Perform point convolution on the fourth encoded data, and split the result of point convolution to obtain the first split data and the second split data; Attention-encoded data is obtained based on the first split data and the second split data.
2. The method according to claim 1, characterized in that, The process of obtaining the first parameter matrix and the second parameter matrix includes: Obtain a pre-trained image processing model; Based on the features of different pixel positions in the two-dimensional plane in the pre-trained image processing model, a first parameter matrix is generated; A second parameter matrix is generated based on the differences in how different features are affected by absolute position in the pre-trained image processing model.
3. The method according to claim 2, characterized in that, The step of determining the adaptive position encoding matrix based on the first parameter matrix and the second parameter matrix includes: Multiplying the second parameter matrix by the first parameter matrix yields the adaptive position encoding matrix.
4. The method according to claim 1, characterized in that, The step of encoding the first encoded data according to the adaptive position encoding matrix to obtain the third encoded data includes: The third encoded data is obtained by summing the adaptive positional encoding matrix with the first encoded data.
5. The method according to claim 1, characterized in that, The process of performing channel-dilated convolution on the attention-encoded data to obtain second-encoded data includes: The attention-encoded data is subjected to dilated convolution to obtain dilated encoded data; The extended coded data is divided according to the channel dimension to obtain coded data of different dimensions; The encoded data of different dimensions are subjected to spatial convolution at different scales, and the results of spatial convolution are fused to obtain the encoded data to be shrunk. The data to be shrunk is subjected to a shrinking convolution to obtain the second coded data.
6. The method according to claim 1, characterized in that, Before acquiring the image data to be processed, the following steps are included: Acquire the first video data from underground; The first video data is cropped and down-resolution to obtain the second video data; The second video data is normalized and converted into floating-point format to obtain the image data to be processed.
7. The method according to claim 6, characterized in that, The step of inferring the enhanced target image based on the first encoded data and the second encoded data includes: The first encoded data and the second encoded data are fused to obtain fused encoded data; The fused encoded data is decoded using a decoder to obtain a desmeared image. The decontamination image is upsampled to obtain the target image.
8. A downhole image enhancement device based on spatiotemporal reconstruction, characterized in that, include: The first acquisition module is used to acquire downhole image data to be processed, and to encode and reshape the image data to be processed based on the encoder to obtain first encoded data; The second acquisition module is used to acquire the first parameter matrix and the second parameter matrix, and determine the adaptive position encoding matrix based on the first parameter matrix and the second parameter matrix; The third acquisition module is used to infer the second encoded data based on the adaptive position encoding matrix and the first encoded data; An image enhancement module is used to infer the enhanced target image based on the first encoded data and the second encoded data; The third acquisition module is further configured to: The first encoded data is encoded according to the adaptive position encoding matrix to obtain the third encoded data; The third encoded data is processed using a spatiotemporal attention mechanism to obtain attention-encoded data; The attention-encoded data is subjected to channel-dilated convolution to obtain the second encoded data; The third acquisition module is also used for: Perform one or more channel convolutions on the third encoded data, and obtain the fourth encoded data based on the results of the channel convolutions; Perform point convolution on the fourth encoded data, and split the result of point convolution to obtain the first split data and the second split data; Attention-encoded data is obtained based on the first split data and the second split data.