Downhole video restoration method, device and readable storage medium

By using dark channel image restoration model in downhole video processing, the atmospheric light value and transmittance are determined, and the model is updated in real time to adapt to environmental changes, the problem of low video surveillance picture quality in the existing technology is solved, and efficient restoration of downhole videos is achieved and stability improvement.

CN118470615BActive Publication Date: 2024-11-22SANY INTELLIGENT EQUIP CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410927575.1
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-07-11
Publication Date
2024-11-22
Estimated Expiration
2044-07-11

AI Technical Summary

Technical Problem

The existing technology cannot be effectively applied to downhole video processing in different environments, resulting in low quality of video surveillance images and cannot meet the needs of real-time video restoration.

Method used

Using a method based on the dark channel image restoration model, we obtain the original image of the downhole video, determine the atmospheric light value and transmittance, build a dark channel image restoration model, update the model in real time to adapt to environmental changes, and improve the restoration effect of the video.

Benefits of technology

Real-time and efficient recovery of downhole videos is achieved, the negative impact of dust and fog on video quality is alleviated, the clarity and contrast of the video surveillance screen is improved, and it is suitable for changes in different environments.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118470615B_ABST
    Figure CN118470615B_ABST
Patent Text Reader

Abstract

The present invention provides a method, device and readable storage medium for restoring underground videos, which relates to the field of video processing technology, and aims to solve or improve the technical problem in the prior art that conventional video processing methods cannot be applied to different environments. The method for restoring underground videos includes: obtaining underground videos; after receiving a preset instruction, determining that the current image of the underground video is the original image; determining the atmospheric light value based on the original image; determining the transmittance based on the atmospheric light value and the original image; constructing a dark channel image restoration model based on the atmospheric light value, the original image and the transmittance; restoring the underground video based on the dark channel image restoration model. The present invention can restore underground videos in real time by applying a dark channel priori algorithm to the video restoration task of a fully-mechanized mining face, and effectively alleviates the negative impact of mining dust and fog on the quality of video monitoring images.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of video processing technology, and in particular to a method, device and readable storage medium for restoring underground videos. Background Art

[0002] The quality of video surveillance images of fully mechanized mining working faces is mainly affected by two factors: on the one hand, the underground environment is complex and changeable, and a large amount of coal dust and water mist generated during mining have scattering, refraction and reflection effects on light. The above light is mixed with the light reflected by the target, causing the image capture equipment to capture images. The sharpness and contrast are greatly reduced; on the other hand, the weak lighting conditions underground cause the image to be dim, the noise increases, and the details are blurred. In this case, it is particularly important to improve the clarity of the video picture and grasp the production situation of the working face in real time. In addition, as the key to the intelligent development of coal mines, the underground vision system is inevitably constrained by the low quality of the original material. Enhancing the video quality can provide a prerequisite for the good application of machine vision.

[0003] In principle, downhole image enhancement technology can be roughly divided into three categories: pure image processing methods, physical model driven methods and deep learning methods.

[0004] Image processing methods do not consider or rarely consider the scenarios in which the algorithms are applied, and only process the images through various image processing methods such as various transformations, morphological operations, filtering, interpolation, equalization, nonlinear operations, etc. These methods are effective but have limited effects and can no longer meet the needs of various current tasks for image clarity.

[0005] The physical model driven method introduces the real physical model into the image processing process. In essence, it models the process from clear to blurred images, and then seeks various parameters in the model through various inversion methods, and finally obtains a clear restored image in reverse. For example, for the scene of underground dust and fog obstruction, the common physical model is the atmospheric scattering model. As long as the atmospheric light value and transmittance are solved, the clear image can be restored. Since the physical model driven method takes into account the real physical process, the effect is better than the traditional image processing method. It is for this reason that the physical model driven image enhancement solution has high computational complexity and low efficiency.

[0006] Deep learning methods utilize the powerful mapping capabilities of neural networks, design appropriate network structures, and solidify the operators that transform images from blurry to clear in the network weights through training with large-scale data sets, thus achieving efficient enhancement of images. However, such methods face some problems. In the early stage, a lot of time and effort needs to be invested in building data sets and training models. The complexity of the above process leads to mixed results of such methods. In addition, large models have poor interpretability, unclear principles, poor portability, and face generalization problems in different scenarios.

[0007] In summary, all kinds of processing methods have their potential problems. Therefore, it is urgent to design a real-time video restoration method suitable for fully mechanized mining working faces to make up for the shortcomings of current video processing methods and thus improve the video restoration effect. Summary of the invention

[0008] The present invention aims to solve or improve the technical problem in the prior art that conventional video processing methods cannot be applied to different environments.

[0009] A first aspect of the present invention is to provide a method for restoring underground video.

[0010] A second aspect of the present invention is to provide a downhole video restoration device.

[0011] The third aspect of the present invention is to provide another downhole video restoration device.

[0012] A fourth aspect of the present invention is to provide a readable storage medium.

[0013] The first aspect of the present invention provides a method for restoring downhole video, comprising: acquiring downhole video; after receiving a preset instruction, determining that the current image of the downhole video is the original image; determining the atmospheric light value based on the original image; determining the transmittance based on the atmospheric light value and the original image; constructing a dark channel image restoration model based on the atmospheric light value, the original image and the transmittance; and restoring the downhole video based on the dark channel image restoration model.

[0014] The downhole video restoration method provided by the present invention first obtains the downhole video, and after receiving the preset instruction, determines that the current image of the downhole video is the original image; wherein the preset instruction can be a control instruction issued by the user through the interface interaction unit, for example, when the downhole video restoration device is started, the image of the first frame or the second frame is the original image, and then the atmospheric light value of the downhole is determined based on the original image, and then the transmittance is determined based on the atmospheric light value and the original image, and finally a dark channel image restoration model is constructed based on the atmospheric light value, the original image and the transmittance, so as to restore the downhole video according to the dark channel image restoration model. The downhole video restoration method provided by the present invention can determine the atmospheric light value of the downhole by collecting the original image of the downhole video, so as to restore the downhole image. By applying the dark channel prior algorithm to the video restoration task of the comprehensive mining working face, the downhole video can be restored in real time, effectively alleviating the negative impact of mining dust and fog on the quality of video monitoring images.

[0015] In some technical schemes, optionally, the downhole video restoration method also includes: acquiring a real-time image of the downhole video once every preset number of frames, and constructing a grayscale histogram of the real-time image; calculating the difference value of the grayscale histogram of two real-time images with similar time sequences, wherein similar time sequences refer to two adjacent real-time images in time order; when the difference value is greater than a preset difference value, determining that the latter of the two real-time images is a new original image, and based on the new original image, recalculating the atmospheric light value and transmittance, and reconstructing the dark channel image restoration model.

[0016] The present invention adopts the strategy of video frame calculation, presets the number of frames per interval, obtains a real-time image of the fully mechanized mining face video, and updates the dark channel image restoration model when the video picture changes drastically, thereby improving the stability and generalization performance of the entire solution, achieving the effect of real-time enhancement, and being able to adapt to changes in different environments with higher accuracy. However, in the currently disclosed technology, under normal circumstances, the model processing process is the same for different environments, which cannot be applied to video adjustment in different environments and has low accuracy.

[0017] In some technical solutions, optionally, the preset number of frames is greater than or equal to 5 frames and less than or equal to 15 frames.

[0018] In the technical solution, the preset number of frames is greater than or equal to 5 frames and less than or equal to 15 frames. For example, the preset number of frames is equal to 10 frames, that is, an image of the downhole video is captured once every 10 adjacent frames, and then the difference value of the grayscale histogram of the two adjacent images is determined. If the difference value is greater than the preset difference value, it means that the change of the environment is more drastic. At this time, the dark channel image restoration model is immediately updated, so that the downhole video restoration method of the present invention can adapt to the change of the environment and has higher adaptability.

[0019] In some technical solutions, optionally, the preset difference value is greater than or equal to 3% and less than or equal to 7%.

[0020] In this technical solution, the preset difference value is greater than or equal to 3% and less than or equal to 7%. For example, the preset difference value is equal to 5%. In this way, when the difference value of the grayscale histogram of two adjacent images is greater than or equal to 5%, it means that the environment has changed drastically. At this time, the dark channel image restoration model is updated to improve the accuracy. If the difference value is less than 5%, it means that the environment has not changed drastically, and the historical model data can be used at this time.

[0021] In some technical solutions, optionally, two real-time images with similar time sequence include a first image and a second image, and the difference value of the grayscale histogram of the two real-time images is calculated by the following relationship:

[0022] ;

[0023] ;

[0024] ;

[0025] ;

[0026] Where x is the pixel of the first image, y is the pixel of the second image, and h(H x ,H y ) is the histogram coefficient, is the number of pixel values ​​i among all pixels in the first image, For all The average value of is the number of pixel values ​​i among all pixels in the second image, For all The average value of , ∑ is the cumulative symbol, is the arithmetic square root, s(x,y) is the grayscale coefficient, x m is the pixel value of the mth pixel of the first image, is the average pixel value of all pixels in the first image, y m is the pixel value of the mth pixel of the second image, is the average pixel value of all pixels of the second image, n is the number of pixels of the first image or the second image, m(x, y) is the difference measure, α and β are weight constants, and Q is the difference value.

[0027] In some technical solutions, optionally, the step of restoring the downhole video based on the dark channel image restoration model specifically includes: converting the processing file of the dark channel image restoration model into ONNX (Open Neural Network Exchange) format (an open neural network exchange format), the processing file including the calculation formula of the atmospheric light value, the calculation formula of the transmittance and the restoration formula of the original image; serializing the processing file in the ONNX format into an engine file through TensorRT (tensor stream processor); inserting the engine file into the video processing model; and restoring the downhole video based on the video processing model after the engine file is inserted.

[0028] In this technical solution, the dark channel prior algorithm of the dark channel image restoration model is integrated into the video processing model to achieve end-to-end real-time video enhancement processing. Of course, in addition to serializing the dark channel image restoration model into an engine file and then inserting it into the video processing model, you can also call the DeepStream image preprocessing plug-in API (Application Programming Interface) to implement the dark channel prior algorithm, that is, to achieve image enhancement in the preprocessing stage of general computer vision tasks. Alternatively, you can first implement the dark channel prior algorithm, and then concatenate the DeepStream video stream to achieve general computer vision tasks and image enhancement at the same time. Both methods can achieve real-time video enhancement.

[0029] In some technical solutions, optionally, in the step of determining the transmittance based on the atmospheric light value and the original image, the atmospheric light value, the original image and the transmittance satisfy the following relationship:

[0030] ;

[0031] Where t(x) is the transmittance, Ω(x) is a rectangular area centered on pixel x, y is the pixel point in the Ω(x) area, C represents one of the R channel, G channel or B channel, A C is the atmospheric light value under the C channel, I C (y) is the pixel value of y in the original image under the C channel, is the minimum value of the ratio of the pixel value of y in the original image to the atmospheric light value under the C channel, For three channels The minimum value of It is a value between [0, 1].

[0032] In this technical solution, the entire image can be first divided into multiple regions according to n×n, each region is Ω(x), and the minimum value of the ratio of the pixel value of y on the original image to the atmospheric light value in each region is obtained, that is, in each channel, there must be a y pixel point in the Ω(x) region, whose pixel value on the original image has the smallest ratio to the atmospheric light value, so there will be three minimum values ​​under three channels, and then the minimum ratio under one of the three channels is taken twice, that is, the minimum value is taken again among the three minimum values ​​to obtain the transmittance of the Ω(x) region. For different regions, the transmittance of different regions can be obtained through this calculation formula.

[0033] In some technical solutions, optionally, in constructing a dark channel image restoration model, the relationship between the atmospheric light value, the original image, the transmittance and the restored image is as follows:

[0034] ;

[0035] Among them, x is the pixel point, J C (x) is the pixel value of the restored image under the C channel, I C (x) is the pixel value of the image before restoration under the C channel, A C is the atmospheric light value under the C channel, t(x) is the transmittance, t0 is 0.1, and max(t(x), t0) is the larger value between t(x) and t0.

[0036] In some technical schemes, optionally, the step of determining the atmospheric light value based on the original image specifically includes: constructing the R channel, G channel and B channel of the original image; determining the minimum pixel value of each pixel on the original image in the R channel, G channel or B channel; constructing a dark channel based on the minimum pixel value of all pixels; sorting all pixels on the dark channel from large to small according to pixel value; determining the first preset number of pixels as target pixels, that is, pixels with relatively strong light as target pixels, the preset number can be the first 0.1% of pixels in the sorting, and calculating the average pixel values ​​of all target pixels on the R channel, G channel and B channel in turn, and determining each average value as the atmospheric light value of the corresponding channel.

[0037] The second aspect of the present invention provides a downhole video restoration device, including: an acquisition unit, used to acquire downhole video; a processing unit, used to determine that the current image of the downhole video is the original image after receiving a preset instruction; the processing unit is also used to determine the atmospheric light value based on the original image; the processing unit is also used to determine the transmittance based on the atmospheric light value and the original image; the processing unit is also used to construct a dark channel image restoration model based on the atmospheric light value, the original image and the transmittance; the processing unit is also used to restore the downhole video based on the dark channel image restoration model.

[0038] The third aspect of the present invention provides a downhole video restoration device, including a memory and a processor, wherein a computer program or instruction is stored in the memory, and when the processor executes the computer program or instruction, a downhole video restoration method provided by any technical solution of the first aspect of the present application is implemented.

[0039] The fourth aspect of the present invention provides a readable storage medium, on which a program or instruction is stored. When a processor executes the program or instruction, a method for restoring downhole video provided by any technical solution of the first aspect of the present application is implemented.

[0040] Additional aspects and advantages of the present invention will become apparent in the following description or may be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] The above and / or additional aspects and advantages of the embodiments of the present invention will become apparent and easily understood from the description of the embodiments in conjunction with the following drawings, in which:

[0042] Figure 1 A first flow chart of a method for restoring downhole video provided by an embodiment of the present invention is shown;

[0043] Figure 2 A second flow chart of a method for restoring downhole video provided by an embodiment of the present invention is shown;

[0044] Figure 3 A structural block diagram of a downhole video restoration device provided by an embodiment of the present invention is shown;

[0045] Figure 4 A structural block diagram of another downhole video restoration device provided by an embodiment of the present invention is shown;

[0046] Figure 5 A histogram of a downhole image before restoration by a downhole video restoration method provided by an embodiment of the present invention is shown;

[0047] Figure 6 The histogram of the downhole image restored by the downhole video restoration method provided by the embodiment of the present invention is shown.

[0048] in, Figure 3 and Figure 4 The corresponding relationship between the reference numerals and component names in the figure is:

[0049] 1 a restoration device for a first downhole video, 13 an acquisition unit, 14 a processing unit, 2 a restoration device for a second downhole video, 22 a processor, 24 a memory. DETAILED DESCRIPTION

[0050] In order to more clearly understand the above aspects, features and advantages of the embodiments of the present invention, the embodiments of the present invention are further described in detail below in conjunction with the accompanying drawings and specific implementation methods. It should be noted that the embodiments of the present application and the features in the embodiments can be combined with each other without conflict.

[0051] like Figure 1 As shown, a method for restoring downhole video provided by an embodiment of the present invention includes the following steps:

[0052] S102: Acquire underground video;

[0053] S104: after receiving the preset instruction, determining that the current image of the downhole video is the original image;

[0054] S106: determining an atmospheric light value based on the original image;

[0055] S108: Determine transmittance based on the atmospheric light value and the original image;

[0056] S110: Construct a dark channel image restoration model based on the atmospheric light value, the original image and the transmittance;

[0057] S112: Restoring the underground video based on the dark channel image restoration model.

[0058] The method for restoring downhole video provided by the present invention includes first acquiring the downhole video, and after receiving a preset instruction, determining that the current image of the downhole video is the original image; wherein the preset instruction may be a control instruction issued by the user through an interface interaction unit, and then determining the atmospheric light value based on the original image, and then determining the transmittance based on the atmospheric light value and the original image, and finally constructing a dark channel image restoration model based on the atmospheric light value, the original image and the transmittance, thereby restoring the downhole video according to the dark channel image restoration model. The method for restoring downhole video provided by the present invention can determine the atmospheric light value in the well by acquiring the original image of the downhole video, thereby restoring the downhole image. wherein the histogram of the downhole image before restoration is as shown in Figure 5 As shown in Figure 2, the histogram of the restored downhole image is as follows: Figure 6 By applying the dark channel prior algorithm to the video restoration task of the fully mechanized mining face, the underground video can be restored in real time, effectively alleviating the negative impact of mining dust and fog on the quality of video surveillance images.

[0059] In some embodiments, optionally, the method for restoring downhole video also includes: acquiring a real-time image of the downhole video once every preset number of frames, and constructing a grayscale histogram of the real-time image; calculating the difference value of the grayscale histograms of two real-time images with similar timing, wherein similar timing refers to two adjacent real-time images in time order; when the difference value is greater than a preset difference value, determining that the latter of the two real-time images is a new original image, and based on the new original image, recalculating the atmospheric light value and transmittance, and reconstructing the dark channel image restoration model.

[0060] The present invention adopts the strategy of video frame calculation, presets the number of frames per interval, obtains a real-time image of the fully mechanized mining face video, and updates the dark channel image restoration model when the video picture changes drastically, thereby improving the stability and generalization performance of the entire solution, achieving the effect of real-time enhancement, and being able to adapt to changes in different environments with higher accuracy. However, in the currently disclosed technology, under normal circumstances, the model processing process is the same for different environments, which cannot be applied to video adjustment in different environments and has low accuracy.

[0061] In some embodiments, optionally, the preset number of frames is greater than or equal to 5 frames and less than or equal to 15 frames.

[0062] In this embodiment, the preset number of frames is greater than or equal to 5 frames, and less than or equal to 15 frames. For example, the preset number of frames is equal to 10 frames, that is, the image of the downhole video is captured once every 10 adjacent frames, and then the difference value of the grayscale histogram of the two adjacent images is determined. If the difference value is greater than the preset difference value, it means that the change of the environment is more drastic. At this time, the dark channel image restoration model is immediately updated, so that the downhole video restoration method of the present invention can adapt to the change of the environment and has higher adaptability. Of course, in order to reduce the amount of calculation, combined with the actual scene change law, a frame can also be extracted at fixed intervals for inter-frame difference verification. For example, the time interval is generally greater than or equal to 10 seconds and less than or equal to 30 seconds.

[0063] In some embodiments, optionally, the preset difference value is greater than or equal to 3% and less than or equal to 7%.

[0064] In this embodiment, the preset difference value is greater than or equal to 3% and less than or equal to 7%. For example, the preset difference value is equal to 5%. In this way, when the difference value of the grayscale histogram of two adjacent images is greater than or equal to 5%, it means that the change of the environment is relatively drastic. At this time, the dark channel image restoration model is updated to improve the accuracy. If the difference value is less than 5%, it means that the change of the environment is not drastic, and the historical model data can be used at this time.

[0065] In some embodiments, optionally, two real-time images with similar time sequence include a first image and a second image, and the difference value of the grayscale histogram of the two real-time images is calculated by the following relationship:

[0066] ;

[0067] ;

[0068] ;

[0069] ;

[0070] Where x is the pixel of the first image, y is the pixel of the second image, and h(H x ,H y ) is the histogram coefficient, is the number of pixel values ​​i among all pixels in the first image, For all The average value of is the number of pixel values ​​i among all pixels in the second image, For all The average value of , ∑ is the cumulative symbol, is the arithmetic square root, s(x,y) is the grayscale coefficient, x m is the pixel value of the mth pixel of the first image, is the average pixel value of all pixels in the first image, y m is the pixel value of the mth pixel of the second image, is the average pixel value of all pixels of the second image, n is the number of pixels of the first image or the second image, m(x, y) is the difference measure, α and β are weight constants, and Q is the difference value.

[0071] In some embodiments, optionally, the step of restoring the downhole video based on the dark channel image restoration model specifically includes: converting the processing file of the dark channel image restoration model into ONNX (Open Neural Network Exchange) format (an open neural network exchange format), the processing file including the calculation formula of the atmospheric light value, the calculation formula of the transmittance and the restoration formula of the original image; serializing the processing file in the ONNX format into an engine file through TensorRT; inserting the engine file into the video processing model; and restoring the downhole video based on the video processing model after the engine file is inserted.

[0072] In this embodiment, the dark channel prior algorithm of the dark channel image restoration model is integrated into the video processing model to achieve end-to-end real-time video enhancement processing. Of course, in addition to serializing the dark channel image restoration model into an engine file and then inserting it into the video processing model, you can also call the DeepStream image preprocessing plug-in API (Application Programming Interface) to implement the dark channel prior algorithm, that is, to achieve image enhancement in the preprocessing stage of general computer vision tasks. Alternatively, you can first implement the dark channel prior algorithm, and then connect the DeepStream video stream in series to achieve general computer vision tasks and image enhancement at the same time. Both methods can achieve real-time enhancement of the video.

[0073] In some embodiments, optionally, in the step of determining the transmittance based on the atmospheric light value and the original image, the atmospheric light value, the original image and the transmittance satisfy the following relationship:

[0074] ;

[0075] Where t(x) is the transmittance, Ω(x) is a rectangular area centered on pixel x, y is the pixel point in the Ω(x) area, C represents one of the R channel, G channel or B channel, A Cis the atmospheric light value under the C channel, I C (y) is the pixel value of y in the original image under the C channel, is the minimum value of the ratio of the pixel value of y in the original image to the atmospheric light value under the C channel, For three channels The minimum value of It is a value between [0, 1].

[0076] In this embodiment, the entire image can be first divided into multiple regions according to n×n, each region is Ω(x), and the minimum value of the ratio of the pixel value of y on the original image to the atmospheric light value in each region is obtained, that is, in each channel, there must be a y pixel point in the Ω(x) region, whose pixel value on the original image has the smallest ratio to the atmospheric light value, so there will be three minimum values ​​under three channels, and then the minimum ratio under one of the three channels is taken twice, that is, the minimum value is taken again among the three minimum values ​​to obtain the transmittance of the Ω(x) region. For different regions, the transmittance of different regions can be obtained by this calculation formula.

[0077] In some embodiments, optionally, in constructing a dark channel image restoration model, the relationship between the atmospheric light value, the original image, the transmittance and the restored image is as follows:

[0078] ;

[0079] Among them, x is the pixel point, J C (x) is the pixel value of the restored image under the C channel, I C (x) is the pixel value of the image before restoration under the C channel, A C is the atmospheric light value under the C channel, t(x) is the transmittance, t0 is 0.1, and max(t(x), t0) is the larger value between t(x) and t0.

[0080] In some embodiments, optionally, the step of determining the atmospheric light value based on the original image specifically includes: constructing the R channel, G channel and B channel of the original image; determining the minimum pixel value of each pixel on the original image in the R channel, G channel or B channel; constructing a dark channel based on the minimum pixel value of all pixels; sorting all pixels on the dark channel from large to small according to pixel value; determining the first preset number of pixels as target pixels, that is, pixels with relatively strong light as target pixels, and the preset number can be the first 0.1% of pixels in the sorting; calculating the average pixel values ​​of all target pixels on the R channel, G channel and B channel in turn, and determining each average value as the atmospheric light value of the corresponding channel.

[0081] Next, the applicant introduces the downhole video restoration method of the present invention from another perspective.

[0082] First of all, it should be stated that the following step S2 of the present invention involves the reasoning process of the calculation formula. The corresponding relationship between the formula symbols and the meanings of all the formulas mentioned in step S2 (step S2 includes S21, S22, S23 and S24) is as follows:

[0083] J dark (x) is the dark channel image, r is the R channel, g is the G channel, b is the B channel, C is one of the R channel, G channel or B channel, x is a pixel, I(x) is the pixel value of the image before restoration, J(x) is the pixel value of the image after restoration, t(x) is the transmittance, A is the atmospheric light value, min represents the minimum value, max represents the maximum value, y is the pixel point in the Ω(x) area, Ω(x) is a rectangular area centered on the pixel point x, t0 is 0.1, I(y) is the pixel value of y in the original image, and J(y) is the pixel value of y in the restored image;

[0084] In addition, when C is used as a superscript, it represents the conditions under the C channel, for example, J C (x) is the pixel value of the restored image under the C channel, and the rest are similar.

[0085] like Figure 2 As shown, another embodiment of the present invention provides a method for restoring downhole video, comprising the following steps:

[0086] S2: Establish a dark channel image restoration model.

[0087] Among them, this step mainly includes the following four steps: S21, S22, S23 and S24.

[0088] S21: Calculate the dark channel from the original image.

[0089] Among them, the dark channel is a basic assumption, that is, in the local area of ​​most images, some pixels will always have at least one color channel with a very low value. The dark channel of the image can be expressed as follows:

[0090] ; Dark channel a priori theory states:

[0091] (That is, the pixel value in the dark channel approaches 0);

[0092] The dark channel is actually obtained by taking the minimum value of the three channels, R channel, G channel and B channel, to form a grayscale image, and then performing a minimum value filter. A key feature of the dark channel is that in the pixel area without dust and fog, the pixel value of the dark channel will be very low, close to 0. In the image with dust and fog, the dust and fog area will be reflected in the dark channel, because the dust and fog will scatter light, resulting in a relatively high pixel value in the dark channel.

[0093] S22: Estimate the atmospheric light value using the dark channel.

[0094] Select the top 0.1% of pixels in the dark channel sorted by brightness, and calculate the average values ​​of the three channels, R, G, and B, in the original image at these points. That is, the average value of these pixels in each channel, R, G, or B, is the atmospheric light value of the corresponding channel.

[0095] S23: Get the transmittance from the dark channel.

[0096] Specifically, the dust cloud formation model described by the following formula is selected:

[0097] ;

[0098] Transform the above formula to get:

[0099] ;

[0100] Assuming that the transmittance in each channel is constant and the value of A is given in step S22, the above equation is subjected to two minimum operations:

[0101] The first minimization operation yields:

[0102] ;

[0103] Divide both sides of the equation by A C get:

[0104] ;

[0105] The second minimum operation yields:

[0106] ;

[0107] Since in the above formula is the restored image to be sought under the C channel, so its dark channel should be consistent with the prior, that is:

[0108] ;

[0109] Then we have:

[0110] ;

[0111] Thus we can deduce:

[0112] ;

[0113] This is the estimated transmittance.

[0114] In practical applications, the above formula is corrected to obtain the transmittance expressed by the following formula:

[0115] ;

[0116] in It is a value between [0, 1], usually 0.95.

[0117] S24: Process the original image according to the atmospheric light value and transmittance to restore a clear image.

[0118] The specific calculation formula is as follows:

[0119] ;

[0120] Among them, t0 prevents the image from being overexposed, and is generally set to 0.1.

[0121] S3: Serialize the dark channel image restoration model into an engine file.

[0122] Among them, this step mainly includes the following three steps: S31, S32 and S33.

[0123] S31: Encapsulate the dark channel image enhancement algorithm into a custom network model.

[0124] Specifically, first create a class in the PyTorch framework (a deep learning framework) that inherits from nn.Module (a base class of a neural network module); initialize the model's layers and parameters in the model class's constructor init; in the init method, instantiate all required layers and parameters and use them as class attributes, implement the forward method (a method in the PyTorch framework), that is, implement the dark channel prior image enhancement algorithm, and integrate the model's layers and forward propagation logic into a complete class.

[0125] S32: Convert the above network model into ONNX format.

[0126] Specifically, before exporting the model, make sure the model is set to evaluation mode; create a set of input tensors whose dimensions and types should match the input of the model; use the torch.onnx.export function (a function that exports to ONNX format) to export the model, input tensors, output file name, and other optional parameters; use ONNX's API (Application Programming Interface) to check whether the ONNX model is valid; to verify that the output of the model is consistent, use ONNX Runtime (an open source engine for optimizing and accelerating machine learning inference and training) to run the exported ONNX model and compare it with the output of PyTorch.

[0127] S33: Serialize the above ONNX format model to the DeepStream engine.

[0128] Specifically, creating an engine using the TensorRT (tensor stream processor) API usually involves creating a builder (a class or interface responsible for creating and configuring objects), loading the ONNX model, building the engine, and serializing it.

[0129] S4: Integrate the serialized engine file into the DeepStream video stream processing framework.

[0130] Specifically, the serialized TensorRT engine is used in the DeepStream pipeline (a deep learning video analysis platform developed by NVIDIA).

[0131] S5: Adaptive adjustment of dark channel prior algorithm during real-time video restoration processing.

[0132] This step mainly includes the following two steps: S51 and S52.

[0133] S51: Calculate the grayscale histogram of the image every 10 frames.

[0134] S52: Calculate the statistical difference between the two grayscale histograms.

[0135] If the statistical difference between the two grayscale histograms (i.e., the difference value of the grayscale histogram of the present invention) is greater than 5%, it means that the environment has changed dramatically and the atmospheric light value and transmittance need to be recalculated. If the statistical difference between the two grayscale histograms is small, it means that the environment has not changed much, and the historical model data is used.

[0136] It should be noted that in the actual production process, factors such as underground personnel activities and equipment operation will cause drastic changes in the environment, which are particularly evident in the following aspects. 1. The lights carried by personnel cause large brightness changes in the local video frame; 2. The operation of the equipment brings a large amount of coal dust, which reduces the quality of the video frame; 3. The rotation of the pan-tilt camera causes changes in the scene. Changes in these factors are ultimately reflected in changes in parameters such as video frame brightness and contrast. Therefore, the distribution histogram is used to characterize the differences in video frame brightness and contrast, and the image correlation coefficient is used to characterize the differences in video frame background and object structure.

[0137] The step of calculating the statistical difference between the two grayscale histograms before and after S52 includes:

[0138] Step S521: Calculate the grayscale histograms of the previous and next images.

[0139] Step S522: Calculate the correlation coefficient of the grayscale histogram.

[0140] ;

[0141] Where x is the pixel of the first image, y is the pixel of the second image, and h(H x ,H y ) is the histogram coefficient, is the number of pixel values ​​i among all pixels in the first image, For all The average value of is the number of pixel values ​​i among all pixels in the second image, For all The average value of , ∑ is the cumulative symbol, is the arithmetic square root.

[0142] Step S523: Calculate the grayscale coefficient.

[0143] ;

[0144] Among them, s(x,y) is the grayscale coefficient, x m is the pixel value of the mth pixel of the first image, is the average pixel value of all pixels in the first image, y m is the pixel value of the mth pixel of the second image, is the average of the pixel values ​​of all pixels in the second image, n is the number of pixels in the first image or the second image, m(x,y) is the difference measure, ∑ is the cumulative symbol, is the arithmetic square root.

[0145] Step S524: Calculate the difference value.

[0146] By adjusting the two weights using α and β, the difference measure m satisfies symmetry and boundedness, and its maximum value is 1. The closer it is to 1, the higher the similarity between frames is, which is calculated by the following formula:

[0147] ;

[0148] Among them, m(x,y) is the difference measure, h(H x ,H y ) is the histogram coefficient, s(x,y) is the grayscale coefficient, α and β are weight constants, and the sum of α and β is equal to 1.

[0149] Finally, the difference value is calculated according to the difference measure:

[0150] ; Where Q is the difference value;

[0151] When the difference measure of x and y between the current and the next two frames is less than a threshold, for example, 0.95, it means that the difference value Q is greater than 5%. The atmospheric light value and transmittance are recalculated. If the difference between the previous and the next two frames is small, the historical model data is used.

[0152] The effects of the downhole video restoration method of the present invention are mainly in the following aspects:

[0153] (1) The present invention applies the dark channel prior algorithm to the video restoration task of the fully mechanized mining face, effectively alleviating the negative impact of mining dust and fog on the quality of video monitoring images, and as a preprocessing method, provides high-quality material input for the application of computer vision algorithms;

[0154] (2) The present invention uses the PyTorch framework and the TensorRT framework to serialize the dark channel prior algorithm into an engine file, and integrates it into the DeepStream video stream processing framework, thereby realizing end-to-end real-time video restoration processing. At the same time, the dark channel prior algorithm can be used as an "enhancement layer" plug-in compatible with other computer vision network models. Among them, PyTorch is used to convert the dark channel prior algorithm into ONNX format, TensorRT is used to serialize the ONNX format model into an engine file, and the dark channel prior engine file is integrated into the DeepStream inference plug-in to realize real-time video restoration;

[0155] (3) The present invention adopts a strategy of video frame-to-frame calculation and updates the dark channel prior model only when the video picture changes drastically, thereby improving the stability and generalization performance of the entire scheme and achieving a real-time enhancement effect.

[0156] like Figure 3As shown, the second aspect of the present invention provides a downhole video restoration device, and the first downhole video restoration device 1 includes: an acquisition unit 13, used to acquire the downhole video; a processing unit 14, used to determine that the current image of the downhole video is the original image after receiving a preset instruction; the processing unit 14 is also used to determine the atmospheric light value based on the original image; the processing unit 14 is also used to determine the transmittance based on the atmospheric light value and the original image; the processing unit 14 is also used to construct a dark channel image restoration model based on the atmospheric light value, the original image and the transmittance; the processing unit 14 is also used to restore the downhole video based on the dark channel image restoration model.

[0157] like Figure 4 As shown, the third aspect of the present invention provides another downhole video restoration device, the second downhole video restoration device 2 includes a memory 24 and a processor 22, the memory 24 stores a computer program or instructions, and when the processor 22 executes the computer program or instructions, it implements the downhole video restoration method provided in any embodiment of the first aspect of the present application.

[0158] The fourth aspect of the present invention provides a readable storage medium, on which a program or instruction is stored. When a processor executes the program or instruction, a method for restoring a downhole video as provided in any embodiment of the first aspect of the present application is implemented.

[0159] In the embodiments according to the present invention, the terms "first", "second", and "third" are only used for description and cannot be understood as indicating or implying relative importance; the term "plurality" refers to two or more, unless otherwise clearly defined. The terms "installed", "connected", "connected", "fixed" and the like should be understood in a broad sense. For example, "connected" can be a fixed connection, a detachable connection, or an integral connection; "connected" can be a direct connection or an indirect connection through an intermediate medium. For those of ordinary skill in the art, the specific meanings of the above terms in the embodiments according to the present invention can be understood according to specific circumstances.

[0160] In addition, although each operation is described in a specific order, this should be understood as requiring such operation to be performed in a specific order or in a sequential order, or requiring that all illustrated operations should be performed to obtain desired results. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present invention. Some features described in the context of a separate embodiment can also be implemented in a single implementation in combination. On the contrary, the various features described in the context of a single implementation can also be implemented in multiple implementations individually or in any suitable sub-combination mode.

[0161] Although the subject matter has been described in the language of specific structural features and / or method logical actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims.

[0162] The above are only preferred embodiments according to the embodiments of the present invention, and are not intended to limit the embodiments of the present invention. For those skilled in the art, various modifications and changes may be made according to the embodiments of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the embodiments of the present invention shall be included in the protection scope of the embodiments of the present invention.

Claims

1. A method for restoring underground video, It is characterized in that include: Get downhole video; After receiving a preset instruction, determining that the current image of the downhole video is the original image, the preset instruction is a control instruction issued by the user through the interface interaction unit; determining an atmospheric light value based on the original image; Determine transmittance based on the atmospheric light value and the original image; Constructing a dark channel image restoration model based on the atmospheric light value, the original image and the transmittance; Restoring the downhole video based on the dark channel image restoration model; Acquire a real-time image of the downhole video once at a preset frame number interval, and construct a grayscale histogram of the real-time image, wherein the preset frame number is greater than or equal to 5 frames and less than or equal to 15 frames; Calculate the difference value of the grayscale histogram of two real-time images with similar time series; When the difference value is greater than a preset difference value, determining the latter of the two real-time images as a new original image, and recalculating the atmospheric light value and transmittance based on the new original image, and reconstructing the dark channel image restoration model, wherein the preset difference value is greater than or equal to 3% and less than or equal to 7%; The two real-time images with similar time sequence include a first image and a second image, and the difference value of the grayscale histogram of the two real-time images is calculated by the following relationship: ; ; ; ; Where x is the pixel of the first image, y is the pixel of the second image, and h(H x ,H y ) is the histogram coefficient, is the number of pixel values ​​i among all pixels in the first image, For all The average value of is the number of pixel values ​​i among all pixels in the second image, For all The average value of , ∑ is the cumulative symbol, is the arithmetic square root, s(x,y) is the grayscale coefficient, x m is the pixel value of the mth pixel of the first image, is the average pixel value of all pixels in the first image, y m is the pixel value of the mth pixel of the second image, is the average pixel value of all pixels of the second image, n is the number of pixels of the first image or the second image, m(x,y) is the difference measure, α and β are weight constants, and Q is the difference value; The step of restoring the downhole video based on the dark channel image restoration model specifically includes: Converting a processing file of the dark channel image restoration model into an ONNX format, wherein the processing file includes a calculation formula for atmospheric light value, a calculation formula for transmittance, and a restoration formula for the original image; Serialize the ONNX format processing file into an engine file through TensorRT; Inserting the engine file into the video processing model; Based on the video processing model after the engine file is inserted, the downhole video is restored.

2. The downhole video restoration method according to claim 1, It is characterized in that In the step of determining the transmittance based on the atmospheric light value and the original image, the atmospheric light value, the original image and the transmittance satisfy the following relationship: ; Where t(x) is the transmittance, Ω(x) is a rectangular area centered on pixel x, y is the pixel point in the Ω(x) area, C represents one of the R channel, G channel or B channel, A C is the atmospheric light value under the C channel, I C (y) is the pixel value of y in the original image under the C channel, is the minimum value of the ratio of the pixel value of y in the original image to the atmospheric light value under the C channel, For three channels The minimum value of It is a value between [0, 1].

3. The downhole video restoration method according to claim 1, It is characterized in that In the dark channel image restoration model, the relationship between the atmospheric light value, the original image, the transmittance and the restored image is as follows: ; Among them, x is the pixel point, J C (x) is the pixel value of the restored image under the C channel, I C (x) is the pixel value of the image before restoration under the C channel, A C is the atmospheric light value under the C channel, t(x) is the transmittance, t 0 is 0.1, max(t(x), t 0 ) is t(x) and t 0 Take the larger value.

4. The downhole video restoration method according to claim 1, It is characterized in that The step of determining the atmospheric light value based on the original image specifically includes: Constructing the R channel, G channel and B channel of the original image; Determine the minimum pixel value of each pixel on the original image in the R channel, the G channel or the B channel; Construct a dark channel based on the minimum pixel value of all pixels; Sort all the pixels on the dark channel from large to small according to pixel values; Determine a preset number of pixels as target pixels; The average values ​​of the pixel values ​​of all the target pixels on the R channel, the G channel, and the B channel are calculated in sequence, and each average value is determined as the atmospheric light value of the corresponding channel.

5. A downhole video restoration device, It is characterized in that include: An acquisition unit, used for acquiring downhole video; A processing unit, configured to determine that the current image of the downhole video is the original image after receiving a preset instruction, wherein the preset instruction is a control instruction issued by the user through the interface interaction unit; The processing unit is further configured to determine an atmospheric light value based on the original image; The processing unit is further used to determine the transmittance based on the atmospheric light value and the original image; The processing unit is further used to construct a dark channel image restoration model based on the atmospheric light value, the original image and the transmittance; The processing unit is further used to restore the downhole video based on the dark channel image restoration model; Acquire a real-time image of the downhole video once at a preset frame number interval, and construct a grayscale histogram of the real-time image, wherein the preset frame number is greater than or equal to 5 frames and less than or equal to 15 frames; Calculate the difference value of the grayscale histogram of two real-time images with similar time sequence; when the difference value is greater than a preset difference value, determine that the latter of the two real-time images is a new original image, and recalculate the atmospheric light value and transmittance based on the new original image, and reconstruct the dark channel image restoration model; Wherein, the two real-time images with similar time sequence include a first image and a second image, and the processing unit can calculate the difference value of the grayscale histogram of the two real-time images by the following relationship: ; ; ; ; Where x is the pixel of the first image, y is the pixel of the second image, and h(H x ,H y ) is the histogram coefficient, is the number of pixel values ​​i among all pixels in the first image, For all The average value of is the number of pixel values ​​i among all pixels in the second image, For all The average value of , ∑ is the cumulative symbol, is the arithmetic square root, s(x,y) is the grayscale coefficient, x m is the pixel value of the mth pixel of the first image, is the average pixel value of all pixels in the first image, y m is the pixel value of the mth pixel of the second image, is the average pixel value of all pixels of the second image, n is the number of pixels of the first image or the second image, m(x,y) is the difference measure, α and β are weight constants, and Q is the difference value; When the processing unit restores the downhole video based on the dark channel image restoration model, it is specifically used to: Converting a processing file of the dark channel image restoration model into an ONNX format, wherein the processing file includes a calculation formula for atmospheric light value, a calculation formula for transmittance, and a restoration formula for the original image; Serialize the ONNX format processing file into an engine file through TensorRT; Inserting the engine file into the video processing model; Based on the video processing model after the engine file is inserted, the downhole video is restored.

6. A downhole video restoration device, It is characterized in that include: A memory and a processor, wherein the memory stores a computer program or an instruction, and the processor implements the downhole video restoration method as described in any one of claims 1 to 4 when executing the computer program or the instruction.

7. A readable storage medium, It is characterized in that The readable storage medium stores a program or an instruction, and when the processor executes the program or the instruction, the method for restoring the downhole video according to any one of claims 1 to 4 is implemented.