A method for processing downhole monitoring video data based on image frame similarity screening

Abstract

The application discloses a kind of based on image frame similarity screening underground monitoring video data processing method, it is related to image processing technical field, and dynamic picture video segment and static picture video segment are obtained by splitting monitoring video;Determine the first image frame in dynamic picture video segment as key frame and obtain the second image frame with key frame similarity greater than preset threshold, first image frame and second image frame are all as dynamic storage image frame;Any image frame is selected as static storage image frame from static picture video segment and is stored together with dynamic storage image frame and image frame key information as the storage data of entire monitoring video and is stored.The static part selects any frame image, and dynamic part selects key frame, according to similarity comparison, the image frame similar to key frame is excluded, and the image frame reserved represents the key information picture of dynamic part.The image frame obtained retains the key information of monitoring video, and the effective use of storage space is realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to a downhole monitoring video data processing method based on image frame similarity screening. BACKGROUND

[0002] Coal mine safety production is related to the safe development of the mine and the lives of the workers. One ventilation and three prevention is the top priority of coal mine safety. With the development of the mine and the increase of the range, the ventilation system becomes more and more complex, the ventilation and prevention facilities become more and more, the mine ventilation and prevention inspection work becomes more and more heavy, and the goaf situation becomes more and more complex. Safety accidents occur every year due to the failure of the inspection personnel to inspect in place and in time, and the management of the on-site supervision inspection personnel is increasingly difficult.

[0003] In order to ensure the safety production of the coal mine in the traditional technology, various video monitoring devices are generally arranged in the downhole, and the video monitoring devices transmit the downhole monitoring video to the uphole monitoring room in real time. In general, the monitoring video is displayed on the screen after being transmitted to the monitoring room, and the video is stored. However, most of the video pictures in the downhole are regular operation videos, and the non-operation time is a long time of static pictures, which are useless video information. Storing useless information not only wastes storage space, but also affects the efficiency of the later verification. Chinese patent CN117112833B discloses a video static frame filtering method and device based on storage space optimization, which screens out non-key frames of the static frame sequence, estimates the key frame structure similarity index, and screens out key frames with low similarity. The method reduces the storage cost of the data generated by the monitoring video and improves the storage efficiency, avoids the occupation of a large amount of storage space in the storage process, causes the low efficiency, and still maintains the quality of the video by retaining the key frames with the most representative and information richness.

[0004] The method provided in the above patent can process the video to a certain extent, but the above method retains the image frames with high similarity index with the key by the similarity index in order to ensure the richness of the video. Although it does not affect the quality of the video, too many similar videos still have too much information redundancy, which still causes waste of storage space. SUMMARY

[0005] In order to solve the above technical problems, the present application provides the following technical solutions:

[0006] In a first aspect, the present application provides a downhole monitoring video data processing method based on image frame similarity screening, comprising:

[0007] Splitting the monitoring video to obtain dynamic picture video segments and static picture video segments;

[0008] determining a first image frame in the dynamic picture video segment as a key frame and obtaining a second image frame with a similarity greater than a preset threshold to the key frame, the first image frame and the second image frame are both stored as dynamic stored image frames;

[0009] selecting any image frame from the static picture video segment as a static stored image frame and storing the dynamic stored image frames and the static stored image frame together with image frame key information as storage data of the entire monitoring video.

[0010] In a possible implementation, the splitting of the monitoring video to obtain the dynamic picture video segment and the static picture video segment comprises:

[0011] determining a downhole monitoring device and a key operation area in the monitoring video;

[0012] if there is no change in the indicator light interface of the downhole monitoring device and no downhole personnel intervention operation in the key operation area within a preset time period, the above video segment is a static picture video segment; otherwise, the above video segment is a dynamic picture video segment.

[0013] In a possible implementation, the detection of the indicator light interface of the downhole monitoring device comprises:

[0014] high-pass filtering the image frame to extract edge and line information in the image frame;

[0015] calculating a gradient map of the image frame to obtain gradient information in the image frame;

[0016] non-maximum suppression is performed on the gradient map to eliminate overlap between adjacent key points;

[0017] mean filtering is performed on the gradient map to eliminate noise;

[0018] peak detection is performed on the mean filtered gradient map to find a key area on the downhole monitoring device.

[0019] In a possible implementation, a model for detecting the key area of the downhole monitoring device is:

[0020]

[0021]

[0022] wherein: is a gradient vector, is a gradient map, is an original image, is a high-pass filter, is a horizontal gradient, is a vertical gradient, is a gradient operator.

[0023] In a possible implementation, the identification of the downhole personnel in the key operation area comprises:

[0024] Read the video file using the cv2.VideoCapture function;

[0025] Create an MOG2 background subtractor object by cv2.createBackgroundSubtractorMOG2;

[0026] Read each frame of the video through an infinite loop, and process the current frame using the background subtractor to obtain the foreground mask;

[0027] Apply morphological opening operation to the foreground mask to remove small noise and separate the adhered objects;

[0028] Find the contour in the processed foreground mask;

[0029] Iterate through all the found contours, calculate the perimeter of the contour, if the perimeter is greater than a preset value, consider it as a moving downhole personnel, and calculate its bounding rectangle, draw a rectangular frame on the original frame to identify the downhole personnel.

[0030] In a possible implementation, the determining the first image frame in the dynamic picture video segment as a key frame and obtaining a second image frame with similarity greater than a preset threshold to the key frame, the first image frame and the second image frame are both storage image frames, comprising:

[0031] If the indicator light interface in the dynamic picture video segment appears a pre-warning and / or the device in the key operation area exists human intervention, the picture at the current time is taken as the first image frame;

[0032] The second image frame is determined through similarity comparison, the first image frame is taken as a storage selected frame, and the second image frame is converted into a key frame;

[0033] Continue to perform similarity comparison until all image frames in the dynamic picture video segment are compared, and all storage image frames are obtained.

[0034] In a possible implementation, the determining the second image frame through similarity comparison comprises:

[0035] Reduce the size of the first image frame and the compared image frame to simplify the calculation of DCT: img = cv2.resize(img, (long, width));

[0036] Convert the image frame into a grayscale image to further simplify the calculation: gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY);

[0037] Convert the grayscale image into a floating-point type and perform DCT transformation to obtain a DCT coefficient matrix: dct = cv2.dct(np.float32(gray));

[0038] Downsize the DCT coefficient matrix to retain only the part that presents the lowest frequency in the image frame to obtain a new DCT coefficient matrix;

[0039] Calculate the DCT mean value of the new DCT coefficient matrix;

[0040] Calculate the hash value, set 0 or 1 to the hash value of 64 bits, greater than or equal to the DCT mean value is set to 1, less than the DCT mean value is set to 0, combined together, which constitutes a 64-bit integer as the fingerprint information of the image frame;

[0041] Compare the fingerprint information of the first image frame and the comparison image frame, if the difference of the fingerprint information is greater than the preset threshold, the comparison image frame is taken as the second image frame.

[0042] In a possible implementation, any image frame is selected from the static picture video segment as a static storage image frame and is stored together with the dynamic storage image frame and image frame key information as the storage data of the entire monitoring video, including:

[0043] Determine the time interval information of the static picture video segment and the time node information of each dynamic storage image frame;

[0044] Combine the underground position corresponding to the monitoring video with the time interval information of the static picture video segment to determine the image frame key information of the static storage image frame;

[0045] Combine the underground position corresponding to the monitoring video with the time node information of each dynamic storage image frame to determine the image frame key information of different dynamic storage image frames;

[0046] Associate the image frame key information to the corresponding storage image frame to generate the storage data of the entire monitoring video.

[0047] In the embodiment of the present application, the static part of the monitoring video basically does not have key information, and any selected frame image can represent it; after the key frame is selected according to the preset condition, the image frames similar to the key frame are excluded according to the similarity comparison, and the remaining image frames basically represent the key information pictures of the dynamic part. The image frames obtained above are stored as the data of the monitoring video, which can realize the effective use of the storage space while retaining the key information of the monitoring video. BRIEF DESCRIPTION OF DRAWINGS

[0048] Figure 1 A flowchart of a downhole monitoring video data processing method based on image frame similarity screening provided by the embodiment of the present application is shown in the figure.

[0049] Figure 2 A static part of a monitoring video provided by the embodiment of the present application is shown in the figure.

[0050] Figure 3 A dynamic part of a monitoring video provided by the embodiment of the present application is shown in the figure.

[0051] Figure 4 A monitoring video segment division provided by the embodiment of the present application is shown in the figure.

[0052] Figure 5 A dynamic picture video segment frame sequence provided by the embodiment of the present application is shown in the figure.

[0053] Figure 6 An image frame difference comparison provided by the embodiment of the present application is shown in the figure. DETAILED DESCRIPTION

[0054] The present scheme will be described below in combination with the accompanying drawings and specific embodiments.

[0055] Referring to Figure 1 The downhole monitoring video data processing method based on image frame similarity screening provided by the embodiment of the present application comprises:

[0056] S101, splitting the monitoring video to obtain a dynamic picture video segment and a static picture video segment.

[0057] The coal mine underground is a special working environment, with flammable and explosive combustible gas and corrosive gas, humidity, water, large amount of mine dust, large voltage fluctuation of power grid, serious electromagnetic interference, small space, long monitoring distance. Therefore, a large number of monitoring devices are arranged in the coal mine underground. The monitoring devices are mainly used to monitor the concentration of methane, carbon monoxide, carbon dioxide, oxygen, hydrogen sulfide, mine dust, wind speed, wind pressure, humidity, temperature, power feeding state, air door state, air duct state, local fan start-stop, main fan start-stop, etc.

[0058] Downhole monitoring equipment monitors key work areas. In addition, to ensure timely understanding of the status of the monitoring equipment, there is also dedicated monitoring equipment. The indicator light interface of the monitoring equipment is the most representative of its working status. The indicator light interface can not only represent the status of the monitoring equipment, but also indicate the detected danger information.

[0059] Therefore, in this embodiment, after identifying the downhole monitoring equipment and key operating areas in the monitoring video, the dynamic and static images in the monitoring video are determined based on the above two aspects. If the indicator light interface of the downhole monitoring equipment does not change within a preset time period and there is no downhole personnel interfering with the operation in the key operating area, then the above video segment is considered a static image video segment; otherwise, the above video segment is considered a dynamic image video segment. Figure 2 and Figure 3 As shown, Figure 2 The middle section contains downhole monitoring equipment and downhole operation equipment. Since both the monitoring equipment and the downhole operation equipment are operating normally, this type of video segment is a still image video segment. However, Figure 3 The video clips containing underground workers are dynamic video clips.

[0060] by Figure 4 For example, in the initial video segment a, the indicator lights on the monitoring equipment do not change, and no personnel are present in the key work area. However, in the adjacent video segment b, personnel intervention occurs. In video segment c, although no one is present, the indicator lights display an alarm. Therefore, Figure 2 In this context, video segment 'a' is considered a static video segment, while video segments 'b' and 'c' are considered dynamic video segments.

[0061] Specifically, in this embodiment, the indicator light interface of the downhole monitoring equipment is detected, including: high-pass filtering the monitoring image frame to extract edge and line information from the image frame; calculating the gradient map of the image frame to obtain gradient information; performing non-maximum suppression on the gradient map to eliminate overlap between adjacent key points; performing mean filtering on the gradient map to eliminate noise; and performing peak detection on the mean-filtered gradient map to find key areas on the downhole monitoring equipment.

[0062] The model for detecting key areas of downhole monitoring equipment in this embodiment is as follows:

[0063]

[0064]

[0065] in: It is the gradient vector. It is a gradient map. is a raw image, is a high-pass filter, is a horizontal gradient, is a vertical gradient, is a gradient operator.

[0066] In the embodiment, the identification of the downhole personnel in the key operation area includes: reading a video file by using a cv2.VideoCapture function; creating an MOG2 background subtractor object by using a cv2.createBackgroundSubtractorMOG2 function. Each frame of the video is read through an infinite loop, and the current frame is processed by using the background subtractor to obtain a foreground mask; the morphological opening operation is applied to the foreground mask to remove small noise and separate the adhered objects; the contours are searched in the processed foreground mask; all the found contours are traversed, the perimeter of the contour is calculated, if the perimeter is greater than a preset value, it is considered that a moving downhole personnel exists, and the bounding rectangle thereof is calculated, and a rectangular frame is drawn on the original frame to identify the downhole personnel.

[0067] S102, determine a first image frame in the dynamic picture video segment as a key frame and obtain a second image frame with a similarity greater than a preset threshold to the key frame, and the first image frame and the second image frame are both stored as dynamic image frames.

[0068] If the indicator interface in the dynamic picture video segment appears a pre-warning and / or the device in the key operation area exists human intervention. In order to find out the first image frame as the key frame, in the embodiment, each image frame in the dynamic picture video segment is extracted, and the first image frame with the indicator interface of the monitoring device appearing an alarm or the downhole personnel is selected as the preferred key frame in the embodiment.

[0069] The second image frame is determined through the similarity comparison, and the first image frame is stored as a selected frame. In the embodiment, the frame similarity comparison is performed according to the time sequence of the dynamic picture video segment according to the selected key frame. When the second image frame with a predetermined difference from the first image frame is selected, the third image frame meeting the difference requirement with the second image frame needs to be further searched, and the same is true for the subsequent image frames. Therefore, after the first image frame is stored as the selected frame, the second image frame is converted into the key frame, and the similarity comparison is continuously performed until all the image frames in the dynamic picture video segment are compared, and all the stored image frames are obtained.

[0070] Referring to Figure 5, an illustrative dynamic picture video segment frame sequence, if the D image frame is selected as the key frame, first compare to the right side to determine the F image frame as the second image frame, then continue to compare to the right side with the F image frame as the key frame. After the same right side comparison is completed, continue to compare to the left side with the D image frame. The above is only an illustrative example, and the comparison order is not specifically limited.

[0071] It should be noted that, in order to illustrate that the D image frame and the F image frame are only separated by 1 image frame, in practice, there can be multiple, and the reason why some of the dynamic pictures are filtered out is that the similarity with the key frame is large, and the key frame can represent this series of image frames. For example, if the key frame shows that the downhole personnel intervene the equipment operation, and the downhole personnel remain unchanged for several seconds, then the image frames corresponding to the several seconds are almost the same as the key frame, and need to be filtered out. The other cases are the same as described above, and will not be repeated here.

[0072] Specifically, in the embodiment, the second image frame is determined through similarity comparison, including: reducing the size of the first image frame and the comparison image frame to simplify the calculation of DCT: img = cv2.resize(img, (long, wide)). The image frame is converted into a gray image to further simplify the calculation amount: gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY). The gray image is converted into a floating-point type, and then DCT transformation is performed to obtain a DCT coefficient matrix: dct = cv2.dct(np.float32(gray)). The DCT coefficient matrix is reduced to only retain the part presenting the lowest frequency in the image frame to obtain a new DCT coefficient matrix, and the DCT mean value of the new DCT coefficient matrix is calculated. The hash value is calculated, and 0 or 1 is set to the hash value of 64 bits. Greater than or equal to the DCT mean value is set to 1, and less than the DCT mean value is set to 0. Combined together, a 64-bit integer is obtained as the fingerprint information of the image frame. The fingerprint information of the first image frame and the comparison image frame is compared, and if the difference degree of the fingerprint information is greater than a preset threshold, the comparison image frame is taken as the second image frame, as shown in Figure 6 .

[0073] S103, any image frame is selected from the static picture video segment as a static storage image frame, and the dynamic storage image frame is stored together with the image frame key information as the storage data of the entire monitoring video.

[0074] In this embodiment, for the static picture video segment, only one image frame is selected as the static storage image frame, and thus the static picture video segment represents the interval of the entire static picture video segment. For the dynamic picture video segment, the determined dynamic storage image frame can be continuous or can have a short interval, but the time interval is generally not large. Therefore, in this embodiment, the time interval information of the static picture video segment and the time node information of each dynamic storage image frame are determined.

[0075] The above only determines the time information of the image frame, and the position information is also very important. In the later stage, the state events of different positions can be queried during the downhole event query. In this embodiment, the downhole position corresponding to the monitoring video is further determined, the image frame key information of the static storage image frame is determined by combining the downhole position corresponding to the monitoring video and the time interval information of the static picture video segment, and the image frame key information of different dynamic storage image frames is determined by combining the downhole position corresponding to the monitoring video and the time node information of each dynamic storage image frame. Then, the two different image frame key information are respectively associated to the corresponding storage image frame to generate the storage data of the entire monitoring video.

[0076] After the monitoring video processing method in the above embodiment is used, the static part of the monitoring video basically does not have key information, and any selected image frame can represent the static part. After the key frame is selected according to the preset condition, the image frames similar to the key frame are excluded according to the similarity comparison, and the remaining image frames basically represent the key information pictures of the dynamic part. The image frames obtained in the above are stored as the data of the monitoring video, so that the effective use of the storage space can be realized while the key information of the monitoring video is retained.

[0077] In the embodiments of the present application, “at least one” refers to one or more, and “multiple” refers to two or more. “And / or” describes the association relationship between the associated objects, which means that there can be three kinds of relationships, for example, A and / or B, which means that A exists alone, A and B exist together, and B exists alone. Wherein A, B can be singular or plural. The character “ / ” generally represents that the associated objects before and after it are in an “or” relationship. “At least one of the following” and the like means any combination of these items, including any combination of single or multiple items. For example, at least one of a, b and c can represent: a, b, c, a-b, a-c, b-c, or a-b-c, wherein a, b, c can be single or multiple.

[0078] The above description is only the specific implementation of the present application. Any person skilled in the art can easily think of changes or replacements within the technical range disclosed in the present application, which should be covered in the protection range of the present application. The protection range of the present application should be subject to the protection range of the claims.

Claims

1. A method for processing downhole monitoring video data based on image frame similarity rejection, the method comprising: receiving a plurality of video frames from a video stream; determining a similarity measure for each of the plurality of video frames; and rejecting a video frame from the video stream based on the similarity measure for the video frame. The method comprises the following steps: Splitting the monitoring video to obtain dynamic picture video segments and static picture video segments, comprising: Determining the downhole monitoring device and the key operation area in the monitoring video; If the indicator light interface of the downhole monitoring device does not change and there is no downhole personnel intervention operation in the key operation area within a preset time period, the above-mentioned video segment is regarded as a static picture video segment; otherwise, the above-mentioned video segment is regarded as a dynamic picture video segment; Determining a first image frame in the dynamic picture video segment as a key frame and obtaining a second image frame with a similarity less than a preset threshold to the key frame, wherein the first image frame and the second image frame are both dynamic storage image frames, comprising: If the indicator light interface in the dynamic picture video segment appears a pre-warning and / or the device in the key operation area exists human intervention, the picture at the current time is regarded as the first image frame; The second image frame is determined through similarity comparison, the first image frame is regarded as a storage selected frame, and the second image frame is converted into a key frame; The similarity comparison is continuously performed until all the image frames in the dynamic picture video segment are compared, and all the storage image frames are obtained; Any image frame is selected from the static picture video segment as a static storage image frame, and the dynamic storage image frame and the image frame key information are stored as the storage data of the entire monitoring video.

2. The method of claim 1, wherein, The method for detecting the indicator light interface of the downhole monitoring device comprises the following steps: High-pass filtering the image frame to extract the edge and line information in the image frame; Calculating the gradient graph of the image frame to obtain the gradient information in the image frame; Non-maximum suppression is performed on the gradient graph to eliminate the overlap between adjacent key points; Mean filtering is performed on the gradient graph to eliminate noise; Peak detection is performed on the gradient graph after mean filtering to find the key region on the downhole monitoring device.

3. The method of claim 2, wherein the method further comprises: The model for detecting the key region of the downhole monitoring device comprises: wherein: is a gradient vector, is a gradient map, is an original image, is a high-pass filter, is a horizontal gradient, is a vertical gradient, is a gradient operator.

4. The method of claim 1, wherein, The method for identifying the downhole personnel in the key operation area comprises the following steps: Reading the video file by using the cv2.VideoCapture function; Creating an MOG2 background subtractor object by using cv2.createBackgroundSubtractorMOG2; Reading each frame of the video through an infinite loop, and processing the current frame by using the background subtractor to obtain a foreground mask; Applying morphological opening operation to the foreground mask to remove small noise and separate the adhered objects; Finding the contour in the processed foreground mask; Traversing all the found contours, calculating the perimeter of the contour, and if the perimeter is greater than a preset value, considering that it is a moving downhole personnel, and calculating the bounding rectangle, and drawing a rectangular frame on the original frame to identify the downhole personnel.

5. The method of claim 1, wherein, The method for determining the second image frame through similarity comparison comprises the following steps: Reducing the size of the first image frame and the comparison image frame to simplify the calculation of DCT: img = cv2.resize(img,(long, wide)); Converting the image frame into a gray image to further simplify the calculation amount: gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY). Convert the gray image to float type, and then perform DCT transformation to obtain a DCT coefficient matrix: dct = cv2.dct(np.float32(gray)); Downsize the DCT coefficient matrix to retain only the part presenting the lowest frequency in the image frame to obtain a new DCT coefficient matrix; Calculate the DCT mean value of the new DCT coefficient matrix; Calculate the hash value, set 0 or 1 to the hash value of 64 bits, set 1 if greater than or equal to the DCT mean value, and set 0 if less than the DCT mean value, and combine them together to form a 64-bit integer as the fingerprint information of the image frame; Compare the fingerprint information of the first image frame and the comparison image frame, if the difference degree of the fingerprint information is greater than a preset threshold, the comparison image frame is taken as the second image frame.

6. The method of claim 1, wherein, Select any image frame from the static picture video segment as a static storage image frame and store it together with the dynamic storage image frame and image frame key information as the storage data of the entire monitoring video, including: Determine the time interval information of the static picture video segment and the time node information of each dynamic storage image frame; Determine the image frame key information of the static storage image frame by combining the underground position corresponding to the monitoring video with the time interval information of the static picture video segment; Determine the image frame key information of different dynamic storage image frames by combining the underground position corresponding to the monitoring video with the time node information of each dynamic storage image frame; Associate the image frame key information to the corresponding storage image frame to generate the storage data of the entire monitoring video.

Images