Underground monitoring video data processing method based on image frame similarity screening

Abstract

The invention discloses an underground monitoring video data processing method based on image frame similarity screening, and relates to the technical field of image processing, and the method comprises the steps: splitting a monitoring video to obtain a dynamic image video segment and a static image video segment; determining a first image frame in the dynamic picture video segment as a key frame and acquiring a second image frame whose similarity with the key frame is greater than a preset threshold, wherein the first image frame and the second image frame are both used as dynamic storage image frames; and selecting any image frame from the static picture video segment as a static storage image frame, and storing the static storage image frame, the dynamic storage image frame and the image frame key information as storage data of the whole monitoring video. The static part can be represented by selecting any frame of image, after the key frame is selected by the dynamic part, the image frame similar to the key frame is eliminated according to similarity comparison, and the retained image frame represents the key information picture of the dynamic part. The key information of the monitoring video is reserved in the obtained image frame, and the effective utilization of the storage space is realized.

Description

Technical Field

[0001] The present application relates to the technical field of image processing, and in particular to a method for processing underground monitoring video data based on image frame similarity screening. Background Art

[0002] Coal mine production safety is related to the safe development of the mine and the lives of employees. One ventilation and three defenses are the top priorities of coal mine safety. With the development of mines, the scope has expanded, the ventilation system has become more and more complex, and there are more and more ventilation and defense facilities. The inspection work of mine ventilation and defense is becoming more and more arduous, and the situation in the goaf is becoming more and more complicated. Safety accidents caused by the inspection personnel's inadequate inspection and untimely inspection occur from time to time every year, and the management difficulty of on-site supervision of inspection personnel is increasing day by day.

[0003] In order to ensure the safe production of coal mines in traditional technology, various video monitoring equipment are generally deployed underground, and the video monitoring equipment will transmit the underground monitoring video to the monitoring room above the well in real time. Generally, after the monitoring video is transmitted to the monitoring room, it will be displayed on the screen on the one hand, and the video will be stored on the other hand. However, most of the video images underground are regular operation videos, and the non-operation time is a long static image, which is useless video information. Storing useless information not only wastes storage space, but also affects the efficiency of later verification. Chinese patent CN117112833B discloses a video static frame filtering method and device based on storage space optimization, which screens out non-key frames of static frame sequences, estimates the key frame structure similarity index, and screens out key frames with low similarity. It achieves the goal of reducing the storage cost of data generated by monitoring videos and improving storage efficiency, avoiding the situation where a large amount of storage space is occupied during the storage process, resulting in low efficiency, while retaining the most representative and information-rich key frames, and still maintaining the quality of the video.

[0004] The method provided by the above patent can process the video to a certain extent, but the above method uses the similarity index to ensure the richness of the video and retains the image frames with high similarity index with the key. Although it does not affect the quality of the video, too many videos with high similarity will still have too much information redundancy and still waste storage space. Summary of the invention

[0005] In order to solve the above technical problems, this application proposes the following technical solutions: In a first aspect, an embodiment of the present application provides a method for processing underground monitoring video data based on image frame similarity screening, comprising: Splitting the surveillance video to obtain dynamic image video segments and static image video segments; Determine a first image frame in the dynamic picture video segment as a key frame and obtain a second image frame whose similarity to the key frame is greater than a preset threshold, wherein the first image frame and the second image frame are both used as dynamic storage image frames; Any image frame is selected from the static picture video segment as a static storage image frame and is stored together with the dynamic storage image frame plus the image frame key information as the storage data of the entire monitoring video.

[0006] In a possible implementation, splitting the surveillance video to obtain dynamic picture video segments and static picture video segments includes: Determine the underground monitoring equipment and key operation areas in the monitoring video; If the indicator light interface of the underground monitoring equipment does not change within the preset time period and there is no underground personnel intervening in the key operation area, the above video segment will be used as a static picture video segment; otherwise, the above video segment will be used as a dynamic picture video segment.

[0007] In a possible implementation, detecting the indicator light interface of the downhole monitoring equipment includes: Performing high-pass filtering on the image frame to extract edge and line information in the image frame; Calculate the gradient map of the image frame to obtain gradient information in the image frame; Perform non-maximum suppression on the gradient map to eliminate the overlap between adjacent key points; Perform mean filtering on the gradient image to eliminate noise; The mean-filtered gradient map is used for peak detection to find the key areas on the downhole monitoring equipment.

[0008] In a possible implementation, the model for detecting the key areas of the downhole monitoring equipment is: in: is the gradient vector, is the gradient map, is the original image, is a high-pass filter, is the horizontal gradient, is the vertical gradient, is the gradient operator.

[0009] In a possible implementation, identification of underground personnel in key operation areas includes: Use cv2.VideoCapture function to read the video file; Create a MOG2 background subtractor object through cv2.createBackgroundSubtractorMOG2; Read each frame of the video through an infinite loop and process the current frame using the background subtractor to obtain the foreground mask; Apply morphological opening to the foreground mask to remove small noise points and separate stuck objects; Find contours in the processed foreground mask; Traverse all found contours and calculate the perimeter of the contour. If the perimeter is greater than a preset value, it is considered to be a moving underground person, and its bounding rectangle is calculated. A rectangular box is drawn on the original frame to identify the underground person.

[0010] In a possible implementation, determining the first image frame in the dynamic picture video segment as a key frame and acquiring a second image frame having a similarity with the key frame greater than a preset threshold, wherein both the first image frame and the second image frame are used as storage image frames, includes: If an alarm is displayed on the indicator light interface in the dynamic picture video segment and / or there is human intervention in the equipment in the key operation area, the picture at the current moment is used as the first image frame; The second image frame is determined by similarity comparison, the first image frame is used as a storage selected frame, and the second image frame is converted into a key frame; The similarity comparison is continued until all image frames in the dynamic picture video segment are compared, and all stored image frames are obtained.

[0011] In a possible implementation manner, determining the second image frame after similarity comparison includes: Reduce the size of the first image frame and the comparison image frame to simplify the calculation of DCT: img = cv2.resize(img, (length, width)); Convert the image frame into a grayscale image to further simplify the calculation: gray = cv2.cvtColor(img,cv2.COLOR_BGR2GRAY); Convert the grayscale image to floating point type, and then perform DCT transformation to obtain the DCT coefficient matrix: dct = cv2.dct(np.float32(gray)); The DCT coefficient matrix is ​​reduced to retain only the lowest frequency portion in the image frame to obtain a new DCT coefficient matrix; Calculate the DCT mean of the new DCT coefficient matrix; Calculate the hash value, set the hash value from 0 or 1 to 64 bits, set the hash value greater than or equal to the DCT mean value to 1, and set the hash value less than the DCT mean value to 0, and combine them together to form a 64-bit integer as the fingerprint information of the image frame; The fingerprint information of the first image frame and the comparison image frame are compared, and if the difference of the fingerprint information is greater than a preset threshold, the comparison image frame is used as the second image frame.

[0012] In a possible implementation, any image frame is selected from the static picture video segment as a static storage image frame and stored together with the dynamic storage image frame plus the image frame key information as storage data of the entire monitoring video, including: Determine the time interval information of the static picture video segment and the time node information of each of the dynamically stored image frames; Determine the image frame key information of the static storage image frame by combining the underground position corresponding to the monitoring video with the time interval information of the static picture video segment; Determine the image frame key information of different dynamically stored image frames by combining the downhole position corresponding to the monitoring video with the time node information of each dynamically stored image frame; The image frame key information is associated with the corresponding stored image frame to generate storage data of the entire monitoring video.

[0013] In the embodiment of the present application, there is basically no key information in the static part of the surveillance video, and any frame of the image can be selected to represent it; after the key frame is selected according to the preset conditions, the image frames similar to the key frame are excluded according to the similarity comparison, and the image frames retained basically represent the key information screen of the dynamic part. The above-obtained image frames are stored as the data of the surveillance video, which can achieve effective use of storage space while retaining the key information of the surveillance video. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 A schematic flow chart of a method for processing underground monitoring video data based on image frame similarity screening provided in an embodiment of the present application; Figure 2 A schematic diagram of a static portion of a surveillance video provided in an embodiment of the present application; Figure 3 A schematic diagram of the dynamic part of the surveillance video provided in the embodiment of the present application; Figure 4 A schematic diagram of dividing surveillance video segments provided in an embodiment of the present application; Figure 5 A schematic diagram of a frame sequence of a dynamic picture video segment provided in an embodiment of the present application; Figure 6A schematic diagram of image frame difference comparison provided in an embodiment of the present application. DETAILED DESCRIPTION

[0015] The present solution is described below in conjunction with the accompanying drawings and specific implementation methods.

[0016] See also Figure 1 , the underground monitoring video data processing method based on image frame similarity screening provided in this embodiment includes: S101, split the surveillance video to obtain dynamic picture video segments and static picture video segments.

[0017] Coal mines are a special working environment with flammable and explosive combustible gases and corrosive gases, humidity, water, large mine dust, large voltage fluctuations in the power grid, serious electromagnetic interference, narrow space, and long monitoring distance. Therefore, a large number of monitoring equipment are deployed underground in coal mines. The monitoring equipment is mainly used to monitor methane concentration, carbon monoxide concentration, carbon dioxide concentration, oxygen concentration, hydrogen sulfide concentration, mine dust concentration, wind speed, wind pressure, humidity, temperature, power supply status, damper status, wind tube status, local fan start and stop, main fan start and stop, etc.

[0018] On the one hand, underground monitoring equipment will monitor key operating areas. At the same time, in order to realize timely understanding of the status of the monitoring equipment, there will also be special monitoring equipment to monitor the monitoring equipment. Among them, the indicator light interface of the monitoring equipment can best represent the working status of the monitoring equipment. The indicator light interface can not only represent the status of the monitoring equipment, but also indicate the dangerous information monitored.

[0019] Therefore, in this embodiment, after determining the underground monitoring equipment and key operation areas in the monitoring video, the dynamic picture and static picture in the monitoring video are determined based on the above two aspects. If the indicator light interface of the underground monitoring equipment does not change within the preset time period and there is no underground personnel intervening in the key operation area, the above video segment is used as a static picture video segment, otherwise the above video segment is used as a dynamic picture video segment. Figure 2 and Figure 3 As shown, Figure 2 The video segments in the middle are underground monitoring equipment and underground operating equipment. Both the monitoring equipment and underground operating equipment are operating normally. Such video segments are static video segments. Figure 3 There are underground workers in the video, so this video segment is a dynamic picture video segment.

[0020] by Figure 4For example, in the beginning of video segment a, the indicator light interface of the monitoring equipment did not change, and no underground personnel appeared in the key operation area. However, in the video segment b adjacent to video segment a, underground personnel intervened in the operation, and in video segment c, although no one appeared, the indicator light interface showed an alarm. Therefore, Figure 2 The video segment a in the figure is used as a static picture video segment, and the video segments b and c are used as dynamic picture video segments.

[0021] Specifically, in this embodiment, the indicator light interface of the underground monitoring equipment is detected, including: high-pass filtering the monitoring image frame to extract edge and line information in the image frame. Calculate the gradient map of the image frame to obtain the gradient information in the image frame; perform non-maximum suppression on the gradient map to eliminate the overlap between adjacent key points; perform mean filtering on the gradient map to eliminate noise; perform peak detection on the gradient map after mean filtering to find the key area on the underground monitoring equipment.

[0022] The model for detecting the key areas of the downhole monitoring equipment in this embodiment is: in: is the gradient vector, is the gradient map, is the original image, is a high-pass filter, is the horizontal gradient, is the vertical gradient, is the gradient operator.

[0023] In this embodiment, the identification of underground personnel in key operation areas includes: using the cv2.VideoCapture function to read the video file; creating a MOG2 background subtractor object through cv2.createBackgroundSubtractorMOG2. Read each frame of the video through an infinite loop, and use the background subtractor to process the current frame to obtain a foreground mask; apply morphological opening operations to the foreground mask to remove small noise points and separate adhered objects; find contours in the processed foreground mask; traverse all found contours and calculate the perimeter of the contour. If the perimeter is greater than a preset value, it is considered to be a moving underground personnel, and its boundary rectangle is calculated, and a rectangular box is drawn on the original frame to identify the underground personnel.

[0024] S102, determining a first image frame in the dynamic picture video segment as a key frame and acquiring a second image frame whose similarity to the key frame is greater than a preset threshold, wherein the first image frame and the second image frame are both used as dynamic storage image frames.

[0025] If an alarm appears on the indicator light interface in the dynamic picture video segment and / or there is human intervention in the equipment in the key operation area. In order to find the first image frame as the key frame, this embodiment first extracts each image frame in the dynamic picture video segment, and selects the first image frame in which there are underground personnel or the indicator light interface of the monitoring equipment shows an alarm from multiple image frames as the preferred key frame of this embodiment.

[0026] The second image frame is determined after similarity comparison, and the first image frame is used as the selected frame for storage. In this embodiment, frame similarity comparison is performed according to the selected key frame in accordance with the timing of the dynamic picture video segment. After the second image frame having a predetermined difference with the first image frame is selected, it is necessary to further use the second image frame as a reference to find a third image frame that meets the difference requirement with the second image frame, and so on. Therefore, after the first image frame is used as the selected frame for storage, the second image frame is converted into a key frame, and the similarity comparison is continued until all image frames in the dynamic picture video segment are compared, and all stored image frames are obtained.

[0027] See also Figure 5 , a schematic frame sequence of a dynamic picture video segment, if the D image frame is selected as the key frame at the beginning, the F image frame is first compared to the right to determine that the F image frame is the second image frame, and then the subsequent F image frame is used as the key frame to continue to compare to the right. After the right comparison is completed, the D image frame is continued to be compared to the left. The above is just an illustrative example, and the comparison order is not specifically limited.

[0028] It should be noted that, in order to explain that there is only one image frame between the D image frame and the F image frame, there may be multiple image frames in reality. The reason why some of them are screened out for the same dynamic picture is that the screened out images are more similar to the key frames, and the key frames can represent this series of image frames. For example, if an underground worker intervenes in the equipment operation in the key frame, and the underground worker keeps the same action for the next few seconds, then the corresponding image frames in these few seconds are almost the same as the key frames, and they need to be screened out. Other situations are the same as above, and no longer need to be repeated.

[0029] Specifically, in this embodiment, the second image frame is determined after similarity comparison, including: reducing the size of the first image frame and the comparison image frame to simplify the calculation of DCT: img = cv2.resize(img, (length, width)). Convert the image frame into a grayscale image to further simplify the calculation amount: gray = cv2.cvtColor(img,cv2.COLOR_BGR2GRAY). Convert the grayscale image to floating point type, and then perform DCT transformation to obtain a DCT coefficient matrix: dct = cv2.dct(np.float32(gray)). Reduce the DCT coefficient matrix to retain only the lowest frequency part in the image frame to obtain a new DCT coefficient matrix, and calculate the DCT mean of the new DCT coefficient matrix. Calculate the hash value, set the hash value of 0 or 1 to 64 bits, set it to 1 if it is greater than or equal to the DCT mean, and set it to 0 if it is less than the DCT mean, and combine them together to form a 64-bit integer as the fingerprint information of the image frame. Compare the fingerprint information of the first image frame and the comparison image frame, if the difference of the fingerprint information is greater than a preset threshold, use the comparison image frame as the second image frame, such as Figure 6 shown.

[0030] S103, selecting any image frame from the static picture video segment as a static storage image frame and storing it together with the dynamic storage image frame plus the image frame key information as storage data of the entire monitoring video.

[0031] In this embodiment, only one frame of image is selected as the static storage image frame for the static image video segment, so it represents the interval of the entire static image video segment. For the dynamic image video segment, the dynamic storage image frames determined may be continuous or may have a short interval, but the time interval is generally not large. Therefore, in this embodiment, the time interval information of the static image video segment and the time node information of each of the dynamic storage image frames are determined.

[0032] The above only determines the time information of the image frame. The position information is also very important. When performing underground event inquiries later, you can query the status events at different locations. In this embodiment, the underground position corresponding to the monitoring video is further determined, and the underground position corresponding to the monitoring video is combined with the time interval information of the static picture video segment to determine the image frame key information of the static storage image frame, and the underground position corresponding to the monitoring video is combined with the time node information of each of the dynamic storage image frames to determine the image frame key information of different dynamic storage image frames. Then, the two different image frame key information are respectively associated with the corresponding storage image frames to generate the storage data of the entire monitoring video.

[0033] After adopting the monitoring video processing method of the above embodiment, combined with the fact that there is basically no key information in the static part of the monitoring video, any frame of the image can be selected to represent it. After the key frame is selected according to the preset conditions for the dynamic part, the image frames similar to the key frame are excluded according to the similarity comparison, and the image frames retained basically represent the key information screen of the dynamic part. The above-obtained image frames are stored as data of the monitoring video, which can achieve effective use of storage space while retaining the key information of the monitoring video.

[0034] In the embodiments of the present application, "at least one" refers to one or more, and "plurality" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent the existence of A alone, the existence of A and B at the same time, and the existence of B alone. Among them, A and B can be singular or plural. The character " / " generally indicates that the previous and subsequent associated objects are in an "or" relationship. "At least one of the following" and similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b and c can be represented by: a, b, c, ab, ac, bc, or abc, where a, b, c can be single or multiple.

[0035] The above is only a specific implementation of the present application. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. The protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A method for processing underground monitoring video data based on image frame similarity screening, characterized in that: include: Splitting the surveillance video to obtain dynamic image video segments and static image video segments; Determine a first image frame in the dynamic picture video segment as a key frame and obtain a second image frame whose similarity to the key frame is greater than a preset threshold, wherein the first image frame and the second image frame are both used as dynamic storage image frames; Any image frame is selected from the static picture video segment as a static storage image frame and is stored together with the dynamic storage image frame plus the image frame key information as the storage data of the entire monitoring video.

2. The method for processing underground monitoring video data based on image frame similarity screening according to claim 1, characterized in that: The step of splitting the surveillance video to obtain dynamic image video segments and static image video segments includes: Determine the underground monitoring equipment and key operation areas in the monitoring video; If the indicator light interface of the underground monitoring equipment does not change within the preset time period and there is no underground personnel intervening in the key operation area, the above video segment will be used as a static picture video segment; otherwise, the above video segment will be used as a dynamic picture video segment.

3. The method for processing underground monitoring video data based on image frame similarity screening according to claim 2, characterized in that: Test the indicator light interface of the underground monitoring equipment, including: Performing high-pass filtering on the image frame to extract edge and line information in the image frame; Calculate the gradient map of the image frame to obtain gradient information in the image frame; Perform non-maximum suppression on the gradient map to eliminate the overlap between adjacent key points; Perform mean filtering on the gradient image to eliminate noise; The mean-filtered gradient map is used for peak detection to find the key areas on the downhole monitoring equipment.

4. The method for processing underground monitoring video data based on image frame similarity screening according to claim 3 is characterized in that: The model for key area detection of underground monitoring equipment is: in: is the gradient vector, is the gradient map, is the original image, is a high-pass filter, is the horizontal gradient, is the vertical gradient, is the gradient operator.

5. The method for processing underground monitoring video data based on image frame similarity screening according to claim 2, characterized in that: Identification of underground personnel in key operating areas, including: Use cv2.VideoCapture function to read the video file; Create a MOG2 background subtractor object through cv2.createBackgroundSubtractorMOG2; Read each frame of the video through an infinite loop and process the current frame using the background subtractor to obtain the foreground mask; Apply morphological opening to the foreground mask to remove small noise points and separate stuck objects; Find contours in the processed foreground mask; Traverse all found contours and calculate the perimeter of the contour. If the perimeter is greater than a preset value, it is considered to be a moving underground person, and its bounding rectangle is calculated. A rectangular box is drawn on the original frame to identify the underground person.

6. The method for processing underground monitoring video data based on image frame similarity screening according to claim 1, characterized in that: The step of determining the first image frame in the dynamic picture video segment as a key frame and acquiring a second image frame whose similarity to the key frame is greater than a preset threshold, wherein both the first image frame and the second image frame are used as storage image frames, includes: If an alarm is displayed on the indicator light interface in the dynamic picture video segment and / or there is human intervention in the equipment in the key operation area, the picture at the current moment is used as the first image frame; The second image frame is determined by similarity comparison, the first image frame is used as a storage selected frame, and the second image frame is converted into a key frame; The similarity comparison is continued until all image frames in the dynamic picture video segment are compared, and all stored image frames are obtained.

7. The method for processing underground monitoring video data based on image frame similarity screening according to claim 6, characterized in that: Determining the second image frame after similarity comparison includes: Reduce the size of the first image frame and the comparison image frame to simplify the calculation of DCT: img = cv2.resize(img, (length, width)); Convert the image frame into a grayscale image to further simplify the calculation: gray = cv2.cvtColor(img,cv2.COLOR_BGR2GRAY); Convert the grayscale image to floating point type, and then perform DCT transformation to obtain the DCT coefficient matrix: dct = cv2.dct(np.float32(gray)); The DCT coefficient matrix is ​​reduced to retain only the lowest frequency portion in the image frame to obtain a new DCT coefficient matrix; Calculate the DCT mean of the new DCT coefficient matrix; Calculate the hash value, set the hash value from 0 or 1 to 64 bits, set the hash value greater than or equal to the DCT mean value to 1, and set the hash value less than the DCT mean value to 0, and combine them together to form a 64-bit integer as the fingerprint information of the image frame; The fingerprint information of the first image frame and the comparison image frame are compared, and if the difference of the fingerprint information is greater than a preset threshold, the comparison image frame is used as the second image frame.

8. The method for processing underground monitoring video data based on image frame similarity screening according to claim 1, characterized in that: Selecting any image frame from the static picture video segment as a static storage image frame and storing it together with the dynamic storage image frame plus the image frame key information as the storage data of the entire monitoring video, including: Determine the time interval information of the static picture video segment and the time node information of each of the dynamically stored image frames; Determine the image frame key information of the static storage image frame by combining the underground position corresponding to the monitoring video with the time interval information of the static picture video segment; Determine the image frame key information of different dynamically stored image frames by combining the downhole position corresponding to the monitoring video with the time node information of each dynamically stored image frame; The image frame key information is associated with the corresponding stored image frame to generate storage data of the entire monitoring video.

Images