A safety production monitoring method and system based on video data

Abstract

The application relates to the field of video compression technology, in particular to a safety production monitoring method and system based on video data, which comprises the following steps: collecting video frame data of a coal mine site; detecting the video frame data by using a target detection network to obtain the confidence of the target; obtaining the confidence level of the target by using the confidence of the target; determining high-confidence targets and low-confidence targets based on the confidence level of the target; obtaining the determination quantity of each video frame being a key frame according to the high-confidence target; screening out first key frames based on the determination quantity of each video frame being a key frame; obtaining the determination quantity of the low-confidence target being a key target by using the low-confidence target in the remaining video frames; obtaining second key frames based on the determination quantity of the low-confidence target being a key target; and compressing and transmitting the first key frame data and the second key frame data. The application can improve the transmission efficiency and reliability of the video monitoring system.

Description

Technical Field

[0001] This invention relates to the field of video compression technology, and specifically to a method and system for safety production monitoring based on video data. Background Technology

[0002] In the field of coal mine safety monitoring, video surveillance systems are a crucial means of ensuring safety during underground operations. However, underground network bandwidth is typically severely limited, making it difficult to achieve stable real-time transmission of full-volume high-definition video. To address this issue, selective video transmission becomes an essential choice.

[0003] Currently, the main video transmission method adopted is based on deep learning target recognition: by deploying a target detection network (such as YOLO), the video stream is analyzed in real time, and the video data is only uploaded when an object of interest (such as a person or vehicle) is identified, thereby improving bandwidth utilization efficiency.

[0004] However, these methods still have significant shortcomings in the complex real-world coal mine monitoring scenarios. Their transmission decisions rely excessively on a single confidence index output by the target detection network, which primarily reflects the probability of a target's presence and does not adequately consider its reliability in the underground environment. Especially when facing high-risk targets with blurred features and uncertain shapes due to smoke, the detection confidence level is often low and fluctuates significantly. If filtering is based solely on a fixed threshold, such low-confidence but high-risk targets are easily misclassified as background noise and discarded, leading to serious missed safety alarms and consequently, the low monitoring reliability and low transmission efficiency of existing methods. Summary of the Invention

[0005] This invention provides a method and system for safety production monitoring based on video data to solve existing problems.

[0006] The present invention provides a safety production monitoring method and system based on video data, which adopts the following technical solution:

[0007] One embodiment of the present invention provides a method for safety production monitoring based on video data, the method comprising the following steps:

[0008] Collect video frame data from the coal mine site;

[0009] An object detection network is used to detect video frame data and obtain the confidence score of each object in each video frame;

[0010] By utilizing the confidence level of each target in each video frame, the confidence level of each target in each video frame is obtained.

[0011] Based on the confidence level of each target in each video frame, high-confidence targets and low-confidence targets in each video frame are determined.

[0012] The determination factor for each video frame as a keyframe is obtained based on the high-confidence target in each video frame;

[0013] Based on the determination that each video frame is a keyframe, the first keyframe is selected from all video frames; where the first keyframe is the keyframe selected for the first time from all video frames.

[0014] By utilizing low-confidence targets in the remaining video frames excluding the first keyframe, the determination metric for low-confidence targets as key targets is obtained.

[0015] Based on the determination that low-confidence targets are key targets, the second key frame is obtained; wherein, the second key frame is a key frame that is further selected from all video frames;

[0016] The first and second keyframe data are compressed and transmitted.

[0017] Furthermore, the specific steps for obtaining the confidence level of each target in each video frame by utilizing the confidence level of each target in each video frame are as follows:

[0018] For the current target in the current video frame, obtain the confidence entropy value and the confidence mean value of the target in the first preset number of video frames before the current frame;

[0019] The difference is obtained by subtracting the preset confidence threshold from the mean confidence score. Then, the difference is multiplied by the reciprocal of the confidence entropy value to calculate the confidence level of the current target in the current video frame.

[0020] Obtain the confidence level of each target in each video frame.

[0021] Furthermore, the specific steps for obtaining the confidence entropy value of the target in the first preset number of video frames prior to the current frame are as follows:

[0022] Obtain the confidence score of the target in the first preset number of video frames before the current frame, and sort them according to time sequence to obtain a confidence score sequence;

[0023] The confidence range is divided into several equal-width intervals, and the distribution frequency of the confidence sequence in each interval is counted. Then, the confidence entropy value of the target in the first preset number of video frames before the current frame is calculated according to the information entropy formula.

[0024] Furthermore, the specific steps for determining high-confidence targets and low-confidence targets in each video frame based on the confidence level of each target in each video frame are as follows:

[0025] The normalized value of the confidence level of the current target in the current video frame is compared with the preset confidence level threshold.

[0026] If the normalized value of the confidence level of the target is greater than the preset confidence level threshold, then the target is judged to be a high-confidence target.

[0027] If the normalized value of the confidence level of the target is less than or equal to the preset confidence level threshold, then the target is determined to be a low-confidence target.

[0028] Furthermore, the specific steps for obtaining the determination quantity of each video frame as a keyframe based on the high-confidence target in each video frame are as follows:

[0029] Choose any high-confidence target in the current video frame as the first target;

[0030] Calculate the ratio of the length to the shortest bounding box of the first target in the current video frame, and use this ratio as the current length-to-shortest ratio of the first target;

[0031] Obtain the length-to-length ratio of the first target in a second preset number of video frames before the current frame, and use each length-to-length ratio as the historical length-to-length ratio of the first target;

[0032] Calculate the difference between the current length-to-short ratio of the first target and each historical length-to-short ratio, and sum them to obtain the sum of the length-to-short ratio differences of the first target;

[0033] Summing the differences in the length ratios of all first targets yields the first sum value;

[0034] The confidence scores of all first objectives are summed to obtain the second sum.

[0035] Multiplying the second sum by the first sum yields the determination value for the current video frame as a keyframe.

[0036] Furthermore, the specific steps for selecting the first keyframe from all video frames based on the determination that each video frame is a keyframe are as follows:

[0037] The determination value for each video frame as a keyframe is compared with a preset first determination value threshold:

[0038] If the number of times a video frame is determined to be a keyframe is greater than a preset first determination threshold, then the video frame is determined to be the first keyframe, and the third preset number of video frames thereafter are also determined to be the first keyframes.

[0039] Furthermore, the specific steps for obtaining the determination metric that a low-confidence target is a key target by utilizing low-confidence targets in the remaining video frames excluding the first keyframe include the following:

[0040] Each video frame except the first keyframe is designated as the first video frame.

[0041] Identify any low-confidence target in the current first video frame as the second target;

[0042] Based on the connected components of the recognition box of the second target, obtain the gray density value of the connected components of the second target;

[0043] An optical flow algorithm is used to obtain the motion direction angle value of the connected region of the recognition box of the second target, and the absolute value of the difference between the angle value and 90° is calculated. The absolute value of the difference is recorded as the first absolute value.

[0044] Divide the gray density value of the connected component of the second target by the first absolute value to calculate the determination value of the second target as a key target.

[0045] Furthermore, the specific steps for obtaining the grayscale density value of the connected component of the second target based on the recognition box of the second target are as follows:

[0046] Obtain the connected component of the recognition box of the second target, as well as the grayscale value of all pixels in the connected component and the total number of pixels;

[0047] Based on the gray values ​​of all pixels within the connected component, obtain the gray range of the connected component;

[0048] The grayscale range is divided into several intervals, and the number of pixels whose grayscale values ​​fall into each interval is counted.

[0049] The ratio of the number of pixels in the interval with the most pixels to the total number of pixels in the connected component is determined as the gray density value of the connected component of the second target.

[0050] Furthermore, the specific steps for obtaining the second keyframe based on the determination that the low-confidence target is the key target are as follows:

[0051] If the determination quantity of at least one second target as a key target in the current first video frame is greater than the preset second determination quantity threshold, then the first video frame is determined to be the second key frame, and the fourth preset number of video frames thereafter are also determined to be the second key frames.

[0052] One embodiment of the present invention provides a safety production monitoring system based on video data, the system comprising the following modules:

[0053] The acquisition module is used to acquire video frame data from the coal mine site.

[0054] The analysis module is used to detect video frame data using an object detection network and obtain the confidence score of each object in each video frame.

[0055] By utilizing the confidence level of each target in each video frame, the confidence level of each target in each video frame is obtained.

[0056] Based on the confidence level of each target in each video frame, high-confidence targets and low-confidence targets in each video frame are determined.

[0057] The determination factor for each video frame as a keyframe is obtained based on the high-confidence target in each video frame;

[0058] Based on the determination that each video frame is a keyframe, the first keyframe is selected from all video frames; where the first keyframe is the keyframe selected for the first time from all video frames.

[0059] By utilizing low-confidence targets in the remaining video frames excluding the first keyframe, the determination metric for low-confidence targets as key targets is obtained.

[0060] Based on the determination that low-confidence targets are key targets, the second key frame is obtained; wherein, the second key frame is a key frame that is further selected from all video frames;

[0061] The transmission module is used to compress and transmit the first keyframe data and the second keyframe data.

[0062] The beneficial effects of the technical solution of this invention are as follows: This invention proposes a safety production monitoring method and system based on video data. By deploying a YOLO network with target detection capabilities to analyze video streams in real time, the system first distinguishes targets based on their confidence level: for high-confidence targets, abnormal behavior is detected by continuously monitoring changes in the aspect ratio of their bounding boxes; for low-confidence targets, feature enhancement analysis is further performed by combining their grayscale density and optical flow direction to determine their safety risks. Finally, the system integrates the judgment information between multiple frames and dynamically selects video frames containing key events or abnormal states for transmission. This method significantly reduces network bandwidth consumption and improves the transmission efficiency and reliability of the video surveillance system while ensuring that no safety production events are missed. Attached Figure Description

[0063] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0064] Figure 1 This is a flowchart illustrating the steps of a safety production monitoring method based on video data according to the present invention.

[0065] Figure 2 This is a block diagram of a safety production monitoring system based on video data according to the present invention. Detailed Implementation

[0066] To further illustrate the technical means and effects adopted by the present invention to achieve its intended purpose, the following, in conjunction with the accompanying drawings and preferred embodiments, details the specific implementation, structure, features, and effects of a video data-based safety production monitoring method and system proposed according to the present invention. In the following description, different "one embodiment" or "another embodiment" do not necessarily refer to the same embodiment. Furthermore, specific features, structures, or characteristics in one or more embodiments can be combined in any suitable form.

[0067] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0068] The following description, in conjunction with the accompanying drawings, details a specific solution for a video data-based safety production monitoring method and system provided by the present invention.

[0069] Please see Figure 1 The diagram illustrates a flowchart of a video data-based safety production monitoring method according to an embodiment of the present invention. The method includes the following steps:

[0070] Step S001: Collect video frame data from the coal mine site.

[0071] It should be noted that the video-based safety production monitoring methods and systems primarily monitor data related to the coal mine safety production process.

[0072] Video surveillance data is data streamed as video (generally dynamic, continuous video data). Its content includes real-time images of the coal mine site captured by cameras or monitoring equipment, including the working environment inside the mine, the activities of workers, and the operating status of equipment. This video data may contain monitoring from different perspectives, targeting different areas within the coal mine (such as production areas, equipment areas, transportation areas, etc.).

[0073] Image frame data, specifically video frame data from the coal mine site, is data typed as static image frame data (image files). Its content consists of each frame of image data extracted from the video stream. These static images can be used for subsequent analysis, such as personnel behavior analysis and equipment status monitoring, facilitating further processing and identification.

[0074] Typical data streams include:

[0075] Data acquisition layer: Explosion-proof underground camera → Fiber optic ring network → Ground monitoring center.

[0076] Processing layer, edge computing: Real-time analysis (such as personnel safety helmet detection) is performed in the explosion-proof computer downhole to reduce the amount of data transmitted uplink.

[0077] Central server: performs multi-channel video analysis, event aggregation, and big data analysis.

[0078] Storage strategy: Raw video: Distributed storage (such as Ceph) or video cloud storage, retained for 7-30 days (compliance requirements); Structured data: Time series database (InfluxDB) or relational database (PostgreSQL).

[0079] Alerts and events: Message queue (Kafka) + event database (Elasticsearch for easy retrieval).

[0080] In coal mine scenarios, video stream data is transmitted via on-site hardware. However, due to network bandwidth limitations, not all raw video streams can be transmitted to the ground monitoring center in real time, leading to distortion and packet loss. Therefore, to address this issue, a selective video transmission strategy—content analysis-based intelligent change transmission based on event-driven transmission—is needed. This strategy transmits only important video segments (such as abnormal events or key areas) to improve transmission efficiency. This invention achieves this by configuring data stream transmission equipment and a YOLO recognition network with target bounding rectangle marking capabilities to collect video stream data to be transmitted within the coal mine. The video stream to be transmitted is acquired, and the inter-frame YOLO target category confidence is calculated. Based on high confidence, the target recognition box length-to-short threshold ratio is determined. Based on the low confidence target grayscale density value and optical flow direction, a non-target determination is obtained. By combining multi-frame determination information, the transmittable video stream is identified, improving video stream transmission efficiency.

[0081] Specifically, the video stream acquisition equipment configuration is as follows:

[0082] Equipment deployment and connection: Install video acquisition equipment such as industrial cameras in key areas of the coal mine and connect them to the system via network.

[0083] Parameter configuration: Set network parameters (such as IP address), video parameters (such as resolution 1920×1080, frame rate 25FPS) and encoding format (such as H.264) for the video capture device.

[0084] Streaming media transmission settings: Enable streaming media protocols such as RTSP or HTTP to ensure stable output of real-time video streams over the network.

[0085] Connectivity verification: Test the video stream to ensure that it is stably accessible, the video is continuous, and the latency is within a controllable range.

[0086] Step S002: Use an object detection network to detect video frame data and obtain the confidence score of each object in each video frame.

[0087] It should be noted that the target detection network in this embodiment is the YOLO network. When the input video frame is processed using a target detection network (such as YOLO), the network outputs a comprehensive confidence score for each target in the frame. This confidence score is obtained by multiplying the probability of the target's existence by the probability that the target belongs to a specific category. Its value ranges from 0 to 1, directly reflecting the degree of confidence that the target is correctly identified as belonging to its category in the video frame.

[0088] Specifically, the configuration and deployment of the target detection network:

[0089] Model selection and training: The YOLOv5 object detection network was adopted. Publicly available pre-trained models can be used, and transfer learning can be further performed using coal mine scene datasets (including targets such as people, vehicles, flames, and smoke) to improve the model's recognition accuracy in the underground environment.

[0090] Inference parameter settings: Configure network inference parameters, including: input image size (e.g., 640×640), confidence threshold (e.g., 0.25, used to filter weakly detected targets), non-maximum suppression (NMS) IoU threshold (e.g., 0.45, used to merge overlapping boxes), and enable bounding box drawing.

[0091] Computational acceleration: Enable GPU acceleration (such as CUDA) in the deployment environment to improve network inference speed and meet real-time requirements.

[0092] Video stream access and processing: The acquired real-time video stream (RTSP / HTTP stream) is input into the configured YOLO network. The network analyzes each frame of the image in real time and outputs structured detection results containing the target category, location bounding box, and its corresponding confidence score.

[0093] Step S003: Utilize the confidence level of each target in each video frame to obtain the confidence level of each target in each video frame.

[0094] It should be noted that the video stream data to be transmitted obtained from the above steps includes several image frames captured by industrial cameras inside the coal mine.

[0095] Due to the limitations of hardware conditions within coal mines, continuously high frame rate transmission becomes an uncontrollable pressure on bandwidth, leading to transmission distortion and packet loss. To eliminate this problem, a selective video transmission strategy is determined, namely, intelligent change transmission based on content analysis of event-driven transmission. Therefore, the accuracy and fidelity of video stream data selection become primary conditions. To meet these conditions, efficient selection of interest frame data streams is achieved through obvious behavior analysis and state recognition within the coal mine, aiming to achieve accurate selection and reduce the proportion of video transmission bandwidth. Therefore, firstly, a YOLO network is used to mark visible targets within video frames and obtain the confidence level of each target.

[0096] The confidence level is related to the identifiability of the target; the higher the identifiability, the higher the confidence level, and the clearer the bounding box of the YOLO network. The reason for the varying confidence levels is that in a coal mine environment, the situation within video data frames is quite complex, including potential personnel information and real-time actions (walking, standing, and falling, etc.), as well as potential equipment malfunctions, smoke diffusion, and other issues. For smoke-related safety issues, the YOLO network struggles to identify and label them. Therefore, it is necessary to provide comprehensive identification and detection capabilities beyond the YOLO network's recognition and judgment to prevent the loss of important frames due to recognition problems.

[0097] Furthermore, calculate the YOLO confidence score of the j-th target within the i-th frame of the video stream data. The confidence score is normalized; the closer it is to 1, the higher the confidence score, and the more likely it is to be a moving or environmentally-based target (such as people or equipment). When the confidence score changes dynamically and is less than half its value (0.5), it is considered low confidence. This is because, in an environment, the most prominent characteristic of smoke is a sudden increase in local grayscale density. Smoke often appears as a blurry area with uneven light reflection, and its grayscale values ​​change rapidly. Therefore, it is usually blurry, with indistinct edges, and exhibits significant color and brightness variations relative to the surrounding background. Thus, it displays dynamic changes in confidence and a generally low confidence level.

[0098] Step S003 further includes steps S0031-S0033:

[0099] Step S0031: For the current target in the current video frame, obtain the confidence entropy value and the confidence mean value of the target in the first preset number of video frames before the current frame.

[0100] It should be noted that the first preset quantity is set according to the specific situation, and it is preferred to be 10 here.

[0101] The specific steps for obtaining the confidence entropy value of the target in the first preset number of video frames preceding the current frame are as follows:

[0102] The confidence score of the target in the first preset number of video frames before the current frame is obtained and sorted according to the time sequence to obtain the confidence score sequence.

[0103] It should be noted that: for the j-th target in the i-th frame, the confidence of the target in the 10 consecutive frames before the i-th frame is obtained, and the confidence of the target in each frame is sorted to obtain the confidence time series sequence.

[0104] The confidence range is divided into several equal-width intervals, and the distribution frequency of the confidence sequence in each interval is counted. Then, the confidence entropy value of the target in the first preset number of video frames before the current frame is calculated according to the information entropy formula.

[0105] It should be noted that the confidence level range is [0, 1].

[0106] For example, the confidence level sequence is: [0.92, 0.88, 0.85, 0.90, 0.87, 0.15, 0.10, 0.08, 0.89, 0.91].

[0107] Divide [0, 1] into 10 intervals (interval width 0.1): [0.0, 0.1), [0.1, 0.2), ..., [0.9, 1.0].

[0108] Statistical distribution frequency: Confidence level of values ​​falling in the interval [0.0, 0.1) is 0.08, with a frequency of 0.1; confidence level of values ​​falling in the interval [0.1, 0.2) is 0.15, 0.10, with a frequency of 0.2; confidence level of values ​​falling in the interval [0.8, 0.9) is 0.88, 0.85, 0.87, 0.89, with a frequency of 0.4; confidence level of values ​​falling in the interval [0.9, 1.0] is 0.92, 0.90, 0.91, with a frequency of 0.3; the number of values ​​falling in the remaining intervals is 0.

[0109] Calculate information entropy: = That is, the confidence entropy value is 1.279. Let be the distribution frequency of the v-th interval.

[0110] Calculate the confidence mean: Calculate the mean of all elements in the sequence.

[0111] Step S0032: Subtract the preset confidence threshold from the mean confidence value to obtain the difference, and then multiply the difference by the reciprocal of the confidence entropy value to calculate the confidence level of the current target in the current video frame.

[0112] Specifically, determine the first Frame number The confidence level corresponding to each target:

[0113]

[0114] in, Indicates the first Frame number The confidence level of each target This represents the average confidence level of the target within 10 frames, which is the average confidence level of the target in the first preset number of video frames before the current frame; 0.5 represents the confidence threshold, which can be set according to specific circumstances. The confidence entropy value is calculated by taking the target confidence level in the previous 10 frames as a sample, which is the confidence entropy value of the target in the first preset number of video frames before the current frame.

[0115] It should be noted that: First, by calculating the entropy value, the specific degree of uncertainty in confidence level changes within a 10-frame range of the video stream data was determined. Higher uncertainty corresponds to a higher entropy value, and consequently, a lower confidence level. This aligns with the confidence level change characteristics when targets such as smoke, which are difficult for YOLO to capture, appear in a coal mine scene.

[0116] Secondly, the former ( -0.5) represents the difference between the confidence level and 0.5. The larger the difference, the further the mean of the confidence level is from the threshold, and the higher the confidence level. Conversely, if the difference is less than 0, the score is negative, and the confidence level is lower, which is consistent with the judgment logic for the mean of the confidence level.

[0117] Step S0033: Obtain the confidence level of each target in each video frame.

[0118] Step S004: Based on the confidence level of each target in each video frame, determine the high-confidence targets and low-confidence targets in each video frame.

[0119] Specifically, it includes:

[0120] The normalized value of the confidence level of the current target in the current video frame is compared with the preset confidence level threshold.

[0121] If the normalized value of the confidence level of the target is greater than the preset confidence level threshold, then the target is determined to be a high-confidence target.

[0122] It should be noted that the confidence threshold is set according to specific circumstances, and 0.5 is preferred here. After normalization, a value greater than 0.5 is considered high confidence, and vice versa.

[0123] If the normalized value of the confidence level of the target is less than or equal to the preset confidence level threshold, then the target is determined to be a low-confidence target.

[0124] Step S005: Obtain the determination value of each video frame as a keyframe based on the high confidence target in each video frame.

[0125] It's important to note that for high-confidence segments, the targets are considered stable, such as personnel or permanently stationed equipment within a coal mine. Therefore, the decision metric for selecting these video stream frames for transmission becomes: a significant change in the aspect ratio of the YOLO bounding box within a given timeframe. This is because when the target recognition confidence is high, indicating high clarity and that it is either a person or equipment, abnormal behavior within the mine, such as falls or equipment movement, necessitates selection as a key frame. Thus, falls and specific equipment movements cause changes in the aspect ratio of the YOLO network's bounding box, which can then be used as a decision metric.

[0126] Step S005 further includes steps S0051-S0057:

[0127] Step S0051: Select any high-confidence target in the current video frame as the first target.

[0128] It should be noted that the distribution of high-confidence targets and low-confidence targets differs within each video frame. First, the first batch of keyframes is selected based on the high-confidence targets in each video frame. Then, a second batch of keyframes is selected from the remaining video frames based on the low-confidence targets in each video frame.

[0129] Step S0052: Calculate the ratio of the length to the shortness of the bounding box of the first target in the current video frame, and use the ratio as the current length-to-shortness ratio of the first target.

[0130] It should be noted that: obtaining the first Intra-frame The bounding boxes for a high-confidence target (the first target) have corresponding box lengths. and short frame The ratio is obtained as follows: ,in For the first The current length-to-width ratio of a high-confidence target (the first target). For example, when a person falls, the length of the bounding box becomes shorter and wider, so the corresponding ratio will decrease; conversely, when a person is standing, the ratio will increase, which is also a keyframe.

[0131] Step S0053: Obtain the length-to-length ratio of the first target in a second preset number of video frames before the current frame, and use each length-to-length ratio as the historical length-to-length ratio of the first target.

[0132] It should be noted that the second preset quantity is set according to specific circumstances, and is preferably 10 here. The first target is obtained at the... The length ratio of each frame within the first 10 frames is the historical length ratio of the first target.

[0133] Step S0054: Calculate the difference between the current length-to-short ratio of the first target and each historical length-to-short ratio, and sum them to obtain the sum of the length-to-short ratio differences of the first target.

[0134] Specifically, Indicates the first Intra-frame The sum of the length-to-short ratio differences of each target within the previous 10 frames is the sum of the length-to-short ratio differences of the first target. If a target has not appeared in several frames within the previous 10 frames, only the targets that have appeared are counted.

[0135] Step S0055: Sum the length-to-short ratio differences of all first targets to obtain the first sum value.

[0136] Specifically, The first sum, Indicates the first The number of first targets within a frame.

[0137] Step S0056: Sum the confidence scores of all first objectives to obtain the second sum.

[0138] Specifically, This is the second sum.

[0139] Step S0057: Multiply the second sum by the first sum to obtain the determination value that the current video frame is a keyframe.

[0140] Specifically, for the high-confidence portion, when the corresponding recognition-to-decision ratio changes, it can be considered that a keyframe has appeared, and its determination value is:

[0141]

[0142] in, Indicates the first Video frames are used to determine keyframes.

[0143] For the summation part, when the length-to-short ratio of an object changes continuously over multiple frames, the sum of its differences... The results are relatively high, and multiple targets and values ​​are also present. The higher the confidence level, the higher the corresponding decision value, and the more likely it is to be selected as a key frame for transmission. At the same time, when the confidence level is high, the target has high observability (important personnel and equipment safety, etc.), so the higher the corresponding value, the higher the decision value.

[0144] Step S006: Based on the determination that each video frame is a keyframe, select the first keyframe from all video frames; wherein, the first keyframe is the keyframe selected for the first time from all video frames.

[0145] Specifically, it includes:

[0146] The determination value for each video frame as a keyframe is compared with a preset first determination value threshold:

[0147] If the number of times a video frame is determined to be a keyframe is greater than a preset first determination threshold, then the video frame is determined to be the first keyframe, and the third preset number of video frames thereafter are also determined to be the first keyframes.

[0148] It should be noted that: the first judgment threshold is set according to the specific situation, and is preferably 0.8 here. The third preset quantity is set according to the specific situation, and is preferably 10 here.

[0149] If the judgment value of a certain frame is greater than 0.8, then the frame and the following 10 frames are considered key frames, and the images of the frame and the following 10 frames are regarded as key transmission items of the video stream.

[0150] Step S007: Using the low-confidence targets in the remaining video frames excluding the first keyframe, obtain the determination value of low-confidence targets as key targets.

[0151] It should be noted that for the low-confidence portion, due to the limitations of the target's characteristics—for example, the edges of the smoke region are unclear, and the changes in grayscale values ​​are relatively gradual—the circumscribed rectangle of the smoke is insufficient to accurately reflect its actual area. To achieve effective and comprehensive identification, this non-target determination quantity is determined by calculating its corresponding grayscale density and optical flow direction.

[0152] Step S007 further includes steps S0071-S0075:

[0153] Step S0071: Determine each video frame other than the first keyframe as the first video frame.

[0154] It should be noted that the first batch of video frames has already been filtered, and then the remaining video frames are filtered again.

[0155] Step S0072: Identify any low-confidence target in the current first video frame as the second target.

[0156] Step S0073: Based on the connected components of the recognition box of the second target, obtain the gray density value of the connected components of the second target.

[0157] It should be noted that: the connected components within the video stream data frame corresponding to low confidence (i.e., the connected components obtained by Canny operator edge detection within the unstable recognition box) are obtained, and their corresponding grayscale density values ​​are analyzed.

[0158] Step S0073 further includes steps S0731-S0734:

[0159] Step S0731: Obtain the connected component of the recognition box of the second target, as well as the grayscale value of all pixels in the connected component and the total number of pixels.

[0160] Specifically, for each low-confidence target, the connected component corresponding to the target is first extracted using the image area covered by its bounding box as a reference. Then, the gray values ​​of all pixels in the connected component are obtained in the grayscale image, and the total number of pixels contained in the connected component is counted.

[0161] Step S0732: Based on the gray values ​​of all pixels within the connected component, obtain the gray range of the connected component.

[0162] Step S0733: Divide the grayscale range into several intervals and count the number of pixels whose grayscale values ​​fall into each interval.

[0163] Specifically, the grayscale range within the connected domain is divided into 10 parts, each corresponding to the number of pixels q.

[0164] Step S0734: Determine the gray density value of the connected component of the second target as the ratio of the number of pixels in the interval with the largest number of pixels to the total number of pixels in the connected component.

[0165] Specifically, obtain the maximum number of connected components corresponding to the grayscale range. That is, the number of pixels corresponding to the interval with the most pixels.

[0166] Calculate the gray density value of the connected components:

[0167]

[0168] in, This represents the gray density value of the connected component of the e-th low-confidence target (i.e., non-target), which is the gray density value of the connected component of the second target. This represents the total number of pixels in the connected component.

[0169] The larger the value, the larger the area of ​​the gray range with the highest proportion of quantity within the connected region, and the higher the probability that it has the characteristics of smoke: "unclear edges and relatively gentle changes in gray values".

[0170] Step S0074: Use the optical flow algorithm to obtain the motion direction angle value of the connected region of the recognition box of the second target, and calculate the absolute value of the difference between the angle value and 90°, and record the absolute value of the difference as the first absolute value.

[0171] It should be noted that optical flow algorithms calculate the motion vector field of each pixel by analyzing the temporal and dimensional changes in pixel intensity in consecutive video frames, thereby estimating the motion pattern and direction of the target in the image. Dense optical flow algorithms (such as the Farneback algorithm) are used to analyze consecutive video frames containing the second target, calculating the motion vector of each pixel. Based on the connected component mask of the second target, the motion vectors corresponding to all pixels within that region are extracted, and the average horizontal and vertical displacements of these vectors are calculated. Then, the average motion direction angle of the pixels in that connected component is calculated using the arctangent function. This angle value represents the overall motion direction of the target in the image plane. Optical flow algorithms are well-known technologies and will not be elaborated upon here.

[0172] The motion direction angle value of the connected component of the recognition box of the second target is denoted as... The closer this value is to 90°, the more vertical it is, and the more the direction of the optical flow matches the characteristics of weak interferences such as smoke. This is because smoke tends to rise in a coal mine environment.

[0173] It is denoted as the first absolute value.

[0174] Step S0075: Divide the gray density value of the connected component of the second target by the first absolute value to calculate the determination value of the second target as a key target.

[0175] Specifically, the second objective is the metric for determining key objectives:

[0176]

[0177] in, Indicates the first One non-target judgment quantity, that is, the judgment quantity that the second target is a key target; This represents the difference between the optical flow direction and the rising and drifting characteristics of targets such as smoke in a coal mine scene. The smaller this difference, the closer the optical flow direction is to the actual non-target direction, and the higher the corresponding non-target determination. A higher grayscale density value indicates a higher probability and degree of concentration of this part within a certain grayscale range, which better matches the presence of dangerous situations like smoke that the YOLO network cannot detect or identify in a coal mine. Therefore, in this case, the priority for frame retention and transmission is higher.

[0178] Step S008: Based on the determination that the low-confidence target is a key target, obtain the second key frame; wherein, the second key frame is a key frame that is selected again from all video frames.

[0179] Specifically, it includes:

[0180] If the determination quantity of at least one second target as a key target in the current first video frame is greater than the preset second determination quantity threshold, then the first video frame is determined to be the second key frame, and the fourth preset number of video frames thereafter are also determined to be the second key frames.

[0181] It should be noted that: the second judgment threshold is set according to the specific situation, and is preferably 0.8 here; the fourth preset quantity is set according to the specific situation, and is preferably 10 here.

[0182] If at least one non-target determination value is greater than 0.8 within a frame, it can be determined that an important video frame needs to be transmitted. This frame and the following 10 frames are taken as key frames and used as key transmission items in the video stream.

[0183] Step S009: Compress and transmit the first keyframe data and the second keyframe data.

[0184] It should be noted that: This concludes the selective transmission strategy for coal mine video frames based on the actual environment within the coal mine. All video frames to be transmitted are acquired and sent to the safety monitoring terminal as real-time monitoring frames.

[0185] The transmission process is as follows:

[0186] Video frame encoding: First, the selected video frames are encoded (H.264) to compress the video data and reduce the transmission bandwidth requirements.

[0187] Data Packaging: Packing the encoded frame data into a transmission format, encapsulating video frames into RTSP data packets.

[0188] Network transmission: The packaged data is transmitted from the sending end to the monitoring terminal via a network (Wi-Fi). The TCP / IP protocol is used during transmission to ensure reliable data delivery.

[0189] Data decoding: After receiving video data, the monitoring terminal will decode it to recover the video frames.

[0190] Display output: The decoded video frames will be rendered and displayed on the monitoring terminal for users to view.

[0191] Please see Figure 2 The diagram illustrates a block diagram of a video data-based safety production monitoring system according to an embodiment of the present invention. The system includes the following modules:

[0192] Acquisition module 100 is used to acquire video frame data from the coal mine site;

[0193] The analysis module 200 is used to detect video frame data using an object detection network and obtain the confidence score of each object in each video frame.

[0194] By utilizing the confidence level of each target in each video frame, the confidence level of each target in each video frame is obtained.

[0195] Based on the confidence level of each target in each video frame, high-confidence targets and low-confidence targets in each video frame are determined.

[0196] The determination factor for each video frame as a keyframe is obtained based on the high-confidence target in each video frame;

[0197] Based on the determination that each video frame is a keyframe, the first keyframe is selected from all video frames; where the first keyframe is the keyframe selected for the first time from all video frames.

[0198] By utilizing low-confidence targets in the remaining video frames excluding the first keyframe, the determination metric for low-confidence targets as key targets is obtained.

[0199] Based on the determination that low-confidence targets are key targets, the second key frame is obtained; wherein, the second key frame is a key frame that is further selected from all video frames;

[0200] The transmission module 300 is used to compress and transmit the first keyframe data and the second keyframe data.

[0201] During the transmission of video stream data in coal mines, limitations in hardware bandwidth lead to phenomena such as video loss. This embodiment improves the transmission efficiency of coal mine video data by performing partial information analysis on the video stream to obtain high-information-content video stream data that can be transmitted.

[0202] During the recognition of transmittable video stream data, due to the limitations of the accuracy of the external rectangular recognition box of the YOLO network, some abnormal environmental phenomena such as smoke cannot be detected, which may lead to the security risk of important video frames not being transmitted. This embodiment combines optical flow method for analysis, which improves the overall recognition accuracy.

[0203] During video transmission, the transmittable video stream data is obtained by judging the judgment quantity within the interval, thus achieving the purpose of high-efficiency transmission of video stream data.

[0204] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for safety production monitoring based on video data, characterized in that, The method includes the following steps: Collect video frame data from the coal mine site; An object detection network is used to detect video frame data and obtain the confidence score of each object in each video frame; By utilizing the confidence level of each target in each video frame, the confidence level of each target in each video frame is obtained. Based on the confidence level of each target in each video frame, high-confidence targets and low-confidence targets in each video frame are determined. The criteria for determining whether a video frame is a keyframe are obtained based on the high-confidence target in each video frame, specifically including: Choose any high-confidence target in the current video frame as the first target; Calculate the ratio of the length to the shortest bounding box of the first target in the current video frame, and use this ratio as the current length-to-shortest ratio of the first target; Obtain the length-to-length ratio of the first target in a second preset number of video frames before the current frame, and use each length-to-length ratio as the historical length-to-length ratio of the first target; Calculate the difference between the current length-to-short ratio of the first target and each historical length-to-short ratio, and sum them to obtain the sum of the length-to-short ratio differences of the first target; Summing the differences in the length ratios of all first targets yields the first sum value; The confidence scores of all first objectives are summed to obtain the second sum. Multiply the second sum by the first sum to obtain the determination value that the current video frame is a keyframe; Based on the determination that each video frame is a keyframe, the first keyframe is selected from all video frames; where the first keyframe is the keyframe selected for the first time from all video frames. By utilizing low-confidence targets in the remaining video frames excluding the first keyframe, a determination metric for identifying low-confidence targets as key targets is obtained, specifically including: Each video frame except the first keyframe is designated as the first video frame. Identify any low-confidence target in the current first video frame as the second target; Based on the connected components of the recognition box of the second target, the gray-level density values ​​of the connected components of the second target are obtained, specifically including: Obtain the connected component of the recognition box of the second target, as well as the grayscale value of all pixels in the connected component and the total number of pixels; Based on the gray values ​​of all pixels within the connected component, obtain the gray range of the connected component; The grayscale range is divided into several intervals, and the number of pixels whose grayscale values ​​fall into each interval is counted. The ratio of the number of pixels in the interval with the most pixels to the total number of pixels in the connected component is determined as the gray density value of the connected component of the second target. An optical flow algorithm is used to obtain the motion direction angle value of the connected region of the recognition box of the second target, and the absolute value of the difference between the angle value and 90° is calculated. The absolute value of the difference is recorded as the first absolute value. Divide the gray density value of the connected component of the second target by the first absolute value to calculate the determination value of the second target as a key target; Based on the determination that low-confidence targets are key targets, the second key frame is obtained; wherein, the second key frame is a key frame that is further selected from all video frames; The first and second keyframe data are compressed and transmitted.

2. The safety production monitoring method based on video data according to claim 1, characterized in that, The specific steps for obtaining the confidence level of each target in each video frame by utilizing the confidence level of each target in each video frame are as follows: For the current target in the current video frame, obtain the confidence entropy value and the confidence mean value of the target in the first preset number of video frames before the current frame; The difference is obtained by subtracting the preset confidence threshold from the mean confidence score. Then, the difference is multiplied by the reciprocal of the confidence entropy value to calculate the confidence level of the current target in the current video frame. Obtain the confidence level of each target in each video frame.

3. The safety production monitoring method based on video data according to claim 2, characterized in that, The specific steps for obtaining the confidence entropy value of the target in a first preset number of video frames before the current frame are as follows: Obtain the confidence score of the target in the first preset number of video frames before the current frame, and sort them according to time sequence to obtain a confidence score sequence; The confidence range is divided into several equal-width intervals, and the distribution frequency of the confidence sequence in each interval is counted. Then, the confidence entropy value of the target in the first preset number of video frames before the current frame is calculated according to the information entropy formula.

4. The safety production monitoring method based on video data according to claim 1, characterized in that, The specific steps for determining high-confidence targets and low-confidence targets in each video frame based on the confidence level of each target in each video frame are as follows: The normalized value of the confidence level of the current target in the current video frame is compared with the preset confidence level threshold. If the normalized value of the confidence level of the target is greater than the preset confidence level threshold, then the target is judged to be a high-confidence target. If the normalized value of the confidence level of the target is less than or equal to the preset confidence level threshold, then the target is determined to be a low-confidence target.

5. The safety production monitoring method based on video data according to claim 1, characterized in that, The specific steps for selecting the first keyframe from all video frames based on the determination that each video frame is a keyframe are as follows: The determination value for each video frame as a keyframe is compared with a preset first determination value threshold: If the number of times a video frame is determined to be a keyframe is greater than a preset first determination threshold, then the video frame is determined to be the first keyframe, and the third preset number of video frames thereafter are also determined to be the first keyframes.

6. The safety production monitoring method based on video data according to claim 1, characterized in that, The specific steps for obtaining the second keyframe based on the determination that the low-confidence target is the key target are as follows: If the determination quantity of at least one second target as a key target in the current first video frame is greater than the preset second determination quantity threshold, then the first video frame is determined to be the second key frame, and the fourth preset number of video frames thereafter are also determined to be the second key frames.

7. A video data-based safety production monitoring system, used in accordance with any one of claims 1 to 6, characterized in that, The system includes the following modules: The acquisition module is used to acquire video frame data from the coal mine site. The analysis module is used to detect video frame data using an object detection network and obtain the confidence score of each object in each video frame. By utilizing the confidence level of each target in each video frame, the confidence level of each target in each video frame is obtained. Based on the confidence level of each target in each video frame, high-confidence targets and low-confidence targets in each video frame are determined. The determination factor for each video frame as a keyframe is obtained based on the high-confidence target in each video frame; Based on the determination that each video frame is a keyframe, the first keyframe is selected from all video frames; where the first keyframe is the keyframe selected for the first time from all video frames. By utilizing low-confidence targets in the remaining video frames excluding the first keyframe, the determination metric for low-confidence targets as key targets is obtained. Based on the determination that low-confidence targets are key targets, the second key frame is obtained; wherein, the second key frame is a key frame that is further selected from all video frames; The transmission module is used to compress and transmit the first keyframe data and the second keyframe data.

Images