Coal mine monitoring video alarm intelligent analysis system and method based on Yolov5 and 3DCNN

Abstract

The invention relates to the technical field of coal mine monitoring video alarm intelligent analysis, in particular to a coal mine monitoring video alarm intelligent analysis system and method based on Yolov5 and 3DCNN, and the system comprises a data processing module and a model reasoning service module which are electrically connected. The data processing module comprises interface service, evidence video request, video formatting processing and model data processing; the model reasoning service module comprises a dispatching room vacancy early warning and rechecking model and a belt operation abnormity early warning and rechecking model; through the coal mine monitoring video alarm intelligent analysis system based on YoloV5 and 3DCNN, model reasoning and data processing decoupling are independent, data processing processes of interface service data processing, video data requesting and downloading, video data unified processing and the like are integrated through a data processing module, the data processing process is abstracted, a unified data processing mode is provided, and the data processing efficiency is improved. And processing the data with different sources and different formats into a uniform data format under the same evaluation index.

Description

Technical Field

[0001] The present invention relates to the technical field of coal mine monitoring video alarm intelligent analysis, and in particular to a coal mine monitoring video alarm intelligent analysis system and method based on Yolov5 and 3DCNN. Background Art

[0002] Coal mine production monitoring video is located in a remote area, due to the poor status of the local network and monitoring equipment, poor lighting conditions at night, and infrequent personnel flow. Therefore, the monitoring equipment's own video target detection and motion analysis capabilities make it easy for factors such as sleeping and obstruction to cause missed detection when identifying personnel in the monitoring area. When the identification equipment is running, it is easy for natural factors such as wind, sand, mosquitoes, or human factors such as vehicles, pedestrians and other equipment to cause erroneous motion detection.

[0003] Among the existing machine vision analysis capabilities, video motion detection is usually implemented based on frame difference, optical flow, 2D convolutional neural network, long short-term memory network (LSTM), and the latest dynamic pattern decomposition (DMD) method. Since the data for the review and analysis problem are false alarms from different types of motion detection algorithms used by different manufacturers, false alarms are often the result of using frame difference and optical flow methods. The method combining 2D convolutional neural network with LSTM and dynamic pattern decomposition are based on the recognition of image features and feature sequences. The motion detection performance is not good for coal mining equipment such as transmission belts, roller rotation, etc., which have small motion amplitudes and are part of the scene. The recognition model is prone to mistake the motion detection problem for the classification and recognition problem of the scene. Summary of the invention

[0004] The purpose of the present invention is to provide a coal mine monitoring video alarm intelligent analysis system and method based on Yolov5 and 3DCNN to solve the problems raised in the above background technology.

[0005] To achieve the above object, the present invention provides the following technical solutions: An intelligent analysis method for coal mine monitoring video alarm based on Yolov5 and 3DCNN includes a data processing module and a model reasoning service module. The data processing module and the model reasoning service module are electrically connected. The data processing module includes interface services, supporting video requests, video formatting processing and model data processing. The model reasoning service module includes a dispatch room vacant post warning re-inspection model and a belt operation abnormality warning re-inspection model.

[0006] An intelligent analysis method for coal mine monitoring video alarm based on Yolov5 and 3DCNN includes interface service, supporting video request, video formatting processing, model data processing, dispatch room vacancy warning re-inspection model and belt operation abnormality warning re-inspection model. The interface service includes the following steps: S1. Receive the alarm record information requested by the interface, perform format and integrity verification on the alarm record information, and receive the early warning recheck result message returned by the model data processing module for interface return.

[0007] As a preferred solution of the present invention, the supporting video request includes the following steps: S2. Analyze the alarm record information, and according to the coal mine code, alarm date, alarm type and other information in the alarm record, request the supporting video query interface in the video cloud storage platform, query and match the supporting video resource address URL closest to the alarm date, and download the supporting video file according to the video URL.

[0008] As a preferred solution of the present invention, the video formatting process includes the following steps: S3. Responsible for performing video integrity check, video interpolation and frame filling, video serialization and other video file processing on the supporting video files, and unifying the video files to reduce the model inference speed. The video difference frame filling method is to unify the sequence with image height and width as the interpolation dimension. The calculation process is to take the original image of (250, H, W, C) and use H as the coordinate system. The H (250, W, C) are all resized to (61, W, C). Then the result after H dimension unification is (61, H, W, C), or take W as the coordinate system. The W (250, H, C) are all resized to (61, H, C). Then the result after W dimension unification is (61, H, W, C).

[0009] As a preferred solution of the present invention, the model data processing includes the following steps: S4. Unify the format and transfer scheduling of data, including formatting and encrypting the warning record information received by the interface service, serializing the video images, unifying the image size, and enhancing the images of the unified video files to form the video image sequence imgs as the model input data, organizing the reasoning results of the warning re-inspection model and structuring the messages, and visualizing the sampling of the re-inspection result image sequence.

[0010] As a preferred solution of the present invention, the dispatch room vacancy warning re-inspection model includes the following steps: S5. The dispatch room vacant post warning re-inspection model takes the dispatch room video image sequence imgs, the detection threshold person_t, and the number of alarm frames warning_flames as input, detects and locates human bodies, human heads, arms and other human and edge targets in each image img in the monitoring dispatch room video image sequence imgs, limits the detection of non-human targets according to the detection threshold person_t, and counts the number of images in which human targets are detected warning_nums. Finally, a threshold judgment is made according to the number of alarm frames warning_flames. When warning_nums>warning_flames, imgs is judged to be a false alarm, otherwise it is a correct alarm.

[0011] As a preferred solution of the present invention, the belt operation abnormality early warning re-inspection model includes the following steps: S6. The model adopts the 3D-CNN video classification model algorithm. It takes the belt movement alarm video image sequence imgs and the recognition threshold belt_t as input, and performs a binary classification task of motion (static) and stillness (moving) on ​​the input video image sequence imgs. The 3D convolutional neural network model infers and calculates the predicted values ​​pred of motion and stillness, and screens the results through the recognition threshold belt_t. When pred>=belt_t, imgs is judged to be in motion and a correct alarm is given. Otherwise, it is still and a false alarm is given.

[0012] As a preferred solution of the present invention, the steps of the video classification model algorithm are: S7.3DCNN network captures dynamic information encoded in multiple adjacent frames by performing three-dimensional convolutions of different scales. Based on the feature extraction capability of 3D convolutional neural network, the image sequence is cropped to 320×240 pixels, and RGB three channels are used as the input of 3D-CNN. In the task, 15 frames are used as the length of the video subsequence. First, the sequence is input into the data reinforcement hardwired module, and three channel information (grayscale, horizontal axis gradient (gradient-x), vertical axis gradient (gradient-y)) is extracted from each frame. The information of the three channels can be directly calculated and obtained for each frame; The S8.3D convolution block consists of four layers of convolution in series. The first layer uses a linear activation function with a filter size of 5×5×5. The dimension of the first convolution layer is 13×118×158×32, where the first dimension represents temporal information (sl=13), the second dimension (h=118) and the third dimension (w=158) represent spatial information, and the fourth dimension (c=32) represents channel information. In order to reduce the dimensions of time and space, the step size of this layer is set to 3, and the number of filters is 32. In order to avoid overfitting and reduce information redundancy, each layer is To perform a dropout operation, according to the dropout principle, the dropout operation means randomly discarding a certain number of neurons in the layer (and the connections between them) with a certain probability. In order to speed up the convergence of model training, each layer must be normalized. For the 2nd, 3rd, and 4th layers, the same parameters are used for setting. The difference is that the filter size is changed to 3×3×3, 3×3×3, and 1×1×1, and the number of feature maps is 48 and 64 respectively. In addition, the ReLU function is used as the activation function; S9. After the 3D convolution block, a 128-dimensional fully connected layer is used to extract classification features of the short-term spatiotemporal feature map, and finally perform a binary classification task of whether it is motion or not. It should be noted that the 4th layer in the 3D convolution block uses a 1×1×1 convolution kernel, which is also equivalent to a certain degree of fully connected operation.

[0013] Compared with the prior art, the present invention has the following beneficial effects: 1. In the present invention, through the coal mine monitoring video alarm intelligent analysis system based on YoloV5 and 3DCNN, model reasoning and data processing are decoupled and independent, and the data processing processes such as interface service data processing, video data request and download, and video data unified processing are integrated through the data processing module. The data processing process is abstracted and a unified data processing mode is provided to process data from different sources and in different formats into a unified data format under the same evaluation index; through the model reasoning service module, the deep learning algorithms such as YoloV5 target detection and 3DCNN video classification are integrated as the technical basis to realize the artificial intelligence model reasoning such as the vacant post alarm re-inspection in the dispatching room and the abnormal operation alarm re-inspection of the belt equipment, providing the system with a core algorithm capability base, and the system realizes a universal, highly mobile, efficient and reliable video alarm intelligent analysis system in the coal mine production monitoring scenario.

[0014] 2. In the present invention, the system uses target detection and video classification capabilities as the basis of AI algorithms, designs a network model structure that takes both general and specific aspects into consideration according to application scenarios, introduces adaptive models to meet low computing power scenarios, builds an independent model reasoning module for the system, abstracts data processing and data services in the system into independent data processing functions, isolates model reasoning from data processing, reduces system redundancy and improves service speed; enables the system to design unconventional data processing logic and data receiving methods to meet the needs of more application scenarios, and can also integrate more model capabilities into the model reasoning module at a low cost, while minimizing the impact on existing businesses, and also enables the system to be applied to application scenarios with more complex business needs, such as personnel behavior norms identification and key personnel monitoring. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 This is a system architecture diagram of the present invention; Figure 2 This is the 3DCNN classification network model diagram of the present invention; Figure 3 The alarm is caused by the movement of personnel in the present invention, and the 3DCNN model recognizes it as a static image; Figure 4 The 3DCNN model recognizes the alarm caused by foreign object interference at night as a static image; Figure 5 For the difficult sample of the belt idling without load in the present invention, the 3DCNN model recognizes it as a motion graph.

[0016] In the figure: 1. Data processing module; 101. Interface service; 102. Supporting video request; 103. Video formatting processing; 2. Model reasoning service module; 201. Dispatching room vacancy warning re-inspection model; 202. Belt operation abnormality warning re-inspection model. DETAILED DESCRIPTION

[0017] The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0018] For examples, see Figure 1-Figure 5 , the present invention provides a technical solution: A coal mine monitoring video alarm intelligent analysis system based on Yolov5 and 3DCNN includes a data processing module 1 and a model reasoning service module 2. The data processing module 1 and the model reasoning service module 2 are electrically connected. The data processing module 1 includes an interface service 101, a supporting video request 102, a video formatting process 103 and a model data process 104. The model reasoning service module 2 includes a dispatch room vacancy warning re-inspection model 201 and a belt operation abnormality warning re-inspection model 202.

[0019] A coal mine monitoring video alarm intelligent analysis method based on Yolov5 and 3DCNN includes an interface service 101, a supporting video request 102, a video formatting process 103, a model data process 104, a dispatch room vacancy warning re-inspection model 201 and a belt operation abnormality warning re-inspection model 202. The interface service includes the following steps: S1. Receive the alarm record information received by the interface request, and perform format and integrity check of the alarm record information, and receive the warning recheck result message returned by the model data processing module for interface return; The corroboration video request 102 includes the following steps: S2. Parse the alarm record information, and request the video cloud storage platform to query the video query interface according to the coal mine code, alarm date, alarm type and other information in the alarm record, query and match the closest video resource address URL to the alarm date, and download the video file according to the video URL; The video formatting process 103 includes the following steps: S3. Responsible for performing video integrity check, video interpolation and frame filling, video serialization and other video file processing on the supporting video files, and unifying the video files to reduce the model inference speed. The video difference frame filling method is to unify the sequence with image height and width as the interpolation dimension. The calculation process is to take the original image of (250, H, W, C) and use H as the coordinate system. The H (250, W, C) are all resized to (61, W, C). Then the result after H dimension unification is (61, H, W, C), or take W as the coordinate system. The W (250, H, C) are all resized to (61, H, C). Then the result after W dimension unification is (61, H, W, C).

[0020] Among them, the data processing processes such as interface service 101 data processing, video data request and download, and video data unified processing are integrated through the data processing module 1, the data processing process is abstracted and a unified data processing mode is provided, and data from different sources and in different formats are processed into a unified data format under the same evaluation index; through the model reasoning service module 2, with the fusion of YoloV5 target detection and 3DCNN video classification and other deep learning algorithms as the technical basis, the dispatching room vacancy warning re-inspection, belt equipment operation abnormal warning re-inspection and other artificial intelligence model reasoning are realized, providing the system with a core algorithm capability base, and the system realizes a general, highly mobile, efficient and reliable video alarm intelligent analysis system in the coal mine production monitoring scenario.

[0021] In this embodiment, Figure 1-Figure 5 As shown, the model data processing 104 includes the following steps: S4. Unify the format and transfer scheduling of data, including formatting and encrypting the warning record information received by the interface service, serializing the video images, unifying the image size, and enhancing the images of the video files after unified processing, forming the video image sequence imgs as the model input data, organizing the reasoning results of the warning re-inspection model and structuring the messages, and visualizing the sampling of the re-inspection result image sequence; The dispatch room vacancy warning re-check model 201 includes the following steps: S5. The dispatch room vacancy warning re-inspection model takes the dispatch room video image sequence imgs, the detection threshold person_t, and the number of warning frames warning_flames as input, detects and locates the human body, human head, arm and other human and edge targets in each image img in the monitoring dispatch room video image sequence imgs, limits the detection of non-human targets according to the detection threshold person_t, and counts the number of images of detected human targets warning_nums, and finally makes a threshold judgment according to the number of warning frames warning_flames. When warning_nums>warning_flames, it determines that imgs is a false alarm, otherwise it is a correct alarm; The belt operation abnormality early warning re-inspection model 202 includes the following steps: S6. The model uses the 3D-CNN video classification model algorithm. Based on the belt motion alarm video image sequence imgs and the recognition threshold belt_t as input, the input video image sequence imgs is subjected to a binary classification task of motion (static) and stillness (moving). The 3D convolutional neural network model infers and calculates the predicted values ​​pred of motion and stillness, and screens the results through the recognition threshold belt_t. When pred>=belt_t, imgs is judged to be in motion and a correct alarm is given. Otherwise, it is still and a false alarm is given. The steps of the video classification model algorithm are: S7.3DCNN network captures dynamic information encoded in multiple adjacent frames by performing three-dimensional convolutions of different scales. Based on the feature extraction capability of 3D convolutional neural network, the image sequence is cropped to 320×240 pixels, and RGB three channels are used as the input of 3D-CNN. In the task, 15 frames are used as the length of the video subsequence. First, the sequence is input into the data enhancement hardwired module, and three channel information (grayscale, horizontal axis gradient (gradient-x), vertical axis gradient (gradient-y)) is extracted from each frame. The information of the three channels can be directly calculated and obtained for each frame; The S8.3D convolution block consists of four layers of convolution in series. The first layer uses a linear activation function with a filter size of 5×5×5. The dimension of the first convolution layer is 13×118×158×32, where the first dimension represents temporal information (sl=13), the second dimension (h=118) and the third dimension (w=158) represent spatial information, and the fourth dimension (c=32) represents channel information. In order to reduce the dimensions of time and space, the step size of this layer is set to 3, and the number of filters is 32. In order to avoid overfitting and reduce information redundancy, each layer is To perform a dropout operation, according to the dropout principle, the dropout operation means randomly discarding a certain number of neurons in the layer (and the connections between them) with a certain probability. In order to speed up the convergence of model training, each layer must be normalized. For the 2nd, 3rd, and 4th layers, the same parameters are used for setting. The difference is that the filter size is changed to 3×3×3, 3×3×3, and 1×1×1, and the number of feature maps is 48 and 64 respectively. In addition, the ReLU function is used as the activation function; S9. After the 3D convolution block, a 128-dimensional fully connected layer is used to extract classification features of the short-term spatiotemporal feature map, and finally perform a binary classification task of whether it is motion or not. It should be noted that the 4th layer in the 3D convolution block uses a 1×1×1 convolution kernel, which is also equivalent to a certain degree of fully connected operation.

[0022] The working process of the present invention: When the coal mine monitoring video alarm intelligent analysis system and method based on Yolov5 and 3DCNN designed by the present invention is working, it is composed of a data processing module 1 and a model reasoning service module 2, which jointly complete the video alarm intelligent analysis task in the coal mine monitoring scene. The data processing module 1 performs unified image and video formatting processing on the alarm data of each coal mine scene, and is responsible for video data request, service data reception and message return. The model reasoning service module 2 performs analysis model reasoning on each coal mine alarm supporting video, including video human body detection model reasoning for the unmanned vacant post warning in the dispatching room, and video motion recognition model reasoning for the coal mine shaft mouth conveyor belt equipment movement warning, realizing a coal mine monitoring video alarm intelligent analysis work with high recognition accuracy and good mobility.

[0023] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A coal mine monitoring video alarm intelligent analysis system based on Yolov5 and 3DCNN, comprising a data processing module (1) and a model reasoning service module (2), characterized in that: The data processing module (1) and the model inference service module (2) are electrically connected, the data processing module (1) comprising an interface service (101), a supporting video request (102), a video formatting process (103) and a model data process (104), and the model inference service module (2) comprising a dispatch room vacancy warning re-inspection model (201) and a belt operation abnormality warning re-inspection model (202).

2. A method for intelligent analysis of coal mine monitoring video alarms based on Yolov5 and 3DCNN, applicable to the coal mine monitoring video alarm intelligent analysis system based on Yolov5 and 3DCNN described in claim 1, comprising an interface service (101), a supporting video request (102), a video formatting process (103), a model data process (104), a dispatch room vacancy warning re-inspection model (201) and a belt operation abnormality warning re-inspection model (202), characterized in that: The interface service comprises the following steps: S1. Receive the alarm record information requested by the interface, perform format and integrity verification on the alarm record information, and receive the early warning recheck result message returned by the model data processing module for interface return.

3. According to claim 2, a coal mine monitoring video alarm intelligent analysis method based on Yolov5 and 3DCNN is characterized in that: The corroboration video request (102) comprises the following steps: S2. Analyze the alarm record information, and according to the coal mine code, alarm date, alarm type and other information in the alarm record, request the supporting video query interface in the video cloud storage platform, query and match the supporting video resource address URL closest to the alarm date, and download the supporting video file according to the video URL.

4. According to claim 2, a coal mine monitoring video alarm intelligent analysis method based on Yolov5 and 3DCNN is characterized in that: The video formatting process (103) includes the following steps: S3. Responsible for performing video integrity check, video interpolation and frame filling, video serialization and other video file processing on the supporting video files, and unifying the video files to reduce the model inference speed. The video difference frame filling method is to unify the sequence with image height and width as the interpolation dimension. The calculation process is to take the original image of (250, H, W, C) and use H as the coordinate system. The H (250, W, C) are all resized to (61, W, C). Then the result after H dimension unification is (61, H, W, C), or take W as the coordinate system. The W (250, H, C) are all resized to (61, H, C). Then the result after W dimension unification is (61, H, W, C).

5. According to claim 2, a coal mine monitoring video alarm intelligent analysis method based on Yolov5 and 3DCNN is characterized in that: The model data processing (104) includes the following steps: S4. Unify the format and transfer scheduling of data, including formatting and encrypting the warning record information received by the interface service, serializing the video images, unifying the image size, and enhancing the images of the unified video files to form the video image sequence imgs as the model input data, organizing the reasoning results of the warning re-inspection model and structuring the messages, and visualizing the sampling of the re-inspection result image sequence.

6. The method for intelligent analysis of coal mine monitoring video alarms based on Yolov5 and 3DCNN according to claim 2 is characterized in that: The dispatch room vacancy warning re-check model (201) comprises the following steps: S5. The dispatch room vacant post warning re-inspection model takes the dispatch room video image sequence imgs, the detection threshold person_t, and the number of alarm frames warning_flames as input, detects and locates human bodies, human heads, arms and other human and edge targets in each image img in the monitoring dispatch room video image sequence imgs, limits the detection of non-human targets according to the detection threshold person_t, and counts the number of images in which human targets are detected warning_nums. Finally, a threshold judgment is made according to the number of alarm frames warning_flames. When warning_nums>warning_flames, imgs is judged to be a false alarm, otherwise it is a correct alarm.

7. The method for intelligent analysis of coal mine monitoring video alarms based on Yolov5 and 3DCNN according to claim 2 is characterized in that: The belt operation abnormality early warning re-inspection model (202) comprises the following steps: S6. The model adopts the 3D-CNN video classification model algorithm. It takes the belt movement alarm video image sequence imgs and the recognition threshold belt_t as input, and performs a binary classification task of motion (static) and stillness (moving) on ​​the input video image sequence imgs. The 3D convolutional neural network model infers and calculates the predicted values ​​pred of motion and stillness, and screens the results through the recognition threshold belt_t. When pred>=belt_t, imgs is judged to be in motion and a correct alarm is given. Otherwise, it is still and a false alarm is given.

8. The method for intelligent analysis of coal mine monitoring video alarms based on Yolov5 and 3DCNN according to claim 7 is characterized in that: The steps of the video classification model algorithm are: S7.3DCNN network captures dynamic information encoded in multiple adjacent frames by performing three-dimensional convolutions of different scales. Based on the feature extraction capability of 3D convolutional neural network, the image sequence is cropped to 320×240 pixels, and RGB three channels are used as the input of 3D-CNN. In the task, 15 frames are used as the length of the video subsequence. First, the sequence is input into the data enhancement hardwired module, and three channel information (grayscale, horizontal axis gradient (gradient-x), vertical axis gradient (gradient-y)) is extracted from each frame. The information of the three channels can be directly calculated and obtained for each frame; The S8.3D convolution block consists of four layers of convolution in series. The first layer uses a linear activation function with a filter size of 5×5×5. The dimension of the first convolution layer is 13×118×158×32, where the first dimension represents temporal information (sl=13), the second dimension (h=118) and the third dimension (w=158) represent spatial information, and the fourth dimension (c=32) represents channel information. In order to reduce the dimensions of time and space, the step size of this layer is set to 3, and the number of filters is 32. In order to avoid overfitting and reduce information redundancy, each layer is To perform a dropout operation, according to the dropout principle, the dropout operation means randomly discarding a certain number of neurons in the layer (and the connections between them) with a certain probability. In order to speed up the convergence of model training, each layer must be normalized. For the 2nd, 3rd, and 4th layers, the same parameters are used for setting. The difference is that the filter size is changed to 3×3×3, 3×3×3, and 1×1×1, and the number of feature maps is 48 and 64 respectively. In addition, the ReLU function is used as the activation function; S9. After the 3D convolution block, a 128-dimensional fully connected layer is used to extract classification features of the short-term spatiotemporal feature map, and finally perform a binary classification task of whether it is motion or not. It should be noted that the 4th layer in the 3D convolution block uses a 1×1×1 convolution kernel, which is also equivalent to a certain degree of fully connected operation.

Images