Coal mine safety early warning method and device and electronic equipment

Abstract

The invention provides a coal mine safety early warning method and device and electronic equipment. The method comprises the following steps: acquiring underground image data of a coal mine; analyzing the image data through a target detection algorithm to obtain a pedestrian position and a vehicle position, tracking the pedestrian position through a target tracking algorithm to obtain a pedestrian trajectory, and tracking the vehicle position through the target tracking algorithm to obtain a vehicle trajectory; and determining whether the vehicle is in a moving state according to the vehicle track, determining the distance and the relative position between the vehicle and the pedestrian according to the pedestrian track and the vehicle track under the condition that the vehicle is in the moving state, and determining whether the vehicle is safe according to the distance and the relative position. According to the invention, the problem of high arrangement cost of a coal mine underground man-vehicle identification method in the prior art is solved.

Description

Technical Field

[0001] The present application relates to the technical field of underground safety management of coal mines, and in particular to a coal mine safety early warning method, device, computer-readable storage medium and electronic device. Background Art

[0002] The underground environment of coal mines is complex, and the safety management of pedestrians and vehicles has always been a key issue in coal mine production. In the transportation tunnels underground in coal mines, because the tunnels are narrow and the line of sight is obstructed, it is very easy for casualties to occur in underground auxiliary transportation. In order to effectively avoid the coexistence of people and vehicles in underground tunnels and reduce the probability of safety accidents in transportation tunnels, technical means can be used to strengthen the supervision of pedestrians and vehicles. At present, it mainly relies on manual viewing of video surveillance to determine whether there is a situation of people and vehicles traveling together. This method not only consumes a lot of human resources and is inefficient, but also belongs to passive identification and cannot achieve real-time alarm. There are currently two types of research on automatic identification of pedestrians but not vehicles underground in coal mines. The first is based on sensors. Due to the complex underground environment and the high noise level when the vehicle is running, it is difficult for the driver to determine whether there is someone near the vehicle through the sensor's alarm. The second is based on underground wireless communication technology, such as RFID (Radio Frequency Identification, also known as electronic tags, wireless radio frequency identification, is a communication technology that can identify specific targets and read and write related data through radio signals without the need to establish mechanical or optical contact between the identification system and the specific target), UWB (Ultra-Wideband, UWB positioning technology is a technology that uses large-bandwidth electromagnetic wave signals for precise positioning), etc. The positioning card is installed in the vehicle. If a pedestrian approaches the device, the distance is determined by the positioning card of the pedestrian and the vehicle itself, thereby determining whether there is a situation where people and vehicles are traveling together. However, this method has problems such as limited communication distance, high layout cost, and expensive card reader.

[0003] Therefore, a method for identifying unsafe behaviors of pedestrians but not vehicles in coal mines is needed to warn the status of vehicles and pedestrians in time. Summary of the invention

[0004] The main purpose of the present application is to provide a coal mine safety early warning method, device, computer-readable storage medium and electronic device to at least solve the problem of high deployment cost of the coal mine underground human and vehicle identification method in the prior art.

[0005] In order to achieve the above-mentioned purpose, according to one aspect of the present application, a coal mine safety early warning method is provided, comprising: acquiring image data underground in a coal mine, wherein the image data includes at least pedestrians and vehicles; analyzing the image data through a target detection algorithm to obtain pedestrian positions and vehicle positions, and tracking the pedestrian positions through a target tracking algorithm to obtain pedestrian trajectories, and tracking the vehicle positions through a target tracking algorithm to obtain vehicle trajectories, wherein the target detection algorithm is obtained by training a machine learning model using multiple sets of data, and each set of data in the multiple sets of data includes the image data and an image containing pedestrian labels and vehicle labels; determining whether the vehicle is in a moving state according to the vehicle trajectory, and when the vehicle is in the moving state, determining the distance and relative position between the vehicle and the pedestrian according to the pedestrian trajectory and the vehicle trajectory, and determining whether it is safe according to the distance and the relative position, wherein the relative position includes the pedestrian in front of the vehicle, the pedestrian to the side of the vehicle, and the pedestrian behind the vehicle.

[0006] Optionally, the position of the pedestrian is tracked by a target tracking algorithm to obtain the pedestrian trajectory, and the position of the vehicle is tracked by a target tracking algorithm to obtain the vehicle trajectory, including: acquiring a first frame of image data, and determining an initial state vector of the target in the first frame of image data, wherein the initial state vector includes at least the center point coordinates of the target, and the targets include the pedestrian and the vehicle; predicting the position and speed of the target in the next frame of image data to obtain a predicted state vector, and updating the motion trajectory of the target according to the predicted state vector to obtain the pedestrian trajectory and the vehicle trajectory.

[0007] Optionally, determining whether the vehicle is in a moving state based on the vehicle trajectory includes: obtaining the center coordinates of the vehicle and the detection frame area of ​​the vehicle in the first frame image data to obtain initial center coordinates and initial area; obtaining the center coordinates of the vehicle and the detection frame area of ​​the vehicle in the current frame image data to obtain current center coordinates and current area; calculating the absolute value of the difference between the current center coordinates and the initial center coordinates to obtain a center coordinate difference, and calculating the absolute value of the difference between the initial area and the current area to obtain an area difference; when the center coordinate difference is greater than a preset coordinate threshold and / or the area difference is greater than a preset area threshold, determining that the vehicle is in the moving state.

[0008] Optionally, determining the distance and relative position between the vehicle and the pedestrian based on the pedestrian trajectory and the vehicle trajectory includes: determining the driving direction of the vehicle based on the vehicle trajectory, calculating the angle between the pedestrian trajectory and the driving direction to determine the relative position between the pedestrian and the vehicle; obtaining the coordinate position of the pedestrian and the coordinate position of the vehicle, calculating the absolute value of the difference between the coordinate position of the pedestrian and the coordinate position of the vehicle, and obtaining the distance.

[0009] Optionally, determining whether it is safe based on the distance and the relative position includes one of the following: determining that it is unsafe when the relative position is that the pedestrian is in front of the vehicle and the distance is less than a distance threshold; determining that it is unsafe when the relative position is that the pedestrian is to the side of the vehicle and the distance is less than a distance threshold.

[0010] Optionally, after acquiring the image data of the underground coal mine, the method further comprises: performing image enhancement on the image data to obtain image data after image enhancement; and performing image cleaning on the image data after image enhancement.

[0011] Optionally, before analyzing the image data through a target detection algorithm, the method also includes: labeling pedestrians and vehicles in the image data to obtain an image including the pedestrian labels and the vehicle labels; constructing an initial target detection model, and training the initial target detection model using the image data and the image including the pedestrian labels and the vehicle labels to obtain the target detection model.

[0012] According to another aspect of the present application, a coal mine safety early warning device is provided, comprising: an acquisition unit, for acquiring image data underground in a coal mine, wherein the image data includes at least pedestrians and vehicles; a tracking unit, for analyzing the image data through a target detection algorithm to obtain pedestrian positions and vehicle positions, and tracking the pedestrian positions through a target tracking algorithm to obtain pedestrian trajectories, and tracking the vehicle positions through a target tracking algorithm to obtain vehicle trajectories, wherein the target detection algorithm is obtained by training a machine learning model using multiple sets of data, and each set of data in the multiple sets of data includes the image data and images containing pedestrian labels and vehicle labels; a determination unit, for determining whether the vehicle is in a moving state according to the vehicle trajectory, and when the vehicle is in the moving state, determining the distance and relative position between the vehicle and the pedestrian according to the pedestrian trajectory and the vehicle trajectory, and determining whether it is safe according to the distance and the relative position, wherein the relative position includes the pedestrian in front of the vehicle, the pedestrian to the side of the vehicle, and the pedestrian behind the vehicle.

[0013] According to another aspect of the present application, a computer-readable storage medium is provided, the computer-readable storage medium including a stored program, wherein when the program is executed, the device where the computer-readable storage medium is located is controlled to execute any one of the coal mine safety early warning methods.

[0014] According to another aspect of the present application, an electronic device includes: one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and the one or more programs include a method for executing any one of the coal mine safety early warning methods.

[0015] The technical solution of the present application is applied to obtain image data of underground coal mines, wherein the image data includes at least pedestrians and vehicles; the image data is analyzed by a target detection algorithm to obtain the position of pedestrians and vehicles, and the position of pedestrians is tracked by a target tracking algorithm to obtain the pedestrian trajectory, and the position of vehicles is tracked by a target tracking algorithm to obtain the vehicle trajectory, wherein the target detection algorithm is obtained by training a machine learning model using multiple sets of data, and each set of data in the multiple sets of data includes image data and images containing pedestrian labels and vehicle labels; according to the vehicle trajectory, it is determined whether the vehicle is in a moving state, and when the vehicle is in a moving state, the distance and relative position between the vehicle and the pedestrian are determined according to the pedestrian trajectory and the vehicle trajectory, and whether it is safe is determined according to the distance and relative position. Compared with the prior art, in which the pedestrian and vehicle recognition technology needs to consume a lot of manpower and material resources to arrange, the image analysis is performed by the target detection algorithm, and the position and distance between the vehicle and the pedestrian are determined, and whether it is safe is further determined, which can save manpower and material resources and accurately judge the safety of coal mines. Therefore, it can solve the problem of high arrangement cost of the method for identifying people and vehicles in underground coal mines in the prior art, and achieve the effect of accurately identifying the safety of coal mines. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The drawings constituting part of the present application are used to provide a further understanding of the present application. The illustrative embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0017] Figure 1 A hardware structure block diagram of a mobile terminal for executing a coal mine safety early warning method provided in an embodiment of the present application is shown;

[0018] Figure 2 A schematic diagram of a process of a coal mine safety early warning method provided in an embodiment of the present application is shown;

[0019] Figure 3 A C2F module network structure diagram provided in an embodiment of the present application is shown;

[0020] Figure 4 An Efficient-RepGFPN network structure diagram provided by an embodiment of the present application is shown;

[0021] Figure 5 A schematic diagram of a PR curve of a verification model provided in an embodiment of the present application is shown;

[0022] Figure 6 A pedestrian and vehicle running trajectory tracking result diagram provided by an embodiment of the present application is shown;

[0023] Figure 7 A result diagram of pedestrian and vehicle unsafe state recognition provided by an embodiment of the present application is shown;

[0024] Figure 8 A structural diagram of an image preprocessing module provided in an embodiment of the present application is shown;

[0025] Fig. 9 A test result diagram of a pedestrian and vehicle target detection model provided by an embodiment of the present application is shown;

[0026] Fig.10 A specific coal mine safety early warning method provided by an embodiment of the present application is shown;

[0027] Fig.11 A structural block diagram of a coal mine safety early warning device provided in an embodiment of the present application is shown.

[0028] The above drawings include the following reference numerals:

[0029] 102, processor; 104, memory; 106, transmission device; 108, input and output devices. DETAILED DESCRIPTION

[0030] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0031] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0032] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present application described here. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0033] As introduced in the background technology, the existing method for identifying people and vehicles in coal mines has high deployment costs. In order to solve the problem of high deployment costs of methods for identifying people and vehicles in coal mines, the embodiments of the present application provide a coal mine safety early warning method, device, computer-readable storage medium and electronic device.

[0034] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0035] The method embodiments provided in the embodiments of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Taking running on a mobile terminal as an example, Figure 1 1 is a hardware structure block diagram of a mobile terminal of a coal mine safety early warning method according to an embodiment of the present invention. Figure 1 As shown, the mobile terminal may include one or more ( Figure 1 Only one is shown in the figure) a processor 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA) and a memory 104 for storing data, wherein the mobile terminal may also include a transmission device 106 and an input / output device 108 for communication functions. It can be understood by those skilled in the art that Figure 1 The structure shown is for illustration only and does not limit the structure of the mobile terminal. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations shown.

[0036] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the coal mine safety early warning method in the embodiment of the present invention. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, the above method is implemented. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories can be connected to the mobile terminal via a network. Examples of the above-mentioned network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof. The transmission device 106 is used to receive or send data via a network. The above-mentioned specific example of the network may include a wireless network provided by a communication provider of the mobile terminal. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 may be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0037] In this embodiment, a coal mine safety early warning method running on a mobile terminal, a computer terminal or a similar computing device is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and, although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0038] Figure 2 Flowchart of the coal mine safety early warning method according to the embodiment of the present application. Figure 2 As shown, the method comprises the following steps:

[0039] Step S201, acquiring image data of a coal mine, wherein the image data at least includes pedestrians and vehicles;

[0040] Specifically, high-resolution cameras deployed at key locations underground in coal mines are used to collect image data of pedestrians and vehicles.

[0041] Step S202, analyzing the image data by a target detection algorithm to obtain a pedestrian position and a vehicle position, and tracking the pedestrian position by a target tracking algorithm to obtain a pedestrian trajectory, and tracking the vehicle position by a target tracking algorithm to obtain a vehicle trajectory, wherein the target detection algorithm is obtained by training a machine learning model using multiple sets of data, and each set of data in the multiple sets of data includes the image data and an image including a pedestrian label and a vehicle label;

[0042] Specifically, the target detection algorithm is used to detect pedestrians and vehicles in the image data. The target tracking algorithm is used to achieve continuous and stable trajectory tracking of pedestrians and vehicles in underground tunnels, accurately record the motion state of each detection target, reduce tracking interruption problems caused by target occlusion, overlap and other factors, and obtain pedestrian and vehicle trajectories. The target detection algorithm adopts the improved yolov5 model. On the basis of the yolov5 model, the C2F lightweight module is integrated. Through branch processing, feature fusion and other operations, the representation ability of the network is improved, and Neck is replaced by Efficient-RepGFPN to improve the multi-scale detection ability of the model. At the same time, the loss function is optimized to further improve the detection accuracy of the model. Specific description of improved yolov5 structure: The improvements of yolov5 network include: for the structure in the Backbone network, the C3 module is replaced by the C2F module, while ensuring the lightweight of the model, richer gradient flow information is obtained, thereby improving the detection accuracy of the model; Neck is replaced by Efficient-RepGFPN, and multi-scale features are more efficiently fused through feature repetition and global feature pyramid pooling, capturing high-level semantic information and low-level spatial position information; the loss function is changed to FocalEIoU to further improve the detection accuracy of the model. The C2F module convolves and pools feature maps of different scales to obtain feature maps of multiple different scales; for each scale of feature maps, convolution and upsampling operations are used to adjust its scale to a uniform size; the adjusted feature maps are spliced ​​to obtain a fused feature map; some convolution operations and feature attention mechanisms are performed on the fused feature maps to further improve the representation ability and detection accuracy of the feature maps. The network structure diagram of the C2F module is shown below. Figure 3 Efficient-RepGFPN is a feature pyramid network based on feature repetition and global feature pyramid pooling. The network structure is shown in the figure below. Figure 4. Specifically, it includes the following modules: Feature repetition module: repeat the feature map multiple times and splice it with the original feature map to improve the resolution and richness of the feature map. Conventional feature pyramid module: use multiple convolutional layers and pooling layers to build a feature pyramid to improve the scale and richness of the feature map. Global feature pyramid pooling module: use different pooling scales to extract global features and splice them with the conventional feature pyramid to improve the richness and generalization ability of the feature map. Feature fusion module: use feature attention mechanism and feature cascade to fuse feature maps of various scales, thereby improving the representation ability and accuracy of the feature map. FocalEIoU is an improved IoU calculation method. FocalEIoU introduces a category weight and sample weight based on the traditional IoU calculation method, which can better handle the situation of category imbalance, thereby improving the detection accuracy. Specifically, the calculation method of FocalEIoU is as follows:

[0043]

[0044] Among them, n represents the number of intersections between the detection box and the real box; w i represents the weight of each category; f i Indicates the frequency of each category, which can be estimated by the number of samples; γ represents a hyperparameter used to balance the weights of positive and negative samples. The training set is sent to the improved yolov5 model, and the model parameters are optimized using the batch stochastic gradient descent method. Specific instructions for model training: Use the GPU server to train the improved yolov5 model, train 300 epochs based on the empirical value to obtain a better model weight, and verify the PR curve of the model, which represents the relationship between accuracy and recall, such as Figure 5 shown.

[0045] Step S203, determining whether the vehicle is in a moving state based on the vehicle trajectory. When the vehicle is in the moving state, determining the distance and relative position between the vehicle and the pedestrian based on the pedestrian trajectory and the vehicle trajectory, and determining whether it is safe based on the distance and the relative position, wherein the relative position includes the pedestrian in front of the vehicle, the pedestrian to the side of the vehicle, and the pedestrian behind the vehicle.

[0046] Specifically, if the vehicle is in a moving state, the vehicle's driving direction and the relative position of the pedestrian and the vehicle are further determined, the distance between the pedestrian and the vehicle and the corresponding safety distance threshold are calculated, and the two are compared to determine whether the pedestrian and the vehicle are in an unsafe state. If so, an alarm is automatically generated.

[0047] Through this embodiment, image data of underground coal mine is obtained, and the image data includes at least pedestrians and vehicles; the image data is analyzed by target detection algorithm to obtain the position of pedestrians and vehicles, and the position of pedestrians is tracked by target tracking algorithm to obtain the pedestrian trajectory, and the position of vehicles is tracked by target tracking algorithm to obtain the vehicle trajectory, and the target detection algorithm is obtained by training multiple sets of data through machine learning model, and each set of data in multiple sets of data includes image data and images containing pedestrian labels and vehicle labels; according to the vehicle trajectory, it is determined whether the vehicle is in a moving state, and when the vehicle is in a moving state, the distance and relative position between the vehicle and the pedestrian are determined according to the pedestrian trajectory and the vehicle trajectory, and whether it is safe is determined according to the distance and relative position. Compared with the prior art, the pedestrian and vehicle recognition technology needs to consume a lot of manpower and material resources to arrange, in this way, image analysis is performed by target detection algorithm, and the position and distance of the vehicle and pedestrian are determined, and whether it is safe is further determined, which can save manpower and material resources, and accurately judge the safety of coal mines. Therefore, it can solve the problem of high cost of arrangement of the method for identifying people and vehicles in underground coal mines in the prior art, and achieve the effect of accurately identifying the safety of coal mines.

[0048] In the specific implementation process, the above step S202 tracks the position of the pedestrian through the target tracking algorithm to obtain the pedestrian trajectory, and tracks the position of the vehicle through the target tracking algorithm to obtain the vehicle trajectory, which can be achieved through the following steps: Step S2022: Acquire the first frame of image data, and determine the initial state vector of the target in the first frame of image data, wherein the initial state vector at least includes the center point coordinates of the target, and the target includes the pedestrian and the vehicle; Step S2023: Predict the position and speed of the target in the next frame of image data to obtain the predicted state vector, and update the motion trajectory of the target according to the predicted state vector to obtain the pedestrian trajectory and the vehicle trajectory. This method determines the trajectory of pedestrians and vehicles through each frame of image data, so that the trajectory can be accurately determined.

[0049] Specifically, the overall process of tracking the trajectory of pedestrians and vehicles is as follows: 1) Construct a track Tracks for each target in the first frame detection result, and initialize the Kalman parameters of the target, represented by the state vector x = [cx, cy, r, h, vx, vy, vr, vh], where (cx, cy) is the coordinate of the center point of the target, y is the aspect ratio of the target, h is the height of the target, and vx, vy, vr, vh represent the change speed of the four variables respectively. 2) For each identified pedestrian and vehicle target, use the pre-trained deep learning model to extract its appearance features, and maintain its category and feature information to reduce the probability of frequent switching of the target ID. 3) Use the Kalman filter to predict the position and speed of Tracks in the next frame. 4) Perform cascade matching (appearance information) and IOU matching (motion information) on the Tracks predicted in the previous frame and the detection box of the current frame, and build a cost matrix based on this. The cost matrix is ​​used as the input of the Hungarian algorithm to obtain the optimal matching result, thereby updating the tracking trajectory. 5) During the entire video frame processing process, Tracks is maintained and updated, and the tracking results of each target are finally output, including the location, ID and other relevant information of each target. Figure 6 The figure shows the results of tracking the trajectory of pedestrians and vehicles. The figure shows that "person 0.96 3" and "person 0.981" represent pedestrians, and "person 0.97 2" represents a vehicle.

[0050] In some optional embodiments, the above step S203 determines whether the vehicle is in a moving state according to the vehicle trajectory, which can be achieved by the following steps: Step S2031: Obtain the center coordinates of the vehicle and the detection frame area of ​​the vehicle in the first frame image data to obtain the initial center coordinates and the initial area; Step S2032: Obtain the center coordinates of the vehicle and the detection frame area of ​​the vehicle in the current frame image data to obtain the current center coordinates and the current area; Step S2033: Calculate the absolute value of the difference between the current center coordinate and the initial center coordinate to obtain the center coordinate difference, and calculate the absolute value of the difference between the initial area and the current area to obtain the area difference; Step S2034: When the center coordinate difference is greater than the preset coordinate threshold and / or the area difference is greater than the preset area threshold, determine that the vehicle is in the moving state. This method can accurately determine whether the vehicle is in a moving state through the coordinate difference and the area difference.

[0051] Specifically, the motion trajectory information of pedestrians and vehicles is analyzed and the center point coordinates are recorded using the trajectory list V_Tracks={(V_Xi, V_Yi)|i=0,1,2,...,n}, where i represents the number of detections and (V_Xi, V_Yi) represents the center point coordinates of the target vehicle detected for the i-th time. The area list V_Area is used to record the area of ​​the vehicle detection box in each frame. When the center point coordinates of a vehicle are tracked for a total of 10 frames, the distance and area change between the center point of the vehicle in the current frame and the first frame are compared. If the relative movement distance of the vehicle (normalized to the image height) exceeds the preset threshold dist_thresh, or the change in the vehicle area exceeds another preset threshold area_thresh, the vehicle is recorded to have moved. If the vehicle movement is recorded for 5 consecutive times, it is determined that the vehicle is in operation. The target vehicle moving distance calculation formula is:

[0052]

[0053] Where (x1, y1) is the coordinate of the center point of the vehicle in the first frame, (x2, y2) is the coordinate of the center point of the vehicle in the current frame, and h is the image height. The calculation formula for the target vehicle area change is:

[0054]

[0055] Where (box[0], box[1]) represents the coordinates of the lower left corner of the vehicle detection box, (box[2], box[3]) represents the coordinates of the upper right corner of the vehicle detection box, area0 represents the area of ​​the vehicle detection box in the first frame, and h and w represent the image height and width.

[0056] In some optional implementations, step S203 determines the distance and relative position between the vehicle and the pedestrian according to the pedestrian trajectory and the vehicle trajectory, which can also be achieved by the following steps: step S2035: determine the driving direction of the vehicle according to the vehicle trajectory, calculate the angle between the pedestrian trajectory and the driving direction to determine the relative position between the pedestrian and the vehicle; step S2036: obtain the coordinate position of the pedestrian and the coordinate position of the vehicle, calculate the absolute value of the difference between the coordinate position of the pedestrian and the coordinate position of the vehicle, and obtain the distance. This method can accurately determine the distance and relative position between the vehicle and the pedestrian through the driving direction and angle of the above-mentioned vehicle.

[0057] In the specific implementation process, 1) Determine the vehicle's driving direction: When the vehicle is in operation, determine whether the vehicle is moving horizontally or vertically by the angle of the vehicle's movement. The angle calculation formula between the vehicle's driving route and the x-axis is:

[0058]

[0059] Where (x1, y1) and (x2, y2) are the center coordinates of the vehicle in the first frame and the current frame respectively, arctan(x) is the inverse tangent function, and deg(x) is the radian conversion angle function.

[0060] 2) Determine the vehicle's orientation: Calculate the angle with the vehicle's moving direction. The angle calculation formula is:

[0061]

[0062] Where a, b, and c represent the three sides of the triangle formed by the center points of the vehicle in the first frame and the current frame, and the center point of the pedestrian in the current frame, respectively. Arccos(x) is the inverse cosine function. The formula for calculating the side length is:

[0063]

[0064] Threshold determination method: when the angle θ is less than 20°, the person is judged to be in front of the vehicle; when the angle is 20° less than θ less than 90°, the person is judged to be on the side of the vehicle; when the angle θ is greater than 90°, the person is judged to be behind the vehicle. The above thresholds are determined based on the on-site application results of Shendong underground mines.

[0065] 3) Pedestrian-vehicle distance calculation: When a person is in front of or beside the vehicle, there is a safety hazard, and the distance between the two must be calculated based on the coordinate positions of the vehicle and pedestrian in the current frame. When the distance is less than the preset safety threshold, it is determined that the pedestrian and vehicle are in an unsafe state.

[0066] In particular, because the underground tunnels are narrow, the required safety distances for pedestrians and vehicles in different directions of the tunnels are also different. For example, when a person is on the side of the car, the safety threshold of the distance between people and vehicles is relatively small, and when a person is diagonally in front of the car, the threshold of the distance between people and vehicles can be slightly larger. Therefore, based on the cases of unsafe behaviors in underground tunnels of coal mines, the present invention sets safety distance thresholds for different situations respectively, and repeatedly corrects them through experiments. At the same time, because the images taken by the camera have the effect of being larger near and smaller far away, the pixels occupied by the same person at the far end of the image are much smaller than the pixels occupied by the near end of the image, and the distance between people and vehicles will also change accordingly. In order to reduce this influence, the present invention uses the height of a person as the reference system, and obtains a mapping function by calculating the ratio of the pixel height of the person in the image to the actual height, and uses the mapping function to map the actual safety distance between people and vehicles to the pixel safety distance. The calculation formula for the pixel safety distance threshold between people and vehicles is as follows:

[0067]

[0068] Among them, h pixel is the pixel height of a person, h actualis the actual height of the person (the actual height is the average height of Shenhua miners), h0 is the image height, β is the safe distance between people and vehicles specified in the coal mine, and λ is used to adjust the contribution of the distance between people and the camera to the pixel safety distance threshold.

[0069] In some optional implementations, determining the distance and the relative position to determine whether it is safe includes one of the following: determining that it is unsafe when the relative position is that the pedestrian is in front of the vehicle and the distance is less than a distance threshold; determining that it is unsafe when the relative position is that the pedestrian is to the side of the vehicle and the distance is less than a distance threshold. The method can quickly and accurately determine whether pedestrians and vehicles in a coal mine are safe through the above steps.

[0070] Figure 7 This is the result diagram of pedestrian and vehicle unsafe state recognition. Figure 7 (a) is the previous frame image, Figure 7 (b) is the next frame. When the pedestrian and vehicle are judged to be in an unsafe state, the upper left corner of the vehicle detection frame shows "Pedestrians are not vehicles". The platform immediately pushes the warning information to the relevant personnel and saves the unsafe behavior video for subsequent analysis.

[0071] In some optional implementations, after acquiring the image data of the coal mine, the method further includes: performing image enhancement on the image data to obtain the image data after image enhancement; and performing image cleaning on the image data after image enhancement. In this way, the image can be pre-processed into the format of the algorithm model to facilitate subsequent training of the algorithm model.

[0072] Specifically, due to the complex environment of coal mines, there are often problems such as low illumination, dust and fog, lack of color information, and uneven illumination. These unfavorable factors will seriously affect the image quality, increase the learning difficulty of the target detection model, and make it difficult to accurately capture the typical features of the image. In order to solve the above problems, a differentiable filter with adjustable hyperparameters is used to preprocess the image from the aspects of dehazing, white balance (WB), gamma, contrast enhancement, tone adjustment, and sharpening. The preprocessing module structure is as follows: Figure 8As shown. In order to avoid the neural network consuming a lot of computing resources, the high-resolution image obtained by the camera needs to be downsampled to obtain a low-resolution image of 256×256. The low-resolution image is then sent to a small convolutional neural network (CNN-S), which passes through five convolutional modules and two fully connected layers in sequence to predict the optimal hyperparameters of the filter, and the filter is applied to the original resolution image to improve the image quality. Among them, the convolution module in the CNN-S network consists of five convolutional modules and two fully connected layers. Each convolutional module includes a 3×3 convolutional layer with a stride of 2 and a leakyRelu layer. The last fully connected layer outputs the predicted value of the hyperparameter. After the image is sent to the image preprocessing module for image enhancement operations such as brightness enhancement, hue adjustment, and defogging, the image is cleaned to remove images with blurred and unclear targets. The LabelImg tool is then used to annotate the image. In order to improve the accuracy of the model, some locations similar to vehicles and pedestrians are annotated as negative samples. Finally, an xml file is generated, and the training set and test set are divided proportionally.

[0073] In some optional implementations, before analyzing the image data by a target detection algorithm, the method further includes: labeling pedestrians and vehicles in the image data to obtain an image including the pedestrian label and the vehicle label; constructing an initial target detection model, and training the initial target detection model using the image data and the image including the pedestrian label and the vehicle label to obtain the target detection model. The method trains the target detection model through the above steps, so that the model can be trained to be a model suitable for detecting pedestrians and vehicles, so as to facilitate subsequent target detection.

[0074] In the specific implementation process, the training set is sent to the improved yolov5 model, and the batch stochastic gradient descent method is used to optimize the model parameters. Specific instructions for model training: Use the GPU server to train the improved yolov5 model, train 300 epochs based on the empirical value to obtain a better model weight, and verify the PR curve of the model, such as Figure 5 As shown. The trained model is used to detect pedestrians and vehicles in the video frame. Specific description of target detection: After the training model is completed, a weight file is obtained. This weight file is used to test the model and mAp calculation to obtain a detection rate of 92.52% for pedestrians and vehicles. Fig. 9 The following is the test result of the pedestrian and vehicle target detection model. Fig. 9 (a) is the image before detection. Fig. 9 (b) is the image after detection using the improved yolov5 model.

[0075] In order to enable those skilled in the art to more clearly understand the technical solution of the present application, the implementation process of the coal mine safety early warning method of the present application will be described in detail below in combination with specific embodiments.

[0076] This embodiment relates to a specific coal mine safety early warning method, such as Fig.10 As shown, the following steps are included:

[0077] Step S1: Create a coal mine underground miner and vehicle dataset;

[0078] Step S2: construct an improved yolov5 model, including the C2F module, the Efficient-RepGFPN module and the loss function improvement;

[0079] Step S3: train the improved yolov5 network model to obtain weights;

[0080] Step S4: Use the trained improved yolov5 network model to detect miners (pedestrians) and vehicles in the video stream data;

[0081] Step S5: Track the movement trajectories of miners and vehicles and construct a trajectory list;

[0082] Step S6: judging whether the vehicle is moving according to the coordinates of the center point of the vehicle and the change in area;

[0083] Step S7: judging whether the miner and the vehicle are in an unsafe state according to the moving direction of the vehicle and the distance between the person and the vehicle, and if so, giving an alarm.

[0084] The embodiment of the present application also provides a coal mine safety early warning device. It should be noted that the coal mine safety early warning device of the embodiment of the present application can be used to execute the coal mine safety early warning method provided by the embodiment of the present application. The device is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware of a predetermined function. Although the device described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceived.

[0085] The following is an introduction to the coal mine safety early warning device provided in the embodiment of the present application.

[0086] Fig.11 Schematic diagram of a coal mine safety warning device according to an embodiment of the present application. Fig.11 As shown, the device comprises:

[0087] An acquisition unit 10 is used to acquire image data of underground coal mines, wherein the image data at least includes pedestrians and vehicles;

[0088] Specifically, high-resolution cameras deployed at key locations underground in coal mines are used to collect image data of pedestrians and vehicles.

[0089] A tracking unit 20, configured to analyze the image data by a target detection algorithm to obtain a pedestrian position and a vehicle position, and to track the pedestrian position by a target tracking algorithm to obtain a pedestrian trajectory, and to track the vehicle position by a target tracking algorithm to obtain a vehicle trajectory, wherein the target detection algorithm is obtained by training a machine learning model using multiple sets of data, and each set of data in the multiple sets of data includes the image data and an image including a pedestrian label and a vehicle label;

[0090] Specifically, the target detection algorithm is used to detect pedestrians and vehicles in the image data. The target tracking algorithm is used to achieve continuous and stable trajectory tracking of pedestrians and vehicles in underground tunnels, accurately record the motion state of each detection target, reduce tracking interruption problems caused by target occlusion, overlap and other factors, and obtain pedestrian and vehicle trajectories. The target detection algorithm adopts the improved yolov5 model. On the basis of the yolov5 model, the C2F lightweight module is integrated. Through branch processing, feature fusion and other operations, the representation ability of the network is improved, and Neck is replaced by Efficient-RepGFPN to improve the multi-scale detection ability of the model. At the same time, the loss function is optimized to further improve the detection accuracy of the model. Specific description of improved yolov5 structure: The improvements of yolov5 network include: for the structure in the Backbone network, the C3 module is replaced by the C2F module, while ensuring the lightweight of the model, richer gradient flow information is obtained, thereby improving the detection accuracy of the model; Neck is replaced by Efficient-RepGFPN, and multi-scale features are more efficiently fused through feature repetition and global feature pyramid pooling, capturing high-level semantic information and low-level spatial position information; the loss function is changed to FocalEIoU to further improve the detection accuracy of the model. The C2F module convolves and pools feature maps of different scales to obtain feature maps of multiple different scales; for each scale of feature maps, convolution and upsampling operations are used to adjust its scale to a uniform size; the adjusted feature maps are spliced ​​to obtain a fused feature map; some convolution operations and feature attention mechanisms are performed on the fused feature maps to further improve the representation ability and detection accuracy of the feature maps. The network structure diagram of the C2F module is shown below. Figure 3 Efficient-RepGFPN is a feature pyramid network based on feature repetition and global feature pyramid pooling. The network structure is shown in the figure Figure 4. Specifically, it includes the following modules: Feature repetition module: repeat the feature map multiple times and splice it with the original feature map to improve the resolution and richness of the feature map. Conventional feature pyramid module: use multiple convolutional layers and pooling layers to build a feature pyramid to improve the scale and richness of the feature map. Global feature pyramid pooling module: use different pooling scales to extract global features and splice them with the conventional feature pyramid to improve the richness and generalization ability of the feature map. Feature fusion module: use feature attention mechanism and feature cascade to fuse feature maps of various scales, thereby improving the representation ability and accuracy of the feature map. FocalEIoU is an improved IoU calculation method. FocalEIoU introduces a category weight and sample weight based on the traditional IoU calculation method, which can better handle the situation of category imbalance, thereby improving the detection accuracy. Specifically, the calculation method of FocalEIoU is as follows:

[0091]

[0092] Among them, n represents the number of intersections between the detection box and the real box; w i represents the weight of each category; f i Indicates the frequency of each category, which can be estimated by the number of samples; γ represents a hyperparameter used to balance the weights of positive and negative samples. The training set is sent to the improved yolov5 model, and the model parameters are optimized using the batch stochastic gradient descent method. Specific instructions for model training: Use the GPU server to train the improved yolov5 model, train 300 epochs based on the empirical value to obtain a better model weight, and verify the PR curve of the model, which represents the relationship between accuracy and recall, such as Figure 5 shown.

[0093] The determination unit 30 is used to determine whether the vehicle is in a moving state according to the vehicle trajectory. When the vehicle is in the moving state, the distance and relative position between the vehicle and the pedestrian are determined according to the pedestrian trajectory and the vehicle trajectory, and whether it is safe according to the distance and the relative position, wherein the relative position includes the pedestrian in front of the vehicle, the pedestrian to the side of the vehicle, and the pedestrian behind the vehicle.

[0094] Specifically, if the vehicle is in a moving state, the vehicle's driving direction and the relative position of the pedestrian and the vehicle are further determined, the distance between the pedestrian and the vehicle and the corresponding safety distance threshold are calculated, and the two are compared to determine whether the pedestrian and the vehicle are in an unsafe state. If so, an alarm is automatically generated.

[0095] Through this embodiment, image data of underground coal mine is obtained, and the image data includes at least pedestrians and vehicles; the image data is analyzed by target detection algorithm to obtain the position of pedestrians and vehicles, and the position of pedestrians is tracked by target tracking algorithm to obtain the pedestrian trajectory, and the position of vehicles is tracked by target tracking algorithm to obtain the vehicle trajectory, and the target detection algorithm is obtained by training multiple sets of data through machine learning model, and each set of data in multiple sets of data includes image data and images containing pedestrian labels and vehicle labels; according to the vehicle trajectory, it is determined whether the vehicle is in a moving state, and when the vehicle is in a moving state, the distance and relative position between the vehicle and the pedestrian are determined according to the pedestrian trajectory and the vehicle trajectory, and whether it is safe is determined according to the distance and relative position. Compared with the prior art, the pedestrian and vehicle recognition technology needs to consume a lot of manpower and material resources to arrange, in this way, image analysis is performed by target detection algorithm, and the position and distance of the vehicle and pedestrian are determined, and whether it is safe is further determined, which can save manpower and material resources, and accurately judge the safety of coal mines. Therefore, it can solve the problem of high cost of arrangement of the method for identifying people and vehicles in underground coal mines in the prior art, and achieve the effect of accurately identifying the safety of coal mines.

[0096] In the specific implementation process, the tracking unit includes a determination module and an update module. The determination module is used to obtain the first frame of image data and determine the initial state vector of the target in the first frame of image data, wherein the initial state vector at least includes the center point coordinates of the target, and the target includes the pedestrian and the vehicle; the update module is used to predict the position and speed of the target in the next frame of image data to obtain the predicted state vector, and update the motion trajectory of the target according to the predicted state vector to obtain the trajectory of the pedestrian and the trajectory of the vehicle. This method determines the trajectory of pedestrians and vehicles through each frame of image data, so that the trajectory can be accurately determined.

[0097] Specifically, the overall process of tracking the trajectory of pedestrians and vehicles is as follows: 1) Construct a track Tracks for each target in the first frame detection result, and initialize the Kalman parameters of the target, represented by the state vector x = [cx, cy, r, h, vx, vy, vr, vh], where (cx, cy) is the coordinate of the center point of the target, y is the aspect ratio of the target, h is the height of the target, and vx, vy, vr, vh represent the change speed of the four variables respectively. 2) For each identified pedestrian and vehicle target, use the pre-trained deep learning model to extract its appearance features, and maintain its category and feature information to reduce the probability of frequent switching of the target ID. 3) Use the Kalman filter to predict the position and speed of Tracks in the next frame. 4) Perform cascade matching (appearance information) and IOU matching (motion information) on the Tracks predicted in the previous frame and the detection box of the current frame, and build a cost matrix based on this. The cost matrix is ​​used as the input of the Hungarian algorithm to obtain the optimal matching result, thereby updating the tracking trajectory. 5) During the entire video frame processing process, Tracks is maintained and updated, and the tracking results of each target are finally output, including the location, ID and other relevant information of each target. Figure 6 The figure shows the results of tracking the trajectory of pedestrians and vehicles. The figure shows that "person 0.96 3" and "person 0.981" represent pedestrians, and "person 0.97 2" represents a vehicle.

[0098] In some optional embodiments, the determination unit includes a first acquisition module, a second acquisition module, a first calculation module and a first determination module, wherein the first acquisition module is used to acquire the center coordinates of the vehicle and the detection frame area of ​​the vehicle in the first frame image data to obtain the initial center coordinates and the initial area; the second acquisition module is used to acquire the center coordinates of the vehicle and the detection frame area of ​​the vehicle in the current frame image data to obtain the current center coordinates and the current area; the first calculation module is used to calculate the absolute value of the difference between the current center coordinates and the initial center coordinates to obtain the center coordinate difference, and calculate the absolute value of the difference between the initial area and the current area to obtain the area difference; the first determination module is used to determine that the vehicle is in the mobile state when the center coordinate difference is greater than a preset coordinate threshold and / or the area difference is greater than a preset area threshold. This method can accurately determine whether the vehicle is in the mobile state through the coordinate difference and the area difference.

[0099] Specifically, the motion trajectory information of pedestrians and vehicles is analyzed and the center point coordinates are recorded using the trajectory list V_Tracks={(V_Xi, V_Yi)|i=0,1,2,...,n}, where i represents the number of detections and (V_Xi, V_Yi) represents the center point coordinates of the target vehicle detected for the i-th time. The area list V_Area is used to record the area of ​​the vehicle detection box in each frame. When the center point coordinates of a vehicle are tracked for a total of 10 frames, the distance and area change between the center point of the vehicle in the current frame and the first frame are compared. If the relative movement distance of the vehicle (normalized to the image height) exceeds the preset threshold dist_thresh, or the change in the vehicle area exceeds another preset threshold area_thresh, the vehicle is recorded to have moved. If the vehicle movement is recorded for 5 consecutive times, it is determined that the vehicle is in operation. The target vehicle moving distance calculation formula is:

[0100]

[0101] Where (x1, y1) is the coordinate of the center point of the vehicle in the first frame, (x2, y2) is the coordinate of the center point of the vehicle in the current frame, and h is the image height. The calculation formula for the target vehicle area change is:

[0102]

[0103] Where (box[0], box[1]) represents the coordinates of the lower left corner of the vehicle detection box, (box[2], box[3]) represents the coordinates of the upper right corner of the vehicle detection box, area0 represents the area of ​​the vehicle detection box in the first frame, and h and w represent the image height and width.

[0104] In some optional embodiments, the determination unit includes a second calculation module and a third calculation module, the second calculation module is used to determine the driving direction of the vehicle according to the vehicle trajectory, calculate the angle between the pedestrian trajectory and the driving direction, so as to determine the relative position between the pedestrian and the vehicle; the third calculation module is used to obtain the coordinate position of the pedestrian and the coordinate position of the vehicle, calculate the absolute value of the difference between the coordinate position of the pedestrian and the coordinate position of the vehicle, and obtain the distance. This method can accurately determine the distance and relative position between the vehicle and the pedestrian through the driving direction and angle of the above-mentioned vehicle.

[0105] In the specific implementation process, 1) Determine the vehicle's driving direction: When the vehicle is in operation, determine whether the vehicle is moving horizontally or vertically by the angle of the vehicle's movement. The angle calculation formula between the vehicle's driving route and the x-axis is:

[0106]

[0107] Where (x1, y1) and (x2, y2) are the center coordinates of the vehicle in the first frame and the current frame respectively, arctan(x) is the inverse tangent function, and deg(x) is the radian conversion angle function.

[0108] 2) Determine the vehicle's orientation: Calculate the angle with the vehicle's moving direction. The angle calculation formula is:

[0109]

[0110] Where a, b, and c represent the three sides of the triangle formed by the center points of the vehicle in the first frame and the current frame, and the center point of the pedestrian in the current frame, respectively. Arccos(x) is the inverse cosine function. The formula for calculating the side length is:

[0111]

[0112] Threshold determination method: when the angle θ is less than 20°, the person is judged to be in front of the vehicle; when the angle is 20° less than θ less than 90°, the person is judged to be on the side of the vehicle; when the angle θ is greater than 90°, the person is judged to be behind the vehicle. The above thresholds are determined based on the on-site application results of Shendong underground mines.

[0113] 3) Pedestrian-vehicle distance calculation: When a person is in front of or beside the vehicle, there is a safety hazard, and the distance between the two must be calculated based on the coordinate positions of the vehicle and pedestrian in the current frame. When the distance is less than the preset safety threshold, it is determined that the pedestrian and vehicle are in an unsafe state.

[0114] In particular, because the underground tunnels are narrow, the required safety distances for pedestrians and vehicles in different directions of the tunnels are also different. For example, when a person is on the side of the car, the safety threshold of the distance between people and vehicles is relatively small, and when a person is diagonally in front of the car, the threshold of the distance between people and vehicles can be slightly larger. Therefore, based on the cases of unsafe behaviors in underground tunnels of coal mines, the present invention sets safety distance thresholds for different situations respectively, and repeatedly corrects them through experiments. At the same time, because the images taken by the camera have the effect of being larger near and smaller far away, the pixels occupied by the same person at the far end of the image are much smaller than the pixels occupied by the near end of the image, and the distance between people and vehicles will also change accordingly. In order to reduce this influence, the present invention uses the height of a person as the reference system, and obtains a mapping function by calculating the ratio of the pixel height of the person in the image to the actual height, and uses the mapping function to map the actual safety distance between people and vehicles to the pixel safety distance. The calculation formula for the pixel safety distance threshold between people and vehicles is as follows:

[0115]

[0116] Among them, h pixel is the pixel height of a person, h actualis the actual height of the person (the actual height is the average height of Shenhua miners), h0 is the image height, β is the safe distance between people and vehicles specified in the coal mine, and λ is used to adjust the contribution of the distance between people and the camera to the pixel safety distance threshold.

[0117] In some optional embodiments, the determination unit further includes a second determination module and a third determination module, the second determination module is used to determine that it is unsafe when the relative position is that the pedestrian is in front of the vehicle and the distance is less than a distance threshold; the third determination module is used to determine that it is unsafe when the relative position is that the pedestrian is to the side of the vehicle and the distance is less than a distance threshold. The method can quickly and accurately determine whether pedestrians and vehicles in a coal mine are safe through the above steps.

[0118] Figure 7 This is the result diagram of pedestrian and vehicle unsafe state recognition. Figure 7 (a) is the previous frame image, Figure 7 (b) is the next frame. When the pedestrian and vehicle are judged to be in an unsafe state, the upper left corner of the vehicle detection frame shows "Pedestrians are not vehicles". The platform immediately pushes the warning information to the relevant personnel and saves the unsafe behavior video for subsequent analysis.

[0119] In some optional embodiments, after acquiring the image data of the coal mine, the method further includes an image enhancement unit and an image cleaning unit, wherein the image enhancement unit is used to perform image enhancement on the image data to obtain the image data after image enhancement; and the image cleaning unit is used to perform image cleaning on the image data after image enhancement. In this way, the image can be pre-processed into the format of the algorithm model for subsequent training of the algorithm model.

[0120] Specifically, due to the complex environment of coal mines, there are often problems such as low illumination, dust and fog, lack of color information, and uneven illumination. These unfavorable factors will seriously affect the image quality, increase the learning difficulty of the target detection model, and make it difficult to accurately capture the typical features of the image. In order to solve the above problems, a differentiable filter with adjustable hyperparameters is used to preprocess the image from the aspects of dehazing, white balance (WB), gamma, contrast enhancement, tone adjustment, and sharpening. The preprocessing module structure is as follows: Figure 8As shown. In order to avoid the neural network consuming a lot of computing resources, the high-resolution image obtained by the camera needs to be downsampled to obtain a low-resolution image of 256×256. The low-resolution image is then sent to a small convolutional neural network (CNN-S), which passes through five convolutional modules and two fully connected layers in sequence to predict the optimal hyperparameters of the filter, and the filter is applied to the original resolution image to improve the image quality. Among them, the convolution module in the CNN-S network consists of five convolutional modules and two fully connected layers. Each convolutional module includes a 3×3 convolutional layer with a stride of 2 and a leakyRelu layer. The last fully connected layer outputs the predicted value of the hyperparameter. After the image is sent to the image preprocessing module for image enhancement operations such as brightness enhancement, hue adjustment, and defogging, the image is cleaned to remove images with blurred and unclear targets. The LabelImg tool is then used to annotate the image. In order to improve the accuracy of the model, some locations similar to vehicles and pedestrians are annotated as negative samples. Finally, an xml file is generated, and the training set and test set are divided proportionally.

[0121] In some optional embodiments, the device further includes a labeling unit and a training unit, the labeling unit is used to label pedestrians and vehicles in the image data to obtain images including the pedestrian labels and the vehicle labels; the training unit is used to construct an initial target detection model, and the image data and the images including the pedestrian labels and the vehicle labels are used to train the initial target detection model to obtain the target detection model. The method trains the target detection model through the above steps, so that the model can be trained to be a model suitable for the detection of pedestrians and vehicles, so as to facilitate subsequent target detection.

[0122] In the specific implementation process, the training set is sent to the improved yolov5 model, and the batch stochastic gradient descent method is used to optimize the model parameters. Specific instructions for model training: Use the GPU server to train the improved yolov5 model, train 300 epochs based on the empirical value to obtain a better model weight, and verify the PR curve of the model, such as Figure 5 As shown. The trained model is used to detect pedestrians and vehicles in the video frame. Specific description of target detection: After the training model is completed, a weight file is obtained. This weight file is used to test the model and mAp calculation to obtain a detection rate of 92.52% for pedestrians and vehicles. Fig. 9 The following is the test result of the pedestrian and vehicle target detection model. Fig. 9 (a) is the image before detection. Fig. 9 (b) is the image after detection using the improved yolov5 model.

[0123] The coal mine safety early warning device includes a processor and a memory, and the acquisition unit, tracking unit, and determination unit are all stored in the memory as program units, and the processor executes the program units stored in the memory to realize the corresponding functions. The modules are all located in the same processor; or, the modules are located in different processors in any combination.

[0124] The processor includes a kernel, and the kernel retrieves the corresponding program unit from the memory. One or more kernels can be set, and the problem of high layout cost of the underground human and vehicle identification method in the coal mine in the prior art can be solved by adjusting the kernel parameters.

[0125] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0126] An embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium includes a stored program, wherein when the program is executed, the device where the computer-readable storage medium is located is controlled to execute the coal mine safety early warning method.

[0127] Specifically, coal mine safety early warning methods include:

[0128] Step S201, acquiring image data of a coal mine, wherein the image data at least includes pedestrians and vehicles;

[0129] Step S202, analyzing the image data by a target detection algorithm to obtain a pedestrian position and a vehicle position, and tracking the pedestrian position by a target tracking algorithm to obtain a pedestrian trajectory, and tracking the vehicle position by a target tracking algorithm to obtain a vehicle trajectory, wherein the target detection algorithm is obtained by training a machine learning model using multiple sets of data, and each set of data in the multiple sets of data includes the image data and an image including a pedestrian label and a vehicle label;

[0130] Step S203, determining whether the vehicle is in a moving state based on the vehicle trajectory. When the vehicle is in the moving state, determining the distance and relative position between the vehicle and the pedestrian based on the pedestrian trajectory and the vehicle trajectory, and determining whether it is safe based on the distance and the relative position, wherein the relative position includes the pedestrian in front of the vehicle, the pedestrian to the side of the vehicle, and the pedestrian behind the vehicle.

[0131] An embodiment of the present invention provides an electronic device, including a processor, a memory, and a program stored in the memory and executable on the processor, wherein when the processor executes the program, at least the following steps are implemented:

[0132] Step S201, acquiring image data of a coal mine, wherein the image data at least includes pedestrians and vehicles;

[0133] Step S202, analyzing the image data by a target detection algorithm to obtain a pedestrian position and a vehicle position, and tracking the pedestrian position by a target tracking algorithm to obtain a pedestrian trajectory, and tracking the vehicle position by a target tracking algorithm to obtain a vehicle trajectory, wherein the target detection algorithm is obtained by training a machine learning model using multiple sets of data, and each set of data in the multiple sets of data includes the image data and an image including a pedestrian label and a vehicle label;

[0134] Step S203, determining whether the vehicle is in a moving state based on the vehicle trajectory. When the vehicle is in the moving state, determining the distance and relative position between the vehicle and the pedestrian based on the pedestrian trajectory and the vehicle trajectory, and determining whether it is safe based on the distance and the relative position, wherein the relative position includes the pedestrian in front of the vehicle, the pedestrian to the side of the vehicle, and the pedestrian behind the vehicle.

[0135] The present application also provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of the method described in each embodiment of the present application:

[0136] Step S201, acquiring image data of a coal mine, wherein the image data at least includes pedestrians and vehicles;

[0137] Step S202, analyzing the image data by a target detection algorithm to obtain a pedestrian position and a vehicle position, and tracking the pedestrian position by a target tracking algorithm to obtain a pedestrian trajectory, and tracking the vehicle position by a target tracking algorithm to obtain a vehicle trajectory, wherein the target detection algorithm is obtained by training a machine learning model using multiple sets of data, and each set of data in the multiple sets of data includes the image data and an image including a pedestrian label and a vehicle label;

[0138] Step S203, determining whether the vehicle is in a moving state based on the vehicle trajectory. When the vehicle is in the moving state, determining the distance and relative position between the vehicle and the pedestrian based on the pedestrian trajectory and the vehicle trajectory, and determining whether it is safe based on the distance and the relative position, wherein the relative position includes the pedestrian in front of the vehicle, the pedestrian to the side of the vehicle, and the pedestrian behind the vehicle.

[0139] Obviously, those skilled in the art should understand that the above modules or steps of the present invention can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order than here, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present invention is not limited to any specific combination of hardware and software.

[0140] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0141] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0142] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0143] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0144] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0145] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0146] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0147] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0148] From the above description, it can be seen that the above embodiments of the present application achieve the following technical effects:

[0149] 1) The coal mine safety early warning method of the present application obtains image data of underground coal mines, and the image data includes at least pedestrians and vehicles; the image data is analyzed by a target detection algorithm to obtain the position of pedestrians and vehicles, and the position of pedestrians is tracked by a target tracking algorithm to obtain the pedestrian trajectory, and the position of vehicles is tracked by a target tracking algorithm to obtain the vehicle trajectory, and the target detection algorithm is obtained by training a machine learning model using multiple sets of data, and each set of data in the multiple sets of data includes image data and images containing pedestrian labels and vehicle labels; according to the vehicle trajectory, it is determined whether the vehicle is in a moving state, and when the vehicle is in a moving state, the distance and relative position between the vehicle and the pedestrian are determined according to the pedestrian trajectory and the vehicle trajectory, and whether it is safe is determined according to the distance and relative position. Compared with the prior art, the pedestrian and vehicle recognition technology needs to consume a lot of manpower and material resources to arrange, so that the image analysis is performed by the target detection algorithm, and the position and distance of the vehicle and the pedestrian are determined, and whether it is safe is further determined, which can save manpower and material resources, and accurately judge the safety of coal mines. Therefore, it can solve the problem of high cost of arrangement of the human-vehicle recognition method in underground coal mines in the prior art, and achieve the effect of accurately identifying the safety of coal mines.

[0150] The above description is only the preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A coal mine safety early warning method, characterized in that: include: Acquire image data of a coal mine, wherein the image data at least includes pedestrians and vehicles; The image data is analyzed by a target detection algorithm to obtain a pedestrian position and a vehicle position, and the pedestrian position is tracked by a target tracking algorithm to obtain a pedestrian trajectory, and the vehicle position is tracked by a target tracking algorithm to obtain a vehicle trajectory, wherein the target detection algorithm is obtained by training a machine learning model using multiple sets of data, and each set of data in the multiple sets of data includes the image data and an image including a pedestrian label and a vehicle label; Determine whether the vehicle is in a moving state based on the vehicle trajectory. When the vehicle is in the moving state, determine the distance and relative position between the vehicle and the pedestrian based on the pedestrian trajectory and the vehicle trajectory, and determine whether it is safe based on the distance and the relative position, wherein the relative position includes the pedestrian in front of the vehicle, the pedestrian to the side of the vehicle, and the pedestrian behind the vehicle.

2. The coal mine safety early warning method according to claim 1, characterized in that: Tracking the position of the pedestrian by a target tracking algorithm to obtain a pedestrian trajectory, and tracking the position of the vehicle by a target tracking algorithm to obtain a vehicle trajectory, including: Acquire a first frame of image data, and determine an initial state vector of a target in the first frame of image data, wherein the initial state vector at least includes the center point coordinates of the target, and the target includes the pedestrian and the vehicle; The position and speed of the target in the next frame of image data are predicted to obtain a predicted state vector, and the motion trajectory of the target is updated according to the predicted state vector to obtain the pedestrian trajectory and the vehicle trajectory.

3. The coal mine safety early warning method according to claim 1, characterized in that: Determining whether the vehicle is in a moving state according to the vehicle trajectory includes: Acquire the center coordinates of the vehicle and the detection frame area of ​​the vehicle in the first frame of image data to obtain initial center coordinates and initial area; Obtaining the center coordinates of the vehicle and the detection frame area of ​​the vehicle in the current frame image data to obtain the current center coordinates and the current area; Calculating the absolute value of the difference between the current center coordinates and the initial center coordinates to obtain a center coordinate difference, and calculating the absolute value of the difference between the initial area and the current area to obtain an area difference; When the center coordinate difference is greater than a preset coordinate threshold and / or the area difference is greater than a preset area threshold, it is determined that the vehicle is in the moving state.

4. The coal mine safety early warning method according to claim 1, characterized in that: Determining the distance and relative position between the vehicle and the pedestrian according to the pedestrian trajectory and the vehicle trajectory includes: Determine the driving direction of the vehicle according to the vehicle trajectory, and calculate the angle between the pedestrian trajectory and the driving direction to determine the relative position of the pedestrian and the vehicle; The coordinate position of the pedestrian and the coordinate position of the vehicle are obtained, and the absolute value of the difference between the coordinate position of the pedestrian and the coordinate position of the vehicle is calculated to obtain the distance.

5. The coal mine safety early warning method according to claim 1, characterized in that: Determining whether it is safe according to the distance and the relative position includes one of the following: When the relative position is that the pedestrian is in front of the vehicle and the distance is less than a distance threshold, determining that it is unsafe; When the relative position is that the pedestrian is at the side of the vehicle and the distance is less than a distance threshold, it is determined to be unsafe.

6. The coal mine safety early warning method according to claim 1, characterized in that: After acquiring the image data of the coal mine, the method further includes: Performing image enhancement on the image data to obtain image data after image enhancement; Image cleaning is performed on the image data after image enhancement.

7. The coal mine safety early warning method according to claim 1, characterized in that: Before analyzing the image data by the target detection algorithm, the method further includes: Labeling pedestrians and vehicles in the image data to obtain an image including the pedestrian label and the vehicle label; An initial target detection model is constructed, and the image data and images containing pedestrian labels and vehicle labels are used to train the initial target detection model to obtain the target detection model.

8. A coal mine safety early warning device, characterized in that: include: An acquisition unit, used to acquire image data of underground coal mine, wherein the image data at least includes pedestrians and vehicles; A tracking unit, configured to analyze the image data by a target detection algorithm to obtain a pedestrian position and a vehicle position, and to track the pedestrian position by a target tracking algorithm to obtain a pedestrian trajectory, and to track the vehicle position by a target tracking algorithm to obtain a vehicle trajectory, wherein the target detection algorithm is obtained by training a machine learning model using multiple sets of data, and each set of data in the multiple sets of data includes the image data and an image including a pedestrian label and a vehicle label; A determination unit is used to determine whether the vehicle is in a moving state according to the vehicle trajectory, and when the vehicle is in the moving state, determine the distance and relative position between the vehicle and the pedestrian according to the pedestrian trajectory and the vehicle trajectory, and determine whether it is safe according to the distance and the relative position, wherein the relative position includes the pedestrian in front of the vehicle, the pedestrian to the side of the vehicle, and the pedestrian behind the vehicle.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored program, wherein when the program is executed, the device where the computer-readable storage medium is located is controlled to execute the coal mine safety early warning method according to any one of claims 1 to 7.

10. An electronic device, characterized in that: include: One or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and the one or more programs include a method for executing the coal mine safety early warning method described in any one of claims 1 to 7.

Images