Underground coal mine personnel intrusion identification method and device

Abstract

The invention provides an underground coal mine personnel intrusion identification method and device, and the method comprises the steps: obtaining a current frame image, and determining a depth image corresponding to the current frame image through employing a monocular depth estimation method; detecting pedestrian targets in the current frame image to obtain a detection frame corresponding to at least one pedestrian target in the current frame image, and extracting skeleton key points based on the current frame image of the area where each detection frame is located to obtain two-dimensional coordinates of the skeleton key points corresponding to the pedestrian targets in each detection frame; determining three-dimensional coordinates of skeleton key points of each pedestrian target according to the two-dimensional coordinates of the skeleton key points of the corresponding pedestrian target in each detection frame, the depth image and an internal reference matrix of the image acquisition equipment; and according to the three-dimensional coordinates of the skeleton key points of each detection frame and / or each pedestrian target, whether each pedestrian target invades the target space electronic fence area is determined, so that the accuracy of underground coal mine pedestrian detection can be improved, and the false alarm rate is reduced.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a method and device for identifying personnel intrusion in underground coal mines. Background Art

[0002] In dangerous environments such as underground coal mines, ensuring the safety of workers is of vital importance. The underground environment is relatively complex and there are many unsafe factors. Preventing underground workers from entering areas with unsafe factors is an important measure to ensure the safety of underground workers.

[0003] Coal mines are usually equipped with visible light cameras to monitor unsafe factors in the underground environment. However, manual monitoring of camera images is usually inefficient, with low detection rates, and it is difficult to concentrate on monitoring camera images for a long time. Nowadays, artificial intelligence technology has been widely used in many fields, and pedestrian detection based on visible light images has also been applied in many fields. However, most of the existing pedestrian detection methods are aimed at general scenes on the ground, and the accuracy is not high when applied in the dim and blurred environment of coal mines, resulting in a large number of false alarms in the electronic fence intrusion detection results. Summary of the invention

[0004] The present invention provides a method and device for identifying personnel intrusion in a coal mine, so as to solve one of the technical problems in the related art at least to a certain extent. The technical solution of the present invention is as follows:

[0005] According to a first aspect of an embodiment of the present invention, a method for identifying intrusion of personnel in an underground coal mine is provided, comprising: obtaining a current frame image acquired by an image acquisition device arranged in an underground coal mine, and determining a depth image corresponding to the current frame image by using a monocular depth estimation method; detecting pedestrian targets in the current frame image to obtain a detection frame corresponding to at least one pedestrian target in the current frame image, and extracting skeleton key points based on the current frame image in the area where each detection frame is located to obtain two-dimensional coordinates of the skeleton key points of the corresponding pedestrian targets in each detection frame; wherein the detection frame is a matrix frame that minimally surrounds the corresponding pedestrian targets; determining the three-dimensional coordinates of the skeleton key points of each pedestrian target according to the two-dimensional coordinates of the skeleton key points of the corresponding pedestrian targets in each detection frame, the depth image and the intrinsic reference matrix of the image acquisition device; determining whether each pedestrian target has invaded a target space electronic fence area according to the three-dimensional coordinates of the skeleton key points of each detection frame and / or each pedestrian target; wherein the target space electronic fence is determined based on the depth image and a preset area to be determined.

[0006] According to a second aspect of an embodiment of the present invention, a coal mine underground personnel intrusion identification device is provided, comprising: a first processing module, used to obtain a current frame image collected by an image acquisition device arranged in the coal mine, and determine a depth image corresponding to the current frame image by using a monocular depth estimation method; a second processing module, used to detect pedestrian targets in the current frame image to obtain a detection frame corresponding to at least one pedestrian target in the current frame image, and extract skeleton key points based on the current frame image in the area where each detection frame is located to obtain two-dimensional coordinates of the skeleton key points of the corresponding pedestrian targets in each detection frame; wherein the detection frame is a matrix frame that minimally surrounds the corresponding pedestrian targets; a first determination module, used to determine the three-dimensional coordinates of the skeleton key points of each pedestrian target according to the two-dimensional coordinates of the skeleton key points of the corresponding pedestrian targets in each detection frame, the depth image and the intrinsic reference matrix of the image acquisition device; a second determination module, used to determine whether each pedestrian target invades the target space electronic fence area according to the three-dimensional coordinates of the skeleton key points of each detection frame and / or each pedestrian target; wherein the target space electronic fence is determined based on the depth image and a preset area to be determined.

[0007] According to a third aspect of an embodiment of the present invention, there is provided an electronic device, comprising: a processor; and a memory for storing instructions executable by the processor; wherein the processor is configured to execute the instructions to implement a method for identifying intrusion of personnel underground in a coal mine as described in the embodiment of the first aspect of the present invention.

[0008] According to the fourth aspect of an embodiment of the present invention, a computer-readable storage medium is provided. When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the method for identifying personnel intrusion underground in a coal mine as described in the embodiment of the first aspect of the present invention.

[0009] According to a fifth aspect of an embodiment of the present invention, a computer program product is provided, comprising: a computer program, which, when executed by a processor, implements the method for identifying personnel intrusion underground in a coal mine as described in the embodiment of the first aspect of the present invention.

[0010] The technical solution provided by the embodiments of the present invention brings at least the following beneficial effects:

[0011] In the technical scheme, a current frame image acquired by an image acquisition device arranged in an underground coal mine is obtained, and a monocular depth estimation method is used to determine a depth image corresponding to the current frame image; pedestrian targets in the current frame image are detected to obtain a detection frame corresponding to at least one pedestrian target in the current frame image, and skeleton key points are extracted based on the current frame image of the area where each detection frame is located to obtain the two-dimensional coordinates of the skeleton key points of the corresponding pedestrian targets in each detection frame; wherein the detection frame is a matrix frame that minimally surrounds the corresponding pedestrian targets; the skeleton key points of each pedestrian target are determined according to the two-dimensional coordinates of the skeleton key points of the corresponding pedestrian targets in each detection frame, the depth image and the internal reference matrix of the image acquisition device. three-dimensional coordinates of key points; determining whether each pedestrian target invades the target space electronic fence area according to the three-dimensional coordinates of the skeleton key points of each detection frame and / or each pedestrian target; wherein the target space electronic fence is determined based on the depth image and the preset area to be determined. Therefore, in view of the problem that a large number of false alarms occur in the existing pedestrian detection method when it is applied in the dim and blurred environment of the coal mine, resulting in a large number of false alarms in the electronic fence intrusion detection, the present invention combines monocular depth estimation with the human skeleton to perform three-dimensional position tracking on the pedestrian target in the current frame image collected by the image acquisition device arranged in the coal mine, thereby improving the accuracy of pedestrian detection in the coal mine and reducing the false alarm rate of electronic fence intrusion detection.

[0012] It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The accompanying drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present invention, and together with the description are used to explain the principles of the present invention, and do not constitute improper limitations of the present invention.

[0014] Figure 1 It is a schematic flow chart of a method for identifying personnel intrusion in underground coal mines according to a first embodiment of the present invention;

[0015] Figure 2 is a flow chart of a method for identifying personnel intrusion in underground coal mines according to a second embodiment of the present invention;

[0016] Figure 3 is a schematic flow chart of a method for identifying personnel intrusion in underground coal mines according to a third embodiment of the present invention;

[0017] Figure 4 4 is a flow chart of a method for identifying personnel intrusion in underground coal mines according to a fourth embodiment of the present invention;

[0018] Figure 5 1 is a flow chart of a method for identifying personnel intrusion in underground coal mines according to a fifth embodiment of the present invention;

[0019] Figure 6 A schematic diagram of the principle of identifying personnel intrusion in a coal mine provided by the sixth embodiment of the present invention;

[0020] Figure 7 1 is a schematic diagram of the structure of a coal mine underground personnel intrusion identification device shown in the seventh embodiment of the present invention;

[0021] Figure 8 It is a schematic structural diagram of an electronic device 800 according to an exemplary embodiment of the present invention. DETAILED DESCRIPTION

[0022] In order to enable ordinary persons in the art to better understand the technical solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings.

[0023] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Instead, they are merely examples of devices and methods consistent with some aspects of the present invention as detailed in the appended claims.

[0024] It should be noted that in the technical solution of the present invention, the collection, storage, use, processing, transmission, provision and disclosure of user personal information are all carried out with the user's consent, and comply with the relevant laws and regulations and do not violate public order and good morals.

[0025] Unlike open spaces, underground coal mines are closed, narrow, dark, and have many obstructions, which poses a great challenge to the detection of underground coal mine workers. Most of the existing pedestrian detection methods are aimed at general scenes on the ground. When applied in the dim and blurred environment of coal mines, the accuracy is not high and often produces many false alarms, which brings considerable trouble to subsequent pedestrian tracking and other further processing, and also causes a high false alarm rate for electronic fence intrusion detection.

[0026] In view of the above problems, the present invention proposes a method for identifying personnel intrusion in underground coal mines. By combining monocular depth estimation with the human skeleton, more robust target tracking methods, motion detection, multi-point detection and other means are realized. While ensuring the accuracy of the alarm, the position of personnel in the prohibited area can be monitored and tracked in real time, and the alarm can be issued in time and necessary measures can be taken, thereby minimizing the possibility of accidents.

[0027] The following describes a method and device for identifying intrusion of personnel underground in a coal mine according to an embodiment of the present invention with reference to the accompanying drawings.

[0028] Figure 1 It is a flow chart of a method for identifying personnel intrusion in underground coal mines according to the first embodiment of the present invention.

[0029] The embodiment of the present invention takes the coal mine underground personnel intrusion identification method as an example in which it is configured in a coal mine underground personnel intrusion identification device. The coal mine underground personnel intrusion identification device can be applied to any electronic device so that the electronic device can perform the coal mine underground personnel intrusion identification function.

[0030] Among them, the electronic device can be any device with computing capabilities, such as a personal computer, a mobile terminal, a server (or cloud), etc. The mobile terminal can be, for example, a vehicle-mounted device, a mobile phone, a tablet computer, a personal digital assistant, a wearable device, and other hardware devices with various operating systems, touch screens and / or display screens.

[0031] like Figure 1 As shown, the coal mine underground personnel intrusion identification method includes the following steps:

[0032] Step 101, obtaining a current frame image acquired by an image acquisition device arranged underground in a coal mine, and using a monocular depth estimation method to determine a depth image corresponding to the current frame image.

[0033] The image acquisition device may be any device for capturing image data, such as a camera, a camcorder, a still camera, or other devices with a camera function (such as a mobile phone, a tablet computer, etc.).

[0034] In the embodiment of the present invention, the coal mine underground personnel intrusion identification device can obtain the current frame image collected by the image acquisition device arranged in the coal mine in various public, legal and compliant ways. For example, during the process of the image acquisition device collecting images, the image collected by the image acquisition device can be obtained in real time through network transmission.

[0035] Among them, depth images, also known as range images, are a special way of representing images. The core of these images is to record the distance (depth) value from the image acquisition device to each point in the scene as a pixel value. This type of image directly reflects the geometric shape of the visible surface of the scene and is a projection of three-dimensional spatial information on a two-dimensional plane. It is of great significance for understanding the spatial structure of objects and performing three-dimensional reconstruction.

[0036] In the present invention, in order to better track the position of pedestrians, a monocular depth estimation method is used to estimate the depth of the current frame image. As a possible implementation method, the current frame image can be input into a monocular depth estimation network to obtain a depth image corresponding to the current frame image, from which the depth information d of each pixel point p can be obtained. p .

[0037] Step 102, detect the pedestrian targets in the current frame image to obtain a detection frame corresponding to at least one pedestrian target in the current frame image, and extract skeleton key points based on the current frame image in the area where each detection frame is located to obtain the two-dimensional coordinates of the skeleton key points corresponding to the pedestrian targets in each detection frame.

[0038] In an embodiment of the present invention, the coal mine underground personnel intrusion identification device can detect pedestrian targets in the current frame image, thereby obtaining a detection frame corresponding to at least one pedestrian target in the current frame image, wherein the detection frame is a matrix frame that minimally surrounds the corresponding pedestrian target. Optionally, the detection frame can be output in the form of a quaternion, and the four numbers included represent the positions of the left, top, right, and bottom of the matrix frame that minimally surrounds the corresponding pedestrian target, respectively represented by l, t, r, and b, then the upper left corner point of the rectangular frame can be represented as p (lt) =[l,t] T , the lower right corner point can be represented as p (rb) =[r,b] T , the center point can be expressed as p (center) =[(l+r) / 2,(t+b) / 2] T .

[0039] As a possible implementation method, a deep neural network can be trained using a coal mine pedestrian dataset to obtain a pedestrian detection network optimized using the coal mine pedestrian dataset, so as to detect pedestrian targets in the current frame image using the pedestrian detection network. Specifically, the current frame image can be input into the pedestrian detection network to obtain a detection box corresponding to at least one pedestrian target in the current frame image output by the pedestrian detection network.

[0040] In the present invention, in order to better track the position of pedestrians, after obtaining the detection frame corresponding to each pedestrian target, the skeleton key points are extracted based on the current frame image of the area where each detection frame is located, so as to obtain the two-dimensional coordinates of the skeleton key points corresponding to the pedestrian target in each detection frame. Among them, the skeleton key points can include 17 human skeleton key points such as nose, left eye, right eye, left ear, right ear, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left hip, right hip, left knee, right knee, left ankle, and right ankle.

[0041] As a possible implementation, for any detection box, the current frame image of the area where the detection box is located can be cropped and input into the skeleton key point extraction network to obtain the two-dimensional coordinates of the skeleton key points of the corresponding pedestrian target in the detection box.

[0042] Step 103: According to the two-dimensional coordinates of the skeleton key points of the corresponding pedestrian targets in each detection box, the depth image, and the internal parameter matrix of the image acquisition device, determine the three-dimensional coordinates of the skeleton key points of each pedestrian target.

[0043] As a possible implementation, the depth information d of each pixel point p can be obtained from the depth image corresponding to the current frame image p , and then using the internal parameter matrix K of the image acquisition device, the following calculation formula is used to determine the three-dimensional spatial coordinates of any pixel point p in the current frame image:

[0044] x = d p K -1 p

[0045] Among them, the coordinates of the pixel point p are given in the form of homogeneous coordinates.

[0046] In summary, since the two-dimensional coordinates of the skeleton key points of the corresponding pedestrian targets in each detection box are pixel points p one by one, the above calculation formula can be used to determine the three-dimensional coordinates of the skeleton key points of each pedestrian target.

[0047] Step 104: According to the three-dimensional coordinates of each detection box and / or the skeleton key points of each pedestrian target, determine whether each pedestrian target invades the target space electronic fence area.

[0048] Among them, the target space electronic fence is determined based on the depth image and the preset area to be determined.

[0049] Among them, the area to be determined can be any region of interest (ROI) preset by the user in the current frame image.

[0050] Among them, the ROI is a closed figure composed of an ordered two-dimensional point sequence, and the area on the right hand side along the point sequence is the internal area of the closed figure. Let the ordered two-dimensional point sequence be p (ROI),i , i = 1, 2, …, L (ROI) The length of is L (ROI) , for any pixel point p in the current frame image, if the pixel point p is adjacent to the points in any ordered two-dimensional point sequence, that is, i < L (ROI) , j = i + 1 and i = L (ROI) , j = 1, there is: the pixel point p is at the pixel point p (ROI),i and the pixel point p (ROI),jIf the right hand side of the directed line segment is formed, it can be determined that the pixel point p is inside the ROI.

[0051] It should be noted that the ROI preset by the user in the current frame image is a plane area in the current frame image. In order to achieve the tracking effect in space, the present invention will use monocular depth estimation to extract the area in space corresponding to the ROI (target space electronic area) according to the preset ROI, so as to perform intrusion detection in the target space electronic area.

[0052] As a possible implementation method, the process of determining the target space electronic fence can be: determining the three-dimensional coordinates of each point in the two-dimensional point sequence according to the depth image and the two-dimensional coordinates of each point in the two-dimensional point sequence constituting the area to be determined; using Hough transform to extract multiple planes in the area to be determined, and taking the number of inner points of each plane as the area occupied by the corresponding plane in the area to be determined; selecting the plane with the largest area from each plane as the ground plane, and obtaining the normal vector of the ground plane and the three-dimensional coordinates of the target intersection; wherein the target intersection is the intersection of the straight line from the coordinate origin along the normal vector direction of the ground plane and the ground plane; based on the normal vector of the ground plane and the three-dimensional coordinates of the target intersection, project the three-dimensional coordinates of each point in the two-dimensional point sequence onto the ground plane to obtain the target space electronic fence.

[0053] For example, first, according to the depth image corresponding to the current frame image, the depth information d of each pixel p in the POI is extracted. p , determine the 3D coordinates of each pixel point p in the POI; then use the Hough transform to extract all planes and their normal vectors, and calculate the number of inner points of each plane as the area occupied by the plane in the ROI; then select the plane with the largest area as the ground plane P, and obtain its normal vector n P , and the intersection o of the straight line along the normal vector direction of the ground plane and the ground plane; finally, the three-dimensional coordinates x of the ordered two-dimensional point sequence constituting the ROI are calculated using the following formula s,(ROI),i Projection onto the ground plane:

[0054] x g,(ROI),i =x s,(ROI),i -(x g,(ROI),i -o)n P

[0055] The ROI on the ground plane is obtained and used as the basis for pedestrian electronic fence intrusion detection, that is, the target space electronic fence.

[0056] In an embodiment of the present invention, after obtaining each detection frame and the three-dimensional coordinates of the skeletal key points of each pedestrian target, it is possible to determine whether each pedestrian target has invaded the target space electronic fence area based on each detection frame and / or the three-dimensional coordinates of the skeletal key points of each pedestrian target.

[0057] As a possible implementation method, the skeleton key points include at least left and right ankle skeleton key points. For any pedestrian target, when the confidences of the left and right ankle skeleton key points of the pedestrian target are both greater than the confidence threshold, the three-dimensional coordinates of the ankle skeleton key points with higher confidence among the left and right ankle skeleton key points of the pedestrian target are used as the foot position of the pedestrian target; when the confidences of the left and right ankle skeleton key points of the pedestrian target are not both greater than the confidence threshold, the three-dimensional coordinates of the midpoint of the lower side of the detection frame corresponding to the pedestrian target are used as the foot position of the pedestrian target; wherein the three-dimensional coordinates of the midpoint of the lower side of the detection frame corresponding to the pedestrian target are determined based on the two-dimensional coordinates of the midpoint of the lower side of the detection frame corresponding to the pedestrian target, the depth image and the intrinsic parameter matrix of the image acquisition device; the foot position of the pedestrian target is projected onto the ground plane to determine whether the foot position of the pedestrian target is within the target space electronic fence area, and when the foot position of the pedestrian target is within the target space electronic fence area, it is determined that the pedestrian target has invaded the target space electronic fence area; wherein the ground plane is the ground plane determined in the process of determining the target space electronic fence area.

[0058] That is to say, the foot position of the pedestrian target is used as a reference in the present invention to determine whether the pedestrian target has invaded the target space electronic fence area. In the case where the confidence of the left and right ankle bone key points is higher than the confidence threshold, the present invention will use the ankle bone key points with higher confidence as the foot position of the pedestrian target, otherwise it is assumed that the midpoint of the bottom of the detection frame corresponding to the pedestrian target is the foot position. Afterwards, the three-dimensional coordinates of the determined foot position are projected onto the ground plane to determine whether the pedestrian target is in the target space electronic fence area. If it is in the target space electronic fence area, it is considered that there is an intrusion condition.

[0059] The coal mine underground personnel intrusion identification method of the embodiment of the present invention obtains the current frame image collected by the image acquisition device arranged in the coal mine, and uses the monocular depth estimation method to determine the depth image corresponding to the current frame image; detects the pedestrian target in the current frame image to obtain a detection frame corresponding to at least one pedestrian target in the current frame image, and extracts the skeleton key points based on the current frame image of the area where each detection frame is located to obtain the two-dimensional coordinates of the skeleton key points of the corresponding pedestrian target in each detection frame; wherein the detection frame is a matrix frame that minimally surrounds the corresponding pedestrian target; and determines each row according to the two-dimensional coordinates of the skeleton key points of the corresponding pedestrian target in each detection frame, the depth image and the intrinsic parameter matrix of the image acquisition device. The three-dimensional coordinates of the skeleton key points of the human target; according to the three-dimensional coordinates of the skeleton key points of each detection frame and / or each pedestrian target, determine whether each pedestrian target invades the target space electronic fence area; wherein the target space electronic fence is determined based on the depth image and the preset area to be determined. Therefore, in view of the problem that the existing pedestrian detection method has a large number of false alarms when applied in the dim and blurred environment of the coal mine, resulting in a large number of false alarms in the electronic fence intrusion detection, the present invention combines monocular depth estimation with the human skeleton to perform three-dimensional position tracking on the pedestrian target in the current frame image collected by the image acquisition device arranged in the coal mine, which can improve the accuracy of pedestrian detection in the coal mine and reduce the false alarm rate of electronic fence intrusion detection.

[0060] It should be noted that in the present invention, before determining whether each pedestrian target invades the target space electronic fence area according to the three-dimensional coordinates of the skeleton key points of each detection frame and / or each pedestrian target, pedestrian tracking can also be performed to remove non-pedestrian targets from each pedestrian target. Figure 2 , which explains this process.

[0061] Figure 2 It is a flow chart of a method for identifying personnel intrusion in underground coal mines according to a second embodiment of the present invention.

[0062] like Figure 2 As shown, the coal mine underground personnel intrusion identification method includes the following steps:

[0063] Step 201, obtain a current frame image captured by an image capture device arranged underground in a coal mine, and use a monocular depth estimation method to determine a depth image corresponding to the current frame image.

[0064] Step 202, detect the pedestrian targets in the current frame image to obtain a detection frame corresponding to at least one pedestrian target in the current frame image, and extract skeleton key points based on the current frame image in the area where each detection frame is located to obtain the two-dimensional coordinates of the skeleton key points corresponding to the pedestrian targets in each detection frame.

[0065] Step 203, determining the three-dimensional coordinates of the skeleton key points of each pedestrian target according to the two-dimensional coordinates of the skeleton key points corresponding to the pedestrian target in each detection frame, the depth image and the intrinsic parameter matrix of the image acquisition device.

[0066] Step 204, based on multiple frame images before the current frame image, use Kalman filtering to track the detection frame of the pedestrian target to obtain a predicted detection frame corresponding to at least one predicted pedestrian target in the current frame image, and / or use Kalman filtering to track the skeletal key points of the pedestrian target to obtain the three-dimensional coordinate prediction value of the skeletal key points of at least one predicted pedestrian target in the current frame image.

[0067] In order to track and filter the position of pedestrians, the present invention uses a Kalman filtering method to track the detection frame of the pedestrian target and / or the skeleton key points of the pedestrian target.

[0068] As a possible implementation method, the detection frame is composed of the two-dimensional coordinates of the left, top, right and bottom sides of the matrix frame that minimally encloses the corresponding pedestrian target, so that the two-dimensional coordinates of the upper left corner point and the lower right corner point of the detection frame corresponding to at least one pedestrian target in the multiple frames before the current frame image can be used as the state vector, the second-order difference of the first point to be tracked is used as the first preset vector, and the first-order difference is unchanged as the target, and the first motion equation is constructed:

[0069]

[0070] Among them, x i Indicates the two-dimensional coordinates of the first point to be tracked in the current frame image, x i-1 Indicates the two-dimensional coordinates of the first point to be tracked in the previous frame image before the current frame image, x i-2 represents the two-dimensional coordinates of the first point to be tracked in the previous frame of the previous image before the current frame of the image,

[0071]

[0072] A x is a first state transfer matrix, indicating that the second-order difference of the first point to be tracked is a first preset vector, and the first-order difference remains unchanged, wherein I is a unit matrix; wherein the first point to be tracked is the upper left corner point or the lower right corner point of a detection frame corresponding to any pedestrian target in multiple frames of images before the current frame image, the two-dimensional coordinates of the upper left corner point of any detection frame are determined based on the two-dimensional coordinates of the left and upper sides of the detection frame, and the two-dimensional coordinates of the lower right corner point of any detection frame are determined based on the two-dimensional coordinates of the right and lower sides of the detection frame;

[0073] Determine, according to the first motion equation, a predicted detection frame corresponding to at least one predicted pedestrian target in the current frame image; and / or,

[0074] The second motion equation can be constructed by taking the three-dimensional coordinates of the skeleton key points of at least one pedestrian target in the multiple frames before the current frame as the state vector, the second-order difference of the second point to be tracked as the second preset vector, and the first-order difference unchanged as the target:

[0075]

[0076] Among them, x′ i Indicates the three-dimensional coordinates of the second point to be tracked in the current frame image, x′ i-1 Indicates the 3D coordinates of the second point to be tracked in the previous frame before the current frame, x′ i-2 represents the three-dimensional coordinates of the second point to be tracked in the previous frame image before the current frame image,

[0077]

[0078] A′ x is a second state transfer matrix, indicating that the second-order difference of the second point to be tracked is a second preset vector, and the first-order difference remains unchanged, wherein I is a unit matrix; wherein the second point to be tracked is any skeletal key point of any pedestrian target in the multiple frames before the current frame image;

[0079] According to the second motion equation, a three-dimensional coordinate prediction value of at least one skeleton key point of a predicted pedestrian target in the current frame image is determined.

[0080] Step 205, based on each predicted detection frame, the three-dimensional coordinate prediction value of the skeletal key points of each predicted pedestrian target, and at least one of the three-dimensional coordinates of the skeletal key points of each detection frame and each pedestrian target, use the Hungarian matching algorithm to match each pedestrian target with each predicted pedestrian target to remove non-pedestrian targets from each pedestrian target.

[0081] The present invention uses the Hungarian matching algorithm to determine the matching degree between each pedestrian target in the previous frame image and each pedestrian target in the current frame image, and matches the pedestrian targets between the two frames.

[0082] As a possible implementation method, for any pair of pedestrian targets and predicted pedestrian targets to be matched, the number of skeleton key points of the set pairs of pedestrian targets and predicted pedestrian targets whose confidence is higher than the confidence threshold is determined; when the number of points is not less than the point threshold, the Euclidean distance of each pair of skeleton key points of the pedestrian target and the predicted pedestrian target is used as the judgment criterion, and the Hungarian matching algorithm is used to match each pedestrian target and each predicted pedestrian target to remove non-pedestrian targets from each pedestrian target; when the number of points is less than the point threshold, the intersection and union ratio of the detection box corresponding to the pedestrian target and the predicted detection box corresponding to the predicted pedestrian target is used as the judgment criterion, and the Hungarian matching algorithm is used to match each pedestrian target and each predicted pedestrian target to remove non-pedestrian targets from each pedestrian target.

[0083] That is, in the process of pedestrian target matching, the Euclidean distance of the skeleton key points and the intersection and union ratio of the target's minimum enclosing rectangular box are used as measurement criteria.

[0084] For example, after using Kalman filtering to obtain the predicted detection box corresponding to at least one predicted pedestrian target in the current frame image and the three-dimensional coordinate prediction value of the skeleton key point of at least one predicted pedestrian target in the current frame image, the Hungarian matching algorithm is used to match the skeleton key points using the Euclidean distance of the skeleton key points as the judgment criterion. Specifically, the confidence c of 9 pairs of human skeleton key points such as the nose, left eye, right eye, left ear, right ear, left shoulder, right shoulder, left hip, and right hip in the two targets to be matched (pedestrian target and predicted pedestrian target) is used. i Above the confidence threshold T c The average Euclidean distance l of the points 1 Determine whether it matches:

[0085]

[0086] Among them, w∈(nose,leye,reye,lear,rear,lshoulder,rshoulder,lhip,rhip).

[0087] If the confidence level c i Above the confidence threshold T c If the number of points is less than 2 (the point count threshold), the degree of matching is measured by the intersection-over-union ratio of the minimum bounding rectangles of the two targets to be matched to determine whether they match.

[0088] Furthermore, after obtaining the matched detection target, the state vector and covariance matrix of the Kalman filter are updated.

[0089] In addition, tracking and exit tracking thresholds are set during tracking. Tracking is confirmed only when the target is tracked for multiple consecutive frames, and target tracking loss is confirmed only when the target is lost for multiple consecutive frames, to prevent fluctuations in target detection results in a short period of time from affecting target tracking.

[0090] Step 206, determining whether each pedestrian target invades the target space electronic fence area according to the three-dimensional coordinates of the skeleton key points of each detection frame and / or each pedestrian target.

[0091] It should be noted that the execution process of steps 201 to 203 and step 206 can be implemented in any manner in the embodiments of the present invention, and the embodiments of the present invention do not limit this and will not be described in detail.

[0092] The coal mine underground personnel intrusion identification method of the embodiment of the present invention uses Kalman filtering to track the detection frame of the pedestrian target according to multiple frame images before the current frame image to obtain the predicted detection frame corresponding to at least one predicted pedestrian target in the current frame image, and / or uses Kalman filtering to track the skeleton key points of the pedestrian target to obtain the three-dimensional coordinate prediction value of the skeleton key points of at least one predicted pedestrian target in the current frame image; according to each predicted detection frame, the three-dimensional coordinate prediction value of the skeleton key points of each predicted pedestrian target, and at least one of the three-dimensional coordinates of each detection frame and the skeleton key points of each pedestrian target, the Hungarian matching algorithm is used to match each pedestrian target and each predicted pedestrian target to remove non-pedestrian targets in each pedestrian target, thereby using Kalman filtering to track and filter the pedestrian position, and on the basis of Hungarian matching, the corner points and skeleton points of the minimum enclosing rectangular frame are used as the matching judgment basis of the Hungarian matching algorithm to achieve target tracking, and the non-pedestrian targets in each pedestrian target can be accurately removed, thereby improving the accuracy of pedestrian detection.

[0093] It should be noted that in the present invention, before determining whether each pedestrian target invades the target space electronic fence area according to the three-dimensional coordinates of the skeleton key points of each detection frame and / or each pedestrian target, in addition to pedestrian tracking, target motion confirmation can also be performed to further remove non-pedestrian targets from each pedestrian target. Figure 3 , which explains this process.

[0094] Figure 3 It is a flow chart of a method for identifying personnel intrusion in underground coal mines shown in the third embodiment of the present invention.

[0095] like Figure 3 As shown, the coal mine underground personnel intrusion identification method includes the following steps:

[0096] Step 301, obtain a current frame image captured by an image capture device arranged underground in a coal mine, and use a monocular depth estimation method to determine a depth image corresponding to the current frame image.

[0097] Step 302, detect the pedestrian targets in the current frame image to obtain a detection frame corresponding to at least one pedestrian target in the current frame image, and extract skeleton key points based on the current frame image in the area where each detection frame is located to obtain the two-dimensional coordinates of the skeleton key points corresponding to the pedestrian targets in each detection frame.

[0098] Step 303, determining the three-dimensional coordinates of the skeleton key points of each pedestrian target according to the two-dimensional coordinates of the skeleton key points corresponding to the pedestrian target in each detection frame, the depth image and the intrinsic parameter matrix of the image acquisition device.

[0099] Step 304, for any pedestrian target, determine the target motion confirmation index corresponding to the pedestrian target based on the detection frame corresponding to the pedestrian target in at least one frame image before the current frame image and the detection frame corresponding to the pedestrian target in the current frame image.

[0100] The pedestrian detection results may contain false detections, where some areas that look like pedestrians are detected as not pedestrians. In the coal mine environment, these false detection results are often static objects in the environment. Therefore, in order to remove these false detection results, the present invention will also perform target motion confirmation after pedestrian tracking.

[0101] As a possible implementation method, the detection frame is composed of the two-dimensional coordinates of the left, top, right and bottom sides of the matrix frame that minimally encloses the corresponding pedestrian target, so that the variance of each left side, the variance of each top side, the variance of each right side and the variance of each bottom side can be determined according to the two-dimensional coordinates of the left, top, right and bottom sides of the detection frame corresponding to the candidate pedestrian target in at least one frame image before the current frame image, and the two-dimensional coordinates of the left, top, right and bottom sides of the detection frame corresponding to the candidate pedestrian target in the current frame image; the first target variance with the largest variance is determined from the variances of each left side and each top side, and the second target variance with the largest variance is determined from the variances of each right side and each bottom side; the minimum value of the first target variance and the second target variance is used as the target motion confirmation indicator.

[0102] For example, suppose the quadruple of all detection results of the pedestrian tracking target from the time it is detected to the latest tracking result arranged in time order is (l i ,t i ,r i ,b i ) correspond to the left, top, right, and bottom of the rectangular box, respectively, where i represents the i-th detection result. The target motion confirmation index m is used to confirm the motion amplitude of the detection result:

[0103] m=min(max(var i (l i ),var i (t i )),max(var i (r i ),var i (b i )))

[0104] Among them, max is the maximum value, min is the minimum value, and var is the variance.

[0105] Step 305 : when the target motion confirmation index is not greater than the target motion confirmation threshold, the pedestrian target is determined to be a non-pedestrian target, and the pedestrian target is removed from the pedestrian targets.

[0106] In the embodiment of the present invention, when the target motion confirmation index corresponding to the pedestrian target is not greater than the target motion confirmation threshold, the pedestrian target is considered to be a non-pedestrian target, and at this time, the pedestrian target needs to be removed from the pedestrian targets. On the contrary, when the target motion confirmation index corresponding to the pedestrian target is greater than the target motion confirmation threshold, the pedestrian target is considered to be a pedestrian target, and at this time, there is no need to remove the pedestrian target.

[0107] Step 306, determining whether each pedestrian target invades the target space electronic fence area according to the three-dimensional coordinates of the skeleton key points of each detection frame and / or each pedestrian target.

[0108] It should be noted that the execution process of steps 301 to 303 and step 306 can be implemented in any manner in the embodiments of the present invention, and the embodiments of the present invention do not limit this and will not be described in detail.

[0109] The method for identifying personnel intrusion underground in a coal mine according to an embodiment of the present invention determines, for any pedestrian target, a target motion confirmation index corresponding to the pedestrian target based on a detection frame corresponding to the pedestrian target in at least one frame image before the current frame image, and a detection frame corresponding to the pedestrian target in the current frame image; when the target motion confirmation index is not greater than the target motion confirmation threshold, the pedestrian target is determined to be a non-pedestrian target, and the pedestrian target is removed from all pedestrian targets. Thus, by using the four corner points of the minimum enclosing rectangular frame to perform motion target confirmation, some misdetected targets can be eliminated, thereby improving the accuracy of pedestrian detection in coal mines and reducing false alarms in the challenging environment underground in coal mines.

[0110] It should be noted that, in the present invention, an alarm can also be issued when any pedestrian target invades the target space electronic fence. Figure 4 , which explains this process.

[0111] Figure 4 It is a flow chart of a method for identifying personnel intrusion in underground coal mines shown in the fourth embodiment of the present invention.

[0112] like Figure 4 As shown, the coal mine underground personnel intrusion identification method includes the following steps:

[0113] Step 401, obtain a current frame image captured by an image capture device arranged underground in a coal mine, and use a monocular depth estimation method to determine a depth image corresponding to the current frame image.

[0114] Step 402, detect the pedestrian targets in the current frame image to obtain a detection frame corresponding to at least one pedestrian target in the current frame image, and extract skeleton key points based on the current frame image in the area where each detection frame is located to obtain the two-dimensional coordinates of the skeleton key points corresponding to the pedestrian targets in each detection frame.

[0115] Step 403, determining the three-dimensional coordinates of the skeleton key points of each pedestrian target according to the two-dimensional coordinates of the skeleton key points corresponding to the pedestrian target in each detection frame, the depth image and the intrinsic parameter matrix of the image acquisition device.

[0116] Step 404, determining whether each pedestrian target invades the target space electronic fence area according to the three-dimensional coordinates of the skeleton key points of each detection frame and / or each pedestrian target.

[0117] Step 405: When any pedestrian target invades the target space electronic fence, an alarm is issued.

[0118] In the embodiment of the present invention, an alarm may be issued when any pedestrian target invades the electronic fence of the target space.

[0119] It should be noted that visual detection inevitably results in false alarms and jittery detection results. In order to make the alarm smoother, the present invention also filters the intrusion results before the alarm.

[0120] As a possible implementation method, an exponential moving average can be used to determine the alarm signal at the current moment based on the alarm signal at the previous moment and the intrusion signal at the current moment; wherein the alarm signal is used to indicate whether an alarm is issued, and the intrusion signal is used to indicate whether someone has invaded the electronic fence of the target space.

[0121] For example, suppose a i is the alarm signal at time i, indicating whether an alarm is needed. is the intrusion signal at time i, indicating whether someone has invaded the electronic fence area.

[0122]

[0123] Among them, η ranges from 0<η<1 and is the coefficient of the exponential moving average.

[0124] It should be noted that the execution process of steps 401 to 404 can be implemented in any manner in the embodiments of the present invention, and the embodiments of the present invention do not limit this and will not be described in detail.

[0125] The method for identifying intrusion of personnel in underground coal mines according to an embodiment of the present invention generates an alarm when any pedestrian target intrudes into the electronic fence of the target space, and before generating the alarm, uses an exponential moving average to filter the intrusion result to avoid false alarms and reduce the false alarm rate.

[0126] In order to clearly illustrate the above embodiment, an example is now given for illustration.

[0127] Figure 5 It is a flow chart of a method for identifying personnel intrusion in underground coal mines shown in the fifth embodiment of the present invention.

[0128] Figure 6 This is a schematic diagram of the principle of identifying personnel intrusion in an underground coal mine provided by the sixth embodiment of the present invention.

[0129] like Figure 5 and Figure 6 As shown, the present invention designs a simple and effective method for identifying personnel intrusion in coal mines in closed, narrow, dim and blurred environments. The present invention uses a pedestrian detection network optimized by a coal mine underground data set to detect pedestrians in coal mine underground images, and uses monocular depth estimation and human skeletons to track the three-dimensional position of the detected target, then eliminates false targets through motion detection, confirms intrusion through multiple points, and alarms the intruder target.

[0130] Based on any embodiment of the present invention, Figure 5 and Figure 6 As shown, the coal mine underground personnel intrusion identification method of the embodiment of the present invention can also be implemented based on the following steps:

[0131] Step 1: Monocular depth estimation

[0132] In order to better track the position of pedestrians, the present invention will use a monocular depth estimation method to estimate the depth of the current frame image, thereby obtaining the approximate depth of the current frame. The current frame is input into the monocular depth estimation network to obtain the depth image of the current frame, from which the depth information d of each pixel point p can be obtained. p .

[0133] Using the intrinsic parameter matrix K of the image acquisition device, any pixel point p can be restored to the three-dimensional space coordinates

[0134] x=dp K -1 p

[0135] Among them, the coordinates of the pixel point p are given in the form of homogeneous coordinates.

[0136] Step 2 Spatial electronic fence area extraction

[0137] The present invention requires inputting a ROI as the area to be determined. The ROI is a closed figure composed of an ordered two-dimensional point sequence, and the area on the right hand side along the order of the point sequence is the inner area of the closed figure. Let the ordered two-dimensional point sequence be p (ROI),i , i = 1, 2,..., L (ROI) The length of which is L (ROI) , for any pixel point p in the current frame image, if the pixel point p is relative to the adjacent points in any ordered two-dimensional point sequence, that is, i < L (ROI) , j = i + 1 and i = L (ROI) , j = 1, there is: the pixel point p is on the right hand side of the directed line segment formed by the pixel point p (ROI),i and the pixel point p (ROI),j , then it can be determined that the pixel point p is located inside the ROI.

[0138] The ROI preset by the user in the current frame image is a planar area in the current frame image. In order to achieve the tracking effect in space, the present invention will utilize monocular depth estimation, extract the corresponding area in space according to the input ROI area, and perform electronic fence intrusion detection on the area in space.

[0139] Specifically, first extract the depth of the pixels inside the input ROI in the image and restore it to three-dimensional space coordinates; then use the Hough transform to extract all planes and their normal vectors, and calculate the number of inliers of each plane as the area occupied by the plane in the ROI area; then select the plane with the largest area as the ground plane P, and obtain its normal vector n P , and the intersection point o of the straight line along the normal vector direction of the ground plane from the coordinate origin and the ground plane; finally project the three-dimensional coordinates x s,(ROI),i of the ordered two-dimensional point sequence constituting the ROI onto the ground plane:

[0140] x g,(ROI),i = x s,(ROI),i -(x g,(ROI),i - o)n P

[0141] Obtain the ROI on the ground plane, and use this as the basis for pedestrian electronic fence intrusion detection, that is, the target spatial electronic fence.

[0142] Step 3 Pedestrian detection

[0143] Accurate pedestrian detection is a prerequisite for pedestrian tracking and intrusion detection. The present invention uses a coal mine pedestrian data set to train a deep neural network, and then infers the input image to achieve the detection of pedestrian targets in the visible light image of the coal mine. The image collected by the visible light camera in the coal mine is input into the pedestrian detection network to obtain the position of the pedestrian target. The target position is output in the form of multiple quadruples, and the four numbers contained in it represent the positions of the left, top, right, and bottom of the minimum enclosing rectangular box of the target, respectively, represented by l, t, r, and b; the upper left corner point of the rectangular box can be represented as p (lt) =[l,t] T , the lower right corner point can be represented as p (rb) =[r,b] T , the center point can be expressed as p (center) =[(l+r) / 2,(t+b) / 2] T .

[0144] Step 4: Extraction of skeleton key points

[0145] Skeleton key point extraction is usually used for pedestrian posture analysis. In order to achieve better pedestrian tracking, the present invention extracts the skeleton key points of pedestrians. The image of the area of ​​the pedestrian enclosing rectangular frame obtained by pedestrian detection is cut out and input into the skeleton key point extraction network to obtain the skeleton key points of the pedestrian, including the nose, left eye, right eye, left ear, right ear, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left hip, right hip, left knee, right knee, left ankle, right ankle and other 17 human skeleton key point positions, which are p respectively. (nose) 、p (leye) 、p (reye) 、p (lear) 、p (rear) 、p (lshoulder) 、p (rshoulder) 、p (lelbow) 、p (relbow) 、p (lwrist) 、p (rwrist) 、p (lhip) 、p (rhip) 、p (lknee) 、p (rknee) 、p (lankle) 、p (rankle) .

[0146] Step 5 Kalman filter

[0147] In order to track and filter the position of pedestrians, the present invention uses the Kalman filter method to track and filter the key points of pedestrian skeletons. The present invention uses the key points of pedestrian skeletons as the state vector, assumes that the second-order difference of the point to be tracked is 0, and maintains the first-order difference. Let x i is the coordinate of the three-dimensional point to be tracked at the current time i, then its second-order motion equation is

[0148]

[0149] Among them, x i-1 is the three-dimensional coordinate of the previous moment, i.e., moment i-1, x i-2 is the three-dimensional coordinate at time i-2,

[0150]

[0151] is the state transfer matrix, indicating that the second-order difference of the point to be tracked is 0, the first-order difference is maintained, and I is the unit matrix.

[0152] In addition to tracking and filtering the key points of the human skeleton, the present invention also uses Kalman filtering to track the minimum enclosing rectangular box obtained by pedestrian detection. Specifically, the upper left corner point and the lower right corner point of the minimum enclosing rectangular box are tracked, and the state vector is formed by its two-dimensional coordinates. The assumption of its motion equation is the same as the motion equation of the key points of the skeleton, that is, the second-order difference of the points to be tracked is 0.

[0153] Step 6 Pedestrian tracking

[0154] After obtaining the detection frame using the detection network, the present invention uses the Hungarian matching algorithm to determine the matching degree of each target in the previous frame with each target in the current frame, and matches the pedestrian targets between the two frames. When matching targets, the present invention uses the Euclidean distance of the skeleton key points and the intersection and union ratio of the target's minimum enclosing rectangular frame as the measurement criteria.

[0155] First, the minimum bounding rectangle and skeleton key points of the target in the previous frame are predicted based on the second-order motion equation, and its position in the current frame is predicted to obtain the minimum bounding rectangle P of the target in the previous frame. i-1 and the bone key point x i-1 The predicted value in the current frame

[0156] Then, the Hungarian matching algorithm is used to match the key points of the skeleton using the Euclidean distance as the criterion. Specifically, the confidence c of the nine pairs of key points of the human skeleton, such as the nose, left eye, right eye, left ear, right ear, left shoulder, right shoulder, left hip, and right hip, in the two targets to be matched is used. i Above the confidence threshold T c The average Euclidean distance l of the points 1 Determine whether it matches:

[0157]

[0158] Among them, w∈(nose,leye,reye,lear,rear,lshoulder,rshoulder,lhip,rhip).

[0159] If the confidence level c i Above the confidence threshold T c If the number of points is less than 2, the intersection and union ratio of the minimum bounding rectangles of the two targets to be matched is used to measure the matching degree to determine whether they match.

[0160] After obtaining the matched detection target, the state vector and covariance matrix of the Kalman filter will be updated.

[0161] Tracking and exit tracking thresholds are also set during tracking. Tracking is confirmed only when the target is tracked for multiple consecutive frames, and target tracking loss is confirmed only when the target is lost for multiple consecutive frames, to prevent fluctuations in target detection results in a short period of time from affecting target tracking.

[0162] Step 7 Target Movement Confirmation

[0163] The pedestrian detection results may contain false detections, and some areas that look like pedestrians may be detected as not pedestrians. In the underground environment, these false detection results are often static objects in the environment. Therefore, in order to remove these false detection results, the present invention will also perform target motion confirmation after pedestrian tracking, using the changes in the four numbers of the detection results during the tracking process as confirmation indicators.

[0164] Suppose the quadruple of all detection results of the pedestrian tracking target from the time it is detected to the latest tracking result arranged in time order is (l i ,t i ,r i ,b i ) correspond to the left, top, right, and bottom of the rectangular box, respectively, where i represents the i-th detection result. The target motion confirmation index m is used to confirm the motion amplitude of the detection result:

[0165] m=min(max(var i (l i ),var i (t i )),max(var i (r i ),var i (b i )))

[0166] Among them, max is the maximum value, min is the minimum value, and var is the variance. When m is greater than the target motion confirmation threshold, the tracked target is confirmed as a pedestrian target.

[0167] Step 8: Intrusion Confirmation

[0168] After obtaining the detection frame of the pedestrian target, it is necessary to determine whether the target has invaded. The present invention uses the foot position as a reference to determine whether the pedestrian has invaded the electronic fence area. i Above the confidence threshold T c In this case, the present invention will use the ankle bone key point with higher confidence as the foot position, otherwise it is assumed that the midpoint of the lower edge of the pedestrian target detection frame is the foot position.

[0169] The present invention projects the three-dimensional coordinates of the foot onto the ground plane, and then determines whether the pedestrian's foot is within the electronic fence area. If it is within the electronic fence, it is considered that there is an intrusion situation.

[0170] Step 9: Alarm filtering

[0171] Visual detection will inevitably result in false alarms and jittery detection results. In order to make the alarm smoother, the present invention will filter the intrusion results before the alarm. The present invention uses exponential moving average (EMA) to filter the intrusion results.

[0172] Specifically, let a i is the alarm signal at time i, indicating whether an alarm is needed. is the intrusion signal at time i, indicating whether someone has invaded the electronic fence area.

[0173]

[0174] Among them, η ranges from 0<η<1 and is the coefficient of the exponential moving average.

[0175] In summary, the position of the electronic fence area in the space is extracted to achieve more accurate electronic fence intrusion detection; the three-dimensional coordinates of the skeleton key points are restored by monocular depth estimation, and then the Kalman filter is used to track and filter the pedestrian position; on the basis of Hungarian matching, the corner points of the minimum enclosing rectangular box and the skeleton points are used as the matching judgment basis of the Hungarian matching algorithm to achieve target tracking; the four corner points of the minimum enclosing rectangular box are used to confirm the moving target and eliminate some misdetected targets. Therefore, when the pedestrian detection algorithm faces the narrow and dim conditions of the underground environment of the coal mine, there are a large number of false alarms, which leads to the problem of more false alarms in the electronic fence intrusion detection. The present invention optimizes the pre-trained pedestrian detection model using the underground coal mine data set, and then uses the monocular depth estimation to restore the pedestrian position to three-dimensional coordinates and track it. Then, the moving target confirmation is applied, and the alarm result is filtered to determine the target of the intrusion space electronic fence area, thereby improving the accuracy of pedestrian detection in the coal mine and reducing the false alarms of the challenging environment in the coal mine.

[0176] Corresponding to the coal mine underground personnel intrusion identification method provided in the above embodiment, the present invention also provides a coal mine underground personnel intrusion identification device. Since the coal mine underground personnel intrusion identification device provided in the embodiment of the present invention corresponds to the coal mine underground personnel intrusion identification method provided in the above embodiment, the implementation method of the coal mine underground personnel intrusion identification method is also applicable to the coal mine underground personnel intrusion identification device provided in the embodiment of the present invention, and will not be described in detail in the embodiment of the present invention.

[0177] Figure 7 It is a schematic diagram of the structure of a coal mine underground personnel intrusion identification device shown in the seventh embodiment of the present invention.

[0178] like Figure 7 As shown, the coal mine underground personnel intrusion identification device 700 includes: a first processing module 710, a second processing module 720, a first determination module 730 and a second determination module 740.

[0179] Among them, the first processing module 710 is used to obtain the current frame image collected by the image acquisition device arranged in the coal mine, and use the monocular depth estimation method to determine the depth image corresponding to the current frame image; the second processing module 720 is used to detect the pedestrian target in the current frame image to obtain the detection frame corresponding to at least one pedestrian target in the current frame image, and extract the skeleton key points based on the current frame image in the area where each detection frame is located to obtain the two-dimensional coordinates of the skeleton key points of the corresponding pedestrian target in each detection frame; wherein the detection frame is a matrix frame that minimally surrounds the corresponding pedestrian target; the first determination module 730 is used to determine the three-dimensional coordinates of the skeleton key points of each pedestrian target according to the two-dimensional coordinates of the skeleton key points of the corresponding pedestrian target in each detection frame, the depth image and the internal reference matrix of the image acquisition device; the second determination module 740 is used to determine whether each pedestrian target invades the target space electronic fence area according to the three-dimensional coordinates of the skeleton key points of each detection frame and / or each pedestrian target; wherein the target space electronic fence is determined based on the depth image and the preset area to be determined.

[0180] As a possible implementation method of an embodiment of the present invention, the skeletal key points include at least left and right ankle skeletal key points; the second determination module 740 is also used to: for any pedestrian target, when the confidences of the left and right ankle skeletal key points of the pedestrian target are both greater than the confidence threshold, use the three-dimensional coordinates of the ankle skeletal key points with higher confidence among the left and right ankle skeletal key points of the pedestrian target as the foot position of the pedestrian target; when the confidences of the left and right ankle skeletal key points of the pedestrian target are not both greater than the confidence threshold, use the three-dimensional coordinates of the lower midpoint of the detection box corresponding to the pedestrian target as the pedestrian. The foot position of the target; wherein the three-dimensional coordinates of the midpoint of the lower side of the detection frame corresponding to the pedestrian target are determined based on the two-dimensional coordinates of the midpoint of the lower side of the detection frame corresponding to the pedestrian target, the depth image and the internal parameter matrix of the image acquisition device; the foot position of the pedestrian target is projected onto the ground plane to determine whether the foot position of the pedestrian target is within the electronic fence area of ​​the target space, and when the foot position of the pedestrian target is within the electronic fence area of ​​the target space, it is determined that the pedestrian target has invaded the electronic fence area of ​​the target space; wherein the ground plane is the ground plane determined in the process of determining the electronic fence area of ​​the target space.

[0181] As a possible implementation method of an embodiment of the present invention, the above-mentioned device also includes: a tracking module, which is used to track the detection frame of the pedestrian target using Kalman filtering based on multiple frame images before the current frame image to obtain a predicted detection frame corresponding to at least one predicted pedestrian target in the current frame image, and / or, to track the skeletal key points of the pedestrian target using Kalman filtering to obtain a three-dimensional coordinate prediction value of the skeletal key points of at least one predicted pedestrian target in the current frame image; a matching module, which is used to match each pedestrian target and each predicted pedestrian target using the Hungarian matching algorithm based on at least one of each predicted detection frame, the three-dimensional coordinate prediction value of the skeletal key points of each predicted pedestrian target, and each detection frame and the three-dimensional coordinates of the skeletal key points of each pedestrian target to remove non-pedestrian targets from each pedestrian target.

[0182] As a possible implementation method of an embodiment of the present invention, the detection frame is composed of the two-dimensional coordinates of the left, top, right and bottom sides of the matrix frame that minimally encloses the corresponding pedestrian target; the tracking module is further used to: use the two-dimensional coordinates of the upper left corner point and the lower right corner point of the detection frame corresponding to at least one pedestrian target in the multiple frames before the current frame image as the state vector, use the second-order difference of the first point to be tracked as the first preset vector, and the first-order difference unchanged as the target, to construct the first motion equation:

[0183]

[0184] Among them, x i Indicates the two-dimensional coordinates of the first point to be tracked in the current frame image, x i-1Indicates the two-dimensional coordinates of the first point to be tracked in the previous frame image before the current frame image, x i-2 represents the two-dimensional coordinates of the first point to be tracked in the previous frame of the previous image before the current frame of the image,

[0185]

[0186] A x is the first state transfer matrix, indicating that the second-order difference of the first point to be tracked is the first preset vector, and the first-order difference remains unchanged, wherein I is the unit matrix; wherein the first point to be tracked is the upper left corner point or the lower right corner point of the detection box corresponding to any pedestrian target in the multiple frames before the current frame image, the two-dimensional coordinates of the upper left corner point of any detection box are determined based on the two-dimensional coordinates of the left and top sides of the detection box, and the two-dimensional coordinates of the lower right corner point of any detection box are determined based on the two-dimensional coordinates of the right and bottom sides of the detection box; according to the first motion equation, determine the predicted detection box corresponding to at least one predicted pedestrian target in the current frame image; and / or, take the three-dimensional coordinates of the skeleton key points of at least one pedestrian target in the multiple frames before the current frame image as the state vector, take the second-order difference of the second point to be tracked as the second preset vector, and the first-order difference remains unchanged as the target, and construct the second motion equation:

[0187]

[0188] Among them, x′ i Indicates the three-dimensional coordinates of the second point to be tracked in the current frame image, x′ i-1 Indicates the 3D coordinates of the second point to be tracked in the previous frame before the current frame, x′ i-2 represents the three-dimensional coordinates of the second point to be tracked in the previous frame image before the current frame image,

[0189]

[0190] A′ x is the second state transfer matrix, indicating that the second-order difference of the second point to be tracked is the second preset vector, and the first-order difference remains unchanged, wherein I is the unit matrix; wherein the second point to be tracked is any skeletal key point of any pedestrian target in the multiple frames before the current frame image; according to the second motion equation, the three-dimensional coordinate prediction value of the skeletal key point of at least one predicted pedestrian target in the current frame image is determined.

[0191] As a possible implementation method of an embodiment of the present invention, the matching module is also used to: for any pair of pedestrian targets and predicted pedestrian targets to be matched, determine the number of skeletal key points of the set pairs of pedestrian targets and predicted pedestrian targets whose confidence is higher than a confidence threshold; when the number of points is not less than the point threshold, use the Euclidean distance of each pair of skeletal key points of the pedestrian target and the predicted pedestrian target as a judgment criterion, and adopt the Hungarian matching algorithm to match each pedestrian target and each predicted pedestrian target to remove non-pedestrian targets from each pedestrian target; when the number of points is less than the point threshold, use the intersection-and-union ratio of the detection box corresponding to the pedestrian target and the predicted detection box corresponding to the predicted pedestrian target as a judgment criterion, and adopt the Hungarian matching algorithm to match each pedestrian target and each predicted pedestrian target to remove non-pedestrian targets from each pedestrian target.

[0192] As a possible implementation method of an embodiment of the present invention, the above-mentioned device also includes: a third determination module, which is used to determine, for any pedestrian target, a target motion confirmation index corresponding to the pedestrian target based on a detection frame corresponding to the pedestrian target in at least one frame image before the current frame image, and a detection frame corresponding to the pedestrian target in the current frame image; a third processing module, which is used to determine that the pedestrian target is a non-pedestrian target and remove the pedestrian target from all pedestrian targets when the target motion confirmation index corresponding to the pedestrian target is not greater than a target motion confirmation threshold.

[0193] As a possible implementation method of an embodiment of the present invention, the detection frame is composed of the two-dimensional coordinates of the left, top, right and bottom sides of the matrix frame that minimally encloses the corresponding pedestrian target; the third determination module is also used to: determine the variance of each left side, the variance of each top side, the variance of each right side and the variance of each bottom side according to the two-dimensional coordinates of the left, top, right and bottom sides of the detection frame corresponding to the candidate pedestrian target in at least one frame image before the current frame image, and the two-dimensional coordinates of the left, top, right and bottom sides of the detection frame corresponding to the candidate pedestrian target in the current frame image; determine the first target variance with the largest variance from the variances of each left side and each top side, and determine the second target variance with the largest variance from the variances of each right side and each bottom side; and use the minimum value between the first target variance and the second target variance as the target motion confirmation indicator.

[0194] As a possible implementation method of an embodiment of the present invention, the above-mentioned device also includes: a fourth determination module, which is used to determine the three-dimensional coordinates of each point in the two-dimensional point sequence based on the depth image and the two-dimensional coordinates of each point in the two-dimensional point sequence constituting the area to be determined; a fourth processing module, which is used to extract multiple planes in the area to be determined using Hough transform, and use the number of inner points of each plane as the area occupied by the corresponding plane in the area to be determined; a fifth processing module, which is used to select the plane with the largest area from each plane as the ground plane, and obtain the normal vector of the ground plane and the three-dimensional coordinates of the target intersection; wherein the target intersection is the intersection of the straight line from the coordinate origin along the normal vector direction of the ground plane and the ground plane; a projection module, which is used to project the three-dimensional coordinates of each point in the two-dimensional point sequence onto the ground plane based on the normal vector of the ground plane and the three-dimensional coordinates of the target intersection, so as to obtain the target space electronic fence.

[0195] As a possible implementation manner of the embodiment of the present invention, the above-mentioned device further includes: an alarm module, which is used to issue an alarm when any pedestrian target invades the electronic fence of the target space.

[0196] As a possible implementation method of an embodiment of the present invention, the above-mentioned device also includes: a fifth determination module, which is used to use an exponential moving average to determine the alarm signal at the current moment based on the alarm signal at the previous moment and the intrusion signal at the current moment; wherein the alarm signal is used to indicate whether to issue an alarm, and the intrusion signal is used to indicate whether someone has invaded the electronic fence of the target space.

[0197] The coal mine underground personnel intrusion identification device of the embodiment of the present invention obtains the current frame image collected by the image acquisition device arranged in the coal mine, and uses the monocular depth estimation method to determine the depth image corresponding to the current frame image; detects the pedestrian target in the current frame image to obtain a detection frame corresponding to at least one pedestrian target in the current frame image, and extracts the skeleton key points based on the current frame image of the area where each detection frame is located to obtain the two-dimensional coordinates of the skeleton key points of the corresponding pedestrian target in each detection frame; wherein the detection frame is a matrix frame that minimally surrounds the corresponding pedestrian target; and determines each row according to the two-dimensional coordinates of the skeleton key points of the corresponding pedestrian target in each detection frame, the depth image and the intrinsic parameter matrix of the image acquisition device. The three-dimensional coordinates of the skeleton key points of the human target; according to the three-dimensional coordinates of the skeleton key points of each detection frame and / or each pedestrian target, determine whether each pedestrian target invades the target space electronic fence area; wherein the target space electronic fence is determined based on the depth image and the preset area to be determined. Therefore, in view of the problem that the existing pedestrian detection method has a large number of false alarms when applied in the dim and blurred environment of the coal mine, resulting in a large number of false alarms in the electronic fence intrusion detection, the present invention combines monocular depth estimation with the human skeleton to perform three-dimensional position tracking on the pedestrian target in the current frame image collected by the image acquisition device arranged in the coal mine, which can improve the accuracy of pedestrian detection in the coal mine and reduce the false alarm rate of electronic fence intrusion detection.

[0198] In an exemplary embodiment, the present invention also provides an electronic device.

[0199] Among them, electronic equipment includes:

[0200] processor;

[0201] a memory for storing processor-executable instructions;

[0202] The processor is configured to execute instructions to implement the method for identifying personnel intrusion in underground coal mines as proposed in any of the aforementioned embodiments.

[0203] As an example, Figure 8 is a schematic diagram of the structure of an electronic device 800 shown in an exemplary embodiment of the present invention. Figure 8 As shown, the electronic device 800 may further include:

[0204] The memory 810 and the processor 820, a bus 830 connecting different components (including the memory 810 and the processor 820), the memory 810 stores a computer program, and when the processor 820 executes the program, the method for identifying underground personnel intrusion in a coal mine described in an embodiment of the present invention is implemented.

[0205] Bus 830 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processor or a local bus using any of a variety of bus architectures. For example, these architectures include but are not limited to Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MAC) bus, Enhanced ISA bus, Video Electronics Standards Association (VESA) local bus and Peripheral Component Interconnect (PCI) bus.

[0206] The electronic device 800 typically includes a variety of electronic device readable media. These media can be any available media that can be accessed by the electronic device 800, including volatile and non-volatile media, removable and non-removable media.

[0207] The memory 810 may also include computer system readable media in the form of volatile memory, such as random access memory (RAM) 840 and / or cache memory 850. The server 800 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, the storage system 860 may be used to read and write non-removable, non-volatile magnetic media ( Figure 8 not shown, often referred to as a "hard drive packager"). Although Figure 8 Not shown in the figure, a disk driver package for reading and writing a removable non-volatile disk (e.g., a "floppy disk"), and an optical disk driver package for reading and writing a removable non-volatile optical disk (e.g., a CD-ROM, a DVD-ROM, or other optical media) may be provided. In these cases, each driver package can be connected to the bus 830 via one or more data medium interfaces. The memory 810 may include at least one program product having a set (e.g., at least one) of program modules that are configured to perform the functions of the embodiments of the present invention.

[0208] A program / utility 880 having a set (at least one) of program modules 870 may be stored, for example, in the memory 810, such program modules 870 including but not limited to an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment. The program modules 870 generally perform the functions and / or methods of the embodiments described herein.

[0209] The electronic device 800 may also communicate with one or more external devices 890 (e.g., keyboard, pointing device, display 891, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 800, and / or communicate with any device that enables the electronic device 800 to communicate with one or more other computing devices (e.g., network card, modem, etc.). Such communication may be performed through an input / output (I / O) interface 892. In addition, the electronic device 800 may also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN) and / or public network, such as the Internet) through a network adapter 893. As shown, the network adapter 893 communicates with other modules of the electronic device 800 through a bus 830. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 800, including but not limited to: microcode, device driver package, redundant processing unit, external disk driver package array, RAID system, tape driver package, and data backup storage system, etc.

[0210] The processor 820 executes various functional applications and data processing by running the programs stored in the memory 810 .

[0211] It should be noted that the implementation process and technical principles of the electronic device of this embodiment refer to the aforementioned explanation of the coal mine underground personnel intrusion identification method of the embodiment of the present invention, and will not be repeated here.

[0212] In an exemplary embodiment, the present invention further provides a computer-readable storage medium including instructions, such as a memory including instructions, and the above instructions can be executed by a processor of an electronic device to complete the coal mine underground personnel intrusion identification method proposed in any of the above embodiments. Optionally, the computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0213] In an exemplary embodiment, the present invention further provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the method for identifying personnel intrusion in underground coal mines proposed in any of the above embodiments.

[0214] Those skilled in the art will readily appreciate other embodiments of the present invention after considering the specification and practicing the invention disclosed herein. The present invention is intended to cover any variations, uses or adaptations of the present invention that follow the general principles of the present invention and include common knowledge or customary techniques in the art not disclosed by the present invention. The description and examples are to be regarded as exemplary only, and the true scope and spirit of the present invention is indicated by the following claims.

[0215] It should be understood that the present invention is not limited to the exact construction that has been described above and shown in the drawings and that various modifications and changes may be made without departing from the scope thereof. The scope of the present invention is limited only by the appended claims.

Claims

1. A method for identifying personnel intrusion in underground coal mines, characterized in that: include: Acquire a current frame image acquired by an image acquisition device arranged in an underground coal mine, and determine a depth image corresponding to the current frame image by using a monocular depth estimation method; Detecting pedestrian targets in the current frame image to obtain a detection frame corresponding to at least one pedestrian target in the current frame image, and extracting skeleton key points based on the current frame image in the area where each detection frame is located to obtain two-dimensional coordinates of the skeleton key points corresponding to the pedestrian target in each detection frame; wherein the detection frame is a matrix frame that minimally surrounds the corresponding pedestrian target; Determine the three-dimensional coordinates of the skeleton key points of each pedestrian target according to the two-dimensional coordinates of the skeleton key points of the pedestrian target in each detection frame, the depth image and the intrinsic parameter matrix of the image acquisition device; According to the three-dimensional coordinates of the skeleton key points of each detection frame and / or each pedestrian target, determine whether each pedestrian target invades the target space electronic fence area; wherein the target space electronic fence is determined based on the depth image and the preset area to be determined.

2. The method according to claim 1, characterized in that The skeleton key points at least include left and right ankle skeleton key points; determining whether each pedestrian target invades the target space electronic fence area according to the three-dimensional coordinates of the skeleton key points of each detection frame and / or each pedestrian target, includes: For any of the pedestrian targets, when the confidences of the left and right ankle bone key points of the pedestrian target are both greater than the confidence threshold, the three-dimensional coordinates of the ankle bone key points with higher confidence among the left and right ankle bone key points of the pedestrian target are used as the foot position of the pedestrian target; In the case where the confidences of the left and right ankle bone key points of the pedestrian target are not both greater than the confidence threshold, the three-dimensional coordinates of the midpoint of the lower side of the detection frame corresponding to the pedestrian target are used as the foot position of the pedestrian target; wherein the three-dimensional coordinates of the midpoint of the lower side of the detection frame corresponding to the pedestrian target are determined based on the two-dimensional coordinates of the midpoint of the lower side of the detection frame corresponding to the pedestrian target, the depth image, and the intrinsic parameter matrix of the image acquisition device; The foot position of the pedestrian target is projected onto the ground plane to determine whether the foot position of the pedestrian target is within the target space electronic fence area, and when the foot position of the pedestrian target is within the target space electronic fence area, it is determined that the pedestrian target has invaded the target space electronic fence area; wherein the ground plane is the ground plane determined in the process of determining the target space electronic fence area.

3. The method according to claim 1, characterized in that Before determining whether each of the pedestrian targets invades the target space electronic fence area according to the three-dimensional coordinates of the skeleton key points of each of the detection frames and / or each of the pedestrian targets, the method further includes: According to multiple frames of images before the current frame image, a detection frame of a pedestrian target is tracked using a Kalman filter to obtain a predicted detection frame corresponding to at least one predicted pedestrian target in the current frame image, and / or, a skeletal key point of a pedestrian target is tracked using a Kalman filter to obtain a three-dimensional coordinate prediction value of a skeletal key point of at least one predicted pedestrian target in the current frame image; Based on each of the predicted detection frames, the predicted three-dimensional coordinate values ​​of the skeletal key points of each predicted pedestrian target, and at least one of the three-dimensional coordinates of each of the detection frames and the skeletal key points of each pedestrian target, the Hungarian matching algorithm is used to match each of the pedestrian targets and each of the predicted pedestrian targets to remove non-pedestrian targets from each of the pedestrian targets.

4. The method according to claim 3, characterized in that The detection frame is composed of the two-dimensional coordinates of the left, top, right and bottom sides of the matrix frame that minimally encloses the corresponding pedestrian target; the detection frame of the pedestrian target is tracked by Kalman filtering according to the multiple frames before the current frame image to obtain the predicted detection frame corresponding to at least one predicted pedestrian target in the current frame image, and / or the skeleton key points of the pedestrian target are tracked by Kalman filtering to obtain the three-dimensional coordinate prediction value of the skeleton key points of at least one predicted pedestrian target in the current frame image, including: The two-dimensional coordinates of the upper left corner and the lower right corner of the detection box corresponding to at least one pedestrian target in the multiple frames before the current frame image are used as the state vector, the second-order difference of the first point to be tracked is used as the first preset vector, and the first-order difference is used as the target to construct the first motion equation: Among them, x i represents the two-dimensional coordinates of the first point to be tracked in the current frame image, x i-1 represents the two-dimensional coordinates of the first point to be tracked in the previous frame image before the current frame image, x i-2 represents the two-dimensional coordinates of the first point to be tracked in the previous frame image before the current frame image, A x is a first state transfer matrix, indicating that the second-order difference of the first point to be tracked is a first preset vector, and the first-order difference remains unchanged, wherein I is a unit matrix; wherein the first point to be tracked is the upper left corner point or the lower right corner point of a detection frame corresponding to any of the pedestrian targets in the multiple frames of images before the current frame image, the two-dimensional coordinates of the upper left corner point of any of the detection frames are determined based on the two-dimensional coordinates of the left and top sides of the detection frame, and the two-dimensional coordinates of the lower right corner point of any of the detection frames are determined based on the two-dimensional coordinates of the right and bottom sides of the detection frame; determining, according to the first motion equation, a predicted detection frame corresponding to at least one predicted pedestrian target in the current frame image; and / or, The three-dimensional coordinates of the skeleton key points of at least one pedestrian target in the multiple frames before the current frame image are used as the state vector, the second-order difference of the second point to be tracked is used as the second preset vector, and the first-order difference is unchanged as the target, and the second motion equation is constructed: Among them, x′ i represents the three-dimensional coordinates of the second point to be tracked in the current frame image, x i-1 ' represents the three-dimensional coordinates of the second point to be tracked in the previous frame image before the current frame image, x i-2 ' represents the three-dimensional coordinates of the second point to be tracked in the previous frame image before the current frame image, A′ x is a second state transfer matrix, indicating that the second-order difference of the second point to be tracked is a second preset vector, and the first-order difference remains unchanged, wherein I is a unit matrix; wherein the second point to be tracked is any skeletal key point of any pedestrian target in the multiple frames of images before the current frame image; According to the second motion equation, a three-dimensional coordinate prediction value of at least one predicted skeletal key point of a pedestrian target in the current frame image is determined.

5. The method according to claim 3, characterized in that: The method further comprises: matching each pedestrian target with each predicted pedestrian target using a Hungarian matching algorithm according to at least one of the predicted detection frames, the predicted three-dimensional coordinates of the skeleton key points of each predicted pedestrian target, and the three-dimensional coordinates of the skeleton key points of each detection frame and each pedestrian target, so as to remove non-pedestrian targets from each pedestrian target, including: For any pair of the pedestrian target and the predicted pedestrian target to be matched, determine the number of skeleton key points of the set pair of skeleton key points of the pedestrian target and the predicted pedestrian target whose confidence is higher than a confidence threshold; When the number of points is not less than the point threshold, the Euclidean distance of each pair of skeleton key points between the pedestrian target and the predicted pedestrian target is used as a judgment criterion, and the Hungarian matching algorithm is used to match each pedestrian target with each predicted pedestrian target, so as to remove non-pedestrian targets from each pedestrian target; When the number of points is less than the point threshold, the intersection-and-union ratio of the detection box corresponding to the pedestrian target and the predicted detection box corresponding to the predicted pedestrian target is used as the judgment criterion, and the Hungarian matching algorithm is used to match each pedestrian target and each predicted pedestrian target to remove non-pedestrian targets from each pedestrian target.

6. The method according to claim 1, characterized in that Before determining whether each of the pedestrian targets invades the target space electronic fence area according to the three-dimensional coordinates of the skeleton key points of each of the detection frames and / or each of the pedestrian targets, the method further includes: For any of the pedestrian targets, determine a target motion confirmation index corresponding to the pedestrian target according to a detection frame corresponding to the pedestrian target in at least one frame image before the current frame image and a detection frame corresponding to the pedestrian target in the current frame image; When the target motion confirmation index corresponding to the pedestrian target is not greater than the target motion confirmation threshold, the pedestrian target is determined to be a non-pedestrian target, and the pedestrian target is removed from the pedestrian targets.

7. The method according to claim 6, characterized in that The detection frame is composed of two-dimensional coordinates of the left, top, right and bottom sides of a matrix frame that minimally encloses the corresponding pedestrian target; the target motion confirmation index corresponding to the pedestrian target is determined according to the detection frame corresponding to the candidate pedestrian target in at least one frame image before the current frame image and the detection frame corresponding to the candidate pedestrian target in the current frame image, including: Determine the variance of each left side, the variance of each top side, the variance of each right side, and the variance of each bottom side according to the two-dimensional coordinates of the left side, the top side, the right side, and the bottom side of the detection box corresponding to the candidate pedestrian target in at least one frame image before the current frame image, and the two-dimensional coordinates of the left side, the top side, the right side, and the bottom side of the detection box corresponding to the candidate pedestrian target in the current frame image; Determine a first target variance with the largest variance from the variances of the left sides and the variances of the upper sides, and determine a second target variance with the largest variance from the variances of the right sides and the variances of the lower sides; The minimum value of the first target variance and the second target variance is used as the target motion confirmation index.

8. The method according to claim 1, characterized in that The process of determining the target space electronic fence includes: Determine the three-dimensional coordinates of each point in the two-dimensional point sequence according to the depth image and the two-dimensional coordinates of each point in the two-dimensional point sequence constituting the area to be determined; Extracting multiple planes in the region to be determined by using Hough transform, and taking the number of inner points of each plane as the area occupied by the corresponding plane in the region to be determined; Selecting the plane with the largest area from the planes as the ground plane, and obtaining the three-dimensional coordinates of the normal vector of the ground plane and the target intersection point; wherein the target intersection point is the intersection point of the straight line from the coordinate origin along the direction of the normal vector of the ground plane and the ground plane; According to the normal vector of the ground plane and the three-dimensional coordinates of the target intersection point, the three-dimensional coordinates of each point in the two-dimensional point sequence are projected onto the ground plane to obtain the target space electronic fence.

9. The method according to any one of claims 1 to 8, characterized in that The method further comprises: When any pedestrian target invades the target space electronic fence, an alarm is issued.

10. The method according to claim 9, characterized in that Before the alarm is issued, the method further includes: Exponential moving average is used to determine the alarm signal at the current moment based on the alarm signal at the previous moment and the intrusion signal at the current moment; wherein the alarm signal is used to indicate whether to alarm, and the intrusion signal is used to indicate whether someone has invaded the electronic fence of the target space.

11. A coal mine underground personnel intrusion identification device, characterized in that: include: The first processing module is used to obtain a current frame image acquired by an image acquisition device arranged in the coal mine, and determine a depth image corresponding to the current frame image by using a monocular depth estimation method; The second processing module is used to detect pedestrian targets in the current frame image to obtain a detection frame corresponding to at least one pedestrian target in the current frame image, and extract skeleton key points based on the current frame image in the area where each detection frame is located to obtain two-dimensional coordinates of the skeleton key points corresponding to the pedestrian target in each detection frame; wherein the detection frame is a matrix frame that minimally surrounds the corresponding pedestrian target; A first determination module is used to determine the three-dimensional coordinates of the skeleton key points of each pedestrian target according to the two-dimensional coordinates of the skeleton key points corresponding to the pedestrian target in each detection frame, the depth image and the intrinsic parameter matrix of the image acquisition device; The second determination module is used to determine whether each pedestrian target invades the target space electronic fence area based on the three-dimensional coordinates of the skeletal key points of each detection frame and / or each pedestrian target; wherein the target space electronic fence is determined based on the depth image and the preset area to be determined.

Images