Method and system for detecting pedestrian intrusion in mine environment

Abstract

A method and system for detecting pedestrian intrusion in a mine environment comprises four stages of initialization, pedestrian recognition, pedestrian detection and intrusion judgment, a deep learning model is applied in the pedestrian recognition and pedestrian detection stages, and a key point judgment method is adopted in the intrusion judgment stage. The initialization stage is used for setting and comprises the steps of defining an electronic fence, selecting a key point type and setting a maximum detection number. The pedestrian recognition stage is used for screening, and enters the pedestrian detection stage after pedestrians are recognized in video frames. And the pedestrian detection stage is used for positioning, returning to the pedestrian recognition stage if no pedestrian can be detected when the maximum detection number is exceeded, and otherwise, entering the intrusion judgment stage. In the intrusion judgment stage, whether pedestrians intrude into the electronic fence is judged according to whether the key points are in the electronic fence, and an alarm is given if the pedestrians intrude into the electronic fence. According to the method and the system, detection is carried out after identification, pedestrian-free video segments are quickly processed by using a simple identification model, the calling times of a complex detection model are reduced, and the consumption of computing resources is reduced.

Description

Technical Field

[0001] The present invention relates to the field of intelligent monitoring, and in particular to a method and system for detecting pedestrian intrusion in a mine environment. Background Art

[0002] Pedestrian intrusion detection is a security monitoring technology that uses various sensors to collect data, and uses various types of data processing technologies and pedestrian identification and detection technologies to determine the type and location of pedestrians to prevent targets from entering specific areas. In the working environment of mines, pedestrian intrusion detection technology is of great significance for ensuring the safety of miners and improving work efficiency. Traditional manual detection methods mainly involve watching surveillance videos or on-site inspections, which not only consume a lot of manpower and time, but also cause workers to become slack due to long-term monitoring and inspections, resulting in false detections and serious consequences. Although intelligent security monitoring technologies have been emerging with the development of deep learning technology, there are very few image-based pedestrian intrusion detections suitable for mine working environments. The main reason is that the complex environment of mines, such as dim vision, strong light pollution, and diffuse mine dust, poses a major challenge to pedestrian intrusion detection. Summary of the invention

[0003] Based on this, it is necessary to provide a method for detecting pedestrian intrusion in a mine environment that can improve detection performance.

[0004] At the same time, a system for detecting pedestrian intrusion in a mine environment is provided, which can improve the detection performance.

[0005] A method for detecting pedestrian intrusion in a mine environment, comprising:

[0006] The initialization phase includes:

[0007] Draw an electronic fence: Draw an electronic fence in the video scene;

[0008] Select key point type: Select the key point type suitable for the scene;

[0009] The pedestrian recognition stage includes:

[0010] Extracting the current video frame, including extracting and identifying the video frame: extracting the current moment frame image from the acquired video stream;

[0011] Preprocessing, including pedestrian recognition preprocessing: preprocessing the frame image;

[0012] Pedestrian recognition model reasoning: The preprocessed image data is input into the pedestrian recognition model with low computational complexity for pedestrian recognition, and the reasoning calculation is performed to output the classification label;

[0013] Determine the recognition result: If the classification label output by the pedestrian recognition model is a pedestrian, then enter the pedestrian detection stage, otherwise continue with pedestrian recognition;

[0014] The pedestrian detection stage includes:

[0015] Set the number of detections: Set the calculation variable n, which is used to record the number of pedestrian detections;

[0016] Pedestrian detection model inference: The image data is input into the pedestrian detection model with high computational complexity, and the target bounding box of the current video frame is obtained through inference calculation;

[0017] Determine the detection result: If the pedestrian detection model outputs a target bounding box, it enters the intrusion judgment stage, otherwise it records the number of detections;

[0018] Enter the judgment phase, which includes:

[0019] Calculate key points: Calculate the key point positions in the target bounding box according to the selected key point type;

[0020] Determine key points: Determine whether the key points are within the designated electronic fence. If the key points are within the designated electronic fence, an alarm is triggered. If the key points are not within the designated electronic fence, the pedestrian detection phase is entered.

[0021] Alarm: Issue prompts or warnings to pedestrian targets, or send an alarm to the central console.

[0022] In a preferred embodiment, the initialization stage further includes setting a maximum number of detections: setting a variable M to be the maximum number of calls to the pedestrian detection model;

[0023] The pedestrian detection stage also includes:

[0024] Record the number of detections: If the pedestrian detection model does not output a bounding box, the counting variable n is increased by 1, and the number of pedestrian detections increases by one;

[0025] Determine the number of detections: Determine whether the counting variable n is less than the maximum detection number M. If the counting variable n is less than the maximum detection number, pedestrian detection is performed, otherwise the pedestrian recognition stage is entered.

[0026] In a preferred embodiment, the extracting the current video frame further comprises extracting the detection video frame: extracting the current moment frame image from the acquired video stream;

[0027] The pedestrian recognition preprocessing includes: image scaling and pixel value standardization;

[0028] The preprocessing includes: pedestrian detection preprocessing set at the pedestrian detection stage and before the pedestrian recognition model inference: image scaling and pixel value normalization;

[0029] The image scaling is proportional scaling, which includes: selecting the longest side of the height and width of the image; dividing the original target side length by the longest side of the image to calculate the scaling factor; multiplying the scaling factor by the shortest side of the image to calculate the target length of the shortest side of the image; scaling the image to scale the longest side of the image to the original target side length, and scaling the shortest side of the image to the calculated target length; subtracting the target length of the shortest side from the original target side length and dividing by 2 to calculate the width of the image edge to be filled; filling the image edge with a fixed pixel value so that the filled image becomes a square image with the original target side length;

[0030] The pixel value normalization includes: calculating the mean and standard deviation of each channel of all images in the training set, or directly using the pre-calculated pixel mean and standard deviation of the pre-training set; subtracting the pixel mean from each pixel of all images in the training set and dividing it by the pixel standard deviation;

[0031] The pixel value normalization divides all pixels in the image by 255 to calculate a normalized pixel value, thereby adjusting the image data to a value range from 0 to 1.

[0032] In a preferred embodiment, the preprocessing further includes: data enhancement, which is used in the image preprocessing stage of the training and testing stages of the pedestrian recognition model and the pedestrian detection model. The data enhancement includes: one or more of random grayscale conversion, random haze synthesis, and random spot addition data enhancement; the data enhancement of the random grayscale conversion includes:

[0033] Generate weight vector: Randomly generate a three-channel weight vector [w R ,w G ,w B ], which satisfies w R +w G +w B =1, which can be expressed as

[0034]

[0035] in, represents uniform distribution, a and b are the lower and upper limits of uniform distribution respectively;

[0036] Generate weight matrix: with three-channel weight w R ,w G ,w B is the mean, with random number std W is the standard deviation, and randomly generates three-channel weight matrices W R ,W G ,W B, expressed mathematically as

[0037]

[0038] in, Represents normal distribution, μ, σ are the mean and standard deviation of normal distribution, std W According to statistics, it is set as a random number between 0 and 1, expressed as a mathematical formula:

[0039]

[0040] Calculate the noise-free grayscale image: Convert the color image to a grayscale image Gray, the calculation formula is

[0041] Gray=W R *Red+W G *Green+W B *Blue,

[0042] Among them, Red, Green, and Blue are three-channel images of color images;

[0043] Find the maximum value: Find the maximum pixel value gray of the grayscale image Gray at this time max and minimum pixel value gray min , expressed mathematically as

[0044]

[0045] Among them, x, y represent the row and column index of the image, that is, the pixel coordinates of the image, and max x,y and min x,y Respectively represent the maximum and minimum values ​​at all pixel coordinates (x, y);

[0046] Generate a proportional matrix: take 1 as the mean and random number std S is the standard deviation, and randomly generates a three-channel ratio matrix S R ,S G ,S B , expressed mathematically as

[0047]

[0048] Among them, std S According to statistics, it is set as a random number between 0 and 0.03, which is expressed as a mathematical formula:

[0049]

[0050] Calculate the noisy grayscale image: Calculate the three-channel grayscale image Gray with noise deviation new , the calculation formula is

[0051]

[0052] Among them, Red new ,Green new ,Blue new They are new grayscale images Gray new Three-channel image of;

[0053] Standardize grayscale image: new Standardize according to the maximum pixel value gray max and minimum pixel value gray min Using the maximum and minimum value normalization, the calculation formula is:

[0054]

[0055] Normalize grayscale image: Grayscale image pixels Gray std (x,y) is rounded and expressed as

[0056] Gray rnd (x,y)=round(Gray std (x,y)),

[0057] Among them, round(·) represents the rounding operation, and then the grayscale image pixel Gray rnd The range of (x,y) is limited to between 0 and 255, which can be expressed as

[0058]

[0059] The random haze synthesis includes:

[0060] Random generation parameters: Randomly generate atmospheric light intensity A, whose value range is between 0 and 255, expressed by mathematical formula:

[0061] A=round(a), The attenuation coefficient β is randomly generated, and the value range is set between 0 and 0.1, which can be expressed as follows:

[0062]

[0063] Randomly select a pixel coordinate in the image as the center of the haze (X H ,Y H ), expressed mathematically as

[0064]

[0065] Among them, Width and Height are the width and height of the haze-free image J respectively;

[0066] Determine the farthest pixel: Determine the distance from the center of the haze (X H ,Y H )The farthest pixel coordinate (X F ,Y F ), for a rectangular image, the farthest pixel coordinate (X F ,Y F ) must be one of the four vertex coordinates of the image. If X H Greater than Width / 2, then X F is 0, otherwise X F For Width, if Y F Greater than Height / 2, then X F is 0, otherwise X F is Height, which can be expressed as

[0067]

[0068] Calculate the size of the haze: Calculate the center of the haze (X H ,Y H ) and the farthest pixel coordinate (X F ,Y F )’s Euclidean distance dist F , the calculation formula is

[0069]

[0070] According to the farthest distance dist F Set the haze size, the calculation formula is:

[0071]

[0072] Calculate the distance matrix: Calculate the distance from each pixel (x, y) in the image to the center of the haze (X H ,Y H ) is calculated as follows:

[0073]

[0074] Use the above distance matrix dist to generate haze spreading from the center to the surrounding areas. If we use the inverted distance matrix dist R , then the haze is generated from the surrounding to the center, and the inversion distance of each pixel is dist R (x,y) is determined by the maximum distance dist F Subtract the original distance dist(x,y) to obtain the calculation formula:

[0075] dist R (x,y)=dist F -dist(x,y),

[0076] Calculate the transparency matrix: Calculate the propagation distance d(x,y) of the reflected light of each pixel in the haze. The calculation formula is

[0077] d(x,y)=size-0.04*dist(x,y),

[0078] Then calculate the transparency of each pixel t(x,y), the calculation formula is

[0079] t(x,y)=e -β *d(x ,y) ,

[0080] Where d(x,y) is the distance that the reflected light corresponding to each pixel (x,y) in the image passes through the haze to reach the image sensor, and β is called the attenuation coefficient;

[0081] Calculate the haze image: First normalize the atmospheric light intensity A and image pixel J(x,y), the calculation formula is:

[0082]

[0083] Then calculate the normalized haze image pixel I(x,y) nml , the calculation formula is

[0084] I(x,y) nml =I(x,y) nml *t(x,y)+A nml *(1-t(x,y)), finally inverse normalize the haze image pixel I(x,y) nml , the calculation formula is

[0085] I(x,y)=I(x,y) nml *255.0;

[0086] Normalize the haze image: Round off the haze image pixel I(x,y) and express it as follows:

[0087] I(x,y) rnd =round(I(x,y)),

[0088] Then the range of pixel I(x,y) is limited to between 0 and 255, which can be expressed as

[0089]

[0090] The random light spot addition includes:

[0091] Random generation parameters: Randomly generate the center of the spot (X C ,Y C ), the value range is within the image size (Width, Height), and is expressed as

[0092]

[0093] The spot radius R is randomly generated, and the value range is set between 20 and 100, which can be expressed as follows:

[0094] R = round(r),

[0095] The randomly generated spot brightness L is set between 220 and 235, which can be expressed as follows:

[0096] L = round(l),

[0097] The blur size sigma is randomly generated, and the value range is set between one-quarter of the spot radius and one-half of the spot radius, which can be expressed as

[0098]

[0099] Set the overlay weight: Set the light spot effect image Effect fh The superposition weight Weight is set to 0.6, and then the spot brightness L before superposition is calculated using the superposition weight Weight PRE , the calculation formula is

[0100] L PRE =Weight*L,Weight=0.6;

[0101] Initialize effect image: Generate the initial spot effect image Effect init , initialize the effect image to an all-zero matrix of the same size as the original image Image, for the spot effect image Effect init For each pixel (x,u) in

[0102] Effect init (x,y)=0;

[0103] Generate a light spot effect: Create a light spot effect image Effect init Generate a circular light spot, the center of which is (X C ,Y C ), the radius is R, and the pixel value inside the circle is L PRE , expressed mathematically as

[0104]

[0105] Generate halo effect: Use Gaussian blur to generate the halo around the spot. The standard deviation of Gaussian blur is the blur size sigma. The filter size ksize′ of Gaussian blur is calculated based on the standard deviation sigma. The calculation formula is:

[0106] ksize′=round(6*sigma+1),

[0107] Ensure that the filter size ksize is a positive odd number. If the remainder of ksize′ divided by 2 is 0, it means that ksize′ is an even number, so add 1 to it to make it an odd number; if the remainder of ksize′ divided by 2 is 1, it means that ksize′ is an odd number, so keep it as the original value, which can be expressed as the mathematical formula:

[0108]

[0109] Among them, x mod 2 means to find the remainder of x divided by 2, sigma is the standard deviation, ksize is the filter size, and the spot effect image Effect f Perform Gaussian blur, expressed as

[0110] Effect fh =GaussianBlur(Effect f ,ksize,sigma),

[0111] Calculate the mask image: Calculate the mask of the spot area. First, generate an image with the spot effect effect. fh The initial mask of the same size, for each pixel in the mask Mask, now has

[0112] Mask(x,y)=0,

[0113] For the light spot effect image Effect fh For each pixel in, if the pixel value Effect fh If (x, y) is greater than 0, the pixel value at the corresponding position in the mask is set to 1, otherwise it is set to 0, which can be expressed as

[0114]

[0115] Overlay effect image: Overlay light spot effect image Effect fh The superposition method of these two images is weighted summation. The weight of the original image is fixed at 1.0. The spot effect image is fhThe weight is Weight, and the result image after adding the spot effect is Result. The calculation formula is

[0116] Result = Image * 1.0 + Effect fh *Weight;

[0117] Normalize the result image: Round the result image pixels Result(x,y) to the nearest integer, which can be expressed as

[0118] Result rnd (x,y)=round(Result(x,y)),

[0119] Then the pixel Result rnd The range of (x,y) is limited to between 0 and 255, which can be expressed as

[0120]

[0121] Limit the brightness of the light spot: limit the maximum pixel value of the light spot area to the light spot brightness L. When the pixel value of the mask Mask(x,y) is 1, it means the Result at the corresponding position clip (x,y) is in the spot area, then judge the Result clip Is the pixel value of (x, y) greater than L? If yes, change the pixel value of the position to L, otherwise keep the original pixel value; when the pixel value of Mask(x, y) is 0, it means the Result of the corresponding position clip (x, y) is not in the spot area, no additional changes are made, and the mathematical formula is expressed as

[0122]

[0123] Among them, min(·) represents the minimum value operation.

[0124] In a preferred embodiment, the pedestrian detection model reasoning includes: extracting features from the input image, outputting feature maps of different sizes, predicting the target bounding box on the feature map, and in the feature extraction process, using a convolutional neural network to iteratively calculate the feature map through a convolutional layer and a sub-layer, reducing the size of the feature map, increasing the number of channels, and outputting multiple feature maps of different resolutions that are reduced in size ratio to the original image. Pre-setting multiple anchor frames with different aspect ratios and no center position for each feature map according to the size, taking each pixel position of each feature map as the target center, and predicting different target bounding boxes based on different anchor frames.

[0125] In a preferred embodiment, the predicted target bounding box is a parameter of the predicted target bounding box, including a normalized center offset x i ,yi , anchor frame scaling ratio w i ,h i , target confidence p conf.i , category probability p cls1.i ,p cls2.i ,..., where i is the bounding box number of the detection layer,

[0126] Let the number of categories to be detected be nc=C, then the number of parameters for each bounding box is no=5+nc, the height and width of the feature map in the detection layer are ny,nx respectively, the number of anchor boxes is na=3, the number of bounding boxes in this detection layer is na*ny*nx=N, and the number of bounding box parameters is na*ny*nx*no, then all target bounding boxes pred predicted by this detection layer are

[0127]

[0128] In the above formula, the target confidence p conf.N Indicates the probability that there is a target in the bounding box N, p clsC.N represents the probability that the target in the bounding box N is of the Cth category,

[0129] The target center of the target bounding box is calculated based on the center offset. The target bounding box is generated at each pixel position of the feature map. The target center of each bounding box is the pixel coordinate when the bounding box is generated. The feature map is obtained by downsampling the input image. The reduced bounding box is detected from the feature map, and the bounding box is enlarged to a size suitable for the input image. The center offset is used to make it accurate to a decimal.

[0130] The internal coordinate axes of the grid are the pixel coordinates of the target center (grid X ,grid Y ) as the origin, with the right and downward directions of the positive X and Y axes respectively, to further determine the exact position of the target center (bbox X ,bbox Y ), center bias X ,bias Y ) is the target center (bbox X ,bbox Y ) in the relative position of the coordinate axis inside the grid, which will be the same as the center coordinate (bbox X ,bbox Y )The two adjacent pixel coordinates (bbox X ±1,bbox Y ±1) is also considered as the target center, center bias X ,bias Y ) changes from [0,1.0] to [-0.5,1.5], and the predicted normalized center bias xi ,y i The value range is [0,1.0]. To get the real center bias X ,bias Y ) expands its value range, and the calculation formula is

[0131]

[0132] Then find the target center of the bounding box (bbox X ,bbox Y ), the calculation formula is

[0133]

[0134] In the above formula, scale is the magnification of the bounding box, which is obtained according to the reduction factor of the feature map in the detection layer compared to the original input image.

[0135] The size of the target bounding box is calculated based on the preset anchor box. The anchor box is designed according to the size of the feature map in the detection layer. The larger the size of the feature map in the detection layer, the larger the size of the anchor box used. The anchor box is calculated, and the sizes of all target bounding boxes in the training set are clustered to generate a set of optimal anchor box sizes. During target detection, the preset anchor box is modified by size scaling to make it approach the real target bounding box, and the anchor box scaling ratio w is predicted. i ,h i , the value range is [0,1.0], and the size of the anchor box used by the bounding box i is anchor W ,anchor H , then the size of the target bounding box bbox W ,bbox H The calculation formula is

[0136]

[0137] From the above formula, we can see that the width and height of the target bounding box bbox W ,bbox H The maximum value of the anchor box width and height is anchor W ,anchor H 4 times,

[0138] Determine the target category and target confidence p conf.i Used to determine whether there is a target in the bounding box i, the target confidence p conf.iIf the value is greater than the confidence threshold, preferably 0.5, it means that there is a target in the bounding box, otherwise it means that there is no target in the bounding box. The target refers to the target within the detection category range. The target outside the detection category range is regarded as the background by the pedestrian detection model. The pedestrian detection model predicts the series category probability p clsN.i ,p cls2.i ,..., is used to determine the category to which target i belongs. When there is a target in the bounding box i, the category corresponding to the largest category probability is selected as the category to which the target belongs.

[0139] In a preferred embodiment, the pedestrian detection model inference uses non-maximum suppression as a post-processing algorithm. Multiple bounding boxes will be generated at the same target position, while the same target has only one bounding box. The predicted bounding boxes will overlap with each other and there will be redundancy. After the bounding boxes are predicted, non-maximum suppression is performed on these bounding boxes. Non-maximum suppression suppresses redundant bounding boxes and finds the best target bounding box, which specifically includes:

[0140] Set threshold: Set the confidence threshold Thres Conf Intersection and union ratio threshold Thres IOU ;

[0141] Filter: Sort all bounding boxes according to the target confidence, and discard those whose target confidence is less than the confidence threshold Thres Conf The bounding box of

[0142] Best selection: select the bounding box with the highest target confidence and use it as the best bounding box;

[0143] Discard: Calculate the intersection and union ratio of the best bounding box and other bounding boxes, and discard those whose intersection and union ratio is greater than the intersection and union ratio threshold Thres IOU Other bounding boxes of

[0144] Repeat processing: Repeat the optimal selection step and the discarding step until all bounding boxes are processed.

[0145] In a preferred embodiment, the target bounding box is a rectangular bounding box whose sides are parallel to the four sides of the image and is represented by four parameters. The bounding box has a confidence and a category label associated with it. The confidence is assigned by the pedestrian detection model, and the category label is predicted by the pedestrian detection model, indicating the type of the target in the bounding box. The bounding box is the bounding box of the target in the image. When the target in the image is reduced or enlarged, the bounding box is proportionally reduced or enlarged. When annotating the image, determine the upper left corner and lower right corner of the target boundary to mark a target. There are two ways to represent the target bounding box. One is the center point plus width and height representation (x, y, w, h), (x, y) is the center point of the bounding box, w, h are the width and height of the bounding box; the other is the coordinate representation (x 1 ,y 1 ,x2 ,y 2 ), (x 1 ,y 1 ) is the upper left corner of the bounding box, (x 2 ,y 2 ) is the lower right corner of the bounding box. The two bounding box representation methods can be converted to each other. The calculation formula for converting the center point plus width and height representation method to the coordinate representation method is:

[0146]

[0147] The calculation formula for converting the coordinate representation to the center point plus width and height representation is:

[0148]

[0149] In the step of calculating the key points, when the pedestrian in the scene is standing or walking, the midpoint of the lower boundary of the target bounding box is selected as the key point; when it is necessary to prevent the pedestrian from climbing into or rolling into the alarm area, the midpoint of the upper boundary of the target bounding box is selected as the key point; when it is necessary to prevent the pedestrian from accidentally falling into or rolling into the alarm area, multiple key points are selected at different positions of the target bounding box, such as the midpoints of the upper, lower, left and right boundaries of the target bounding box can be selected as the key points.

[0150] If the key point is located at the midpoint of the lower boundary of the bounding box, the key point coordinates (x K ,y K ) is calculated as

[0151]

[0152] Similarly, if the key point is located at the midpoint of the upper boundary of the bounding box, the key point coordinates (x K ,y K ) is calculated as

[0153]

[0154] If the key point is located at the middle of the left boundary of the bounding box, the key point coordinates (x K ,y K ) is calculated as

[0155]

[0156] If the key point is located at the middle of the right boundary of the bounding box, the key point coordinates (x K ,y K ) is calculated as

[0157]

[0158] In a preferred embodiment, the pedestrian recognition model and the pedestrian detection model adopt a deep learning algorithm of a convolutional neural network, including a convolution layer, a downsampling layer, and an activation function. The pedestrian recognition includes: extracting features from the input image to obtain multiple feature maps of different sizes, then compressing these two-dimensional feature maps into one-dimensional features, and finally using a fully connected layer to predict the category probability of the image; for each category to be recognized in the pedestrian recognition model, the output layer has a corresponding category probability, and the value range of all category probabilities is [0,1]. The model takes the category corresponding to the maximum category probability as the predicted image category.

[0159] In the pedestrian recognition step, the pedestrian recognition model uses 0.4-0.5 as the decision threshold. If the model prediction value is greater than or equal to the decision threshold, the classification label is assigned to 1, and if it is less than the decision threshold, the classification label is assigned to 0.

[0160] In the judgment of the recognition result: if the classification label is 1, it is considered that there is a suspected pedestrian target in the current frame image, and a pedestrian detection model with large computational complexity is started to perform pedestrian detection; if the classification label is 0, the pedestrian recognition model is continued.

[0161] A system for detecting pedestrian intrusion in a mine environment, comprising:

[0162] Initialization module, including:

[0163] Electronic fence drawing unit: draw an electronic fence in the video scene;

[0164] Select key point type unit: Select the key point type suitable for the scene;

[0165] Pedestrian recognition module, including:

[0166] Extracting the current video frame unit, including extracting and identifying the video frame unit: extracting the current moment frame image from the collected video stream;

[0167] A preprocessing unit, including a pedestrian recognition preprocessing unit: preprocessing the frame image;

[0168] Pedestrian recognition model inference unit: inputs the preprocessed image data into the pedestrian recognition model with low computational complexity for pedestrian recognition, performs inference calculation and outputs classification labels;

[0169] Identification result judgment unit: If the classification label output by the pedestrian recognition model is a pedestrian, then the pedestrian detection phase is entered, otherwise pedestrian recognition continues;

[0170] Pedestrian detection module, including:

[0171] Set the detection number unit: set the calculation variable n, which is used to record the number of pedestrian detections;

[0172] Pedestrian detection model inference unit: inputs image data into the pedestrian detection model with large computational complexity, and obtains the target bounding box of the current video frame through inference calculation.

[0173] Detection result judgment unit: If the pedestrian detection model outputs a target bounding box, it enters the intrusion judgment module, otherwise the number of detections is recorded;

[0174] The intrusion judgment module includes:

[0175] Calculate key point unit: calculate the key point position in the target bounding box according to the selected key point type;

[0176] Key point judgment unit: judge whether the key point is in the designated electronic fence. If the key point is in the designated electronic fence, it will enter the alarm unit; if the key point is not in the designated electronic fence, it will enter the pedestrian detection module;

[0177] Alarm unit: issues prompts or warnings to pedestrian targets, or sends an alarm to the central console.

[0178] The method and system for detecting pedestrian intrusion in the above-mentioned mine environment adopt the method of first identifying and then detecting, and use a simple recognition model to quickly process the video segments without pedestrians, reduce the number of calls to complex detection models, and reduce the consumption of computing resources; the intrusion judgment method based on key points determines whether the pedestrian target has intruded by judging whether the key points in the pedestrian encirclement box are within the electronic fence, effectively reducing the misjudgment of intrusion caused by perspective projection, and the key point judgment method is used to improve the accuracy and robustness of intrusion detection.

[0179] In addition, in view of the complex environment of mines, the pedestrian recognition model and the pedestrian detection model use the data enhancement algorithm proposed in the present invention during the training and testing stages. Random grayscale conversion: In view of the dim field of vision in mines, the present invention proposes a data enhancement algorithm for random grayscale conversion, which enhances the network model's recognition or detection capabilities for night grayscale images by simulating grayscale images in night photography mode. Random haze synthesis: In view of the diffuse dust in mines, the present invention proposes a data enhancement algorithm for random haze synthesis to enhance the network model's ability to resist haze interference. Random light spot addition: In view of the strong light pollution in mines, the present invention proposes a data enhancement algorithm for random light spot addition to enhance the network model's ability to resist light spot interference. BRIEF DESCRIPTION OF THE DRAWINGS

[0180] Figure 1 A partial flow chart of a method for detecting pedestrian intrusion in a mine environment according to an embodiment of the present invention;

[0181] Figure 2 A schematic diagram of pedestrian intrusion detection according to an embodiment of the present invention;

[0182] Figure 3 The figure is a schematic diagram of the training and testing process of a pedestrian recognition model and a pedestrian detection model according to an embodiment of the present invention. DETAILED DESCRIPTION

[0183] like Figure 1 As shown, a method for detecting pedestrian intrusion in a mine environment according to an embodiment of the present invention includes:

[0184] Initialization phase, initialization of some settings needed for subsequent phases, including

[0185] Step S101, defining an electronic fence: defining an electronic fence in a video scene;

[0186] Step S103, select key points: select a key point type suitable for the scene;

[0187] Step S105, setting the maximum number of detections: setting the maximum number of detections M, i.e. the maximum number of calls to the pedestrian detection model; by setting this variable, the duration of the pedestrian detection phase can be indirectly changed;

[0188] The main task of the pedestrian recognition stage is to filter video segments for the pedestrian detection stage and notify the pedestrian detection stage that there are pedestrian targets in the current video scene, including:

[0189] Step S107, extracting and identifying video frames: extracting the current moment frame image from the collected video stream;

[0190] Step S109, pedestrian recognition preprocessing: preprocessing the image, including image scaling and pixel value standardization;

[0191] Step S111, pedestrian recognition model reasoning: input the preprocessed image data into a small pedestrian recognition model, perform pedestrian recognition based on deep learning, and perform reasoning calculation to output a classification label; the pedestrian recognition model uses 0.4-0.5 as a decision threshold, and if the model prediction value is greater than or equal to the decision threshold, the classification label is classified as 1, and if it is less than the decision threshold, the classification label is classified as 0;

[0192] Step S113, judging the recognition result: judging whether the pedestrian recognition model outputs a pedestrian classification label. If the classification label is 1, it is considered that there is a suspected pedestrian target in the current frame image, and the pedestrian detection stage is entered; if the classification label is 0, there is no pedestrian target, and pedestrian recognition continues;

[0193] The pedestrian detection stage is the core part of the pedestrian intrusion detection algorithm of the present invention. The main task of this stage is to determine the position of the pedestrian in the video frame and predict the pedestrian bounding box; it includes:

[0194] Step S115, setting the number of detections: setting a counting variable n to 0, the counting variable records the number of pedestrian detections, that is, the number of calls to the pedestrian detection model;

[0195] Step S117, extracting the detection video frame: extracting the frame image at the current moment from the collected video stream;

[0196] Step S119, pedestrian detection preprocessing: preprocessing the image, including image scaling and pixel value normalization;

[0197] Step S121, pedestrian detection model reasoning: input the image data into the pedestrian detection model, perform reasoning calculation based on deep learning through the pedestrian detection model, and obtain the target bounding box of the current video frame after reasoning calculation; this step is the key to the pedestrian intrusion judgment algorithm, and the calculation amount of the pedestrian detection model is greater than that of the pedestrian recognition model; the calculation amount of this embodiment is relative, which is the relative size of the pedestrian detection model with respect to the pedestrian recognition model;

[0198] Step S123, judging the detection result: judging whether the pedestrian detection model outputs a target bounding box. If the pedestrian detection model outputs a target bounding box, it is considered that there is a pedestrian target in the current frame image, and the intrusion judgment stage is entered. If the pedestrian detection model does not output a target bounding box, the number of detections is recorded.

[0199] Step S125, recording the number of detections: adding 1 to the counting variable n, indicating that the number of pedestrian detections increases by one;

[0200] Step S127, determining the number of detections: determining whether the calculated variable n is less than the preset maximum number of detections M, if the calculated variable n is less than the preset maximum number of detections M, then performing pedestrian detection, if not, then returning to the pedestrian recognition stage;

[0201] In the intrusion judgment stage, the main task is to determine whether the pedestrian has intruded into the electronic fence. The judgment method adopts the key point judgment method, which includes:

[0202] Step S129, calculating key points: calculating the key point positions in the target bounding box according to the selected key point type;

[0203] Step S131, determining the key point: determining whether the key point is in the electronic fence; if the key point is in the electronic fence, an alarm is issued; if the key point is not in the electronic fence, the system returns to the pedestrian detection stage;

[0204] Step S133, alarm: issuing a prompt or warning to the pedestrian target, or sending an alarm to the central control console.

[0205] The algorithm flow of the present invention is to perform pedestrian recognition before pedestrian detection. Specifically, in the algorithm flow of the intrusion detection program, a simple recognition model is added before the complex detection model. The program first uses a pedestrian recognition model with small computational complexity and high speed to determine whether there is a pedestrian target in the current video. If there is, the pedestrian detection model with large computational complexity and higher accuracy is called to further determine the specific location of the pedestrian target. In fact, after adding the pedestrian recognition model, the pedestrian intrusion detection algorithm flow of the present invention can be divided into three stages: pedestrian recognition, pedestrian detection and intrusion judgment. The program first enters the pedestrian recognition stage and enables the pedestrian recognition model. The main task of this stage is to filter out the video segments with pedestrian targets. When the pedestrian recognition model outputs the pedestrian classification label, the program enters the pedestrian detection stage and enables the pedestrian detection model. The main task of this stage is to determine the location of the pedestrian target. When the pedestrian detection model outputs the pedestrian enclosing box, the program enters the intrusion judgment stage and enables the key point judgment method. The main task of this stage is to determine whether the pedestrian has intruded into the electronic fence. In addition, in the pedestrian detection stage, the algorithm sets the upper limit of the pedestrian detection model, that is, the maximum number of detections M, which indirectly sets the duration of pedestrian detection. When the pedestrian detection model continuously detects M frames without outputting a pedestrian bounding box, the program returns to the pedestrian recognition stage.

[0206] The key point judgment method adopted by the present invention, that is, the intrusion judgment method based on key points, needs to select one or more appropriate key points according to the application scenario and actual requirements before calculating the key point position. For example, when the pedestrians in the scene are usually standing or walking, the midpoint of the lower boundary of the pedestrian enclosing box can be selected as the key point. When it is necessary to prevent pedestrians from climbing into or rolling into the alarm area, the midpoint of the upper boundary of the pedestrian enclosing box can be selected as the key point. When it is necessary to prevent pedestrians from accidentally falling into or rolling into the alarm area, multiple key points can be selected at different positions of the enclosing box, such as all (up, down, left, and right) boundary midpoints of the enclosing box can be selected as key points. The traditional intersection judgment method is prone to false intrusion detection because it does not take into account the problem of object overlap due to perspective projection. When projecting from three-dimensional space to a two-dimensional image, two targets will overlap with each other in the image even if they are at different distances in space. The key point judgment method used in the present invention can avoid misjudgment in such situations. Refer to Figure 2 , because the pedestrian bounding box intersects with the electronic fence at this time, the intrusion detection program using the intersection judgment method will mistakenly believe that the pedestrian in the image has entered the electronic fence. On the contrary, since the key point at the bottom of the pedestrian bounding box is not in the electronic fence at this time, the intrusion detection program using the key point judgment method will not have such a false alarm.

[0207] In view of the problem that the existing technology occupies a lot of computing resources and has a long delay time, the present invention proposes a program solution of first recognition and then detection. A pedestrian recognition part with a smaller computational workload is added before the pedestrian detection part with a larger computational workload. The solution uses a simple recognition model to quickly process video segments without pedestrians, reduces the number of calls to complex detection models, and reduces the consumption of computing resources.

[0208] The present invention aims to solve the problem that the intrusion judgment method in the prior art is prone to misjudgment, and provides an intrusion judgment method based on key points. The method judges whether a pedestrian target has intruded by judging whether the key points in the pedestrian encirclement box are within the electronic fence. Compared with the intrusion judgment method based on intersection, the key point judgment method has high flexibility and strong adaptability. It can not only simplify the intrusion judgment logic, but also avoid misjudgment caused by pedestrian targets blocking the electronic fence, and effectively improve the accuracy and robustness of intrusion detection.

[0209] The preprocessing of the present invention also includes: data enhancement. The data enhancement of the present invention includes: data enhancement of random grayscale conversion, random haze synthesis, and random light spot addition.

[0210] The data enhancement of random grayscale conversion in the present invention is proposed for the dim vision in the mine. The camera used in the present invention will enable the night photography mode when the light is insufficient, and output grayscale images instead of color images. Therefore, the model needs to input a certain amount of grayscale images during training to adapt to the night photography mode of the camera and reduce the dependence on image color.

[0211] The data enhancement algorithm of random grayscale conversion of the present invention enhances the recognition or detection ability of the network model for night grayscale images. The steps of data enhancement of random grayscale conversion are as follows:

[0212] Generate weight vector: Randomly generate a three-channel weight vector [w R ,w G ,w B ], which satisfies w R +w G +w B =1, which can be expressed as

[0213]

[0214] in, represents uniform distribution, a and b are the lower and upper limits of uniform distribution respectively;

[0215] Generate weight matrix: with three-channel weight w R ,w G ,w B is the mean, with random number std W is the standard deviation, and randomly generates three-channel weight matrices WR ,W G ,W B , expressed mathematically as

[0216]

[0217] in, Represents normal distribution, μ, σ are the mean and standard deviation of normal distribution, std W According to statistics, it is set as a random number between 0 and 1, expressed as a mathematical formula:

[0218]

[0219] Calculate the noise-free grayscale image: Convert the color image to a grayscale image Gray, the calculation formula is

[0220] Gray=W R *Red+W G *Green+W B *Blue,

[0221] Among them, Red, Green, and Blue are three-channel images of color images;

[0222] Find the maximum value: Find the maximum pixel value gray of the grayscale image Gray at this time max and minimum pixel value gray min , expressed mathematically as

[0223]

[0224] Among them, x, y represent the row and column index of the image, that is, the pixel coordinates of the image, and max x,y and min x,y Respectively represent the maximum and minimum values ​​at all pixel coordinates (x, y);

[0225] Generate a proportional matrix: take 1 as the mean and random number std S is the standard deviation, and randomly generates a three-channel ratio matrix S R ,S G ,S B , expressed mathematically as

[0226]

[0227] Among them, std S According to statistics, it is set as a random number between 0 and 0.03, which is expressed as a mathematical formula:

[0228] Calculate the noisy grayscale image: Calculate the three-channel grayscale image Gray with noise deviationnew , the calculation formula is

[0229]

[0230] Among them, Red new ,Green new ,Blue new They are new grayscale images Gray new Three-channel image of;

[0231] Standardize grayscale image: Grayscale image Gray new Standardize according to the maximum pixel value gray max and minimum pixel value gray min _Use the maximum and minimum values ​​for standardization, and the calculation formula is:

[0232]

[0233] Normalize grayscale image: Grayscale image pixels Gray std (x,y) is rounded and expressed as

[0234] Gray rnd (x,y)=round(Gray std (x,y)),

[0235] Among them, round(·) represents the rounding operation, and then the grayscale image pixel Gray rnd The range of (x,y) is limited to between 0 and 255, which can be expressed as

[0236]

[0237] The random haze synthesis algorithm of the present invention is proposed for the situation where mine dust is diffused in mines. During the mining and excavation operations in mines, the cutting and crushing of ore rocks will generate a large amount of dust. In addition, the underground mining space is small and the ventilation conditions are poor, which causes the mine dust to remain in the working environment for a long time to form haze, causing significant obstacles and interference to pedestrian intrusion detection. The data enhancement algorithm of the random haze synthesis of the present invention enhances the ability of the network model to resist haze interference. The algorithm principle of random haze synthesis is described as follows.

[0238] The scattering effect of particles in the atmosphere is the main cause of haze. When light propagates in the atmosphere, due to the scattering of tiny particles and gas molecules in the air, the light intensity will attenuate as the propagation distance increases. According to Lambert-Beer's law, the light intensity in the atmosphere can be expressed as

[0239] I(d)=I 0 e -βd ,

[0240] Among them, I 0 is the initial light intensity, d is the distance the light travels, and β is the scattering coefficient, which describes the scattering ability of the medium to light of different wavelengths.

[0241] In the atmospheric scattering model, the light source received by the image sensor on a hazy day mainly comes from two parts: one is the light reflected by the object, which passes through the haze and reaches the image sensor after being attenuated by particles; the other is the atmospheric light formed by sunlight scattered by particles. Therefore, the haze image can be regarded as a synthesis of two images: a haze-free image and a pure haze image. The synthesis formula of the haze image based on the atmospheric scattering model is:

[0242] I(x,y)=J(x,y)*t(x,y)+A*(1-t(x,y)),

[0243] Among them, I(x,y) represents the pixel of the haze image, J(x,y) represents the pixel of the non-haze image, and A is the atmospheric light intensity, which is used to control the brightness and color of the haze. t(x,y) is the transmittance, that is, the transparency of the pixel, which can be obtained according to the Lambert-Beer law. The pixel J(x,y) of the non-haze image can be regarded as the initial light intensity I 0 , then the calculation formula of transmittance t(x,y) is

[0244] t(x,y)=e -β *d(x ,y) ,

[0245] Among them, d(x,y) is the distance that the reflected light corresponding to each pixel (x,y) in the image passes through the haze to reach the image sensor. β is called the attenuation coefficient, which reflects the concentration of haze in the atmosphere. The thicker the haze, the greater the attenuation coefficient.

[0246] Based on the above synthesis formula of haze image, the algorithm steps of random haze synthesis are described as follows.

[0247] Random generation parameters: Randomly generate atmospheric light intensity A, whose value range is between 0 and 255, expressed by mathematical formula:

[0248] A=round(a), The attenuation coefficient β is randomly generated, and the value range is set between 0 and 0.1, which can be expressed as follows:

[0249]

[0250] Randomly select a pixel coordinate in the image as the center of the haze (X H ,Y H ), expressed mathematically as

[0251]

[0252] Among them, Width and Height are the width and height of the haze-free image J respectively;

[0253] Determine the farthest pixel: Determine the distance from the center of the haze (X H ,Y H )The farthest pixel coordinate (X F ,Y F ), for a rectangular image, the farthest pixel coordinate (X F ,Y F ) must be one of the four vertex coordinates of the image. If X H Greater than Width / 2, then X F is 0, otherwise X F For Width, if Y F Greater than Height / 2, then X F is 0, otherwise X F is Height, which can be expressed as

[0254]

[0255] Calculate the size of the haze: Calculate the center of the haze (X H ,Y H ) and the farthest pixel coordinate (X F ,Y F )’s Euclidean distance dist F , the calculation formula is

[0256]

[0257] According to the farthest distance dist F Set the haze size, the calculation formula is:

[0258]

[0259] Calculate the distance matrix: Calculate the distance from each pixel (x, y) in the image to the center of the haze (X H ,Y H ) is calculated as follows:

[0260]

[0261] Use the above distance matrix dist to generate haze spreading from the center to the surrounding areas. If we use the inverted distance matrix dist R , then the haze is generated from the surrounding to the center, and the inversion distance of each pixel is dist R (x,y) is determined by the maximum distance dist F Subtract the original distance dist(x,y) to obtain the calculation formula:

[0262] dist R (x,y)=dist F -dist(x,y),

[0263] Calculate the transparency matrix: Calculate the propagation distance d(x,y) of the reflected light of each pixel in the haze. The calculation formula is

[0264] d(x,y)=size-0.04*dist(x,y),

[0265] Then calculate the transparency of each pixel t(x,y), the calculation formula is

[0266] t(x,y)=e -β*d(x,y) ,

[0267] Where d(x,y) is the distance that the reflected light corresponding to each pixel (x,y) in the image passes through the haze to reach the image sensor, and β is called the attenuation coefficient;

[0268] Calculate the haze image: First normalize the atmospheric light intensity A and image pixel J(x,y), the calculation formula is:

[0269]

[0270] Then calculate the normalized haze image pixel I(x,y) nml , the calculation formula is

[0271] I(x,y) nml =J(x,y) nml *t(x,y)+A nml *(1-t(x,y)), finally inverse normalize the haze image pixel I(x,y) nml , the calculation formula is

[0272] I(x,y)=I(x,y) nml *255.0;

[0273] Normalize the haze image: Round off the haze image pixel I(x,y) and express it as follows:

[0274] I(x,y) rnd =round(I(x,y)),

[0275] Then the range of pixel I(x,y) is limited to between 0 and 255, which can be expressed as

[0276]

[0277] The random light spot addition algorithm of the present invention is proposed for the situation of strong light pollution in mines. In order to ensure the safety and work efficiency of miners, high-intensity lighting equipment is used in mines, including miner's cap lamps, searchlights, work lights, etc. When strong light directly shines on the lens of the camera, light spots and halos of varying sizes will be formed on the captured image. These light spots will seriously interfere with the normal operation of the pedestrian intrusion detection program. To this end, the data enhancement algorithm for random light spot addition of the present invention enhances the ability of the network model to resist light spot interference. The algorithm steps for random light spot addition are described as follows:

[0278] Random generation parameters: Randomly generate the center of the spot (X C ,Y C ), the value range is within the image size (Width, Height), and is expressed as

[0279]

[0280] The spot radius R is randomly generated, and the value range is set between 20 and 100, which can be expressed as follows:

[0281] R = round(r),

[0282] The randomly generated spot brightness L is set between 220 and 235, which can be expressed as follows:

[0283] L = round(l),

[0284] The blur size sigma is randomly generated, and the value range is set between one-quarter of the spot radius and one-half of the spot radius, which can be expressed as

[0285] sigma=round(σ′),

[0286] Set the overlay weight: Set the light spot effect image Effect fh The superposition weight Weight is set to 0.6, and then the spot brightness L before superposition is calculated using the superposition weight Weight PRE , the calculation formula is

[0287] L PRE =Weight*L,Weight=0.6;

[0288] Initialize effect image: Generate the initial spot effect image Effect init , initialize the effect image to an all-zero matrix of the same size as the original image Image, for the spot effect image Effect initFor each pixel (x,y) in

[0289] Effect init (x,y)=0;

[0290] Generate a light spot effect: Create a light spot effect image Effect init Generate a circular light spot, the center of which is (X C ,Y C ), the radius is R, and the pixel value inside the circle is L PRE , expressed mathematically as

[0291]

[0292] Generate halo effect: Use Gaussian blur to generate the halo around the spot. The standard deviation of Gaussian blur is the blur size sigma. The filter size ksize′ of Gaussian blur is calculated based on the standard deviation sigma. The calculation formula is:

[0293] ksize′=round(6*sigma+1),

[0294] Ensure that the filter size ksize is a positive odd number. If the remainder of ksize′ divided by 2 is 0, it means that ksize′ is an even number, so add 1 to it to make it an odd number; if the remainder of ksize′ divided by 2 is 1, it means that ksize′ is an odd number, so keep it as the original value, which can be expressed as the mathematical formula:

[0295]

[0296] Among them, x mod 2 means to find the remainder of x divided by 2, sigma is the standard deviation, ksize is the filter size, and the spot effect image Effect f Perform Gaussian blur, expressed as

[0297] Effect fh =GaussianBlur(Effect f ,ksize,sigma),

[0298] Calculate the mask image: Calculate the mask of the spot area. First, generate an image with the spot effect effect. fh The initial mask of the same size, for each pixel in the mask Mask, now has

[0299] Mask(x,y)=0,

[0300] For the light spot effect image Effect fh For each pixel in, if the pixel value Effect fhIf (x, y) is greater than 0, the pixel value at the corresponding position in the mask is set to 1, otherwise it is set to 0, which can be expressed as

[0301]

[0302] Overlay effect image: Overlay light spot effect image Effect fh The superposition method of these two images is weighted summation. The weight of the original image is fixed at 1.0. The spot effect image is fh The weight is Weight, and the result image after adding the spot effect is Result. The calculation formula is

[0303] Result=Image*1.0+Effect fh *Weight;

[0304] Normalize the result image: Round the result image pixels Result(x,y) to the nearest integer, which can be expressed as

[0305] Result rnd (x,y)=round(Result(x,y)),

[0306] Then the pixel Result rnd The range of (x,y) is limited to between 0 and 255, which can be expressed as

[0307]

[0308] Limit the brightness of the light spot: limit the maximum pixel value of the light spot area to the light spot brightness L. When the pixel value of the mask Mask(x,y) is 1, it means the Result at the corresponding position clip (x,y) is in the spot area, then judge the Result clip Is the pixel value of (x, y) greater than L? If yes, change the pixel value of the position to L, otherwise keep the original pixel value; when the pixel value of Mask(x, y) is 0, it means the Result of the corresponding position clip (x, y) is not in the spot area, no additional changes are made, and the mathematical formula is expressed as

[0309]

[0310] Among them, min(·) represents the minimum value operation.

[0311] In order to solve the problem of poor detection accuracy in the prior art due to the complex environment of mines, the present invention provides three data enhancement methods, namely, random grayscale conversion, random haze synthesis and random spot addition data enhancement. These data enhancement methods expand the pedestrian data set by simulating the mine environment, reduce the cost of collecting training data, and thus enhance the detection performance and reliability of the model.

[0312] In step S101, the electronic fence can be automatically generated, or you can use your finger or touch pen to click on the vertices of the electronic fence in the video scene on the display touch screen. The program sequentially connects the vertices to draw a closed polygon. The area within the polygon is the intrusion alarm area, and the edge of the polygon is the electronic fence.

[0313] Pedestrian intrusion judgment can pre-set several key point position options for users to choose according to actual scenarios, or it can directly display a pedestrian bounding box diagram, allowing users to select, change or cancel key point positions by clicking with their fingers or a stylus.

[0314] Preprocessing includes: pedestrian recognition preprocessing before pedestrian recognition, including image scaling and pixel value normalization. Image preprocessing also includes: pedestrian detection preprocessing before pedestrian detection, including image scaling and pixel value normalization.

[0315] The image scaling is to enlarge or reduce the image to a square image with a certain side length. Since directly scaling the image will cause the object in the image to deform, the present invention uses proportional scaling. Pixel value standardization makes the image data have similar scale and distribution. Pixel value normalization adjusts the image data to a value range from 0 to 1.

[0316] Specifically, the proportional scaling steps are: 1) selecting the longest side of the height and width of the image; 2) dividing the original target side length by the longest side of the image to calculate the scaling factor; 3) multiplying the scaling factor by the shortest side of the image to calculate the target length of the shortest side of the image; 4) scaling the image to scale the longest side of the image to the original target side length, and the shortest side of the image to its calculated target length; 5) subtracting the target length of the shortest side from the original target side length and dividing by 2 to calculate the width of the image edge that needs to be filled; 6) filling the image edge with a fixed pixel value such as 0 to make the filled image a square image with the original target side length.

[0317] The steps for pixel value normalization are: 1) calculate the mean and standard deviation for each channel of all images in the training set, or directly use the provided pre-training dataset to calculate the pixel mean and standard deviation; 2) subtract the calculated pixel mean from each pixel of all images in the training set; 3) divide each pixel of all images in the training set by the calculated pixel standard deviation.

[0318] Specifically, since the rounding range of image pixels is from 0 to 255, the step of normalizing pixel values ​​is to divide all pixels in the image by 255 to calculate the normalized pixel values.

[0319] The present invention can use EfficientNet-Lite as a pedestrian recognition model with low computational complexity; or YOLOv5 can be selected as a pedestrian detection model with high computational complexity. In addition to the above three data enhancement algorithms, the present invention can also use other commonly used data enhancement algorithms when training the model, such as image flipping, image cropping, affine transformation, noise addition, blur processing, brightness adjustment, image splicing, etc.

[0320] The main task of pedestrian recognition of the present invention is to filter out videos within a period of time for pedestrian detection, rather than a video frame. After the pedestrian recognition model recognizes that there are pedestrians in the current video frame, the present invention starts the pedestrian detection model, and then the pedestrian detection model detects the video frame extracted by the next program, rather than detecting the video frame currently identified as having pedestrians. Therefore, in order to ensure the effect of subsequent pedestrian detection, when the pedestrian target enters the image, the video frame should be identified, and then the pedestrian detection model is enabled to continuously detect the position of the pedestrian target. However, these pedestrian targets that have just entered the image usually do not have a complete human body shape, while most images in general pedestrian data sets have a relatively complete full body or upper body human body shape. For this reason, a data enhancement algorithm for image cropping can be used to crop out some special training images, which contain suspected targets of various incomplete human body shapes, such as the left half of the body or the right half of the body, to enhance the real-time and reliability of the pedestrian recognition model.

[0321] Since subsequent pedestrian detection can serve as a guarantee, the pedestrian detection model can slightly lower the decision threshold when outputting classification labels, and relax the standard for the pedestrian recognition model to identify images as pedestrians, thereby reducing the missed detection rate of the pedestrian intrusion detection program. For example, during training, the pedestrian recognition model uses 0.5 as the decision threshold. If the model prediction value is greater than or equal to 0.5, the classification label is classified as 1, and if it is less than 0.5, the classification label is classified as 0. In actual reasoning, 0.4 can be used as the decision threshold. Some misdetected targets due to the lowering of the decision threshold can be handed over to the subsequent pedestrian detection model with higher accuracy for resolution.

[0322] The key points used in the intrusion judgment stage of the present invention are calculated according to the preset key point type after the pedestrian detection model predicts the pedestrian bounding box. When the pedestrians in the scene are usually standing or walking, the midpoint of the lower boundary of the pedestrian bounding box can be selected as the key point. When it is necessary to prevent pedestrians from climbing into or rolling into the alarm area, the midpoint of the upper boundary of the pedestrian bounding box can be selected as the key point. When it is necessary to prevent pedestrians from accidentally falling into or rolling into the alarm area, multiple key points can be selected at different positions of the bounding box, such as the midpoints of all (up, down, left, and right) boundaries can be selected as key points. Regarding the calculation method of the key point position, it is assumed that the parameters of the pedestrian bounding box output by the pedestrian detection model are (x, y, w, h), where x is the horizontal coordinate of the target center, y is the vertical coordinate of the target center, w is the width of the bounding box, and h is the height of the bounding box. The coordinate of the upper left corner of the pedestrian bounding box (x 1 ,y 1 ) and the lower right corner coordinate (x 2 ,y 2 ) are calculated by

[0323]

[0324] If the key point is located at the midpoint of the lower boundary of the bounding box, the key point coordinates (x K ,y K ) is calculated as

[0325]

[0326] Similarly, if the key point is located at the midpoint of the upper boundary of the bounding box, the key point coordinates (x K ,y K ) is calculated as

[0327]

[0328] If the key point is located at the middle of the left boundary of the bounding box, the key point coordinates (x K ,y K ) is calculated as

[0329]

[0330] If the key point is located at the middle of the right boundary of the bounding box, the key point coordinates (x K ,y K ) is calculated as

[0331]

[0332] The present invention can use a variety of ways to give prompts or warnings, including using voice to give prompts, using a buzzer to give warnings, using a display to give warnings, etc.

[0333] The purpose of the pedestrian recognition model of the present invention is to identify pedestrian targets in video frames. Once the program recognizes a pedestrian, it means that a pedestrian has passed near the electronic fence, and then the pedestrian detection model is started. The purpose of the pedestrian detection model is to determine the position of the pedestrian in the image, because the subsequent intrusion judgment requires the pixel coordinates of the pedestrian and the electronic fence.

[0334] The pedestrian recognition model and pedestrian detection model of the present invention are deep learning models. Deep learning models can only be used after training. Training is the process of the model learning patterns and rules from the provided data. The pedestrian recognition model and pedestrian detection model used in the present invention are both supervised learning. The input data and its real label (containing the correct answer) are required for training. The training and testing (verification) process of the deep learning model is as follows: Figure 3 shown.

[0335] The steps of the model training phase are as follows:

[0336] Image preprocessing: Before inputting the image into the model, preprocessing is usually required to improve the performance and stability of the model. Image preprocessing includes pixel normalization, pixel standardization, image scaling, data augmentation, etc. These operations help reduce noise during model training and improve the generalization ability of the model.

[0337] Image input: Before model training, the data set generally needs to be screened and checked to ensure that the input image meets the task to be achieved by the current model.

[0338] Model inference calculation: mainly performs forward propagation to obtain predicted output. Forward propagation is the process of calculating the output based on the input data and current parameters. In this step, we need to select the corresponding deep learning model according to different computer vision tasks. For example, the pedestrian recognition task should use the image recognition model, and the pedestrian inspection task should use the target inspection model. These models are designed for different computer vision tasks. The corresponding network structure.

[0339] Result output: Different computer vision tasks have different prediction results, which can be mainly divided into parameter output and image output. For example, the pedestrian recognition and pedestrian detection of the present invention are both parameter outputs. Pedestrian recognition outputs the category of the image (with or without pedestrians), and pedestrian detection outputs the position of the bounding box and the category of the target in the bounding box.

[0340] Loss calculation: Evaluate the difference between the model's predicted output and the true label. The model uses a loss function to define and quantify these differences, such as mean squared error loss or cross entropy loss. Generally, the smaller the calculated loss value, the closer the model's prediction is to the true label.

[0341] Gradient calculation: Use the back-propagation algorithm to calculate the derivatives of the loss function with respect to the model parameters. These gradients indicate the rate of change of the loss function in the parameter space and are used to guide the direction and magnitude of parameter updates.

[0342] Parameter update: Adjust the parameters within the model based on the gradient and learning rate. This process is usually implemented using gradient descent or its variants. By iteratively updating the parameters, the model gradually learns the parameter values ​​that minimize the loss function.

[0343] Iterative optimization: Usually, each image in the dataset needs to be input repeatedly. Through repeated image input, the parameters inside the model are continuously optimized iteratively, so that the results of the model inference calculation are constantly close to the real label. The performance of the model will also gradually improve in the iteration until a sufficient number of iterations are trained.

[0344] The steps of the model testing (verification) phase are as follows:

[0345] Image preprocessing: In addition to data augmentation, image preprocessing in the test phase should be consistent with that in training. Data augmentation can be used or not in the test phase, but it is not used in actual applications.

[0346] Image input: The test (validation) set used in the test (validation) phase cannot contain the images used in the training phase. Before image training, the dataset needs to be divided into a training dataset and a test (validation) dataset.

[0347] Model inference calculation: mainly performs forward propagation to obtain prediction output. The trained model is used in the testing phase.

[0348] Result output: If necessary, you can draw a graph to verify the results, such as the target detection model.

[0349] Evaluation index calculation: Evaluation index is used to evaluate the performance of the model in a quantitative way. Different evaluation indexes can be analyzed to identify the defects and advantages of the model, so as to determine whether the model meets the expected requirements.

[0350] The data enhancement algorithm of one embodiment of the present invention includes: image cropping. The image cropping of this embodiment is to select a rectangular area in the image, crop the area as a new image, and discard other image areas. The steps of image cropping are as follows: first, the cropping size is determined. The cropping size (the height and width of the cropping area) of the data enhancement algorithm is randomly generated, and its value range needs to be limited to the image size (the height and width of the image). Next, the cropping position is determined. The randomly generated cropping position cannot exceed the image size, so it is necessary to calculate the value range of the cropping position based on the cropping size and the image size. Finally, the cropping operation is performed to generate a cropped image based on the cropping size and the cropping position. If pedestrian detection is performed, the label needs to be re-made. After the image is cropped, the position of the target relative to the image may change, and the target may even disappear due to the cropping operation. To ensure that the training results are correct, it is necessary to confirm whether the original target exists and recalculate the bounding box of the remaining target, and make a new label based on the calculation results.

[0351] The image flipping of an embodiment of the present invention is to flip the image upside down or left to right. For the deep learning model, the image is a new image after flipping. If pedestrian detection is started, the target bounding box needs to be recalculated after the image is flipped.

[0352] The affine transformation of an embodiment of the present invention is to rotate, translate, scale or tilt the image, and the graphics in the image before and after the affine transformation are similar. Since these operations can be uniformly represented by an affine matrix, they are collectively referred to as affine transformations. If pedestrian detection is performed, the target bounding box needs to be recalculated after the affine transformation.

[0353] The noise addition in one embodiment of the present invention is to add noise to the image. The noise types include Gaussian noise, Poisson noise, salt and pepper noise, etc.

[0354] The blur processing of an embodiment of the present invention performs blur processing on an image. Blurring types include mean blur, Gaussian blur, motion blur, compression blur, and the like.

[0355] The brightness adjustment of an embodiment of the present invention mainly adjusts the pixel values ​​in the image as a whole. If the pixel values ​​are adjusted higher as a whole, the image brightness is increased, and if the pixel values ​​are adjusted lower as a whole, the image brightness is decreased.

[0356] The image stitching of one embodiment of the present invention refers to a mosaic data enhancement algorithm of a pedestrian detection model for target detection tasks. The algorithm randomly selects four pictures, performs data enhancement on each image, including image cropping, affine transformation, brightness adjustment, etc., and then stitches the four images into a new image. During the stitching, it is also necessary to recalculate the target bounding box and create a label for the new image.

[0357] The data enhancement algorithm can be applied in the image preprocessing step of the model training or verification stage. The main purpose is to enrich the diversity of the training set images and increase the generalization ability of the model. The data enhancement algorithm is not used in the actual application of the model, that is, there is no data enhancement algorithm in the preprocessing step of the pedestrian intrusion detection of the present invention. The above data enhancement algorithms are independent of each other and can be selected and used in any order. For the data enhancement algorithm for image splicing, the data enhancement processing included therein can be flexibly changed, and it can be processed first and then spliced, or it can be spliced ​​first and then processed.

[0358] Pedestrian detection in this embodiment is the core of the pedestrian intrusion detection method, which is responsible for confirming the existence of pedestrians and determining the location of pedestrians. To this end, a large amount of calculation needs to be allocated to the pedestrian detection model to ensure the accuracy of this step. Pedestrian recognition is before pedestrian detection. It is responsible for screening video frames without pedestrians, reducing unnecessary pedestrian detection, and improving the overall efficiency of the algorithm. Therefore, pedestrian recognition cannot determine the final result of the pedestrian intrusion detection method, and the amount of calculation allocated to the pedestrian recognition model is relatively small.

[0359] The pedestrian recognition in this embodiment belongs to the image recognition task in computer vision, while the pedestrian detection belongs to the target detection task in computer vision. Although the input of both tasks is an image, the output of the image recognition task is only the classification label of the entire image, while the target detection task not only outputs the classification label of the target in the image, but also outputs the position parameters of the target in the image. Therefore, from a theoretical analysis, the target detection task is more complex and challenging than the image recognition task. Under the condition of equivalent performance, the computational complexity of the pedestrian detection model will be greater than that of the pedestrian recognition model.

[0360] In fact, image preprocessing mainly plays a role in the model training stage, making the model training faster and better. Different image preprocessing operations have different functions. Image preprocessing includes the following:

[0361] Pixel normalization: Scale the pixel values ​​of an image to a specific range, such as [0,1] or [-1,1]. This process limits the image pixels to a certain range, makes the image data have the same scale, and reduces the impact of different data ranges on model training. This operation helps reduce the computational complexity of the model and improve the stability and convergence speed of model training.

[0362] Pixel normalization: also known as zero mean normalization, subtracting the mean from each pixel value and dividing it by its standard deviation. This process makes the pixel values ​​of the image have zero mean and unit variance, making the image data have the same distribution, reducing the impact of different data distributions on model training. This operation helps reduce the computational complexity of the model and improve the stability and convergence speed of model training.

[0363] Image scaling: This is mainly about adjusting the size of the image to suit the requirements of the model input.

[0364] Data augmentation: Increase the size and diversity of the training dataset by generating variations of the image, including flipping, rotation, translation, scaling, cropping, etc. Data augmentation helps the model learn more stable features, reduce overfitting and improve generalization capabilities.

[0365] Image preprocessing is also required when applying the model in practice, mainly to maintain data consistency. Because the model is trained using preprocessed image data, the input image must also be preprocessed in practice to obtain similar results as in training. Data augmentation is to increase the diversity of the training data set.

[0366] Detection preprocessing is the preprocessing that needs to be performed before the image is input into the pedestrian detection model. Recognition preprocessing is the preprocessing that needs to be performed before the image is input into the pedestrian recognition model. Pre-detection preprocessing uses pixel normalization, while pre-recognition preprocessing uses pixel standardization.

[0367] The pedestrian detection steps of the present invention are as follows: First, the pedestrian detection model extracts features from the input image, outputs multiple feature maps of different sizes, and then predicts bounding boxes on these feature maps. Among them, one feature map corresponds to one detection layer, and the pixels in each feature map are also called a grid. In the process of feature extraction, the convolutional neural network iteratively calculates the feature map through the convolution layer and the downsampling layer, and the size (i.e., resolution) of the feature map will continue to shrink, and the number of channels will continue to increase. Outputting multiple feature maps of different sizes refers to outputting multiple feature maps with a resolution of a fraction of the size of the original image. For example, the pedestrian detection model has 4 detection layers, and the resolution of the feature map in each detection layer is 1 / 8, 1 / 16, 1 / 32, and 1 / 64 of the original input image, respectively.

[0368] The pedestrian detection model is based on anchor boxes for target detection. Before use, three bounding boxes with different aspect ratios and no center position (i.e., anchor boxes) need to be set for each feature map according to the size. Each pixel position of each feature map is taken as the target center, and three different target bounding boxes are predicted based on the three anchor boxes. Predicting the target bounding box means predicting a set of parameters that can represent the target bounding box, including the normalized center offset x i ,y i , anchor frame scaling ratio w i ,h i , target confidence p conf.i And the class probability p cls1.i ,p cls2.i ,..., where i is the bounding box number of the detection layer.

[0369] Let the number of categories to be detected be nc=C, then the number of parameters for each bounding box is no=5+nc, the height and width of the feature map in the detection layer are nu, nx respectively, and the number of anchor boxes is na=3. Therefore, the number of bounding boxes in the detection layer is na*ny*nx=N, and the number of bounding box parameters is na*ny*nx*no, then all target bounding boxes pred predicted by the detection layer are

[0370]

[0371] In the above formula, the target confidence p conf.N Indicates the probability that there is a target in the bounding box N, p clsC.N Represents the probability that the target in the bounding box N is of the Cth category.

[0372] The target center of the target bounding box is calculated based on the center offset. Because the target bounding box is generated at each pixel position in the feature map, the target center of each bounding box is determined, which is the pixel coordinate when the bounding box is generated. In addition, because the feature map is obtained by downsampling the input image, the bounding box detected from the feature map is also a reduced size, and the bounding box needs to be enlarged to a size suitable for the input image. However, enlarging the pixel coordinates of the target center will cause the error to increase. For example, if the original pixel coordinate error is 1, the error will increase to 8 after enlarging 8 times. Therefore, the integer target center is not accurate enough, and the center offset needs to be used to make it accurate to a decimal.

[0373] The internal coordinate axes of the grid are based on the pixel coordinates of the target center (grid X ,grid Y ) is the origin, with the right and the downward directions being the positive directions of the X and Y axes respectively, which are used to further determine the precise position of the target center (bbox X ,bbox Y ). Center bias X ,bias Y ) is the target center (bbox X ,bbox Y ) is the relative position of the coordinate axis inside the grid. To improve the accuracy and robustness of target detection, the pedestrian detection model will be compared with the center coordinate (bbox X ,bbox Y )The two adjacent pixel coordinates (bbox X ±1,bbox Y ±1) is also considered as the target center, resulting in center bias. X ,bias Y ) changes from [0,1.0] to [-0.5,1.5]. The normalized center bias x predicted by the pedestrian detection model i ,yi The value range is [0,1.0]. To get the real center bias X ,bias Y ) needs to expand its value range, and the calculation formula of this process is:

[0374]

[0375] Then we can find the target center of the bounding box (bbox X ,bbox Y ), the calculation formula is

[0376]

[0377] In the above formula, scale is the magnification of the bounding box, which can be obtained according to the reduction factor of the feature map in the detection layer compared to the original input image.

[0378] The size of the target bounding box is calculated based on the preset anchor box. The anchor box is designed according to the size of the feature map in the detection layer. The larger the size of the feature map in the detection layer, the larger the size of the anchor box used. The pedestrian detection model automatically calculates the anchor box. It performs a cluster analysis on the sizes of all target bounding boxes in the training set and generates a set of optimal anchor box sizes. When detecting pedestrians, the pedestrian detection model modifies the preset anchor box by scaling it so that it keeps approaching the true target bounding box. To this end, the pedestrian detection model predicts the anchor box scaling ratio w i ,h i , the value range is [0,1.0]. Note that the size of the anchor box used by bounding box i is anchor W ,anchor H , then the size of the target bounding box bbox W ,bbox H The calculation formula is

[0379]

[0380] From the above formula, we can see that the width and height of the target bounding box bbox W ,bbox H The maximum value of the anchor box width and height is anchor W ,anchor H 4 times of that.

[0381] The target detection task not only needs to predict the target location, but also needs to determine the target category. Of course, not all bounding boxes contain targets, and the target confidence p conf.i Used to determine whether there is a target in the bounding box i. Target confidence p conf.iIf the confidence threshold is greater than 0.5, it means that there is a target in the bounding box, otherwise it means that there is no target in the bounding box. The so-called "target" refers to the target within the detection category range (physical existence). As for the targets outside the detection category range, they are all background for the target detection model. In addition, the target detection model also predicts a series of category probabilities p clsN.i ,p cls2.i ,..., are used to determine the category to which target i belongs. These category probabilities are meaningful only when there is a target in the bounding box i. The category corresponding to the largest category probability is selected as the category to which the target belongs.

[0382] The pedestrian detection model uses non-maximum suppression as a post-processing algorithm. Since the pedestrian detection model will generate a large number of bounding boxes at the same target position, and the same target can only have one bounding box, the predicted bounding boxes will overlap with each other and there must be redundancy. Therefore, after the pedestrian monitoring model predicts the bounding boxes, these bounding boxes need to be non-maximum suppressed. The main function of non-maximum suppression is to suppress a large number of redundant bounding boxes and find the best target bounding box. The specific steps are as follows:

[0383] Set threshold: Set the confidence threshold Thres Conf Intersection and union ratio threshold Thres IOU ;

[0384] Filter: Sort all bounding boxes according to the target confidence, and discard those whose target confidence is less than the confidence threshold Thres Conf The bounding box of

[0385] Best selection: Select the bounding box with the highest target confidence and use it as the best bounding box.

[0386] Discard: Calculate the intersection and union ratio of the best bounding box and other bounding boxes, and discard those whose intersection and union ratio is greater than the intersection and union ratio threshold Thres IOU Other bounding boxes of

[0387] Repeat processing: Repeat the optimization and discarding steps until all bounding boxes are processed.

[0388] Regarding the characteristics of the bounding box, the target bounding box is a rectangular bounding box whose sides are parallel to the four sides of the image and have no rotation angle. Although such a rectangular bounding box may not accurately match the shape of the target, it can be simply represented using four parameters and is easy to calculate. In addition, the bounding box also has confidence and category labels associated with it. Among them, the confidence is assigned by the target detection model, indicating the model's certainty about the detection result. The category label is predicted by the target detection model and indicates the type of target in the bounding box. The bounding box is the bounding box of the target in the image. When the target in the image is reduced or enlarged, the bounding box should also be reduced or enlarged proportionally. In addition, the rectangular bounding box also has the characteristic of easy annotation. When annotating an image, we only need to determine the upper left corner and lower right corner of the target boundary to quickly annotate a target. Compared with other polygonal bounding boxes, although the rectangular bounding box cannot accurately describe the outline of the target, it requires fewer vertices to be determined, which saves time and effort in annotation. When a large number of images need to be annotated, the rectangular bounding box is the least time-consuming option.

[0389] There are two ways to represent the target bounding box. One is the center point plus width and height representation (x, y, w, h), where (x, y) is the center point of the bounding box, and w, h is the width and height of the bounding box; the other is the coordinate representation (x 1 ,y 1 ,x 2 ,y 2 ), (x 1 ,y 1 ) is the upper left corner of the bounding box, (x 2 ,y 2 ) is the lower right corner of the bounding box. The two bounding box representation methods can be converted to each other. The calculation formula for converting the center point plus width and height representation to the coordinate representation is:

[0390]

[0391] The calculation formula for converting the coordinate representation to the center point plus width and height representation is:

[0392]

[0393] When the pedestrian recognition model of the present invention performs pedestrian recognition, it extracts features from the input image to obtain multiple feature maps of different sizes, then compresses these two-dimensional feature maps into one-dimensional features, and finally uses a fully connected layer to predict the category probability of the image. For each category that needs to be recognized in the pedestrian recognition model, the output layer must have a corresponding category probability. The value range of all category probabilities is [0,1], and the model takes the category corresponding to the maximum category probability as the predicted image category. Compared with the pedestrian detection model, the pedestrian recognition model does not need to predict the bounding box position and size for the image classification task, and the input image is actually the selected target.

[0394] Since subsequent pedestrian detection can serve as a guarantee, the pedestrian recognition model can slightly lower the decision threshold when outputting classification labels, thereby reducing the missed detection rate of the pedestrian intrusion detection program. That is, in order to prevent missed detection, the present invention will relax the standard for the pedestrian recognition model to identify the target as a pedestrian, and will label the target that is not a pedestrian with a pedestrian classification label, and then use the pedestrian detection model with better performance in the subsequent steps to detect the real pedestrian target from these suspected pedestrian targets. Therefore, the program only recognizes the prediction results detected by the pedestrian detection model, and the pedestrian recognition model is only pre-screened. The screened video frames are only "suspected to exist" pedestrian targets, which need to rely on the pedestrian detection model for further confirmation.

[0395] After the pedestrian recognition model identifies that there is a suspected pedestrian target in the current video frame, the present invention activates the pedestrian detection model to detect the video frame extracted by the program next time, instead of starting the pedestrian detection model to detect the video frame suspected of having a pedestrian target. The so-called screening can be said to be screening a video segment rather than a video frame. The pedestrian recognition step is mainly to notify the program that a pedestrian is likely to enter the video, that is, a pedestrian target is likely to appear in the video within a period of time from now on. After that, the program will respond to the notification and start the pedestrian detection model to detect the video segment within a certain period of time.

[0396] Setting the maximum number of detections M of the pedestrian detection model is actually indirectly setting the length of the pedestrian detection start time. The purpose of the number judgment is to check whether the current number of detections n exceeds the maximum number of detections M. Next, two situations are handled. One is that the pedestrian detection model does not detect pedestrian targets. Each time a pedestrian target is not detected in a video frame, the counting variable n is increased once until the maximum number of detections M is reached. The program will determine that there is indeed no pedestrian target and return to pedestrian recognition. The second is that the pedestrian detection model has detected pedestrian targets. Each time a pedestrian target is detected in a video frame, an intrusion judgment is required. If the program determines that the pedestrian has not intruded into the electronic fence, the counting variable n is cleared and the detection continues in the pedestrian detection stage until the program determines that the pedestrian has intruded into the electronic fence or the pedestrian leaves the video range. If the pedestrian leaves the video range, it will not return to the pedestrian recognition stage immediately, but will enter the above-mentioned situation 1 until the maximum number of detections M is reached, and the program will return to the pedestrian recognition stage. In short, in the pedestrian detection stage, if the pedestrian detection model does not detect a pedestrian target in the current frame, the counting variable n will be increased by 1. If the pedestrian detection model detects a pedestrian target in the current frame, it will make an intrusion judgment and then the counting variable will be reset to zero. Until the pedestrian breaks into the electronic fence, the program will send a warning signal, or the counting variable reaches the maximum number of detections M, and the program will return to the pedestrian recognition stage.

[0397] The pedestrian detection model and pedestrian recognition model of the present invention are both convolutional neural networks. The reasoning calculation of the model is determined by the network structure. Designing the structure of the convolutional neural network is to design the calculation process of the neural network. The convolutional neural network mainly includes basic modules such as convolution layer, downsampling layer and activation function, and the arrangement and combination of them determines the reasoning calculation of the convolutional neural network.

[0398] The pedestrian recognition model and pedestrian detection model of an embodiment of the present invention use a loss function to calculate the loss during the training and testing process. The loss function is an error function used to estimate the predicted value and the true value of the model. Different loss functions are used according to different tasks. The pedestrian recognition model of the present invention has only a classification loss L cls The loss function of the pedestrian detection model of the present invention is divided into classification loss L cls , positioning loss L loc and confidence loss L conf Among them, the classification loss L cls Use focal loss, which is calculated as follows:

[0399]

[0400] Among them, α i is a balancing factor used to balance the ratio of positive and negative samples. γ is an adjustment factor used to focus on samples that are difficult to classify and to weaken the contribution of simple samples to the loss. is the one-hot encoding of the true label, is the predicted probability. Confidence loss L conf It also uses focal loss, which is calculated as follows:

[0401]

[0402] Among them, α i is the balancing factor, γ is the regulating factor, is the true value of the target confidence, YOLOv5 uses intersection-over-union instead, is the target confidence of the prediction.

[0403] Positioning loss L loc It is more complicated. It uses CIoU Loss, and the calculation formula is:

[0404]

[0405] Among them, ρ is the Euclidean distance between the center points of the predicted box and the real box, c is the diagonal length of the minimum closed area containing the two boxes, v is the loss term of the aspect ratio, and α is the balance factor. The calculation formula of the aspect ratio loss term v is

[0406]

[0407] in, is the height and width of the real frame, and is the height and width of the predicted box. Then, the calculation formula of the balance factor α is

[0408]

[0409] Where v is the loss term of the aspect ratio calculated above. The calculation formula for the intersection over union ratio IoU is

[0410]

[0411] Here, the bounding boxes A and B are the predicted box and the true box, respectively.

[0412] Finally, the total loss L of the pedestrian detection model is total is the classification loss L cls , positioning loss L loc and confidence loss L conf The weighted sum of is calculated as

[0413] L total =λ cls L cls +λ loc L loc +λ conf L conf ,

[0414] Among them, λ cls ,λ loc ,λ conf are the weight coefficients of each loss respectively.

[0415] For pedestrian recognition, accuracy is mainly used as the evaluation index of the model. For pedestrian detection, mean Average Precision (mAP) is used as the evaluation index of the model. The mean average precision (mAP) is the average of the average precision (AP) of all categories, and its calculation formula is

[0416]

[0417] Among them, C represents the total number of categories, AP i Represents the average precision of the i-th category. Average precision AP iIt is obtained by drawing the precision-recall curve and calculating the area under the curve (AUC). When the evaluation index of the model is close to the optimal value, the model is considered to be practical, and the maximum fluctuation value of the evaluation index is 5%. In addition, the prediction results of the model need to be drawn on the input image, and the specific detection performance of the model needs to be observed by human eyes.

[0418] If the model overfits during the training phase, regularization and dropout techniques can be used. Regularization and dropout techniques are mainly used to prevent overfitting during the training phase, and are usually not used during the test phase to evaluate the performance of the model without any regularization. The pedestrian recognition model of the present invention uses dropout technology, and the inference calculations in the training phase and the test phase are different.

[0419] The method and system for detecting pedestrian intrusion in a mine environment of the present invention realizes pedestrian intrusion detection in a complex environment of a mine, improves safety and operating efficiency, improves the accuracy of pedestrian detection in the mine, reduces false alarms in the intrusion judgment stage, and thus enhances the operating safety of miners. At the same time, due to the reduced reliance on manual monitoring, the operating efficiency is also improved. The resource consumption and cost are reduced, the algorithm flow of pedestrian intrusion detection is optimized, the computing load on resource-constrained devices is reduced, the energy consumption and equipment requirements are reduced, and the long-term human monitoring cost is reduced. The detection performance and anti-interference ability are enhanced, and the pedestrian data set is expanded at a low cost through the data enhancement technology that simulates the mine environment, the detection performance and anti-interference ability of the model are enhanced, and its stable operation in the complex environment of the mine is ensured. The present invention is easy to integrate and deploy, can be compatible with the existing monitoring system, and is easy to be quickly deployed and integrated in the mine without large-scale transformation of existing facilities. The present invention provides a more reliable and efficient pedestrian intrusion detection method, which helps to improve the effectiveness of the existing safety monitoring system and brings practical value and improvement to mine safety management.

[0420] A system for detecting pedestrian intrusion in a mine environment according to an embodiment of the present invention includes:

[0421] Initialization module, which initializes some settings needed in subsequent stages, including:

[0422] Electronic fence drawing unit: draw an electronic fence in the video scene;

[0423] Select key point unit: Select the key point type suitable for the scene;

[0424] Set the maximum number of detections unit: Set the maximum number of detections M, that is, the maximum number of calls to the pedestrian detection model; by setting this variable, the duration of the pedestrian detection phase can be indirectly changed;

[0425] The pedestrian recognition module is mainly responsible for filtering video segments for the pedestrian detection stage and notifying the pedestrian detection stage that there are pedestrian targets in the current video scene, including:

[0426] Extract and identify video frame unit: extract the current frame image from the acquired video stream;

[0427] Pedestrian recognition preprocessing unit: preprocess the image, including image scaling and pixel value standardization;

[0428] Pedestrian recognition model inference unit: input the preprocessed image data into the pedestrian recognition model with a small calculation, perform pedestrian recognition based on deep learning, and perform inference calculation to output the classification label; the pedestrian recognition model uses 0.4-0.5 as the decision threshold. If the model prediction value is greater than or equal to the decision threshold, the classification label is classified as 1, and if it is less than the decision threshold, the classification label is classified as 0;

[0429] Recognition result judgment unit: judge whether the pedestrian recognition model outputs a pedestrian classification label. If the classification label is 1, it is considered that there is a suspected pedestrian target in the current frame image and the pedestrian detection stage is entered; if the classification label is 0, there is no pedestrian target and pedestrian recognition continues;

[0430] The pedestrian detection module is the core part of the pedestrian intrusion detection algorithm of the present invention. The main task of this stage is to determine the position of the pedestrian in the video frame and predict the pedestrian bounding box, including:

[0431] Set the detection number unit: set the counting variable n to 0, which records the number of pedestrian detections, that is, the number of calls to the pedestrian detection model;

[0432] Extraction and detection video frame unit: extract the frame image at the current moment from the collected video stream;

[0433] Pedestrian detection preprocessing unit: preprocess the image, including image scaling and pixel value normalization;

[0434] Pedestrian detection model inference unit: input image data into the pedestrian detection model, perform inference calculation based on deep learning through the pedestrian detection model, and obtain the target bounding box of the current video frame after inference calculation; this step is the key to the pedestrian intrusion judgment algorithm, and the calculation amount of the pedestrian detection model is greater than that of the pedestrian recognition model; the calculation amount of this embodiment is relative, which is the relative size of the pedestrian detection model relative to the pedestrian recognition model;

[0435] Detection result judgment unit: judge whether the pedestrian detection model outputs a target bounding box. If the pedestrian detection model outputs a target bounding box, it is considered that there is a pedestrian target in the current frame image, and the intrusion judgment stage is entered. If the pedestrian detection model does not output a target bounding box, the number of detections is recorded;

[0436] Intrusion judgment module: The main task is to judge whether pedestrians have intruded into the electronic fence. The judgment method uses the key point judgment method instead of the common bounding box intersection method; including:

[0437] Recording detection times unit: adding 1 to the counting variable n, indicating that the number of pedestrian detections increases by one;

[0438] Detection times determination unit: determines whether the calculated variable n is less than the preset maximum detection times M. If the calculated variable n is less than the preset maximum detection times M, pedestrian detection is performed; if not, the pedestrian recognition stage is entered;

[0439] Intrusion judgment module:

[0440] Calculate key point unit: calculate the key point position in the target bounding box according to the selected key point type;

[0441] Key point determination unit: determines whether the key point is in the electronic fence; if the key point is in the designated electronic fence, an alarm is issued; if the key point is not in the electronic fence, the pedestrian detection stage is entered;

[0442] Alarm unit: issues prompts or warnings to pedestrian targets, or sends an alarm to the central console.

[0443] The preprocessing of the present invention also includes: data enhancement. The data enhancement of the present invention includes: data enhancement of random grayscale conversion, random haze synthesis, and random light spot addition.

[0444] The data enhancement of random grayscale conversion in the present invention is proposed for the dim vision in the mine. The camera used in the present invention will enable the night photography mode when the light is insufficient, and output grayscale images instead of color images. Therefore, the model needs to input a certain amount of grayscale images during training to adapt to the night photography mode of the camera and reduce the dependence on image color.

[0445] The pedestrian detection model inference unit of the present invention operates as follows: First, the pedestrian detection model extracts features from the input image, outputs multiple feature maps of different sizes, and then predicts bounding boxes on these feature maps. Among them, one feature map corresponds to one detection layer, and the pixels in each feature map are also called a grid. In the process of feature extraction, the convolutional neural network iteratively calculates the feature map through the convolution layer and the downsampling layer, and the size (i.e., resolution) of the feature map will continue to shrink, and the number of channels will continue to increase. Outputting multiple feature maps of different sizes refers to outputting multiple feature maps with a resolution of a fraction of the size of the original image. For example, the pedestrian detection model has 4 detection layers, and the resolutions of the feature maps in each detection layer are 1 / 8, 1 / 16, 1 / 32, and 1 / 64 of the original input image, respectively.

[0446] The present invention adopts recognition before detection, and uses a simple recognition model to quickly process video segments without pedestrians, reducing the number of calls to complex detection models and reducing the consumption of computing resources. The key point-based intrusion judgment method determines whether the pedestrian target has intruded by judging whether the key points in the pedestrian encirclement box are within the electronic fence, effectively reducing the misjudgment of intrusion caused by perspective projection, and adopts the key point judgment method to improve the accuracy and robustness of intrusion detection. Data enhancement method for complex mine environments, random grayscale conversion: for the dim vision in the mine, the present invention proposes a data enhancement algorithm for random grayscale conversion, which enhances the recognition or detection ability of the network model for night grayscale images by simulating grayscale images in night photography mode. Random haze synthesis: for the situation of mine dust diffusion in the mine, the present invention proposes a data enhancement algorithm for random haze synthesis to enhance the ability of the network model to resist haze interference. Random light spot addition: for the situation of strong light pollution in the mine, the present invention proposes a data enhancement algorithm for random light spot addition to enhance the ability of the network model to resist light spot interference.

[0447] Based on the above ideal embodiments of this application, the relevant staff can make various changes and modifications without departing from the technical concept of this application through the above description. The technical scope of this application is not limited to the content in the specification, and its technical scope must be determined according to the scope of the claims.

[0448] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0449] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0450] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0451] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

Claims

1. A method for detecting pedestrian intrusion in a mine environment, characterized in that: include: The initialization phase includes: Draw an electronic fence: Draw an electronic fence in the video scene; Select key point type: Select the key point type suitable for the scene; The pedestrian recognition stage includes: Extracting the current video frame, including extracting and identifying the video frame: extracting the current moment frame image from the acquired video stream; Preprocessing, including pedestrian recognition preprocessing: preprocessing the frame image; Pedestrian recognition model reasoning: The preprocessed image data is input into the pedestrian recognition model with low computational complexity for pedestrian recognition, and the reasoning calculation is performed to output the classification label; Determine the recognition result: If the classification label output by the pedestrian recognition model is a pedestrian, then enter the pedestrian detection stage, otherwise continue with pedestrian recognition; The pedestrian detection stage includes: Set the number of detections: Set the calculation variable n, which is used to record the number of pedestrian detections; Pedestrian detection model inference: Input the image data into the pedestrian detection model with large computational complexity, and infer the calculation to obtain the target bounding box of the current video frame. Determine the detection result: If the pedestrian detection model outputs a target bounding box, it enters the intrusion judgment stage, otherwise it records the number of detections; Enter the judgment phase, which includes: Calculate key points: Calculate the key point positions in the target bounding box according to the selected key point type; Determine key points: Determine whether the key points are within the designated electronic fence. If the key points are within the designated electronic fence, an alarm is triggered. If the key points are not within the designated electronic fence, the pedestrian detection phase is entered. Alarm: Issue prompts or warnings to pedestrian targets, or send an alarm to the central console.

2. The method for detecting pedestrian intrusion in a mine environment according to claim 1, characterized in that: Also includes: The initialization stage also includes setting the maximum number of detections: setting a variable M to the maximum number of calls to the pedestrian detection model; The pedestrian detection stage also includes: Record the number of detections: If the pedestrian detection model does not output a bounding box, the counting variable n is increased by 1, and the number of pedestrian detections increases by one; Determine the number of detections: Determine whether the counting variable n is less than the maximum detection number M. If the counting variable n is less than the maximum detection number, pedestrian detection is performed, otherwise the pedestrian recognition stage is entered.

3. The method for detecting pedestrian intrusion in a mine environment according to claim 1, characterized in that: The extracting of the current video frame also includes extracting the detection video frame: extracting the current moment frame image from the collected video stream; The pedestrian recognition preprocessing includes: image scaling and pixel value standardization; The preprocessing also includes pedestrian detection preprocessing set in the pedestrian detection stage and before the pedestrian recognition model inference, including: image scaling and pixel value normalization; The image scaling is proportional scaling, which includes: selecting the longest side of the height and width of the image; dividing the original target side length by the longest side of the image to calculate the scaling factor; multiplying the scaling factor by the shortest side of the image to calculate the target length of the shortest side of the image; scaling the image to scale the longest side of the image to the original target side length, and scaling the shortest side of the image to the calculated target length; subtracting the target length of the shortest side from the original target side length and dividing by 2 to calculate the width of the image edge that needs to be filled; filling the image edge with a fixed pixel value so that the filled image becomes a square image with the original target side length; The pixel value normalization includes: calculating the mean and standard deviation of each channel of all images in the training set, or directly using the pre-calculated pixel mean and standard deviation of the pre-training set; subtracting the pixel mean from each pixel of all images in the training set and dividing it by the pixel standard deviation; The pixel value normalization divides all pixels in the image by 255 to calculate a normalized pixel value, thereby adjusting the image data to a value range from 0 to 1.

4. The method for detecting pedestrian intrusion in a mine environment according to claim 1, characterized in that: The preprocessing also includes: data enhancement, which is used for image preprocessing in the training and testing stages of pedestrian recognition models and pedestrian detection models. The data enhancement includes: one or more of random grayscale conversion, random haze synthesis, and random light spot addition; the data enhancement of random grayscale conversion includes: Generate weight vector: Randomly generate a three-channel weight vector [w R ,w G ,w B ], which satisfies w R +w G +w B =1, which can be expressed as in, represents uniform distribution, a and b are the lower and upper limits of uniform distribution respectively; Generate weight matrix: with three-channel weight w R ,w G ,w B is the mean, with random number std W is the standard deviation, and randomly generates three-channel weight matrices W R ,W G ,W B , expressed mathematically as in, Represents normal distribution, μ, σ are the mean and standard deviation of normal distribution, std W According to statistics, it is set as a random number between 0 and 1, expressed as a mathematical formula: Calculate the noise-free grayscale image: Convert the color image to the grayscale image Gray, the calculation formula is Gray = W R *Red+W G *Green+W B *Blue, Among them, Red, Green, and Blue are three-channel images of color images; Find the maximum value: Find the maximum pixel value gray of the grayscale image Gray at this time max and minimum pixel value gray min , expressed mathematically as Among them, x, y represent the row and column index of the image, that is, the pixel coordinates of the image, and max x,y and min x,y Respectively represent the maximum and minimum values ​​at all pixel coordinates (x, y); Generate a proportional matrix: take 1 as the mean and random number std S is the standard deviation, randomly generates a three-channel ratio matrix s R ,S G ,S B , expressed mathematically as Among them, std S According to statistics, it is set as a random number between 0 and 0.03, which is expressed as a mathematical formula: Calculate the noisy grayscale image: Calculate the three-channel grayscale image Gray with noise deviation new , the calculation formula is Among them, Red new ,Green new ,Blue new They are new grayscale images Gray new Three-channel image; Standardized grayscale image: Grayscale image Gray new Standardize according to the maximum pixel value gray max _ and minimum pixel value gray min Using the maximum and minimum value normalization, the calculation formula is: Normalize grayscale image: Grayscale image pixels Gray std (x, y) is rounded and expressed as Gray rnd (x,y)=round(Gray std (x,y)), Among them, round(·) represents the rounding operation, and then the grayscale image pixel Gray rnd The range of (x,y) is limited to between 0 and 255, which can be expressed as The random haze synthesis includes: Random generation parameters: Randomly generate atmospheric light intensity A, whose value range is between 0 and 255, expressed by mathematical formula: The attenuation coefficient β is randomly generated, and the value range is set between 0 and 0.1, which can be expressed as follows: Randomly select a pixel coordinate in the image as the center of the haze (X H ,Y H ), expressed mathematically as Among them, Widht and Height are the width and height of the haze-free image J respectively; Determine the farthest pixel: Determine the distance from the center of the haze (X H ,Y H )The farthest pixel coordinate (X F ,Y F ), for a rectangular image, the farthest pixel coordinate (X F ,Y F ) must be one of the four vertex coordinates of the image. If X H Greater than Width / 2, then X F is 0, otherwise X F For Width, if Y F Greater than Height / 2, then X F is 0, otherwise X F is Height, which can be expressed as Calculate the size of the haze: Calculate the center of the haze (X H ,Y H ) and the farthest pixel coordinate (X F ,Y F )’s Euclidean distance dist F , the calculation formula is According to the farthest distance dist F Set the haze size, the calculation formula is: Calculate the distance matrix: Calculate the distance from each pixel (x, y) in the image to the center of the haze (X H ,Y H ) is calculated as follows: Use the above distance matrix dist to generate haze spreading from the center to the surrounding areas. If you use the inverted distance matrix dist R , then the haze is generated from the surrounding to the center, and the inversion distance of each pixel is dist R (x,y) is determined by the maximum distance dist F Subtract the original distance dist(x,y) to obtain the calculation formula: dist R (x,y)=dist F -dist(x,y), Calculate the transparency matrix: Calculate the propagation distance d(x,y) of the reflected light of each pixel in the haze. The calculation formula is d(x,y)=size-0.04*dist(x,y), Then calculate the transparency of each pixel t(x,y), the calculation formula is t(x,y)=e -β*d(x,y) , Where d(x,y) is the distance that the reflected light corresponding to each pixel (x,y) in the image passes through the haze to reach the image sensor, and β is called the attenuation coefficient; Calculate the haze image: First normalize the atmospheric light intensity A and image pixel J(x,y), the calculation formula is: Then calculate the normalized haze image pixel I(x,y) nml , the calculation formula is I(x,y) nml =J(x,y) nml *t(x,y)+A nml *(1-t(x,y)), Finally, the inverse normalized haze image pixel x(x,y) nml , the calculation formula is I(x,y)=I(x,y) nml *255.0; Normalize the haze image: Round off the haze image pixel I(x,y) and express it as follows: I(x,y) rnd =round(I(x,y)), Then the range of pixel I(x,y) is limited to between 0 and 255, which can be expressed as The random light spot addition includes: Random generation parameters: Randomly generate the center of the spot (X C ,Y C ), the value range is within the image size (Width, Height), and is expressed as The spot radius R is randomly generated, and the value range is set between 20 and 100, which can be expressed as follows: The randomly generated spot brightness L is set between 220 and 235, which can be expressed as follows: The blur size sigma is randomly generated, and the value range is set between one-quarter of the spot radius and one-half of the spot radius, which can be expressed as Set the overlay weight: Set the light spot effect image Effect fh The superposition weight Weight is set to 0.6, and then the spot brightness L before superposition is calculated using the superposition weight Weight PRE , the calculation formula is L PRE =Weight*L,Weight=0.6; Initialize effect image: Generate the initial spot effect image Effect init , initialize the effect image to an all-zero matrix of the same size as the original image Image, for the spot effect image Effect init For each pixel (x,y) in Effect init (x,y)=0; Generate a light spot effect: Create a light spot effect image Effect init Generate a circular light spot, the center of which is (X C ,Y C ), the radius is R, and the pixel value inside the circle is L PRE , expressed mathematically as Generate halo effect: Use Gaussian blur to generate the halo around the spot. The standard deviation of Gaussian blur is the blur size sigma. The filter size ksize′ of Gaussian blur is calculated based on the standard deviation sigma. The calculation formula is: ksize′=round(6*sigma+1), Ensure that the filter size ksize is a positive odd number. If the remainder of ksize′ divided by 2 is 0, it means that ksize′ is an even number, so add 1 to it to make it an odd number; if the remainder of ksize′ divided by 2 is 1, it means that ksize′ is an odd number, so keep it as the original value, which can be expressed as the mathematical formula: Among them, xmod2 means to find the remainder of x divided by 2, sigma is the standard deviation, ksize is the filter size, and the spot effect image Effect f Perform Gaussian blur, expressed as Effect fh =GaussianBlur(Effect f ,ksize,sigma), Calculate the mask image: Calculate the spot area mask Msk, first generate a spot effect image Effect fh The initial mask of the same size, for each pixel in the mask Mask, now has Mask(x,y)=0, For the light spot effect image Effect fh For each pixel in, if the pixel value Effect fh If (x, y) is greater than 0, the pixel value at the corresponding position in the mask is set to 1, otherwise it is set to 0, which can be expressed as Overlay effect image: Overlay light spot effect image Effect fh The superposition method of these two images is weighted summation. The weight of the original image is fixed at 1.

0. The spot effect image is fh The weight is Weight, and the result image after adding the spot effect is Result. The calculation formula is Result=Image*1.0+Effect fh *Weight; Normalize the result image: Round the result image pixels Result(x,y) to the nearest integer, which can be expressed as Result rnd (x,y)=round(Result(x,y)), Then the pixel Result rnd The range of (x,y) is limited to between 0 and 255, which can be expressed as Limit the brightness of the light spot: limit the maximum pixel value of the light spot area to the light spot brightness L. When the pixel value of the mask Mask(x,y) is 1, it means the Result at the corresponding position clip (x,y) is in the spot area, then judge the Result clip Is the pixel value of (x, y) greater than L? If yes, change the pixel value of the position to L, otherwise keep the original pixel value; when the pixel value of Mask(x, y) is 0, it means the Result of the corresponding position clip (x, y) is not in the spot area, no additional changes are made, and the mathematical formula is expressed as Among them, min(·) represents the minimum value operation.

5. The method for detecting pedestrian intrusion in a mine environment according to any one of claims 1 to 4, characterized in that: The pedestrian detection model reasoning includes: extracting features from an input image, outputting feature maps of different sizes, predicting a target bounding box on the feature map, and in the feature extraction process, using a convolutional neural network to iteratively calculate the feature map through a convolutional layer and a downsampling layer, reducing the size of the feature map, increasing the number of channels, and outputting multiple feature maps of different resolutions that are reduced in size ratio to the original image, pre-setting multiple anchor frames with different aspect ratios and no center position for each feature map according to the size, taking each pixel position of each feature map as the target center, and predicting different target bounding boxes based on different anchor frames.

6. The method for detecting pedestrian intrusion in a mine environment according to claim 5, characterized in that: The predicted target bounding box is the parameter of the predicted target bounding box, including the normalized center offset x i ,y i , anchor frame scaling ratio w i ,h i , target confidence p conf.i , category probability p cls1.i ,p cls2.i ,..., where i is the bounding box number of the detection layer, Let the number of categories to be detected be nc=C, then the number of parameters for each bounding box is no=5+nc, the height and width of the feature map in the detection layer are ny,nx respectively, the number of anchor boxes is na=3, the number of bounding boxes in this detection layer is na*ny*nx=N, and the number of bounding box parameters is na*ny*nx*no, then all target bounding boxes pred predicted by this detection layer are In the above formula, the target confidence p conf.N Indicates the probability that there is a target in the bounding box N, p clsC.N represents the probability that the target in the bounding box N is of the Cth category, The target center of the target bounding box is calculated based on the center offset. The target bounding box is generated at each pixel position of the feature map. The target center of each bounding box is the pixel coordinate when the bounding box is generated. The feature map is obtained by downsampling the input image. The reduced bounding box is detected from the feature map, and the bounding box is enlarged to a size suitable for the input image. The center offset is used to make it accurate to a decimal. The internal coordinate axes of the grid are the pixel coordinates of the target center (grid X ,grid Y ) as the origin, with the right and downward directions of the positive X and Y axes respectively, to further determine the exact position of the target center (bbox X ,bbox Y ), center bias X ,bias Y ) is the target center (bbox X ,bbox Y ) in the relative position of the coordinate axis inside the grid, which will be the same as the center coordinate (bbox X ,bbox Y )The two adjacent pixel coordinates (bbox X ±1,bbox Y ±1) is also considered as the target center, center bias X ,bias Y ) changes from [0,1.0] to [-0.5,1.5], and the predicted normalized center bias x i ,y i The value range is [0,1.0]. To get the real center bias X ,bias Y ) expands its value range, and the calculation formula is Then find the target center of the bounding box (bbox X ,bbox Y ), the calculation formula is In the above formula, scale is the magnification of the bounding box, which is obtained according to the reduction factor of the feature map in the detection layer compared to the original input image. The size of the target bounding box is calculated based on the preset anchor box. The anchor box is designed according to the size of the feature map in the detection layer. The larger the size of the feature map in the detection layer, the larger the size of the anchor box used. The anchor box is calculated, and the sizes of all target bounding boxes in the training set are clustered to generate a set of optimal anchor box sizes. During target detection, the preset anchor box is modified by size scaling to make it approach the real target bounding box, and the anchor box scaling ratio w is predicted. i ,h i , the value range is [0,1.0], and the size of the anchor box used by the bounding box i is anchor W ,anchor H , then the size of the target bounding box bbox W ,bbox H The calculation formula is From the above formula, we can see that the width and height of the target bounding box bbox W ,bbox H The maximum value of the anchor box width and height is anchor W ,anchor H 4 times, Determine the target category and target confidence p conf.i Used to determine whether there is a target in the bounding box i, the target confidence p conf.i If it is greater than the confidence threshold, it means that there is a target in the bounding box, otherwise it means that there is no target in the bounding box. The target refers to the target within the detection category range. The target outside the detection category range is regarded as the background by the pedestrian detection model. The pedestrian detection model predicts the series category probability p clsN.i ,p cls2.i ,..., is used to determine the category to which target i belongs. When there is a target in the bounding box i, the category corresponding to the largest category probability is selected as the category to which the target belongs.

7. The method for detecting pedestrian intrusion in a mine environment according to claim 6, characterized in that: The pedestrian detection uses non-maximum suppression as a post-processing algorithm. Multiple bounding boxes will be generated at the same target position, while the same target has only one bounding box. The predicted bounding boxes will overlap with each other and there will be redundancy. After the bounding boxes are predicted, non-maximum suppression is performed on these bounding boxes. Non-maximum suppression suppresses redundant bounding boxes and finds the best target bounding box, specifically including: Set threshold: Set the confidence threshold Thres Conf Intersection and union ratio threshold Thres IOU ; Filter: Sort all bounding boxes according to the target confidence, and discard those whose target confidence is less than the confidence threshold Thres Conf The bounding box of Best selection: select the bounding box with the highest target confidence and use it as the best bounding box; Discard: Calculate the intersection and union ratio of the best bounding box and other bounding boxes, and discard those whose intersection and union ratio is greater than the intersection and union ratio threshold Thres IOU Other bounding boxes of Repeat processing: Repeat the optimal selection step and the discarding step until all bounding boxes are processed.

8. The method for detecting pedestrian intrusion in a mine environment according to any one of claims 1 to 4, characterized in that: The target bounding box is a rectangular bounding box whose sides are parallel to the four sides of the image and is represented by four parameters. The bounding box has a confidence and a category label associated with it. The confidence is assigned by the pedestrian detection model, and the category label is predicted by the pedestrian detection model, indicating the type of the target in the bounding box. The bounding box is the bounding box of the target in the image. When the target in the image is reduced or enlarged, the bounding box is proportionally reduced or enlarged. When annotating an image, determine the upper left corner and the lower right corner of the target boundary to mark a target. There are two ways to represent the target bounding box. One is the center point plus width and height representation (x, y, w, h), (x, y) is the center point of the bounding box, w, h is the width and height of the bounding box; the other is the coordinate representation (x1, y1, x2, y2). (x1, y1) is the upper left corner of the bounding box, and (x2, y2) is the lower right corner of the bounding box. The two bounding box representation methods can be converted to each other. The calculation formula for converting the center point plus width and height representation method to the coordinate representation method is: The calculation formula for converting the coordinate representation to the center point plus width and height representation is: In the step of calculating the key points, when the pedestrian in the scene is standing or walking, the midpoint of the lower boundary of the target bounding box is selected as the key point; when it is necessary to prevent the pedestrian from climbing into or rolling into the alarm area, the midpoint of the upper boundary of the target bounding box is selected as the key point; when it is necessary to prevent the pedestrian from accidentally falling into or rolling into the alarm area, multiple key points are selected at different positions of the target bounding box, and the midpoints of the upper, lower, left and right boundaries of the target bounding box are selected as the key points. If the key point is located at the midpoint of the lower boundary of the bounding box, the key point coordinates (x K ,y K ) is calculated as Similarly, if the key point is located at the midpoint of the upper boundary of the bounding box, the key point coordinates (x K ,y K ) is calculated as If the key point is located at the middle of the left boundary of the bounding box, the key point coordinates (x K ,y K ) is calculated as If the key point is located at the middle of the right boundary of the bounding box, the key point coordinates (x K ,y K ) is calculated as 9. The method for detecting pedestrian intrusion in a mine environment according to any one of claims 1 to 4, characterized in that: The pedestrian recognition model and the pedestrian detection model adopt a deep learning algorithm of a convolutional neural network, including a convolution layer, a downsampling layer, and an activation function. The pedestrian recognition includes: extracting features from the input image to obtain multiple feature maps of different sizes, then compressing these two-dimensional feature maps into one-dimensional features, and finally using a fully connected layer to predict the category probability of the image; for each category to be recognized in the pedestrian recognition model, the output layer has a corresponding category probability, and the value range of all category probabilities is [0,1]. The model takes the category corresponding to the maximum category probability as the predicted image category. In the pedestrian recognition model reasoning, the pedestrian recognition model uses 0.4-0.5 as the decision threshold. If the model prediction value is greater than or equal to the decision threshold, the classification label is classified as 1, and if it is less than the decision threshold, the classification label is classified as 0. In the judgment and recognition result: if the classification label is 1, it is considered that there is a suspected pedestrian target in the current frame image, and the pedestrian detection model with large computational complexity is started to perform pedestrian detection; if the classification label is 0, the pedestrian recognition model continues to run for pedestrian recognition.

10. A system for detecting pedestrian intrusion in a mine environment, characterized in that: include Initialization module, including: Electronic fence drawing unit: draw an electronic fence in the video scene; Select key point type unit: Select the key point type suitable for the scene; Pedestrian recognition module, including: Extracting the current video frame unit, including extracting and identifying the video frame unit: extracting the current moment frame image from the collected video stream; A preprocessing unit, including a pedestrian recognition preprocessing unit: preprocessing the frame image; Pedestrian recognition model inference unit: inputs the preprocessed image data into the pedestrian recognition model with low computational complexity for pedestrian recognition, performs inference calculation and outputs classification labels; Identification result judgment unit: If the classification label output by the pedestrian recognition model is a pedestrian, then the pedestrian detection phase is entered, otherwise pedestrian recognition continues; Pedestrian detection module, including: Set the detection number unit: set the calculation variable n, which is used to record the number of pedestrian detections; Pedestrian detection model inference unit: inputs image data into the pedestrian detection model with large computational complexity, and obtains the target bounding box of the current video frame through inference calculation. Detection result judgment unit: If the pedestrian detection model outputs a target bounding box, it enters the intrusion judgment module, otherwise the number of detections is recorded; The intrusion judgment module includes: Calculate key point unit: calculate the key point position in the target bounding box according to the selected key point type; determine key point unit: determine whether the key point is in the designated electronic fence. If the key point is in the designated electronic fence, it will enter the alarm unit; if the key point is not in the designated electronic fence, it will enter the pedestrian detection module; Alarm unit: issues prompts or warnings to pedestrian targets, or sends an alarm to the central console.

Images