Real-time statistical system and method for human flow in closed mine based on deep learning

Abstract

The application provides a kind of based on deep learning's stop production mine people flow real-time statistics system and method, it is related to image recognition technical field, the system includes: target detection module and multi-target tracking module;Target detection module includes multi-scale enhancement module, the local information and global information of the image to be measured are extracted using multi-scale enhancement module, and the target feature of the image to be measured is generated, so as to enhance the scale and ability of feature extraction;The image to be measured is generated based on the real-time monitoring video data collected in stop production mine, and the multi-target tracking module is used to match the target feature with the image frame in real-time monitoring video data, determine the target image existing in real-time monitoring video, so as to count the real-time people flow condition of stop production mine, so as to improve the target matching precision.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image recognition, in particular to a real-time people flow statistical system and method for closed mines based on deep learning. BACKGROUND

[0002] In order to effectively prevent illegal production and operation of coal mines and curb safety accidents, according to the requirements of mine safety production planning, strict control measures are taken for closed and rectified mines, and illegal subcontracting of underground backout projects is strictly prohibited. For long-term closed and production coal mines, it is strictly forbidden to open secretly during the day and at night. Under such control background, in order to further implement the safety management of closed mines, a large amount of manpower and material resources need to be invested. The traditional mine supervision method is mainly dominated by people, which has the problems that the professional of law enforcement personnel cannot completely match the professional involved in coal mines, and the examination mechanism is not perfect.

[0003] With the rapid development of machine vision technology, using computer vision to replace human eyes has become a trend of intelligentization in various industries. The existing people flow statistical model uses yolov5 for training and detection, uses simple distance tracking logic for tracking to realize counting function, uses Haar features to extract target, uses Adaboost algorithm to obtain a head detection classifier, or uses Histogram of Oriented Gradient (HOG) and Support Vector Machine (SVM) to detect the head and shoulder of the target block, and counts the number of pedestrians passing through the scene design line by means of the direction information of the moving target and the number of pedestrians contained in the target block. However, the existing people flow statistical method has the problems of poor feature extraction capability and low target matching precision. SUMMARY

[0004] Therefore, the present application aims to provide a real-time people flow statistical system and method for closed mines based on deep learning to alleviate the above problems.

[0005] In order to achieve the above purpose, the technical solutions adopted by the embodiments of the present application are as follows:

[0006] In a first aspect, the embodiments of the present application provide a real-time people flow statistical system for closed mines based on deep learning, which comprises a target detection module and a multi-target tracking module.

[0007] The target detection module comprises a multi-scale enhancement module, which is used to extract local information and global information of the to-be-tested image and generate target features of the to-be-tested image. The to-be-tested image is generated based on the collected real-time monitoring video data of the closed mine.

[0008] The multi-target tracking module is configured to match the target feature with image frames in the real-time monitoring video data, and determine that the target image exists in the real-time monitoring video, so as to count the real-time flow of the closed mine.

[0009] In some embodiments, the method further comprises: an image acquisition module; the image acquisition module is configured to acquire real-time monitoring video data of the closed mine, and send the real-time monitoring video data to the target detection module based on a time sequence; the real-time monitoring video data comprises a plurality of continuous frames of test images.

[0010] In some embodiments, the multi-scale enhancement module comprises a feature integration sub-module, and the feature integration sub-module comprises a first branch and a second branch; the first branch is configured to extract local information of a feature; and the second branch is configured to extract global information of the feature.

[0011] In some embodiments, the first branch comprises three cascaded hollow convolution blocks; each hollow convolution block comprises a first convolution layer, a hollow convolution layer, and a second convolution layer; the size of the first convolution layer is 1*1, the size of the second convolution layer is 3*3, and the size of the hollow convolution layer is 3*3; the second branch comprises a third convolution layer and a feature extractor encoding layer; and the size of the third convolution layer is 3*3.

[0012] In some embodiments, the multi-scale enhancement module further comprises a fusion sub-module, and the fusion sub-module is configured to fuse the local information output by the first branch and the global information output by the second branch, and generate a target feature of the test image.

[0013] In a second aspect, an embodiment of the present application provides a method for real-time counting of flow of people in a closed mine based on deep learning, comprising:

[0014] extracting local information and global information of a test image by using a multi-scale enhancement module, and generating a target feature of the test image; the test image is generated based on acquired real-time monitoring video data of the closed mine;

[0015] matching the target feature with image frames in the real-time monitoring video data by using a multi-target tracking module, and determining that the target image exists in the real-time monitoring video, so as to count the real-time flow of people in the closed mine.

[0016] In some embodiments, the method further comprises: acquiring real-time monitoring video data of the closed mine by using an image acquisition module, and sending the real-time monitoring video data to the target detection module based on a time sequence; and the real-time monitoring video data comprises a plurality of continuous frames of test images.

[0017] In some embodiments, the multi-scale enhancement module comprises a feature integration submodule and a fusion submodule; the feature integration submodule comprises a first branch and a second branch, the first branch is configured to extract local information of features, and the second branch is configured to extract global information of features; the fusion submodule is configured to fuse the local information output by the first branch and the global information output by the second branch, and generate target features of the image to be tested.

[0018] In a third aspect, an electronic device is provided, which comprises a memory and a processor, the memory stores a computer program capable of running on the processor, and the processor implements the steps of the method of any one of the first aspect when executing the computer program.

[0019] In a fourth aspect, a computer readable storage medium is provided, which stores computer executable instructions, and the computer executable instructions cause a processor to execute the method of any one of the first aspect when the computer executable instructions are invoked and executed by the processor.

[0020] The application provides a deep learning-based real-time human flow statistical system and method for a closed mine, which comprises a target detection module and a multi-target tracking module; the target detection module comprises a multi-scale enhancement module, which is used to extract local information and global information of an image to be tested, and generate target features of the image to be tested, so as to enhance the scale and capability of feature extraction; the image to be tested is generated based on real-time monitoring video data collected from the closed mine, and the multi-target tracking module is used to match the target features with image frames in the real-time monitoring video data, determine the target image existing in the real-time monitoring video, and count the real-time human flow of the closed mine, so as to improve the target matching accuracy. BRIEF DESCRIPTION OF DRAWINGS

[0021] In order to more clearly illustrate the specific embodiments of the application or the technical solutions in the prior art, the drawings needed in the specific embodiments or the prior art description will be briefly introduced as follows. Obviously, the drawings in the following description are some embodiments of the application, and those skilled in the art can also obtain other drawings according to these drawings without creative labor.

[0022] Figure 1 A structural schematic diagram of a deep learning-based real-time human flow statistical system for a closed mine provided by an embodiment of the application is shown in the figure.

[0023] Figure 2 A structural schematic diagram of a multi-scale enhancement module provided by an embodiment of the application is shown in the figure.

[0024] Figure 3A structural schematic diagram of a feature integration sub-module provided for an embodiment of the present application is shown in the figure.

[0025] Figure 4 A result schematic diagram of feature extraction provided for an embodiment of the present application is shown in the figure.

[0026] Figure 5 A count schematic diagram provided for an embodiment of the present application is shown in the figure.

[0027] Figure 6 A structural schematic diagram of an electronic device provided for an embodiment of the present application is shown in the figure. DETAILED DESCRIPTION

[0028] In order to make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the drawings can be arranged and designed in various different configurations.

[0029] Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the claimed present application, but only represents selected embodiments of the present application. All other embodiments obtained by those of ordinary skill in the art based on the embodiments in the present application without creative labor are within the scope of protection of the present application.

[0030] It should be noted that: similar labels and letters represent similar items in the following drawings, therefore, once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings.

[0031] In order to effectively prevent illegal production operation of coal mines and curb the occurrence of safety production accidents, according to the requirements of mine safety production planning, the mine safety informatization construction project proposes to perfect the framework of mine safety production informatization standard specification system, standardize and guide the construction of mine safety production comprehensive information system. Strictly control the measures of stopping production and closing down the mine, and strictly prohibit illegal subcontracting of underground backout engineering; for long-term closed and suspended coal mines, strict prevention of opening secretly during the day and closing secretly at night; for technical transformation coal mines, relevant agencies should strictly check against the design. Under such control background, in order to further implement the safety management of closed mines, a large amount of manpower and material resources need to be invested.

[0032] The traditional mine supervision method is mainly human-led, which has problems such as that the professional of law enforcement personnel cannot completely match the professional involved in the coal mine, the law enforcement personnel are not enough, the responsibility is not clear, and the law enforcement personnel examination mechanism is not perfect. Under the rapid development of technologies such as 'Internet +', big data and Internet of Things, researching supervision and inspection theory, working mode, coordination of supervision and inspection and participation of third-party institutions under the background of 'Internet +' is the future development direction.

[0033] With the rapid development of machine vision technology, using computer vision to replace the human eye has become a trend of intelligentization in various industries. Existing people flow statistical models or use yolov5 for training and detection, and use simple distance tracking logic for tracking to realize counting function, or use Haar features of extracted targets, use Adaboost algorithm to obtain a head detection classifier, or use HOG and SVM to detect the head and shoulder of a person in a target block, and count the pedestrians passing through the scene design line by means of the direction information of the moving target and the number of pedestrians contained in the target block. However, the existing people flow statistical method has problems of poor feature extraction capability and low target matching precision.

[0034] Based on this, the embodiment of the present application provides a kind of based on deep learning's stop production mine people flow real-time statistical system and method, to alleviate the feature extraction capability poor, target matching precision low problem existing in prior art.

[0035] For the convenience of understanding this embodiment, first of all, a kind of based on deep learning's stop production mine people flow real-time statistical system disclosed by the embodiment of the present application is introduced in detail, referring to Figure 1 The structure diagram of a kind of based on deep learning's stop production mine people flow real-time statistical system is shown in the figure, the system includes: target detection module 110 and multi-target tracking module 120.

[0036] In this embodiment, the target detection module 110 includes a multi-scale enhancement module, which is used to extract local information and global information of the test image and generate target features of the test image. The test image is generated based on the collected real-time monitoring video data of the stop production mine.

[0037] The multi-target tracking module 120 is used to match the target features with the image frames in the real-time monitoring video data, determine the existence of target portraits in the real-time monitoring video, so as to count the real-time people flow of the stop production mine.

[0038] In one embodiment, the deep learning-based stop production mine people flow real-time statistical system can also include an image acquisition module 130; the image acquisition module is used to acquire real-time monitoring video data of the stop production mine, and send the real-time monitoring video data based on time sequence to the target detection module; the real-time monitoring video data includes a plurality of continuous frame test images.

[0039] The data input of the system is completed by the image acquisition module. The video monitoring data of the stopped mine portal is collected in real time by the image acquisition module in advance, images are formed, and the images are transmitted to the target detection module. The data format of the transmitted data can be an image format or a video stream format. The transmitted data is the data input of the entire system.

[0040] The hardware corresponding to the target detection module can be a domestic embedded neural network processing unit (NPU). The NPU adopts a “data-driven parallel computing” architecture and is particularly good at processing massive multimedia data such as videos and images. The domestic NPU has the advantages of high performance, low power consumption, and small size, and can be deployed on many cameras, unmanned aerial vehicles, and other devices.

[0041] The target detection module is usually based on Yolov5. The original Yolov5 structure is: “cspdarknet53+spp (main stem) + panet (neck) + yolo head (head)”. The cspdarknet53 is a main stem network for extracting features, and the spp is connected to the last layer of the main stem network to form multi-scale features. The panet has a bidirectional feature pyramid structure. First, the high-level features are propagated from top to bottom to enhance the bottom-level features. This part mainly transfers the semantic information in the high-level features to the bottom-level features to increase the detection performance of small and medium-sized objects (because in target detection, it is generally believed that shallow features contain a large amount of shape, texture, and other detailed information and have a small receptive field, so they are suitable for detecting small and medium-sized objects; high-level features have a large receptive field, so they are suitable for detecting large-sized objects. However, the semantic information in the shallow features is insufficient, resulting in a large number of false positives and missed detections of small and medium-sized objects. Therefore, introducing semantic information from high-level features into bottom-level features will help improve the detection performance of small and medium-sized objects). Second, the bottom-level features are propagated from bottom to top to enhance the high-level features to improve the positioning ability of the high-level features. The yolo head mainly uses a convolution layer with a convolution kernel of 3 to classify and locate the target.

[0042] However, in this application, the feature extraction capability of the spp module in yolo5 is not sufficient. If the spp cannot deeply mine the semantic information contained in the high-level features of the main stem network, the enhancement of the bottom-level features in the top-down branch of the panet will be limited. Therefore, in order to obtain more expressive hierarchical features, the spp part is replaced by a multi-scale enhancement module (MSEM) in this embodiment.

[0043] In an embodiment, the multi-scale enhancement module comprises a feature integration submodule, the feature integration submodule comprises a first branch and a second branch, the first branch is used to extract local information of features, and the second branch is used to extract global information of features.

[0044] As a specific example, refer to Figure 2 As shown in a structural schematic diagram of a multi-scale enhancement module, input features are features output by a last layer of a backbone network, and a basic component unit of the multi-scale enhancement module (MSEM) is a global and local integration module (GLIM).

[0045] In an embodiment, the first branch comprises three cascaded hollow convolution blocks; the hollow convolution block comprises a first convolution layer, a hollow convolution layer and a second convolution layer; the size of the first convolution layer is 1*1, the size of the second convolution layer is 3*3, and the size of the hollow convolution layer is 3*3; the second branch comprises a third convolution layer and a feature extractor encoder layer (Transformer-encoder); and the size of the third convolution layer is 3*3.

[0046] In an embodiment, the multi-scale enhancement module further comprises a fusion submodule, the fusion submodule is used to fuse local information output by the first branch and global information output by the second branch, and generate target features of the image to be tested.

[0047] As a specific example, refer to Figure 3 As shown in a structural schematic diagram of a feature integration submodule, the module comprises an upper branch and a lower branch, the upper branch is used to extract local information of features. In the branch, three hollow convolution blocks with different hollow rates are cascaded. In each hollow convolution block, features are first passed through a 1*1 convolution layer, then passed through a 3*3 hollow convolution with a hollow rate d, and finally passed through a normal 3*3 convolution. It should be noted that in this hollow convolution block, a residual structure is adopted to add the input and the output to obtain the final output. The lower branch is used to extract global information of features. As can be seen from the figure, features are first sent to a normal convolution layer, and then sent to a Transformer-encoder to extract long-distance dependency relationships between each pixel, i.e., global features. The features in the upper and lower branches are spliced and sent to a 1*1 convolution for fusion, so as to obtain the final features of the branch. Finally, the final features of the three branches are spliced and sent to a 1*1 convolution for fusion, and the obtained features are taken as the final output of the MSEM module. The specific parameters involved in the three GLIM branches in the MSEM are shown in Table 1.

[0048] Table 1: Parameter list in MSEM

[0049]

[0050] Through the above structural design, the semantic information of the features obtained by the MSEM is stronger than that of the features in the spp, so that the features can be enhanced in the top-down propagation process to obtain features with stronger expression ability, thereby improving the detection effect of the multi-scale target.

[0051] Generally, multi-target tracking is regarded as a data association problem, and association is performed across detection results in a sequence of video frames. In order to solve the data association problem, the tracker uses various methods to model the motion process and the appearance features of the moving target. In the embodiment, the multi-target tracking algorithm used is DeepSORT, which is the predecessor of SORT, which stands for Simple Online And Realtime Tracking. The biggest feature of DeepSort is to add appearance information, which borrows the ReID field model to extract features and reduces the number of ID switches. The processing flow of DeepSORT for each frame includes: (1) the detector obtains the bbox; (2) generate detections; (3) Kalman filter prediction; (4) use the Hungarian algorithm to match the predicted tracks and the detections in the current frame (cascade matching and IOU matching); (5) Kalman filter update.

[0052] For DeepSORT, a linear constant velocity model independent of other objects and camera motion is used to calculate the inter-frame displacement of each object. The SORT algorithm used by each target uses an 8-dimensional state space: x = [u, v, r, h, x°, y°, r°, h°] T , where (u, v) represents the center point of the track box, the aspect ratio r, the height h, and the relative speed in the image coordinates. After associating the detected target, the state of the target is updated using the detected bounding box, and the velocity component is optimized and solved through the Kalman filter framework. If there is no detection associated with the target, its state is simply predicted using a linear velocity model without correction. When associating frame-by-frame data, first, predict the new position of each target in the current frame to estimate its bounding box shape; second, calculate the assignment cost matrix from the intersection over union of each detection and all existing predicted bounding boxes; finally, use the Hungarian algorithm to optimize and solve the assignment.

[0053] However, in the above description process, only by calculating the intersection over union between each detection and all existing predicted bounding boxes is a rough matching method. This method does not consider the information inside the detection box when matching, which is easy to cause id assignment error. Therefore, in the actual matching process, the feature extraction module (FEM) proposed in this embodiment is used to extract the features of the two detection boxes respectively, and the Euclidean distance between the features is constrained to measure the similarity of the content inside the detection box. The result of FEM is shown in Figure 4 As can be seen from Figure 4 , the module extracts local features and global features of the detection box for corresponding matching. The specific operation is as follows:

[0054] The patch corresponding to the detection box is used to extract the global feature by ResNet50, and then the patch is divided into a 3*3 grid, and VGG16 is used to extract the feature of each grid (i.e. local feature). In this way, one detection box corresponds to 10 features. The Euclidean distance of the global features of the two detection boxes is calculated, and the features of each small block corresponding in space are also calculated. Finally, the 10 Euclidean distances are added to obtain the final distance D, and the matching result of D and IOU is multiplied. Set the threshold value to 0.45, if greater than the threshold value, the matching is successful; if less than the threshold value, it proves that the two detection boxes are not matched successfully.

[0055] In one embodiment, the above-mentioned deep learning-based real-time human flow statistics system of closed mine can also include a counting module, which uses a vector intersection discrimination method to count the number of people, thereby realizing the number of people.

[0056] In the implementation part of the counting function, first, a preset counting line is determined in the image, and the vector formed by the motion of the pedestrian between different frames is calculated with the vector formed by the counting line to judge whether it intersects, thereby realizing counting. Referring to the counting principle diagram shown in Figure 5 , the specific description is as follows:

[0057] Suppose the directed line segment of the counting line drawn in the image is The directed line segment formed between the front and back frames of the pedestrian is In theory, if and intersect, then point C and point D are on the two sides of the vector , and point A and point B are on the two sides of the vector . Then, if the following conditions are met:

[0058]

[0059] then the pedestrian target crosses the line from bottom to top, and the number of people entering increases by 1; otherwise, it crosses the line from top to bottom, and the number of people leaving increases by 1.

[0060] The embodiment provides a deep learning-based real-time personnel flow statistical system for a closed mine, which comprises a target detection module and a multi-target tracking module; the target detection module comprises a multi-scale enhancement module, which is used for extracting local information and global information of a to-be-detected image and generating target features of the to-be-detected image, so as to enhance the scale and capability of feature extraction; the to-be-detected image is generated based on collected real-time monitoring video data of the closed mine; the multi-target tracking module is used for matching the target features with image frames in the real-time monitoring video data, determining that there are target images in the real-time monitoring video, and facilitating the statistics of real-time personnel flow of the closed mine, so as to improve the target matching accuracy.

[0061] In addition, the embodiment of the present application also provides a deep learning-based real-time personnel flow statistical method for a closed mine, which is mainly applied to the field of coal mine safety and closed coal mine supervision. Supervising the coal mines that have been closed for a long time, are under construction or are being closed is an important work of mine safety supervision. The method provided by the present application mainly combines the closed mine entrance monitoring video information, image features, NPU computing units and the like to construct a complete real-time personnel flow statistical system for the closed mine.

[0062] Next, a deep learning-based real-time personnel flow statistical system method for a closed mine disclosed by the embodiment of the present application will be described in detail. The method can be executed by an electronic device and mainly comprises the following steps S110 to S120:

[0063] S110: a multi-scale enhancement module is used to extract local information and global information of a to-be-detected image and generate target features of the to-be-detected image; the to-be-detected image is generated based on collected real-time monitoring video data of the closed mine;

[0064] S120: a multi-target tracking module is used to match the target features with image frames in the real-time monitoring video data, determine that there are target images in the real-time monitoring video, and facilitate the statistics of real-time personnel flow of the closed mine.

[0065] In one embodiment, the method further comprises: collecting real-time monitoring video data of the closed mine by using an image collection module and sending the real-time monitoring video data to the target detection module based on time sequence; the real-time monitoring video data comprises a plurality of continuous frames of to-be-detected images.

[0066] In one embodiment, the multi-scale enhancement module comprises a feature integration sub-module and a fusion sub-module; the feature integration sub-module comprises a first branch and a second branch, the first branch is used to extract local information of features, and the second branch is used to extract global information of features; the fusion sub-module is used to fuse the local information output by the first branch and the global information output by the second branch, and generate target features of the to-be-detected image.

[0067] The main difference between the multi-target tracking method in this embodiment and the DeepSORT algorithm is that when two boxes are matched, DeepSORT only considers the IOU of the two boxes, while our method also considers the features inside the boxes. The FEM is used to propose features of the two boxes to optimize the matching process of the boxes.

[0068] The application uses a high-definition camera to collect mine entrance information, uses a target detection algorithm to realize person recognition, and uses a tracking algorithm to realize tracking of personnel entering the mine. The method is easy to promote, convenient to implement, and has great application value.

[0069] As a specific example, the embodiment combines a target detection algorithm and a tracking algorithm, uses the output of the target detection algorithm as the input of the tracking algorithm, uses the output of the tracking algorithm as the input of the counting model, and finally outputs the number of people entering and leaving the mine from the counting model. The relevant evidence video and the number of statistics are sent to the intelligent supervision system, and when the number of people exceeds the limit, an alarm information is sent to the relevant management personnel, and the management personnel make relevant operations according to the alarm information. The following briefly introduces the related algorithms and the combined process.

[0070] 1) A multi-scale context enhancement module (MSEM) is proposed to improve yolo5. In this embodiment, yolo5 algorithm is used as the basic target detection algorithm. The multi-scale feature extraction part in yolo5 is improved to help improve the detection effect of objects of different scales. Compared with the simple spp in yolo5, the MSEM module in this paper can extract more rich high-level semantic information. The MSEM module fuses local features and global features, and designs multiple branches of different scales to fully extract the multi-scale expression ability of features, thereby enhancing the detection effect of the detection model for objects of different scales, which is more conducive to the subsequent tracking algorithm.

[0071] 2) Improve the IOU matching process of the target tracking algorithm. The DeepSORT multi-target tracking algorithm is used in this embodiment. One of the steps in this algorithm is to calculate the IOU between the detection box output by the detection algorithm and the target box predicted by the Hungarian algorithm to assign an id to the target. However, this method only uses the spatial position information of the object, without considering the information inside the box. In this case, id assignment errors may occur when the object is occluded or overlapped, causing tracking errors. Therefore, this paper considers the content inside the box in addition to the calculation of IOU, thereby avoiding such situations.

[0072] 3) Designed a simple and efficient people counting module. In order to ensure the real-time performance of the system, this embodiment does not choose to use a complex people counting module based on deep learning for counting. Instead, this paper designs a method based on counting line, and combines with tracking algorithm, so as to realize the people counting function.

[0073] The application is based on target detection algorithm, combined with target tracking algorithm using image recognition technology, the running hardware can be Atlas 200 intelligent terminal, proposes an intelligent mine management scheme for reducing coal mine management cost and reducing human management failure. The target detection algorithm and tracking algorithm are combined at the mine mouth of the closed mine, the access counting is realized, the evidence video is uploaded in real time, the overrun warning is realized, and the domestic NPU is used for image calculation to obtain the detection result, which has the effects of high performance, low power consumption, small size and low cost.

[0074] The application embodiment also provides an electronic device, specifically, the electronic device includes a processor and a storage device; the storage device stores a computer program, and the computer program executes the method of any one of the above embodiments when the processor runs.

[0075] Figure 6 A structural schematic diagram of an electronic device provided by the embodiment of the application is provided, and the electronic device 400 includes a processor 40, a memory 41, a bus 42 and a communication interface 43, the processor 40, the communication interface 43 and the memory 41 are connected through the bus 42; the processor 40 is used for executing the executable modules stored in the memory 41, such as a computer program.

[0076] The memory 41 may include a high-speed random access memory (RAM, Random Access Memory) and may also include a non-volatile memory, such as at least one disk memory. The communication connection between the system network element and at least one other network element is realized through at least one communication interface 43 (which can be wired or wireless), and the Internet, a wide area network, a local area network, a metropolitan area network, etc. can be used.

[0077] The bus 42 can be an ISA bus, a PCI bus or an EISA bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For the convenience of representation, Figure 6 Only one bidirectional arrow is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0078] The memory 41 is configured to store a program, and the processor 40 executes the program after receiving an execution instruction. The method performed by the device for defining a flow process according to any of the embodiments of the present application can be applied to the processor 40 or implemented by the processor 40.

[0079] The processor 40 can be an integrated circuit chip with a processing capability of signals. In the implementation process, each step of the above method can be completed by the integrated logic circuit of hardware in the processor 40 or the instruction in the form of software. The processor 40 described above can be a general processor, including a central processing unit (CPU), a network processor (NP), etc.; can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. Each method, step and logic block diagram disclosed in the embodiments of the present application can be implemented or executed. The general processor can be a microprocessor or the processor can also be any conventional processor. The steps of the method disclosed in combination with the embodiments of the present application can be directly embodied as a hardware code processor for execution, or a combination of hardware and software modules in the code processor for execution. The software module can be located in a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register or other mature storage medium in the art. The storage medium is located in the memory 41, and the processor 40 reads the information in the memory 41 and combines the hardware to complete the steps of the above method.

[0080] Corresponding to the above method, the embodiment of the present application further provides a computer readable storage medium, the computer readable storage medium stores machine executable instructions, when the processor calls and runs the computer executable instructions, the computer executable instructions make the processor run the steps of the above method.

[0081] In the embodiments of the present application, it should be understood that the disclosed apparatus and method can be implemented in other manners. The embodiments described above are merely exemplary, for example, the division of the units is only a logical function division, and there can be another division manner in actual implementation; for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed mutual couplings or direct couplings or communication connections can be indirect couplings or communication connections through some interfaces, and electrical, mechanical or other forms.

[0082] The units described as separate components can or can not be physically separate, and the components displayed as units can or can not be physical units, i.e., can be located in one place, or can be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purposes of the embodiments of the present application.

[0083] In addition, each functional unit in the embodiments of the present application can be integrated in one processing unit, or each unit can be a physically independent unit, or two or more units can be integrated in one unit.

[0084] If the functions are implemented in the form of software function units and sold or used as independent products, they can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application, in essence or the parts that make contributions to the prior art, or parts of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes a plurality of instructions for causing a computer device (which can be a personal computer, an electronic device, or a network device, etc.) to execute all or part of the steps of the methods described in the embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0085] It should be noted that: similar reference numerals and letters in the drawings represent similar items, and therefore, once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings. In addition, the terms "first", "second", "third" and the like are used only to distinguish descriptions, and cannot be understood as indicating or implying relative importance.

[0086] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, and are not intended to limit the present application; although the present application has been described in detail with reference to the above embodiments, those skilled in the art should understand that the technical solutions recorded in the above embodiments can be modified, or some or all of the technical features can be replaced by equivalents; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A deep learning-based real-time statistical system for human flow in a closed mine, characterized by, Comprise: Target detection module, multi-target tracking module and counting module; The target detection module comprises a multi-scale enhancement module, which is used to extract local information and global information of the to-be-detected image and generate target features of the to-be-detected image; The multi-scale enhancement module comprises a feature integration submodule and a fusion submodule, the feature integration submodule comprises a first branch and a second branch, the first branch extracts local information through three cascaded hollow convolution blocks, and the second branch extracts global information through a convolution layer and a Transformer feature extractor encoding layer; the fusion submodule fuses the local information and the global information to generate target features; the to-be-detected image is generated based on the collected real-time monitoring video data of the closed mine; Wherein, the multi-scale enhancement module comprises three local and global feature integration modules GLIM1, GLIM2 and GLIM3; the multi-scale enhancement module is also used for splicing and fusing the final features output by each of the local and global feature integration modules, and the obtained features are taken as the final output of the multi-scale enhancement module; The multi-target tracking module is used to match the target features with image frames in the real-time monitoring video data, determine that there are target images in the real-time monitoring video, so as to count the real-time flow of the closed mine; The counting module is used to determine a preset counting line in the image, calculate whether the vector generated by the motion of the pedestrian between different frames intersects with the vector formed by the counting line, so as to realize counting.

2. The system of claim 1, wherein, Also include: Image acquisition module; The image acquisition module is used to collect real-time monitoring video data of the closed mine, and send the real-time monitoring video data based on time sequence to the target detection module; the real-time monitoring video data comprises a plurality of continuous frames of to-be-detected images.

3. The system of claim 2, wherein, The first branch comprises three cascaded hollow convolution blocks; the hollow convolution block comprises a first convolution layer, a hollow convolution layer and a second convolution layer; the size of the first convolution layer is 1*1, the size of the second convolution layer is 3*3, and the size of the hollow convolution layer is 3*3; The second branch comprises a third convolution layer and a feature extractor encoding layer; the size of the third convolution layer is 3*3.

4. A deep learning-based real-time statistical method for human flow in a closed mine, characterized by, The method is applied to the deep learning-based real-time flow counting system of the closed mine according to any one of claims 1 to 3, and the method comprises: Extracting local information and global information of the to-be-detected image by using a multi-scale enhancement module, and generating target features of the to-be-detected image; the to-be-detected image is generated based on the collected real-time monitoring video data of the closed mine; The multi-scale enhancement module comprises a feature integration submodule and a fusion submodule; the feature integration submodule comprises a first branch and a second branch, the first branch is used to extract local information of features, and the second branch is used to extract global information of features; the fusion submodule is used to fuse the local information output by the first branch and the global information output by the second branch, to generate target features of the to-be-detected image; The multi-scale enhancement module comprises three local and global feature integration modules GLIM1, GLIM2 and GLIM3; the multi-scale enhancement module is further configured to splice and fuse final features output by the local and global feature integration modules, and take the obtained features as final output of the multi-scale enhancement module; The multi-target tracking module is used to match the target features with image frames in the real-time monitoring video data, so as to determine that a target person exists in the real-time monitoring video, and to facilitate the counting of real-time people flow of the closed mine. The counting module is used to determine a preset counting line in the image, to calculate whether a vector generated by motion of a pedestrian between different frames intersects with a vector formed by the counting line, and to realize counting.

5. The method of claim 4, wherein, Further comprising: The image acquisition module is used to acquire real-time monitoring video data of the closed mine, and to send the real-time monitoring video data to the target detection module based on time sequence. The real-time monitoring video data comprises a plurality of continuous frames of to-be-detected images.

6. An electronic device comprising a memory, a processor, the memory having stored therein a computer program executable on the processor, characterized in that, The processor executes the computer program to realize steps of the method in any one of claims 4 to 5.

7. A computer readable storage medium characterized by The computer readable storage medium stores machine executable instructions, and the machine executable instructions, when called and executed by the processor, cause the processor to execute the method in any one of claims 4 to 5.

Images