Mine car overhanging detection method and device based on target detection and multi-target tracking

Abstract

The present application belongs to the technical field of mine car over-hanging transportation detection, and discloses a mine car over-hanging detection method and device based on target detection and multi-target tracking. The method comprises: frame sampling of a transportation video to generate an image queue; inputting the image queue into a target detection model to obtain an identification queue, and adding a detection box to any identification sample; adding a tracking box to any first frame identification sample corresponding to any mine car in the identification queue based on a multi-target tracking algorithm to track; updating the position of each tracking box in the next frame identification sample; when there is a detection box corresponding to the position of the tracking box, it is judged that the corresponding car has appeared before, otherwise it is the first appearing car and a tracking box is added; repeating the previous step until the target detection model detects the last frame identification sample corresponding to the mine car, and counting the total number of tracking box IDs to determine whether the mine car hanging exceeds the preset hanging threshold. The present application can effectively avoid mine car over-hanging and improve the safety of mine car transportation.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of mine car overhanging detection, and particularly relates to a mine car overhanging detection method and device based on target detection and multi-target tracking. BACKGROUND

[0002] In the process of coal mining, mine cars such as endless rope winches or electric locomotives are needed to undertake corresponding transportation tasks in the roadway. At the same time, the number of carriages carried by the mine car needs to be counted to prevent exceeding the maximum transportation capacity and causing transportation accidents, thereby ensuring the safe production of mining.

[0003] The existing carriage number statistical detection is mostly based on video monitoring and manual judgment. That is, the video data corresponding to the actual transportation information is obtained through a camera, and the video data is viewed by the human eye to confirm whether the corresponding mine car is overhanging. However, due to the high speed of the mine car, the mine car appears in the field of view of the camera for about 2-4 seconds in general. And due to the actual production needs, the transportation of the mine car is random, that is, it always appears randomly in the camera. Therefore, in the existing carriage number statistical detection process, not only is the workload of the relevant technical personnel large, but also there are always missed detection and false detection, so that transportation accidents caused by mine car overhanging cannot be effectively avoided. SUMMARY

[0004] The present application aims to provide a mine car overhanging detection method and device based on target detection and multi-target tracking, to solve the technical problem that there is always missed detection and false detection in the existing mine car overhanging detection, so that the corresponding transportation accidents cannot be effectively avoided.

[0005] To achieve the above-mentioned purpose, the present application proposes the following technical solutions:

[0006] The mine car overhanging detection method based on target detection and multi-target tracking comprises:

[0007] Frame sampling is performed on the transportation video of the mine car to form a plurality of image samples, and each image sample is arranged in a time sequence in a normal order to generate an image queue; wherein the mine car runs on a track laid in a transportation roadway, and the transportation video is obtained by a camera installed in the transportation roadway;

[0008] The image queue is input into a target detection model to obtain an identification queue, and a detection frame is added to each identification sample in the identification queue in turn; at the same time, a tracking frame is added to the first frame identification sample corresponding to any mine car based on a multi-target tracking algorithm for tracking; wherein each tracking frame is assigned a unique ID;

[0009] updating the position of each tracking frame in the next frame of identification samples; when there is a detection frame corresponding to the tracking frame position, it is determined that the corresponding car is an existing car, otherwise it is a first appearing car and a tracking frame is added for tracking;

[0010] repeating the previous step until the target detection model detects the last frame of identification samples corresponding to any mine car, and counting the total number of tracking frame IDs to determine whether the mine car loading exceeds the preset loading threshold.

[0011] Further, after counting the total number of tracking frame IDs to determine whether the mine car loading exceeds the preset loading threshold, it includes:

[0012] displaying the total number of tracking frame IDs on a display platform; and adding differentiated annotation information to the total number of IDs based on the judgment result of whether the mine car loading exceeds the loading threshold.

[0013] Further, after counting the total number of tracking frame IDs to determine whether the mine car loading exceeds the preset loading threshold, it includes:

[0014] When the mine car loading exceeds the preset loading threshold, extract the corresponding image sample in the image queue to generate an alarm video and store it;

[0015] sending the alarm video to a remote control terminal.

[0016] Further, when updating the position of each tracking frame in the next frame of identification samples, it includes:

[0017] determining whether the tracking frame has been successfully tracked, if not, recording the number of tracking failures, and when the number of failures is greater than a preset number of threshold, the tracking frame is removed in the frame of identification samples; otherwise, the following steps are continued:

[0018] based on calculating whether the first coincidence rate of the tracking frame and the left boundary frame of the identification sample is greater than a preset first coincidence threshold; wherein, R1 is the coordinates of the left boundary frame, B x is the coordinates of the tracking frame;

[0019] based on calculating whether the second coincidence rate of the tracking frame and the right boundary frame of the identification sample is greater than a preset second coincidence threshold; wherein, R2 is the coordinates of the right boundary frame, B x is the coordinates of the tracking frame;

[0020] When the first coincidence rate is greater than the first coincidence threshold, or the second coincidence rate is greater than the second coincidence threshold, the tracking frame is removed in the frame of the identification sample.

[0021] Further, when there is a detection frame corresponding to the position of the tracking frame, it is judged that the corresponding car body is a car body that has appeared before, otherwise it is a car body that appears for the first time and a tracking frame is added for tracking; comprising:

[0022] Traverse all the tracking frames, calculate and take the maximum value; wherein A is the coordinates of any detection frame in the identification sample;

[0023] When the maximum value of lx is greater than a preset intersection-over-union threshold, it indicates that there is a detection frame corresponding to the position of the tracking frame; otherwise, it indicates that it is a car body that appears for the first time.

[0024] The mine car over-hanging detection device based on target detection and multi-target tracking comprises:

[0025] A first acquisition module is configured to frame sample a transport video of a mine car to form a plurality of image samples, and arrange the image samples in a time sequence in ascending order to generate an image queue; wherein the mine car runs on a track laid in a transport main roadway, and the transport video is acquired by a camera installed in the transport main roadway;

[0026] A detection and tracking module is configured to input the image queue into a target detection model to obtain an identification queue, and add detection frames to each identification sample in the identification queue in turn; at the same time, a tracking frame is added to a first frame of identification sample corresponding to any mine car for tracking based on a multi-target tracking algorithm; wherein each tracking frame is assigned a unique ID;

[0027] An update and discrimination module is configured to update the position of each tracking frame in a next frame of identification sample; when there is a detection frame corresponding to the position of the tracking frame, it is judged that the corresponding car body is a car body that has appeared before, otherwise it is a car body that appears for the first time and a tracking frame is added for tracking;

[0028] A statistical output module is configured to repeatedly call the update and discrimination module until the target detection model detects the last frame of identification sample corresponding to the any mine car, and then count the total number of IDs of the tracking frames to judge whether the mine car is over-hanging the preset hanging threshold.

[0029] Further, comprising:

[0030] A labeling module is configured to display the total number of IDs of the tracking frames on a display platform, and add differentiated labeling information to the total number of IDs based on the judgment result of whether the mine car is over-hanging the hanging threshold.

[0031] Further comprising:

[0032] An alarm generation module is configured to extract a corresponding image sample from the image queue to generate an alarm video and store the alarm video when the ore car is mounted beyond a preset mounting threshold.

[0033] A sending module is configured to send the alarm video to a remote control terminal.

[0034] Further comprising:

[0035] A first judgment module is configured to judge whether the tracking frame has been successfully tracked, and if not, record the number of tracking failures, and if the number of failures is greater than a preset number threshold, then eliminate the tracking frame in the frame recognition sample; otherwise, the following modules are sequentially called:

[0036] A first calculation module is configured to calculate, based on whether the first overlap rate of the tracking frame and the left boundary frame of the recognition sample is greater than a preset first overlap threshold; wherein R1 is the left boundary frame coordinate, B x is the tracking frame coordinate;

[0037] A second calculation module is configured to calculate, based on whether the second overlap rate of the tracking frame and the right boundary frame of the recognition sample is greater than a preset second overlap threshold; wherein R2 is the right boundary frame coordinate, B x is the tracking frame coordinate;

[0038] A first elimination module is configured to eliminate the tracking frame in the frame recognition sample when the first overlap rate is greater than the first overlap threshold, or the second overlap rate is greater than the second overlap threshold.

[0039] Further comprising:

[0040] A third calculation module is configured to traverse all the tracking frames, calculate and take the maximum value; wherein A is the coordinate of any detection frame in the recognition sample;

[0041] A second judgment module is configured to indicate that there is a detection frame corresponding to the position of the tracking frame when the maximum value of lx is greater than a preset intersection-over-union threshold; otherwise, it indicates that it is the first appearing car body.

[0042] Beneficial effects:

[0043] According to the technical scheme, the mine car over-hanging detection method based on target detection and multi-target tracking is provided to improve the defects of low detection efficiency, missed detection and false detection in the existing video-based and artificial detection.

[0044] The method comprises: frame sampling a transport video to form a plurality of image samples, and arranging each image sample in a time sequence to generate an image queue; wherein the mine car runs on a track laid in a transport main roadway, and the transport video is obtained by a camera installed in the transport main roadway. The image queue is input into a target detection model to obtain an identification queue, and a detection box is added to any identification sample in the identification queue. A tracking box is added to a first frame identification sample corresponding to any mine car in the identification queue based on a multi-target tracking algorithm for tracking; wherein each tracking box is assigned a unique ID. The position of each tracking box in the next frame identification sample is updated; when there is a detection box corresponding to the position of the tracking box, it is judged that the corresponding car body is a car body that has appeared before, otherwise it is a car body that appears for the first time and a tracking box is added for tracking. Repeat the above step until the target detection model detects the last frame identification sample corresponding to the any mine car, and count the total number of tracking box IDs to determine whether the mine car loading exceeds the preset loading threshold.

[0045] According to the above method, since the image queue is arranged in a time sequence, when it is input into the target detection model, it can be detected in a first-in first-out and last-in last-out manner according to the actual running situation of the mine car. In the specific detection, the first frame identification sample corresponding to any mine car is taken as the tracking starting point, and a tracking box is added for tracking; and the position of the tracking box in the next frame identification sample is updated. At this time, in the target detection and multi-target tracking process, when the detection box in the next frame has a tracking box corresponding to its position (the tracking box is obtained by tracking calculation in the multi-target tracking algorithm), it indicates that the car body corresponding to the detection box has been tracked. At the same time, since each tracking box is assigned a unique ID, it also indicates that the ID of the tracked detection box has been counted. When the detection box in the next frame does not have a tracking box corresponding to its position, it indicates that the car body corresponding to the detection box has not been tracked, i.e. it is a newly appeared car body. At this time, a tracking box is added to it and a tracking box ID is assigned. When the tracking box is iteratively updated to the last frame identification sample of the mine car, it indicates that the column of mine cars has been identified and tracked. Since the tracking box ID is uniquely assigned, the total number of tracking box IDs finally counted is the actual number of mine car loadings. Based on this, the detection of whether the mine car is over-hanging can be performed.

[0046] As the above analysis shows, this technical solution, based on a combination of target detection and multi-target tracking algorithms, achieves the counting of mine car bodies, effectively improving upon existing technologies which suffer from high manpower consumption and, in particular, false detections and missed detections. This, in turn, effectively ensures the safety of underground mine transportation.

[0047] It should be understood that all combinations of the foregoing concepts and the additional concepts described in more detail below can be considered part of the inventive subject matter of this disclosure, provided that such concepts do not contradict each other.

[0048] The foregoing and other aspects, embodiments, and features of the teachings of the present invention will be more fully understood from the following description in conjunction with the accompanying drawings. Other additional aspects of the invention, such as features and / or beneficial effects of exemplary embodiments, will become apparent from the following description or may be learned through practice of specific embodiments according to the teachings of the present invention. Attached Figure Description

[0049] The accompanying drawings are not intended to be drawn to scale. In the drawings, each identical or nearly identical component shown in the various figures may be denoted by the same reference numeral. For clarity, not every component is labeled in each figure. Embodiments of various aspects of the invention will now be described by way of example and with reference to the accompanying drawings, wherein:

[0050] Figure 1 This is a flowchart of the mine car overloading detection method described in this embodiment;

[0051] Figure 2 for Figure 1 The flowchart of the image sample processing in the document;

[0052] Figure 3 A flowchart showing the direction of entry for any mining truck;

[0053] Figure 4 for Figure 1 The flowchart shows the process of repeatedly updating the tracking box of the identified sample in the next frame;

[0054] Figure 5 for Figure 1 The flowchart for determining whether a detection box exists corresponding to the position of the tracking box is shown.

[0055] Figure 6 for Figure 1 Flowchart of the process after determining whether an overload has occurred;

[0056] Figure 7 for Figure 1 Another processing flowchart for determining whether an overload has occurred;

[0057] Figure 8 To conduct Figure 1A training flowchart of the target detection model. DETAILED DESCRIPTION

[0058] In order to make the objects, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions of the embodiments of the present application will be described clearly and completely below with reference to the drawings of the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the described embodiments of the present application, all other embodiments obtained by a person of ordinary skill in the art without any inventive effort fall within the scope of protection of the present application. Unless otherwise defined, the technical terms or scientific terms used herein should have the usual meanings understood by a person of ordinary skill in the art.

[0059] The terms "first", "second", and similar terms used in the patent application specification and claims of the present application do not denote any order, quantity, or importance, but are only used to distinguish different components. Similarly, the singular forms "a", "an", and "the" and the like do not denote a quantity limitation, but denote the existence of at least one, unless the context clearly indicates otherwise. The terms "comprise", "comprising", and the like mean that the elements or objects appearing before "comprise" or "comprising" cover the features, integers, steps, operations, elements, and / or components listed after "comprise" or "comprising", and do not exclude the existence or addition of one or more other features, integers, steps, operations, elements, components, and / or sets thereof. "Up", "down", "left", "right", and the like are only used to represent relative positional relationships, which may change accordingly when the absolute position of the described object changes.

[0060] Detecting whether a mine car is overhanging is a key to ensure the safety of underground operation. In the existing mine car overhanging detection method, the transportation situation is obtained through the transportation video from the camera, and the judgment is made manually. Therefore, in the specific implementation, there is a large amount of personnel work, low work efficiency, and always missing detection and wrong detection. The embodiment aims to provide a mine car overhanging detection method based on target detection and multi-target tracking to improve the above-mentioned defects in the existing mine car overhanging detection and ensure the safety of underground transportation.

[0061] The mine car overhanging detection method based on target detection and multi-target tracking disclosed in the embodiment will be further described below with reference to the drawings.

[0062] In combination with Figure 1 As shown in the figure, the method is performed according to the following steps:

[0063] Step S102, frame sampling is performed on the transport video to form a plurality of image samples, and the image samples are arranged in a time sequence in a normal order to generate an image queue; wherein the mine car runs on a track laid in a transport main roadway, and the transport video is obtained by a camera installed in the transport main roadway.

[0064] In a specific implementation, in order to enable the camera to shoot a clear transport video and ensure that the movement direction of the car body in the obtained image sample is a horizontal direction to facilitate subsequent identification and tracking, the pitch angle range of the camera is set to 45°-75°.

[0065] In frame sampling, a preset fixed frame sampling frequency is used. In order to ensure that the entry of the first car body of each train and the exit of the last car body are always sampled, the frame sampling frequency is set to: wherein M is the number of images shot by the camera per second, T is the movement time of the mine car in the field of view of the camera, and N is a preset mounting threshold of the mine car. In this embodiment, the number of images shot by the camera per second is 24, that is, M=24 images / s; the movement time of the mine car in the field of view of the camera is generally 2-4 seconds, and is specifically 3 seconds, that is, T=3 seconds.

[0066] As an alternative implementation, in order to prevent the defects of the image samples from causing subsequent identification and tracking abnormalities, the image samples are preprocessed as follows before the image queue is generated: Figure 2 As shown in the figure, before the image queue is generated, the image samples are further preprocessed as follows:

[0067] Step S102.2, similar samples in each image sample are removed based on a structural similarity algorithm.

[0068] Step S102.4, the sharpness of each image sample is calculated based on a Laplace energy function to remove blurred samples in each image sample.

[0069] At this time, through steps S102.2 to S102.4, the image samples of the generated image queue are all clear image samples without repetition.

[0070] Step S104, the image queue is input into a target detection model to obtain an identification queue, and a detection box is added to each identification sample in the identification queue in sequence; at the same time, a tracking box is added to a first frame identification sample corresponding to any mine car based on a multi-target tracking algorithm for tracking; wherein each tracking box is assigned a unique ID.

[0071] In a specific implementation, a text box is further drawn above each detection box to display the score of the corresponding car body detected.

[0072] For the mine car in the haulage roadway, the mine car running in any direction runs on a track. In the specific implementation, in order to facilitate the ID allocation of the tracking frame for subsequent detection result statistics. It is also necessary to know the running direction of any mine car, that is, to obtain the image sample based on which the mine car is driven from the left side of the image sample or from the right side of the image sample.

[0073] Therefore, as an optional implementation, in combination with Figure 3 As shown, in step S104, when adding the detection frame for each identification sample, it also includes:

[0074] Step S104.2, taking all the detection frames in the first frame identification sample as a whole and obtaining the center point coordinates; wherein the upper left corner of the identification sample is defined as the coordinate origin, the horizontal direction to the right is the positive direction of the X axis, and the vertical direction downward is the positive direction of the Y axis.

[0075] Step S104.4, calculating the first distance between the center point coordinates and the left side of the identification sample, and the second distance between the center point coordinates and the right side of the identification sample.

[0076] Step S104.6, when the first distance is greater than the second distance, it is judged that the mine car is driven from the left side of the image sample; otherwise, it is driven from the right side of the image sample.

[0077] At this time, as a specific implementation, when the ID allocation of each tracking frame is performed, the ID allocation is performed from small to large along the driving direction of the train.

[0078] As a more preferred implementation, for the ID allocation of the tracking frame, the ID difference between adjacent two tracking frames is also 1 starting from ID 1.

[0079] Step S106, updating the position of each tracking frame in the next frame identification sample; when there is a detection frame corresponding to the tracking frame position, it is judged that the corresponding car is a car that has appeared before, otherwise it is a car that appears for the first time and adds a tracking frame for tracking.

[0080] In combination with Figure 4 As shown, when the tracking frame of the next frame identification sample is updated each time, in order to reduce CPU consumption and improve tracking effectiveness; it also includes:

[0081] Step S106.2, judging whether the tracking frame has been successfully tracked, if not, recording the failure number of tracking failure, and when the failure number is greater than a preset number threshold, eliminating the tracking frame in the frame identification sample; otherwise, continue to:

[0082] Step S106.4, based on calculating whether the first coincidence rate of the tracking frame and the left boundary frame of the identified sample is greater than a preset first coincidence threshold; wherein, the R1 is the left boundary frame coordinate, B x is the tracking frame coordinate;

[0083] Step S106.6, based on calculating whether the second coincidence rate of the tracking frame and the right boundary frame of the identified sample is greater than a preset second coincidence threshold; wherein, the R2 is the right boundary frame coordinate, B x is the tracking frame coordinate;

[0084] Step S106.8, when the first coincidence rate is greater than the first coincidence threshold, or the second coincidence rate is greater than the second coincidence threshold, then the tracking frame is eliminated in the frame identified sample.

[0085] At this time, the tracking frame that has been repeatedly tracked and the tracking frame that always fails to track can be eliminated in subsequent tracking frame updating, so as to improve the tracking effectiveness. Specifically, the tracking frame that always fails to track may be a detection frame marked error failure target, which is beneficial to improve the final mounting statistical accuracy at this time; it may also be a tracking frame corresponding to a train car that has left, so as to avoid invalid tracking.

[0086] In combination with Figure 5 shown, as a specific embodiment, based on steps S106.2-S106.8, for judging whether there is a detection frame corresponding to the tracking frame position, comprising:

[0087] Step S106.10.2, traversing all the tracking frames, calculating and taking the maximum value; wherein, A is the coordinate of any detection frame in the identified sample;

[0088] Step S106.10.4, when the maximum value of the lx is greater than a preset intersection-over-union threshold, it is indicated that there is a detection frame corresponding to the tracking frame position; otherwise, it is indicated that it is the first appearing car skin.

[0089] Step S108, repeating the previous step until the target detection model detects the last frame of the identified sample corresponding to the any train car, and then counting the total number of IDs of the tracking frames to judge whether the train car mounting exceeds a preset mounting threshold.

[0090] As a specific embodiment, when the ID assignment of each tracking frame is performed, the ID assignment is performed from small to large along the driving direction of the train. And when the ID of the first tracking frame is 1, the ID difference between adjacent two tracking frames is also 1, when counting the total number of IDs of the tracking frames, only the last ID needs to be obtained.

[0091] In a specific implementation, for the first frame recognition sample corresponding to any mine car in step S104 and the last frame recognition sample corresponding to the any mine car in step S108, the following method is used to obtain them:

[0092] When a plurality of continuous frames are not detected, it is indicated that the last frame before the plurality of continuous frames is the last frame recognition sample of the previous row of mine cars, and the first frame after the plurality of continuous frames is the first frame recognition sample of the next row of mine cars.

[0093] In combination with Figure 6 As shown in the figure, in order to facilitate the confirmation of the over-hanging situation, as one of the optional embodiments, after step S108, the following steps are further included:

[0094] Step S110, display the total number of IDs of the tracking frames on a display platform; and add differentiated annotation information to the total number of IDs based on the judgment result of whether the mine car hanging exceeds the hanging threshold.

[0095] In a specific implementation, when over-hanging occurs, the total number of IDs is annotated using a red text box; and when over-hanging does not occur, the total number of IDs is annotated using a blue text box.

[0096] In combination with Figure 7 As shown in the figure, in order to facilitate the confirmation of the over-hanging situation, as another optional embodiment, after step S108, the following steps are further included:

[0097] Step S110.2', when the mine car hanging exceeds the preset hanging threshold, extract the corresponding image sample in the image queue to generate an alarm video and store it.

[0098] Step S110.4', send the alarm video to a remote control end.

[0099] At this time, the technician at the remote end can also obtain the actual situation when over-hanging occurs to perform abnormal confirmation processing.

[0100] In combination with Figure 8 As one of the specific embodiments, in the entire over-hanging detection process, in order to improve the detection accuracy, the following method is used to train the target detection model involved in the embodiment:

[0101] Step S202, collect a plurality of image samples, and remove similar images and blurred images by the method shown in steps S102.2 to S102.4.

[0102] Step S204, dividing the image samples into a training set and a validation set; inputting the training set into a yolov4 neural network for training, and inputting the validation set into the aforementioned trained neural network for training adjustment.

[0103] In this step, the image samples are 3000, of which the training set is 2400 and the validation set is 600.

[0104] Step S206, repeating the above process until the optimal target detection model is obtained.

[0105] In this step, the maximum number of iterations is 100000. In this embodiment, the optimal target detection model is obtained when training to 50000. In this embodiment, the target detection model is specifically a yolov4 target detection network.

[0106] The above method process can be run in a processor, or can also be stored in a memory (or called computer readable medium), the computer readable medium includes permanent and non-permanent, removable and non-removable media can be realized by any algorithm or technology information storage. Information can be computer readable instructions, data structure, program module or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read only memory (ROM), electrically erasable programmable read only memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage device or any other non-transmission medium, which can be used to store information that can be accessed by a computing device. According to the definition in this paper, computer readable medium does not include transitory computer readable medium, such as modulated data signal and carrier wave.

[0107] These computer programs can also be loaded into a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to generate computer implemented processing, so that the instructions executed on the computer or other programmable device provide a process for implementing the steps specified in the flow Figure 1 One flow or multiple flows and / or blocks Figure 1 One block or multiple blocks, the steps of the function specified in the block can be realized by different modules.

[0108] The embodiment provides such a device, which is called a mine car overhanging detection device based on target detection and multi-target tracking. The device comprises in sequence:

[0109] The first acquisition module is configured to frame sample a transport video of a mine car to form a plurality of image samples, and arrange the image samples in a time sequence in a normal order to generate an image queue; wherein the mine car runs on a track laid in a transport main roadway, and the transport video is acquired by a camera installed on a wall of the transport main roadway.

[0110] The detection and tracking module is configured to input the image queue into a target detection model to obtain an identification queue, and sequentially add a detection frame to each identification sample in the identification queue; meanwhile, a tracking frame is added to a first frame identification sample corresponding to any mine car based on a multi-target tracking algorithm for tracking; wherein each tracking frame is assigned a unique ID.

[0111] The update and discrimination module is configured to update the position of each tracking frame in a next frame identification sample; when there is a detection frame corresponding to the position of the tracking frame, it is determined that the corresponding car is a car that has appeared before, otherwise it is a car that appears for the first time and a tracking frame is added for tracking.

[0112] The statistical output module is configured to repeatedly call the update and discrimination module until the target detection model detects a last frame identification sample corresponding to the any mine car, and then count the total number of IDs of the tracking frames to determine whether the loading of the mine car exceeds a preset loading threshold.

[0113] The device is used to implement the functions of the methods described in the above embodiments. Each module in the device corresponds to each step in the method, and has been described in the method and will not be repeated here.

[0114] For example, in order to facilitate the rapid acquisition of overloading, the device further includes, after the statistical output module:

[0115] The labeling module is configured to display the total number of IDs of the tracking frames on a display platform, and add differentiated labeling information to the total number of IDs based on the determination result of whether the loading of the mine car exceeds the loading threshold.

[0116] As an alternative embodiment, the device can further include, after the statistical output module:

[0117] The alarm generation module is configured to, when the loading of the mine car exceeds the preset loading threshold, extract the corresponding image sample in the image queue to generate an alarm video and store it;

[0118] The sending module is configured to send the alarm video to a remote control end.

[0119] At this time, the control personnel at the remote end can also acquire real-time overloading and perform abnormal processing.

[0120] For example, in order to reduce CPU consumption and improve tracking effectiveness, the device further comprises:

[0121] A first judging module is configured to judge whether the tracking frame has been successfully tracked or not, and if not, record the number of tracking failures, and if the number of failures is greater than a preset number threshold, eliminate the tracking frame in the frame recognition sample; otherwise, call the following modules:

[0122] A first calculating module is configured to calculate, based on whether the first coincidence rate of the tracking frame and the left boundary frame of the recognition sample is greater than a preset first coincidence threshold; wherein R1 is the left boundary frame coordinate, B x is the tracking frame coordinate.

[0123] A second calculating module is configured to calculate, based on whether the second coincidence rate of the tracking frame and the right boundary frame of the recognition sample is greater than a preset second coincidence threshold; wherein R2 is the right boundary frame coordinate, B x is the tracking frame coordinate.

[0124] A first eliminating module is configured to eliminate the tracking frame in the frame recognition sample when the first coincidence rate is greater than the first coincidence threshold, or the second coincidence rate is greater than the second coincidence threshold.

[0125] Therefore, as a specific embodiment, the device further comprises:

[0126] A third calculating module is configured to traverse all the tracking frames, and calculate and take the maximum value; wherein A is the coordinate of any detection frame in the recognition sample.

[0127] A second judging module is configured to indicate that there is a detection frame corresponding to the tracking frame position when the maximum value of lx is greater than a preset intersection-over-union threshold; otherwise, it is a first appearing car body.

[0128] Although the present application has been disclosed with reference to the preferred embodiments above, it is not intended to limit the present application. Those skilled in the art can make various modifications and improvements without departing from the spirit and scope of the present application. Therefore, the protection scope of the present application shall be subject to the scope defined by the claims.

Claims

1. A method for detecting overhanging of a mine car based on target detection and multi-target tracking, characterized in that, The method comprises the following steps: frame sampling is performed on a transport video of a mine car to form a plurality of image samples, and each of the image samples is arranged in a positive time sequence to generate an image queue; wherein the mine car runs on a track laid in a transport roadway, and the transport video is obtained by a camera installed in the transport roadway; the image queue is input into a target detection model to obtain an identification queue, and a detection box is sequentially added to each identification sample in the identification queue; at the same time, a tracking box is added to a first frame identification sample corresponding to any mine car based on a multi-target tracking algorithm for tracking; wherein each tracking box is assigned a unique ID; the position of each tracking box in the next frame identification sample is updated; when there is a detection box corresponding to the position of the tracking box, it is determined that the corresponding car body is a car body that has appeared before, otherwise it is a car body that appears for the first time and a tracking box is added for tracking; wherein the updating of the position of each tracking box in the next frame identification sample comprises: determining whether the tracking box has been successfully tracked, if not, recording the number of failures of tracking, and when the number of failures is greater than a preset number threshold, the tracking box is removed in the frame identification sample; otherwise, the following steps are continued: Based on whether the first coincidence rate of the tracking frame and the left boundary frame of the identified sample is greater than a preset first coincidence threshold; wherein, the R1 is the coordinate of the left boundary frame, B x is the coordinate of the tracking frame; Based on whether the second coincidence rate of the tracking frame and the right boundary frame of the identified sample is greater than a preset second coincidence threshold; wherein, the R2 is the coordinate of the right boundary frame, B x is the coordinate of the tracking frame. when the first coincidence rate is greater than the first coincidence threshold, or the second coincidence rate is greater than the second coincidence threshold, the tracking box is removed in the frame identification sample; wherein, when there is a detection box corresponding to the position of the tracking box, it is determined that the corresponding car body is a car body that has appeared before, otherwise it is a car body that appears for the first time and a tracking box is added for tracking, which comprises: Traverse all the tracking frames, calculate Take the maximum value; wherein A is the coordinates of any detection frame in the identified sample when the maximum value of the lx is greater than a preset intersection-over-union threshold, it indicates that there is a detection box corresponding to the position of the tracking box; otherwise, it indicates that it is a car body that appears for the first time; repeat the previous step until the target detection model detects the last frame identification sample corresponding to the any mine car, and count the total number of IDs of the tracking boxes to determine whether the loading of the mine car exceeds a preset loading threshold.

2. The method according to claim 1, wherein, After the total number of IDs of the tracking boxes is counted to determine whether the loading of the mine car exceeds the preset loading threshold, the following steps are included: display the total number of IDs of the tracking boxes on a display platform; and add differentiated annotation information to the total number of IDs based on the determination result of whether the loading of the mine car exceeds the loading threshold.

3. The method according to claim 1, wherein, After the total number of IDs of the tracking boxes is counted to determine whether the loading of the mine car exceeds the preset loading threshold, the following steps are included: when the loading of the mine car exceeds the preset loading threshold, extract the corresponding image sample in the image queue to generate an alarm video and store it; send the alarm video to a remote control terminal.

4. The mine car overhang detection device based on target detection and multi-target tracking, characterized in that, The method comprises the following steps: a first acquisition module is configured to perform frame sampling on a transport video of a mine car to form a plurality of image samples, and each of the image samples is arranged in a positive time sequence to generate an image queue; wherein the mine car runs on a track laid in a transport roadway, and the transport video is obtained by a camera installed in the transport roadway; The detection and tracking module is configured to input the image queue into a target detection model to obtain an identification queue, and add a detection frame to each identification sample in the identification queue in sequence. Meanwhile, a tracking frame is added to a first frame of identification sample corresponding to any mine car based on a multi-target tracking algorithm for tracking. Each tracking frame is assigned a unique ID. The update and discrimination module is configured to update the position of each tracking frame in a next frame of identification sample. When a detection frame corresponding to the position of the tracking frame exists, it is determined that the corresponding car body is a car body that has appeared before, otherwise, it is a car body that appears for the first time and a tracking frame is added for tracking. The update and discrimination module includes: The first judgment module is configured to determine whether the tracking frame has been successfully tracked. If not, record the number of tracking failures, and when the number of failures is greater than a preset number threshold, remove the tracking frame in the frame of identification sample. Otherwise, the following modules are called in sequence: The first calculation module is configured to calculate, based on whether the first coincidence rate of the tracking frame and the left boundary frame of the identified sample is greater than a preset first coincidence threshold; wherein, the R1 is the left boundary frame coordinate, B x is the tracking frame coordinate. The second calculation module is used for... Calculate whether the second overlap rate between the tracking box and the right bounding box of the identified sample is greater than a preset second overlap threshold; where R2 is the coordinate of the right bounding box, and B... x The coordinates of the tracking box; The first removal module is configured to remove the tracking frame in the frame of identification sample when the first coincidence rate is greater than the first coincidence threshold, or the second coincidence rate is greater than the second coincidence threshold. The third calculation module is configured to traverse all the tracking frames and calculate and take the maximum value; wherein A is the coordinate of any detection frame in the identification sample. The second judgment module is configured to determine that there is a detection frame corresponding to the position of the tracking frame when the maximum value of lx is greater than a preset intersection over union threshold. Otherwise, it is a car body that appears for the first time. The statistical output module is configured to repeatedly call the update and discrimination module until the target detection model detects the last frame of identification sample corresponding to any mine car, and then counts the total number of IDs of the tracking frame to determine whether the load of the mine car exceeds a preset load threshold.

5. The mine car overhang detection device based on target detection and multi-target tracking according to claim 4, characterized in that, The labeling module is configured to display the total number of IDs of the tracking frame on a display platform. Based on the result of determining whether the load of the mine car exceeds the load threshold, the labeling information is added to the total number of IDs. The alarm generation module is configured to extract the corresponding image sample in the image queue to generate an alarm video and store it when the load of the mine car exceeds the preset load threshold.

6. The mine car overhang detection device based on target detection and multi-target tracking according to claim 4, characterized in that, The sending module is configured to send the alarm video to a remote control terminal. ​ ​

Images