Coal mining machine follow-up frame detection method, device and system
Through monitoring video, the position of the coal miner roller and guard plate is detected, the operating status of the coal miner and the condition of the machine shifting, the safety accident caused by the coal miner failing to follow the machine shifting in time, and the automatic alarm and Guarantee of production safety.
Patent Information
- Application Number
- CN202011631011.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2020-12-30
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2040-12-30
AI Technical Summary
If the coal mining machine fails to move the frame or guard the machine in time during operation, the detection capacity is insufficient, which can easily lead to safety production liability accidents.
By obtaining the monitoring video of the coal mining working surface, performing target detection to determine the position information of the coal mining machine drum and guard plate, determining whether the coal mining machine is in operation, and determining whether the carrier is moving the frame in time based on the number of guard plates. , use the video acquisition module, object detection module and operation judgment module to achieve automatic detection.
It realizes automatic identification of the machine-moving status in the operating state of the coal miner, promptly alarm, avoid the occurrence of safety production responsibility accidents, and ensure the safety of coal mines.
Smart Images

Figure CN112712035B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of coal mine safety technology, and in particular to a method, device and system for detecting a coal mining machine following a machine frame shift. Background Art
[0002] Production safety is the foundation and guarantee for the development of an enterprise, and is an important concept that every enterprise should implement in production and operation. In the field of coal mining, safe production is the cornerstone of the coal mining industry. For coal mining production, the state, enterprises, and coal mining enterprises have formulated detailed laws, regulations, and operating regulations. However, when enterprises often focus on the pursuit of efficient production, they often violate regulations and pose safety hazards.
[0003] During the coal mining process, the coal mining machine on the fully mechanized working face is constantly advancing, and it is necessary to promptly move the frame with the machine and protect the side walls accordingly. The side walls are protected by the side guard plates, which are metal plates specially used to protect the side walls of the coal mining face. There are two states under electric control. One state is side wall protection, where the metal plate is attached to the side wall of the coal to prevent the coal from falling off and play a protective role; the other state is no side wall protection, where the metal plate does not contact the coal wall and does not play a protective role.
[0004] During the mining process, due to illegal operations and other reasons, there are cases where the frame is not moved or the side is not protected in time. The current detection capabilities are insufficient, which makes it easy for production safety accidents to occur. Summary of the invention
[0005] The invention solves the problem that the detection capability is insufficient when the coal mining machine is not moved or protected in time, which easily leads to production safety liability accidents.
[0006] In order to solve the above problems, the present invention provides a method for detecting the follow-up frame movement of a coal mining machine, the method comprising: acquiring a monitoring video of a coal mining face; performing target detection on image frames of the monitoring video to obtain target detection results of a coal mining machine drum and a side guard plate; judging whether the coal mining machine is in operation according to the target detection results of the coal mining machine drum; if in operation, determining whether to follow the machine and move the side guard plate in time according to the target detection results of the side guard plate.
[0007] Optionally, judging whether the coal mining machine is in operation according to the target detection result includes: obtaining first position information of the roller in the first image frame and second position information of the roller in the second image frame in the target detection result; the first image frame and the second image frame are separated by a first preset number of image frames; determining the number of pixels moved in the horizontal direction of the roller according to the first position information and the second position information as a first horizontal displacement; if the ratio of the first horizontal displacement to the number of pixels in the horizontal direction of the image frame is greater than a first preset movement tolerance, determining that the coal mining machine is in operation; or, determining the number of pixels moved in the vertical direction of the roller according to the first position information and the second position information as a first vertical displacement; if the ratio of the first vertical displacement to the number of pixels in the vertical direction of the image frame is greater than a second preset movement tolerance, determining that the coal mining machine is in operation.
[0008] Optionally, the method also includes: if the second position information of the roller does not exist in the second image frame, extracting the third position information of the third image frame; the first image frame and the third image frame are separated by a second preset number of image frames, the second preset number is n times the first preset number, and n is a positive integer greater than or equal to 1; according to the first position information and the third position information, the number of pixels moved by the roller in the horizontal direction is determined as the second horizontal displacement; if the ratio of the second horizontal displacement to the number of pixels in the horizontal direction of the image frame is greater than a third preset movement tolerance, it is determined that the coal mining machine is in operation; the third preset movement tolerance is n times the first movement tolerance; or, according to the first position information and the third position information, the number of pixels moved by the roller in the vertical direction is determined as the second vertical displacement; if the ratio of the second vertical displacement to the number of pixels in the vertical direction of the image frame is greater than a fourth preset movement tolerance, it is determined that the coal mining machine is in operation; the fourth preset movement tolerance is n times the second movement tolerance.
[0009] Optionally, determining whether to move the rack with the machine in time based on the target detection result of the guard plates includes: if the number of guard plates in the target detection result of the guard plates is greater than or equal to a first threshold, determining that the rack is moved with the machine in time; if the number of guard plates in the target detection result of the guard plates is less than or equal to a second threshold, determining that the rack is not moved with the machine in time; wherein the second threshold is less than the first threshold.
[0010] Optionally, the method also includes: if the number of guard plates in the target detection result of the guard plates is greater than the second threshold and less than or equal to a third threshold, it is determined to be a following-machine frame transfer in progress state or a following-machine frame transfer failure state; wherein the third threshold is less than the first threshold.
[0011] Optionally, before judging whether the coal mining machine is in operation based on the target detection result of the coal mining machine drum, the method further includes: judging whether there are rollers and / or guard plates in the target detection result of at least one of the image frames within a preset time length; if at least one of the image frames within the preset time length includes rollers and / or guard plates, determining that the acquisition area of the monitoring video is correct; if each of the image frames within the preset time length does not include rollers and guard plates, determining that the acquisition area of the monitoring video is incorrect.
[0012] Optionally, the method also includes: obtaining training samples corresponding to image frames of historical monitoring videos of the coal mining face; the training samples include the image frames after the position and classification of the target objects have been marked, and the target objects include at least one of the following: drum, rocker arm, coal flow, and guard plate; and obtaining a target detection model by training according to the training samples.
[0013] The present invention provides a coal mining machine follow-up frame shifting detection device, the device comprising: a video acquisition module, used to acquire monitoring video of a coal mining face; a target detection module, used to perform target detection on image frames of the monitoring video to obtain target detection results of a coal mining machine drum and a side guard plate; an operation judgment module, used to judge whether the coal mining machine is in an operating state according to the target detection result of the coal mining machine drum; and a follow-up frame shifting determination module, used to determine whether to follow the machine shift the side guard plate in time according to the target detection result of the side guard plate if the coal mining machine is in an operating state.
[0014] Optionally, the operation judgment module is specifically used to: if the number of guard plates in the target detection result of the guard plates is greater than or equal to a first threshold, determine that the rack is moved in time with the machine; if the number of guard plates in the target detection result of the guard plates is less than or equal to a second threshold, determine that the rack is not moved in time with the machine; wherein the second threshold is less than the first threshold.
[0015] The present invention provides a coal mining machine frame shifting detection system, comprising a camera and a processor; the camera is used to collect monitoring videos; the processor is used to execute the above-mentioned coal mining machine frame shifting detection method.
[0016] In the embodiment of the present invention, when the coal mining machine is in operation, the target detection result of the side guard plate can be used to determine whether the frame is moved in time with the machine, and the state of moving the frame in time with the machine can be automatically identified and detected, so that an alarm can be issued when it is found that the frame is not moved in time with the machine, thereby ensuring safe production and avoiding the occurrence of safety accidents. BRIEF DESCRIPTION OF THE DRAWINGS
[0017] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.
[0018] Figure 1 It is a structural schematic diagram of a coal mining machine follow-up frame shifting detection system in one embodiment of the present invention;
[0019] Figure 2 It is a schematic flow chart of a method for detecting a coal mining machine frame shifting according to an embodiment of the present invention;
[0020] Figure 3 A schematic diagram of the position of a target object in an image frame in one embodiment of the present invention;
[0021] Figure 4 It is a schematic diagram of a target detection result of a fully mechanized mining face in one embodiment of the present invention;
[0022] Figure 5 It is a schematic diagram of another detection result of the side guard plate of a fully mechanized mining working face in one embodiment of the present invention;
[0023] Figure 6 It is a schematic diagram of another detection result of the side guard plate of a fully mechanized mining working face in one embodiment of the present invention;
[0024] Figure 7 The present invention is a schematic structural diagram of a coal mining machine frame movement detection device in one embodiment of the present invention.
[0025] Description of reference numerals:
[0026] 101-network camera; 102-processor; 701-video acquisition module; 702-target detection module; 703-operation judgment module; 704-follow-machine shifting determination module. DETAILED DESCRIPTION
[0027] In order to make the above-mentioned objects, features and advantages of the present invention more clearly understood, the specific embodiments of the present invention are described in detail below in conjunction with the accompanying drawings. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.
[0028] See also Figure 1The structural diagram of the coal mining machine frame shift detection system shown includes a network camera 101 and a deep learning server 102; the network camera 101 is used to collect monitoring video of the coal mining face, and the deep learning server 102 is used to execute the coal mining machine frame shift detection method. Optionally, the deep learning server 102 can be a graphics processing unit (GPU).
[0029] In the production environment, the network camera is fixedly deployed on one side of the coal mining face so that the drum of the coal mining machine and the multiple side guards of the hydraulic support can be monitored through the network camera. The deep learning server is connected to the network camera through the network and obtains the monitoring video stream from the network camera in real time. At the same time, the target detection technology in computer vision is used to detect the position of the drum of the coal mining machine and judge whether the coal mining machine is in operation by the displacement of the drum anchor frame in the image. In the operation state, it is judged whether there is a side guard behind the coal mining machine. When the side guard is identified, it means that the side guard has been timely protected and the frame has been moved with the machine in time. If the side guard is not identified, it means that the side guard has not been timely protected and the frame has not been moved with the machine in time.
[0030] Figure 2 1 is a schematic flow chart of a method for detecting a coal mining machine frame shifting according to an embodiment of the present invention, the method comprising:
[0031] S202, obtaining monitoring video of the coal mining face.
[0032] S204, performing target detection on the image frames of the monitoring video to obtain target detection results of the coal mining machine drum and the side guard plate. The target detection results may include target category information and target position information of each target object.
[0033] Optionally, target detection of image frames is performed based on a pre-trained target detection model to obtain target detection results of multiple image frames of the surveillance video. Pre-training the target detection model may include the following steps:
[0034] (1) Obtain training samples corresponding to image frames of historical monitoring videos of coal mining working faces. The training samples include image frames with the positions and classifications of target objects annotated. The target objects include at least one of the following: drum, rocker arm, coal flow, and guard plate. Figure 3 The schematic diagram of the position of the target object in the image frame shown shows that one image frame may include a drum, a rocker arm, a coal flow and a guard plate.
[0035] After collecting historical surveillance videos, the location and classification information of the target objects that need to be tracked and identified are annotated and trained. After the model converges, the optimal model is extracted for deployment. The deployed model can identify the category information and location information of the above target objects. Among the above target objects, some target objects will move relative to the main body of the coal mining machine during the operation of the coal mining machine. For example, the above drum gradually moves backward during the operation of the coal mining machine. Therefore, the target objects with the above characteristics can represent the operating status of the coal mining machine.
[0036] (2) Train the target detection model based on the above training samples. Figure 4 The schematic diagram of the target detection result of the fully mechanized mining face shown in FIG. shows the types and positions of the drum, rocker arm, and coal flow, which are represented by anchor boxes. The anchor box is a set of boxes predefined in the target detection algorithm, and its width and height match the width and height of the object in the dataset.
[0037] S206, judging whether the coal shearer is in operation according to the target detection result of the coal shearer drum.
[0038] Based on the target detection results of multiple image frames of the monitoring video, by comparing the displacement of the drum in different image frames, if its displacement is greater than the preset threshold, it is determined that the coal mining machine is in operation.
[0039] S208: If the system is in operation, determine whether to move the rack in time according to the target detection result of the guard plate.
[0040] When the coal mining machine is in operation, as the excavation continues, it is necessary to promptly follow the machine to move the frame and control the side guard plates to fit the side wall of the working face or to retract them. When setting the camera monitoring angle, the monitored collection area needs to include the roller and multiple side guard plates. When the following machine to move the frame is completed, multiple side guard plates fit the side wall of the working face, and the side guard plates appear in the above-mentioned collection area; when the following machine to move the frame is not completed in time, the side guard plates are in a retracted state, and the side guard plates leave the above-mentioned collection area. Therefore, by judging the number of side guard plates in the monitoring video image frame, it can be determined whether the following machine to move the frame in time.
[0041] The coal mining machine following-machine frame shifting detection method provided in the embodiment of the present invention can determine whether the coal mining machine following-machine frame shifting is timely through the target detection result of the side guard plate when the coal mining machine is in operation, and automatically identify and detect the following-machine frame shifting status, so that an alarm can be issued when it is found that the following-machine frame shifting is not timely, thereby ensuring safe production and avoiding the occurrence of safety production liability accidents.
[0042] The following specifically describes the process of determining whether the coal mining machine is in operation, including:
[0043] (1) Obtaining first position information of the drum in a first image frame and second position information of the drum in a second image frame in the target detection result. The first image frame and the second image frame are separated by a first preset number of image frames.
[0044] (2) Determine the number of pixels that the roller moves in the horizontal direction according to the first position information and the second position information as the first horizontal displacement. If the ratio of the first horizontal displacement to the number of pixels in the horizontal direction of the image frame is greater than the preset movement tolerance, it is determined that the coal mining machine is in operation. Alternatively, determine the number of pixels that the roller moves in the vertical direction according to the first position information and the second position information as the first vertical displacement; if the ratio of the first vertical displacement to the number of pixels in the vertical direction of the image frame is greater than the second preset movement tolerance, it is determined that the coal mining machine is in operation. The first preset movement tolerance may be equal to or unequal to the second preset movement tolerance.
[0045] The preset movement tolerance is the ratio of the projection of the moving track of the center point of the drum in the horizontal and vertical directions to the length or width of the image frame. A detection is performed every fixed number of frames. When the horizontal or vertical position of the drum in two adjacent detections exceeds the preset movement tolerance, it is determined that the coal mining machine is running.
[0046] The information format of the roller position obtained based on target detection is: (x, y, w, h), where x and y are the normalized center point coordinates, and w and h are the normalized target width and target height.
[0047] The drum anchor frame a and anchor frame a' are obtained by identifying the original image frame. The anchor frame a corresponds to a certain image frame, and the anchor frame a' corresponds to an image frame after t frames.
[0048] If the initial center coordinates of the roller are (x, y), and the center coordinates of the feature object become (x', y') after t frames, then the pixel points of the roller movement are Δx=xx', Δy=yy'. The length and width of the image frame are expressed in pixels as W and H respectively. If or It is determined that the coal mining machine is running, and TH is the movement tolerance threshold.
[0049] Considering the complex environment of the fully mechanized mining face, there is a possibility that the roller is blocked by debris such as coal dust, resulting in the roller not being detected during target detection, or the roller not being detected due to the target detection model's misrecognition of the image frame. Therefore, after the roller is detected in the first image frame, if the roller is not detected in the second image frame, the next image frame that is spaced from the second image frame by the preset number or the next image frame that is spaced from the second image frame by a multiple of the preset number can be selected backward. Based on this, the above method can also include the following steps:
[0050] (1) If the second position information of the roller does not exist in the second image frame, the third position information of the third image frame is extracted; the first image frame and the third image frame are separated by a second preset number of image frames, the second preset number is n times the first preset number, and n is a positive integer greater than or equal to 1.
[0051] (2) Determine the number of pixels that the roller moves in the horizontal direction according to the first position information and the third position information as the second horizontal displacement. If the ratio of the second horizontal displacement to the number of pixels in the horizontal direction of the image frame is greater than a third preset movement tolerance, it is determined that the coal mining machine is in operation. The third preset movement tolerance is n times the first movement tolerance.
[0052] (3) Determine the number of pixels that the roller moves in the vertical direction according to the first position information and the third position information as the second vertical displacement. If the ratio of the second vertical displacement to the number of pixels in the vertical direction of the image frame is greater than a fourth preset movement tolerance, it is determined that the coal mining machine is in operation. The fourth preset movement tolerance is n times the second movement tolerance.
[0053] The following specifically introduces whether to determine whether to move the rack in time based on the number of guard plates. It can be divided into the following situations:
[0054] If the number of guard plates in the target detection result of the guard plate is greater than or equal to the first threshold, it is determined that the frame should be moved with the machine in time. According to the actual working scene of the coal mining machine, the shooting angle and range of the network camera, it can be determined how many guard plates should exist in an image frame in the state of timely guarding and moving with the machine. Figure 5 The schematic diagram of the detection results of the guard plates on the comprehensive mining working face shown in the figure shows that under the conditions of timely guarding and following machine frame shifting, there should be three guard plates in one image frame, so the first threshold can be set to 3.
[0055] If the number of guard plates in the target detection result of the guard plates is less than or equal to the second threshold, it is determined that the frame is not moved in time with the aircraft; wherein the second threshold is less than the first threshold. Similar to the determination method of the first threshold, it can be determined how many guard plates should be present in an image frame when the guard plates are not moved in time. Figure 4 The detection result shown shows that when the side guards are not timely provided and the frame is not moved with the machine, the number of side guards in an image frame is 0, so the second threshold value can be set to 0.
[0056] Considering that it takes a certain amount of time to fold and release the side guard plates during the following machine frame transfer process, the number of side guard plates in the detection result changes between the first threshold and the second threshold; and when there is a failure in the following machine frame transfer, for example, some side guard plates fail to fold or release, the number of side guard plates in the detection result will also change between the first threshold and the second threshold, so judgment can be made for the above situation. Based on this, the above method also includes the following steps:
[0057] If the number of guard plates in the target detection result of the guard plates is greater than the second threshold and less than or equal to the third threshold, it is determined that the following machine frame transfer is in progress or the following machine frame transfer is faulty. Among them, the above-mentioned third threshold is less than the first threshold. Continuing with the previous example, the first threshold is set to 3 and the second threshold is set to 0, then the third threshold can be set to 2. When the number of guard plates is 2 or 1, it can be determined that the following machine frame transfer is in progress or the following machine frame transfer is faulty, and the monitoring personnel can be reminded to perform manual identification. See Figure 6 The schematic diagram of the detection results of the guard plate of the comprehensive mining working face shown in the figure shows that when the following machine frame moving is in progress or the following machine frame moving fails, there is a guard plate in an image frame.
[0058] Considering that the network camera may be affected by factors such as coal mining machine operation and human touch, causing its shooting angle or position to change, the unobtainable monitoring video cannot perform the above target detection process at all. Therefore, before the above S206, a step of confirming whether the network camera angle is correct can also be set, as follows:
[0059] First, it is determined whether there are rollers and / or side guards in the target detection results of at least one image frame within a preset time length. If at least one image frame within the preset time length includes rollers and / or side guards, it is determined that the acquisition area of the surveillance video is correct; if each image frame within the preset time length does not include rollers and side guards, it is determined that the acquisition area of the surveillance video is wrong.
[0060] Specifically, the preset time length may be 30 seconds or 1 minute. If the roller and / or the side guard plate can be detected in any image frame within the preset time length, it means that the network camera angle is correct. If the network camera angle is incorrect, an alarm may be issued.
[0061] When performing target detection on the image frames of the surveillance video, the image frames of the surveillance video may be converted to a preset resolution to obtain converted image frames; then, the converted image frames are input into the target detection model to obtain target detection results. Optionally, the server extracts the image frames in the memory and unifies their image resolutions to 416*416*3.
[0062] The embodiment of the present invention can identify whether the coal mining machine in the fully-mechanized working face has timely followed the machine frame transfer during the coal mining process, and use computer vision technology to automatically identify the operating status of the coal mining machine, and determine whether the machine frame transfer is timely based on the working status. Furthermore, the detection results can be connected to the coal mine safety production monitoring platform according to demand to issue violation alarms, timely discover and stop violations, and ensure coal mine safety production.
[0063] Figure 7 1 is a schematic structural diagram of a coal mining machine frame shift detection device in one embodiment of the present invention, the device comprising:
[0064] The video acquisition module 701 is used to acquire the monitoring video of the coal mining face;
[0065] The target detection module 702 is used to perform target detection on the image frames of the monitoring video to obtain target detection results of the coal mining machine drum and the side guard plate;
[0066] An operation judgment module 703 is used to judge whether the coal shearer is in operation according to the target detection result of the coal shearer drum;
[0067] The following-machine-shifting determination module 704 is used to determine whether to follow the machine to shift the rack in time according to the target detection result of the side guard plate if the machine is in operation state.
[0068] The coal mining machine following-machine frame shifting detection device provided in this embodiment can determine whether the coal mining machine has followed the machine and shifted the frame in time through the target detection results of the side guard plate when the coal mining machine is in operation, and automatically identify and detect the following-machine frame shifting status, so that it can alarm when it is found that the following-machine frame shifting is not timely, thereby ensuring safe production and avoiding the occurrence of safety production liability accidents.
[0069] Optionally, as an embodiment, the operation determination module 703 is specifically configured to:
[0070] Acquire first position information of the roller in the first image frame and second position information of the roller in the second image frame in the target detection result; the first image frame and the second image frame are separated by a first preset number of image frames;
[0071] Determine the number of pixels that the roller moves in the horizontal direction according to the first position information and the second position information as a first horizontal displacement; if the ratio of the first horizontal displacement to the number of pixels in the horizontal direction of the image frame is greater than a first preset movement tolerance, determine that the coal mining machine is in operation; or,
[0072] The number of pixels that the roller moves in the vertical direction is determined according to the first position information and the second position information as the first vertical displacement; if the ratio of the first vertical displacement to the number of pixels in the vertical direction of the image frame is greater than the second preset movement tolerance, it is determined that the coal mining machine is in operation.
[0073] Optionally, as an embodiment, the operation judgment module 703 is also used for: if the second position information of the roller does not exist in the second image frame, then extracting the third position information of the third image frame; the first image frame and the third image frame are separated by a second preset number of image frames, the second preset number is n times the first preset number, and n is a positive integer greater than or equal to 1; according to the first position information and the third position information, the number of pixels moved by the roller in the horizontal direction is determined as the second horizontal displacement; if the ratio of the second horizontal displacement to the number of pixels in the horizontal direction of the image frame is greater than a third preset movement tolerance, then it is determined that the coal mining machine is in operation; the third preset movement tolerance is n times the first movement tolerance; or, according to the first position information and the third position information, the number of pixels moved by the roller in the vertical direction is determined as the second vertical displacement; if the ratio of the second vertical displacement to the number of pixels in the vertical direction of the image frame is greater than a fourth preset movement tolerance, then it is determined that the coal mining machine is in operation; the fourth preset movement tolerance is n times the second movement tolerance.
[0074] Optionally, as an embodiment, the operation judgment module 704 is specifically used to: if the number of guard plates in the target detection result of the guard plates is greater than or equal to a first threshold, determine that the rack is moved in time with the machine; if the number of guard plates in the target detection result of the guard plates is less than or equal to a second threshold, determine that the rack is not moved in time with the machine; wherein the second threshold is less than the first threshold.
[0075] Optionally, as an embodiment, the operation judgment module 704 is also used for: if the number of guard plates in the target detection result of the guard plates is greater than the second threshold and less than or equal to the third threshold, it is determined to be a following-machine frame transfer state or a following-machine frame transfer failure state; wherein the third threshold is less than the first threshold.
[0076] Optionally, as an embodiment, the device also includes a monitoring area judgment module, which is used to: judge whether there are rollers and / or guard plates in the target detection results of at least one of the image frames within a preset time length; if at least one of the image frames within the preset time length includes rollers and / or guard plates, it is determined that the acquisition area of the monitoring video is correct; if each of the image frames within the preset time length does not include rollers and guard plates, it is determined that the acquisition area of the monitoring video is incorrect.
[0077] Optionally, as an embodiment, the device also includes a model training module, which is used to: obtain training samples corresponding to image frames of historical monitoring videos of the coal mining face; the training samples include the image frames after the position and classification of the target objects have been marked, and the target objects include at least one of the following: drum, rocker arm, coal flow, and guard plate; and obtain a target detection model by training according to the training samples.
[0078] An embodiment of the present invention also provides a coal mining machine frame shift detection system, comprising a camera and a processor; the camera is used to collect monitoring video; the processor is used to execute the above-mentioned coal mining machine frame shift detection method.
[0079] The coal mining machine frame movement detection device provided in the above embodiment can realize each process in the embodiment of the above coal mining machine frame movement detection method, and will not be described again here to avoid repetition.
[0080] The embodiment of the present invention also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, each process of the above-mentioned coal mining machine following machine frame shift detection method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it is not repeated here. The computer-readable storage medium is, for example, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.
[0081] Of course, those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the control device through a computer, and the program can be stored in a computer-readable storage medium. When the program is executed, it may include the processes of the above-mentioned method embodiments, wherein the storage medium may be a memory, a disk, an optical disk, etc.
[0082] In this article, relational terms such as first and second, etc. are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the statement "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.
[0083] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.
[0084] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A method for detecting the movement of a coal mining machine. It is characterized in that The method comprises: Obtain monitoring video of the coal mining face; Performing target detection on the image frames of the monitoring video to obtain target detection results of the coal mining machine drum and the side guard plate; judging whether the coal shearer is in operation according to the target detection result of the coal shearer drum; If it is in the running state, whether to move the rack in time with the machine is determined according to the target detection result of the guard plate; The step of determining whether to timely follow the machine and move the rack according to the target detection result of the guard plate comprises: If the number of guard plates in the target detection result of the guard plates is greater than or equal to the first threshold, it is determined that the frame is moved in time with the machine; if the number of guard plates in the target detection result of the guard plates is less than or equal to the second threshold, it is determined that the frame is not moved in time with the machine; wherein the second threshold is less than the first threshold; if the number of guard plates in the target detection result of the guard plates is greater than the second threshold and less than or equal to the third threshold, it is determined that the frame is in the state of moving with the machine or the state of failure of moving with the machine; wherein the third threshold is less than the first threshold; The acquisition area of the monitoring video includes the roller and a plurality of side guard plates; The step of judging whether the coal mining machine is in operation according to the target detection result includes: Obtain the first position information of the roller in the first image frame and the second position information of the roller in the second image frame in the target detection result; the first image frame and the second image frame are separated by a first preset number of image frames; determine the number of pixels moved in the horizontal direction of the roller according to the first position information and the second position information as a first horizontal displacement; if the ratio of the first horizontal displacement to the number of pixels in the horizontal direction of the image frame is greater than a first preset movement tolerance, determine that the coal mining machine is in operation; or, determine the number of pixels moved in the vertical direction of the roller according to the first position information and the second position information as a first vertical displacement; if the ratio of the first vertical displacement to the number of pixels in the vertical direction of the image frame is greater than a second preset movement tolerance, determine that the coal mining machine is in operation; The method further comprises: If the second position information of the roller does not exist in the second image frame, the third position information of the third image frame is extracted; the first image frame and the third image frame are separated by a second preset number of image frames, the second preset number is n times the first preset number, and n is a positive integer greater than or equal to 1; the number of pixels moved by the roller in the horizontal direction is determined as the second horizontal displacement according to the first position information and the third position information; if the ratio of the second horizontal displacement to the number of pixels in the horizontal direction of the image frame is greater than the third preset movement tolerance, it is determined that the coal mining machine is in operation; the third preset movement tolerance is n times the first preset movement tolerance; or, the number of pixels moved by the roller in the vertical direction is determined as the second vertical displacement according to the first position information and the third position information; if the ratio of the second vertical displacement to the number of pixels in the vertical direction of the image frame is greater than the fourth preset movement tolerance, it is determined that the coal mining machine is in operation; the fourth preset movement tolerance is n times the second preset movement tolerance.
2. The method according to claim 1, It is characterized in that Before judging whether the coal shearer is in operation according to the target detection result of the coal shearer drum, the method further comprises: Determine whether a roller and / or a side guard plate exists in the target detection result of at least one of the image frames within a preset time period; If at least one of the image frames within the preset time period includes a roller and / or a side guard plate, it is determined that the acquisition area of the surveillance video is correct; If each of the image frames within the preset time period does not include the roller and the guard plate, it is determined that the acquisition area of the monitoring video is wrong.
3. The method according to any one of claims 1 to 2, It is characterized in that The method further comprises: Acquire training samples corresponding to image frames of historical monitoring video of the coal mining face; the training samples include the image frames after the positions and classifications of target objects have been marked, and the target objects include at least one of the following: a drum, a rocker arm, a coal flow, and a side guard plate; The target detection model is obtained by training according to the training samples.
4. A coal mining machine follow-up frame detection device, It is characterized in that The device comprises: A video acquisition module is used to acquire monitoring videos of the coal mining face; A target detection module is used to perform target detection on the image frames of the monitoring video to obtain target detection results of the coal mining machine drum and the side guard plate; An operation judgment module, used for judging whether the coal shearer is in operation state according to the target detection result of the coal shearer drum; A module for determining whether to move the rack in time with the machine, if the machine is in operation, according to the target detection result of the side guard plate; The following machine rack shift determination module is specifically used for: If the number of guard plates in the target detection result of the guard plates is greater than or equal to the first threshold, it is determined that the frame is moved in time with the machine; if the number of guard plates in the target detection result of the guard plates is less than or equal to the second threshold, it is determined that the frame is not moved in time with the machine; wherein the second threshold is less than the first threshold; if the number of guard plates in the target detection result of the guard plates is greater than the second threshold and less than or equal to the third threshold, it is determined that the frame is in the state of moving with the machine or the state of failure of moving with the machine; wherein the third threshold is less than the first threshold; The acquisition area of the monitoring video includes the roller and a plurality of side guard plates; The operation judgment module is specifically used for: Obtain the first position information of the roller in the first image frame and the second position information of the roller in the second image frame in the target detection result; the first image frame and the second image frame are separated by a first preset number of image frames; determine the number of pixels moved in the horizontal direction of the roller according to the first position information and the second position information as a first horizontal displacement; if the ratio of the first horizontal displacement to the number of pixels in the horizontal direction of the image frame is greater than a first preset movement tolerance, determine that the coal mining machine is in operation; or, determine the number of pixels moved in the vertical direction of the roller according to the first position information and the second position information as a first vertical displacement; if the ratio of the first vertical displacement to the number of pixels in the vertical direction of the image frame is greater than a second preset movement tolerance, determine that the coal mining machine is in operation; The operation judgment module is also used for: If the second position information of the roller does not exist in the second image frame, the third position information of the third image frame is extracted; the first image frame and the third image frame are separated by a second preset number of image frames, the second preset number is n times the first preset number, and n is a positive integer greater than or equal to 1; the number of pixels moved by the roller in the horizontal direction is determined as the second horizontal displacement according to the first position information and the third position information; if the ratio of the second horizontal displacement to the number of pixels in the horizontal direction of the image frame is greater than the third preset movement tolerance, it is determined that the coal mining machine is in operation; the third preset movement tolerance is n times the first preset movement tolerance; or, the number of pixels moved by the roller in the vertical direction is determined as the second vertical displacement according to the first position information and the third position information; if the ratio of the second vertical displacement to the number of pixels in the vertical direction of the image frame is greater than the fourth preset movement tolerance, it is determined that the coal mining machine is in operation; the fourth preset movement tolerance is n times the second preset movement tolerance.
5. A coal mining machine and machine frame detection system, It is characterized in that Including camera and processor; The camera is used to collect surveillance video; The processor is used to execute the coal mining machine follow-up frame detection method as described in any one of claims 1-3.
