Intelligent identification video following system for coal mine fully-mechanized face equipment

Abstract

The invention provides an intelligent identification video tracking system for coal mine fully-mechanized face equipment, and relates to the technical field of video image processing. The system comprises a coal mining machine operation data interface, an image real-time processing assembly, a central position output assembly, a holder control assembly and a display assembly, the coal mining machine operation data interface is connected with the holder control assembly, and the image real-time processing assembly is connected with the display assembly. Therefore, automatic machine-following positioning operation of images monitored by the cameras can be realized, the consistency of dynamic images displayed by the plurality of cameras is ensured, the video machine-following display effect is improved, an operator can obtain visual environment reconstruction images more conveniently, and the operation condition of a fully-mechanized face can be obtained better and more quickly.

Description

Technical Field

[0001] The present application relates to the technical field of video image processing, and in particular to an intelligent recognition video tracking system for equipment on a fully mechanized mining surface in a coal mine. Background Art

[0002] The fully mechanized coal mining face is a working face that uses comprehensive mechanized coal mining methods to mine coal, including coal mining machines, scraper conveyors, hydraulic supports and other equipment. Since the fully mechanized coal mining face is an important part of coal mining, in order to ensure the production safety and production continuity of the fully mechanized coal mining face, it is necessary to use cameras to monitor the working conditions of the fully mechanized coal mining face in real time.

[0003] Since there are many cameras in the fully mechanized mining face, if the shooting images of all cameras are displayed, it is difficult for operators to quickly obtain target information, and the work efficiency is low. In the prior art, the shooting images of several cameras before and after the coal mining machine's travel position can be displayed on the control center screen, reducing the number of displayed images. However, in the prior art, the movement trajectory and stop position of the camera pan / tilt are not confirmed, which easily leads to confusion in the camera image display. It is difficult for remote control personnel to distinguish the corresponding relationship between the image in the picture and the mining face position, and it is impossible to analyze the real-time status of the mining face as a whole, which seriously restricts the development of remote control of fully mechanized mining faces. Summary of the invention

[0004] The present application aims to solve one of the technical problems in the related art at least to some extent.

[0005] To this end, the first purpose of this application is to propose a coal mine comprehensive mining face equipment intelligent identification video tracking system to realize the control of the pan-tilt camera automatic tracking and positioning operation, and improve the stability and performance of video monitoring of the coal mine comprehensive mining face.

[0006] The second purpose of this application is to propose a method for intelligent identification video tracking of coal mine fully mechanized mining face equipment.

[0007] The third objective of the present application is to provide an electronic device.

[0008] A fourth objective of the present application is to provide a computer-readable storage medium.

[0009] A fifth object of the present application is to provide a computer program product.

[0010] To achieve the above-mentioned purpose, the first embodiment of the present application proposes a coal mine fully mechanized mining face equipment intelligent identification video tracking system, comprising:

[0011] A coal mining machine operation data interface, an image real-time processing component, a center position output component, a pan-tilt control component, and a display component, wherein the coal mining machine operation data interface is connected to the pan-tilt control component, and the image real-time processing component is connected to the display component;

[0012] The coal mining machine operation data interface is used to receive the current position information, operation direction and operation speed of the coal mining machine;

[0013] The real-time image processing component is used to obtain the image data collected by each target camera in the target camera queue corresponding to the position information, and process the image data to determine the bounding box corresponding to the coal mining machine cutting head on the image data, and the first coordinates corresponding to the bounding box;

[0014] The center position output component is used to determine the second coordinate of the center of the coal mining machine based on the first coordinate corresponding to the bounding box;

[0015] The pan / tilt control component is used to generate a follow-up instruction according to the second coordinate, the running speed and the running direction, and send the follow-up instruction to the pan / tilt of the target camera;

[0016] The display component is used to splice and display the image data and the corresponding boundary box of the coal mining machine cutting head based on the target camera queue.

[0017] To achieve the above-mentioned purpose, the second embodiment of the present application proposes a method for intelligently identifying video tracking of coal mine fully mechanized mining face equipment, comprising:

[0018] Receive the current position information, running direction and running speed of the coal mining machine;

[0019] Obtaining image data collected by each target camera in a target camera queue corresponding to the position information;

[0020] Processing the image data to determine a bounding box corresponding to the shearer cutting head on the image data and a first coordinate corresponding to the bounding box;

[0021] Determining a second coordinate corresponding to the center of the coal mining machine according to the first coordinate corresponding to the bounding box;

[0022] A follow-up instruction is generated according to the center position information, the running speed and the running direction, and is sent to the target camera.

[0023] To achieve the above-mentioned purpose, the third aspect embodiment of the present application proposes an electronic device, comprising: a processor; and a memory communicatively connected to the processor; the memory stores computer execution instructions; the processor executes the computer execution instructions stored in the memory, so that the processor can execute the method for intelligent identification video following of coal mine comprehensive mining face equipment described in the above-mentioned second aspect embodiment.

[0024] To achieve the above-mentioned purpose, the fourth aspect embodiment of the present application proposes a computer-readable storage medium on which a computer program is stored, and the computer instructions are used to enable the computer to execute the method for intelligent identification video tracking of coal mine comprehensive mining face equipment described in the above-mentioned second aspect embodiment.

[0025] To achieve the above-mentioned purpose, the fifth aspect embodiment of the present application proposes a computer program product, including a computer program, which, when executed by a processor, implements the method for intelligent identification video tracking of coal mine comprehensive mining face equipment described in the above-mentioned second aspect embodiment.

[0026] The application provides a system and method for intelligent identification of video tracking of coal mine fully mechanized mining equipment. By acquiring the shooting images of some cameras that can capture coal mining machines from all cameras in the fully mechanized mining face, the automatic splicing display of the whole image of the coal mining machine is realized, and the cutting head of the coal mining machine is detected in the shooting images of each camera. Based on the detection results, the position of the coal mining machine in the camera shooting range is calculated, and a tracking instruction is generated to control the pan / tilt of the camera to realize real-time tracking of the coal mining machine for synchronous operation. Automatic tracking and positioning operation of the camera monitoring screen can be realized, the consistency of the dynamic images displayed by multiple cameras can be ensured, the effect of video tracking display can be improved, and it is more convenient for operators to obtain intuitive environmental reconstruction images, and the operating status of the fully mechanized mining face can be better and faster obtained.

[0027] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0029] Figure 1 A schematic diagram of the structure of a coal mine fully mechanized mining face equipment intelligent identification video tracking system provided in an embodiment of the present application;

[0030] Figure 2 A schematic diagram of the structure of a real-time image processing component provided in an embodiment of the present application;

[0031] Figure 3A schematic diagram of the structure of another intelligent recognition video tracking system for coal mine fully mechanized mining face equipment provided in an embodiment of the present application;

[0032] Figure 4 A schematic diagram of equipment for a fully mechanized coal mining face provided in an embodiment of the present application;

[0033] Figure 5 A schematic diagram of a process of intelligent video tracking method for fully mechanized coal mining face equipment provided in an embodiment of the present application;

[0034] Figure 6 A schematic diagram of a recognition model training process provided in an embodiment of the present application. DETAILED DESCRIPTION

[0035] Embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.

[0036] The following describes the intelligent recognition video tracking system and method for coal mine fully mechanized mining face equipment according to an embodiment of the present application with reference to the accompanying drawings.

[0037] It should be noted that in the following embodiments of the present application, the intelligent identification video tracking system for coal mine fully mechanized mining face equipment can be simplified to a video tracking system for the convenience of description.

[0038] Figure 1 : is a structural schematic diagram of a coal mine fully mechanized mining face equipment intelligent identification video tracking system according to an exemplary embodiment. Figure 1 As shown, the video tracking system 100 of the embodiment of the present application includes:

[0039] Coal mining machine operation data interface 10, image real-time processing component 20, center position output component 30, pan-tilt control component 40 and display component 50. Among them, the coal mining machine operation data interface 10 is connected to the pan-tilt control component 30, and the image real-time processing component 20 is connected to the display component 50.

[0040] In some embodiments, the coal mining machine operation data interface 10 is used to receive the current position information, operation direction and operation speed of the coal mining machine.

[0041] In this application, with the principle of resource integration and camera reuse, and the purpose of avoiding duplication, the video tracking system can complete remote monitoring through the existing pan-tilt camera of the comprehensive mining face in actual application. A camera is usually installed every three hydraulic supports on the comprehensive mining face of a coal mine. When the coal mining machine is turned on, the images of all pan-tilt cameras on the comprehensive mining face are polled and detected, so that the specific position information of the coal mining machine at the current moment can be located according to whether the coal mining machine is recognized in the images taken by each camera. Since the shooting range of each camera corresponds to a certain number of hydraulic supports, the current position information of the coal mining machine can be the position number interval of the hydraulic support corresponding to the camera currently shooting the coal mining machine. The relevant status information of the coal mining machine, as well as the running direction and running speed, can also be measured by the sensor installed on the camera. Then the camera can transmit the current position information, running direction and running speed of the coal mining machine to the video tracking system through the coal mining machine operation data interface 10.

[0042] It should be noted that due to other reasons such as the on-site environment and communication, the given coal mining machine location information may be inaccurate. For this reason, in this application, the coal mining machine location information needs to be calibrated based on the position identified and located by the pan-tilt camera. If it exceeds the calibration range, it can be determined that the calibration is abnormal and an alarm will be issued.

[0043] In some embodiments, the real-time image processing component 20 is used to obtain the image data collected by each target camera in the target camera queue corresponding to the position information, and process the image data to determine the bounding box corresponding to the coal mining machine cutting head on the image data, and the first coordinate corresponding to the bounding box.

[0044] In some embodiments, after obtaining the current position information of the coal mining machine, all cameras whose position information is within its shooting range can be determined, and the target camera queue can be determined according to the installation order of these cameras in the fully mechanized mining face. The number of target cameras in the target camera queue may be 4, 6, and so on. Then, the existing network video recorder (NVR) server of the fully mechanized mining face can be directly connected to obtain the video streams of all target cameras, or the video stream can be directly obtained from the target camera (a network camera based on the Internet Protocol (IP)). Then, the video stream is parsed into a single frame image, that is, the image data collected by each target camera can be obtained.

[0045] In some embodiments, after obtaining the image data, the image data can be identified and segmented, the area of ​​the coal mining machine cutting head in the original image can be segmented out, the bounding box corresponding to the coal mining machine cutting head can be obtained, and the first coordinate corresponding to the bounding box can be obtained based on the position of the bounding box in the original image.

[0046] The bounding box is a rectangular box used to enclose all pixel points in the image that represent the cutting head of the coal mining machine. The first coordinates corresponding to the bounding box can be the coordinates of the upper left vertex and the lower right vertex of the bounding box, or the coordinates of the lower left vertex and the upper right vertex, or the coordinates of all vertices, which is not limited in this application.

[0047] It should be noted that image segmentation is a process of dividing an image into several specific regions with unique properties, assigning the same number to the pixels in each region, and then extracting a region with feature matching according to the features of the target extraction object (in this application, the shearer cutting head). In this application, any image segmentation method can be used, such as a threshold-based segmentation method, a region-based segmentation method, an edge-based segmentation method, and a segmentation method based on a specific theory.

[0048] It should be noted that during the operation of the coal mining machine, the tunnel may be filled with smoke, water vapor, etc., and the collected image data contains a lot of noise. At this time, when the image data is processed, the bounding box obtained may have a large error and the image is not clear, which will also intuitively reduce the visualization effect of the entire monitoring system.

[0049] Therefore, in this application, an unsupervised deep learning image denoising method can be used as the basis of Generative Adversarial Networks (GAN), and a learning data set is constructed by GAN to generate simulated real noise images to perform tunnel image denoising. And the "smoke consistency loss" is introduced to maintain the color of the simulated noise image and to make the distribution of the simulated noise more consistent with the distribution of noise in the real image by correcting the discriminator. In addition, the wavelet threshold denoising principle can also be used, and the image and noise have different statistical characteristics after wavelet transformation. The energy of the image information corresponds to the wavelet coefficients with larger copies, which are mainly concentrated in the low-frequency part, while the energy of the noise is distributed in the entire wavelet domain, corresponding to the wavelet coefficients with smaller amplitudes. By setting a suitable threshold, the image noise can be removed, effectively solving the problem of image feature loss in a high dust environment.

[0050] In some embodiments, the center position output component 30 is used to determine the second coordinate corresponding to the center of the coal mining machine based on the first coordinate corresponding to the bounding box.

[0051] The second coordinate is the position of the center of the coal mining machine in the coordinate system established for the fully mechanized mining face of the coal mine. At the same time, the second coordinate is consistent for all target cameras.

[0052] In the embodiment of the present application, since all target cameras in the target camera queue are in order, the shearer cutting head circled by the boundary box in the image data collected by the target camera can be determined as two cutting heads at the front and rear ends of the shearer. The second coordinate corresponding to the center of the shearer can be obtained through certain geometric transformations or mathematical calculations based on the first coordinates of the front and rear cutting heads in the images of each target camera, combined with the position of the target camera in the fully mechanized mining face and the shooting angle.

[0053] In some embodiments, the pan / tilt control component 40 is used to generate a follow-up instruction according to the second coordinate, the running speed and the running direction, and send the follow-up instruction to the pan / tilt of the target camera.

[0054] In an embodiment of the present application, the second coordinate can be combined with the running speed and running direction to determine the position of the coal mining machine in the current shooting range of each target camera in real time, and generate a following machine instruction to control the shooting range of the target camera to move toward the position of the coal mining machine, track and shoot the coal mining machine in real time, and output image data.

[0055] It should be noted that due to the large vibration of the coal mining machine during operation and the high dust in the tunnel, the camera may shake unnecessarily, which may easily lead to a large difference between the results of the detection of the cutting head in two adjacent frames of images collected by the same camera or in the images collected by two adjacent cameras at the same time, thus affecting the smoothness of the camera and the machine shooting. Therefore, a filter tracking algorithm can be used to ensure the continuity and stability of the recognition results of the cutting head in different frames of the same camera. And in order to further save computing resources and reduce the computing responsibility of the graphics processing unit (GPU), a kernel correlation filter algorithm (Kernel Correlation Filter, KCF) can be used.

[0056] In some embodiments, the display component 50 is used to stitch and display the image data and the corresponding bounding box of the coal mining machine cutting head based on the target camera queue.

[0057] In the embodiment of the present application, the image data collected by each target camera can be sequentially spliced ​​according to the order of all target cameras in the target camera queue, and displayed in the display component (such as a display screen, etc.) of the video tracking system, or can also be sent to the display screen of the industrial computer in the underground chute monitoring center for display. And the corresponding boundary box of each coal mining machine cutting head determined by the above-mentioned image real-time processing component 20 can be displayed at the first coordinate position of the corresponding image data. This allows the operator to obtain a more intuitive environmental reconstruction image and better and faster understand the operating status of the fully mechanized mining face.

[0058] In the present application, during the operation or shutdown of the coal mining machine, the pan-tilt camera screen can automatically switch to the coal mining machine position. At the same time, the pan-tilt camera within the coal mining machine area locks each pan-tilt camera screen to match, and can display the corresponding pan-tilt camera dynamic image screen of the coal mining machine on the large screen.

[0059] The intelligent identification video tracking system for coal mine fully mechanized mining face equipment provided in the embodiment of the present application realizes automatic splicing and display of the image of the whole screen of the coal mining machine by acquiring the shooting screen of some cameras that can shoot the coal mining machine among all the cameras in the fully mechanized mining face, and detects the cutting head of the coal mining machine in the shooting screen of each camera, calculates the position of the coal mining machine in the shooting range of the camera based on the detection result, generates a tracking instruction, and controls the pan / tilt of the camera to realize real-time tracking of the coal mining machine for synchronous operation. It can realize automatic tracking and positioning operation of the camera monitoring screen, ensure the consistency of the dynamic screen displayed by multiple cameras, improve the effect of video tracking display, make it easier for operators to obtain intuitive environmental reconstruction images, and better and faster obtain the operating status of the fully mechanized mining face.

[0060] In some embodiments, Figure 2 As shown, the real-time image processing component 20 may include but is not limited to: an image input unit 21 , a feature extraction unit 22 and an image recognition unit 23 .

[0061] In some embodiments, the image input unit 21 is used to obtain the first image data captured by each target camera in the target camera queue corresponding to the current position information of the coal mining machine.

[0062] The first image data is the picture currently captured by each target camera in the target camera queue. The first image data may include the cutting head of the coal mining machine, or may not include the cutting head of the coal mining machine.

[0063] In the embodiment of the present application, the existing NVR server of the comprehensive mining face can be directly connected to obtain the video stream of each target camera, or the video stream can be directly obtained from the target camera (a network camera based on the Internet Protocol (IP)). Then, the video stream is parsed into a single frame image to obtain the first image data collected by each target camera.

[0064] Then, in the image input unit 21, the first image data may be input into the classification model to determine that at least one second image data of the shearer cutting head is contained in the first image data.

[0065] Among them, the classification model can be any training model based on a deep learning algorithm, which can identify whether the image data contains a coal mining machine cutting head, and divide all the first image data into two categories according to the recognition result, and obtain at least one second image data that contains a coal mining machine cutting head in the first image data.

[0066] In some embodiments, the feature extraction unit 22 is used to extract features from each second image data to obtain a feature map.

[0067] In the embodiment of the present application, the feature extraction unit 22 may use a convolutional neural network to extract features from the second image data. The feature extraction unit 22 may first input the second image data into the convolutional neural network, perform a convolution operation on the second image data using a convolution kernel, then perform a pooling operation on the convolved image to reduce the size of the feature map, and finally fully connect all pooling layers to obtain a feature extraction map.

[0068] In some embodiments, the image recognition unit 23 is used to input the feature map and the second image data into a pre-trained recognition model to obtain a bounding box corresponding to the cutting head on the second image data and a first coordinate corresponding to the bounding box in the second image data.

[0069] In the embodiment of the present application, the image recognition unit 23 can use a pre-trained recognition model to perform a series of processing operations on the feature map and the second image data to achieve the purpose of detecting and identifying the shearer cutting head. The recognition model can use a sliding window to generate multiple anchor frames of different sizes and sizes that match the shearer cutting head for each pixel position of the feature map. Each anchor frame can be represented by (x 1 ,y 1 ,x 2 ,y 2 ) means, where (x 1 ,y 1 ) and (x 2 ,y 2 ) represents the coordinates of the upper left corner and the lower right corner of the anchor box. Then, the softmax function can be used to classify the cropping head and the background, and determine the positive sample score corresponding to each anchor box. The positive sample score is the probability that the pixel in the anchor box is the cropping head. All anchor boxes are sorted according to the positive sample score, and the anchor boxes with positive sample scores higher than the threshold are regressed and fine-tuned. The anchor boxes with low overlap with other anchor boxes and small sizes are non-maximum suppressed and mapped to the last layer of convolutional feature map of the convolutional neural network. The category of each anchor box is calculated through the fully connected layer and softmax. That is, the anchor box with the category of cropping head and the highest corresponding positive sample score can be determined as the bounding box corresponding to the cropping head, and the anchor box's (x 1 ,y 1 ) and (x2 ,y 2 ) is the first coordinate corresponding to the bounding box in the second image data.

[0070] In some embodiments, the pan / tilt control component 40 can also be used to generate an instruction for controlling the posture of any target camera to return to the initial posture and send it to any target camera when the shooting range of any target camera does not include the coal mining machine.

[0071] In the embodiment of the present application, after the pan / tilt of the target camera receives the follow-up command, it will control the shooting angle of the target camera to change with the operation of the coal mining machine. After the coal mining machine runs for a certain distance, it may exceed the maximum shooting range of the target camera. At this time, it is necessary to control the shooting angle of the target camera to return to the correct position, or adjust the camera posture to the preset initial posture. This can ensure that the overlap of the shooting ranges of two adjacent non-follow-up monitoring cameras is minimized, which is convenient for controlling the camera to follow the machine next time and reducing the time consumed by recalculating the camera angle to generate the follow-up command.

[0072] Figure 3 FIG. 1 is a structural schematic diagram of another intelligent recognition video tracking system for fully mechanized mining face equipment in a coal mine according to an exemplary embodiment. Figure 3 As shown, based on the above-mentioned embodiment, the video tracking system 100 of the embodiment of the present application may further include: a guard plate status monitoring component 60 and a personnel safety monitoring component 70, wherein the guard plate monitoring component 60 is connected to the display component 50, and the personnel safety monitoring component 70 is also connected to the display component 50.

[0073] In some embodiments, the guard plate status monitoring component 60 is used to solve the posture of multiple guard plates at preset positions based on the image data in the display component, and generate an alarm message and display it in the display component when it is determined that any guard plate has not reached the preset state.

[0074] The preset position refers to a preset number of positions before and after the coal mining machine, each position corresponds to a side guard plate, and the preset number can be determined according to the actual needs of the fully mechanized mining face, for example, it can be 3. The preset state refers to the posture of the side guard plate in the correct supported state, or the posture of the side guard plate in the correct retracted state.

[0075] Combine the following Figure 4 The following is an example of the status monitoring of the guard plate. Figure 4 A schematic diagram of equipment for a fully mechanized coal mining face provided in an embodiment of the present application.

[0076] Depend on Figure 4It can be seen that during the coal mining machine's movement and coal cutting process, the three guard plates before the front drum of the coal mining machine need to be in a recovered state, and the three guard plates after the rear drum need to be in a propped-up state. If the guard plates are not recovered or propped up, it may affect the normal operation of the comprehensive mining face. Therefore, in this application, it is necessary to monitor the state of the guard plates to determine whether the guard plates are recovered. Based on the monitoring image of the hydraulic support guard plates in the image data in the display component, the posture of the guard plates can be solved by image processing technology. When the posture of any guard plate does not reach the preset state, it can be determined that any guard plate has not been recovered. A switch alarm signal can be output for any guard plate (two states: recovered / unrecovered), and an alarm message can be displayed in the display component at the same time. The alarm message can include the number of the hydraulic support corresponding to any guard plate.

[0077] It should be noted that Figure 4 As shown in the fully mechanized mining face, the five PTZ cameras corresponding to the position of the coal mining machine are used to follow the machine to shoot the video of the coal mining machine, and the videos collected by each camera are spliced ​​and displayed. Since the status monitoring of the side guard plate is carried out together with the operation of the coal mining machine, the status monitoring of the side guard plate can be completed directly in the camera screen following the machine, but since the priority of coal mining machine tracking is higher than that of side guard plate monitoring, Figure 4 The cameras at both ends of the five PTZ cameras are used to monitor the status of the guard plates (raised or retracted).

[0078] In some embodiments, the personnel safety monitoring component 70 is used to identify and mark the personnel appearing in the image data collected by all cameras in the comprehensive mining face, and generate and display an alarm message when a person is identified in any image data.

[0079] In this application, the personnel who appear in the screen area of ​​all PTZ cameras in the comprehensive mining face can be automatically identified and framed, so as to achieve efficient video supervision and regulation, and realize real-time, all-weather unattended, automatic monitoring, alarm linkage and triggered recording of abnormal dynamic entry and exit conditions in the monitoring area. In the case where a person is identified in any image data, the any image data can be displayed in the display component 50, and the person can be framed in a rectangular frame in the any image data, and then an alarm can be issued by means of lights, voices and pop-up alarm information on the display screen, so as to promptly remind personnel to evacuate and avoid, and ensure the safety of the comprehensive mining face of the coal mine.

[0080] like Figure 4 As shown, except for the camera used for follow-up shooting, all other cameras in the comprehensive mining face can be used for personnel safety monitoring.

[0081] The intelligent recognition video tracking system for coal mine comprehensive mining face equipment provided in the embodiment of the present application uses the video images captured by the camera to monitor the recovery status of the guard plates and whether the personnel are in the comprehensive mining face working area, so as to realize remote monitoring and early warning, ensure that the guard plates are retracted in time when the coal mining machine passes by, avoid collision with the coal mining machine, protect the equipment from damage, and promptly detect and prevent personnel from entering the dangerous area to avoid personal injury accidents.

[0082] Based on the above embodiments, Figure 5 The present invention provides a flow chart of a method for intelligently identifying video tracking of coal mine fully mechanized mining equipment. Figure 5 As shown, the method is applicable to the above-mentioned coal mine fully mechanized mining face equipment intelligent identification video tracking system, and the method includes but is not limited to the following steps:

[0083] Step 501, receiving the current position information, running direction and running speed of the coal mining machine.

[0084] In some embodiments, the current position information, running direction and running speed of the coal mining machine can be received through the coal mining machine running data interface 102 in the above embodiment. For a detailed description, please refer to the relevant contents in the above embodiment, which will not be repeated here.

[0085] Step 502: Acquire image data collected by each target camera in the target camera queue corresponding to the position information.

[0086] In some embodiments, the image data collected by each target camera in the target camera queue corresponding to the position information can be obtained through the image real-time processing component 20 in the above embodiment. For a detailed description, please refer to the relevant contents in the above embodiment, which will not be repeated here.

[0087] Step 503, processing the image data to determine a bounding box corresponding to the shearer cutting head on the image data and a first coordinate corresponding to the bounding box.

[0088] In some embodiments, the image data can be processed by the real-time image processing component 20 in the above embodiment to determine the bounding box corresponding to the coal mining machine cutting head on the image data and the first coordinates corresponding to the bounding box. For a detailed description, please refer to the relevant contents in the above embodiment, which will not be repeated here.

[0089] Step 504, determining the second coordinate corresponding to the center of the coal mining machine according to the first coordinate corresponding to the bounding box.

[0090] In some embodiments, the second coordinate corresponding to the center of the coal mining machine can be determined according to the first coordinate corresponding to the boundary box by the center position output component 30 in the above embodiment. Specific descriptions can be found in the relevant contents of the above embodiment, which will not be repeated here.

[0091] Step 505: Generate a follow-up instruction based on the second coordinate, the running speed, and the running direction, and send it to the target camera.

[0092] In some embodiments, the pan / tilt control component 40 in the above embodiment can generate a follow-up instruction according to the second coordinate, the running speed and the running direction, and send it to the target camera. For a detailed description, please refer to the relevant contents in the above embodiment, which will not be repeated here.

[0093] In some embodiments, the display component 50 in the above embodiment can also be used to splice and display the image data and the corresponding boundary box of the coal mining machine cutting head based on the target camera queue.

[0094] In some embodiments, the first image data collected by each target camera in the target camera queue corresponding to the current position information of the coal mining machine can be obtained. Then, the first image data is input into the classification model to determine that at least one second image data of the coal mining machine cutting head is included in the first image data. After that, feature extraction is performed on each second image data to obtain a feature map. Finally, the feature map and the second image data are input into a pre-trained recognition model to obtain a bounding box corresponding to the cutting head on the second image data, and a first coordinate corresponding to the bounding box in the second image data.

[0095] It should be noted that the explanation of the aforementioned embodiment of the system for intelligent identification and video tracking of equipment in coal mine comprehensive mining face is also applicable to a method for intelligent identification and video tracking of equipment in coal mine comprehensive mining face of this embodiment, which will not be repeated here.

[0096] In this embodiment, the images of some cameras that can capture the coal mining machine are obtained from all cameras on the fully mechanized mining face, so as to realize the automatic splicing display of the whole image of the coal mining machine, and the cutting head of the coal mining machine is detected in the shooting images of each camera, and the position of the coal mining machine in the camera shooting range is calculated based on the detection results, and the following machine instruction is generated to control the pan / tilt of the camera to realize real-time tracking of the coal mining machine for synchronous operation. The automatic following machine positioning operation of the camera monitoring screen can be realized, the consistency of the dynamic images displayed by multiple cameras can be ensured, the effect of video following machine display can be improved, and it is more convenient for operators to obtain intuitive environmental reconstruction images, so as to obtain the operating status of the fully mechanized mining face better and faster.

[0097] It should be noted that, in the embodiment of the present application, a training process of a recognition model for detecting a shearer cutting head in image data is also included. Figure 6 A flow chart of a recognition model training process provided in an embodiment of the present application. Figure 6 As shown, the training process of the recognition model may include but is not limited to the following steps:

[0098] Step 601, obtaining a training data set.

[0099] The training data set includes a plurality of sample images including a cutting head of a coal mining machine, a reference bounding box corresponding to the cutting head in each sample image, and reference coordinates corresponding to the reference bounding box.

[0100] It should be noted that the reference bounding box corresponding to the cropping head in each sample image and the reference coordinates corresponding to the reference bounding box may be manually annotated and are the actual boundaries of the cropping head in the image.

[0101] It should be noted that due to the special working scene of the coal mining machine, it is difficult to obtain a large amount of sample data. In order to ensure the accuracy and recall rate of the recognition model for target detection, data enhancement methods can be used to expand the samples.

[0102] Optionally, each sample image in the training data set may be transformed to obtain a new sample image, thereby expanding the training data set, wherein the transformation includes at least one of the following methods: horizontal flipping, cropping, and rotation.

[0103] It should be noted that when each sample image in the training data set is transformed, the reference bounding box in the original sample data may also change along with the sample image.

[0104] Alternatively, a first image corresponding to a reference bounding box in a sample image may be extracted, and the first image may be synthesized with a second image excluding the shearer cutting head to obtain a new sample image, thereby expanding the training data set.

[0105] It should be noted that the first image is the image of the cutting head. The first image is synthesized with the second image that does not contain the cutting head of the coal mining machine, and a mix-up enhancement method using linear interpolation is used to increase the robustness of the sample images in the training data set. At this time, the reference bounding box corresponding to the newly obtained sample data is the boundary of the first image.

[0106] Step 602: input each sample image into the initial recognition model to obtain a predicted bounding box output by the initial recognition model and predicted coordinates corresponding to the predicted bounding box.

[0107] Among them, the initial recognition model can be a model built based on a general target detection algorithm, and deep learning can be used to achieve end-to-end target detection.

[0108] In the embodiment of the present application, a convolutional neural network can be used to extract features from each sample data. Then, the initial recognition model can use a sliding window to generate multiple anchor frames of different sizes and sizes that meet the requirements of the coal mining machine cutting head for each pixel position of the feature map corresponding to the sample image. Each anchor frame can be represented by (x 1 ,y 1 ,x 2 ,y 2 ) means, where (x 1 ,y 1 ) and (x 2 ,y 2 ) represents the coordinates of the upper left corner and the lower right corner of the anchor box. Then, the normalized exponential (softmax) function can be used to classify the cropping head and the background, and determine the positive sample score corresponding to each anchor box. The positive sample score is the probability that the pixel in the anchor box is the cropping head. All anchor boxes are sorted according to the positive sample score, and the anchor boxes with positive sample scores higher than the threshold are regressed and fine-tuned. The anchor boxes with low overlap with other anchor boxes and small sizes are non-maximum suppressed and mapped to the last layer of convolutional feature map of the convolutional neural network. The category of each anchor box is calculated through the fully connected layer and softmax. That is, the anchor box with the category of cropping head and the highest corresponding positive sample score can be the predicted bounding box output by the initial recognition model, and the anchor box's (x 1 ,y 1 ) and (x 2 ,y 2 ) is the predicted coordinate corresponding to the predicted bounding box.

[0109] It should be noted that the general target detection algorithm predicts the offset based on the anchor box. The preset anchor box ratio in the existing target detection algorithm usually does not match the ratio of the coal mining machine cutting head. Therefore, in this application, the anchor box with the proportional size that conforms to the coal mining machine cutting head is re-preset.

[0110] Step 603: Determine the difference between the predicted bounding box and the reference bounding box according to the reference coordinates and the predicted coordinates.

[0111] In the embodiment of the present application, in order to make the bounding box more accurate and consistent with the real bounding box of the coal mining machine cutting head, it is necessary to calculate the difference between the predicted bounding box output by the recognition model and the reference bounding box, so as to fine-tune the anchor box with a higher positive sample score obtained by the recognition model, so that the predicted bounding box is closer to the real boundary of the coal mining machine cutting head.

[0112] Step 604: Update the initial recognition model according to the difference until a trained recognition model is obtained.

[0113] For example, the predicted bounding box is A = (A x ,A y ,A w ,A h ), the reference bounding box is G = (G x ,G y ,G w ,G h ), assuming that there is a transformation F, so that the predicted bounding box and the reference bounding box have the following relationship as shown in equation (1):

[0114] F(A x ,A y ,A w ,A h )=(G′ x ,G′ y ,G′ w ,G′ h ) (1)

[0115] Among them, (G x ,G y ,G w ,G h )≈(G′ x ,G′ y ,G′ w ,G′ h ), that is, a bounding box that almost coincides with the reference bounding box.

[0116] Since the difference between the anchor box and the reference bounding box is small after the anchor box is filtered, the relationship between the predicted bounding box and the reference bounding box can be regarded as a linear transformation relationship, so the coordinates can be obtained by the following transformations (2) and (3):

[0117] Formula (2) indicates that the translation transformation is performed first:

[0118] G′ x =A w *d x (x)+A x

[0119] G′ y =A h *d y (x)+A y (2)

[0120] Formula (3) indicates that the scaling transformation is performed again:

[0121] G′ w =A w *exp(dw (x)

[0122] G′ h =A h *exp (d h (x)) (3)

[0123] Then we can use linear regression to find dx(A),dy(A),dw(A),d h (A). Formula (1) is essentially a linear mapping, that is, Y = W · X, where X is the input feature vector, its parameter is W, and Y is the true value. For the anchor box problem, the input X is the convolution feature map, defined as φ, and there is also the transformation between the anchor box and the true box, defined as (t x ,t y ,t h ,t w ), represented by T. Therefore, the following formula (4) can be obtained:

[0124] d * (A) = W * T ·φ(A) (4)

[0125] where d * (A) represents the predicted value of the model output. is a parameter, and φ(A) is the feature vector composed of the convolution feature map corresponding to the anchor box. a are the parameters of the predicted bounding box and the anchor box, respectively. (y, w, h) are the parameters of the predicted bounding box and the anchor box, respectively. Thus, the difference between the predicted bounding box and the reference bounding box determined by formula (4) can be used to determine how to regress and fine-tune the anchor box, thereby updating the initial recognition model and continuously correcting the anchor box on the input feature map until the difference between the predicted bounding box and the reference bounding box output by the recognition model is minimized. Then, the training can be terminated to obtain the trained recognition model.

[0126] In this embodiment, by training the recognition model used to detect the cutting head of the coal mining machine in the image, the cutting head boundary box output by the recognition model is made to be more consistent with the actual cutting head boundary, which provides conditions for improving the accuracy of the camera following instructions, and can more accurately mark the position of the cutting head in the following display video.

[0127] In order to implement the above embodiments, the present application also proposes an electronic device, comprising: a processor, and a memory communicatively connected to the processor; the memory stores computer-executable instructions; the processor executes the computer-executable instructions stored in the memory to implement the method provided by the above embodiments.

[0128] In order to implement the above embodiments, the present application also proposes a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to implement the methods provided by the above embodiments.

[0129] In order to implement the above embodiments, the present application also proposes a computer program product, including a computer program, which implements the methods provided by the above embodiments when executed by a processor.

[0130] The collection, storage, use, processing, transmission, provision and disclosure of user personal information involved in this application are in compliance with relevant laws and regulations and do not violate public order and good morals.

[0131] It should be noted that personal information from users should be collected for legitimate and reasonable purposes and should not be shared or sold outside of these legitimate uses. In addition, such collection / sharing should be carried out after receiving the user's informed consent, including but not limited to notifying the user to read the user agreement / user notice and sign the agreement / authorization including authorization of relevant user information before the user uses the function. In addition, any necessary steps should be taken to protect and safeguard access to such personal information data and ensure that others who have access to personal information data comply with its privacy policy and procedures.

[0132] The present application is expected to provide an implementation scheme for users to selectively block the use or access of personal information data. That is, the present disclosure is expected to provide hardware and / or software to prevent or block access to such personal information data. Once the personal information data is no longer needed, the risk can be minimized by limiting data collection and deleting the data. In addition, when applicable, such personal information is de-identified to protect the privacy of the user.

[0133] In the description of the aforementioned embodiments, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0134] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of this application, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0135] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or more executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present application belong.

[0136] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute the instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purpose of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or processing in other suitable ways if necessary, and then stored in a computer memory.

[0137] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0138] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0139] In addition, each functional unit in each embodiment of the present application may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0140] The storage medium mentioned above may be a read-only memory, a magnetic disk or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limiting the present application. A person of ordinary skill in the art may change, modify, replace and modify the above embodiments within the scope of the present application.

Claims

1. A video tracking system for intelligent identification of coal mine fully mechanized mining equipment, characterized in that: The system comprises: A coal mining machine operation data interface, an image real-time processing component, a center position output component, a pan-tilt control component, and a display component, wherein the coal mining machine operation data interface is connected to the pan-tilt control component, and the image real-time processing component is connected to the display component; The coal mining machine operation data interface is used to receive the current position information, operation direction and operation speed of the coal mining machine; The real-time image processing component is used to obtain the image data collected by each target camera in the target camera queue corresponding to the position information, and process the image data to determine the bounding box corresponding to the coal mining machine cutting head on the image data, and the first coordinates corresponding to the bounding box; The center position output component is used to determine the second coordinate corresponding to the center of the coal mining machine based on the first coordinate corresponding to the bounding box; The pan / tilt control component is used to generate a follow-up instruction according to the second coordinate, the running speed and the running direction, and send the follow-up instruction to the pan / tilt of the target camera; The display component is used to splice and display the image data and the corresponding boundary box of the coal mining machine cutting head based on the target camera queue.

2. The system according to claim 1, characterized in that The real-time image processing component further includes: Image input unit, feature extraction unit and image recognition unit; The image input unit is used to obtain the first image data collected by each target camera in the target camera queue corresponding to the current position information of the coal mining machine, and input the first image data into the classification model to determine that at least one second image data of the coal mining machine cutting head is included in the first image data; The feature extraction unit is used to extract features from each second image data to obtain a feature map; The image recognition unit is used to input the feature map and the second image data into a pre-trained recognition model to obtain a bounding box corresponding to the cutting head on the second image data and a first coordinate corresponding to the bounding box in the second image data.

3. The system according to claim 1, wherein the PTZ control component is further used for: When the coal mining machine is not included in the shooting range of any target camera, a command for controlling the posture of any target camera to return to the initial posture is generated and sent to the any target camera.

4. The system according to claim 1, characterized in that The system further comprises: A guard plate status monitoring component, wherein the guard plate monitoring component is connected to the display component; The guard plate status monitoring component is used to solve the posture of multiple guard plates at preset positions based on the image data in the display component, and generate an alarm message and display it in the display component when it is determined that any of the guard plates has not reached a preset state.

5. The system according to claim 1, characterized in that The system further comprises: A personnel safety monitoring component, wherein the personnel safety monitoring component is connected to the display component; The personnel safety monitoring component is used to identify and mark the personnel appearing in the image data collected by all cameras in the comprehensive mining face, and generate and display an alarm message when a person is identified in any image data.

6. A method for intelligent identification video tracking of coal mine fully mechanized mining equipment, characterized in that: The method comprises: Receive the current position information, running direction and running speed of the coal mining machine; Obtaining image data collected by each target camera in a target camera queue corresponding to the position information; Processing the image data to determine a bounding box corresponding to the shearer cutting head on the image data and a first coordinate corresponding to the bounding box; Determining a second coordinate corresponding to the center of the coal mining machine according to the first coordinate corresponding to the bounding box; A following instruction is generated according to the second coordinate, the running speed and the running direction, and is sent to the target camera.

7. The method according to claim 6, characterized in that After processing the image data and determining the boundary box corresponding to the coal mining machine cutting head on the image data, the method further includes: Based on the target camera queue, the image data and the corresponding boundary box of the coal mining machine cutting head are spliced ​​and displayed.

8. The method according to claim 6, characterized in that The processing of the image data to obtain a bounding box corresponding to the cutting head of the coal mining machine on the image data and coordinates corresponding to the bounding box includes: Obtaining first image data captured by each target camera in a target camera queue corresponding to the current position information of the coal mining machine; Inputting the first image data into a classification model to determine that at least one second image data of the shearer cutting head is included in the first image data; Performing feature extraction on each of the second image data to obtain a feature map; The feature map and the second image data are input into a pre-trained recognition model to obtain a bounding box corresponding to the cutting head on the second image data and a first coordinate corresponding to the bounding box in the second image data.

9. The method according to claim 8, characterized in that Also includes: Acquire a training data set, wherein the training data set includes a plurality of sample images including a cutting head of a coal mining machine, a reference bounding box corresponding to the cutting head in each of the sample images, and reference coordinates corresponding to the reference bounding box; Inputting each sample image into an initial recognition model to obtain a predicted bounding box output by the initial recognition model and predicted coordinates corresponding to the predicted bounding box; Determining a difference between the predicted bounding box and the reference bounding box according to the reference coordinates and the predicted coordinates; The initial recognition model is updated according to the difference until a trained recognition model is obtained.

10. The method according to claim 9, characterized in that After obtaining the training data set, the method further includes: Transforming each sample image in the training data set to obtain a new sample image, and expanding the training data set, wherein the transformation includes at least one of the following methods: horizontal flipping, cropping, and rotation; and / or, A first image corresponding to the reference bounding box in the sample image is extracted, and the first image is synthesized with a second image that does not include the shearer cutting head to obtain a new sample image, thereby expanding the training data set.

Images