Fully mechanized coal mining face tail clearing and transporting control system and method

Abstract

The invention relates to a fully mechanized coal mining face tail clearing and transporting control system and method. The system comprises a coal mining clearing and transporting robot equipment layer, at least two cameras, an intelligent terminal and a sound-light-electricity alarm. Panoramic real-time visual data acquisition is carried out on road conditions at the front end of the coal mining cleaning robot through a fully mechanized coal face tail cleaning control system, a control instruction related to the ground coal quantity and the coal briquette volume is determined, and the cleaning speed of the coal mining cleaning robot and the cutting track of a mechanical arm are calculated. The coal mining cleaning and transporting robot has the function of dynamically adjusting the cleaning and transporting speed based on the ground coal pile; and meanwhile, accurate recognition of large-size coal briquettes is achieved, the cutting track of the mechanical arm is adjusted in a self-adaptive mode according to the recognition result, and accurate crushing of the coal briquettes is completed. The intelligent level of the coal mining cleaning robot is greatly improved, and the coal mining cleaning robot has wide applicability.

Description

Technical Field

[0001] The present application relates to the technical field of coal mining, and in particular to a control system and method for clearing and transporting at the end of a fully mechanized mining face. Background Art

[0002] At the end of the comprehensive mining face, the ground coal piles are scattered in large quantities, and it is usually necessary to concentrate manpower and related equipment to clean up the ground coal piles. The coal mining and transportation robot equipment layer is the main equipment for the removal of ground coal piles. Operators need to control the robot through a remote control device to carry out coal removal and crushing work, but this requires extremely high professional skills from operators. The environment at the end of the comprehensive mining face is often harsh, such as a lot of coal dust, high humidity, and dim light, which brings difficulties to the coal quality identification, operation posture adjustment, and obstacle avoidance of related equipment, thereby greatly increasing the difficulty of removing the ground coal piles, and also bringing great safety risks to operators. Therefore, improving the intelligence level of coal removal and reducing the workload and difficulty of operators in mines are technical problems that need to be solved urgently.

[0003] It should be noted that the above introduction to the technical background is only for the convenience of providing a clear and complete description of the technical solutions of the present application and facilitating the understanding of those skilled in the art. It cannot be considered that the above technical solutions are well known to those skilled in the art simply because they are described in the background technology section of the present application. Summary of the invention

[0004] The purpose of this application is to provide a comprehensive mining face end cleaning control system and method to solve the problems of coal quality identification difficulty, working posture adjustment and obstacle avoidance caused by the low degree of intelligence of coal mining and cleaning equipment in the prior art.

[0005] In order to solve the above problems, a comprehensive mining face end cleaning control system involved in this application adopts the following technical solutions:

[0006] A coal mining and cleaning robot equipment layer, at least two mine-use flameproof and intrinsically safe cameras arranged on the coal mining and cleaning robot equipment layer, a mine-use flameproof and intrinsically safe intelligent terminal arranged on the coal mining and cleaning robot equipment layer, and a mine-use flameproof and intrinsically safe sound and photoelectric alarm arranged on the coal mining and cleaning robot equipment layer, wherein:

[0007] Each of the above-mentioned flameproof and intrinsically safe cameras for mining collects panoramic real-time visual data of the road conditions at the front of the coal mining and removal robot;

[0008] The mine flameproof and intrinsically safe intelligent terminal analyzes the real-time visual data, determines the control instructions related to the amount of coal on the ground and the volume of coal blocks, and identifies the abnormal state of the coal mining and removal robot equipment layer, wherein the coal mining and removal robot equipment layer adjusts the speed and breaks the coal blocks according to the control instructions;

[0009] The mine-used flameproof and intrinsically safe sound and light alarm is used to give an alarm for the action of breaking coal blocks.

[0010] In order to solve the above problems, a method for controlling the end of a fully-mechanized mining face involved in this application is applicable to the control system for controlling the end of a fully-mechanized mining face involved in this application. The method adopts the following technical solutions:

[0011] Obtain initial visual data at the end of fully mechanized mining face mining;

[0012] Based on preprocessing the initial visual data, obtaining first visual data for detecting the amount of coal and second visual data for detecting the volume of coal blocks;

[0013] Based on the first visual data and the second visual data, the cleaning speed of the coal mining and cleaning robot and the cutting trajectory of the robot arm are calculated.

[0014] In order to solve the above problems, a comprehensive mining face end cleaning control device involved in this application adopts the following technical solutions:

[0015] A first acquisition module, the first acquisition module is used to acquire initial visual data of the fully mechanized mining working face;

[0016] a second acquisition module, the second acquisition module being used to acquire first visual data for detecting the amount of coal and second visual data for detecting the volume of coal blocks based on preprocessing the initial visual data;

[0017] A calculation module, wherein the calculation module is used to calculate the coal mining and cleaning robot's cleaning speed and the cutting trajectory of the robot arm based on the first visual data and the second visual data.

[0018] In order to solve the above problems, the present application involves an electronic device, comprising: a processor; a memory for storing executable instructions of the processor; wherein the processor is configured to execute the instructions to implement the end-of-mine working face clearing control method involved in the present application.

[0019] In order to solve the above problems, the present application involves a non-temporary computer-readable storage medium. When the instructions in the storage medium are executed by the processor of an electronic device, the electronic device can execute the comprehensive mining face end clearing control method involved in the present application.

[0020] In order to solve the above-mentioned problem, the present application relates to a computer program product, including a computer program, which, when executed by a processor in a communication device, implements the comprehensive mining face end clearing control method involved in the present application.

[0021] The beneficial effects of this application are as follows:

[0022] Through the comprehensive mining face end cleaning control system, the front-end road conditions of the coal mining and cleaning robot are collected in real time to determine the control instructions related to the ground coal volume and coal block volume, and the cleaning speed of the coal mining and cleaning robot and the cutting trajectory of the mechanical arm are calculated, so that the coal mining and cleaning robot has the function of dynamically adjusting the cleaning speed based on the ground coal pile; at the same time, it can accurately identify large-volume coal blocks, and adaptively adjust the cutting trajectory of the mechanical arm according to the identification results to complete the precise crushing of coal blocks. It greatly improves the intelligence level of the coal mining and cleaning robot and has a wide range of applicability.

[0023] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the technical solution of the embodiment of the present application, the following is a brief introduction to the drawings required for use in the embodiment:

[0025] Figure 1 A schematic diagram of the structure of a comprehensive mining face end cleaning control system provided in an embodiment of the present application;

[0026] Figure 2 A schematic diagram of a method for controlling the end cleaning of a fully mechanized mining face provided in an embodiment of the present application;

[0027] Figure 3 A schematic flow chart of a second method for controlling the end cleaning of a fully mechanized mining face provided in an embodiment of the present application;

[0028] Figure 4 A schematic flow chart of a third method for controlling the end cleaning of a fully mechanized mining face provided in an embodiment of the present application;

[0029] Figure 5 A schematic diagram of the structure of a cleaning control device at the end of a fully mechanized mining face provided in an embodiment of the present application;

[0030] Figure 6 It is a schematic diagram of the operation of the end cleaning control device of the fully mechanized mining working face provided in accordance with the embodiment of the present application;

[0031] Figure 7 A schematic diagram of the structure of an electronic device provided according to an embodiment of the present application;

[0032] Figure 8 It is a schematic diagram of the structure of another electronic device provided according to an embodiment of the present application. DETAILED DESCRIPTION

[0033] In order to make the technical purpose, technical solution and beneficial effects of the present application clearer, the technical solution of the present application is further described below in conjunction with the drawings and specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application, that is, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various different configurations.

[0034] Figure 1 This is a schematic diagram of the structure of a comprehensive mining face end cleaning control system provided in the embodiment of the present application. Figure 1 As shown, the end cleaning control system of the fully mechanized mining working face includes but is not limited to a coal mining and cleaning robot equipment layer (wherein, the coal mining and cleaning robot equipment layer includes a coal mining and cleaning robot, a scraper conveyor, a transfer machine, a crushing device, a mechanical arm for controlling the crushing device, etc.), at least two mine-used flameproof and intrinsically safe cameras (hereinafter referred to as cameras) arranged on the coal mining and cleaning robot equipment layer, a mine-used flameproof and intrinsically safe intelligent terminal (hereinafter referred to as intelligent terminal) arranged on the coal mining and cleaning robot equipment layer, and a mine-used flameproof and intrinsically safe sound and photoelectric alarm (hereinafter referred to as sound and photoelectric alarm) arranged on the coal mining and cleaning robot equipment layer, wherein:

[0035] like Figure 1 As shown in the figure, each camera collects panoramic real-time visual data of the road conditions in front of the coal mining and transportation robot. The intelligent terminal analyzes the real-time visual data, determines the control instructions related to the amount of coal on the ground and the volume of coal blocks, and identifies the abnormal state of the equipment layer of the coal mining and transportation robot. The equipment layer of the coal mining and transportation robot adjusts the speed and breaks the coal blocks according to the control instructions. The sound and light alarm is used to warn of the action of breaking the coal blocks.

[0036] It should be noted that multiple cameras can be distributed at different positions of the coal mining and cleaning robot and provide a 360-degree field of view without blind spots, so that the coal mining and cleaning robot can perceive the surrounding environment in all directions. In this embodiment, two cameras can be used. The two cameras are respectively installed on both sides of the front end of the coal mining and cleaning robot. During the crushing process, the mechanical arm of the crushing device may block the field of view of one of the cameras. However, due to the presence of the other camera, the blocked field of view can be perceived, thereby detecting any potential obstacles or targets on the wall. It should be noted that three cameras can also be used, wherein one camera is set on each side of the front end of the coal mining and cleaning robot, and one camera is set at the front of the top of the coal mining and cleaning robot, so as to further improve the perception of the surrounding environment. On the other hand, setting at least two cameras can also improve the redundancy and reliability of the cleaning control system at the end of the comprehensive mining face. In a complex and harsh working environment, a single camera may not work properly due to obstruction, damage or failure. Through redundant design, even if a camera has a problem, other cameras can still work normally, ensuring that the robot can continuously obtain environmental information and complete the task. This redundant design greatly improves the reliability and stability of the coal mining and cleaning robot. Therefore, the number of cameras and the setting method of the cameras should be selected according to specific needs and usage scenarios. As long as a 360-degree field of view without blind spots can be provided, so that the coal mining and cleaning robot can perceive the surrounding environment in an all-round way, any camera setting method is applicable, and is not limited to this embodiment.

[0037] Furthermore, if Figure 1 As shown, the real-time visual data collected by the camera will be converted into an Ethernet signal format for communication transmission. The camera also supports white light compensation to reduce the adverse effects of the dim underground environment. It also supports video encoding in H.264 and H.265 formats to facilitate subsequent parsing operations. The image resolution of the camera should be set according to specific needs. In this embodiment, the maximum image resolution of real-time visual data is 2560×1440.

[0038] Smart terminals usually have built-in coal quantity monitoring modules and coal block volume detection modules. The coal quantity monitoring module and coal block volume detection module will use relevant algorithms to detect the ground coal quantity and coal block volume in real time, and send corresponding control instructions to the coal mining and transportation robot. The control instructions control the speed adjustment and coal crushing action of the coal mining and transportation robot; at the same time, the sound and light alarm will issue an alarm.

[0039] Furthermore, if Figure 1As shown, the intelligent terminal can use the Gstreamer streaming media service to obtain, decode, convert and other operations on the real-time visual data stream collected by the camera to obtain the RGB image of the perceived environment. If the frame rate is 30 frames per second, a single camera will obtain 30 coal quantity detection results per second. As the amount of data increases, when adjusting the speed of the coal mining and cleaning robot, the control instructions may not respond in time and lag. Therefore, the coal quantity monitoring module is required to perform data cleaning on the RGB image based on the internal algorithm. As an example, the coal quantity detection results can be statistically analyzed within 3 seconds. At the same time, according to the end operation standard of the fully mechanized working face, a first reference value for characterizing a large amount of coal, a second reference value for characterizing a medium amount of coal, and a third reference value for characterizing a small amount of coal are set. Among them, if the number of the first reference value is greater than 60 in the 90 coal quantity detection results, it indicates that the amount of coal on the ground is large; if the number of the second reference value is greater than 60, it indicates that the amount of coal on the ground is moderate; if the number of the third reference value is greater than 60, it indicates that the amount of coal on the ground is small. On the other hand, an image segmentation model can be used (for example, the DeepLabV3 network model in the image segmentation model can be used, and the DeepLabV3 network model includes a backbone layer, a convolutional network layer, and a classification network layer) to detect the amount of coal on the ground. The backbone layer performs deep convolution on the RGB image to extract high-dimensional feature data; the convolutional network layer performs hole convolution on the high-dimensional feature data to obtain two-dimensional output features; the classification network layer is used to parse the two-dimensional output features to obtain the segmentation results of the RGB image and the types of segmentation results, and the coal quantity monitoring module is used to process the segmentation results and the types of segmentation results to obtain a mask image. Two cameras can obtain two mask images, in which the data in the mask image is divided into 0 and 1, 1 indicates that the position of the pixel point is coal, and 0 indicates that the position of the pixel point is not coal. The number of 0s and 1s in the two mask images are counted to determine the three levels of large, moderate, and small ground coal quantity. The specific operation process is not repeated here.

[0040] Then, according to the three levels of ground coal quantity, namely large, moderate and small, the coal mining speed of the coal mining and transportation robot is adjusted by control instructions. When the ground coal quantity is large, the coal mining speed is increased; when the ground coal quantity is moderate, the coal mining speed is kept unchanged; when the ground coal quantity is small, the coal mining speed is reduced.

[0041] While adjusting the coal mining speed, the coal block volume detection module of the intelligent terminal detects the coal block volume in real time. If there are two cameras and the frame rate is 30 frames per second, the coal block volume detection module will obtain 60 volume detection results per second. In the volume detection results, it is assumed that the detection area of ​​the coal volume is (x1, y1, w1, h1), where x1, y1, w1, and h1 represent the coordinates and width and height respectively. By intercepting the image of the detection area, the detection area of ​​the large coal block in the coal block volume is obtained (x2, y2, w2, h2).

[0042] Among them, the detection area of ​​coal quantity and the detection area of ​​large coal blocks (x2, y2, w2, h2) also include image scaling and feature extraction operations. The specific operation process is not repeated here. Next, the following processing can be done: Assuming that the current time is t, the number of large volume coal blocks detected from time t to time (t+1) is The number of large coal blocks detected from time (t+1) to time (t+2) is The number of large coal blocks detected from time (t+2) to time (t+3) is As an example, if all

[0043] ≥40, it indicates that a large volume of coal blocks has been staying in front of the coal mining and removal robot for a long time and cannot be removed, and the large volume of coal blocks needs to be cut; otherwise, it indicates that there is no large volume of coal blocks in front of the coal mining and removal robot. On the other hand, the RGB image can be normalized; and according to the directional gradient histogram feature extraction algorithm, the result of the normalization operation is extracted to obtain the corresponding feature variable; then the target detection algorithm (for example, the YOLOv4 network architecture algorithm can be used) is used to obtain the three parameters of the feature variable category, size and position, and the size of the dimensions occupied by the category, size and position, and according to the confidence judgment and non-maximum suppression and other operations, the volume of the coal block is determined. As an example, by modifying the input layer of the network architecture of YOLOv4, the original RGB image (for example, the feature of the original RGB image is 608×608×3) is converted into the feature input of the directional gradient histogram feature extraction algorithm (the feature input of the directional gradient histogram feature extraction algorithm is 416×416×16), and the feature input of the directional gradient histogram feature extraction algorithm is processed by the feature extraction layer and prediction layer of the directional gradient histogram feature extraction algorithm to output the category, size and position of the target. The output layer has three size dimensions, namely (76×76×18), (19×19×18) and (38×38×18), where the first two dimensions represent the size of the data set to facilitate the detection of multi-scale targets; the size of the third dimension is determined by the target category of the data set. Each target of the directional gradient histogram feature extraction algorithm gives three prediction boxes of different scales, and each prediction box will contain all categories of the data set. In addition, the position size will occupy 4 dimensions and the position confidence will occupy 1 dimension. Then the target category of the data set = 3×(number of categories + 4 + 1). In this embodiment, the number of categories of large media blocks is 1, so the target category of the output layer is equal to 18.

[0044] Furthermore, the confidence level is determined by setting a target confidence threshold and then traversing the target confidence information of the output layer. If the target confidence information is lower than the target confidence threshold, it is considered to have low reliability and is removed from the detection result, thereby ensuring the accuracy of target detection. Non-maximum suppression can be used to remove redundant data, and its specific operation process will not be repeated here.

[0045] On the other hand, according to the coal block volume detection results, the intelligent terminal issues a control command to control the crushing device of the coal mining and transportation robot equipment layer to perform the crushing work. At the same time, the sound and light alarm broadcasts: "Attention! Attention! A large coal block is found and is being crushed."

[0046] In summary, the coal mining and transportation robot can collect panoramic real-time visual data of the road conditions at the front end of the fully mechanized mining face through the cleaning control system, determine the control instructions related to the amount of ground coal and the volume of coal blocks, calculate the cleaning speed of the coal mining and transportation robot and the cutting trajectory of the mechanical arm, so that the coal mining and transportation robot can dynamically adjust the cleaning speed based on the ground coal pile; at the same time, it can accurately identify large-volume coal blocks, and adaptively adjust the cutting trajectory of the mechanical arm according to the identification results to complete the precise crushing of coal blocks. It greatly improves the intelligence level of the coal mining and transportation robot and has a wide range of applicability.

[0047] Figure 2 This is a flow chart of a method for controlling the end cleaning of a fully mechanized working face provided in an embodiment of the present application. Figure 2 As shown, the method is applicable to the end cleaning control system of the fully mechanized mining working face provided in the embodiment of the present application, and the method includes but is not limited to the following steps:

[0048] S201, obtaining initial visual data at the end of mining of the fully mechanized mining face.

[0049] In a feasible implementation, the cleaning work at the end of the fully-mechanized working face is an important part of the coal mining process, which involves the dismantling of the working face equipment, the cleaning of coal, and the subsequent safety treatment. Before the cleaning work begins, a comprehensive safety inspection is required, including the stability of the top and bottom plates of the working face, the integrity of the cables and conveying equipment, and whether there are dangerous substances such as combustible gas and combustible dust. Ensuring the safety of the working area is the primary task of the cleaning work. Therefore, in order to obtain the cleaning speed of the coal mining and cleaning robot and the cutting trajectory of the robotic arm, it is necessary to obtain the initial visual data at the end of the fully-mechanized working face mining, and the initial visual data should cover the key areas and the working surface. In order to meet this requirement, two cameras are usually installed on the coal mining and cleaning robot to collect the front road conditions of the coal mining and cleaning robot in real time, obtain streaming media data, and convert the streaming media data into initial visual data suitable for network communication transmission. At the same time, the camera also supports white light compensation, which is used to reduce the adverse effects of the dim environment caused by the coal dust, high humidity, and dark light at the end of the fully-mechanized working face mining. On the other hand, the camera also supports video encoding formats such as H.264 and H.265, which can ensure excellent data network transmission capabilities and adaptability at the end of mining in a dim environment. Furthermore, based on the collected initial visual data, it can also identify and classify the end of mining at the comprehensive mining face, such as key elements such as coal seams, roofs, ground coal piles, support pressures of hydraulic supports, and cutting speeds of the mechanical arms of coal mining and transportation robots. It can also analyze abnormal conditions, such as roof fall, equipment failures (for example, failures of hydraulic supports, failures of fans, etc.), and can issue warning information through the warning equipment set on the coal mining and transportation robot. It should be noted that the warning equipment can be an audible and visual alarm, and the warning information issued by the audible and visual alarm can be customized according to the actual environment at the end of the comprehensive mining face.

[0050] S202, based on preprocessing the initial visual data, obtaining first visual data for detecting the amount of coal and second visual data for detecting the volume of coal blocks.

[0051] In a feasible implementation, the initial visual data contains a variety of information such as ambient lighting changes, shadows, coal pile morphology, coal block size and distribution. In the process of detecting the amount of coal and the volume of coal blocks based on the initial visual data, the preprocessing step is crucial because it can significantly improve the effect of subsequent image analysis and processing. First, the color image in the initial visual data is converted into a grayscale image, because the grayscale image is simpler to process and is usually sufficient to meet the needs of coal quantity detection and volume estimation. Therefore, this operation can remove color information and reduce the amount of data processed. Next, a noise filtering operation is performed. Filtering techniques (such as Gaussian filtering, median filtering, etc.) can be used to remove noise in the initial visual data, such as dust, random pixels, etc., to improve data quality. Next, an edge detection operation is performed. The contour edge of the coal pile or coal block can be extracted by an edge detection algorithm (such as Canny edge detector, Sobel operator, etc.). Edge information is crucial for subsequent coal quantity estimation and volume calculation. Then, image segmentation techniques (such as threshold segmentation, region-based segmentation, graph-based segmentation, etc.) are used to separate the coal pile area from the background area. This operation may need to be adjusted according to the color, brightness and other characteristics of the coal pile.

[0052] Furthermore, global features are extracted from the segmented coal pile visual information, such as the projected area of ​​the coal pile, the grayscale mean, etc. These features can be used to estimate the total amount of coal. The extracted global features are used as the first visual data for training and prediction of a coal quantity detection model (the coal quantity detection model can be set based on a machine learning model, a regression model, etc., and the specific setting process is not repeated here).

[0053] Further identifying independent coal blocks from the segmented coal pile visual information requires more sophisticated image segmentation or shape-based recognition algorithms. As an example, semantic segmentation can be used to assign a specific category label to each pixel in the coal pile visual information, and then the category labels are accurately classified based on the convolutional neural network. The classification results can be used to estimate the volume of the coal blocks. Furthermore, for each identified coal block, its volume is estimated based on its shape (assuming it is a cube, ellipsoid, etc.) and size (length, width, height, etc.). The mentioned data of each coal block and related image features (such as contours, shape parameters, etc.) are then used as the second visual data, which is used for the training and prediction of the coal block volume detection model (such as a classification model, regression model, etc.).

[0054] S203, calculating the coal mining and cleaning robot's cleaning speed and the cutting trajectory of the robot arm based on the first visual data and the second visual data.

[0055] In a feasible implementation, the first visual data and the second visual data are integrated, and these data come from different perception networks, such as cameras, laser radars, inertial measurement units, etc. For the inertial measurement unit, the SLAM (Simultaneous Localization and Mapping) technology can be used to estimate the position and posture of the coal mining and cleaning robot; according to the position of the robot, its relative position and direction relative to the ground coal pile are calculated. Then, the movement of the coal block or coal pile in the first visual data is analyzed, which can be achieved by the pixel changes between consecutive frames. Then, the moving speed of the coal block or coal pile is calculated using the optical flow method or feature point tracking technology. Combined with the relative position and direction of the coal mining and cleaning robot, the moving speed of the coal block or coal pile is converted into the cleaning speed of the coal mining and cleaning robot.

[0056] According to the second vision data, the surface contours of the coal pile and obstacles (such as rocks) on the ground are identified. Using the path planning algorithm, a collision-free path is planned for the robot arm from the current position to the target cutting point. Considering the dynamic characteristics and physical limitations of the robot arm (such as joint angles, speed limits, etc.), the path is smoothed to generate the final cutting trajectory.

[0057] When the coal mining and clearing robot performs clearing and cutting tasks, it continuously monitors the first visual data and the second visual data. Based on real-time data feedback, it dynamically adjusts the clearing speed and cutting trajectory to adapt to uncertain factors such as coal seam changes and obstacles. Machine learning or optimization algorithms can also be used to continuously optimize the clearing and cutting strategies to improve the working efficiency and safety of the coal mining and clearing robot.

[0058] Furthermore, the calculated cleaning speed and the cutting trajectory of the robot arm can be output to the control system of the coal mining robot. As an example, the control system can be an intelligent terminal. In this embodiment, the intelligent terminal drives the coal mining and cleaning robot to perform corresponding cleaning and cutting operations according to the instructions. On the other hand, the intelligent terminal can also receive the streaming media data of the camera and convert the streaming media data into initial visual data suitable for network communication transmission; and control the sound and light alarm to issue warning information through instructions.

[0059] In summary, the coal mining and transportation robot can collect panoramic real-time visual data of the road conditions at the front end of the fully mechanized mining face through the cleaning control system, determine the control instructions related to the amount of ground coal and the volume of coal blocks, calculate the cleaning speed of the coal mining and transportation robot and the cutting trajectory of the mechanical arm, so that the coal mining and transportation robot can dynamically adjust the cleaning speed based on the ground coal pile; at the same time, it can accurately identify large-volume coal blocks, and adaptively adjust the cutting trajectory of the mechanical arm according to the identification results to complete the precise crushing of coal blocks. It greatly improves the intelligence level of the coal mining and transportation robot and has a wide range of applicability.

[0060] Figure 3 This is a flow chart of the second method for controlling the end cleaning of a fully mechanized mining face provided in the embodiment of the present application. Figure 3 As shown, the method is applicable to the end cleaning control system of the fully mechanized mining working face provided in the embodiment of the present application, and the method includes but is not limited to the following steps:

[0061] S301, determine the visual information that needs to be captured based on the geological environment at the end of mining of the fully mechanized mining face.

[0062] In a feasible implementation, the geological environment at the end of the comprehensive mining face is identified to obtain the texture information of the ground coal pile. Based on the geological mapping of the texture information, the outline of the ground coal pile is determined, wherein, based on the image processing operation, the texture information is feature analyzed to obtain the spatial arrangement and distribution law of the ground coal pile; according to the spatial arrangement and distribution law, the scale for mapping is determined; based on the scale, the texture information is geologically mapped (geological mapping aims to identify the geological structural characteristics and the geological conditions of mineral formation and occurrence in the working area or the discovered mining area through systematic geological observation and research, and provide data basis for further prospecting or exploration work) to determine the outline of the ground coal pile.

[0063] After determining the outline of the coal pile on the ground, the visual information collection scheme is determined based on feature extraction of the outline. The outline can be deeply learned based on a convolutional neural network to obtain geometric features and shape features after noise reduction. The visual information collection scheme is determined based on the geometric features and shape features. It should be noted that the visual information collection scheme involves multiple links such as the selection, deployment, data transmission, processing and analysis of perception equipment.

[0064] After determining the visual information collection plan, determine the collection points, configuration parameters of the collection equipment, collection path and emergency plan based on the visual information collection plan.

[0065] It should be noted that in the geological environment at the end of mining in the fully mechanized working face, the visual information that needs to be captured is crucial to ensure the safety, efficiency and accuracy of the mining process. This visual information includes geological structure characteristics, working face status, equipment operation status, lighting and visibility, safety hazards and abnormal phenomena, video monitoring systems, etc. Among them:

[0066] Geological structural characteristics include rock strata distribution, faults and folds, among which rock strata distribution includes rock strata thickness, bedding, joints and other characteristics to assess the stability of the rock strata and possible geological risks; faults and folds often have a significant impact on the mining process and may lead to problems such as rock strata breakage and reduced stability.

[0067] The working face status includes the status of the support and roof, coal wall and goaf, etc. Among them, the status of the support and roof is used to observe the supporting status of the support, whether there are signs of roof collapse or sinking, and the contact between the support and the roof, to ensure the overall stability of the working face; the coal wall and goaf are used to monitor the stability of the coal wall, including whether the coal wall is flat, whether there is spalling or collapse, to prevent the collapse of the goaf from causing impact on the working face.

[0068] The equipment operating status includes observing the operating status of the coal mining machine and scraper conveyor, including whether the equipment is operating normally, whether there is any jamming or wear, and the coordinated operation between the equipment; checking whether the hydraulic system of the hydraulic support is stable, whether the pressure is sufficient, whether the support moves according to the predetermined trajectory, etc., to ensure the reliability of the support system.

[0069] Lighting and visibility include lighting conditions, dust and water mist, etc. Among them, since a low-light environment may be encountered at the end of mining, it is necessary to ensure sufficient lighting conditions to clearly observe the working surface conditions. You can consider using lighting equipment with adjustable brightness to improve the lighting effect; monitor and reduce the impact of dust and water mist on the visual environment of the working surface, and improve visibility by using dust removal equipment and optimizing the spray system.

[0070] Safety hazards and abnormal phenomena include timely discovery and recording of potential safety hazards on the working face, such as cracks in the roof, skewed supports, equipment failures, etc., so that timely measures can be taken to deal with them; pay attention to observing and recording abnormal phenomena that occur on the working face, such as sudden changes in the coal wall, abnormal sounds in the roof, etc., which may be harbingers of changes in geological conditions.

[0071] The video surveillance system includes the use of a high-definition video surveillance system to monitor the work surface in real time, capturing and recording visual information of key areas.

[0072] S302, based on the visual information, obtaining the initial visual data at the end of mining of the fully mechanized mining face.

[0073] In a feasible implementation, based on visual information, a sensing device is used to perform dynamic video acquisition of the fully-mechanized mining face to obtain streaming media data. As an example, a camera can be used to obtain streaming media data of the environment in front of the coal mining and removal robot. Based on editing and format conversion of the streaming media data, the initial visual data at the end of the fully-mechanized mining face is obtained. As an example, the collected streaming media data can be encoded and compressed and converted into Ethernet optical signals, Ethernet electrical signals, and power carrier communication signals, and based on the standard RTSP video stream transmission protocol (full name Real Time Streaming Protocol, an important application layer protocol in the TCP / IP protocol system, which is mainly used to control the streaming media server to send multimedia data, such as audio, video, etc.), the Ethernet optical signal, Ethernet electrical signal, and power carrier communication signal are integrated to obtain streaming media data.

[0074] For further details of step S302, please refer to the relevant contents in the above embodiment, which will not be repeated here.

[0075] S303, based on preprocessing the initial visual data, obtaining first visual data for detecting the amount of coal and second visual data for detecting the volume of coal blocks.

[0076] In a feasible implementation, the first data to be detected is obtained based on preprocessing the initial visual data. Based on the deep learning model, the first detection data is subjected to image semantic segmentation to obtain the first visual data for detecting the amount of coal, wherein, based on the backbone network in the deep learning model, the first detection data is subjected to feature extraction to obtain backbone feature data; based on the convolutional network in the deep learning model, the backbone feature data is subjected to dilated convolution to obtain dilated convolution domain data. As an example, the ASPP (Atrous Spatital Pyramid Pooling) network structure can be used to perceive backbone feature data of different scales, and at the same time, dilated convolution is introduced into the network structure. By using dilated convolutions with different sampling rates, the multi-scale problem in the segmentation task is processed in a cascade or parallel manner. Furthermore, for each position i and filter w on the dilated convolution domain data y, the input backbone feature data x is mapped to the dilated convolution domain to obtain the following relationship:

[0077] y[i]=∑x[i+r*k]*w[k]

[0078] Among them, r represents the dilation rate, which is the input signal corresponding to the sample stride; k represents the order of the filter; the dilated convolution can adaptively modify the receptive field of the filter by changing the size of the dilation rate.

[0079] After obtaining the hole convolution domain data, the hole convolution domain data is subjected to image semantic segmentation based on the classification network in the deep learning model to obtain the first visual data for detecting the amount of coal. As an example, the hole convolution domain data can be segmented according to the type of segmentation target to obtain a mask image related to the amount of coal. The mask image can be obtained by the camera of the coal mining and transportation robot. The camera can obtain two mask images for each acquisition operation. Tools such as OpenCV can be used to perform reverse addition operations on the two mask images. If the value of the pixel point is zero, it means that there is no coal in the area corresponding to the pixel point; if the value of the pixel point is non-zero, it means that there is coal in the area corresponding to the pixel point. The non-zero pixel points are counted and compared with the set threshold. According to the comparison result, the amount of coal in the ground coal pile is divided into orders of magnitude. Furthermore, the statistical result of the non-zero pixel points is the first visual data for detecting the amount of coal.

[0080] In a feasible implementation, the second data to be detected is obtained based on preprocessing the initial visual data. The third data to be detected is obtained based on normalizing the second data to be detected (eliminating the dimension and value range differences between the data, making different indicators comparable, and also facilitating subsequent data processing and analysis). Based on the target detection algorithm model, image features are extracted from the third data to be detected to obtain the second visual data for detecting the volume of the coal block. Furthermore, based on the directional gradient histogram network in the target detection algorithm model, feature extraction is performed on the third data to be detected to obtain directional gradient data (the directional gradient data includes the type, position and size of the target); based on the prediction network in the target detection algorithm model, confidence judgment is performed on the directional gradient data (a target confidence threshold is set, and the target confidence information of the output layer is traversed. Parameters below the threshold are considered to have low reliability, so they are removed from the detection results to ensure the accuracy of target detection) and non-maximum suppression is performed (anchor method is used to generate a large number of candidate bounding boxes with different lengths and widths near the target area. To ensure that each target output has a relatively optimal bounding box, after the algorithm predicts and generates the bounding box, a non-maximum suppression operation will be performed for filtering to remove redundant data) to obtain the second visual data for detecting the volume of coal blocks.

[0081] For further details of step S303, please refer to the relevant contents in the above embodiment, which will not be repeated here.

[0082] S304, based on the first visual data and the second visual data, calculating the coal mining and cleaning robot's cleaning speed and the cutting trajectory of the robot arm.

[0083] For further details of step S304, please refer to the relevant contents in the above embodiment, which will not be repeated here.

[0084] In summary, the coal mining and transportation robot can collect panoramic real-time visual data of the road conditions at the front end of the fully mechanized mining face through the cleaning control system, determine the control instructions related to the amount of ground coal and the volume of coal blocks, calculate the cleaning speed of the coal mining and transportation robot and the cutting trajectory of the mechanical arm, so that the coal mining and transportation robot can dynamically adjust the cleaning speed based on the ground coal pile; at the same time, it can accurately identify large-volume coal blocks, and adaptively adjust the cutting trajectory of the mechanical arm according to the identification results to complete the precise crushing of coal blocks. It greatly improves the intelligence level of the coal mining and transportation robot and has a wide range of applicability.

[0085] Figure 4 This is a flow chart of the third method for controlling the end cleaning of a fully mechanized mining face provided in the embodiment of the present application. Figure 4 As shown, the method is applicable to the end cleaning control system of the fully mechanized mining working face provided in the embodiment of the present application, and the method includes but is not limited to the following steps:

[0086] S401, obtaining initial visual data at the end of mining of the fully mechanized mining face, and obtaining streaming media data based on the initial visual data.

[0087] For further details of step S401, please refer to the relevant contents in the above embodiment, which will not be repeated here.

[0088] S402, based on the mode conversion of the streaming media data, an RGB image of the perceived environment is obtained.

[0089] For further details of step S402, please refer to the relevant contents in the above embodiment, which will not be repeated here.

[0090] S403, acquiring a plurality of first detection results based on coal quantity detection on the RGB image.

[0091] For further details of step S403, please refer to the relevant contents in the above embodiment, which will not be repeated here.

[0092] S404, obtaining a second detection result based on data cleaning of the first detection result.

[0093] For further details of step S404, please refer to the relevant contents in the above embodiment, which will not be repeated here.

[0094] S405: Perform statistics on the second detection result within a preset time to obtain a first statistical result.

[0095] For further details of step S405, please refer to the relevant contents in the above embodiment, which will not be repeated here.

[0096] S406: According to the first statistical result, the amount of coal in the ground coal pile is divided into orders of magnitude.

[0097] For further details of step S406, please refer to the relevant contents in the above embodiment, which will not be repeated here.

[0098] S407, adjusting the cleaning speed of the coal mining and cleaning robot according to the specific order of magnitude.

[0099] It should be noted that if there is no need to adjust the coal mining and removal speed of the coal mining and removal robot, the coal mining and removal robot runs at a constant speed and returns to step S401.

[0100] For further details of step S407, please refer to the relevant contents in the above embodiment, which will not be repeated here.

[0101] S408, obtaining a plurality of third detection results based on coal block volume detection on the RGB image.

[0102] For further details of step S408, please refer to the relevant contents in the above embodiment, which will not be repeated here.

[0103] S409: Acquire a fourth detection result based on data cleaning of the third detection result.

[0104] For further details of step S409, please refer to the relevant contents in the above embodiments, which will not be repeated here.

[0105] S410, within a preset time, collect statistics on the fourth detection result to obtain a second statistical result.

[0106] As an example, if no large media block is detected, return to step S401.

[0107] For further details of step S410, please refer to the relevant contents in the above embodiments, which will not be repeated here.

[0108] S411, adjusting the cutting trajectory of the mechanical arm of the coal mining and cleaning robot according to the second statistical result, and outputting corresponding warning information according to the second statistical result.

[0109] For further details of step S411, please refer to the relevant contents in the above embodiments, which will not be repeated here.

[0110] In summary, the coal mining and transportation robot can collect panoramic real-time visual data of the road conditions at the front end of the fully-mechanized mining face through the cleaning control system, determine the control instructions related to the amount of ground coal and the volume of coal blocks, calculate the cleaning speed of the coal mining and transportation robot and the cutting trajectory of the mechanical arm, so that the coal mining and transportation robot has the function of dynamically adjusting the cleaning speed based on the ground coal pile; at the same time, it can accurately identify large-volume coal blocks, and adaptively adjust the cutting trajectory of the mechanical arm according to the identification results to complete the precise crushing of coal blocks. It greatly improves the intelligence level of the coal mining and transportation robot and has a wide range of applicability.

[0111] Figure 5 This is a schematic diagram of the structure of a comprehensive mining face end cleaning control device provided in the embodiment of the present application. Figure 5 As shown, the fully mechanized mining face end cleaning control device 500 comprises:

[0112] A first acquisition module 501, the first acquisition module 501 is used to acquire initial visual data of the fully mechanized mining working face;

[0113] A second acquisition module 502, the second acquisition module 502 is used to acquire first visual data for detecting the amount of coal and second visual data for detecting the volume of coal blocks based on preprocessing the initial visual data;

[0114] The calculation module 503 is used to calculate the coal mining and cleaning robot's cleaning speed and the cutting trajectory of the robot arm based on the first visual data and the second visual data.

[0115] As an example, Figure 6 The schematic diagram of the operation of the end-of-comprehensive mining face cleaning control device provided according to the embodiment of the present application is shown in the figure. As shown in the figure, the end-of-comprehensive mining face cleaning control device 500 obtains the first visual data for detecting the amount of coal and the second visual data for detecting the volume of coal blocks, and calculates the cleaning speed of the coal mining and cleaning robot and the cutting trajectory of the mechanical arm based on the first visual data and the second visual data, and transmits the cleaning speed and the cutting trajectory of the mechanical arm to the intelligent terminal of the coal mining and cleaning robot, which is converted into corresponding control instructions by the intelligent terminal, and based on the control instructions, drives the cleaning speed action of the coal mining and cleaning robot and the cutting action of the mechanical arm.

[0116] Figure 7 The present invention is a schematic diagram of the structure of an electronic device provided according to an embodiment of the present application. Figure 7 The electronic device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0117] like Figure 7As shown, the electronic device 700 includes a processor 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a memory 706 to a random access memory (RAM) 703. In the RAM 703, various programs and data required for the operation of the electronic device 700 are also stored. The processor 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0118] The following components are connected to the I / O interface 705: a memory 706 including a hard disk, etc.; and a communication part 707 including a network interface card such as a LAN (Local Area Network) card, a modem, etc., and the communication part 707 performs communication processing via a network such as the Internet; a drive 608 is also connected to the I / O interface 705 as needed.

[0119] In particular, according to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 707. When the computer program is executed by the processor 701, the above-mentioned functions defined in the method of the present application are executed.

[0120] In an exemplary embodiment, a storage medium including instructions is also provided, such as a memory including instructions, and the instructions can be executed by the processor 701 of the electronic device 700 to complete the above method. Optionally, the storage medium can be a non-transitory computer-readable storage medium, for example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0121] In the present application, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, apparatus, or device. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in combination with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium may be transmitted using any suitable medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0122] Figure 8 It is a schematic diagram of the structure of another electronic device provided according to an embodiment of the present application. Figure 8 The electronic device shown is only an example and should not limit the functions and scope of use of the embodiments of the present application. Figure 8 As shown, the electronic device 800 includes a processor 801 and a memory 802. The memory 802 is used to store program codes, and the processor 801 is connected to the memory 802 to read the program codes from the memory 802 to implement the end cleaning control method of the fully mechanized working face in the above embodiment.

[0123] Optionally, the number of processors 801 may be one or more.

[0124] Optionally, the electronic device may further include an interface 803, and the number of the interface 803 may be multiple. The interface 803 may be connected to an application program and may receive data from an external device such as a sensor.

[0125] Finally, it should be noted that the above embodiments are only used to illustrate but not limit the technical solutions of the present application. Any equivalent substitutions of the present application and any modifications or partial substitutions that do not depart from the spirit and scope of the present application shall be included in the scope of protection of the claims of the present application.

Claims

1. A comprehensive mining face end cleaning control system, characterized in that: include: A coal mining and cleaning robot equipment layer, at least two mine-use flameproof and intrinsically safe cameras arranged on the coal mining and cleaning robot equipment layer, a mine-use flameproof and intrinsically safe intelligent terminal arranged on the coal mining and cleaning robot equipment layer, and a mine-use flameproof and intrinsically safe sound and photoelectric alarm arranged on the coal mining and cleaning robot equipment layer, wherein: Each of the above-mentioned flameproof and intrinsically safe cameras for mining collects panoramic real-time visual data of the road conditions at the front end of the coal mining and removal robot; The mine flameproof and intrinsically safe intelligent terminal analyzes the real-time visual data to determine the control instructions related to the amount of coal on the ground and the volume of coal blocks, wherein the coal mining and removal robot equipment layer adjusts the speed and breaks the coal blocks according to the control instructions; The mine-used flameproof and intrinsically safe sound and light alarm is used to give an alarm for the action of breaking coal blocks.

2. A method for controlling the end cleaning of a fully mechanized working face, characterized in that: Applicable to the end cleaning and transportation control system of the fully mechanized mining face according to claim 1, the method comprises: Obtain initial visual data at the end of fully mechanized mining face mining; Based on preprocessing the initial visual data, obtaining first visual data for detecting the amount of coal and second visual data for detecting the volume of coal blocks; Based on the first visual data and the second visual data, the cleaning speed of the coal mining and cleaning robot and the cutting trajectory of the robot arm are calculated.

3. The method according to claim 2, characterized in that The obtaining of initial visual data at the end of mining of the fully mechanized mining face includes: Determine the visual information that needs to be captured based on the geological environment at the end of mining in the fully mechanized mining face; Based on the visual information, a sensing device is used to collect dynamic video of the fully mechanized mining face to obtain streaming media data; Based on editing and format conversion of the streaming media data, initial visual data at the end of mining of the comprehensive mining face is obtained.

4. The method according to claim 3, characterized in that: The visual information that needs to be captured is determined based on the geological environment at the end of mining in the fully mechanized mining face, including: Identify the geological environment at the end of fully mechanized mining face mining and obtain the texture information of the ground coal pile; Determining the contour of the ground coal pile based on geological mapping of the texture information; Determining a visual information collection scheme based on feature extraction of the contour; According to the visual information collection scheme, the collection points, configuration parameters of the collection equipment, the collection path and the emergency plan are determined.

5. The method according to claim 4, characterized in that The step of determining the contour of the ground coal pile based on geological mapping of the texture information comprises: Based on image processing operations, feature analysis is performed on the texture information to obtain the spatial arrangement and distribution pattern of the ground coal pile; Determining a scale for filling and drawing according to the spatial arrangement and the distribution law; Based on the scale, the texture information is geologically mapped to determine the outline of the ground coal pile.

6. The method according to claim 4, characterized in that The step of determining a visual information acquisition scheme based on feature extraction of the contour includes: Based on a convolutional neural network, deep learning is performed on the contour to obtain geometric features and shape features after noise reduction; A visual information collection scheme is determined according to the geometric features and the shape features.

7. The method according to any one of claims 2 to 6, characterized in that: The step of obtaining first visual data for detecting the amount of coal based on preprocessing the initial visual data includes: Based on preprocessing the initial visual data, obtaining first data to be detected; Based on the deep learning model, image semantic segmentation is performed on the first detection data to obtain first visual data for detecting the amount of coal.

8. The method according to claim 7, characterized in that The performing image semantic segmentation on the first detection data based on the deep learning model to obtain first visual data for detecting the coal quantity includes: Based on the backbone network in the deep learning model, feature extraction is performed on the first detection data to obtain backbone feature data; Based on the convolutional network in the deep learning model, performing dilated convolution on the backbone feature data to obtain dilated convolution domain data; Based on the classification network in the deep learning model, image semantic segmentation is performed on the void convolution domain data to obtain first visual data for detecting the coal quantity.

9. The method according to any one of claims 2 to 6, characterized in that: The method of obtaining second visual data for detecting the volume of coal blocks based on preprocessing the initial visual data includes: Based on preprocessing the initial visual data, obtaining second data to be detected; Based on normalizing the second data to be detected, obtaining third data to be detected; Based on the target detection algorithm model, image features are extracted from the third data to be detected to obtain second visual data for detecting the volume of coal blocks.

10. The method according to claim 9, characterized in that The method of extracting image features from the third data to be detected based on the target detection algorithm model to obtain second visual data for detecting the volume of coal blocks includes: Based on the directional gradient histogram network in the target detection algorithm model, feature extraction is performed on the third data to be detected to obtain directional gradient data; Based on the prediction network in the target detection algorithm model, the confidence judgment and non-maximum suppression are performed on the directional gradient data to obtain the second visual data for detecting the volume of the coal block.

Images