Fully-mechanized face coal flow large block detection method based on laser radar

Abstract

The invention provides a laser radar-based fully-mechanized face coal flow large block detection method and device and electronic equipment. The method comprises the following steps: acquiring point cloud data of a to-be-detected coal flow on a scraper based on a laser radar; performing coordinate system conversion on the point cloud data to obtain target point cloud data; inputting the target point cloud data into a pre-trained large coal detection model, and obtaining a three-dimensional detection frame of large coal in the to-be-detected coal flow; and determining a detection result of the large coal in the to-be-detected coal flow according to the three-dimensional detection frame of the large coal, thereby obtaining the point cloud data based on the laser radar, obtaining more real scale information, obtaining the three-dimensional detection frame of the large coal in the coal flow through the large coal detection model, and obtaining the detection result of the large coal in the to-be-detected coal flow. The detection precision and the detection efficiency of the large coal in the coal flow are improved, and based on the detection result of the large coal, the service life of the scraper conveyor is prolonged, the transportation capacity is improved, and the safety production of a fully mechanized coal mining face is ensured.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a method for detecting large pieces of coal flow in a fully mechanized mining face based on laser radar. Background Art

[0002] In the coal industry, scrapers are mostly used in the coal flow transportation link. Large pieces of coal will be generated during the transportation process of scrapers. Large pieces of coal will affect the safety of production equipment such as scrapers and belt conveyors. In related technologies, binocular ranging technology and machine vision technology are often used to detect large pieces of coal in the coal flow transported by scrapers in comprehensive mining faces. However, the above methods cannot accurately and efficiently obtain the detection results of large pieces of coal flow. How to accurately and efficiently obtain the detection results of large pieces of coal flow has become a problem that needs to be solved urgently. Summary of the invention

[0003] The purpose of this application is to solve one of the technical problems in the above technology at least to a certain extent.

[0004] The first aspect of the present application provides a method for detecting large lumps of coal flow in a fully mechanized mining face based on laser radar, comprising: obtaining point cloud data of the coal flow to be detected on a scraper based on laser radar; performing coordinate system conversion on the point cloud data to obtain target point cloud data; inputting the target point cloud data into a pre-trained large lump coal detection model to obtain a three-dimensional detection frame of the large lumps of coal in the coal flow to be detected; and determining the detection result of the large lumps of coal in the coal flow to be detected based on the three-dimensional detection frame of the large lump of coal.

[0005] The second aspect of the present application provides a laser radar-based large-lump coal flow detection device for a fully mechanized mining face, comprising: a first acquisition module, used to acquire point cloud data of the coal flow to be detected on a scraper based on the laser radar; a conversion module, used to perform coordinate system conversion on the point cloud data to acquire target point cloud data; a second acquisition module, used to input the target point cloud data into a pre-trained large-lump coal detection model to acquire a three-dimensional detection frame of the large-lump coal in the coal flow to be detected; a determination module, which determines the detection result of the large-lump coal in the coal flow to be detected based on the three-dimensional detection frame of the large-lump coal.

[0006] The third aspect embodiment of the present application provides an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute a laser radar-based method for detecting large blocks of coal flow in a comprehensive mining face provided in the first aspect of the present application.

[0007] The fourth aspect embodiment of the present application provides a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable the computer to execute a laser radar-based method for detecting large blocks of coal flow in a comprehensive mining face provided in the first aspect of the present application.

[0008] The fifth aspect embodiment of the present application provides a computer program product. When the instruction processor in the computer program product is executed, a laser radar-based method for detecting large blocks of coal flow in a comprehensive mining face provided in the first aspect of the present application is executed.

[0009] The present application provides a method for detecting large lumps of coal flow in a fully mechanized mining face based on laser radar. The method obtains point cloud data of the coal flow to be detected on a scraper based on laser radar, converts the point cloud data into a coordinate system, obtains target point cloud data, inputs the target point cloud data into a pre-trained large lump coal detection model, obtains a three-dimensional detection frame of large lumps of coal in the coal flow to be detected, and determines the detection result of large lumps of coal in the coal flow to be detected based on the three-dimensional detection frame of the large lumps of coal. Therefore, the present application obtains point cloud data based on laser radar, and can obtain more realistic scale information. The three-dimensional detection frame of large lumps of coal in the coal flow is obtained through the large lump coal detection model, thereby improving the detection accuracy and efficiency of large lumps of coal in the coal flow. The detection result based on the large lump coal is beneficial to improving the service life of the scraper, increasing the transportation capacity, and ensuring the safe production of the fully mechanized mining face.

[0010] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0012] Figure 1 A schematic diagram of a flow chart of a method for detecting large pieces of coal flow in a fully mechanized mining face based on a laser radar according to an embodiment of the present application;

[0013] Figure 2 This is a schematic diagram of the installation of a laser radar according to an embodiment of the present application;

[0014] Figure 3 A schematic diagram of a flow chart of a method for training a large coal detection model according to an embodiment of the present application;

[0015] Figure 4 This is a schematic flow chart of a method for detecting large pieces of coal flow in a fully mechanized mining face based on a laser radar according to another embodiment of the present application;

[0016] Figure 5 A schematic diagram of the structure of a columnar feature network according to an embodiment of the present application;

[0017] Figure 6 A schematic diagram of the structure of a backbone network according to an embodiment of the present application;

[0018] Figure 7 This is a structural schematic diagram of a large coal detection model according to an embodiment of the present application;

[0019] Figure 8 This is a structural schematic diagram of a laser radar-based large-block detection system for coal flow in a fully mechanized mining face according to an embodiment of the present application;

[0020] Fig. 9 This is a schematic structural diagram of a large-block coal flow detection device for a fully mechanized mining face based on a laser radar according to an embodiment of the present application;

[0021] Fig.10 It is a block diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0022] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.

[0023] The following describes, with reference to the accompanying drawings, a laser radar-based method, device, electronic equipment and medium for detecting large pieces of coal flow in a fully mechanized mining face in an embodiment of the present application.

[0024] Figure 1 FIG. 1 is a flow chart of a method for detecting large pieces of coal flow in a fully mechanized mining face based on a laser radar according to an embodiment of the present application. Figure 1 As shown, the method includes:

[0025] S101, obtaining point cloud data of the coal flow to be detected on the scraper based on the laser radar.

[0026] It should be noted that the installation parameters of the laser radar can be obtained, and the laser radar can be installed on the fully mechanized mining face according to the installation parameters to obtain the point cloud data of the coal flow to be detected on the scraper based on the laser radar.

[0027] For example, Figure 2 As shown in the figure, the installation parameters of the laser radar are an installation angle of 40°, an installation height of 4.6m, and a horizontal installation distance of 2m from the scraper. By installing the laser radar on the fully mechanized mining face, the point cloud data of the coal flow to be detected on the scraper can be obtained based on the laser radar.

[0028] In an embodiment of the present application, the point cloud data collected by the laser radar not only includes the point cloud data of the coal flow to be detected on the scraper, but also other background point cloud data. In order to reduce the impact of other background point cloud data on the detection results, and to reduce the amount of input point cloud data and improve the accuracy and timeliness of the detection results of large pieces of coal, the point cloud data collected by the laser radar can be obtained, the position information of the scraper can be obtained, and the point cloud data can be segmented according to the position information to obtain the point cloud data of the coal flow to be detected.

[0029] S102, performing coordinate system conversion on the point cloud data to obtain target point cloud data.

[0030] In an embodiment of the present application, the posture information of the laser radar can be obtained, and the rotation transformation matrix can be determined based on the posture information. According to the rotation transformation matrix, the coordinate system of the point cloud data can be transformed to obtain the target point cloud data.

[0031] For example, the formula can be used to transform the coordinate system of point cloud data to obtain the target point cloud data:

[0032] P new =R new_src P src

[0033] Among them, P new is the target point cloud data, R new_src is the rotation transformation matrix, P src Point cloud data.

[0034] S103, inputting the target point cloud data into a pre-trained large coal detection model to obtain a three-dimensional detection frame of the large coal in the coal flow to be detected.

[0035] Among them, the bulk coal detection model includes a columnar feature network, a backbone network and a detection head.

[0036] In an embodiment of the present application, after acquiring the target point cloud data, the target point cloud data can be input into a large coal detection model, and the target point cloud data can be transformed through a columnar feature network to obtain a pseudo feature map. The pseudo feature map can be convolved through a backbone network to obtain a deep feature map. The deep feature map can be detected through a detection head to obtain a three-dimensional detection frame of large coal in the coal flow to be detected.

[0037] S104, determining the detection result of the large coal in the coal flow to be detected according to the three-dimensional detection frame of the large coal.

[0038] In an embodiment of the present application, the size parameters of the three-dimensional detection frame can be obtained, wherein the size parameters include at least length, width, height and volume, and the detection results of large coal in the coal flow to be detected are determined based on the size parameters and preset large coal size parameter thresholds.

[0039] Optionally, the size parameters can be compared with corresponding large coal size parameter thresholds to obtain size parameter comparison results. In response to at least one size parameter in the size parameter comparison results being greater than or equal to the corresponding large coal size parameter threshold, the detection result is determined to be the presence of large lumps of coal in the coal flow to be detected. In response to all size parameters in the size parameter comparison results being less than the corresponding large coal size parameter threshold, the detection result is determined to be the absence of large lumps of coal in the coal flow to be detected.

[0040] It should be noted that the present application does not limit the setting of the threshold value of the parameter of the bulk coal size, and it can be set according to the actual situation.

[0041] Optionally, you can set the length parameter threshold to 80cm, the width parameter threshold to 80cm, the height parameter threshold to 50cm, and the volume parameter threshold to 1m. 3 .

[0042] For example, for a length parameter threshold of 80 cm, a width parameter threshold of 80 cm, a height parameter threshold of 50 cm, and a volume parameter threshold of 1 m 3 , if the length of the three-dimensional detection frame 1 of the large piece of coal is 90cm, the width is 75cm, the height is 45cm and the volume is 0.8m 3 In this case, the length of the three-dimensional detection frame 1 of the large lump of coal (90 cm) is greater than the length parameter threshold (80 cm), and it is determined that there are large lump of coal in the coal flow to be detected.

[0043] For example, for a length parameter threshold of 80 cm, a width parameter threshold of 80 cm, a height parameter threshold of 50 cm, and a volume parameter threshold of 1 m 3 , if the length of the three-dimensional detection frame 2 of the large piece of coal is 70cm, the width is 65cm, the height is 45cm and the volume is 0.8m 3 In this case, all size parameters of the three-dimensional detection frame 2 of the large lump of coal are smaller than the corresponding large lump of coal size parameter thresholds, and it is determined that there is no large lump of coal in the coal flow to be detected.

[0044] In an embodiment of the present application, it can be determined that the detection result is that there are large pieces of coal in the coal flow to be detected, an early warning signal and an analog control signal are generated, the early warning signal is reported to the server, and the scraper conveyor is controlled to shut down according to the analog control signal.

[0045] The present application proposes a method for detecting large lumps of coal flow in a fully mechanized mining face based on laser radar. The method obtains point cloud data of the coal flow to be detected on a scraper based on laser radar, converts the point cloud data into a coordinate system, obtains target point cloud data, and inputs the target point cloud data into a pre-trained large lump coal detection model to obtain a three-dimensional detection frame of large lumps of coal in the coal flow to be detected. According to the three-dimensional detection frame of the large lump of coal, the detection result of the large lumps of coal in the coal flow to be detected is determined. Therefore, the present application obtains point cloud data based on laser radar, and can obtain more realistic scale information. The three-dimensional detection frame of large lumps of coal in the coal flow is obtained through the large lump coal detection model, which improves the detection accuracy and efficiency of large lumps of coal in the coal flow. Based on the detection result of large lumps of coal, it is beneficial to increase the service life of the scraper, increase the transportation capacity, and ensure the safe production of the fully mechanized mining face.

[0046] The following is an explanation of the training process of the bulk coal detection model proposed in this application.

[0047] In the above embodiment, the specific process of training the large coal detection model can be combined with Figure 3 Further understanding, Figure 3 FIG. 1 is a flow chart of a training method for a large coal detection model according to an embodiment of the present application. Figure 3 As shown, the method includes:

[0048] S301, obtaining sample point cloud data of a sample coal flow on a scraper based on a laser radar, marking large pieces of coal in the sample point cloud data, and obtaining a sample marking frame of the large pieces of coal.

[0049] It should be noted that in order to improve the diversity of training data and the generalization ability of the large coal detection model, the sample annotation box can be rotated and translated to obtain new training data after rotation and translation, and random mirror flipping, global scaling, rotation, and adding simulated noise along the X-axis can be used to enhance the training data.

[0050] Among them, the sample annotation box can be used (x gt ,y gt , z gt , w gt , l gt ,h gt ,θ gt ) is described by a 7-dimensional vector, (x gt ,y gt , z gt ) is the center of the sample annotation box, (w gt , l gt ,h gt ) is the size parameter of the sample annotation box, θ gt The orientation angle of the sample label box.

[0051] S302, inputting the sample point cloud data into the initial bulk coal detection model to be trained to obtain a three-dimensional detection frame of the sample point cloud data.

[0052] Among them, the three-dimensional detection frame can be used (x a ,y a , z a , w a , l a ,h a ,θ a ) is described by a 7-dimensional vector, (x a ,y a , z a ) is the center of the 3D detection frame, (w a , l a ,h a ) is the size parameter of the 3D detection box, θ a is the orientation angle of the 3D detection box.

[0053] S303, iteratively training the initial bulk coal detection model according to the three-dimensional detection frame and the sample annotation frame to obtain a trained bulk coal detection model.

[0054] In an embodiment of the present application, the loss function of the initial large lump coal detection model can be obtained based on the three-dimensional detection box and the sample annotation box, and the model parameters of the initial large lump coal detection model can be adjusted according to the loss function until the loss function meets the training end conditions. The initial large lump coal detection model after the last adjustment of the model parameters is determined as the trained large lump coal detection model.

[0055] Optionally, the positioning loss function, classification loss function and orientation loss function can be obtained based on the three-dimensional detection box and the sample annotation box, and the positioning loss function, classification loss function and orientation loss function can be used as the total loss function of the initial large coal detection model, and the model parameters of the initial large coal detection model can be adjusted according to the total loss function.

[0056] It should be noted that for the positioning loss function L loc , if the sample annotation box (x gt ,y gt , z gt , w gt , l gt ,h gt ,θ gt ) and the 3D detection box (x a ,y a , z a , w a , l a ,h a ,θ a ), positioning loss function L loc The expression can be:

[0057]

[0058] Among them, L loc is the positioning loss function, △x is the center x coordinate offset, d a is the center coordinate x a With the center coordinate y a The distance between them, △y is the center y coordinate offset, △z is the center z coordinate offset, △w is the w width offset, △l is the l length offset, △h is the h height offset, and △θ is the direction angle θ offset.

[0059] It should be noted that for the classification loss function L cls , training through Focal Loss can effectively avoid the impact of unbalanced sample numbers, which is equivalent to giving a larger weight to samples with a small number. The classification loss function L cls The expression can be:

[0060]

[0061] Among them, L cls is the classification loss function, α a is the weight factor, γ is the hyperparameter for adjusting the weight factor, and p a is the predicted probability of the sample by the initial bulk coal detection model.

[0062] It should be noted that for the direction loss function L dir In order to avoid the error of 3D detection box direction judgment, Softmax loss is introduced to learn the direction of 3D detection box. The direction loss function L dir The expression can be:

[0063]

[0064] Among them, L dir is the directional loss function, M is the number of categories, yi is the unique hot encoding of the true category, and pi is the predicted probability of the category.

[0065] In the embodiment of the present application, after obtaining the positioning loss function L loc , classification loss function L cls And the direction loss function L dir After that, the total loss function L of the initial bulk coal detection model can be expressed as:

[0066]

[0067] Among them, L is the total loss function, Npos is the number of positive anchors, L loc is the positioning loss function, βloc is the weight factor of the positioning loss function, L cls is the classification loss function, β cls is the weight factor of the classification loss function, L dir is the direction loss function, β dir is the weight factor of the direction loss function.

[0068] Optionally, a weight factor β for the localization loss function can be set loc =2, weight factor β of classification loss function cls =1, weight factor β of directional loss function dir = 0.2, in order to optimize the loss function, the initial learning rate can be 2×10 -4 The Adam optimizer was used with a learning rate decay of 0.8 every 15 epochs and trained for 160 epochs, using a batch size of 2 for the validation set and a batch size of 4 for the test set submission.

[0069] It should be noted that the present application does not limit the setting of the training end condition, and it can be selected according to the actual situation.

[0070] Optionally, the training end condition can be set as the loss function value being less than a preset loss threshold.

[0071] Optionally, the training end condition may be set as the number of times the parameters of the initial bulk coal detection model are adjusted reaches a preset threshold number of times.

[0072] Optionally, the large coal detection model can be migrated and deployed to the RK1126 edge, making full use of the capabilities of the neural processing unit on the edge to realize applications in related scenarios.

[0073] The present application proposes a method for detecting large lumps of coal flow in a comprehensive mining face based on laser radar. The method obtains sample point cloud data of a sample coal flow on a scraper based on laser radar, labels large lumps of coal in the sample point cloud data, obtains a sample labeling frame of the large lumps of coal, and inputs the sample point cloud data into an initial large lump coal detection model to be trained to obtain a three-dimensional detection frame of the sample point cloud data. The initial large lump coal detection model is iteratively trained based on the three-dimensional detection frame and the sample labeling frame to obtain a trained large lump coal detection model. Therefore, the present application discloses a method for detecting large lumps of coal more accurately and efficiently by obtaining a trained large lump coal detection model, which lays a foundation for subsequently accurately obtaining the detection results of large lumps of coal.

[0074] In the above embodiment, the specific process of inputting the target point cloud data into the pre-trained large coal detection model to obtain the three-dimensional detection frame of the large coal in the coal flow to be detected can be combined with Figure 4 Further understanding, Figure 4This is a flow chart of a method for detecting large pieces of coal flow in a fully mechanized mining face based on a laser radar according to another embodiment of the present application. Figure 4 As shown, the method includes:

[0075] S401, inputting the target point cloud data into the bulk coal detection model, which includes a columnar feature network, a backbone network and a detection head.

[0076] Among them, the large coal detection model is a single-stage and end-to-end point cloud target detection model network (PointPillars). The accuracy of the point cloud target detection model network has been improved, which greatly improves the detection speed, making it possible to deploy and apply the point cloud target detection network in the industry.

[0077] S402, transforming the target point cloud data through a columnar feature network to obtain a pseudo feature map.

[0078] For example, Figure 5 As shown, (1) Meshing: Map the target point cloud data (Point cloud) into a top-down view Figure X On the Y plane, it is divided into specified H×W grids, and the grid is promoted to a pillar along the Z axis. The resolution of the XY grid can be set to 0.16m, the target point cloud data; (2) Dimension expansion: The target point cloud data contains 4-dimensional data (x, y, z, r), where x, y and z are 3-axis data and r is the reflectivity. The 4-dimensional data is expanded to 9-dimensional data (x, y, z, r, x_c, y_c, z_c, x_p, y_p), where x_c, y_c, z_c are the centroid deviations of the point relative to all points in the pillar, and x_p, y_p are the physical center deviations of the point relative to all points in the pillar; (3) Downsampling and padding: For the number of points in each pillar, downsampling is performed when there are more than N points, and zero value padding is performed when there are less than N points, forming a (D, P, N)-dimensional tensor, obtaining stacked pillars, and generating pillar indexes (Pillar index), where D is the expanded data dimension, N is the number of sampling points for each pillar, and P is the total number of pillars. D = 9, N = 100, and P = 12000 can be set; (4) Feature extraction: Use the PointNet network to extract pillar features, and the dimension becomes a (C, P, N) tensor (Learned Features); (5) Pseudo feature map generation: Perform a maximum pooling operation on the number of sampling points N for each pillar to obtain a (C, P) tensor, which is transformed into a (C, H, W) dimensional tensor, that is, a pseudo feature map (Pseudo image) is obtained.

[0079] S403, convolving the pseudo feature map through the backbone network to obtain a deep feature map.

[0080] For example, Figure 6 As shown in the figure, the backbone network consists of two sub-networks: a top-down network extracts features of three scales of pseudo images (1 / 2, 1 / 4, 1 / 8), and the other network performs bottom-up upsampling and decodes feature maps of different scales step by step to obtain feature maps with consistent dimensions, and finally splices the above three feature maps of different scales at the channel scale. A tensor with dimensions of (H / 2, W / 2, 6C) is generated, that is, a deep feature map is obtained.

[0081] S404, using a detection head, detect the deep feature map to obtain a three-dimensional detection frame of large pieces of coal in the coal flow to be detected.

[0082] In an embodiment of the present application, feature extraction and target detection can be performed directly from a bird's eye view (Bird's Eye View, BEV for short), while ignoring the height information in the target point cloud data. The deep feature map is detected by the detection head to obtain a three-dimensional detection frame of large pieces of coal in the coal flow to be detected.

[0083] For example, Figure 7 As shown, the model structure of the large coal detection model includes a columnar feature network (PillarFeature Net), a backbone network (Backbone) namely a two-dimensional convolutional neural network (2D-CNN) and a detection head namely a single shot multibox detector (SSD). The target point cloud data is converted by the columnar feature network to obtain a pseudo feature map, the pseudo feature map is convolved by the backbone network to obtain a deep feature map, and the deep feature map is detected by the detection head to obtain a three-dimensional detection frame of the large coal in the coal flow to be detected.

[0084] The present application proposes a method for detecting large blocks of coal flow in a fully mechanized mining face based on laser radar. The method inputs target point cloud data into a large block coal detection model. The large block coal detection model includes a columnar feature network, a backbone network and a detection head. The target point cloud data is converted into a pseudo feature map through the columnar feature network. The target point cloud data is converted into a pseudo feature map through the columnar feature network. The deep feature map is detected through the detection head to obtain a three-dimensional detection frame of large blocks of coal in the coal flow to be detected. Therefore, the present application can more accurately and efficiently obtain a three-dimensional detection frame of large blocks of coal through the large block coal detection model, thereby improving the detection accuracy and efficiency of large blocks of coal in the coal flow.

[0085] It should be noted that a laser radar-based large-block coal flow detection method for a fully-mechanized mining face proposed in an embodiment of the present application can construct a laser radar-based large-block coal flow detection system for a fully-mechanized mining face.

[0086] For example, a laser radar-based coal flow block detection system for fully mechanized mining faces is Figure 8 As shown, the coal flow point cloud data can be collected by laser radar, and the coal flow point cloud data can be transmitted to the back-end processing unit through the network cable. The back-end processing unit takes RK1126CPU as the core, and obtains the three-dimensional detection frame of the large coal in the coal flow to determine the detection result of the large coal in the coal flow. The detection result is determined to be the presence of large coal in the coal flow, and an early warning signal and an analog control signal are generated. The early warning signal is reported to the ground server through the underground ring network to remind relevant personnel to handle it in real time. At the same time, the alarm signal can be output through the input / output panel of the display, the sound and light alarm lights can be controlled to give on-site alarms, and the analog control signal can be transmitted to the control unit through the target protocol to control the scraper to perform shutdown linkage processing. Among them, the laser radar, the back-end processing unit, and the control unit are generally powered by 12v or 24v power supply.

[0087] Optionally, the historical scraper speed, large coal block data in the coal flow, etc. can be stored in the server, and reports can be set up to accumulate for the current shift, month, and year, and the parameters and control strategies can be continuously optimized and adjusted through data analysis.

[0088] In summary, the present application proposes a method for detecting large pieces of coal flow in a fully mechanized mining face based on a laser radar. By installing a laser radar, the laser radar has a more flexible installation angle and a wider recognition range, which solves the problems of high installation position requirements for two-dimensional image perception and visual depth perception (difficult to install at close range, easy to affect the safety of other equipment), and obstacles to two-dimensional image space perception. The laser radar can still collect complete scraper point cloud data under conditions such as limited installation position in the underground fully mechanized mining face working scene. Compared with binocular ranging sensors, it is more suitable for the scene of large-piece coal flow detection in the fully mechanized mining face. Compared with traditional cameras, it can obtain real scale information and has better anti-interference and higher robustness. By introducing a large piece of coal detection model, the corresponding anchor frame size can be adjusted according to actual needs to improve the overall detection accuracy. The detection accuracy of large pieces of coal flow in the fully mechanized mining face based on the laser radar can reach the centimeter level, which can effectively reduce the influence of factors such as uneven installation position and shape change on the detection results. The accuracy, sensitivity and adaptability of large-piece coal flow detection are higher, and RK1126 is fully utilized. The NPU inference module has a faster detection speed and can achieve real-time large-lump coal target detection under the premise of losing a small amount of accuracy in the model quantization deployment inference. It is not affected by the electromagnetic environment, has embedded algorithm transplantation, supports edge computing, and can greatly reduce the computing pressure of the central server. It can still maintain normal operation when problems occur in the underground ring network. It has the advantages of high stability and high fault tolerance.

[0089] Fig. 9 This is a schematic diagram of the structure of a large block detection device for coal flow in a fully mechanized mining face based on a laser radar according to an embodiment of the present application. Fig. 9 As shown, a laser radar-based coal flow block detection device 900 for fully mechanized mining face includes a first acquisition module 91, a conversion module 92, a second acquisition module 93 and a determination module 94, wherein:

[0090] A first acquisition module 91 is used to acquire point cloud data of the coal flow to be detected on the scraper based on the laser radar;

[0091] A conversion module 92, used to perform coordinate system conversion on the point cloud data to obtain target point cloud data;

[0092] A second acquisition module 93 is used to input the target point cloud data into a pre-trained large coal detection model to obtain a three-dimensional detection frame of large coal in the coal flow to be detected;

[0093] The determination module 94 determines the detection result of the large coal in the coal flow to be detected according to the three-dimensional detection frame of the large coal.

[0094] The second aspect of the present application provides a laser radar-based large-chunk coal flow detection device for a fully mechanized mining face, which also has the following technical features.

[0095] According to one embodiment of the present application, device 900 is used to: obtain sample point cloud data of a sample coal flow on a scraper based on a laser radar, label large lumps of coal in the sample point cloud data, and obtain a sample labeling box of the large lumps of coal; input the sample point cloud data into an initial large lump coal detection model to be trained to obtain a three-dimensional detection box of the sample point cloud data; iteratively train the initial large lump coal detection model based on the three-dimensional detection box and the sample labeling box to obtain a trained large lump coal detection model.

[0096] According to one embodiment of the present application, the second acquisition module 93 is used to: input the target point cloud data into the large coal detection model, the large coal detection model includes a columnar feature network, a backbone network and a detection head; transform the target point cloud data through the columnar feature network to obtain a pseudo feature map; convolve the pseudo feature map through the backbone network to obtain a deep feature map; detect the deep feature map through the detection head to obtain a three-dimensional detection frame of the large coal in the coal flow to be detected.

[0097] According to one embodiment of the present application, a determination module 94 is used to: obtain the size parameters of the three-dimensional detection frame, wherein the size parameters include at least length, width, height and volume; and determine the detection result of the large coal in the coal flow to be detected based on the size parameters and a preset large coal size parameter threshold.

[0098] According to one embodiment of the present application, a determination module 94 is used to: compare the size parameter with a corresponding large coal size parameter threshold to obtain a size parameter comparison result; in response to at least one size parameter in the size parameter comparison result being greater than or equal to the corresponding large coal size parameter threshold, determine that the detection result is that large coal exists in the coal flow to be detected; in response to all size parameters in the size parameter comparison result being less than the corresponding large coal size parameter threshold, determine that the detection result is that no large coal exists in the coal flow to be detected.

[0099] According to one embodiment of the present application, device 900 is used to: determine that the detection result is that large pieces of coal exist in the coal flow to be detected, generate an early warning signal and an analog control signal; report the early warning signal to the server, and control the scraper to stop according to the analog control signal.

[0100] According to one embodiment of the present application, the first acquisition module 91 is used to: acquire point cloud data collected by the laser radar; acquire position information of the scraper, and segment the point cloud data according to the position information to acquire point cloud data of the coal flow to be detected.

[0101] According to one embodiment of the present application, the conversion module 92 is used to: obtain the posture information of the laser radar; determine the rotation transformation matrix based on the posture information; and perform coordinate system conversion on the point cloud data according to the rotation transformation matrix to obtain the target point cloud data.

[0102] The present application proposes a device for detecting large lumps of coal flow in a fully mechanized mining face based on laser radar. The device obtains point cloud data of the coal flow to be detected on a scraper based on laser radar, performs coordinate system conversion on the point cloud data, obtains target point cloud data, inputs the target point cloud data into a pre-trained large lump coal detection model, obtains a three-dimensional detection frame of large lumps of coal in the coal flow to be detected, and determines the detection result of large lumps of coal in the coal flow to be detected based on the three-dimensional detection frame of the large lumps of coal. Therefore, the present application obtains point cloud data based on laser radar, and can obtain more realistic scale information. The three-dimensional detection frame of large lumps of coal in the coal flow is obtained through the large lump coal detection model, thereby improving the detection accuracy and efficiency of large lumps of coal in the coal flow. The detection result based on the large lump coal is beneficial to improving the service life of the scraper, increasing the transportation capacity, and ensuring the safe production of the fully mechanized mining face.

[0103] To achieve the above embodiments, the present application also provides an electronic device, a computer-readable storage medium and a computer program product.

[0104] Fig.10 A block diagram of an electronic device according to an embodiment of the present application, such as Fig.10 As shown, the device 1000 includes a memory 101, a processor 102, and a computer program stored in the memory 101 and executable on the processor 102. When the processor 102 executes the program instructions, the execution is realized. Figures 1 to 8 The embodiment of the invention provides a method for detecting large pieces of coal flow in a fully mechanized mining face based on laser radar.

[0105] In order to implement the above embodiments, the present application also provides a non-transitory computer-readable storage medium storing computer instructions, the computer instructions are used to enable the computer to execute Figures 1 to 8 A method for detecting large pieces of coal flow in a fully mechanized mining face based on laser radar is provided in an embodiment of the present invention.

[0106] In order to implement the above embodiment, the present application also provides a computer program product. When the instruction processor in the computer program product executes, Figures 1 to 8 A method for detecting large pieces of coal flow in a fully mechanized mining face based on laser radar is provided in an embodiment of the present invention.

[0107] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0108] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of this application, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0109] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or more executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present application belong.

[0110] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute the instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purpose of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or processing in other suitable ways if necessary, and then stored in a computer memory.

[0111] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0112] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0113] In addition, each functional unit in each embodiment of the present application may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0114] The storage medium mentioned above may be a read-only memory, a magnetic disk or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limiting the present application. A person of ordinary skill in the art may change, modify, replace and modify the above embodiments within the scope of the present application.

Claims

1. A method for detecting large pieces of coal flow in a fully mechanized mining face based on laser radar, characterized in that: The method comprises: Obtain point cloud data of the coal flow to be detected on the scraper based on LiDAR; Performing coordinate system conversion on the point cloud data to obtain target point cloud data; Inputting the target point cloud data into a pre-trained large coal detection model to obtain a three-dimensional detection frame of large coal in the coal flow to be detected; The detection result of the large lumps of coal in the coal flow to be detected is determined according to the three-dimensional detection frame of the large lumps of coal.

2. The method according to claim 1, characterized in that The training process of the bulk coal detection model includes: Acquire sample point cloud data of a sample coal flow on a scraper based on a laser radar, mark large pieces of coal in the sample point cloud data, and obtain a sample marking frame of the large pieces of coal; Inputting the sample point cloud data into an initial bulk coal detection model to be trained to obtain a three-dimensional detection frame of the sample point cloud data; The initial bulk coal detection model is iteratively trained according to the three-dimensional detection frame and the sample annotation frame to obtain a trained bulk coal detection model.

3. The method according to any one of claims 1 to 2, characterized in that: The step of inputting the target point cloud data into a pre-trained large coal detection model to obtain a three-dimensional detection frame of large coal in the coal flow to be detected includes: Inputting the target point cloud data into the bulk coal detection model, wherein the bulk coal detection model includes a columnar feature network, a backbone network and a detection head; The target point cloud data is transformed by the columnar feature network to obtain a pseudo feature map; Convolving the pseudo feature map through the backbone network to obtain a deep feature map; The deep characteristic map is detected by the detection head to obtain a three-dimensional detection frame of large pieces of coal in the coal flow to be detected.

4. The method according to claim 1, characterized in that: The step of determining the detection result of the large coal in the coal flow to be detected according to the three-dimensional detection frame of the large coal includes: Acquire size parameters of the three-dimensional detection frame, wherein the size parameters include at least length, width, height and volume; The detection result of the large coal in the coal flow to be detected is determined according to the size parameter and a preset large coal size parameter threshold.

5. The method according to claim 4, characterized in that The step of determining the detection result of the large coal in the coal flow to be detected according to the size parameter and a preset large coal size parameter threshold comprises: Comparing the size parameter with a corresponding large coal size parameter threshold to obtain a size parameter comparison result; In response to at least one size parameter in the size parameter comparison result being greater than or equal to the corresponding large coal size parameter threshold, determining that the detection result is that large coal exists in the coal flow to be detected; In response to all the size parameters in the size parameter comparison result being smaller than the corresponding large coal size parameter threshold, it is determined that the detection result is that no large coal exists in the coal flow to be detected.

6. The method according to claim 5, characterized in that The method further comprises: Determining that the detection result is that there are large pieces of coal in the coal flow to be detected, and generating an early warning signal and a simulation control signal; The early warning signal is reported to the server, and the scraper conveyor is controlled to stop according to the simulated control signal.

7. The method according to claim 1, characterized in that The method of obtaining point cloud data of the coal flow to be detected on the scraper based on the laser radar includes: Acquire point cloud data collected by the laser radar; The position information of the scraper is obtained, and the point cloud data is segmented according to the position information to obtain the point cloud data of the coal flow to be detected.

8. The method according to claim 1, characterized in that The step of performing coordinate system conversion on the point cloud data to obtain target point cloud data includes: Obtaining the position information of the laser radar; Determine a rotation transformation matrix according to the posture information; The point cloud data is subjected to a coordinate system transformation according to the rotation transformation matrix to obtain the target point cloud data.

9. A laser radar-based coal flow block detection device for fully mechanized mining face, characterized in that: The device comprises: The first acquisition module is used to acquire point cloud data of the coal flow to be detected on the scraper based on the laser radar; A conversion module, used to perform coordinate system conversion on the point cloud data to obtain target point cloud data; A second acquisition module is used to input the target point cloud data into a pre-trained large coal detection model to obtain a three-dimensional detection frame of large coal in the coal flow to be detected; A determination module determines the detection result of the bulk coal in the coal flow to be detected according to the three-dimensional detection frame of the bulk coal.

10. An electronic device, characterized in that: including a processor and a memory; The processor runs a program corresponding to the executable program code by reading the executable program code stored in the memory, so as to implement the method according to any one of claims 1 to 8.

Images