Large block coal target detection method and device

Abstract

The present application provides a kind of large lump coal target detection method and device, it is related to computer vision technical field, method includes: based on the data set obtained after image enhancement and YOLO-OBB data format marking are carried out to scraper conveyor detection area image and coal lump image, target detection model is trained, generates the obb target detection model of the region detection frame corresponding to the region detection frame of scraper conveyor detection area, coal lump corresponding coal lump detection frame;With the real-time detection area coal flow video of scraper conveyor, the area ratio between the obb region detection frame and the obb coal lump detection frame corresponding to each frame of image is obtained, whether coal lump belongs to large lump coal is judged, and based on the number and displacement of optical flow points in the obb coal lump detection frame of each frame of image, whether coal lump belongs to immobile abnormal coal lump is judged.Therefore, based on obb target detection model, the extraction identification of the obb detection frame of scraper conveyor and coal lump target in video stream is improved, and then the accurate detection classification identification of coal lump is realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, and in particular to a large lump coal target detection method and device, electronic equipment and storage medium. BACKGROUND

[0002] However, in the process of coal mining, large lump coal is prone to be produced, and the crusher cannot break it in time. With the accumulation of time, it may cause the scraper conveyor to be blocked, and in severe cases, the scraper conveyor may be damaged. Therefore, it is of great significance to realize intelligent detection of large lump coal targets in the production process of the scraper conveyor for coal mine production safety and labor reduction and efficiency improvement.

[0003] In related technologies, coal mine large lump coal target detection methods can be roughly divided into three categories: first, relying on manual observation, which requires workers to observe in the observation area for a long time, and the work nature is single, which is easy to cause visual fatigue and even safety accidents; second, using sensors for monitoring, which requires a good sensor monitoring environment and high-precision stable hardware, but due to the poor transportation status of the fully mechanized working face, it is easy to be damaged, low in efficiency and poor in accuracy; third, using computer vision and deep learning technology, but it cannot accurately identify the main detection area of the scraper conveyor and the coal lump area, and cannot judge whether there is an abnormal coal lump. SUMMARY

[0004] The present application aims to at least solve one of the technical problems in the related art to some extent.

[0005] To this end, the first object of the present application is to provide a large lump coal target detection method based on an obb target detection model to improve the extraction and identification of obb detection boxes of the scraper conveyor and coal lump targets in the video stream, and to realize accurate detection and classification identification of coal lumps.

[0006] The second object of the present application is to provide a large lump coal target detection device.

[0007] The third object of the present application is to provide an electronic device.

[0008] The fourth object of the present application is to provide a non-transitory computer readable storage medium storing computer instructions.

[0009] To achieve the above objects, the first aspect of the present application provides a large lump coal target detection method, which comprises:

[0010] Making a data set, wherein the data set is obtained by labeling the enhanced images in YOLO-OBB data format after image enhancement of the scraper conveyor detection area images and coal lump images;

[0011] The data set is subjected to target detection model training to generate an obb target detection model for identifying a region detection frame corresponding to a detection region of the scraper conveyor and a coal block detection frame corresponding to a coal block.

[0012] Based on the obb target detection model, real-time detection region coal flow video of the scraper conveyor is deduced to obtain an obb region detection frame of the real-time detection region of the scraper conveyor and an obb coal block detection frame of the coal block.

[0013] Based on an area ratio between the obb coal block detection frame and the obb region detection frame corresponding to each frame of image in the real-time detection region coal flow video, it is determined whether the coal block belongs to a large coal block greater than a preset size, and based on a number and displacement of optical flow points in the obb coal block detection frame corresponding to each frame of image, it is determined whether the coal block belongs to an abnormal coal block that is not moving.

[0014] To achieve the above purpose, a second aspect embodiment of the present application provides a large coal block target detection device, which comprises:

[0015] A manufacturing module is configured to manufacture a data set, wherein the data set is obtained by performing image enhancement on images of a detection region of the scraper conveyor and images of the coal block, and then performing YOLO-OBB data format labeling on the enhanced images.

[0016] A training module is configured to train the data set by using a target detection model to generate an obb target detection model for identifying a region detection frame corresponding to a detection region of the scraper conveyor and a coal block detection frame corresponding to a coal block.

[0017] A deduction module is configured to deduce real-time detection region coal flow video of the scraper conveyor based on the obb target detection model to obtain an obb region detection frame of the real-time detection region of the scraper conveyor and an obb coal block detection frame of the coal block.

[0018] A judgment module is configured to determine, based on an area ratio between the obb coal block detection frame and the obb region detection frame corresponding to each frame of image in the real-time detection region coal flow video, whether the coal block belongs to a large coal block greater than a preset size, and determine, based on a number and displacement of optical flow points in the obb coal block detection frame corresponding to each frame of image, whether the coal block belongs to an abnormal coal block that is not moving.

[0019] To achieve the above purpose, a third aspect embodiment of the present application provides an electronic device, which comprises at least one processor and a memory connected in communication with the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the method of the first aspect.

[0020] To achieve the above object, the fourth aspect of the present application provides a non-transitory computer readable storage medium storing computer instructions, the computer instructions being used to make the computer execute the method of the first aspect.

[0021] The bulk coal target detection method and device, the electronic device and the storage medium provided by the embodiments of the present application train a target detection model based on a data set obtained by performing image enhancement on a scraper conveyor detection region image and a coal block image and performing YOLO-OBB data format labeling, and generate an obb target detection model for recognizing a detection frame corresponding to a region of the scraper conveyor detection region and a detection frame corresponding to a coal block; the area ratio between the obb region detection frame and the obb coal block detection frame corresponding to each image is obtained by deducing a real-time detection region coal flow video of the scraper conveyor, and it is determined whether the coal block belongs to bulk coal, and it is determined whether the coal block belongs to an abnormal coal block that does not move based on the number and displacement of the optical flow points in the obb coal block detection frame of each image. Therefore, based on the obb target detection model, the extraction and recognition of the obb detection frame of the scraper conveyor and the coal block target in the video stream are improved, and accurate detection, classification and recognition of the coal block are realized.

[0022] Additional aspects and advantages of the present application will be made apparent by the following description and the accompanying drawings. BRIEF DESCRIPTION OF DRAWINGS

[0023] The above and / or additional aspects and advantages of the present application will become apparent and be more readily understood through consideration of the following description, taken in conjunction with the accompanying drawings, in which:

[0024] Figure 1 A flowchart of a bulk coal target detection method provided by an embodiment of the present application;

[0025] Figure 2 A scraper conveyor coal flow video model deduction flowchart provided by an embodiment of the present application;

[0026] Figure 3 An execution flowchart of a bulk coal target detection method provided by an embodiment of the present application;

[0027] Figure 4 A structural schematic diagram of a bulk coal target detection device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0028] Embodiments of the present application are described in detail below, examples of which are shown in the accompanying drawings, in which the same or similar reference signs represent the same or similar elements or elements having the same or similar functions throughout. The embodiments described below by reference to the accompanying drawings are exemplary and are intended to explain the present application, and cannot be understood as limiting the present application.

[0029] It should be noted that the acquisition, storage, use, processing, etc. of data in the technical solutions of the present application comply with relevant provisions of relevant laws and regulations.

[0030] The bulk coal target detection method and device are described below with reference to the accompanying drawings.

[0031] Figure 1 A flowchart of a bulk coal target detection method provided by an embodiment of the present application is shown.

[0032] As shown in Figure 1 , the method comprises the following steps:

[0033] Step 101, making a data set, wherein the data set is obtained by performing image enhancement on the scraper conveyor detection area image and the coal block image, and then performing YOLO-OBB data format labeling on the enhanced image.

[0034] In some embodiments, one implementation of making a data set can be that the scraper conveyor detection area image and the coal block image are acquired by a coal mining face monitoring device; the scraper conveyor detection area image and the coal block image are subjected to enhancement processing to obtain enhanced images, the enhancement processing including Gaussian filter processing, adding Gaussian noise processing, horizontal flipping processing, and 30° and 45° clockwise or counterclockwise rotation processing; and the enhanced images are subjected to YOLO-OBB data format labeling to obtain the data set of the scraper conveyor detection area image and the coal block image subjected to YOLO-OBB data format labeling. In this way, the data set is enriched, and the accuracy of training the target detection model by the data set is improved.

[0035] Further, the Gaussian filter processing of the scraper conveyor detection area image and the coal block image can enhance the processing capability of the target detection model for target motion blur; the addition of Gaussian noise to the scraper conveyor detection area image and the coal block image can enhance the processing capability of the target detection model for coal dust and water mist; the horizontal flipping of the scraper conveyor detection area image and the coal block image can enable the target detection model to learn different directional features of objects; and the 30° and 45° clockwise or counterclockwise rotation of the scraper conveyor detection area image and the coal block image can enable the target detection model to learn features of objects at different angles, simulate the position change of the camera, and enhance the rotation invariance of the target detection model.

[0036] Step 102, training the data set by a target detection model to generate an obb target detection model for recognizing the detection frame of the corresponding region of the scraper conveyor detection area and the detection frame of the corresponding coal block of the coal block.

[0037] In some embodiments, the target detection model can be yolov11, but is not limited thereto.

[0038] In step 103, the real-time detection area coal flow video of the scraper conveyor is deduced based on the obb target detection model to obtain the obb area detection frame of the real-time detection area of the scraper conveyor and the obb coal block detection frame of the coal block.

[0039] In some embodiments, one implementation of deducing the obb area detection frame of the real-time detection area of the scraper conveyor and the obb coal block detection frame of the coal block based on the obb target detection model to the real-time detection area coal flow video of the scraper conveyor can be that the video frame extraction is performed on the real-time detection area coal flow video of the scraper conveyor, denoted as image I, and the bicubic interpolation method is used to perform upsampling and downsampling on the image I respectively to obtain image I' and image I'', wherein the high and wide dimensions of the image I' are 0.5 times of the high and wide dimensions of the image I, and the high and wide dimensions of the image I'' are 2 times of the high and wide dimensions of the image I; the images I, I' and I'' with different high and wide dimensions are input into the trained obb target detection model respectively for deduction to obtain the respective deduction results of I, I' and I'', and the deduction results include the initial obb area detection frame, the four-point coordinate values of the initial obb area detection frame, the confidence of the initial obb area detection frame, the initial obb coal block detection frame, the four-point coordinate values of the initial obb coal block detection frame, and the confidence of the initial obb coal block detection frame; the four-point coordinate values of the initial obb area detection frame and the initial obb coal block detection frame in the deduction result of the image I' are multiplied by 2 uniformly, and the four-point coordinate values of the initial obb area detection frame and the initial obb coal block detection frame in the deduction result of the image I'' are divided by 2 uniformly to obtain the images I, I' and I'' with uniform coordinate scales; the images I, I' and I'' with uniform coordinate scales are input into the obb target detection model to obtain the standby obb area detection frame of the detection area of the scraper conveyor and the standby obb coal block detection frame of the coal block in the target deduction result, and the non-maximum suppression algorithm is used to remove the detection frames with high overlap degree and low confidence of the standby obb area detection frame and the standby obb coal block detection frame in the target deduction result to obtain the obb area detection frame of the real-time detection area of the scraper conveyor and the obb coal block detection frame of the coal block. Thus, the obb area detection frame and the standby obb coal block detection frame are rectangles with a certain rotation angle, which can more accurately describe the main detection area of the scraper conveyor and the direction and size of the coal block.

[0040] The deduction result can be a group of data including (detection frame label name, four-point coordinate values of the detection frame, and detection frame confidence).

[0041] Optionally, for better understanding of the present application, the present application further provides a flow chart of the model deduction of the coal flow video of the scraper conveyor, as shown in Figure 2As shown, specifically, taking the target detection model yolov11 as an example, the real-time detection area coal flow video of the scraper conveyor is subjected to video frame extraction, recorded as an image I, the image I is subjected to image up-sampling and image down-sampling by using a bicubic interpolation method to obtain an image I' and an image I'', the images I, I' and I'' of different high-width sizes are respectively input into the yolov11 for derivation, the derivation results (initial obb area detection frame and initial obb coal block detection frame) of the images I, I' and I'' are subjected to unified coordinate scale (normalization) to obtain the backup obb area detection frame of the scraper conveyor detection area and the backup obb coal block detection frame of the coal block in the target derivation results of the images I, I' and I'' input into the yolov11, and a non-maximum suppression algorithm is used to remove the detection frames with high overlap degree and low backup obb area detection frame confidence and backup obb coal block detection frame confidence in the target derivation results, so as to obtain the obb area detection frame of the real-time detection area of the scraper conveyor and the obb coal block detection frame of the coal block.

[0042] In step 104, based on the area ratio between the obb coal block detection frame and the obb area detection frame corresponding to each image in the real-time detection area coal flow video, it is judged whether the coal block belongs to large lump coal larger than a preset size, and based on the number and displacement of the optical flow points in the obb coal block detection frame corresponding to each image, it is judged whether the coal block belongs to abnormal coal block that is not moving.

[0043] In some embodiments, based on the area ratio between the obb coal block detection frame corresponding to each frame of image in the real-time detection area coal flow video and the obb area detection frame, it is judged whether the coal block belongs to large block coal larger than a preset size, and based on the number and displacement of the optical flow points in the obb coal block detection frame corresponding to each frame of image, it is judged whether the coal block belongs to abnormal coal block that is not moving. One implementation can be that the area ratio between the obb coal block detection frame corresponding to each frame of image in the real-time detection area coal flow video and the obb area detection frame is obtained, and in the case that the area ratio is greater than a set threshold value, the coal block is large block coal equal to or larger than a preset size; in the case that the area ratio is less than the set threshold value, adjacent image frames of the real-time detection area coal flow video are extracted, dense optical flow extraction is performed on the adjacent image frames to obtain initial optical flow points of each frame of image; the number and displacement of the initial optical flow points in the obb coal block detection frame corresponding to each frame of image are counted according to the initial optical flow points of each frame of image; abnormal optical flow points with a displacement less than a set displacement threshold value are removed from the initial optical flow points in the obb coal block detection frame to obtain available optical flow points in the obb coal block detection frame; in the case that the number of optical flow points is greater than or equal to a set number threshold value, the coal block corresponding to the obb coal block detection frame is normal coal block; in the case that the number of optical flow points is less than the set number threshold value and the scraper conveyor equipment is not running, the coal block is normal coal block; in the case that the number of available optical flow points is less than the set number threshold value and the scraper conveyor equipment is in a running state, the coal block belongs to abnormal coal block that is not moving. Thus, these abnormal coal blocks and large block coal in the coal flow transportation are detected and motion estimated, the abnormal coal block that is not moving is identified, an alarm is prompted for manual processing, and in serious cases, the machine is stopped for maintenance to prevent greater harm.

[0044] In the case that the coal block is determined to be abnormal coal block that is not moving, an alarm needs to be prompted for manual inspection and processing to ensure the safe operation of the scraper conveyor equipment.

[0045] The large block coal target detection method of the embodiment of the application is based on the data set obtained by performing image enhancement on the images of the scraper conveyor detection area and the coal block images and performing YOLO-OBB data format labeling, a target detection model is trained, and an obb target detection model for identifying the region detection frame corresponding to the detection area of the scraper conveyor and the coal block detection frame corresponding to the coal block is generated; the area ratio between the obb region detection frame and the obb coal block detection frame corresponding to each frame of image is obtained by deducing the real-time detection area coal flow video of the scraper conveyor, it is judged whether the coal block belongs to large block coal, and based on the number and displacement of the optical flow points in the obb coal block detection frame of each frame of image, it is judged whether the coal block belongs to abnormal coal block that is not moving. Thus, based on the obb target detection model, the extraction and identification of the obb detection frame of the scraper conveyor and the coal block target in the video stream are improved, and accurate detection, classification and identification of the coal block are realized.

[0046] In addition, in some embodiments, the present application also provides an execution flowchart of a large lump coal target detection method, as shown in Figure 3 obtains a real-time detection area coal flow video, extracts adjacent image frames of the real-time detection area coal flow video; each frame of image is input into an obb target detection model for derivation to obtain a derivation result (obb lump coal detection box and obb area detection box), and a dense optical flow of adjacent image frames is extracted to obtain initial optical flow points of each frame of image; in the case that the obb lump coal detection box does not exist in the obb area detection box, the lump coal is not in the detection area; in the case that the obb lump coal detection box exists in the obb area detection box, an area ratio between the obb lump coal detection box and the obb area detection box is calculated, in the case that the area ratio is greater than a set threshold value, then the lump coal is a large lump coal greater than or equal to a preset size; in the case that the area ratio is less than the set threshold value, the number and displacement of the initial optical flow points in the obb lump coal detection box corresponding to each frame of image are counted; abnormal optical flow points with a displacement less than a set displacement threshold value are removed from the initial optical flow points in the obb lump coal detection box to obtain available optical flow points in the obb lump coal detection box; in the case that the number of the optical flow points is greater than or equal to a set number threshold value, then the lump coal corresponding to the obb lump coal detection box is a normal lump coal; in the case that the number of the optical flow points is less than the set number threshold value and the scraper conveyor equipment is not running, then the lump coal is a normal lump coal; in the case that the number of the available optical flow points is less than the set number threshold value and the scraper conveyor equipment is in a running state, then the lump coal belongs to an abnormal lump coal that does not move, an alarm is given for manual inspection and processing, and coal mine production safety is realized.

[0047] In order to realize the above-mentioned embodiments, the present application further provides a large lump coal target detection device.

[0048] Figure 4 A structure schematic diagram of a large lump coal target detection device provided for the embodiments of the present application.

[0049] As shown in Figure 4 The large lump coal target detection device 40 comprises a making module 41, a training module 42, a derivation module 43 and a judgment module 44.

[0050] The making module 41 is used for making a data set, wherein the data set is obtained by performing image enhancement on a scraper conveyor detection area image and a lump coal image, and then performing YOLO-OBB data format labeling on the enhanced image;

[0051] The training module 42 is used for training a target detection model by using the data set to generate an obb target detection model for identifying a scraper conveyor detection area corresponding area detection box and a lump coal corresponding lump coal detection box;

[0052] The derivation module 43 is configured to derive the real-time detection area coal flow video of the scraper conveyor based on the obb target detection model, to obtain an obb area detection frame of the real-time detection area of the scraper conveyor and an obb coal block detection frame of the coal block.

[0053] The judgment module 44 is configured to judge whether the coal block is a large coal block greater than a preset size based on an area ratio between the obb coal block detection frame and the obb area detection frame corresponding to each frame of image in the real-time detection area coal flow video, and judge whether the coal block is an abnormal coal block that is not moving based on a number and displacement of optical flow points in the obb coal block detection frame corresponding to each frame of image.

[0054] Further, in a possible implementation manner of the embodiment of the present application, the production module 41 is specifically configured to:

[0055] obtain the scraper conveyor detection area image and the coal block image through a coal mining face monitoring device;

[0056] perform enhancement processing on the scraper conveyor detection area image and the coal block image to obtain an enhanced image, and the enhancement processing includes Gaussian filtering processing, adding Gaussian noise processing, horizontal flipping processing, and clockwise or counterclockwise rotation by 30° and 45° processing;

[0057] perform YOLO-OBB data format labeling on the enhanced image to obtain a data set of the YOLO-OBB data format labeling on the scraper conveyor detection area image and the coal block image.

[0058] Further, in a possible implementation manner of the embodiment of the present application, the derivation module 43 is specifically configured to:

[0059] perform video frame extraction on the real-time detection area coal flow video of the scraper conveyor, denoted as image I, and perform upsampling and downsampling on the image I through a bicubic interpolation device to obtain image I' and image I'', wherein the height and width size of the image I' is 0.5 times the height and width size of the image I, and the height and width size of the image I'' is 2 times the height and width size of the image I;

[0060] input the images I, I' and I'' of different height and width sizes into the trained obb target detection model respectively for derivation to obtain respective derivation results of the images I, I' and I'', and the derivation result includes an initial obb area detection frame, four coordinate values of the initial obb area detection frame, an initial obb area detection frame confidence, an initial obb coal block detection frame, four coordinate values of the initial obb coal block detection frame, and an initial obb coal block detection frame confidence;

[0061] Multiply the coordinate values ​​of the four points of the initial obb region detection box and the initial obb coal block detection box in the derivation result of image I' by 2, and divide the coordinate values ​​of the four points of the initial obb region detection box and the initial obb coal block detection box in the derivation result of image I” by 2 to obtain images I, I', and I” with a uniform coordinate scale;

[0062] Images I, I', and I" with uniform coordinate scale are input into the obb target detection model to obtain the backup obb region detection boxes and backup obb coal block detection boxes for the scraper conveyor detection area in the target derivation result. The non-maximum suppression algorithm is then used to remove detection boxes with high overlap and low confidence levels for the backup obb region detection boxes and backup obb coal block detection boxes in the target derivation result, so as to obtain the obb region detection boxes and obb coal block detection boxes for the real-time detection area of ​​the scraper conveyor.

[0063] Furthermore, in one possible implementation of this invention, the determining module 44 is specifically used for:

[0064] Obtain the area ratio between the obb coal block detection box and the obb region detection box corresponding to each frame of the real-time detection area coal flow video. If the area ratio is greater than a set threshold, the coal block is a large coal block with a preset size or greater.

[0065] When the area ratio is less than a set threshold, adjacent image frames of the real-time detection area coal flow video are extracted, and dense optical flow is extracted from the adjacent image frames to obtain the initial optical flow points of each frame.

[0066] Based on the initial optical flow points of each frame, the number and displacement of the initial optical flow points in the corresponding OBB coal block detection frame of each frame are counted.

[0067] Abnormal optical flow points in the obb coal block detection frame are removed if the displacement of the initial optical flow point is less than the set displacement threshold, and the usable optical flow points in the obb coal block detection frame are obtained.

[0068] If the number of optical flow points is greater than or equal to a set threshold, then the coal block corresponding to the OBB coal block detection box is a normal coal block.

[0069] If the number of optical flow points is less than a set threshold and the scraper conveyor is not running, then the coal block is a normal coal block.

[0070] If the number of available optical flow points is less than a set threshold and the scraper conveyor is in operation, then the coal block is considered an abnormal, stationary coal block.

[0071] The large lump coal target detection device of the embodiment of the present application is based on the data set obtained by performing image enhancement on the image of the detection area of the scraper conveyor and the image of the coal lump and performing YOLO-OBB data format labeling, training the target detection model, and generating the obb target detection model for recognizing the detection frame corresponding to the region of the detection area of the scraper conveyor and the detection frame corresponding to the coal lump; the area ratio between the obb region detection frame and the obb coal lump detection frame corresponding to each image is obtained by deducing the real-time detection area coal flow video of the scraper conveyor, it is judged whether the coal lump belongs to the large lump coal, and based on the number and displacement of the optical flow points in the obb coal lump detection frame of each image, it is judged whether the coal lump belongs to the abnormal coal lump that does not move. Therefore, based on the obb target detection model, the extraction and recognition of the obb detection frame of the scraper conveyor and the coal lump target in the video stream are improved, and accurate detection, classification and recognition of the coal lump are realized.

[0072] To achieve the above-mentioned embodiments, the present application further provides an electronic device, comprising:

[0073] at least one processor; and

[0074] a memory in communication with the at least one processor; wherein

[0075] The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the foregoing method.

[0076] To achieve the above-mentioned embodiments, the present application further provides a non-transitory computer readable storage medium storing computer instructions, the computer instructions being used to make the computer execute the foregoing method.

[0077] In the description of the present application, the description of the terms "one embodiment", "some embodiments", "an example", "a specific example", or "some examples" means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In the present application, the illustrative description of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any appropriate manner in any one or more embodiments or examples. In addition, the skilled in the art can combine and combine the different embodiments or examples described in the present application and the features of the different embodiments or examples without contradiction.

[0078] Moreover, the terms "first", "second", etc. are used herein only to describe different instances, and do not imply or suggest relative importance or a number of indicated technical features. Thus, features defined with "first", "second" can include at least one of such features, either explicitly or implicitly. In the description of the application, the meaning of "a plurality" is at least two, for example two, three, etc., unless explicitly specified otherwise.

[0079] Any process or method descriptions or blocks in flow charts described herein and elsewhere can be understood as representing modules, segments, or portions of code which include one or more executable instructions for implementing specific logical functions or steps in the process. Alternate implementations are included within the scope of the preferred embodiments of the application in which additional functionality can be added or existing functionality can be modified by, for example, adding, removing or modifying the steps of the process described.

[0080] The logic and / or steps represented in the flow charts described herein and elsewhere can be considered as a sequence of executable instructions, for example, to implement logical functions, which can be embodied in any computer-readable medium for use by or in connection with an instruction execution system, apparatus, or device, such as a computer-based system, processor-containing system, or other system that can fetch instructions from a instruction execution system, apparatus, or device and execute instructions. For purposes of this specification, a "computer-readable medium" can be any apparatus that can contain, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of the computer-readable medium include the following: an electrical connection having one or more wires (electrical apparatus), a portable computer diskette (magnetic apparatus), a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber (optical apparatus), and a portable compact disc read-only memory (CDROM). In addition, the computer-readable medium can even be paper or another suitable medium upon which the program is printed, as the program can be electronically captured, for example, via an optical scanner, then compiled, interpreted, or otherwise processed, and stored in a computer memory in order to be executed.

[0081] It should be understood that each part of the present application can be realized by hardware, software, firmware or their combination. In the above-mentioned embodiments, a plurality of steps or methods can be realized by software or firmware stored in a memory and executed by a suitable instruction execution system. As in another embodiment, if realized by hardware, any one or their combination of the following technologies known in the art can be used: discrete logic circuit with logic gate circuit for implementing logic function on data signal, application specific integrated circuit with suitable combination logic gate circuit, programmable gate array (PGA), field programmable gate array (FPGA) and the like.

[0082] Those skilled in the art can understand that all or part of the steps carried out by the above-mentioned embodiment method can be completed by a program instructing the relevant hardware, and the program can be stored in a computer readable storage medium, and when the program is executed, it includes one of the steps of the method embodiment or a combination thereof.

[0083] In addition, each functional unit in each embodiment of the present application can be integrated in one processing module, or each unit can exist physically, or two or more units can be integrated in one module. The above-mentioned integrated module can be realized in the form of hardware or in the form of software functional module. The integrated module, if realized in the form of software functional module and sold or used as an independent product, can also be stored in a computer readable storage medium.

[0084] The above-mentioned storage medium can be read-only memory, disk or optical disk and the like. Although the embodiments of the present application have been shown and described above, it should be understood that the above-mentioned embodiments are exemplary and cannot be understood as limiting the present application, and those skilled in the art can make changes, modifications, replacements and variations to the above-mentioned embodiments within the scope of the present application.

Claims

1. A method for detecting large coal blocks, characterized in that, The method includes: A dataset is created, wherein the dataset is obtained by image enhancement of images of the scraper conveyor detection area and coal block images, and then the enhanced images are labeled in YOLO-OBB data format; The dataset is used to train an object detection model to generate an OBB object detection model that can identify the detection boxes corresponding to the detection area of ​​the scraper conveyor and the detection boxes corresponding to the coal blocks. Based on the aforementioned obb target detection model, the coal flow video of the real-time detection area of ​​the scraper conveyor is derived to obtain the obb area detection box and the obb coal block detection box of the real-time detection area of ​​the scraper conveyor. Based on the area ratio between the obb coal block detection box and the obb region detection box corresponding to each frame of the real-time coal flow video, it is determined whether the coal block belongs to a large coal block larger than a preset size. Furthermore, based on the number and displacement of optical flow points within the obb coal block detection box corresponding to each frame, it is determined whether the coal block belongs to an immobile abnormal coal block, including: Obtain the area ratio between the obb coal block detection box and the obb region detection box corresponding to each frame of the real-time detection area coal flow video. If the area ratio is greater than a set threshold, the coal block is a large coal block with a preset size or greater. When the area ratio is less than a set threshold, adjacent image frames of the real-time detection area coal flow video are extracted, and dense optical flow is extracted from the adjacent image frames to obtain the initial optical flow points of each frame. Based on the initial optical flow points of each frame, the number and displacement of the initial optical flow points in the corresponding OBB coal block detection frame of each frame are counted. Abnormal optical flow points in the obb coal block detection frame are removed if the displacement of the initial optical flow point is less than the set displacement threshold, and the usable optical flow points in the obb coal block detection frame are obtained. If the number of optical flow points is greater than or equal to a set threshold, then the coal block corresponding to the OBB coal block detection box is a normal coal block. If the number of optical flow points is less than a set threshold and the scraper conveyor is not running, then the coal block is a normal coal block. If the number of available optical flow points is less than a set threshold and the scraper conveyor is in operation, then the coal block is considered an abnormal, stationary coal block.

2. The method according to claim 1, characterized in that, The dataset is created by performing image enhancement on images of the scraper conveyor detection area and coal blocks, and then annotating the enhanced images in YOLO-OBB data format, including: Images of the scraper conveyor detection area and coal blocks are obtained through coal mining face monitoring equipment; The images of the scraper conveyor detection area and the coal block are enhanced to obtain enhanced images. The enhancement process includes Gaussian filtering, adding Gaussian noise, horizontal flipping, and clockwise or counterclockwise rotation of 30° and 45°. The enhanced images are annotated with YOLO-OBB data format to obtain a dataset of YOLO-OBB data format annotated images of the scraper conveyor detection area and coal blocks.

3. The method according to claim 1, characterized in that, The step of deriving the real-time detection area coal flow video of the scraper conveyor based on the obb target detection model to obtain the obb area detection box and the obb coal block detection box of the real-time detection area of ​​the scraper conveyor includes: Video frames were extracted from the real-time monitoring area of ​​the scraper conveyor and denoted as image I. Image I was then upsampled and downsampled using a bicubic interpolation method to obtain image I2. ' and Image I ” Image I ' The height and width dimensions are 0.5 times the height and width dimensions of image I. Image I ” The height and width dimensions are twice the height and width dimensions of image I; Images I and I with different height and width dimensions ' I ” The inputs are respectively fed into the trained obb object detection model for derivation, resulting in I and I. ' I ” The corresponding derivation results include the initial obb region detection box, the coordinate values ​​of the four points of the initial obb region detection box, the confidence level of the initial obb region detection box, the initial obb coal block detection box, the coordinate values ​​of the four points of the initial obb coal block detection box, and the confidence level of the initial obb coal block detection box. Image I ' In the derivation results, the coordinates of the four points of the initial obb region detection box and the initial obb coal block detection box are multiplied by 2, and the image I is... ” In the derivation results, the coordinate values ​​of the four points of the initial OBB region detection box and the initial OBB coal block detection box are uniformly divided by 2 to obtain images I and I with a uniform coordinate scale. ' I ” ; Images I and I with uniform coordinate scale ' I ” The data is input into the obb target detection model to obtain the backup obb region detection boxes and backup obb coal block detection boxes for the scraper conveyor detection area in the target derivation result. The non-maximum suppression algorithm is then used to remove detection boxes with high overlap and low confidence levels for the backup obb region detection boxes and backup obb coal block detection boxes in the target derivation result, so as to obtain the obb region detection boxes and obb coal block detection boxes for the real-time detection area of ​​the scraper conveyor.

4. A device for detecting large coal blocks, characterized in that, The device includes: The creation module is used to create a dataset, wherein the dataset is obtained by image enhancement of the scraper conveyor detection area image and coal block image, and then labeling the enhanced image in YOLO-OBB data format; The training module is used to train the target detection model on the dataset and generate an OBB target detection model that can identify the detection boxes corresponding to the detection area of ​​the scraper conveyor and the detection boxes corresponding to the coal blocks. The derivation module is used to derive the coal flow video of the real-time detection area of ​​the scraper conveyor based on the obb target detection model, so as to obtain the obb area detection box and the obb coal block detection box of the real-time detection area of ​​the scraper conveyor. The judgment module is used to determine whether a coal block is a large coal block larger than a preset size based on the area ratio between the obb coal block detection box and the obb region detection box corresponding to each frame of the real-time detection area coal flow video, and to determine whether a coal block is an immobile abnormal coal block based on the number and displacement of optical flow points in the obb coal block detection box corresponding to each frame of the image. The judgment module is specifically used for: Obtain the area ratio between the obb coal block detection box and the obb region detection box corresponding to each frame of the real-time detection area coal flow video. If the area ratio is greater than a set threshold, the coal block is a large coal block with a preset size or greater. When the area ratio is less than a set threshold, adjacent image frames of the real-time detection area coal flow video are extracted, and dense optical flow is extracted from the adjacent image frames to obtain the initial optical flow points of each frame. Based on the initial optical flow points of each frame, the number and displacement of the initial optical flow points in the corresponding OBB coal block detection frame of each frame are counted. Abnormal optical flow points in the obb coal block detection frame are removed if the displacement of the initial optical flow point is less than the set displacement threshold, and the usable optical flow points in the obb coal block detection frame are obtained. If the number of optical flow points is greater than or equal to a set threshold, then the coal block corresponding to the OBB coal block detection box is a normal coal block. If the number of optical flow points is less than a set threshold and the scraper conveyor is not running, then the coal block is a normal coal block. If the number of available optical flow points is less than a set threshold and the scraper conveyor is in operation, then the coal block is considered an abnormal, stationary coal block.

5. The apparatus according to claim 4, characterized in that, The manufacturing module is specifically used for: Images of the scraper conveyor detection area and coal blocks are obtained through coal mining face monitoring equipment; The images of the scraper conveyor detection area and the coal block are enhanced to obtain enhanced images. The enhancement process includes Gaussian filtering, adding Gaussian noise, horizontal flipping, and clockwise or counterclockwise rotation of 30° and 45°. The enhanced images are annotated with YOLO-OBB data format to obtain a dataset of YOLO-OBB data format annotated images of the scraper conveyor detection area and coal blocks.

6. The apparatus according to claim 4, characterized in that, The derivation module is specifically used for: Video frames were extracted from the real-time monitoring area of ​​the scraper conveyor and denoted as image I. Image I was then upsampled and downsampled using a bicubic interpolation device to obtain image I2. ' and Image I ” Image I ' The height and width dimensions are 0.5 times the height and width dimensions of image I. Image I ” The height and width dimensions are twice the height and width dimensions of image I; Images I and I with different height and width dimensions ' I ” The inputs are respectively fed into the trained obb object detection model for derivation, resulting in I and I. ' I ” The corresponding derivation results include the initial obb region detection box, the coordinate values ​​of the four points of the initial obb region detection box, the confidence level of the initial obb region detection box, the initial obb coal block detection box, the coordinate values ​​of the four points of the initial obb coal block detection box, and the confidence level of the initial obb coal block detection box. Image I ' In the derivation results, the coordinates of the four points of the initial obb region detection box and the initial obb coal block detection box are multiplied by 2, and the image I is... ” In the derivation results, the coordinate values ​​of the four points of the initial OBB region detection box and the initial OBB coal block detection box are uniformly divided by 2 to obtain images I and I with a uniform coordinate scale. ' I ” ; Images I and I with uniform coordinate scale ' I ” The data is input into the obb target detection model to obtain the backup obb region detection boxes and backup obb coal block detection boxes for the scraper conveyor detection area in the target derivation result. The non-maximum suppression algorithm is then used to remove detection boxes with high overlap and low confidence levels for the backup obb region detection boxes and backup obb coal block detection boxes in the target derivation result, so as to obtain the obb region detection boxes and obb coal block detection boxes for the real-time detection area of ​​the scraper conveyor.

7. An electronic device, characterized in that, include: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-3.

8. A non-transitory computer-readable storage medium storing computer instructions, characterized in that, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-3.

Images