Large coal target detection method and device

Abstract

The invention provides a large coal target detection method and device, and relates to the technical field of computer vision, and the method comprises the steps: training a target detection model based on a data set obtained through carrying out the image enhancement of a detection region image and a coal block image of a scraper conveyor and carrying out the YOLO-OBB data format labeling, and obtaining a target detection result; generating an obb target detection model for identifying an area detection frame corresponding to the detection area of the scraper conveyor and a coal block detection frame corresponding to the coal block; the method comprises the following steps: acquiring a real-time detection area coal flow video of a scraper conveyor, deducing the real-time detection area coal flow video of the scraper conveyor to obtain an area ratio between an obb area detection frame and an obb coal briquette detection frame corresponding to each frame of image, judging whether the coal briquette belongs to large coal, and judging whether the coal briquette belongs to fixed abnormal coal briquettes based on the number and displacement of optical flow points in the obb coal briquette detection frame of each frame of image. Therefore, based on the obb target detection model, extraction and identification of the obb detection frame of the scraper conveyor and the coal briquette target in the video stream are improved, and accurate detection, classification and identification of the coal briquettes are further realized.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a method, device, electronic equipment and storage medium for detecting a large lump of coal. Background Art

[0002] However, in the process of coal mining, large pieces of coal are easily produced, and the crusher cannot crush them in time. As time goes by, it may cause the scraper conveyor to be blocked, and in serious cases, it may cause damage to the scraper conveyor. Therefore, realizing intelligent detection of large pieces of coal in the production process of scraper conveyors is of great significance to coal mine production safety and reducing manpower and improving efficiency.

[0003] Among the related technologies, the large coal block target detection methods in coal mines can be roughly divided into three categories. The first is to rely on manual observation, which requires workers to stay in the observation area for a long time. The nature of the work is single, which can easily cause visual fatigue and even safety accidents. The second is to use sensors for monitoring, which requires a good sensor monitoring environment and high-precision and stable hardware. However, due to the poor transportation status of the comprehensive mining working face, there are defects such as easy damage, low efficiency and poor accuracy. The third is to use computer vision and deep learning technology, but it cannot accurately identify the main detection area of ​​the scraper conveyor and the coal block area, nor can it determine whether there are abnormal coal blocks. Summary of the invention

[0004] The present invention aims to solve one of the technical problems in the related art at least to a certain extent.

[0005] To this end, the first purpose of the present invention is to propose a large coal target detection method, based on the obb target detection model, to improve the extraction and recognition of the obb detection frame of the scraper conveyor and coal block targets in the video stream, and then realize the accurate detection, classification and recognition of coal blocks.

[0006] The second object of the present invention is to provide a large coal target detection device.

[0007] A third objective of the present invention is to provide an electronic device.

[0008] A fourth object of the present invention is to provide a non-transitory computer-readable storage medium storing computer instructions.

[0009] To achieve the above-mentioned purpose, a first embodiment of the present invention provides a method for detecting a large lump of coal, the method comprising:

[0010] A data set is prepared, wherein the data set is obtained by performing image enhancement on the scraper conveyor detection area image and the coal block image, and then annotating the enhanced image in the YOLO-OBB data format;

[0011] The data set is used to train a target detection model to generate an obb target detection model that identifies a detection area corresponding to a scraper conveyor and a detection area corresponding to a coal block;

[0012] Based on the obb target detection model, the coal flow video of the scraper conveyor real-time detection area is derived to obtain the obb area detection frame of the scraper conveyor real-time detection area and the obb coal block detection frame of the coal block;

[0013] Based on the area ratio between the obb coal block detection frame corresponding to each frame image in the real-time detection area coal flow video and the obb area detection frame, it is judged whether the coal block is a large piece of coal larger than the preset size, and based on the number and displacement of optical flow points in the obb coal block detection frame corresponding to each frame image, it is judged whether the coal block is an abnormal and stationary coal block.

[0014] To achieve the above-mentioned purpose, a second embodiment of the present invention provides a large coal target detection device, the device comprising:

[0015] A production module, used for producing a data set, wherein the data set is obtained by performing image enhancement on the scraper conveyor detection area image and the coal block image, and then annotating the enhanced image in YOLO-OBB data format;

[0016] A training module, used to train the target detection model with the data set, and generate an OBB target detection model that identifies the detection area of ​​the scraper conveyor corresponding to the area detection frame, and the coal block corresponding to the coal block detection frame;

[0017] A derivation module, used for derivation of the coal flow video in the real-time detection area of ​​the scraper conveyor based on the obb target detection model, so as to obtain the obb area detection frame of the real-time detection area of ​​the scraper conveyor and the obb coal block detection frame of the coal block;

[0018] The judgment module is used to judge whether the coal block is a large piece of coal larger than a preset size based on the area ratio between the obb coal block detection frame corresponding to each frame image in the real-time detection area coal flow video and the obb area detection frame, and to judge whether the coal block is an abnormal and stationary coal block based on the number and displacement of optical flow points in the obb coal block detection frame corresponding to each frame image.

[0019] To achieve the above-mentioned purpose, the third aspect of the present invention proposes an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method described in the first aspect.

[0020] In order to achieve the above-mentioned purpose, the fourth aspect of the present invention proposes a non-transitory computer-readable storage medium storing computer instructions, where the computer instructions are used to enable the computer to execute the method described in the first aspect.

[0021] The large-lump coal target detection method, device, electronic device and storage medium of the embodiment of the present invention are based on the data set obtained by performing image enhancement on the scraper conveyor detection area image and the coal block image and annotating them in YOLO-OBB data format, training the target detection model, generating an obb target detection model that identifies the corresponding regional detection frame of the scraper conveyor detection area and the corresponding coal block detection frame of the coal block; deriving the real-time coal flow video of the scraper conveyor detection area, obtaining the area ratio between the obb regional detection frame and the obb coal block detection frame corresponding to each frame image, judging whether the coal block belongs to a large-lump coal, and judging whether the coal block belongs to an abnormal immobile coal block based on the number and displacement of optical flow points in the obb coal block detection frame of each frame image. Therefore, based on the obb target detection model, the extraction and recognition of the obb detection frame of the scraper conveyor and the coal block target in the video stream are improved, thereby realizing accurate detection, classification and recognition of the coal block.

[0022] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The above and / or additional aspects and advantages of the present invention will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0024] Figure 1 A schematic diagram of a flow chart of a large lump coal target detection method provided by an embodiment of the present invention;

[0025] Figure 2 A flow chart for derivation of a coal flow video model of a scraper conveyor provided by an embodiment of the present invention;

[0026] Figure 3 A flowchart of a method for detecting a large lump of coal provided by an embodiment of the present invention;

[0027] Figure 4 A schematic structural diagram of a large lump coal target detection device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0028] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, and should not be construed as limiting the present invention.

[0029] It should be noted that the acquisition, storage, use, and processing of data in the technical solution of the present invention are in compliance with the relevant provisions of relevant laws and regulations.

[0030] The following describes a method and apparatus for detecting a large lump of coal target according to an embodiment of the present invention with reference to the accompanying drawings.

[0031] Figure 1 A schematic flow chart of a method for detecting a large lump of coal provided in an embodiment of the present invention.

[0032] like Figure 1 As shown, the method comprises the following steps:

[0033] Step 101, making a data set, wherein the data set is obtained by performing image enhancement on the scraper conveyor detection area image and the coal block image, and then annotating the enhanced image in the YOLO-OBB data format.

[0034] In some embodiments, one implementation method of making a data set may be to obtain the scraper conveyor detection area image and the coal block image through the coal mining face monitoring equipment; enhance the scraper conveyor detection area image and the coal block image to obtain an enhanced image, wherein the enhancement processing includes Gaussian filtering, adding Gaussian noise, horizontal flipping, and clockwise or counterclockwise rotation of 30° and 45°; and annotate the enhanced image in YOLO-OBB data format to obtain a data set in which the scraper conveyor detection area image and the coal block image are annotated in YOLO-OBB data format. Thus, the data set is enriched and the accuracy of the target detection model trained by the data set is improved.

[0035] Furthermore, performing Gaussian filtering on the scraper conveyor detection area images and coal lump images can enable the target detection model to enhance its ability to handle target motion blur; adding Gaussian noise processing on the scraper conveyor detection area images and coal lump images can enable the target detection model to enhance its ability to handle coal dust and water mist; horizontally flipping the scraper conveyor detection area images and coal lump images can enable the target detection model to learn the characteristics of objects in different directions; rotating the scraper conveyor detection area images and coal lump images clockwise or counterclockwise by 30° and 45° can enable the target detection model to learn the characteristics of objects at different angles, simulate the change of camera position, and enhance the rotation invariance of the target detection model.

[0036] Step 102, training the target detection model with the data set, generating an OBB target detection model that identifies the scraper conveyor detection area corresponding to the area detection frame, and the coal block corresponding to the coal block detection frame.

[0037] In some embodiments, the target detection model may be yolov11, but is not limited thereto.

[0038] Step 103, deriving the coal flow video in the real-time detection area of ​​the scraper conveyor based on the obb target detection model to obtain the obb area detection frame of the real-time detection area of ​​the scraper conveyor and the obb coal block detection frame of the coal block.

[0039] In some embodiments, based on the obb target detection model, the coal flow video of the real-time detection area of ​​the scraper conveyor is derived to obtain the obb area detection frame of the real-time detection area of ​​the scraper conveyor and the obb coal block detection frame of the coal block. One implementation method can be to extract the video frame of the coal flow video of the real-time detection area of ​​the scraper conveyor, recorded as image I, and use the bicubic interpolation method to upsample and downsample the image I to obtain image I' and image I", wherein the height and width of image I' are 0.5 times the height and width of image I, and the height and width of image I" are 2 times the height and width of image I; images I, I', and I" with different height and width sizes are respectively input into the trained obb target detection model for derivation to obtain the derivation results corresponding to I, I', and I", and the derivation results include the initial obb area detection frame, the coordinate values ​​of the four points of the initial obb area detection frame, the confidence of the initial obb area detection frame, the initial obb coal block detection frame, the coordinate values ​​of the four points of the initial obb coal block detection frame, and the confidence of the initial obb coal block detection frame. ; The sum of the coordinate values ​​of the four points of the initial obb region detection frame and the initial obb coal block detection frame in the derivation result of image I' is uniformly multiplied by 2, and the coordinate values ​​of the four points of the initial obb region detection frame and the initial obb coal block detection frame in the derivation result of image I" are uniformly divided by 2 to obtain images I, I', I" with a unified coordinate scale; the images I, I', I" with a unified coordinate scale are input into the obb target detection model to obtain the spare obb region detection frame of the scraper conveyor detection area and the spare obb coal block detection frame of the coal block in the target derivation result, and the non-maximum suppression algorithm is used to remove the detection frames with high overlap and low confidence of the spare obb region detection frame and the spare obb coal block detection frame in the target derivation result, so as to obtain the obb region detection frame of the scraper conveyor real-time detection area and the obb coal block detection frame of the coal block. Therefore, the obb region detection frame and the spare obb coal block detection frame are rectangles with a certain rotation angle, which can more accurately describe the main detection area of ​​the scraper conveyor, as well as the direction and size of the coal block.

[0040] The derivation result may be a set of data including (detection box label name, coordinate values ​​of four points of the detection box, and detection box confidence).

[0041] Optionally, in order to better understand the present invention, the present invention also provides a scraper conveyor coal flow video model derivation flow chart, such as Figure 2As shown, specifically, taking the target detection model yolov11 as an example, the video frame of the coal flow video in the real-time detection area of ​​the scraper conveyor is extracted, recorded as image I, and the image I is upsampled and downsampled by the bicubic interpolation method to obtain image I' and image I", and the images I, I', and I" of different height and width sizes are respectively input into yolov11 for derivation, and the derivation results of images I, I', and I" (initial obb area detection frame and initial obb coal block detection frame) are unified in coordinate scale (normalization), and the images I, I', and I" with unified coordinate scale are obtained. After inputting the target derivation results of yolov11, the spare obb area detection frame of the scraper conveyor detection area and the spare obb coal block detection frame of the coal block are obtained, and the non-maximum suppression algorithm is used to remove the detection frames with high overlap and low confidence of the spare obb area detection frame and the spare obb coal block detection frame in the target derivation results, so as to obtain the obb area detection frame of the real-time detection area of ​​the scraper conveyor and the obb coal block detection frame of the coal block.

[0042] Step 104, based on the area ratio between the obb coal block detection frame corresponding to each frame image in the real-time detection area coal flow video and the obb area detection frame, judge whether the coal block is a large piece of coal larger than a preset size, and based on the number and displacement of optical flow points in the obb coal block detection frame corresponding to each frame image, judge whether the coal block is a stationary abnormal coal block.

[0043] In some embodiments, based on the area ratio between the obb coal block detection frame and the obb area detection frame corresponding to each frame image in the real-time detection area coal flow video, it is judged whether the coal block is a large piece of coal larger than a preset size, and based on the number and displacement of optical flow points in the obb coal block detection frame corresponding to each frame image, it is judged whether the coal block is an abnormal and stationary coal block. One implementation method may be to obtain the area ratio between the obb coal block detection frame and the obb area detection frame corresponding to each frame image in the real-time detection area coal flow video. When the area ratio is greater than a set threshold, the coal block is a large piece of coal greater than or equal to a preset size; when the area ratio is less than the set threshold, adjacent image frames of the real-time detection area coal flow video are extracted, and the adjacent image frames are subjected to Dense optical flow extraction is used to obtain the initial optical flow points of each frame image; according to the initial optical flow points of each frame image, the number and displacement of the initial optical flow points in the obb coal block detection frame corresponding to each frame image are counted; the abnormal optical flow points whose displacement of the initial optical flow points in the obb coal block detection frame is less than the set displacement threshold are eliminated to obtain the available optical flow points in the obb coal block detection frame; when the number of optical flow points is greater than or equal to the set number threshold, the coal block corresponding to the obb coal block detection frame is a normal coal block; when the number of optical flow points is less than the set number threshold and the scraper conveyor equipment is not running, the coal block is a normal coal block; when the number of available optical flow points is less than the set number threshold and the scraper conveyor equipment is in operation, the coal block is an abnormal coal block that does not move. Therefore, these abnormal coal blocks and lump coal in coal flow transportation are detected and motion estimated, and the abnormal coal blocks that do not move are identified, and alarms are given to prompt manual processing, and the machine is shut down for maintenance in serious cases to prevent greater harm.

[0044] Among them, when it is determined that the coal block is an abnormal and immobile coal block, an alarm is required to remind manual inspection and processing to ensure the safe operation of the scraper conveyor equipment.

[0045] The large-lump coal target detection method of the embodiment of the present invention is based on the data set obtained by performing image enhancement on the scraper conveyor detection area image and the coal block image and annotating them in the YOLO-OBB data format, training the target detection model, generating an obb target detection model that identifies the region detection frame corresponding to the scraper conveyor detection area and the coal block detection frame corresponding to the coal block; deriving the real-time coal flow video of the scraper conveyor detection area, obtaining the area ratio between the obb region detection frame and the obb coal block detection frame corresponding to each frame image, judging whether the coal block belongs to a large-lump coal, and judging whether the coal block belongs to an abnormal immobile coal block based on the number and displacement of optical flow points in the obb coal block detection frame of each frame image. Therefore, based on the obb target detection model, the extraction and recognition of the obb detection frame of the scraper conveyor and the coal block target in the video stream is improved, thereby realizing accurate detection, classification and recognition of the coal block.

[0046] In addition, in some embodiments, the present invention also provides a flowchart of a method for detecting a large coal target, such as Figure 3 As shown, a real-time detection area coal flow video is obtained, and adjacent image frames of the real-time detection area coal flow video are extracted; each frame of the image is input into the obb target detection model for derivation to obtain a derivation result (obb coal block detection frame and obb area detection frame), and dense optical flow extraction is performed on adjacent image frames to obtain the initial optical flow point of each frame of the image; when the obb coal block detection frame does not exist in the obb area detection frame, the coal block is not in the detection area; when the obb coal block detection frame exists in the obb area detection frame, the area ratio between the obb coal block detection frame and the obb area detection frame is calculated. When the area ratio is greater than the set threshold, the coal block is a large piece of coal greater than or equal to a preset size; when the area ratio is less than When the threshold is set, the number and displacement of the initial optical flow points in the obb coal block detection frame corresponding to each frame image are counted; the abnormal optical flow points whose displacement of the initial optical flow points in the obb coal block detection frame is less than the set displacement threshold are eliminated to obtain the available optical flow points in the obb coal block detection frame; when the number of optical flow points is greater than or equal to the set number threshold, the coal block corresponding to the obb coal block detection frame is a normal coal block; when the number of optical flow points is less than the set number threshold and the scraper conveyor equipment is not running, the coal block is a normal coal block; when the number of available optical flow points is less than the set number threshold and the scraper conveyor equipment is in operation, the coal block is an abnormal and immovable coal block, and an alarm is issued for manual inspection and processing to ensure coal mine production safety.

[0047] In order to implement the above embodiment, the present invention also proposes a large coal target detection device.

[0048] Figure 4 A schematic structural diagram of a large lump coal target detection device provided in an embodiment of the present invention.

[0049] like Figure 4 As shown, the large coal target detection device 40 includes: a production module 41, a training module 42, a derivation module 43 and a judgment module 44.

[0050] A production module 41 is used to produce a data set, wherein the data set is obtained by performing image enhancement on the scraper conveyor detection area image and the coal block image, and then annotating the enhanced image in YOLO-OBB data format;

[0051] A training module 42 is used to train the target detection model with the data set to generate an obb target detection model that identifies the detection area of ​​the scraper conveyor corresponding to the area detection frame and the detection frame of the coal block corresponding to the coal block;

[0052] A derivation module 43 is used to derive the coal flow video of the scraper conveyor real-time detection area based on the obb target detection model to obtain the obb area detection frame of the scraper conveyor real-time detection area and the obb coal block detection frame of the coal block;

[0053] The judgment module 44 is used to judge whether the coal block is a large piece of coal larger than a preset size based on the area ratio between the obb coal block detection frame corresponding to each frame image in the real-time detection area coal flow video and the obb area detection frame, and to judge whether the coal block is a stationary abnormal coal block based on the number and displacement of optical flow points in the obb coal block detection frame corresponding to each frame image.

[0054] Furthermore, in a possible implementation of the embodiment of the present invention, the making module 41 is specifically used to:

[0055] Acquire the detection area image of the scraper conveyor and the coal block image through the coal mining face monitoring equipment;

[0056] Performing enhancement processing on the scraper conveyor detection area image and the coal block image to obtain an enhanced image, wherein the enhancement processing includes Gaussian filtering processing, adding Gaussian noise processing, horizontal flipping processing, and clockwise or counterclockwise rotation of 30° and 45° processing;

[0057] The enhanced image is annotated in the YOLO-OBB data format to obtain a data set of scraper conveyor detection area images and coal block images annotated in the YOLO-OBB data format.

[0058] Furthermore, in a possible implementation of the embodiment of the present invention, the derivation module 43 is specifically used to:

[0059] The video frame of the coal flow video in the real-time detection area of ​​the scraper conveyor is extracted and recorded as image I. The image I is upsampled and downsampled by a bicubic interpolation device to obtain image I' and image I", where the height and width of image I' are 0.5 times the height and width of image I, and the height and width of image I" are 2 times the height and width of image I;

[0060] Input images I, I', and I" of different height and width sizes into the trained obb target detection model for derivation, and obtain derivation results corresponding to I, I', and I", respectively, wherein the derivation results include an initial obb region detection frame, four point coordinate values ​​of the initial obb region detection frame, confidence of the initial obb region detection frame, an initial obb coal block detection frame, four point coordinate values ​​of the initial obb coal block detection frame, and confidence of the initial obb coal block detection frame;

[0061] The sum of the coordinate values ​​of the four points of the initial obb region detection frame and the initial obb coal block detection frame in the derivation result of image I' is uniformly multiplied by 2, and the coordinate values ​​of the four points of the initial obb region detection frame and the initial obb coal block detection frame in the derivation result of image I" are uniformly divided by 2 to obtain images I, I', I" with uniform coordinate scales;

[0062] The images I, I', I" with unified coordinate scale are input into the obb target detection model to obtain the spare obb area detection frame of the scraper conveyor detection area and the spare obb coal block detection frame of the coal block in the target derivation result, and the non-maximum suppression algorithm is used to remove the detection frames with high overlap and low confidence of the spare obb area detection frame and the spare obb coal block detection frame in the target derivation result to obtain the obb area detection frame of the scraper conveyor real-time detection area and the obb coal block detection frame of the coal block.

[0063] Furthermore, in a possible implementation of the embodiment of the present invention, the determination module 44 is specifically configured to:

[0064] Obtaining the area ratio between the obb coal block detection frame and the obb area detection frame corresponding to each frame image in the real-time detection area coal flow video. When the area ratio is greater than a set threshold, the coal block is a large piece of coal greater than or equal to a preset size.

[0065] When the area ratio is less than the set threshold, the adjacent image frames of the real-time detection area coal flow video are extracted, and dense optical flow extraction is performed on the adjacent image frames to obtain the initial optical flow points of each frame of the image;

[0066] According to the initial optical flow points of each frame image, the number and displacement of the initial optical flow points in the obb coal block detection frame corresponding to each frame image are counted;

[0067] Eliminate the abnormal optical flow points whose displacement of the initial optical flow points in the obb coal block detection frame is less than the set displacement threshold, and obtain the available optical flow points in the obb coal block detection frame;

[0068] When the number of the optical flow points is greater than or equal to the set number threshold, the coal block corresponding to the obb coal block detection frame is a normal coal block;

[0069] When the number of the optical flow points is less than the set number threshold and the scraper conveyor device is not running, the coal block is a normal coal block;

[0070] When the number of available optical flow points is less than a set number threshold and the scraper conveyor device is in operation, the coal block is an abnormal and immobile coal block.

[0071] The large-lump coal target detection device of the embodiment of the present invention is based on the data set obtained by performing image enhancement on the scraper conveyor detection area image and the coal block image and annotating them in YOLO-OBB data format, training the target detection model, generating an obb target detection model that identifies the corresponding regional detection frame of the scraper conveyor detection area and the corresponding coal block detection frame of the coal block; deriving the real-time coal flow video of the scraper conveyor detection area, obtaining the area ratio between the obb regional detection frame and the obb coal block detection frame corresponding to each frame image, judging whether the coal block belongs to a large-lump coal, and judging whether the coal block belongs to an abnormal immobile coal block based on the number and displacement of optical flow points in the obb coal block detection frame of each frame image. Therefore, based on the obb target detection model, the extraction and recognition of the obb detection frame of the scraper conveyor and the coal block target in the video stream is improved, thereby realizing accurate detection, classification and recognition of the coal block.

[0072] In order to implement the above embodiment, the present invention further provides an electronic device, including:

[0073] at least one processor; and

[0074] a memory communicatively connected to the at least one processor; wherein,

[0075] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can perform the aforementioned method.

[0076] In order to implement the above embodiment, the present invention further proposes a non-transitory computer-readable storage medium storing computer instructions, where the computer instructions are used to enable the computer to execute the above method.

[0077] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0078] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of the present invention, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0079] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code comprising one or more executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present invention includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present invention belong.

[0080] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute the instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purpose of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or processing in other suitable ways if necessary, and then stored in a computer memory.

[0081] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, a plurality of steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0082] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0083] In addition, each functional unit in each embodiment of the present invention may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0084] The storage medium mentioned above may be a read-only memory, a magnetic disk or an optical disk, etc. Although the embodiments of the present invention have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limiting the present invention. A person of ordinary skill in the art may change, modify, replace and modify the above embodiments within the scope of the present invention.

Claims

1. A method for detecting large coal targets, characterized in that: The method comprises: A data set is prepared, wherein the data set is obtained by performing image enhancement on the scraper conveyor detection area image and the coal block image, and then annotating the enhanced image in the YOLO-OBB data format; The data set is used to train a target detection model to generate an obb target detection model that identifies a detection frame corresponding to a scraper conveyor detection area and a detection frame corresponding to a coal block; Based on the obb target detection model, the coal flow video of the scraper conveyor real-time detection area is derived to obtain the obb area detection frame of the scraper conveyor real-time detection area and the obb coal block detection frame of the coal block; Based on the area ratio between the obb coal block detection frame corresponding to each frame image in the real-time detection area coal flow video and the obb area detection frame, it is judged whether the coal block is a large piece of coal larger than the preset size, and based on the number and displacement of optical flow points in the obb coal block detection frame corresponding to each frame image, it is judged whether the coal block is an abnormal and stationary coal block.

2. The method according to claim 1, characterized in that The data set is prepared, wherein the data set is obtained by performing image enhancement on the scraper conveyor detection area image and the coal block image, and annotating the enhanced image in YOLO-OBB data format, including: Acquire the detection area image of the scraper conveyor and the coal block image through the coal mining face monitoring equipment; Performing enhancement processing on the scraper conveyor detection area image and the coal block image to obtain an enhanced image, wherein the enhancement processing includes Gaussian filtering processing, adding Gaussian noise processing, horizontal flipping processing, and clockwise or counterclockwise rotation of 30° and 45° processing; The enhanced image is annotated in the YOLO-OBB data format to obtain a data set of scraper conveyor detection area images and coal block images annotated in the YOLO-OBB data format.

3. The method according to claim 1, characterized in that The method of deducing the coal flow video in the real-time detection area of ​​the scraper conveyor based on the obb target detection model to obtain the obb area detection frame of the real-time detection area of ​​the scraper conveyor and the obb coal block detection frame of the coal block includes: The video frame of the coal flow video in the real-time detection area of ​​the scraper conveyor is extracted and recorded as image I. The image I is upsampled and downsampled by the bicubic interpolation method to obtain image I' and image I", where the height and width of image I' are 0.5 times the height and width of image I, and the height and width of image I" are 2 times the height and width of image I; Input images I, I', and I" of different height and width sizes into the trained obb target detection model for derivation, and obtain derivation results corresponding to I, I', and I", respectively, wherein the derivation results include an initial obb region detection frame, four point coordinate values ​​of the initial obb region detection frame, confidence of the initial obb region detection frame, an initial obb coal block detection frame, four point coordinate values ​​of the initial obb coal block detection frame, and confidence of the initial obb coal block detection frame; The sum of the coordinate values ​​of the four points of the initial obb region detection frame and the initial obb coal block detection frame in the derivation result of image I' is uniformly multiplied by 2, and the coordinate values ​​of the four points of the initial obb region detection frame and the initial obb coal block detection frame in the derivation result of image I" are uniformly divided by 2 to obtain images I, I', I" with uniform coordinate scales; The images I, I', I" with unified coordinate scale are input into the obb target detection model to obtain the spare obb area detection frame of the scraper conveyor detection area and the spare obb coal block detection frame of the coal block in the target derivation result, and the non-maximum suppression algorithm is used to remove the detection frames with high overlap and low confidence of the spare obb area detection frame and the spare obb coal block detection frame in the target derivation result to obtain the obb area detection frame of the scraper conveyor real-time detection area and the obb coal block detection frame of the coal block.

4. The method according to claim 1, characterized in that: The method of judging whether the coal block is a large coal block larger than a preset size based on the area ratio between the obb coal block detection frame and the obb area detection frame corresponding to each frame of the real-time detection area coal flow video, and judging whether the coal block is a static abnormal coal block based on the number and displacement of optical flow points in the obb coal block detection frame corresponding to each frame of the image, includes: Obtaining the area ratio between the obb coal block detection frame and the obb area detection frame corresponding to each frame image in the real-time detection area coal flow video. When the area ratio is greater than a set threshold, the coal block is a large piece of coal greater than or equal to a preset size. When the area ratio is less than the set threshold, the adjacent image frames of the real-time detection area coal flow video are extracted, and dense optical flow extraction is performed on the adjacent image frames to obtain the initial optical flow points of each frame of the image; According to the initial optical flow points of each frame image, the number and displacement of the initial optical flow points in the obb coal block detection frame corresponding to each frame image are counted; Eliminate the abnormal optical flow points whose displacement of the initial optical flow points in the obb coal block detection frame is less than the set displacement threshold, and obtain the available optical flow points in the obb coal block detection frame; When the number of the optical flow points is greater than or equal to the set number threshold, the coal block corresponding to the obb coal block detection frame is a normal coal block; When the number of the optical flow points is less than the set number threshold and the scraper conveyor device is not running, the coal block is a normal coal block; When the number of available optical flow points is less than a set number threshold and the scraper conveyor device is in operation, the coal block is an abnormal and immobile coal block.

5. A large coal target detection device, characterized in that: The device comprises: A production module, used for producing a data set, wherein the data set is obtained by performing image enhancement on the scraper conveyor detection area image and the coal block image, and then annotating the enhanced image in YOLO-OBB data format; A training module, used to train the target detection model with the data set, and generate an OBB target detection model that identifies the detection area of ​​the scraper conveyor corresponding to the area detection frame, and the coal block corresponding to the coal block detection frame; A derivation module, used for derivation of the coal flow video in the real-time detection area of ​​the scraper conveyor based on the obb target detection model, so as to obtain the obb area detection frame of the real-time detection area of ​​the scraper conveyor and the obb coal block detection frame of the coal block; The judgment module is used to judge whether the coal block is a large piece of coal larger than a preset size based on the area ratio between the obb coal block detection frame corresponding to each frame image in the real-time detection area coal flow video and the obb area detection frame, and to judge whether the coal block is an abnormal and stationary coal block based on the number and displacement of optical flow points in the obb coal block detection frame corresponding to each frame image.

6. The device according to claim 5, characterized in that The production module is specifically used for: Acquire the detection area image of the scraper conveyor and the coal block image through the coal mining face monitoring equipment; Performing enhancement processing on the scraper conveyor detection area image and the coal block image to obtain an enhanced image, wherein the enhancement processing includes Gaussian filtering processing, adding Gaussian noise processing, horizontal flipping processing, and clockwise or counterclockwise rotation of 30° and 45° processing; The enhanced image is annotated in the YOLO-OBB data format to obtain a data set of scraper conveyor detection area images and coal block images annotated in the YOLO-OBB data format.

7. The device according to claim 5, characterized in that The derivation module is specifically used for: The video frame of the coal flow video in the real-time detection area of ​​the scraper conveyor is extracted and recorded as image I. The image I is upsampled and downsampled by a bicubic interpolation device to obtain image I' and image I", where the height and width of image I' are 0.5 times the height and width of image I, and the height and width of image I" are 2 times the height and width of image I; Input images I, I', and I" of different height and width sizes into the trained obb target detection model for derivation, and obtain derivation results corresponding to I, I', and I", respectively, wherein the derivation results include an initial obb region detection frame, four point coordinate values ​​of the initial obb region detection frame, confidence of the initial obb region detection frame, an initial obb coal block detection frame, four point coordinate values ​​of the initial obb coal block detection frame, and confidence of the initial obb coal block detection frame; The sum of the coordinate values ​​of the four points of the initial obb region detection frame and the initial obb coal block detection frame in the derivation result of image I' is uniformly multiplied by 2, and the coordinate values ​​of the four points of the initial obb region detection frame and the initial obb coal block detection frame in the derivation result of image I" are uniformly divided by 2 to obtain images I, I', I" with uniform coordinate scales; The images I, I', I" with unified coordinate scale are input into the obb target detection model to obtain the spare obb area detection frame of the scraper conveyor detection area and the spare obb coal block detection frame of the coal block in the target derivation result, and the non-maximum suppression algorithm is used to remove the detection frames with high overlap and low confidence of the spare obb area detection frame and the spare obb coal block detection frame in the target derivation result to obtain the obb area detection frame of the scraper conveyor real-time detection area and the obb coal block detection frame of the coal block.

8. The device according to claim 5, characterized in that The judgment module is specifically used for: Obtaining the area ratio between the obb coal block detection frame and the obb area detection frame corresponding to each frame image in the real-time detection area coal flow video. When the area ratio is greater than a set threshold, the coal block is a large piece of coal greater than or equal to a preset size. When the area ratio is less than the set threshold, the adjacent image frames of the real-time detection area coal flow video are extracted, and dense optical flow extraction is performed on the adjacent image frames to obtain the initial optical flow points of each frame of the image; According to the initial optical flow points of each frame image, the number and displacement of the initial optical flow points in the obb coal block detection frame corresponding to each frame image are counted; Eliminate the abnormal optical flow points whose displacement of the initial optical flow points in the obb coal block detection frame is less than the set displacement threshold, and obtain the available optical flow points in the obb coal block detection frame; When the number of the optical flow points is greater than or equal to the set number threshold, the coal block corresponding to the obb coal block detection frame is a normal coal block; When the number of the optical flow points is less than the set number threshold and the scraper conveyor device is not running, the coal block is a normal coal block; When the number of available optical flow points is less than a set number threshold and the scraper conveyor device is in operation, the coal block is an abnormal and immobile coal block.

9. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 4.

10. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-4.

Images