A machine vision-based method and apparatus for detecting belt misalignment

Abstract

This invention relates to a machine vision-based method and apparatus for detecting belt misalignment. The method includes: obtaining a belt mask in an image using an instance segmentation model; extracting edge points from the belt mask; comparing the positional relationship between each edge point and a pre-set non-misalignment boundary of the belt edge to determine whether the edge point is an anomaly; and determining whether the belt is misaligned based on the number of anomalies. The technical solution provided by this invention determines belt misalignment through edge sampling; occlusion of some points does not affect the determination of misalignment, effectively mitigating the influence of random occlusion.

Description

Technical Field

[0001] This invention relates to the fields of mine safety monitoring and belt conveyor, and in particular to a method and device for detecting belt misalignment based on machine vision. Background Technology

[0002] Belt conveyors, with their advantages of long transport distances and large carrying capacity, are crucial transportation equipment in coal mine production. During operation, factors such as belt slack, uneven ore distribution, and mechanical vibration frequently cause belt misalignment. Belt misalignment can damage equipment components, lead to malfunctions and downtime, impact production efficiency and progress, and cause economic losses; it can even cause the belt to overturn and tear, posing serious safety hazards. Therefore, timely detection of belt misalignment is of great importance for ensuring coal mine production efficiency, reducing economic losses, and minimizing safety risks.

[0003] With the development of network monitoring and intelligent recognition technologies, video monitoring has become a common and efficient method for detecting belt misalignment. It mainly includes the following types: First, the image processing method based on OpenCV. This method has poor adaptability and cannot be well applied to scenarios with poor imaging effects such as poor lighting conditions and dust blurring in underground mines. Second, the target detection method, which judges the belt misalignment by detecting the belt idler. Although this method can be well applied to underground environments, it has problems such as not being able to quantify the degree of belt misalignment, especially its poor adaptability to obstructed scenes. Summary of the Invention

[0004] To overcome the problems existing in related technologies, the present invention provides a method and device for detecting belt misalignment based on machine vision.

[0005] According to a first aspect of the present invention, a machine vision-based belt misalignment detection method is provided, comprising:

[0006] Obtain the belt mask in the image using an instance segmentation model;

[0007] Extract the edge points in the belt mask;

[0008] Compare the positional relationship between each edge point and the pre-set non-deviation boundary of the belt edge to determine whether the edge point is an abnormal point;

[0009] Determine whether the belt is misaligned based on the number of abnormal points.

[0010] Furthermore, the extraction of edge points in the belt mask specifically includes:

[0011] Remove the four corner portions of the belt mask;

[0012] Sampling of edge points is performed uniformly on both sides of the remaining belt mask.

[0013] Furthermore, the step of determining whether the belt is misaligned based on the number of abnormal points specifically includes:

[0014] If the number of edge points on the side of the belt edge that is not the deviation boundary is greater than the preset threshold, then the belt is determined to be deviating to that side.

[0015] Furthermore, the method also includes:

[0016] Calculate the average distance between the abnormal point and the non-deviation boundary of the belt edge, and determine the degree of belt deviation based on the average value.

[0017] According to a second aspect of the present invention, 5. A machine vision-based belt misalignment detection device is provided, comprising:

[0018] The instance segmentation module is used to obtain the belt mask in the image through the instance segmentation model;

[0019] An edge point extraction module is used to extract edge points in the belt mask;

[0020] The anomaly point determination module compares the positional relationship between each edge point and a pre-set non-deviation boundary of the belt edge to determine whether the edge point is an anomaly point.

[0021] The belt misalignment detection module determines whether the belt is misaligned based on the number of abnormal points.

[0022] Furthermore, the edge point extraction module is specifically used for:

[0023] Remove the four corner portions of the belt mask;

[0024] Sampling of edge points is performed uniformly on both sides of the remaining belt mask.

[0025] Furthermore, the deviation detection module is specifically used for:

[0026] If the number of edge points on the side of the belt edge that is not the deviation boundary is greater than the preset threshold, then the belt is determined to be deviating to that side.

[0027] Furthermore, the device also includes:

[0028] The belt misalignment determination module is used to calculate the average distance between the abnormal point and the non-misalignment boundary of the belt edge, and to determine the degree of belt misalignment based on the average value.

[0029] According to a third aspect of the present invention, a terminal device is provided, comprising:

[0030] Processor; and

[0031] A memory that stores executable code, which, when executed by the processor, causes the processor to perform the method described above.

[0032] According to a fourth aspect of the present invention, a non-transitory machine-readable storage medium is provided, on which executable code is stored, which, when executed by a processor of an electronic device, causes the processor to perform the method described above.

[0033] The technical solution provided by the embodiments of the present invention determines belt misalignment by using edge sampling. The obstruction of some points will not affect the determination of misalignment, and can effectively mitigate the impact of random obstruction.

[0034] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit the invention. Attached Figure Description

[0035] The above and other objects, features and advantages of the present invention will become more apparent from the more detailed description of exemplary embodiments of the invention in conjunction with the accompanying drawings, wherein the same reference numerals generally represent the same components in the exemplary embodiments of the invention.

[0036] Figure 1 This is a schematic flowchart illustrating a machine vision-based belt misalignment detection method according to an exemplary embodiment of the present invention.

[0037] Figure 2 This is a structural block diagram of a machine vision-based belt misalignment detection device according to an exemplary embodiment of the present invention.

[0038] Figure 3 This is a schematic diagram of the structure of a computing device according to an exemplary embodiment of the present invention. Detailed Implementation

[0039] Preferred embodiments of the invention will now be described in more detail with reference to the accompanying drawings. While preferred embodiments of the invention are shown in the drawings, it should be understood that the invention can be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that the invention will be thorough and complete, and will fully convey the scope of the invention to those skilled in the art.

[0040] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. The singular forms “a,” “the,” and “the” used in this invention and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0041] It should be understood that although the terms "first," "second," "third," etc., may be used in this invention to describe various information, this information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this invention, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Thus, features defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0042] This invention acquires real-time video of a belt using a fixed camera; determines the non-deviation boundaries of the belt's left and right edges using a two-point method; detects the actual edges of the belt using an instance segmentation algorithm; quickly determines whether the belt is deviating by uniformly sampling points along the edges of the belt segmentation mask; quantifies the degree of belt deviation through pixel analysis; and adapts to random occlusion in the scene through relatively dense sampling. The technical solutions of embodiments of this invention are described in detail below with reference to the accompanying drawings.

[0043] Figure 1 This is a schematic flowchart illustrating a machine vision-based belt misalignment detection method according to an exemplary embodiment of the present invention.

[0044] See Figure 1 The method includes:

[0045] 110. Obtain the belt mask in the image using the instance segmentation model.

[0046] Specifically, in one embodiment, images of the belt area need to be acquired in real time using a camera fixed above the belt. A mask for the belt area is then obtained using a real-time instance segmentation model trained for the specific scenario, such as YOLACT.

[0047] 120. Extract the edge points in the belt mask.

[0048] Specifically, this step extracts edge points based on the belt mask obtained in the previous step, which can be achieved using existing edge analysis methods.

[0049] In practical implementation, it was found that if the belt area is large, analyzing the position of each point on the belt edge individually is time-consuming and cannot meet the requirements for real-time detection. Therefore, this invention proposes uniform sampling of the belt edge, which can significantly reduce the judgment time without affecting the accuracy of the judgment. Furthermore, by uniformly sampling multiple points, unless most points are obscured, the obscuration of some points will not affect the judgment of belt deviation. However, judging belt deviation by identifying idlers through target detection will result in false judgments if any idler is obscured. In addition, it was also found in practical implementation that the segmentation effect at the four corners of the belt is relatively poor. By removing the upper and lower parts of the mask before sampling, the judgment accuracy can be effectively improved.

[0050] Based on the above implementation results, the edge point extraction is further optimized. In one embodiment, this step includes:

[0051] 1201. Remove the four corner portions of the belt mask;

[0052] 1202. Sample edge points evenly on both sides of the remaining belt mask.

[0053] Specifically, first, remove the edge points (e.g., 1 / 10) of the mask that are prone to inaccuracy. Then, take multiple (e.g., 30) points evenly on both sides of the belt mask to determine the belt deviation.

[0054] 130. Compare the positional relationship between each edge point and the pre-set non-deviation boundary of the belt edge to determine whether the edge point is an abnormal point.

[0055] Specifically, in one embodiment, two points are set on the left and right sides of the belt respectively to determine two straight lines, which represent the non-deviation boundaries of the left and right edges of the belt.

[0056] Set a deviation threshold (e.g., 5px, meaning that a point on the belt edge that exceeds the normal boundary by more than 5 pixels is considered an abnormal point of deviation) to determine whether the belt is deviating.

[0057] 140. Determine whether the belt is misaligned based on the number of abnormal points.

[0058] Specifically, this step includes: if the number of edge points on the side of the belt edge that is not the deviation boundary is greater than a preset threshold, then it is determined that the belt is deviating to that side.

[0059] Specifically, if the number of edge points located to the left of the non-deviation boundary of the belt exceeds a preset threshold, the belt is determined to be deviating to the left; if the number of edge points located to the right of the non-deviation boundary of the belt exceeds a preset threshold, the belt is determined to be deviating to the right.

[0060] This invention provides a machine vision-based belt misalignment detection method that uses edge sampling to determine belt misalignment. Occlusion of some points will not affect the determination of misalignment, effectively mitigating the impact of random occlusion.

[0061] Optionally, in this embodiment, the method further includes:

[0062] 150. Calculate the average distance between the abnormal point and the non-deviation boundary of the belt edge, and determine the degree of belt deviation based on the average value.

[0063] Based on the above, it can be seen that the greater the distance between the edge point and the non-deviation boundary of the belt edge, the more severe the deviation. Considering that a single distance can easily lead to misjudgment, this step needs to consider the average distance of all abnormal points. In addition, considering the perspective relationship of the camera, the closer to the belt, the greater the distance. Therefore, when calculating the average, the width of the belt on the horizontal line where the abnormal point is located is taken into account, which can eliminate the problem of excessive pixel distance caused by perspective. In summary, the average distance between the abnormal point and the non-deviation boundary of the belt edge is the sum of the pixel distances between the abnormal point and the straight line to the left of the non-deviation boundary of the belt edge, divided by the sum of the number of abnormal points and the width of each belt.

[0064] In addition, the degree of deviation can also be measured by the actual distance. Specifically, the pixel distance between each abnormal point and the non-deviation boundary of the belt edge is converted into the actual distance according to the pre-calibrated mapping function and the average value is calculated. This can also overcome the image distortion problem caused by perspective.

[0065] This invention provides a belt misalignment detection method based on instance segmentation. It accurately determines whether the belt is misaligned based on the relationship between the belt edge and a set non-misalignment boundary of the belt edge. The degree of belt misalignment is quantified by sampling the distance from the edge point to the set non-misalignment boundary of the belt edge. The method of sampling edge points based on instance segmentation mask can significantly reduce the judgment time while ensuring the accuracy of the judgment, and realize real-time detection. In addition, for sampling points with dense belt mask, by setting an appropriate threshold, the influence of occlusion on vision-based belt misalignment judgment can be effectively mitigated.

[0066] In one embodiment, the specific implementation of the present invention may include the following six steps:

[0067] Step 1: Camera Installation

[0068] The camera needs to be fixedly installed directly above the belt, 2-4 meters from the ground, tilted downwards at a 30-60° angle; ensure the belt is clearly visible to the camera with a resolution greater than 75*75; the scene should be clearly visible, without obvious water mist, dust, overexposure, or underexposure.

[0069] When the camera position and angle change, the belt edge non-deviation boundary in step 2 needs to be reset.

[0070] Step 2: Setting the deviation parameters

[0071] Two points are set on the left and right sides of the belt, respectively, to define two straight lines representing the non-deviation boundaries of the belt's left and right edges. Let the top left corner of the image be the origin, the horizontal axis be the y-axis, and the vertical axis be the x-axis. The two points on the left and right are (x...). left1 ,y left1 ), (x left2 ,y left2 ), (x right1 ,y right11 ), (x right12 ,y right12 The expressions for the two lines on the left and right sides using the two-point method are:

[0072]

[0073] They are respectively denoted as: f left (x) and f right (x).

[0074] In addition, a deviation threshold needs to be set. When the proportion of abnormal sampling points exceeds this threshold, it is determined to be a deviation.

[0075] Step 3: Real-time acquisition of belt mask by instance segmentation model

[0076] The belt segmentation mask is obtained in real time using a real-time instance segmentation model, such as YOLACT. This belt mask is a binary matrix where elements with a value of 1 represent regions that make up the belt, and other regions have a value of 0. Let the image width and height be w and h, and any pixel in the image be p. ij ,i∈[0,w),j∈[0,h), the belt region is represented as R belt Then the belt mask matrix M∈{0,1} w×h It can be represented as:

[0077]

[0078] Step 4: Belt edge sampling

[0079] In practical implementation, it was found that if the belt area is large, analyzing the position of each belt edge point individually results in high time complexity, which cannot meet the requirements for real-time detection. Therefore, this invention proposes uniform sampling of the belt edge, which can significantly reduce the judgment time without affecting the accuracy of the judgment. In addition, it was also found in practical implementation that the segmentation effect at the four corners of the belt is relatively poor. By removing the upper and lower rows of the mask before sampling, the judgment accuracy can be effectively improved. Let the upper and lower range of the belt be [u,d], the number of sampling points be n_all, and the number of upper and lower discard rows be n_discard, then the sampling row index can be represented as:

[0080] Inde_sp[i]=u+n_discard+i×(du-2*n_discard) / n_all,i∈[0,n_all)

[0081] That is, after discarding several rows above and below, n_all points are sampled uniformly.

[0082] Step 5: Determine if the belt is misaligned.

[0083] After obtaining the sampling row index of the belt edge, calculate the relationship between the left and right edges of the belt in each row and the left and right straight lines in step 2. If the edge point of the belt is outside the straight line, it is judged as an anomaly. Then, count the proportion of anomalies. If it exceeds the threshold set in step 2, it is judged as deviation. Let the left and right edge points of the i-th row be (x... i ,y il ), (x i ,y ir Then the ordinate value of the line corresponding to these two points is: f left (x i ) and f right (x i Let count_l be the number of left edge points outside the line and count_r be the number of right edge points outside the line. Then, for any sampling point row index, we have:

[0084]

[0085] Let the belt misalignment threshold set in step 2 be δ. Then the final belt misalignment situation is as follows:

[0086]

[0087] Step 6: Quantify the degree of deviation

[0088] This invention represents the degree of belt misalignment by calculating the average distance between the actual edge ordinate and the ordinate on the calibration line. Let sum_l be the left misalignment distance accumulator, sum_r be the right misalignment distance accumulator, and sum_b be the belt width accumulator. Then, for any sampling point (x...i ,y il ), (x i ,y ir )have:

[0089]

[0090] sum_b+=(f right (x i )-f left (x i ))

[0091] The final degree of deviation can then be expressed as:

[0092]

[0093] Where n_all is the number of sampling points.

[0094] Figure 2 This is a structural block diagram of a machine vision-based belt misalignment detection device according to an exemplary embodiment of the present invention.

[0095] See Figure 2 The system includes:

[0096] The instance segmentation module is used to obtain the belt mask in the image through the instance segmentation model;

[0097] An edge point extraction module is used to extract edge points in the belt mask;

[0098] The anomaly point determination module compares the positional relationship between each edge point and a pre-set non-deviation boundary of the belt edge to determine whether the edge point is an anomaly point.

[0099] The belt misalignment detection module determines whether the belt is misaligned based on the number of abnormal points.

[0100] Optionally, in this embodiment, the edge point extraction module is specifically used for:

[0101] Remove the four corner portions of the belt mask;

[0102] Sampling of edge points is performed uniformly on both sides of the remaining belt mask.

[0103] Optionally, in this embodiment, the deviation detection module is specifically used for:

[0104] If the number of edge points on the side of the belt edge that is not the deviation boundary is greater than the preset threshold, then the belt is determined to be deviating to that side.

[0105] Optionally, in this embodiment, the device further includes:

[0106] The belt misalignment determination module is used to calculate the average distance between the edge point and the non-misalignment boundary of the belt edge, and to determine the degree of belt misalignment based on the average value.

[0107] Regarding the system in the above embodiments, the specific ways in which each module performs operations have been described in detail in the embodiments related to the method, and will not be elaborated further here.

[0108] Figure 3 This is a schematic diagram of the structure of a computing device according to an exemplary embodiment of the present invention.

[0109] See Figure 3 The computing device 300 includes a memory 310 and a processor 320.

[0110] The processor 320 can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor.

[0111] Memory 310 may include various types of storage units, such as system memory, read-only memory (ROM), and permanent storage devices. ROM may store static data or instructions required by the processor 320 or other modules of the computer. Permanent storage devices may be read-write storage devices. Permanent storage devices may be non-volatile storage devices that retain stored instructions and data even when the computer is powered off. In some embodiments, permanent storage devices use mass storage devices (e.g., magnetic or optical disks, flash memory) as permanent storage devices. In other embodiments, permanent storage devices may be removable storage devices (e.g., floppy disks, optical drives). System memory may be a read-write storage device or a volatile read-write storage device, such as dynamic random access memory. System memory may store some or all of the instructions and data required by the processor during operation. Furthermore, memory 310 may include any combination of computer-readable storage media, including various types of semiconductor memory chips (DRAM, SRAM, SDRAM, flash memory, programmable read-only memory), and disks and / or optical disks may also be used. In some embodiments, memory 310 may include a removable storage device that is readable and / or writable, such as a laser disc (CD), a read-only digital multifunction optical disc (e.g., DVD-ROM, dual-layer DVD-ROM), a read-only Blu-ray disc, an ultra-high density optical disc, a flash memory card (e.g., SD card, mini SD card, Micro-SD card, etc.), a magnetic floppy disk, etc. Computer-readable storage media do not contain carrier waves or transient electronic signals transmitted wirelessly or via wired connections.

[0112] The memory 310 stores executable code, which, when processed by the processor 320, can cause the processor 320 to execute part or all of the methods described above.

[0113] Furthermore, the method according to the present invention can also be implemented as a computer program or computer program product, which includes computer program code instructions for performing some or all of the steps in the above-described method of the present invention.

[0114] Alternatively, the present invention can also be implemented as a non-transitory machine-readable storage medium (or computer-readable storage medium, or machine-readable storage medium) storing executable code (or computer program, or computer instruction code) that, when executed by a processor of an electronic device (or computing device, server, etc.), causes the processor to perform some or all of the steps of the method described above according to the present invention.

[0115] The present invention has been described in detail above with reference to the accompanying drawings. In the above embodiments, the descriptions of each embodiment have their own emphasis; for parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments. Those skilled in the art should also understand that the actions and modules involved in the specification are not necessarily essential to the present invention. Furthermore, it is understood that the steps in the method of the embodiments of the present invention can be adjusted, combined, and deleted according to actual needs, and the modules in the device of the embodiments of the present invention can be combined, divided, and deleted according to actual needs.

[0116] Those skilled in the art will also understand that the various exemplary logic blocks, modules, circuits, and algorithm steps described in connection with the disclosure herein can be implemented as electronic hardware, computer software, or a combination of both.

[0117] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems and methods according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0118] The various embodiments of the present invention have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or improvement of the technology in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A machine vision-based method for detecting belt misalignment, characterized in that, include: Obtain the belt mask in the image using an instance segmentation model; Extracting edge points from the belt mask; specifically including: removing the four corner portions of the belt mask; uniformly sampling edge points on both sides of the remaining belt mask; Compare the positional relationship between each edge point and the pre-set non-deviation boundary of the belt edge to determine whether the edge point is an abnormal point; Determine if the belt is misaligned based on the number of abnormal points; Specifically, determining whether the belt is misaligned based on the number of abnormal points includes: If the number of edge points on the side of the belt edge that is not the deviation boundary is greater than the preset threshold, then the belt is determined to be deviating to that side.

2. The method according to claim 1, characterized in that, Also includes: Calculate the average distance between the abnormal point and the non-deviation boundary of the belt edge, and determine the degree of belt deviation based on the average value.

3. A machine vision-based belt misalignment detection device, characterized in that, include: The instance segmentation module is used to obtain the belt mask in the image through the instance segmentation model; The edge point extraction module is used to extract edge points in the belt mask; specifically, it is used to: remove the four corner portions of the belt mask; and uniformly sample edge points on both sides of the remaining belt mask. The anomaly point determination module compares the positional relationship between each edge point and a pre-set non-deviation boundary of the belt edge to determine whether the edge point is an anomaly point. The belt misalignment detection module determines whether the belt is misaligned based on the number of abnormal points. The deviation detection module is specifically used for: If the number of edge points on the side of the belt edge that is not the deviation boundary is greater than the preset threshold, then the belt is determined to be deviating to that side.

4. The apparatus according to claim 3, characterized in that, Also includes: The belt misalignment determination module is used to calculate the average distance between the abnormal point and the non-misalignment boundary of the belt edge, and to determine the degree of belt misalignment based on the average value.

5. A terminal device, characterized in that, include: processor; as well as A memory having executable code stored thereon, which, when executed by the processor, causes the processor to perform the method as described in claim 1 or 2.

6. A non-transitory machine-readable storage medium having executable code stored thereon, characterized in that, When the executable code is executed by the processor of the electronic device, the processor performs the method as described in claim 1 or 2.

Images