A belt deviation detection method and device based on multi-task learning

Abstract

The application provides a belt deviation detection method and device based on multi-task learning, and relates to the field of artificial intelligence; the method comprises the following steps: obtaining a plurality of input images with label information and dividing the images into training set images and test set images; constructing a target detection model based on multi-task learning and training the model by using the training set images and the test set images; obtaining real-time images of belt operation and inputting the images into the target detection model to obtain output images; calculating left and right edge straight line equations of the belt according to targets labeled in the output images, determining the position of a drag roller, and then determining left and right early warning lines of the belt and left and right warning lines of the belt; when the included angle between the left and right early warning lines of the belt and the corresponding side edge straight line equations of the belt does not exceed an included angle threshold value, calculating the distance between the left and right edges of the belt and the corresponding side early warning lines, and then determining the deviation direction of the belt; the application can accurately detect the deviation direction and angle of the belt by simultaneously detecting the drag roller and the belt edge.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, specifically to a method and apparatus for detecting belt misalignment based on multi-task learning. Background Technology

[0002] Belt conveyors are a primary mode of transportation in coal mines. Belt misalignment can lead to material spillage, equipment damage, and even more serious safety accidents, posing a significant safety hazard. Timely and accurate detection of belt misalignment is crucial for accident prevention and ensuring safe production.

[0003] Existing technologies mainly rely on the following methods for detecting belt misalignment: 1) Manual periodic inspection: Inspectors periodically check belt conveyor systems. In coal mines, where belt conveyors are used extensively, this increases labor costs. Furthermore, manual periodic inspections cannot monitor belt operation in real time, resulting in low efficiency and difficulty in ensuring safe production. 2) Optical flow-based methods, such as the belt misalignment detection method proposed in patent application CN117699375A. This method is greatly affected by the environment; when light intensity changes, the detection rate cannot be guaranteed. This is due to the inherent defects of the optical flow method. In coal mines, where the production environment is harsh, this method has poor robustness. 3) Target detection methods: This method determines belt misalignment by detecting the belt rollers. However, this method is prone to missed detections when the rollers are obstructed. Additionally, it cannot determine the direction of belt misalignment. 4) Image segmentation-based methods: such as the machine vision-based belt misalignment detection method proposed in patent application CN117566378A. This method can accurately obtain the edge contour of the belt, but the computational load is relatively large. Furthermore, this method requires pre-setting the belt misalignment boundary, which is quite cumbersome. 5) Based on traditional image processing algorithms, such as the vision-based belt misalignment detection method proposed in patent CN116573366B, which uses the Sobel operator for image edge detection. This method is limited by the environment, has poor generalization ability, and low accuracy, making it difficult to meet production needs.

[0004] Therefore, there is a need to propose a method that can detect belt misalignment in real time without being limited by the environment, so as to improve the detection efficiency of belt misalignment and ensure safe production. Summary of the Invention

[0005] The purpose of this invention is to provide a belt misalignment detection method and device based on multi-task learning. By optimizing the target detection model through a joint loss function, it can simultaneously detect the rollers and belt edges. Furthermore, by calculating the belt edge straight line, warning line, and alarm line, it can accurately detect the belt misalignment direction and angle, providing strong support for ensuring safe production and preventing accidents in enterprises.

[0006] To achieve the above objectives, the present invention proposes the following technical solution:

[0007] Firstly, a belt misalignment detection method based on multi-task learning is proposed, including:

[0008] Acquire several input images with annotation information and divide the input images into training set images and test set images. The input images are captured images of the belt running process, and the annotation information includes the roller, the left side of the belt, and the right side of the belt.

[0009] A target detection model based on multi-task learning is constructed, and the model is trained using the training set images and the test set images; wherein, the target detection model is an improved YOLO model containing a belt edge detection head composed of two fully connected layer networks, and the loss function of the improved YOLO model is a joint loss function of the YOLO model loss function and the belt edge detection loss function;

[0010] A real-time image of the belt running is acquired and input into the target detection model based on multi-task learning to obtain an output image, which is a real-time image after target detection and annotation.

[0011] Based on the targets marked in the output image, calculate the straight line equations of the left and right edges of the belt, determine the position of the drag roller, and then determine the left warning line, left alarm line, right warning line, and right alarm line of the belt.

[0012] Calculate and determine whether the angle between the left warning line of the belt and the straight line equation of the left edge of the belt, and the angle between the right warning line of the belt and the straight line equation of the right edge of the belt do not exceed the angle threshold. When the angles do not exceed the angle threshold, calculate the distance between the two edges of the belt and the corresponding side warning lines respectively, and then determine the belt deviation direction based on the relationship between the distance and the distance threshold.

[0013] Furthermore, the process of calculating the equations of the straight lines on the left and right edges of the belt based on the targets marked in the output image, determining the position of the drag roller, and then determining the left warning line, left alarm line, right warning line, and right alarm line of the belt is as follows:

[0014] Based on the center coordinates of several target boxes of several rollers in the output image, the detected rollers are divided into rollers on the same side and rollers on different sides.

[0015] Based on the output image of the belt edge detection head, calculate the left edge point set and the right edge point set of the belt, and establish the straight line equations of the left edge and the right edge of the belt by fitting the left edge point set and the right edge point set of the belt using the least squares method.

[0016] Calculate the distances between the center of any one of the rollers on the same side and the straight lines of the left and right edges of the belt, respectively, and determine whether the roller is located on the left or right side of the belt based on the distances.

[0017] For the several rollers on the left side of the belt, select the center points of any two rollers on the same side to establish two straight line equations. The two straight lines formed are respectively denoted as the left side warning line and the right side warning line of the belt. The left side warning line of the belt is moved to the left along the X-axis by a distance threshold to obtain the left side alarm line of the belt, and the right side warning line of the belt is moved to the right along the X-axis by a distance threshold to obtain the right side alarm line of the belt.

[0018] Furthermore, the process of calculating the distances between the two edges of the belt and the corresponding side warning lines when the included angles do not exceed the included angle threshold, and then determining the belt deviation direction based on the relationship between the distances and the distance thresholds, is as follows:

[0019] Calculate the distances from the center point of the warning line on the left side of the belt and the center point of the warning line on the right side of the belt to the corresponding edge of the belt, denoted as dwarnL and dwarnR respectively. Then, when dwarnL < 0 and |dwarnL| < L, a warning is issued for the left side of the belt to deviate; where L represents the distance threshold. When dwarnL < 0 and |dwarnL| > L, a warning is issued for the left side of the belt to deviate. When dwarnR < 0 and |dwarnR| < L, a warning is issued for the right side of the belt to deviate. When dwarnR < 0 and |dwarnR| > L, a warning is issued for the right side of the belt to deviate.

[0020] Furthermore, the loss function of the improved YOLO model is calculated as follows:

[0021] belt edge similarity loss function L sim The calculation formula is:

[0022]

[0023] Location loss function L shp The calculation formula is:

[0024] Prob i，j,： =softmax(P i,j，1：w )

[0025]

[0026]

[0027]

[0028] Classification loss function Lcls The calculation formula is:

[0029]

[0030] Where C represents the left and right edges of the belt, C = 2; h represents the anchor frame parameters; P i,j，： Let P be the (w+1)-dimensional vector output by the model, where w = 200, representing the predicted value at the i-th belt edge and the j-th column; i，j+1,： Q represents the predicted value of the (j+1)th column at the i-th belt edge; i，j，： This represents the i-th belt edge, with the j-th column being a uniquely hot-coded label; k is an integer representing the position index; L CE Prob represents the cross-entropy loss function; i，j，： Loc represents the probability value at different positions, ranging from 1 to w. i，j Indicates the location where the belt edge exists; P i,j，1：w The w-dimensional vector output by the model represents the probability of the presence of a belt edge; Prob i，j,k P represents the probability value at the k-th position. i，j，k This represents the probability that the belt edge exists at the model's predicted output value at the k-th position.

[0031] Loss function L for belt edge detection line The calculation formula is:

[0032] L line =L shp +βL cls +L sim

[0033] Where β represents a hyperparameter used to balance the weights of the class loss function in the loss function for belt edge detection;

[0034] The total loss function L of the improved YOLO model total The calculation formula is:

[0035] L total =L obj +αL line

[0036] Among them, L obj This is the loss function for the original YOLO series object detectors.

[0037] Furthermore, the process of calculating the angle between the left warning line of the belt and the equation of the left edge of the belt, and the angle between the right warning line of the belt and the equation of the right edge of the belt, is as follows:

[0038] Let the equation of the warning line on the left or right side of the belt be y = k1x + b1, and the equation of the corresponding belt edge line be y = k2x + b2.

[0039] Then, the angle θ between the warning line and the straight line equation of the corresponding side of the belt edge is calculated as follows:

[0040] tanθ=|(k2-k1) / (1+k1*k2)|

[0041] θ=(tan -1 (tanθ))*(180 / π)

[0042] Where k1 and k2 are the slopes of the two lines forming the angle between the lines to be determined.

[0043] Furthermore, it also includes:

[0044] It provides feedback on the direction of belt misalignment and returns the belt misalignment angle, displays and saves belt misalignment warning or alarm images for data traceability.

[0045] Secondly, a belt misalignment detection device based on multi-task learning is proposed, comprising:

[0046] The acquisition module is used to acquire several input images with annotation information and divide the input images into training set images and test set images. The input images are captured images of the belt running process, and the annotation information includes the roller, the left side of the belt, and the right side of the belt.

[0047] A training module is constructed to build a target detection model based on multi-task learning. The model is trained using the training set images and the test set images. The target detection model is an improved YOLO model that includes a belt edge detection head composed of two fully connected layers. The loss function of the improved YOLO model is a joint loss function of the YOLO model loss function and the belt edge detection loss function.

[0048] The detection module is used to acquire real-time images of the belt running and input them into the target detection model based on multi-task learning to obtain an output image, which is a real-time image after target detection and annotation.

[0049] The calculation and determination module is used to calculate the straight line equations of the left edge and the right edge of the belt based on the targets marked in the output image, determine the position of the drag roller, and then determine the left warning line, the left alarm line, the right warning line, and the right alarm line of the belt.

[0050] The calculation and judgment module is used to calculate and judge whether the angle between the left warning line of the belt and the straight line equation of the left edge of the belt, and the angle between the right warning line of the belt and the straight line equation of the right edge of the belt do not exceed the angle threshold. When the angles do not exceed the angle threshold, the distances between the two edges of the belt and the corresponding side warning lines are calculated respectively, and then the belt deviation direction is determined according to the relationship between the distance and the distance threshold.

[0051] Furthermore, the calculation and determination module calculates the straight line equations of the left and right edges of the belt, determines the position of the roller, and then determines the execution unit for the left warning line, left alarm line, right warning line, and right alarm line of the belt, including:

[0052] The segmentation unit is used to segment the detected rollers into rollers on the same side and rollers on different sides based on the center coordinates of the target boxes of several rollers in the output image.

[0053] The first calculation unit is used to calculate the left edge point set and the right edge point set of the belt based on the output image of the belt edge detection head, and to fit and establish the straight line equation of the left edge of the belt and the straight line equation of the right edge of the belt based on the left edge point set and the right edge point set of the belt using the least squares method.

[0054] The second calculation unit is used to calculate the distance between the center of any one of the rollers on the same side and the straight line between the left edge and the right edge of the belt, respectively, and determine whether the roller is located on the left or right side of the belt based on the distance.

[0055] The unit is used to establish two straight line equations for the center points of any two rollers on the same side of the belt, for a plurality of rollers on the left side of the belt. The two straight lines are denoted as the left side warning line and the right side warning line of the belt, respectively. The left side warning line is moved to the left along the X-axis by a threshold distance to obtain the left side alarm line of the belt, and the right side warning line is moved to the right along the X-axis by a threshold distance to obtain the right side alarm line of the belt.

[0056] Furthermore, the calculation and judgment module calculates the distance between the two edges of the belt and the corresponding side warning line, and then determines the direction of belt deviation based on the relationship between the distance and the distance threshold.

[0057] Calculate the distances from the center point of the warning line on the left side of the belt and the center point of the warning line on the right side of the belt to the corresponding edge of the belt, denoted as dwarnL and dwarnR respectively. Then, when dwarnL < 0 and |dwarnL| < L, a warning is issued for the left side of the belt to deviate; where L represents the distance threshold. When dwarnL < 0 and |dwarnL| > L, a warning is issued for the left side of the belt to deviate. When dwarnR < 0 and |dwarnR| < L, a warning is issued for the right side of the belt to deviate. When dwarnR < 0 and |dwarnR| > L, a warning is issued for the right side of the belt to deviate.

[0058] Thirdly, an electronic device is proposed, comprising a computer program stored in a computer-readable storage medium; when the processor of the electronic device reads the computer program from the computer-readable storage medium, the processor executes the computer program, causing the electronic device to perform the steps of the belt misalignment detection method based on multi-task learning described above.

[0059] As can be seen from the above technical solutions, the technical solutions of the present invention have achieved the following beneficial effects:

[0060] This invention discloses a belt misalignment detection method and apparatus based on multi-task learning. The method includes: acquiring several input images with labeled information and dividing the input images into training set images and test set images; constructing a target detection model based on multi-task learning, and training the model using the training set images and test set images; acquiring real-time images of belt operation and inputting them into the target detection model to obtain output images; calculating the straight line equations of the left and right edges of the belt according to the targets labeled in the output images, determining the position of the drag roller, and then determining the left warning line, left alarm line, right warning line, and right alarm line of the belt; calculating and judging whether the angle between the left and right warning lines of the belt and the corresponding side belt edge straight line equations does not exceed the angle threshold, and when it does not exceed the angle threshold, calculating the distance between the two sides of the belt edge and the corresponding side warning line, and then determining the belt misalignment direction according to the relationship between the distance and the distance threshold; this invention accurately detects the belt misalignment direction and angle by simultaneously detecting the drag roller and the belt edge, calculating the belt edge straight line, warning line, and alarm line; compared with the prior art, it is not limited by environmental changes, the method is simple, has high detection accuracy, and can fully meet production needs.

[0061] The target detection model of this invention is an improved YOLO model. It adds a belt edge detection head composed of two fully connected layers to the original network structure, and introduces a belt edge detection loss function into the model's loss function. By using the joint loss function to optimize network learning, it can simultaneously detect the edges of the rollers and belts, thereby accurately obtaining the belt edge straight line, warning line and alarm line. Compared with the prior art, it can not only detect belt deviation, but also determine the direction and angle of belt deviation.

[0062] It should be understood that all combinations of the foregoing concepts and the additional concepts described in more detail below can be considered part of the inventive subject matter of this disclosure, provided that such concepts do not contradict each other.

[0063] The foregoing and other aspects, embodiments, and features of the teachings of the present invention will be more fully understood from the following description in conjunction with the accompanying drawings. Other additional aspects of the invention, such as features and / or beneficial effects of exemplary embodiments, will become apparent from the following description or may be learned through practice of specific embodiments according to the teachings of the present invention. Attached Figure Description

[0064] The accompanying drawings are not drawn to scale. In the drawings, each identical or nearly identical component shown in the various figures may be denoted by the same reference numeral. For clarity, not every component is labeled in each figure. Embodiments of various aspects of the invention will now be described by way of example and with reference to the accompanying drawings, wherein:

[0065] Figure 1 This is a flowchart of the belt misalignment detection method based on multi-task learning of the present invention;

[0066] Figure 2 This is a flowchart illustrating the calculation of the straight lines on both sides of the belt edge, the warning lines on both sides, and the alarm lines of the present invention.

[0067] Figure 3 This is a flowchart of the method for determining the direction of belt misalignment according to the present invention;

[0068] Figure 4 This is a flowchart of the belt misalignment alarm process of the present invention;

[0069] Figure 5 This is a structural block diagram of the belt misalignment detection device based on multi-task learning according to the present invention;

[0070] Figure 6 A structural block diagram of an electronic device is provided for this invention. Detailed Implementation

[0071] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the described embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention. Unless otherwise defined, the technical or scientific terms used herein should have the ordinary meaning understood by those skilled in the art.

[0072] The terms "first," "second," and similar words used in the specification and claims of this patent application do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Similarly, unless the context clearly indicates otherwise, the singular forms of "an," "a," or "the," etc., do not indicate a quantity limitation, but rather indicate the presence of at least one. Terms such as "comprising" or "including" mean that the element or object preceding "comprising" encompasses the features, wholes, steps, operations, elements, and / or components listed following "comprising" or "including," and do not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components, and / or collections thereof. Terms such as "upper," "lower," "left," and "right" are used only to indicate relative positional relationships; when the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0073] Existing methods for detecting belt misalignment, including manual periodic inspection, optical flow methods, target detection methods, image segmentation methods, and traditional image processing methods, suffer from technical problems such as high labor costs, low inspection efficiency, susceptibility to environmental influences, high false negative rates, and low accuracy. Therefore, this invention aims to develop a new method for detecting belt misalignment. This method improves the target detection method by simultaneously detecting the belt and the roller to obtain the edge line, warning line, and alarm line for determining belt misalignment, thereby efficiently and accurately determining the direction of belt misalignment.

[0074] The belt misalignment detection method and apparatus based on multi-task learning disclosed in this invention will be further described in detail below with reference to the accompanying drawings.

[0075] Combination Figure 1 As shown in the figure, the belt misalignment detection method based on multi-task learning disclosed in this invention includes the following steps:

[0076] Step S102: Obtain several input images with annotation information and divide the input images into training set images and test set images. The input images are images of the belt running process, and the annotation information includes the roller, left side of the belt, and right side of the belt.

[0077] The acquired images of the conveyor belt operation process are then processed, i.e., data annotation is performed. For example, labelme annotation software can be used to complete the data annotation and generate a JSON file for model training.

[0078] Step S104: Construct a target detection model based on multi-task learning, and train the model using the training set images and the test set images; wherein, the target detection model is an improved YOLO model containing a belt edge detection head composed of two fully connected layer networks, and the loss function of the improved YOLO model is a joint loss function of the YOLO model loss function and the belt edge detection loss function;

[0079] In this case, the improved YOLO model retains the original network structure, only adding two fully connected layers to the original YOLO model network's neck structure to form a belt edge detection head for belt edge detection. Its input-output order is 1024->2048->7200. The labels obtained from the generated JSON file of the completed data annotation are converted to YOLO format; the improved YOLO model input data is resized to 640, 640 and normalized; during training, mosaic and Mixup data augmentation, which are unsuitable for belt edge detection, are disabled. During model training, the preset number of row anchor boxes is 18, and the sampling points are 200. The row anchor boxes are as follows:

[0080] [140,168,196,224,252,280,308,336,364,392,420,448,486,514,542,580,608,636].

[0081] To achieve belt edge detection, the solution incorporates a belt edge detection loss function into the improved YOLO model's loss function. By combining the loss functions, the network learning is optimized, enabling simultaneous detection of both the roller and belt edges. Specifically, the calculation process of the improved YOLO model's loss function is as follows:

[0082] belt edge similarity loss function L sim The calculation formula is:

[0083]

[0084] Location loss function L shp The calculation formula is:

[0085] Prob i，j，： =softmax(P i,j，1：w )

[0086]

[0087]

[0088]

[0089] Classification loss function L cls The calculation formula is:

[0090]

[0091] Where C represents the left and right edges of the belt, C = 2; h represents the anchor frame parameters; P i，j，： Let P be the (w+1)-dimensional vector output by the model, where w = 200, representing the predicted value at the i-th belt edge and the j-th column; i，j+1，： Q represents the predicted value of the (j+1)th column at the i-th belt edge; i，j，: This represents the i-th belt edge, with the j-th column being a uniquely hot-coded label; k is an integer representing the position index; L CE Prob represents the cross-entropy loss function; i，j，： Loc represents the probability value at different positions, ranging from 1 to w. i，j Indicates the location where the belt edge exists; P i,j,1：w The w-dimensional vector output by the model represents the probability of the presence of a belt edge; Prob i，j，k P represents the probability value at the k-th position. i，j，k This represents the probability that the belt edge exists at the model's predicted output value at the k-th position.

[0092] Loss function L for belt edge detection line The calculation formula is:

[0093] L line =L shp +βL cls +L sim

[0094] Where β represents a hyperparameter used to balance the weights of the class loss function in the loss function for belt edge detection;

[0095] The total loss function L of the improved YOLO model total The calculation formula is:

[0096] L total =L obj +αL line

[0097] Among them, L obj This is the loss function for the original YOLO series object detectors.

[0098] Step S106: Acquire a real-time image of the belt running and input it into the target detection model based on multi-task learning to obtain an output image. The output image is a real-time image after target detection and annotation. Multiple roller target boxes and belt target boxes are clearly annotated in the output image.

[0099] Step S108: Based on the target marked in the output image, calculate the straight line equation of the left edge of the belt and the straight line equation of the right edge of the belt, determine the position of the drag roller, and then determine the left warning line, the left alarm line, the right warning line, and the right alarm line of the belt.

[0100] The specific implementation process is as follows: Figure 2 As shown, firstly, based on the center coordinates of several target boxes of several rollers in the output image, the detected rollers are divided into rollers on the same side and rollers on different sides, that is, whether the rollers are on the same side is determined according to the magnitude of the center coordinate x; then, based on the output image of the belt edge detection head, that is, based on the divided rollers on the same side or different sides, the left edge point set and the right edge point set of the belt are calculated from the input image, and the left edge line equation and the right edge line equation of the belt are established by fitting the left edge point set and the right edge line equation of the belt based on the least squares method; then, the distance d between the center of any roller on the same side and the left edge line of the belt and the corresponding right edge line of the belt are calculated respectively. iL and d iR The location of the roller on the left or right side of the belt is determined based on the distance; specifically, when d iR >d iL If the condition is met, it indicates that the side roller is the right side roller; otherwise, the side roller is the right side roller. Finally, for several rollers on the left side of the belt, select the center points of any two rollers on the same side to establish two straight line equations. The two straight lines formed are denoted as the left side warning line and the right side warning line of the belt, respectively. The left side warning line of the belt moves a threshold distance L to the left along the X-axis to obtain the left side alarm line of the belt, and the right side warning line of the belt moves a threshold distance L to the right along the X-axis to obtain the right side alarm line of the belt.

[0101] Step S110: Calculate and determine whether the angle between the left warning line of the belt and the straight line equation of the left edge of the belt, and the angle between the right warning line of the belt and the straight line equation of the right edge of the belt do not exceed the angle threshold. When the angles do not exceed the angle threshold, calculate the distance between the two edges of the belt and the corresponding side warning line respectively, and then determine the belt deviation direction according to the relationship between the distance and the distance threshold.

[0102] Combination Figure 3As shown in the figure, the process of determining the belt deviation direction based on the relationship between the distance and the distance threshold is as follows: Calculate the distances from the center points of the left and right warning lines of the belt to the straight lines of the corresponding side belt edges, denoted as dwarnL and dwarnR respectively. Then, when dwarnL < 0 and |dwarnL| < L, it indicates a warning of left belt deviation; where L represents the distance threshold. When dwarnL < 0 and |dwarnL| > L, it indicates an alarm of left belt deviation. When dwarnR < 0 and |dwarnR| < L, it indicates a warning of right belt deviation. When dwarnR < 0 and |dwarnR| > L, it indicates an alarm of right belt deviation.

[0103] In the implementation process of the above step S108, the process of calculating the angles between the left warning line of the belt and the straight line equation of the left edge of the belt, and the angle between the right warning line of the belt and the straight line equation of the right edge of the belt is as follows:

[0104] Denote the straight line equation of the left warning line or the right warning line of the belt as: y = k1x + b1, and the straight line equation of the corresponding side belt edge as: y = k2x + b2;

[0105] Then, the angle θ between the warning line and the straight line equation of the corresponding side belt edge is calculated as:

[0106] tanθ = |(k2 - k1) / (1 + k1 * k2)|

[0107] θ = (tan -1 (tanθ)) * (180 / π)

[0108] Where k1 and k2 are the slopes of the two straight lines for which the included angle is to be determined.

[0109] Combined with Figure 4 As shown in the figure, when the belt deviation detection method based on multi-task learning disclosed in the present invention is applied, it further includes: when it is determined in step S110 that the belt is deviated, feedback the belt deviation direction and return the belt deviation angle, display and save the belt deviation warning or alarm picture for data traceability, and continuously detect the belt running process.

[0110] This invention discloses a belt misalignment detection method based on multi-task learning, which solves the problem of belt misalignment detection. By adding a belt edge detection head to the original network structure of a commonly used target detection algorithm, such as the YOLO model, and introducing a belt edge detection loss function into the loss function, the network learning is optimized through the joint loss function, achieving simultaneous detection of the roller and the belt edge. Furthermore, the warning area and alarm area of ​​belt misalignment are determined by the detected roller, and the direction of belt misalignment is detected by judging whether the straight line of the belt edge is in the warning area or alarm area. Compared with the prior art, the detection method is not only simple and unaffected by the environment, but also has high detection accuracy and low false negative rate.

[0111] In embodiments of this application, an electronic device is also provided, including a computer program stored in a computer-readable storage medium. When the processor of the electronic device reads the computer program from the computer-readable storage medium, the processor executes the computer program, causing the electronic device to perform the steps of the aforementioned belt misalignment detection method based on multi-task learning. Taking an electronic device running on a computer as an example, such as... Figure 6 As shown, the electronic device may include one or more (only one is shown in the figure) processors (processors may include, but are not limited to, microprocessors such as MCUs or programmable logic devices such as FPGAs), a memory for storing data, and a transmission device for communication functions. Those skilled in the art will understand that... Figure 6 The structure shown is for illustrative purposes only and does not limit the structure of the aforementioned electronic device.

[0112] The aforementioned programs may run in a processor or be stored in memory, i.e., in a computer-readable medium. Computer-readable media include both permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. Information may be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transfer medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media do not include temporary computer-readable media such as modulated data signals and carrier waves. These computer programs may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, such that the instructions that execute on the computer or other programmable apparatus provide for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes can be implemented through different modules, corresponding to different method steps.

[0113] In this embodiment, such a device or system is provided, which can be called a belt misalignment detection device based on multi-task learning, such as... Figure 5As shown, the system includes: an acquisition module, used to acquire several input images with annotation information and divide the input images into training set images and test set images, wherein the input images are captured images of the belt running process, and the annotation information includes the roller, left side of the belt, and right side of the belt; a construction and training module, used to construct a target detection model based on multi-task learning, and to train the model using the training set images and the test set images; wherein the target detection model is an improved YOLO model containing a belt edge detection head composed of two fully connected layer networks, and the loss function of the improved YOLO model is a joint loss function of the YOLO model loss function and the belt edge detection loss function; and an acquisition and detection module, used to acquire real-time images of the belt running and input them into the multi-task learning system. The learned target detection model obtains an output image, which is a real-time image after target detection and annotation. A calculation and determination module is used to calculate the equations of the left and right edges of the belt based on the targets annotated in the output image, determine the position of the drag roller, and then determine the left warning line, left alarm line, right warning line, and right alarm line of the belt. A calculation and judgment module is used to calculate and judge whether the angles between the left warning line and the equation of the left edge of the belt, and the angles between the right warning line and the equation of the right edge of the belt, do not exceed an angle threshold. When the angles do not exceed the angle threshold, the distances between the two edges of the belt and the corresponding side warning lines are calculated respectively, and then the belt deviation direction is determined based on the relationship between the distances and distance thresholds.

[0114] The steps of the system for implementing the belt misalignment detection method based on multi-task learning disclosed in the above embodiments have already been described and will not be repeated here.

[0115] For example, the calculation and determination module calculates the straight-line equations of the left and right edges of the belt, determines the positions of the idler rollers, and further determines the execution units of the left warning line, left alarm line, right warning line, and right alarm line of the belt, including: a division unit for dividing the detected idler rollers into same-side idler rollers and different-side idler rollers according to the center coordinates of several idler roller target frames in the output image; a first calculation unit for calculating the left-edge point set and right-edge point set of the belt according to the output image of the belt-edge detection head, and fitting and establishing the straight-line equations of the left and right edges of the belt based on the least squares method according to the left-edge point set and right-edge point set of the belt; a second calculation unit for calculating the distances from the center of any one of the same-side idler rollers to the left-edge straight line and right-edge straight line of the belt respectively, and determining whether the idler roller is on the left or right side of the belt according to the distances; a construction unit for, for several idler rollers on the left side of the belt and several idler rollers on the left side of the belt, respectively selecting the center points of any two same-side idler rollers to establish two straight-line equations, and the two formed straight lines are respectively recorded as the left warning line and right warning line of the belt, and the left warning line of the belt moves leftward along the X-axis by a distance threshold to obtain the left alarm line of the belt, and the right warning line of the belt moves rightward along the X-axis by a distance threshold to obtain the right alarm line of the belt.

[0116] For another example, the process of the calculation and judgment module calculating the distances between the two edges of the belt and the corresponding side warning lines and then determining the belt deviation direction according to the relationship between the distances and the distance threshold is as follows: calculating the distances from the center points of the left warning line and right warning line of the belt to the corresponding side belt-edge straight lines respectively, and recording them as dwarnL and dwarnR; then, when dwarnL < 0 and |dwarnL| < L, it is prompted that the current belt is deviated to the left for warning; where L represents the distance threshold; when dwarnL < 0 and |dwarnL| > L, it is prompted that the current belt is deviated to the left for alarm; when dwarnR < 0 and |dwarnR| < L, it is prompted that the current belt is deviated to the right for warning; when dwarnR < 0 and |dwarnR| > L, it is prompted that the current belt is deviated to the right for alarm.

[0117] For another example, the loss function of the improved YOLO model constructed by the construction and training module is calculated as follows:

[0118] The loss function L of the belt-edge similarity sim The calculation formula is:

[0119]

[0120] The loss function L of the position shp The calculation formula is:

[0121] Prob i，j，：=softmax(P i，j，1：w )

[0122]

[0123]

[0124]

[0125] Classification loss function L cls The calculation formula is:

[0126]

[0127] Where C represents the left and right edges of the belt, C = 2; h represents the anchor frame parameters; P i,j，： Let P be the (w+1)-dimensional vector output by the model, where w = 200, representing the predicted value at the i-th belt edge and the j-th column; i，j+1，： Q represents the predicted value of the (j+1)th column at the i-th belt edge; i，j，： This represents the i-th belt edge, with the j-th column being a uniquely hot-coded label; k is an integer representing the position index; L CE Prob represents the cross-entropy loss function; i，j，： Loc represents the probability value at different positions, ranging from 1 to w. i，j Indicates the location where the belt edge exists; P i，j，1：w The w-dimensional vector output by the model represents the probability of the presence of a belt edge; Prob i，j，k P represents the probability value at the k-th position. i，j，k This represents the probability that the belt edge exists at the model's predicted output value at the k-th position.

[0128] Loss function L for belt edge detection line The calculation formula is:

[0129] L line =L shp +βL cls +L sim

[0130] Where β represents a hyperparameter used to balance the weights of the class loss function in the loss function for belt edge detection;

[0131] The total loss function L of the improved YOLO model total The calculation formula is:

[0132] L total =L obj +αL line

[0133] Among them, L objThis is the loss function for the original YOLO series object detectors.

[0134] For example, the belt misalignment detection device based on multi-task learning disclosed in the embodiments further includes: a feedback storage module, used to provide feedback on the belt misalignment direction and return the belt misalignment angle, and to display and save belt misalignment warning or alarm images for data traceability.

[0135] The belt misalignment detection method and device based on multi-task learning disclosed in the above embodiments of the present invention, through multi-task learning, integrates the original scheme that required two models to detect the roller and the belt separately into a single model for detection, reducing the storage pressure on edge devices. At the same time, this method can achieve the purpose of real-time detection of the belt operation process. Secondly, this method effectively solves the problem that the belt edge cannot be detected when it is obscured by materials. For coal mine scenarios, this method has strong robustness.

[0136] While the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the invention. Those skilled in the art can make various modifications and refinements without departing from the spirit and scope of the invention. Therefore, the scope of protection of the present invention shall be determined by the claims.

Claims

1. A belt misalignment detection method based on multi-task learning, characterized in that, include: Acquire several input images with annotation information and divide the input images into training set images and test set images. The input images are captured images of the belt running process, and the annotation information includes the roller, the left side of the belt, and the right side of the belt. A target detection model based on multi-task learning is constructed, and the model is trained using the training set images and the test set images; wherein, the target detection model is an improved YOLO model containing a belt edge detection head composed of two fully connected layer networks, and the loss function of the improved YOLO model is a joint loss function of the YOLO model loss function and the belt edge detection loss function; A real-time image of the belt running is acquired and input into the target detection model based on multi-task learning to obtain an output image, which is a real-time image after target detection and annotation. Based on the targets marked in the output image, calculate the straight line equations of the left and right edges of the belt, determine the position of the drag roller, and then determine the left warning line, left alarm line, right warning line, and right alarm line of the belt. Calculate and determine whether the angle between the left warning line of the belt and the straight line equation of the left edge of the belt, and the angle between the right warning line of the belt and the straight line equation of the right edge of the belt do not exceed the angle threshold. When the angles do not exceed the angle threshold, calculate the distance between the two edges of the belt and the corresponding side warning line respectively, and then determine the direction of belt deviation based on the relationship between the distance and the distance threshold. The loss function of the improved YOLO model is calculated as follows: belt edge similarity loss function L sim The calculation formula is: Location loss function L shp The calculation formula is: Prob i,j,: =softmax(P i,j,1:w ) Classification loss function L cls The calculation formula is: Where C represents the left and right edges of the belt, C = 2; h represents the anchor frame parameters; P i,j,: Let P be the (w+1)-dimensional vector output by the model, where w = 200, representing the predicted value at the i-th belt edge and the j-th column; i,j+1,: Q represents the predicted value of the (j+1)th column at the i-th belt edge; i,j,: This represents the i-th belt edge, with the j-th column being a uniquely hot-coded label; k is an integer representing the position index; L CE Prob represents the cross-entropy loss function; i,j,: Loc represents the probability value at different positions, ranging from 1 to w. i,j Indicates the location where the belt edge exists; P i,j,1:w The w-dimensional vector output by the model represents the probability of the presence of a belt edge; Prob i,j,k P represents the probability value at the k-th position; i,j,k This represents the probability that the belt edge exists at the model's predicted output value at the k-th position. Loss function L for belt edge detection line The calculation formula is: L line =L shp +βL cls +L sim Where β represents a hyperparameter used to balance the weights of the class loss function in the loss function for belt edge detection.

2. The belt misalignment detection method based on multi-task learning according to claim 1, characterized in that, The process of calculating the equations of the straight lines on the left and right edges of the belt based on the targets marked in the output image, determining the position of the drag roller, and then determining the left warning line, left alarm line, right warning line, and right alarm line of the belt is as follows: Based on the center coordinates of several target boxes of several rollers in the output image, the detected rollers are divided into rollers on the same side and rollers on different sides. Based on the output image of the belt edge detection head, calculate the left edge point set and the right edge point set of the belt, and establish the straight line equations of the left edge and the right edge of the belt by fitting the left edge point set and the right edge point set of the belt using the least squares method. Calculate the distances between the center of any one of the rollers on the same side and the straight lines of the left and right edges of the belt, respectively, and determine whether the roller is located on the left or right side of the belt based on the distances. For the several rollers on the left side of the belt, select the center points of any two rollers on the same side to establish two straight line equations. The two straight lines formed are respectively denoted as the left side warning line and the right side warning line of the belt. The left side warning line of the belt is moved to the left along the X-axis by a distance threshold to obtain the left side alarm line of the belt, and the right side warning line of the belt is moved to the right along the X-axis by a distance threshold to obtain the right side alarm line of the belt.

3. The belt misalignment detection method based on multi-task learning according to claim 1, characterized in that, The process of calculating the distances between the two sides of the belt and the corresponding side warning line when the included angles do not exceed the included angle threshold, and then determining the belt deviation direction based on the relationship between the distances and the distance thresholds, is as follows: Calculate the distances from the center point of the warning line on the left side of the belt and the center point of the warning line on the right side of the belt to the corresponding edge of the belt, denoted as dwarnL and dwarnR respectively. Then, when dwarnL < 0 and |dwarnL| < L, a warning is issued for the left side of the belt to deviate; where L represents the distance threshold. When dwarnL < 0 and |dwarnL| > L, a warning is issued for the left side of the belt to deviate. When dwarnR < 0 and |dwarnR| < L, a warning is issued for the right side of the belt to deviate. When dwarnR < 0 and |dwarnR| > L, a warning is issued for the right side of the belt to deviate.

4. The belt misalignment detection method based on multi-task learning according to claim 1, characterized in that, The process of calculating the angle between the left warning line of the belt and the equation of the left edge of the belt, and the angle between the right warning line of the belt and the equation of the right edge of the belt, is as follows: Let the equation of the warning line on the left or right side of the belt be y = k1x + b1, and the equation of the corresponding belt edge line be y = k2x + b2. Then, the angle θ between the warning line and the straight line equation of the corresponding side of the belt edge is calculated as follows: tanθ=|(k2-k1) / (1+k1*k2)| θ=(tan -1 (tanθ)*(180 / π) Where k1 and k2 are the slopes of the two lines forming the angle between the lines to be determined.

5. The belt misalignment detection method based on multi-task learning according to claim 1, characterized in that, Also includes: It provides feedback on the direction of belt misalignment and returns the belt misalignment angle, displays and saves belt misalignment warning or alarm images for data traceability.

6. A belt misalignment detection device based on multi-task learning, characterized in that, include: The acquisition module is used to acquire several input images with annotation information and divide the input images into training set images and test set images. The input images are captured images of the belt running process, and the annotation information includes the roller, the left side of the belt, and the right side of the belt. A training module is constructed to build a target detection model based on multi-task learning. The model is trained using the training set images and the test set images. The target detection model is an improved YOLO model that includes a belt edge detection head composed of two fully connected layers. The loss function of the improved YOLO model is a joint loss function of the YOLO model loss function and the belt edge detection loss function. The detection module is used to acquire real-time images of the belt running and input them into the target detection model based on multi-task learning to obtain an output image, which is a real-time image after target detection and annotation. The calculation and determination module is used to calculate the straight line equations of the left edge and the right edge of the belt based on the targets marked in the output image, determine the position of the drag roller, and then determine the left warning line, the left alarm line, the right warning line, and the right alarm line of the belt. The calculation and judgment module is used to calculate and judge whether the angle between the left warning line of the belt and the straight line equation of the left edge of the belt, and the angle between the right warning line of the belt and the straight line equation of the right edge of the belt do not exceed the angle threshold. When the angles do not exceed the angle threshold, the distances between the two edges of the belt and the corresponding side warning lines are calculated respectively, and then the belt deviation direction is determined according to the relationship between the distance and the distance threshold. The loss function for improving the YOLO model in the training module is calculated as follows: belt edge similarity loss function L sim The calculation formula is: Location loss function L shp The calculation formula is: Classification loss function L cls The calculation formula is: Where C represents the left and right edges of the belt, C = 2; h represents the anchor frame parameters; P i,j,: Let P be the (w+1)-dimensional vector output by the model, where w = 200, representing the predicted value at the i-th belt edge and the j-th column; i,j+1,: Q represents the predicted value of the (j+1)th column at the i-th belt edge; i,j,: This represents the i-th belt edge, with the j-th column being a uniquely hot-coded label; k is an integer representing the position index; L CE Prob represents the cross-entropy loss function; i,j,: Loc represents the probability value at different positions, ranging from 1 to w. i,j Indicates the location where the belt edge exists; p i,j,1:w The w-dimensional vector output by the model represents the probability of the presence of a belt edge; Prob i,j,k P represents the probability value at the k-th position; i,j,k This represents the probability that the belt edge exists at the model's predicted output value at the k-th position. Loss function L for belt edge detection line The calculation formula is: L line =L shp +βL cls +L sim Where β represents a hyperparameter used to balance the weights of the class loss function in the loss function for belt edge detection.

7. The belt misalignment detection device based on multi-task learning according to claim 6, characterized in that, The calculation and determination module calculates the straight line equations of the left and right edges of the belt, determines the position of the drag roller, and then determines the execution unit for the left warning line, left alarm line, right warning line, and right alarm line of the belt, including: The segmentation unit is used to segment the detected rollers into rollers on the same side and rollers on different sides based on the center coordinates of the target boxes of several rollers in the output image. The first calculation unit is used to calculate the left edge point set and the right edge point set of the belt based on the output image of the belt edge detection head, and to fit and establish the straight line equation of the left edge of the belt and the straight line equation of the right edge of the belt based on the left edge point set and the right edge point set of the belt using the least squares method. The second calculation unit is used to calculate the distance between the center of any one of the rollers on the same side and the straight line between the left edge and the right edge of the belt, respectively, and determine whether the roller is located on the left or right side of the belt based on the distance. The unit is used to establish two straight line equations for the center points of any two rollers on the same side of the belt, for a plurality of rollers on the left side of the belt. The two straight lines are denoted as the left side warning line and the right side warning line of the belt, respectively. The left side warning line is moved to the left along the X-axis by a threshold distance to obtain the left side alarm line of the belt, and the right side warning line is moved to the right along the X-axis by a threshold distance to obtain the right side alarm line of the belt.

8. The belt misalignment detection device based on multi-task learning according to claim 6, characterized in that, The calculation and judgment module calculates the distance between the two edges of the belt and the corresponding side warning line, and then determines the direction of belt deviation based on the relationship between the distance and the distance threshold. Calculate the distances from the center point of the warning line on the left side of the belt and the center point of the warning line on the right side of the belt to the corresponding edge of the belt, denoted as dwarnL and dwarnR respectively. Then, when dwarnL < 0 and |dwarnL| < L, a warning is issued for the left side of the belt to deviate; where L represents the distance threshold. When dwarnL < 0 and |dwarnL| > L, a warning is issued for the left side of the belt to deviate. When dwarnR < 0 and |dwarnR| < L, a warning is issued for the right side of the belt to deviate. When dwarnR < 0 and |dwarnR| > L, a warning is issued for the right side of the belt to deviate.

9. An electronic device, characterized in that, The device includes a computer program stored in a computer-readable storage medium; when a processor of an electronic device reads the computer program from the computer-readable storage medium, the processor executes the computer program, causing the electronic device to perform the steps of the belt misalignment detection method based on multi-task learning as described in any one of claims 1-5.

Images