Belt deviation detection method and device based on multi-task learning

Abstract

The invention provides a belt deviation detection method and device based on multi-task learning, and relates to the field of artificial intelligence. Comprising the steps of obtaining a plurality of input images with annotation information and dividing the input images into training set images and test set images; a target detection model based on multi-task learning is constructed, and the training set image and the test set image are adopted for training; a real-time image of belt operation is obtained and input to the target detection model to obtain an output image; according to the marked target in the output image, calculating a linear equation of left and right edges of the belt, determining the position of a drag roller, and further determining left and right early warning lines of the belt and left and right warning lines of the belt; when the included angles between the left and right early warning lines of the belt and the linear equations of the edges of the corresponding sides do not exceed an included angle threshold value, the distances between the edges of the two sides of the belt and the early warning lines of the corresponding sides are calculated, and then the deviation direction of the belt is judged; the deviation direction and angle of the belt can be accurately detected by simultaneously detecting the dragging roller and the edge of the belt.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a belt deviation detection method and device based on multi-task learning. Background Art

[0002] Belt conveying is a major mode of transportation in coal mines. Belt deviation may cause material scattering, equipment damage, or even more serious safety accidents, which is a major safety hazard in belt conveying. Timely and accurate detection of belt deviation is crucial to preventing accidents and ensuring safe production.

[0003] The existing technology mainly detects belt deviation based on the following methods, including: 1) Based on manual regular inspection, inspectors regularly inspect the belt transportation scenes. Since there are many scenes of using belt transportation in coal mine scenes, this invisibly increases the labor cost; secondly, manual regular inspections cannot monitor the belt operation status in real time, and the inspection efficiency is low, making it difficult to ensure safe production. 2) Based on the optical flow method, such as the optical flow-based belt deviation detection method proposed in patent application CN117699375A. This method is greatly affected by the environment. When the light intensity changes, this method cannot guarantee the belt detection rate. This is caused by the inherent defects of the optical flow method. In the coal mine scene, the production environment is relatively harsh, and the robustness of this method is poor. 3) Based on the target detection method, the belt deviation is judged by detecting the belt drag roller. When the drag roller is blocked, this method is prone to missed detection. In addition, this method has no way to determine the direction of the belt deviation. 4) Image segmentation-based methods: For example, the machine vision-based belt deviation detection method proposed in patent application CN117566378A can accurately obtain the edge contour of the belt, but the amount of calculation is relatively large. At the same time, this method requires pre-setting the belt deviation boundary, which is relatively cumbersome. 5) Based on traditional image processing algorithms, such as the vision-based belt deviation detection method proposed in patent CN116573366B, image edge detection is performed based on the Sobel operator. This method is limited by the environment, has poor generalization and low accuracy, and is difficult to meet production needs.

[0004] Therefore, it is necessary to propose a method that is not restricted by the environment and can perform belt deviation detection in real time, so as to improve the efficiency of belt deviation detection and ensure safe production. Summary of the invention

[0005] The purpose of the present invention is to provide a belt deviation detection method and device based on multi-task learning, which optimizes the target detection model through a joint loss function, realizes the simultaneous detection of the drag roller and the belt edge, and then accurately detects the deviation direction and angle of the belt by calculating the belt edge straight line, warning line and alarm line, thereby providing strong support for ensuring enterprise safety production and preventing accidents.

[0006] To achieve the above object, the present invention proposes the following technical solutions:

[0007] First, a belt deviation detection method based on multi-task learning is proposed, including:

[0008] Acquire a plurality of input images with annotation information and divide the input images into training set images and test set images, wherein the input images are images of the belt running process, and the annotation information includes the drag roller, the left side of the belt, and the right side of the belt;

[0009] Constructing a target detection model based on multi-task learning, and using the training set images and the test set images to train the model; wherein the target detection model is an improved YOLO model including a belt edge detection head composed of two fully connected layer networks, and the loss function of the improved YOLO model is a joint loss function of the YOLO model loss function and the belt edge detection loss function;

[0010] Acquire a real-time image of the belt running and input it into the target detection model based on multi-task learning to obtain an output image, wherein the output image is a real-time image after target detection and annotation;

[0011] According to the targets marked in the output image, the equation of the straight line on the left edge of the belt and the equation of the straight line on the right edge of the belt are calculated to determine the position of the drag roller, and then determine the left warning line of the belt, the left warning line of the belt, the right warning line of the belt, and the right warning line of the belt;

[0012] Calculate and determine whether the angle between the left warning line of the belt and the straight line equation of the left edge of the belt, and the angle between the right warning line of the belt and the straight line equation of the right edge of the belt do not exceed the angle threshold; when the angles do not exceed the angle threshold, calculate the distances between the two side edges of the belt and the corresponding side warning lines respectively, and then determine the direction of belt deviation based on the relationship between the distances and the distance threshold.

[0013] Furthermore, the process of calculating the straight line equation of the left edge of the belt and the straight line equation of the right edge of the belt according to the target marked in the output image, determining the position of the drag roller, and then determining the left warning line of the belt, the left warning line of the belt, the right warning line of the belt, and the right warning line of the belt is:

[0014] According to the center coordinates of the plurality of roller target frames in the output image, the plurality of rollers to be detected are divided into rollers on the same side and rollers on different sides;

[0015] According to the output image of the belt edge detection head, a belt left edge point set and a belt right edge point set are calculated, and a belt left edge straight line equation and a belt right edge straight line equation are established based on the least squares method according to the belt left edge point set and the belt right edge point set;

[0016] Calculate the distance between the center of any one of the same-side drag rollers and the left edge straight line of the belt and the right edge straight line of the belt, and determine whether the drag roller is on the left side or the right side of the belt according to the distance;

[0017] For several rollers on the left side of the belt and several rollers on the right side of the belt, select the center points of any two rollers on the same side to establish two straight line equations, and the two straight lines formed are respectively recorded as the left side warning line of the belt and the right side warning line of the belt. The left side warning line of the belt moves to the left along the X-axis by a threshold distance to obtain the left side warning line of the belt, and the right side warning line of the belt moves to the right along the X-axis by a threshold distance to obtain the right side warning line of the belt.

[0018] Furthermore, when the angles do not exceed the angle threshold, the distances between the two side edges of the belt and the corresponding side warning lines are calculated respectively, and then the process of determining the belt deviation direction according to the relationship between the distance and the distance threshold is as follows:

[0019] The distances between the center points of the warning lines on the left and right sides of the belt and the straight lines of the belt edges on the corresponding sides are calculated respectively, and are recorded as dwarnL and dwarnR. Then, when dwarnL<0 and |dwarnL|<L, a warning of the current left side deviation of the belt is prompted; wherein L represents the distance threshold; when dwarnL<0 and |dwarnL|>L, a warning of the current left side deviation of the belt is prompted; when dwarnR<0 and |dwarnR|<L, a warning of the current right side deviation of the belt is prompted; when dwarnR<0 and |dwarnR|>L, a warning of the current right side deviation of the belt is prompted.

[0020] Furthermore, the loss function of the improved YOLO model is calculated as follows:

[0021] Belt edge similarity loss function L sim The calculation formula is:

[0022]

[0023] Position loss function L shp The calculation formula is:

[0024] Prob i，j,： =softmax(P i,j，1：w )

[0025]

[0026]

[0027]

[0028] Classification loss function Lcls The calculation formula is:

[0029]

[0030] Where C represents the left and right edges of the belt, C = 2; h represents the row anchor box parameter; P i,j，： represents the w+1-dimensional vector output by the model, w=200, representing the predicted value of the i-th belt edge and the j-th column; P i，j+1,： represents the predicted value of the i-th belt edge, j+1th column; Q i，j，： represents the i-th belt edge, the j-th column is a one-hot encoded label; k is an integer, indicating the position index; L CE represents the cross entropy loss function; Prob i，j，： Indicates the probability value of different positions, ranging from 1-w; Loc i，j Indicates the position where the belt edge exists; P i,j，1：w Represents the w-dimensional vector output by the model, the probability of the existence of a belt edge; Prob i，j,k represents the probability value of the kth position; P i，j，k represents the model output prediction value at the kth position, the probability of the existence of a belt edge;

[0031] Loss function L for belt edge detection line The calculation formula is:

[0032] L line =L shp +βL cls +L sim

[0033] Among them, β represents a hyperparameter used to balance the weight of the category loss function in the loss function of belt edge detection;

[0034] The total loss function L of the improved YOLO model total The calculation formula is:

[0035] L total =L obj +αL line

[0036] Among them, L obj is the loss function of the original YOLO series target detector.

[0037] Further, the process of calculating the angle between the left warning line of the belt and the straight line equation of the left edge of the belt, and the angle between the right warning line of the belt and the straight line equation of the right edge of the belt is as follows:

[0038] The linear equation of the warning line on the left side of the belt or the warning line on the right side of the belt is: y=k1x+b1, and the linear equation of the linear equation of the belt edge on the corresponding side is: y=k2x+b2;

[0039] Then, the angle θ between the warning line and the belt edge straight line equation on the corresponding side is calculated as:

[0040] tanθ=|(k2-k1) / (1+k1*k2)|

[0041] θ=(tan -1 (tanθ))*(180 / π)

[0042] Among them, k1 and k2 are the slopes of the two straight lines whose included angle is to be determined.

[0043] Furthermore, it also includes:

[0044] Feedback the belt deviation direction and return the belt deviation angle, display and save the belt deviation warning or alarm picture for data traceability.

[0045] In the second aspect, a belt deviation detection device based on multi-task learning is proposed, comprising:

[0046] An acquisition module is used to acquire a plurality of input images with annotation information and divide the input images into training set images and test set images, wherein the input images are images of the belt running process, and the annotation information includes the drag roller, the left side of the belt, and the right side of the belt;

[0047] Constructing a training module for constructing a target detection model based on multi-task learning, and using the training set images and the test set images to train the model; wherein the target detection model is an improved YOLO model including a belt edge detection head composed of two fully connected layer networks, and the loss function of the improved YOLO model is a joint loss function of the YOLO model loss function and the belt edge detection loss function;

[0048] An acquisition detection module is used to acquire a real-time image of the belt running and input it into the target detection model based on multi-task learning to obtain an output image, wherein the output image is a real-time image after target detection annotation;

[0049] A calculation and determination module is used to calculate the straight line equation of the left edge of the belt and the straight line equation of the right edge of the belt according to the target marked in the output image, determine the position of the drag roller, and then determine the left warning line of the belt, the left warning line of the belt, the right warning line of the belt, and the right warning line of the belt;

[0050] A calculation and judgment module is used to calculate and judge whether the angle between the left warning line of the belt and the straight line equation of the left edge of the belt, and the angle between the right warning line of the belt and the straight line equation of the right edge of the belt do not exceed the angle threshold. When the angles do not exceed the angle threshold, the distances between the two side edges of the belt and the corresponding side warning lines are calculated respectively, and the direction of belt deviation is determined according to the relationship between the distance and the distance threshold.

[0051] Further, the calculation and determination module calculates the straight line equation of the left edge of the belt and the straight line equation of the right edge of the belt, determines the position of the drag roller, and then determines the execution unit of the left warning line of the belt, the left warning line of the belt, the right warning line of the belt, and the right warning line of the belt, including:

[0052] A division unit, used for dividing the detected rollers into rollers on the same side and rollers on different sides according to the center coordinates of the target frames of the rollers in the output image;

[0053] A first calculation unit is used to calculate a belt left edge point set and a belt right edge point set according to the output image of the belt edge detection head, and to establish a belt left edge straight line equation and a belt right edge straight line equation according to the belt left edge point set and the belt right edge point set based on the least square method;

[0054] The second calculation unit is used to calculate the distance between the center of any one of the same-side drag rollers and the left edge straight line of the belt and the right edge straight line of the belt, and determine whether the drag roller is located on the left side or the right side of the belt according to the distance;

[0055] An establishment unit is used to establish two straight line equations for the several rollers on the left side of the belt and the several rollers on the right side of the belt, respectively selecting the center points of any two rollers on the same side, the two formed straight lines are respectively recorded as the left side warning line of the belt and the right side warning line of the belt, the left side warning line of the belt moves to the left along the X-axis by a threshold distance to obtain the left side warning line of the belt, and the right side warning line of the belt moves to the right along the X-axis by a threshold distance to obtain the right side warning line of the belt.

[0056] Furthermore, the calculation and judgment module calculates the distance between the two side edges of the belt and the corresponding side warning line, and then determines the direction of the belt deviation according to the relationship between the distance and the distance threshold. The process is as follows:

[0057] The distances between the center points of the warning lines on the left and right sides of the belt and the straight lines of the belt edges on the corresponding sides are calculated respectively, and are recorded as dwarnL and dwarnR. Then, when dwarnL<0 and |dwarnL|<L, a warning of the current left side deviation of the belt is prompted; wherein L represents the distance threshold; when dwarnL<0 and |dwarnL|>L, a warning of the current left side deviation of the belt is prompted; when dwarnR<0 and |dwarnR|<L, a warning of the current right side deviation of the belt is prompted; when dwarnR<0 and |dwarnR|>L, a warning of the current right side deviation of the belt is prompted.

[0058] In the third aspect, an electronic device is proposed, comprising a computer program, which is stored in a computer-readable storage medium; when a processor of the electronic device reads the computer program from the computer-readable storage medium, the processor executes the computer program, so that the electronic device performs the steps of the above-mentioned belt deviation detection method based on multi-task learning.

[0059] It can be seen from the above technical solutions that the technical solutions of the present invention have the following beneficial effects:

[0060] The invention discloses a belt deviation detection method and device based on multi-task learning, and the method comprises: obtaining a plurality of input images with annotation information and dividing the input images into training set images and test set images; constructing a target detection model based on multi-task learning, and using the training set images and the test set images to train the model; obtaining a real-time image of the belt running and inputting it into the target detection model to obtain an output image; calculating the straight line equation of the left edge of the belt and the straight line equation of the right edge of the belt according to the targets annotated in the output image, determining the position of the drag roller, and then determining the left warning line of the belt, the left alarm line of the belt, the right warning line of the belt, and the right alarm line of the belt; calculating and judging whether the angles between the left and right warning lines of the belt and the straight line equations of the belt edge on the corresponding side do not exceed the angle threshold, and when the angle threshold is not exceeded, respectively calculating the distances between the two side edges of the belt and the corresponding side warning lines, and then determining the belt deviation direction according to the relationship between the distance and the distance threshold; the invention simultaneously detects the drag roller and the belt edge, calculates the belt edge straight line, the warning line and the warning line, and then accurately detects the belt deviation direction and angle; compared with the prior art, the invention is not limited by environmental changes, the method is simple, the detection accuracy is high, and the production needs can be fully met.

[0061] The target detection model of the present invention is an improved YOLO model. A belt edge detection head composed of two fully connected layer networks is added to the original network structure, and a belt edge detection loss function is introduced into the loss function of the model. The joint loss function is used to optimize network learning to achieve simultaneous detection of the drag roller and the belt edge, so as to accurately obtain the belt edge straight line, warning line and alarm line. Compared with the existing technology, it can not only detect the deviation of the belt, but also derive the direction and angle of the belt deviation.

[0062] It should be appreciated that all combinations of the foregoing concepts, as well as additional concepts described in greater detail below, may be considered to be part of the inventive subject matter of the present disclosure, provided such concepts are not mutually inconsistent.

[0063] The foregoing and other aspects, embodiments and features of the present invention can be more fully understood from the following description in conjunction with the accompanying drawings. Other additional aspects of the present invention, such as the features and / or beneficial effects of the exemplary embodiments, will be apparent from the following description or learned from the practice of the specific embodiments according to the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] The accompanying drawings are not drawn to scale according to actual reference objects. In the accompanying drawings, each identical or nearly identical component shown in various figures may be represented by the same reference numeral. For the sake of clarity, not every component is labeled in each figure. Now, embodiments of various aspects of the present invention will be described by way of example and with reference to the accompanying drawings, in which:

[0065] Figure 1 This is a flow chart of a belt deviation detection method based on multi-task learning of the present invention;

[0066] Figure 2 It is a calculation flow chart of the edge straight lines on both sides of the belt, the warning lines on both sides and the alarm lines of the present invention;

[0067] Figure 3 This is a flow chart of the method for determining the belt deviation direction of the present invention;

[0068] Figure 4 This is a flow chart of the belt deviation alarm of the present invention;

[0069] Figure 5 This is a structural block diagram of a belt deviation detection device based on multi-task learning of the present invention;

[0070] Figure 6 A structural block diagram of an electronic device is provided for the present invention. DETAILED DESCRIPTION

[0071] In order to make the purpose, technical solution and advantages of the embodiment of the present invention clearer, the technical solution of the embodiment of the present invention will be clearly and completely described below in conjunction with the drawings of the embodiment of the present invention. Obviously, the described embodiment is a part of the embodiment of the present invention, not all of the embodiments. Based on the described embodiment of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention. Unless otherwise defined, the technical terms or scientific terms used herein should be the common meanings understood by people with general skills in the field to which the present invention belongs.

[0072] The words "first", "second" and similar words used in the patent application specification and claims of the present invention do not indicate any order, quantity or importance, but are only used to distinguish different components. Similarly, unless the context clearly indicates otherwise, the singular form of "a", "an" or "the" and other similar words do not indicate a quantitative limitation, but indicate the existence of at least one. "Include" or "comprise" and other similar words mean that the elements or objects appearing before "include" or "comprise" include the features, wholes, steps, operations, elements and / or components listed after "include" or "comprise", and do not exclude the existence or addition of one or more other features, wholes, steps, operations, elements, components and / or their collections. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0073] Based on the belt deviation detection methods disclosed in the prior art, including manual periodic inspection, optical flow method, target detection method, image segmentation method and traditional image processing method, there are technical problems such as high labor cost, low inspection efficiency, susceptibility to environmental influences, high missed detection rate and low precision. Therefore, the present invention aims to develop a new belt deviation detection method, which improves the target detection method, and simultaneously detects the belt and the drag roller to obtain the edge straight line, warning line and alarm line for judging the belt deviation, so as to efficiently and accurately determine the belt deviation direction.

[0074] The belt deviation detection method and device based on multi-task learning disclosed in the present invention are further specifically introduced below in conjunction with the accompanying drawings.

[0075] Combination Figure 1 As shown, the belt deviation detection method based on multi-task learning disclosed in an embodiment of the present invention includes the following steps:

[0076] Step S102, obtaining a plurality of input images with annotation information and dividing the input images into training set images and test set images, wherein the input images are images of the belt running process, and the annotation information includes the drag roller, the left side of the belt, and the right side of the belt;

[0077] The acquired belt running process images are now processed, that is, data annotation is performed, such as using labelme annotation software to complete data annotation and generate a json file for model training.

[0078] Step S104, constructing a target detection model based on multi-task learning, and using the training set images and the test set images to train the model; wherein the target detection model is an improved YOLO model including a belt edge detection head composed of two fully connected layer networks, and the loss function of the improved YOLO model is a joint loss function of the YOLO model loss function and the belt edge detection loss function;

[0079] In this case, the improved YOLO model retains the original network structure, and only two fully connected layers are added to the neck structure of the original YOLO model network to form a belt edge detection head for belt edge detection, and its input level output is 1024->2048->7200. The labels obtained from the json file generated by the data annotation are converted to YOLO format; the input data of the improved YOLO model is resized to 640,640 and normalized; during training, mosaic and mixup data enhancements that are not suitable for belt edge detection are turned off. During model training, the preset number of row anchor boxes is 18, the sampling points are 200, and the row anchor boxes are:

[0080] [140,168,196,224,252,280,308,336,364,392,420,448,486,514,542,580,608,636].

[0081] In order to realize the detection of the belt edge, the loss function of the belt edge detection is introduced into the loss function of the improved YOLO model. By combining the loss function, the network learning is optimized to realize the simultaneous detection of the roller and the belt edge. Specifically, the loss function calculation process of the improved YOLO model is as follows:

[0082] Belt edge similarity loss function L sim The calculation formula is:

[0083]

[0084] Position loss function L shp The calculation formula is:

[0085] Prob i，j，： =softmax(P i,j，1：w )

[0086]

[0087]

[0088]

[0089] Classification loss function L cls The calculation formula is:

[0090]

[0091] Where C represents the left and right edges of the belt, C = 2; h represents the row anchor box parameter; P i，j，： represents the w+1-dimensional vector output by the model, w=200, representing the predicted value of the i-th belt edge and the j-th column; P i，j+1，： represents the predicted value of the i-th belt edge, j+1th column; Q i，j，: represents the i-th belt edge, the j-th column is a one-hot encoded label; k is an integer, indicating the position index; L CE represents the cross entropy loss function; Prob i，j，： Indicates the probability value of different positions, ranging from 1-w; Loc i，j Indicates the position where the belt edge exists; P i,j,1：w Represents the w-dimensional vector output by the model, the probability of the existence of a belt edge; Prob i，j，k represents the probability value of the kth position; P i，j，k represents the model output prediction value at the kth position, the probability of the existence of a belt edge;

[0092] Loss function L for belt edge detection line The calculation formula is:

[0093] L line =L shp +βL cls +L sim

[0094] Among them, β represents a hyperparameter used to balance the weight of the category loss function in the loss function of belt edge detection;

[0095] The total loss function L of the improved YOLO model total The calculation formula is:

[0096] L total =L obj +αL line

[0097] Among them, L obj is the loss function of the original YOLO series target detector.

[0098] Step S106, obtaining a real-time image of the belt running and inputting it into the target detection model based on multi-task learning to obtain an output image, wherein the output image is a real-time image after target detection and annotation; a plurality of roller target frames and belt target frames are clearly annotated in the output image;

[0099] Step S108, according to the target marked in the output image, the equation of the straight line on the left side of the belt and the equation of the straight line on the right side of the belt are calculated to determine the position of the drag roller, and then determine the left side warning line of the belt, the left side warning line of the belt, the right side warning line of the belt, and the right side warning line of the belt;

[0100] The specific implementation process is as follows Figure 2 As shown, first, according to the center coordinates of the several roller target frames in the output image, the several rollers to be detected are divided into rollers on the same side and rollers on different sides, that is, whether the rollers are on the same side is determined according to the size of the center coordinate x; then, according to the output image of the belt edge detection head, that is, according to the divided rollers on the same side or on different sides, the belt left edge point set and the belt right edge point set are calculated from the input image, and the belt left edge straight line equation and the belt right edge straight line equation are fitted based on the least squares method according to the belt left edge point set and the belt right edge point set; then, the distance between the center of any roller on the same side and the belt left edge straight line and the corresponding belt right edge straight line is calculated, d iL and d iR , according to the distance, it is determined that the drag roller is located on the left side or the right side of the belt; specifically, when d iR >d iL , it indicates that the side roller is the right side roller; otherwise, the side roller is the right side roller; finally, for the several rollers on the left side of the belt and the several rollers on the left side of the belt, the center points of any two rollers on the same side are selected to establish two straight line equations, and the two straight lines formed are respectively recorded as the left side warning line of the belt and the right side warning line of the belt. The left side warning line of the belt moves to the left along the X-axis by a threshold distance, L, to obtain the left side warning line of the belt, and the right side warning line of the belt moves to the right along the X-axis by a threshold distance to obtain the right side warning line of the belt.

[0101] Step S110, calculate and determine whether the angle between the left warning line of the belt and the straight line equation of the left edge of the belt, and the angle between the right warning line of the belt and the straight line equation of the right edge of the belt do not exceed the angle threshold; when the angles do not exceed the angle threshold, calculate the distances between the two side edges of the belt and the corresponding side warning lines respectively, and then determine the direction of belt deviation based on the relationship between the distance and the distance threshold.

[0102] Combination Figure 3As shown in the figure, the process of determining the belt deviation direction based on the relationship between the distance and the distance threshold is as follows: Calculate the distances from the center points of the warning lines on the left and right sides of the belt to the straight lines of the corresponding belt edges, denoted as dwarnL and dwarnR respectively. Then, when dwarnL < 0 and |dwarnL| < L, it indicates a warning of belt deviation to the left; where L represents the distance threshold. When dwarnL < 0 and |dwarnL| > L, it indicates an alarm of belt deviation to the left. When dwarnR < 0 and |dwarnR| < L, it indicates a warning of belt deviation to the right. When dwarnR < 0 and |dwarnR| > L, it indicates an alarm of belt deviation to the right.

[0103] In the implementation process of the above step S108, the process of calculating the included angles between the warning line on the left side of the belt and the straight line equation of the left edge of the belt, and the included angle between the warning line on the right side of the belt and the straight line equation of the right edge of the belt is as follows:

[0104] Denote the straight line equation of the warning line on the left side or the right side of the belt as: y = k1x + b1, and the straight line equation of the corresponding belt edge as: y = k2x + b2;

[0105] Then, the included angle θ between the warning line and the straight line equation of the corresponding belt edge is calculated as:

[0106] tanθ = |(k2 - k1) / (1 + k1*k2)|

[0107] θ = (tan -1 (tanθ))*(180 / π)

[0108] Where k1 and k2 are the slopes of the two straight lines for which the included angle of the straight line is to be determined respectively.

[0109] Combined with Figure 4 As shown in the figure, when the belt deviation detection method based on multi-task learning disclosed in the present invention is applied, it further includes: when it is determined in step S110 that the belt deviates, feedback the belt deviation direction and return the belt deviation angle, display and save the belt deviation warning or alarm picture for data traceability, and continuously detect the belt running process.

[0110] The present invention discloses a belt deviation detection method based on multi-task learning to solve the belt deviation detection problem. By adding a belt edge detection head to the original network structure in a commonly used target detection algorithm, such as the YOLO model, and introducing a belt edge detection loss function in the loss function, the network learning is optimized through the joint loss function to achieve simultaneous detection of the drag roller and the belt edge. Furthermore, the warning area and the alarm area of ​​the belt deviation are determined through the detected drag roller, and whether the belt edge straight line is in the warning area or the alarm area is judged to detect the belt deviation direction. Compared with the prior art, not only is the detection method simple and not affected by the environment, but also the detection accuracy is high and the missed detection rate is low.

[0111] In an embodiment of the present application, an electronic device is also provided, including a computer program, wherein the computer program is stored in a computer-readable storage medium; when a processor of the electronic device reads the computer program from the computer-readable storage medium, the processor executes the computer program, so that the electronic device performs the steps of the belt deviation detection method based on multi-task learning. Taking an electronic device running on a computer as an example, Figure 6 As shown, the electronic device may include one or more (only one is shown in the figure) processors (the processor may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory for storing data, and a transmission device for communication functions. A person skilled in the art can understand that Figure 6 The structure shown is for illustration only and does not limit the structure of the above-mentioned electronic device.

[0112] The above program can be run in the processor, or it can be stored in the memory, that is, in the computer-readable medium, which includes permanent and non-permanent, removable and non-removable media. Information storage can be achieved by any method or technology. Information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined in this article, computer-readable media does not include temporary computer-readable media, such as modulated data signals and carrier waves. These computer programs can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are performed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps of the functions specified in a block or multiple blocks may be implemented by different modules corresponding to different method steps.

[0113] In this embodiment, such a device or system is provided, which can be called a belt deviation detection device based on multi-task learning. Figure 5As shown, it includes: an acquisition module, which is used to acquire several input images with annotation information and divide the input images into training set images and test set images, wherein the input image is a captured image of the belt running process, and the annotation information includes the drag roller, the left side of the belt, and the right side of the belt; a training module is constructed, which is used to construct a target detection model based on multi-task learning, and the model is trained using the training set images and the test set images; wherein the target detection model is an improved YOLO model including a belt edge detection head composed of two fully connected layer networks, and the loss function of the improved YOLO model is a joint loss function of the YOLO model loss function and the belt edge detection loss function; an acquisition detection module is used to acquire real-time images of the belt running and input them into the multi-task learning-based A target detection model is learned to obtain an output image, which is a real-time image after target detection and annotation; a calculation and determination module is used to calculate the straight line equation of the left edge of the belt and the straight line equation of the right edge of the belt according to the targets annotated in the output image, determine the position of the drag roller, and then determine the left warning line of the belt, the left warning line of the belt, the right warning line of the belt, and the right warning line of the belt; a calculation and judgment module is used to calculate and judge whether the angle between the left warning line of the belt and the straight line equation of the left edge of the belt, and the angle between the right warning line of the belt and the straight line equation of the right edge of the belt do not exceed the angle threshold, and when the angles do not exceed the angle threshold, the distances between the two side edges of the belt and the corresponding side warning lines are calculated respectively, and then the belt deviation direction is determined according to the relationship between the distance and the distance threshold.

[0114] The system is used to implement the steps of the belt deviation detection method based on multi-task learning disclosed in the above embodiment, which have been described and will not be repeated here.

[0115] For example, the calculation and determination module calculates the straight line equations of the left and right edges of the belt, determines the positions of the idlers, and further determines the execution units for the left warning line, left alarm line, right warning line, and right alarm line of the belt, including: a division unit for dividing several detected idlers into idlers on the same side and idlers on different sides according to the center coordinates of several idler target frames in the output image; a first calculation unit for calculating the left edge point set and right edge point set of the belt based on the output image of the belt edge detection head, and fitting and establishing the straight line equations of the left edge and right edge of the belt based on the least squares method according to the left edge point set and right edge point set of the belt; a second calculation unit for calculating the distances from the center of any idler on the same side to the left edge straight line and right edge straight line of the belt respectively, and determining whether the idler is on the left or right side of the belt according to the distances; a establishment unit for, for several idlers on the left side of the belt and several idlers on the left side of the belt, respectively selecting the center points of any two idlers on the same side to establish two straight line equations, and the two formed straight lines are respectively recorded as the left warning line and right warning line of the belt, and the left warning line of the belt is moved leftward along the X-axis by a distance threshold to obtain the left alarm line of the belt, and the right warning line of the belt is moved rightward along the X-axis by a distance threshold to obtain the right alarm line of the belt.

[0116] For another example, the process of the calculation and judgment module calculating the distances between the two edges of the belt and the warning lines on the corresponding sides, and then determining the belt deviation direction according to the relationship between the distances and the distance threshold is as follows: calculate the distances from the center points of the left warning line and right warning line of the belt to the straight lines of the corresponding side belt edges respectively, and record them as dwarnL and dwarnR; then, when dwarnL < 0 and |dwarnL| < L, it indicates a warning of left deviation of the current belt; where L represents the distance threshold; when dwarnL < 0 and |dwarnL| > L, it indicates an alarm of left deviation of the current belt; when dwarnR < 0 and |dwarnR| < L, it indicates a warning of right deviation of the current belt; when dwarnR < 0 and |dwarnR| > L, it indicates an alarm of right deviation of the current belt.

[0117] For another example, the loss function of the improved YOLO model constructed by the construction and training module is calculated as follows:

[0118] The loss function L of the belt edge similarity sim The calculation formula is:

[0119]

[0120] The loss function L of the position shp The calculation formula is:

[0121] Prob i，j，：=softmax(P i，j，1：w )

[0122]

[0123]

[0124]

[0125] Classification loss function L cls The calculation formula is:

[0126]

[0127] Where C represents the left and right edges of the belt, C = 2; h represents the row anchor box parameter; P i,j，： represents the w+1-dimensional vector output by the model, w=200, representing the predicted value of the i-th belt edge and the j-th column; P i，j+1，： represents the predicted value of the i-th belt edge, j+1th column; Q i，j，： represents the i-th belt edge, the j-th column is a one-hot encoded label; k is an integer, indicating the position index; L CE represents the cross entropy loss function; Prob i，j，： Indicates the probability value of different positions, ranging from 1-w; Loc i，j Indicates the position where the belt edge exists; P i，j，1：w Represents the w-dimensional vector output by the model, the probability of the existence of a belt edge; Prob i，j，k represents the probability value of the kth position; P i，j，k represents the model output prediction value at the kth position, the probability of the existence of a belt edge;

[0128] Loss function L for belt edge detection line The calculation formula is:

[0129] L line =L shp +βL cls +L sim

[0130] Among them, β represents a hyperparameter used to balance the weight of the category loss function in the loss function of belt edge detection;

[0131] The total loss function L of the improved YOLO model total The calculation formula is:

[0132] L total =L obj +αL line

[0133] Among them, L objis the loss function of the original YOLO series target detector.

[0134] For another example, the belt deviation detection device based on multi-task learning disclosed in the embodiment also includes: a feedback storage module, which is used to feedback the belt deviation direction and return the belt deviation angle, display and save the belt deviation warning or alarm picture for data tracing.

[0135] The belt deviation detection method and device based on multi-task learning disclosed in the above-mentioned implementation scheme of the present invention, through multi-task learning, integrates the solution that originally requires two models to detect the drag roller and the belt respectively into one model for detection, thereby reducing the storage pressure of the edge device. At the same time, the method can achieve the purpose of real-time detection of the belt operation process; secondly, the method effectively solves the problem that the belt edge cannot be detected when the belt edge is blocked by materials. For coal mine scenarios, this method has strong robustness.

[0136] Although the present invention has been disclosed as above with preferred embodiments, it is not intended to limit the present invention. A person with ordinary knowledge in the technical field to which the present invention belongs may make various changes and modifications without departing from the spirit and scope of the present invention. Therefore, the protection scope of the present invention shall be determined by the definition of the claims.

Claims

1. A belt deviation detection method based on multi-task learning, characterized in that: include: Acquire a plurality of input images with annotation information and divide the input images into training set images and test set images, wherein the input images are images of the belt running process, and the annotation information includes the drag roller, the left side of the belt, and the right side of the belt; Constructing a target detection model based on multi-task learning, and using the training set images and the test set images to train the model; wherein the target detection model is an improved YOLO model including a belt edge detection head composed of two fully connected layer networks, and the loss function of the improved YOLO model is a joint loss function of the YOLO model loss function and the belt edge detection loss function; Acquire a real-time image of the belt running and input it into the target detection model based on multi-task learning to obtain an output image, wherein the output image is a real-time image after target detection and annotation; According to the targets marked in the output image, the equation of the straight line on the left edge of the belt and the equation of the straight line on the right edge of the belt are calculated to determine the position of the drag roller, and then determine the left warning line of the belt, the left warning line of the belt, the right warning line of the belt, and the right warning line of the belt; Calculate and determine whether the angle between the left warning line of the belt and the straight line equation of the left edge of the belt, and the angle between the right warning line of the belt and the straight line equation of the right edge of the belt do not exceed the angle threshold; when the angles do not exceed the angle threshold, calculate the distances between the two side edges of the belt and the corresponding side warning lines respectively, and then determine the direction of belt deviation based on the relationship between the distances and the distance threshold.

2. The belt deviation detection method based on multi-task learning according to claim 1 is characterized in that: The process of calculating the straight line equation of the left edge of the belt and the straight line equation of the right edge of the belt according to the target marked in the output image, determining the position of the drag roller, and then determining the left warning line of the belt, the left warning line of the belt, the right warning line of the belt, and the right warning line of the belt is as follows: According to the center coordinates of the plurality of roller target frames in the output image, the plurality of rollers to be detected are divided into rollers on the same side and rollers on different sides; According to the output image of the belt edge detection head, a belt left edge point set and a belt right edge point set are calculated, and a belt left edge straight line equation and a belt right edge straight line equation are established based on the least squares method according to the belt left edge point set and the belt right edge point set; Calculate the distance between the center of any one of the same-side drag rollers and the left edge straight line of the belt and the right edge straight line of the belt, and determine whether the drag roller is on the left side or the right side of the belt according to the distance; For several rollers on the left side of the belt and several rollers on the right side of the belt, select the center points of any two rollers on the same side to establish two straight line equations, and the two straight lines formed are respectively recorded as the left side warning line of the belt and the right side warning line of the belt. The left side warning line of the belt moves to the left along the X-axis by a threshold distance to obtain the left side warning line of the belt, and the right side warning line of the belt moves to the right along the X-axis by a threshold distance to obtain the right side warning line of the belt.

3. The belt deviation detection method based on multi-task learning according to claim 1 is characterized in that: When the angles do not exceed the angle threshold, the distances between the two side edges of the belt and the corresponding side warning lines are calculated respectively, and then the process of determining the belt deviation direction according to the relationship between the distances and the distance threshold is as follows: The distances between the center points of the warning lines on the left and right sides of the belt and the straight lines of the belt edges on the corresponding sides are calculated respectively, and are recorded as dwarnL and dwarnR. Then, when dwarnL<0 and |dwarnL|<L, a warning of the current left side deviation of the belt is prompted; wherein L represents the distance threshold; when dwarnL<0 and |dwarnL|>L, a warning of the current left side deviation of the belt is prompted; when dwarnR<0 and |dwarnR|<L, a warning of the current right side deviation of the belt is prompted; when dwarnR<0 and |dwarnR|>L, a warning of the current right side deviation of the belt is prompted.

4. The belt deviation detection method based on multi-task learning according to claim 1 is characterized in that: The loss function of the improved YOLO model is calculated as follows: Belt edge similarity loss function L sim The calculation formula is: Position loss function L shp The calculation formula is: Prob i,j,: =softmax(P i,j,1:w ) Classification loss function L cls The calculation formula is: Where C represents the left and right edges of the belt, C = 2; h represents the row anchor box parameter; P i,j,: represents the w+1-dimensional vector output by the model, w=200, representing the predicted value of the i-th belt edge and the j-th column; P i,j+1,: represents the predicted value of the i-th belt edge, j+1th column; Q i,j,: represents the i-th belt edge, the j-th column is a one-hot encoded label; k is an integer, indicating the position index; L CE represents the cross entropy loss function; Prob i,j,: Indicates the probability value of different positions, ranging from 1-w; Loc i,j Indicates the position where the belt edge exists; P i,j,1:w Represents the w-dimensional vector output by the model, the probability of the existence of a belt edge; Prob i,j,k represents the probability value of the kth position; P i,j,k represents the model output prediction value at the kth position, the probability of the existence of a belt edge; Loss function L for belt edge detection line The calculation formula is: L line =L shp +βL cls +L sim Among them, β represents a hyperparameter used to balance the weight of the category loss function in the loss function of belt edge detection; The total loss function L of the improved YOLO model total The calculation formula is: L total =L obj +αL line Among them, L obj is the loss function of the original YOLO series target detector.

5. The belt deviation detection method based on multi-task learning according to claim 1 is characterized in that: The process of calculating the angle between the left warning line of the belt and the straight line equation of the left edge of the belt, and the angle between the right warning line of the belt and the straight line equation of the right edge of the belt is as follows: The linear equation of the warning line on the left side of the belt or the warning line on the right side of the belt is: y=k1x+b1, and the linear equation of the linear equation of the belt edge on the corresponding side is: y=k2x+b2; Then, the angle θ between the warning line and the belt edge straight line equation on the corresponding side is calculated as: tanθ=|(k2-k1) / (1+k1*k2)| θ=(tan -1 (tanθ)*(180 / π) Among them, k1 and k2 are the slopes of the two straight lines whose included angle is to be determined.

6. The belt deviation detection method based on multi-task learning according to claim 1 is characterized in that: Also includes: Feedback the belt deviation direction and return the belt deviation angle, display and save the belt deviation warning or alarm picture for data traceability.

7. A belt deviation detection device based on multi-task learning, characterized in that: include: An acquisition module is used to acquire a plurality of input images with annotation information and divide the input images into training set images and test set images, wherein the input images are images of the belt running process, and the annotation information includes the drag roller, the left side of the belt, and the right side of the belt; Constructing a training module for constructing a target detection model based on multi-task learning, and using the training set images and the test set images to train the model; wherein the target detection model is an improved YOLO model including a belt edge detection head composed of two fully connected layer networks, and the loss function of the improved YOLO model is a joint loss function of the YOLO model loss function and the belt edge detection loss function; An acquisition detection module is used to acquire a real-time image of the belt running and input it into the target detection model based on multi-task learning to obtain an output image, wherein the output image is a real-time image after target detection annotation; A calculation and determination module is used to calculate the straight line equation of the left edge of the belt and the straight line equation of the right edge of the belt according to the target marked in the output image, determine the position of the drag roller, and then determine the left warning line of the belt, the left warning line of the belt, the right warning line of the belt, and the right warning line of the belt; A calculation and judgment module is used to calculate and judge whether the angle between the left warning line of the belt and the straight line equation of the left edge of the belt, and the angle between the right warning line of the belt and the straight line equation of the right edge of the belt do not exceed the angle threshold. When the angles do not exceed the angle threshold, the distances between the two side edges of the belt and the corresponding side warning lines are calculated respectively, and then the direction of belt deviation is determined according to the relationship between the distance and the distance threshold.

8. The belt deviation detection device based on multi-task learning according to claim 7 is characterized in that: The calculation and determination module calculates the straight line equation of the left edge of the belt and the straight line equation of the right edge of the belt, determines the position of the drag roller, and then determines the execution unit of the left warning line of the belt, the left warning line of the belt, the right warning line of the belt, and the right warning line of the belt, including: A division unit, used for dividing the detected rollers into rollers on the same side and rollers on different sides according to the center coordinates of the target frames of the rollers in the output image; A first calculation unit is used to calculate a belt left edge point set and a belt right edge point set according to the output image of the belt edge detection head, and to establish a belt left edge straight line equation and a belt right edge straight line equation according to the belt left edge point set and the belt right edge point set based on the least square method; The second calculation unit is used to calculate the distance between the center of any one of the same-side drag rollers and the left edge straight line of the belt and the right edge straight line of the belt, and determine whether the drag roller is located on the left side or the right side of the belt according to the distance; An establishment unit is used to establish two straight line equations for the several rollers on the left side of the belt and the several rollers on the right side of the belt, respectively selecting the center points of any two rollers on the same side, the two formed straight lines are respectively recorded as the left side warning line of the belt and the right side warning line of the belt, the left side warning line of the belt moves to the left along the X-axis by a threshold distance to obtain the left side warning line of the belt, and the right side warning line of the belt moves to the right along the X-axis by a threshold distance to obtain the right side warning line of the belt.

9. The belt deviation detection device based on multi-task learning according to claim 7 is characterized in that: The calculation and judgment module calculates the distance between the two side edges of the belt and the corresponding side warning line, and then determines the direction of the belt deviation according to the relationship between the distance and the distance threshold. The process is: The distances between the center points of the warning lines on the left and right sides of the belt and the straight lines of the belt edges on the corresponding sides are calculated respectively, and are recorded as dwarnL and dwarnR. Then, when dwarnL<0 and |dwarnL|<L, a warning of the current left side deviation of the belt is prompted; wherein L represents the distance threshold; when dwarnL<0 and |dwarnL|>L, a warning of the current left side deviation of the belt is prompted; when dwarnR<0 and |dwarnR|<L, a warning of the current right side deviation of the belt is prompted; when dwarnR<0 and |dwarnR|>L, a warning of the current right side deviation of the belt is prompted.

10. An electronic device, characterized in that: It includes a computer program, which is stored in a computer-readable storage medium; when the processor of an electronic device reads the computer program from the computer-readable storage medium, the processor executes the computer program, so that the electronic device performs the steps of the belt deviation detection method based on multi-task learning as described in any one of claims 1-6.

Images