Belt deviation detection method and device
By using an improved lane detection network and optical flow deep neural network, image information is acquired in real time, which solves the shortcomings of existing belt misalignment detection methods in terms of accuracy, real-time performance and robustness. This achieves high-precision and fast belt misalignment detection, adapting to different detection scenarios.
Patent Information
- Application Number
- CN202210270615.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-03-18
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2042-03-18
AI Technical Summary
Existing belt misalignment detection methods have shortcomings in detection accuracy, real-time performance, and classification accuracy. They are also less robust, unable to achieve precise detection, and difficult to adapt to different detection scenarios.
An improved lane detection network and optical flow deep neural network are adopted. By training the belt edge detection model and the material motion optical flow detection model, image information is acquired in real time to determine the degree of belt deviation. Clustering algorithm and bilinear sputtering technology are used to improve detection accuracy and robustness.
It achieves high-precision and rapid belt misalignment detection, and can adapt to different lighting conditions, camera angles, and hardware equipment conditions, reducing detection costs and improving the robustness and real-time performance of the detection.
Smart Images

Figure CN114742864B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of belt deviation detection, and in particular to a belt deviation detection method, a belt deviation detection device and a machine-readable storage medium. Background Art
[0002] Belt conveyors are widely used in industries such as coal, ports, and electricity, especially in coal mining production, and are one of the main equipment for coal transportation. In the long-term transportation process, belt deviation failures often occur due to impact fatigue and various external factors. Therefore, belt edge detection is a technology with high demand and high standards in industrial production. At present, the mainstream belt deviation detection method adopts the traditional digital image processing technology in image detection, which mainly includes image preprocessing, threshold segmentation processing and edge discrimination. However, this detection method still has many defects in detection accuracy, real-time performance and classification accuracy. The selection of methods between the three processing steps needs to cooperate with each other and be adjusted continuously according to the actual experimental situation. It is difficult to promote it in the face of different detection situations, and end-to-end detection cannot be achieved; in addition, some parameters of preprocessing and image segmentation need to be set manually, which is not adaptive, the detection effect is poor, and it is difficult to improve the detection speed at the same time; the edge discrimination method design does not take into account the classification of boundary lines, and is easily disturbed by noise such as railing lines around the belt and edge lines of coal. The robustness is poor and accurate belt edge detection cannot be achieved. Summary of the invention
[0003] The purpose of the embodiments of the present invention is to provide a method and device for detecting equipment belt deviation, so as to solve the problems of detection accuracy, real-time performance and classification accuracy defects in the belt deviation detection process in the prior art, poor robustness and inability to achieve accurate detection.
[0004] In order to achieve the above object, an embodiment of the present invention provides a belt deviation detection method, wherein the belt is used to carry and transport materials, and the method includes:
[0005] The belt image training data set is input into the improved lane line detection network for model training to obtain the belt edge detection model;
[0006] The flying chair dataset is used to train the improved optical flow deep neural network model and obtain the material motion optical flow detection model;
[0007] Acquire a detection image in real time, wherein the detection image includes belt information and material information;
[0008] Inputting the detection image into a belt edge detection model and a material movement optical flow detection model to obtain a belt edge and a material movement optical flow edge;
[0009] The degree of belt deviation is determined based on the belt edge and the material movement optical flow edge.
[0010] Optionally, the improved lane line detection network is obtained by adding a non-local module to the lane line detection network, and the improved lane line detection network adopts the EfficientDet-D7 network infrastructure;
[0011] The improved optical flow deep neural network is obtained by improving the optical flow deep neural network using a clustering algorithm and a bilinear sputtering technique.
[0012] Optionally, the improved lane detection network includes: an input layer, a feature layer, a feature fusion layer, a convolutional layer, a non-local module and a fully connected layer;
[0013] The non-local module acts on the high-order semantic layer of the convolutional layer;
[0014] There are seven feature layers, and the number of channels are 2, 4, 8, 16, 32, 64, and 128 respectively.
[0015] Optionally, the improved optical flow deep neural network includes: an extraction module, a visual similarity calculation module, and an update iteration module;
[0016] The extraction module is used to extract image features of a first frame image and an adjacent second frame image of the detection image;
[0017] The visual similarity calculation module is used to output a correlation matrix based on the image features of the first frame image and the image features of the adjacent second frame image;
[0018] The update iteration module is used to iteratively update the optical flow field based on the correlation matrix, the last optical flow iteration result, the image features of the first frame image, and the image features of the adjacent second frame image.
[0019] Optionally, clustering algorithms and bilinear sputtering techniques are used to improve the optical flow deep neural network, including:
[0020] Using a k-clustering algorithm, the indexes of k points that best match the feature vector of the first frame image are determined from the feature vector of the adjacent second frame image, and a sparse visual similarity matrix is constructed based on the indexes of the k points;
[0021] Based on the latest optical flow iteration results and the sparse visual similarity matrix, the correlation matrix is obtained.
[0022] Optionally, the method further includes: determining an initial optical flow result, wherein the initial optical flow result is an optical flow result input during the first iteration, and the optical flow result is a non-zero value;
[0023] The determining of the initial optical flow result comprises:
[0024] Sort the similarity values in the sparse visual similarity matrix;
[0025] Determine an index corresponding to a similarity value that is ranked first, wherein the index corresponding to the similarity value includes a corresponding feature vector in a first frame image and a corresponding feature vector in an adjacent second frame image;
[0026] Determine the two-dimensional coordinates of the corresponding feature vector in the first frame image and the two-dimensional coordinates of the corresponding feature vector in the adjacent second frame image in the same coordinate system;
[0027] The two-dimensional coordinates of the corresponding feature vector in the first frame image are initialized based on the two-dimensional coordinates of the corresponding feature vector in the adjacent second frame image to obtain an initial optical flow result.
[0028] Optionally, the optical flow deep neural network is improved using clustering algorithms and bilinear sputtering techniques, including:
[0029] After distance weighting is performed on the points in the preset window among the top n similarity value sorts using the distance weighting method, the density of the correlation matrix is improved based on the bilinear sputtering technology.
[0030] Optionally, determining the degree of belt deviation based on the belt edge and the material movement optical flow edge includes:
[0031] Determine a standard straight line, wherein the standard straight line has a first intersection point and a second intersection point with the edge of the belt, and the standard straight line has a third intersection point and a fourth intersection point with the edge of the optical flow of the material movement;
[0032] The degree of belt deviation is determined based on the difference between the distance value between the first intersection point and the third intersection point and the distance value between the second intersection point and the fourth intersection point.
[0033] A second aspect of the present invention provides a belt deviation detection device, wherein the belt is used to carry and transport materials, and the device comprises:
[0034] The first modeling module is used to input the belt image training data set into the improved lane line detection network for model training to obtain a belt edge detection model;
[0035] The second modeling module is used to train the improved optical flow deep neural network model using the flying chair dataset to obtain a material motion optical flow detection model;
[0036] An acquisition module is used to acquire a detection image in real time, wherein the detection image includes belt information and material information;
[0037] A detection module, used for inputting the detection image into a belt edge detection model and a material motion optical flow detection model to obtain a belt edge and a material motion optical flow edge;
[0038] The output module is used to determine the degree of belt deviation based on the belt edge and the material movement optical flow edge.
[0039] On the other hand, the present invention provides a machine-readable storage medium having instructions stored thereon, the instructions being used to enable a machine to execute the above-mentioned method for detecting belt deviation of an equipment in the present application.
[0040] The present application obtains a belt edge detection model through improved lane line detection network training, which is used to obtain the belt edge; at the same time, a material movement optical flow detection model is obtained based on an improved optical flow deep neural network training, which is used to obtain the material movement optical flow edge; the degree of belt deviation is then confirmed by the obtained belt edge and material movement optical flow edge. The application is easy to debug and promote, and can be applied to belt deviation detection scenarios under different lighting, camera angles, and hardware equipment conditions. There is no need to reselect and debug methods according to actual conditions; and the detection accuracy is high, the detection speed is fast, the robustness is stronger, and the real-time performance is high.
[0041] Other features and advantages of the embodiments of the present invention will be described in detail in the subsequent detailed description. BRIEF DESCRIPTION OF THE DRAWINGS
[0042] The accompanying drawings are used to provide a further understanding of the embodiments of the present invention and constitute a part of the specification. Together with the following specific implementations, they are used to explain the embodiments of the present invention, but do not constitute a limitation on the embodiments of the present invention. In the accompanying drawings:
[0043] Figure 1 It is a flow chart of the equipment belt deviation detection method provided by the present invention;
[0044] Figure 2 is a schematic diagram of the structure of an improved lane detection network provided by the present invention;
[0045] Figure 3 It is a schematic diagram of the structure of the improved optical flow deep neural network provided by the present invention;
[0046] Figure 4 It is a schematic diagram of the positional relationship between the edge of the belt provided by the present invention and the edge of the optical flow of the material movement;
[0047] Figure 5 It is a structural schematic diagram of the equipment belt deviation detection device provided by the present invention.
[0048] Description of Reference Numerals
[0049] 10-first modeling module; 20-second modeling module; 30-acquisition module;
[0050] 40-detection module; 50-output module. DETAILED DESCRIPTION
[0051] The specific implementation of the embodiment of the present invention is described in detail below in conjunction with the accompanying drawings. It should be understood that the specific implementation described here is only used to illustrate and explain the embodiment of the present invention, and is not used to limit the embodiment of the present invention.
[0052] In the embodiments of the present invention, unless otherwise specified, directional words such as "up, down, left, right" generally refer to the directions or positional relationships shown in the accompanying drawings, or the directions or positional relationships in which the inventive product is usually placed when in use.
[0053] The terms “first”, “second”, “third”, etc. are only used for distinguishing descriptions and should not be understood as indicating or implying relative importance.
[0054] The terms "parallel", "vertical", etc. do not mean that the components must be absolutely parallel or vertical, but can be slightly tilted. For example, "parallel" only means that its direction is more parallel than "vertical", and does not mean that the structure must be completely parallel, but can be slightly tilted.
[0055] The terms "horizontal", "vertical", "overhanging" and the like do not mean that the components are required to be absolutely horizontal, vertical or overhanging, but can be slightly tilted. For example, "horizontal" only means that its direction is more horizontal than "vertical", and does not mean that the structure must be completely horizontal, but can be slightly tilted.
[0056] In addition, the terms "roughly" and "basically" are intended to explain that the relevant content does not require absolute precision, but can have certain deviations. For example, "roughly equal" does not only mean absolute equality. Since it is difficult to achieve absolute "equality" in actual production and operation, there are generally certain deviations. Therefore, in addition to absolute equality, "roughly equal" also includes the above-mentioned situation with certain deviations. Taking this as an example, in other cases, unless otherwise specified, the terms "roughly" and "basically" have similar meanings to the above.
[0057] In the description of the present invention, it is also necessary to explain that, unless otherwise clearly specified and limited, the terms "set", "install", "connect", and "connect" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a direct connection, or an indirect connection through an intermediate medium, or it can be the internal communication of two elements. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.
[0058] The present invention first obtains the belt transmission video data from a camera at a fixed position, performs preprocessing by image enhancement and other means, corrects the segmentation results of the Canny operator as data labels, and finally packages them into the data set required for network training and testing. The main algorithm adopted by the present invention is a dual-stream network based on improved PolylaneNet and RAFT. The results of the dual-stream network are integrated to calculate the distance difference between the edges of the belt and the edges of the moving materials, and judge whether the belt is biased. The basic network structure of PolylaneNet is replaced from EfficientNet-b1 to the best-performing EfficientDet-D7, and a non-local module is inserted in the high-level semantic layer to capture remote dependencies and improve the edge extraction accuracy; at the same time, the optical flow extraction performance of the RAFT network is improved by introducing a new optical flow initialization method, using the k-clustering algorithm to reduce the calculation amount of the visual similarity matrix, and using the bilinear sputtering technology instead of the linear interpolation method. The improved RAFT network effectively improves the robustness of the network by extracting motion information, and the improved PolylaneNet further improves the accuracy of belt deviation detection by extracting appearance information. This makes the present invention have the following advantages:
[0059] It is scalable and easy to debug. It eliminates the need for manual design of feature extraction and complicated manual parameter selection, and realizes end-to-end detection. It can be extended to belt deviation detection scenarios under different lighting, camera angles, and hardware equipment conditions, without the need to re-select and debug methods according to actual conditions.
[0060] High precision and high speed. Compared with the existing detection methods based on digital image processing, the dual-stream network used in this belt deviation detection method mines the potential features of belt video data, has richer scene semantic clues, and has higher precision. Deep learning methods can quickly adapt to different scenarios by fine-tuning, with faster model training speed, fewer parameters, faster calculation speed, and high computational efficiency.
[0061] Lower cost and strong robustness. The training and debugging of deep learning methods only require the use of GPUs, the test process consumes low energy, and the images used for training are relatively easy to obtain, so the method is low cost. Traditional methods work well in limited situations, but this belt deviation detection method is more robust. The RAFT network can effectively reduce noise interference such as static railings and coal edge lines. Facing targets of different sizes and distances, and detection scenarios under different conditions, the network can achieve good performance. The classification accuracy is higher.
[0062] Figure 1 It is a flow chart of the equipment belt deviation detection method provided by the present invention, such as Figure 1As shown, this embodiment provides a method for detecting deviation of a belt of an equipment, wherein the belt is used to carry and transport materials, and the method includes:
[0063] Step 101: input the belt image training data set into the improved lane line detection network for model training to obtain a belt edge detection model;
[0064] Step 102: Use the flying chair dataset to train the improved optical flow deep neural network model to obtain a material motion optical flow detection model;
[0065] Step 103: acquiring a detection image in real time, wherein the detection image includes belt information and material information;
[0066] Step 104, inputting the detection image into a belt edge detection model and a material movement optical flow detection model to obtain a belt edge and a material movement optical flow edge;
[0067] Step 105: Determine the degree of belt deviation based on the belt edge and the material movement optical flow edge.
[0068] Specifically, the lane detection network (PolyLaneNet: Lane Estimation via Deep PolynomialRegression), PolylaneNet was originally applied to the lane detection scenario of autonomous driving. It obtains input images from the front-view camera installed in the vehicle and outputs a polynomial representing each lane mark in the image, as well as the domain lane polynomial and the confidence score of each lane. Compared with other networks, it has significant improvements in speed and accuracy. Using the high-performance deep learning network PolylaneNet as the basis, the improved network can well meet the application requirements of belt edge detection after fine-tuning. Optical flow deep neural network (RAFT: Recurrent All-Pairs Field Transforms for Optical Flow, optical flow network with periodic full-pair field transformation) is an optical flow extraction algorithm based on a deep learning network. It has high extraction accuracy and fast speed, and the final motion optical flow is obtained through iterative updates. Among them, the flying chair dataset is a part of the optical flow dataset and is often used for optical flow estimation.
[0069] More specifically, the detection image is input into the belt edge detection model and the material movement optical flow detection model. Two identical detection images can be respectively input into the belt edge detection model and the material movement optical flow detection model. The belt edge is output by the belt edge detection model, and the material movement optical flow detection model outputs the material movement optical flow edge. The belt edge and the material movement optical flow edge are then combined into a picture to facilitate subsequent comparison and obtain the degree of belt deviation.
[0070] In another embodiment, a detection image can be output to the belt edge detection model and the material movement optical flow detection model in sequence, and the detection image can be input into the belt edge detection model to obtain the belt edge in the detection image. The detection image can then be input into the material movement optical flow detection model, and finally an image containing the belt edge and the material movement optical flow edge can be output. Subsequent comparisons can be performed directly on the image to obtain the degree of belt deviation.
[0071] The belt image training data set is a pre-collected image, and the image contains belt information and material information, and can be a collection of images captured from a video shot by a fixed camera.
[0072] Before training and actual detection, the acquired images can be preprocessed, including:
[0073] Step 1: Install a camera just above the belt conveyor, and transmit the video images acquired by the camera in real time through the network and rtsp (real-time streaming) protocol;
[0074] Step 2: Preprocess the video data, extract short video segments from high-pixel long videos, process them into video frames, and perform operations such as cropping, rotating, and translating the video frame images;
[0075] The basic theoretical support for step 2 is:
[0076] The quality of the training set plays an important role in deep learning. In industrial applications, it is common for data to be unevenly distributed and for noise points to exist in the image. Usually, the belt conveyor images obtained from surveillance videos have a single scene, and there are fewer samples with abnormalities (i.e., curved edges). When the ratio of the number of positive and negative sample images is unbalanced, the network will focus on learning the characteristics of a certain distribution, affecting the final performance. In addition, changes in lighting in the environment will also affect the quality of the image. The shape and edge of the conveyor belt, and the linear railings around the conveyor belt may all become noise points in the image. Data preprocessing operations can solve the above problems to a certain extent and improve the effect of deep learning.
[0077] The present invention proposes to implement the embodiment by performing the following processing steps based on this theory:
[0078] The proportions of positive and negative sample images are adjusted through operations such as copying to balance the data distribution and avoid insufficient learning of a certain feature by deep learning.
[0079] Perform data augmentation operations such as cropping, mirror flipping, adjusting color contrast, light intensity, and image angle on the original captured image to simulate belt edge images in different scenarios;
[0080] Linear noise points are randomly added to 1% of the images to make deep network learning more robust.
[0081] In summary, the original image data can be preprocessed through data balanced distribution, data augmentation, adding image noise points, etc., which can improve the segmentation effect of the deep learning network and make the method suitable for different application scenarios.
[0082] Step 3: Perform edge detection using the Canny operator and obtain the true values of n points on both sides of the belt edge after manual verification. At the same time, divide the processed data set into a training set and a validation set according to a reasonable ratio.
[0083] The present invention performs data set annotation and division, and step 3 includes the following sub-steps:
[0084] 3.1. After unifying the format of the dataset images, the belt edge is detected using the Canny operator. The height of the output image is divided into n equal parts and intersected with the two belt edge lines detected to obtain 2*n edge detection point horizontal coordinates with equidistant vertical coordinates. After verification, they are used as data labels to train the improved PolylaneNet network.
[0085] 3.2. To achieve supervised learning, we expand the dataset and use uniform random sampling to divide it into training set, validation set and test set in a ratio of 8:1:1, while keeping them identically distributed.
[0086] 3.3. Package specific data sets for different application scenarios. After loading the trained original network weights, train the network with the data set of the specific scenario and adjust the network weights and other parameters and hyperparameters.
[0087] The labeling method saves labor costs. The division of the data set uses a common standard. The packaging of the fine-tuning data set is the same as that of the coarse-tuning data set. When faced with changing scenarios, it can still show a certain degree of excellence.
[0088] The specific calculation steps are detailed as follows:
[0089] After obtaining the belt monitoring video, the video is frame sampled using a uniform sampling method as the original image of the training set;
[0090] Perform operations such as balancing data distribution, data augmentation, and adding noise points on the acquired images;
[0091] The input of the neural network requires a unified data format. After preprocessing, the data is normalized and cropped to a fixed pixel ratio of 224*224;
[0092] The Canny operator is used to mark the belt edge in the image. The specific process is to first convert the image into a grayscale image, and then use Gaussian filtering to smooth the grayscale image.
[0093] The Gaussian filter function is:
[0094]
[0095] Among them, i ≥ 1, j ≤ (2k + 1), σ 2 is the variance of the Gaussian distribution, e is the natural base, and σ = 1.4, we can get the Gaussian convolution kernel of size 3×3 as:
[0096]
[0097] That is, for a pixel with position (m,n) and grayscale value f(m,n), the grayscale value g(m,n) after Gaussian filtering is the convolution value of the grayscale value of the 3×3 matrix around (m,n) and H. After calculating the gradient value and gradient direction of the smoothed grayscale image, non-maximum values are filtered, and finally upper and lower thresholds are used to detect edges.
[0098] The center points of the above stripes were verified so that the edge detection points were within an acceptable error range (1 cm). The data set was encapsulated as the final data label and the obtained data was divided into training set, validation set and test set in a ratio of 8:1:1.
[0099] Furthermore, the improved lane line detection network is obtained by adding a non-local module to the lane line detection network, and the improved lane line detection network adopts the EfficientDet-D7 network infrastructure;
[0100] Furthermore, the improved lane detection network includes: an input layer, a feature layer, a feature fusion layer, a convolutional layer, a non-local module and a fully connected layer;
[0101] The non-local module acts on the high-order semantic layer of the convolutional layer;
[0102] There are seven feature layers, and the number of channels are 2, 4, 8, 16, 32, 64, and 128 respectively.
[0103] Step 4: Specifically, the improved PolylaneNet network construction of the present invention includes the following sub-steps:
[0104] 4.1. In order to improve the performance of deep learning networks, the basic network of PolylaneNet is replaced by the more superior EfficientDet-D7 from EfficientNet-b1, that is, the coefficients of EfficientNet-b1 are first amplified, and then BiFPN (multi-scale feature fusion) and Compound Scaling are added;
[0105] EfficientDet-D7 still uses the original network framework EfficientNet-b0, which is the same as EfficientNet-b1. It uses a higher network magnification coefficient in the selection of the mixing coefficient to obtain EfficientNet-b7. EfficientNet-b7 has a large amount of calculation, but its accuracy is far higher than EfficientNet-b1, which plays a significant role in accurate feature extraction.
[0106] EfficientDet-D7 adds BiFPN and CompoundScaling to EfficientNet-b7. BiFPN is a feature fusion module added to the original EfficientNet-b7. The previous feature fusion treated features of different scales equally, while BiFPN amplified the influence of the target scale and learned the importance of different input features through the selection and setting of weights. When considering model expansion, EfficientNet emphasized the three elements of network width, depth and resolution, while EfficientDet further expanded it and used EfficientNet as the backbone network to control the scale of the backbone network. At the same time, the number of channels of the added BiFPN, the number of repeated layers, the number of layers in the head part, the resolution of the input image, etc. are controlled. These constitute the extended configuration parameters of EfficientDet, called Compound Scaling.
[0107] 4.2. In order to enable the model to have the function of belt edge detection, the present invention improves the output of the network by changing the polynomial coefficients, vertical offset and prediction confidence score output by the original fully connected layer into a tensor of size 2*n, so that the number of output nodes matches the edge detection points on both sides of the belt;
[0108] 4.3. Add a non-local module (non-local neural network module) to the high-order semantic layer of the improved PolylaneNet network in step 4.2. It directly captures long-range dependencies by calculating the interaction between any two unknowns, without being limited to adjacent points. It maintains more information through eigenvalue weighting operations. Inserting a non-local module improves the effects of visual tasks such as image classification, object detection, and object segmentation to varying degrees. The non-local module is calculated in the following way:
[0109]
[0110]
[0111]
[0112] Among them, y i represents the feature vector of the ith spatial position of a feature map of the deep learning network after non-local processing, x i With x j represents the feature vector of the i-th and j-th spatial positions on the corresponding original feature map, where i,j∈(1,H*W), H and W are the width and height of the input feature map respectively; f(x i ,x j ) is a function for calculating the similarity between any two points; θ(x i )and are all convolution operations; C(x) is the normalization factor, e is a mathematical constant; g(x j ) is the mapping function, g(x j )∈R HW*Cg .
[0113] In the embodiment of the present invention, g(x j ) is simplified to a linear embedding, that is, g(x j )=W g ·x j , where W g is a weight matrix to be learned, and the final feature y i The number of channels is C g .
[0114] Figure 2 : is a schematic diagram of the structure of the improved lane detection network provided by the present invention. In the embodiment, the core network EfficientDet-D7 is used to improve the network output in the belt detection scenario. Figure 2 The meanings of each part are as follows:
[0115] The video frames will first be input into the basic network structure of EfficientNet for feature extraction, where P1-P7 are the bottom-up features extracted by the convolutional layer, and the number of feature layer channels are 2, 4, 8, 16, 32, 64, and 128 respectively.
[0116] Level 3-7 features {P3, P4, P5, P6, P7} are extracted from the basic network structure, and top-down and bottom-up bidirectional feature fusion are repeatedly applied, i.e., the middle BiFPN layer. The number of repetitions weighs the accuracy and speed of the model. The present invention repeats it three times, i.e., three BiFPN layers are used.
[0117] These fused features are used as the input of the following three convolutional layers. In actual operation, the embodiment of the present invention inserts the non-local module into these three high-order semantic layers and converts f(x i ,x j ) is calculated as matrix multiplication and convolution. The non-local module operates on the feature layer and can be easily inserted into any network. In the embodiment, the high-order semantic layer is preferably inserted to reduce the amount of calculation.
[0118] The present invention designs the final fully connected layer output as a tensor of size 2*n, which represents the horizontal coordinates of the edge points detected by the network. In the training phase, the horizontal coordinates of the key points of the belt edge are used as supervision to complete the training. In the inference phase, the least squares method is used for fitting, and the straight line obtained by fitting is used as the final belt edge prediction result.
[0119] Furthermore, the improved optical flow deep neural network is obtained by improving the optical flow deep neural network using a clustering algorithm and a bilinear sputtering technique.
[0120] Figure 3 is a schematic diagram of the structure of the improved optical flow deep neural network provided by the present invention, such as Figure 3 As shown, further, the improved optical flow deep neural network includes: an extraction module, a visual similarity calculation module, and an update iteration module;
[0121] The extraction module is used to extract image features of a first frame image and an adjacent second frame image of the detection image;
[0122] The visual similarity calculation module is used to output a correlation matrix based on the image features of the first frame image and the image features of the adjacent second frame image;
[0123] The update iteration module is used to iteratively update the optical flow field based on the correlation matrix, the last optical flow iteration result, the image features of the first frame image, and the image features of the adjacent second frame image.
[0124] Furthermore, clustering algorithm and bilinear sputtering technology are used to improve the optical flow deep neural network, including:
[0125] Using a k-clustering algorithm, the indexes of k points that best match the feature vector of the first frame image are determined from the feature vector of the adjacent second frame image, and a sparse visual similarity matrix is constructed based on the indexes of the k points;
[0126] Based on the latest optical flow iteration results and the sparse visual similarity matrix, the correlation matrix is obtained.
[0127] Furthermore, the method further comprises: determining an initial optical flow result, wherein the initial optical flow result is an optical flow result inputted in the first iteration, and the optical flow result is a non-zero value;
[0128] The determining of the initial optical flow result comprises:
[0129] Sort the similarity values in the sparse visual similarity matrix;
[0130] Determine an index corresponding to a similarity value that is ranked first, wherein the index corresponding to the similarity value includes a corresponding feature vector in a first frame image and a corresponding feature vector in an adjacent second frame image;
[0131] Determine the two-dimensional coordinates of the corresponding feature vector in the first frame image and the two-dimensional coordinates of the corresponding feature vector in the adjacent second frame image in the same coordinate system;
[0132] The two-dimensional coordinates of the corresponding feature vector in the first frame image are initialized based on the two-dimensional coordinates of the corresponding feature vector in the adjacent second frame image to obtain an initial optical flow result.
[0133] Furthermore, the optical flow deep neural network is improved by using clustering algorithm and bilinear sputtering technology, including:
[0134] After distance weighting is performed on the points in the preset window among the top n similarity value sorts using the distance weighting method, the density of the correlation matrix is improved based on the bilinear sputtering technology.
[0135] In the present invention, the points located in the preset window among the first n points in the similarity value sorting are selected for weighting. In order to achieve the unification of the front and back, n is set to k.
[0136] Specifically, the present invention improves the optical flow deep neural network, including the following sub-steps:
[0137] The improved RAFT network will be used to extract the optical flow map of the material movement in the belt conveyor. At the same time, it can forcefully distinguish static noise and improve the accuracy of classification. The RAFT network is mainly composed of a feature / context extraction module, a visual similarity calculation module, and an update iteration module. The function of the context extraction module is to extract features. Its input is two adjacent frames of RGB (color channel) images. The structure is similar to the feature extraction network, but a different normalization layer is selected to output a feature map (H*W) with a resolution of 1 / 8 of the original image. The visual similarity calculation module takes the features of the two frames of images as input and outputs a correlation matrix; the calculation is divided into two steps. The first step is to calculate the visual similarity between the two video frames. It matches each feature vector of the feature map of the previous frame with all the feature vectors of the feature map of the adjacent frame as the inner product to obtain a visual similarity matrix (H*W*H*W). Finally, pooling operations are performed to obtain images of different resolutions. The visual similarity matrix is used to obtain overall and local features. The role of the context extraction network is still to extract features. Its structure is similar to that of the feature extraction network, but a different normalization layer is selected and only the features of the first frame image are extracted. Second, according to the latest optical flow iteration results, correlation search is performed in the calculated multi-scale 4D correlation body to obtain the correlation matrix. The main body of the network iteratively updates the optical flow field through the GRU (Gated Recurrent Unit). The input of each iteration is the connection of the correlation matrix, the optical flow results obtained in the previous iteration, and the context features (image features of the first frame image, image features of the adjacent second frame image) after convolution upsampling.
[0138] In the first iteration, the initial value of the optical flow is 0, so the matrix corresponding to the current pixel in the visual similarity matrix is directly searched. In subsequent iterations, an optical flow displacement must be applied to the current point first. Since the iteration result is not necessarily an integer, we need to complete the interpolation calculation of the four similarity values on the grid where the displaced point is located before performing the above search operation to obtain 4 correlation matrices (H*W). In RAFT, the optical flow is calculated with full precision from the initialization, and there is no pyramid upsampling process in previous methods, so it has obvious advantages in the fast movement of small objects.
[0139] The iteration formula is as follows:
[0140] z t =σ(Conv 3×3 ([h t-1 ,x t ],W z ))
[0141] r t =σ(Conv 3×3 ([h t-1 ,x t ],W r ))
[0142]
[0143]
[0144] Among them, h t-1 is the latest optical flow iteration result of the previous round, x t Yes t-1 , the connection between the correlation matrix and the context features. In the first iteration, the initial value of the optical flow is 0, Conv 3×3 is a convolution of size 3x3, σ represents the sigmoid activation function, W z , W r , W h Both represent weights. The result of this round of iteration h t It is the residual optical flow relative to the output optical flow of the last iteration, which is added to the optical flow of the last iteration to generate the full-resolution optical flow result f t , which is used to calculate the loss function. The RAFT model performs 12 iterations of optimization in total, and the 12 optical flow results generated are gradually refined. t And the true value of optical flow f gt The L1 norm of is multiplied by the exponentially growing weight as the loss function to update the model parameters, as shown in the following formula:
[0145]
[0146] Specifically, in subsequent iterations, we need to first apply an optical flow displacement to the current point. Since the iteration result is not necessarily an integer, we need to interpolate the four similar values on the grid where the displaced point is located before performing the above search operation to obtain 4 correlation matrices (H*W). In RAFT, the optical flow is calculated with full precision from the initialization, and there is no pyramid upsampling process in previous methods. Therefore, it has obvious advantages in the fast movement of small objects.
[0147] Step 5: To meet the requirements of belt material motion detection for real-time performance and capturing fine structure motion, the present invention makes the following improvements to the RAFT network:
[0148] 5.1. When calculating the visual similarity matrix, the present invention uses the k-clustering algorithm to select the indexes of the k points that best match the feature vectors of the first frame image from the H*W feature vectors of the feature map of the second frame, and only calculates the inner product of these k points and stores them to construct a sparse visual similarity matrix (H*W*k). When performing back propagation, the gradient is only propagated to the k feature vectors selected by the k-clustering module. The space complexity of the coefficient visual similarity matrix that only stores the top-k similarity values is reduced from O(N*W) to O(N*W).2 ) is reduced to O(N), solving the problem of the original RAFT model calculation in which memory consumption grows quadratically with the number of pixels, and to a certain extent freeing up its ability to process high-resolution images.
[0149] 5.2. This invention proposes a new optical flow initialization method. In the similarity search, the index (i, j) of each similarity value in the sparse visual similarity matrix with the top-1 is recorded, where i represents the i-th feature vector of the first video frame feature map F1, and j represents the j-th feature vector of the second video frame feature map F2. The two-dimensional coordinates (x1, x2) and (y1, y2) of the two feature vectors are located respectively, and the optical flow at the i-th position of the first video frame feature map is initialized to (y1-x1, y2-x2). By a priori avoiding falling into the local minimum, a clear direction is provided for the subsequent iteration of the optical flow of small targets.
[0150] 5.3. When scaling the visual similarity matrix, the corresponding k index positions may become non-integer. The present invention uses bilinear sputtering technology to increase the density of the correlation matrix, wherein the similarity value at a non-grid position is bilinearly dispersed to the four nearest integer grids according to the distance. The bilinear sputtering technology changes the matrix dimension and rearranges the values of r points within a fixed radius to form a dense tensor, which not only stores the similarity value weight information of the current iterative optical flow, but also stores the position information through a specific arrangement method, which gives the next iteration direction of the optical flow to a certain extent.
[0151] The improved RAFT network in this embodiment adopts distance weighting. In the process of tensor dimension reshaping in the bilinear sputtering technology, the top-k (ranked before k) points that are not in the preset window (the preset window is a circle with a radius of r, and the center of the circle is the point ranked first in the similarity value) are directly discarded. Such a simple and direct method may cause the loss of information of fast-moving small targets. To this end, this paper weights the top-k (ranked before k) points within a fixed radius r according to the distribution and similarity of the top-k points outside the fixed radius r to speed up iterations. Let a top-k (ranked before k) point within a fixed radius r be x, a top-k point outside the fixed radius r be y, and the Euclidean distance between the two points be d. Then use the Gaussian function to consider the influence of distance on the weight. The formula is as follows:
[0152]
[0153] We record the similarity value of point y as a, then the weight of y to x is:
[0154] X y =1+a*X yd
[0155] The larger the weight, the closer the distance between y and x is or the larger the similarity value of y is. In theory, both cases require that the light flow within a fixed radius r moves toward the x direction that is closer to y. We weight all the top-k (sorted before k) x points within a fixed radius r (preset window) and then perform a bilinear sputtering operation.
[0156] Specifically, when using the Faiss framework for similarity search, the metric can be changed from cosine similarity to Euclidean distance.
[0157] The coordinates of each point in the visual similarity matrix are scaled by 5 different constant coefficients to form a multi-scale sparse correlation pyramid. Then, we bilinearly split the similarity values onto an integer grid and extract correlation values at different resolutions within a local window (i.e., fixed radius r) based on the optical flow result of the previous iteration. A larger resolution provides broader contextual semantic information, and a smaller resolution provides more precise displacement. Finally, we extract the 5*(2r+1) 2 Convert similarity value tensors into dense tensors, change their dimensions and concatenate them to form a tensor of size H*W*c.
[0158] When scaling the visual similarity matrix, the corresponding k index positions may become non-integer, so we need to use bilinear sputtering to increase the density of the correlation matrix, where the similarity values at a non-grid position are bilinearly scattered to the four nearest integer grids according to the distance.
[0159] In addition, the training samples are used as input to train the improved PolylaneNet network to obtain the network weights and form a belt edge detection model for detection. The improved RAFT network is trained with a general flying chair dataset to obtain the network weights, which are then used to extract the optical flow of belt conveyor material movement. This step uses the validation set to measure indicators such as model accuracy.
[0160] Use TensorRT (GPU acceleration tool) to lightweight the model, and select appropriate fine-tuning datasets to train the above model to suit different scenarios.
[0161] The new belt video image test sample is used as the above-trained belt deviation detection model to output the belt edge extraction and classification results in the image; at the same time, the material movement optical flow is output
[0162] Further, based on the belt edge and the material movement optical flow edge, the belt deviation degree is determined, including:
[0163] Determine a standard straight line, wherein the standard straight line has a first intersection point and a second intersection point with the edge of the belt, and the standard straight line has a third intersection point and a fourth intersection point with the edge of the optical flow of the material movement;
[0164] The degree of belt deviation is determined based on the difference between the distance value between the first intersection point and the third intersection point and the distance value between the second intersection point and the fourth intersection point.
[0165] Figure 4 Schematic diagram of the positional relationship between the edge of the belt provided by the present invention and the edge of the optical flow of the material movement, such as Figure 4 As shown, specifically, the standard straight line intersects the edges on both sides of the belt at points a1 and a4, and intersects the optical flow edge of the material movement at points a2 and a3, and the two distances between the edge of the material movement optical flow and the edge of the belt are calculated respectively: d1=a2-a1, d2=a4-a3; (using d1=a3-a1, d2=a4-a2 can also obtain the two distances between the edge of the material movement optical flow and the edge of the belt).
[0166] The distance difference between the two distances is: Δd=|d1-d2|. If Δd is greater than a set threshold, it is determined that the belt is deviating, and Δd is the degree of deviation.
[0167] In another embodiment, the deviation direction of the belt may be determined based on the positive and negative values of the formula Δd=d1-d2.
[0168] The threshold value can also be replaced by a threshold interval. When the distance difference Δd is greater than or less than the value in the threshold interval, the belt is judged to be deviated, and the degree of belt deviation is determined based on the distance difference Δd and the maximum value or minimum value in the threshold interval.
[0169] The standard straight line can be set as an inclined straight line or a relatively parallel line. Preferably, the angle between the standard straight line and the belt edge is equal or the difference between the angles between the standard straight line and the belt edge is less than a preset value.
[0170] Figure 5 Schematic diagram of the structure of the equipment belt deviation detection device provided by the present invention, such as Figure 5 As shown, the second aspect of the present invention provides a belt deviation detection device, wherein the belt is used to carry and transport materials, and the device comprises:
[0171] A first modeling module 10 is used to input the belt image training data set into the improved lane line detection network for model training to obtain a belt edge detection model;
[0172] The second modeling module 20 is used to train the improved optical flow deep neural network model using the flying chair data set to obtain a material motion optical flow detection model;
[0173] An acquisition module 30 is used to acquire a detection image in real time, wherein the detection image includes belt information and material information;
[0174] A detection module 40, used for inputting the detection image into a belt edge detection model and a material movement optical flow detection model to obtain a belt edge and a material movement optical flow edge;
[0175] The output module 50 is used to determine the degree of belt deviation based on the belt edge and the material movement optical flow edge.
[0176] A third aspect of the present invention provides a machine-readable storage medium having instructions stored thereon, the instructions being used to enable a machine to execute the belt deviation detection method described above in the present application.
[0177] The optional implementation modes of the embodiments of the present invention are described in detail above in conjunction with the accompanying drawings. However, the embodiments of the present invention are not limited to the specific details in the above implementation modes. Within the technical concept of the embodiments of the present invention, various simple modifications can be made to the technical scheme of the embodiments of the present invention, and these simple modifications all belong to the protection scope of the embodiments of the present invention.
[0178] It should also be noted that the various specific technical features described in the above specific embodiments can be combined in any suitable manner without contradiction. To avoid unnecessary repetition, the embodiments of the present invention will not further describe various possible combinations.
[0179] Those skilled in the art can understand that all or part of the steps in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a program, and the program is stored in a storage medium, including a number of instructions to enable a single-chip microcomputer, a chip or a processor to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.
[0180] In addition, various implementation modes of the embodiments of the present invention may be arbitrarily combined, and as long as they do not violate the concept of the embodiments of the present invention, they should also be regarded as the contents disclosed by the embodiments of the present invention.
Claims
1. A method for detecting belt deviation, wherein the belt is used to carry and transport materials, characterized in that: The method comprises: Inputting the belt image training data set into the improved lane line detection network for model training to obtain a belt edge detection model; the improved lane line detection network is obtained by adding a non-local module to the lane line detection network, the improved lane line detection network adopts the EfficientDet-D7 network infrastructure, and the fully connected layer output is a 2*n tensor; The improved optical flow deep neural network is trained using the flying chair data set to obtain a material movement optical flow detection model; the improved optical flow deep neural network is obtained by improving the optical flow deep neural network using a clustering algorithm and a bilinear sputtering technique; Acquire a detection image in real time, wherein the detection image includes belt information and material information; Inputting the detection image into a belt edge detection model and a material movement optical flow detection model to obtain a belt edge and a material movement optical flow edge; The degree of belt deviation is determined based on the belt edge and the material movement optical flow edge.
2. The belt deviation detection method according to claim 1, characterized in that: The improved lane detection network includes: an input layer, a feature layer, a feature fusion layer, a convolutional layer, a non-local module and a fully connected layer; The non-local module acts on the high-order semantic layer of the convolutional layer; There are seven feature layers, and the number of channels are 2, 4, 8, 16, 32, 64, and 128 respectively.
3. The belt deviation detection method according to claim 1, characterized in that: The improved optical flow deep neural network includes: an extraction module, a visual similarity calculation module, and an update iteration module; The extraction module is used to extract image features of a first frame image and an adjacent second frame image of the detection image; The visual similarity calculation module is used to output a correlation matrix based on the image features of the first frame image and the image features of the adjacent second frame image; The update iteration module is used to iteratively update the optical flow field based on the correlation matrix, the last optical flow iteration result, the image features of the first frame image, and the image features of the adjacent second frame image.
4. The belt deviation detection method according to claim 3, characterized in that: The optical flow deep neural network is improved by using clustering algorithm and bilinear sputtering technology, including: Using a k-clustering algorithm, the indexes of k points that best match the feature vector of the first frame image are determined from the feature vector of the adjacent second frame image, and a sparse visual similarity matrix is constructed based on the indexes of the k points; Based on the latest optical flow iteration results and the sparse visual similarity matrix, the correlation matrix is obtained.
5. The belt deviation detection method according to claim 4, characterized in that: The method further includes: determining an initial optical flow result, wherein the initial optical flow result is an optical flow result input during the first iteration, and the optical flow result is a non-zero value; The determining of the initial optical flow result comprises: Sort the similarity values in the sparse visual similarity matrix; Determine an index corresponding to a similarity value that is ranked first, wherein the index corresponding to the similarity value includes a corresponding feature vector in a first frame image and a corresponding feature vector in an adjacent second frame image; Determine the two-dimensional coordinates of the corresponding feature vector in the first frame image and the two-dimensional coordinates of the corresponding feature vector in the adjacent second frame image in the same coordinate system; The two-dimensional coordinates of the corresponding feature vector in the first frame image are initialized based on the two-dimensional coordinates of the corresponding feature vector in the adjacent second frame image to obtain an initial optical flow result.
6. The belt deviation detection method according to claim 5, characterized in that: Improvements to the optical flow deep neural network using clustering algorithms and bilinear sputtering techniques also include: After distance weighting is performed on the points in the preset window among the top n similarity value sorts using the distance weighting method, the density of the correlation matrix is improved based on the bilinear sputtering technology.
7. The belt deviation detection method according to claim 1, characterized in that: Based on the belt edge and the material movement optical flow edge, the belt deviation degree is determined, including: Determine a standard straight line, wherein the standard straight line has a first intersection point and a second intersection point with the edge of the belt, and the standard straight line has a third intersection point and a fourth intersection point with the edge of the optical flow of the material movement; The degree of belt deviation is determined based on the difference between the distance value between the first intersection point and the third intersection point and the distance value between the second intersection point and the fourth intersection point.
8. A belt deviation detection device, wherein the belt is used to carry and transport materials, characterized in that: The device comprises: A first modeling module is used to input the belt image training data set into the improved lane line detection network for model training to obtain a belt edge detection model; the improved lane line detection network is obtained by adding a non-local module to the lane line detection network, the improved lane line detection network adopts the EfficientDet-D7 network infrastructure, and the fully connected layer output is a 2*n tensor; The second modeling module is used to train the improved optical flow deep neural network using the flying chair data set to obtain a material movement optical flow detection model; the improved optical flow deep neural network is obtained by improving the optical flow deep neural network using a clustering algorithm and a bilinear sputtering technology; An acquisition module is used to acquire a detection image in real time, wherein the detection image includes belt information and material information; A detection module, used for inputting the detection image into a belt edge detection model and a material motion optical flow detection model to obtain a belt edge and a material motion optical flow edge; The output module is used to determine the degree of belt deviation based on the belt edge and the material movement optical flow edge.
9. A machine-readable storage medium having instructions stored thereon, the instructions being used to enable a machine to execute the belt deviation detection method described in any one of claims 1 to 7 of the present application.
