Conveyor belt deviation visual detection method, device, electronic equipment and storage medium

Abstract

The present invention relates to a method, device, electronic device, and storage medium for visually detecting conveyor belt deviation. The method comprises: acquiring a real-time conveyor belt transport image; inputting the real-time conveyor belt transport image into a fully trained image segmentation model to extract an image covering the conveyor belt boundary curve; determining a first centerline and a second centerline of the image covering the conveyor belt boundary curve, and calculating the pixel distance between the first and second centerlines; and determining the conveyor belt deviation distance corresponding to the pixel distance based on a preset coordinate transformation relationship. This improves detection accuracy and safety.

Description

Technical Field

[0001] The present invention relates to the field of image recognition technology, and in particular to a method, device, electronic equipment and storage medium for visually detecting conveyor belt deviation. Background Art

[0002] In modern transportation production, belt conveyors are suitable for the continuous transport of bulk materials. Their advantages include large transport volumes, long transport distances, low transport energy consumption, and convenient loading and unloading. They are widely used in mining, metallurgy, electricity, chemical industry, and grain industries. With the continuous expansion of production demand, the carrying capacity of conveyor belts has continued to increase, bringing huge economic benefits to the material transportation industry. However, as a key component for traction and load-bearing, the conveyor belt accounts for 40% to 60% of the total cost of the belt conveyor. The conveyor belt can reach up to several thousand meters in length, and its operating status directly affects the stability of the entire production process.

[0003] Belt deviation is one of the most common faults encountered during the actual operation of a belt conveyor, and can include partial deviation, overall deviation, and intermittent deviation. Deformation and aging due to long-term wear and tear, as well as uneven material distribution on the belt, can easily lead to asymmetric stress distribution within the belt during operation, causing the belt to deviate from its original longitudinal centerline, a phenomenon known as belt deviation. During high-speed operation, mild belt deviation can easily cause aging, deformation, and offset wear; moderate belt deviation can easily lead to material overflow, exacerbating friction damage to the belt support rollers and increasing cleaning costs; severe belt deviation can cause large-scale belt tearing, requiring the entire conveyor line to shut down and require replacement, and even result in casualties.

[0004] Currently, companies typically rely on regular manual inspections to detect conveyor belt deviation during production. However, the conveyor belt status estimates provided by inspectors are insufficient to generate reliable data, and statistical analysis of the safety status of the production process is impossible. Furthermore, manual inspections are risky, inefficient, labor-intensive, and lack real-time monitoring capabilities. The repetitive, mechanical work also causes significant fatigue for workers. By the time inspectors discover a conveyor belt failure, it has already caused serious accidents, such as extensive belt tears. Furthermore, inspectors require long-term training, leading to increasing personnel and management costs. Therefore, finding real-time detection of conveyor belt deviation to provide reliable transmission data is a pressing issue. Summary of the Invention

[0005] In view of this, it is necessary to provide a conveyor belt deviation visual detection method, device, electronic equipment and storage medium to detect the conveyor belt deviation distance in real time and improve transportation safety.

[0006] In order to achieve the above objectives, the present invention provides a method for visually detecting conveyor belt deviation, comprising:

[0007] Get real-time transport images of conveyor belts;

[0008] Inputting the real-time transport image of the conveyor belt into a well-trained image segmentation model to extract an image of a curve covering the conveyor belt boundary;

[0009] Determining a first center line and a second center line of the covered conveyor belt boundary curve image, and calculating a pixel distance between the first center line and the second center line;

[0010] Based on a preset coordinate conversion relationship, a conveyor belt offset distance corresponding to the pixel distance is determined.

[0011] Furthermore, the fully trained image segmentation model is trained based on the improved YOLACT network. The training process of the fully trained image segmentation model includes:

[0012] Collecting historical transport images of the conveyor belt, annotating the historical transport images, and constructing a training set and a test set based on the annotated images;

[0013] Inputting the training set into the backbone network in the improved YOLACT network to extract initial feature information;

[0014] Combining the backbone network and the improved FPN feature pyramid structure, the initial feature information is fused based on a preset fusion method to obtain effective feature information;

[0015] jointly calculating the first effective feature information and the second effective feature information in the effective feature information, and outputting a first covering conveyor belt boundary curve image;

[0016] The loss calculation is performed using the first covered conveyor belt boundary curve image and the second covered conveyor belt boundary curve image in the validation set. If the loss function meets the preset loss condition, the image segmentation model is fully trained.

[0017] Furthermore, the backbone network and the improved FPN feature pyramid structure are combined to fuse the initial feature information based on a preset fusion method to obtain effective feature information, including:

[0018] The initial feature information is fused based on a top-down fusion method to obtain intermediate feature information;

[0019] The first initial feature information in the initial feature information is used as a basis for bottom-up fusion, and is combined with the intermediate feature information to perform bottom-up fusion to obtain the effective feature information.

[0020] Furthermore, the first center line of the image covering the conveyor belt boundary curve is the center line of the conveyor belt machine;

[0021] Determining a second center line of the covered conveyor belt boundary curve image includes:

[0022] An image coordinate system is established with the center of the covered conveyor belt boundary curve image as the origin and the first center line as the first direction coordinate axis;

[0023] The conveyor belt width of the covered conveyor belt boundary curve image in the second direction of the image coordinate system is determined, and the center point coordinate set of the conveyor belt width is used as the second center line.

[0024] Furthermore, calculating the pixel distance between the first center line and the second center line includes:

[0025] Pixel coordinates of the first center line and the second center line in the image coordinate system are respectively determined, and a pixel distance between the first center line and the second center line is determined according to the pixel coordinates.

[0026] Furthermore, the preset coordinate transformation relationship includes the transformation relationship between the image coordinate system, the camera coordinate system and the world coordinate system.

[0027] Furthermore, the determining of the conveyor belt offset distance corresponding to the pixel distance based on a preset coordinate conversion relationship includes:

[0028] The conveyor belt offset distance corresponding to the pixel distance is determined based on a proportional relationship included in the preset coordinate conversion relationship, wherein the proportional relationship is determined by the height of the camera from the conveyor belt, camera parameters, pixel width and pixel length.

[0029] In a second aspect, the present invention further provides a conveyor belt deviation visual detection device, comprising:

[0030] An acquisition module is used to acquire real-time transport images of the conveyor belt;

[0031] An extraction module is used to input the real-time transport image of the conveyor belt into a well-trained image segmentation model to extract an image covering the conveyor belt boundary curve;

[0032] a calculation module, configured to determine a first center line and a second center line of the image covering the conveyor belt boundary curve, and calculate a pixel distance between the first center line and the second center line;

[0033] The determination module is used to determine the conveyor belt offset distance corresponding to the pixel distance based on a preset coordinate conversion relationship.

[0034] In a third aspect, the present invention also provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps in the above-mentioned method for visually detecting conveyor belt deviation are implemented.

[0035] In a fourth aspect, the present invention further provides a computer storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the above-mentioned method for visually detecting conveyor belt deviation are implemented.

[0036] The present invention can segment an image covering the conveyor belt boundary by training a complete image segmentation model, which is convenient for controlling the overall offset of the conveyor belt; and by extracting the two center lines of the curve image covering the conveyor belt boundary, and then using the coordinate transformation relationship to determine the offset distance of the conveyor belt, real-time and accurate detection is achieved, and the safety of the detection is also guaranteed. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 A schematic flow chart of an embodiment of a method for visually detecting conveyor belt deviation provided by the present invention;

[0038] Figure 2 A reference image covering a conveyor belt boundary curve image provided by an embodiment of the present invention;

[0039] Figure 3 A flowchart of an image segmentation model training process provided by one embodiment of the present invention;

[0040] Figure 4 A schematic diagram of the structure of an image segmentation model provided by one embodiment of the present invention;

[0041] Figure 5 A diagram of the FPN calculation process provided by one embodiment of the present invention;

[0042] Figure 6 A conversion diagram between camera coordinates and world coordinates provided by an embodiment of the present invention;

[0043] Figure 7 A schematic structural diagram of an embodiment of a conveyor belt deviation visual detection device provided by the present invention;

[0044] Figure 8 The present invention provides a schematic structural diagram of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0045] The preferred embodiments of the present invention will be described in detail below in conjunction with the accompanying drawings, wherein the accompanying drawings constitute a part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, and are not used to limit the scope of the present invention.

[0046] In the description of the present invention, "multiple" means two or more, unless otherwise specifically defined. Reference to "embodiments" herein means that the specific features, structures, or characteristics described in conjunction with the embodiments may be included in at least one embodiment of the present invention. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0047] The present invention proposes to use image recognition technology to monitor the position of the conveyor belt of the belt conveyor in real time. By identifying and locating the material status and position of the conveyor belt, the risk degree of the conveyor belt deviation is comprehensively evaluated, thereby realizing the online diagnosis and risk prediction function of the conveyor belt deviation, and ensuring the safe production and continuous operation of the belt conveyor.

[0048] YOLACT, which stands for "You Only Look At CoefficienTs," is a highly innovative real-time instance segmentation algorithm. YOLACT's contribution lies in its redesign based on a one-stage fully convolutional algorithm, rather than merely modifying Mask R-CNN. This algorithm achieves real-time speed, is easy to deploy, and is widely applicable in various scenarios. The core idea of ​​YOLACT is to concurrently predict a prototype mask for the current image and mask coefficients for each bounding box instance. The instance mask is then generated by linearly combining the prototype and mask coefficients. This parallel prediction eliminates the need for operations like two-stage RoIPool, maintaining high output resolution and resulting in relatively high segmentation accuracy.

[0049] The present invention provides a method, device, electronic equipment and storage medium for visual detection of conveyor belt deviation, which are described below respectively.

[0050] See also Figure 1 , Figure 1 This is a flow chart of an embodiment of a method for visually detecting conveyor belt deviation provided by the present invention. A specific embodiment of the present invention discloses a method for visually detecting conveyor belt deviation, comprising:

[0051] Step S101: Acquire real-time transport images of the conveyor belt;

[0052] Step S102: inputting the real-time transport image of the conveyor belt into a well-trained image segmentation model to extract the curve image covering the conveyor belt boundary;

[0053] Step S103: determining a first center line and a second center line of the image covering the conveyor belt boundary curve, and calculating a pixel distance between the first center line and the second center line;

[0054] Step S104: Based on a preset coordinate conversion relationship, determine the conveyor belt offset distance corresponding to the pixel distance.

[0055] The real-time transport image on the conveyor belt can be captured by a camera, and then the captured real-time transport image is input into a pre-trained image segmentation model for segmentation extraction to obtain a curve image covering the conveyor belt boundary. It can be understood that by extracting the image covering the conveyor belt boundary, the conveyor belt can be recognized as a whole, making it easier to observe whether the conveyor belt is offset. Please refer to Figure 2 , Figure 2 A reference image covering a conveyor belt boundary curve image is provided in one embodiment of the present invention.

[0056] The centerline of the segmented conveyor belt image is then extracted. The centerline of the conveyor belt image includes the machine centerline and the centerline on the conveyor belt image. The degree of conveyor belt offset can be deduced through the intersection angle of the two centerlines. Specifically, the pixel distance between the image centerline and the machine centerline is first calculated. Then, based on the image, camera, and real-world coordinate conversion relationship, the real-world offset distance corresponding to the above pixel distance is determined.

[0057] The present invention can segment an image covering the conveyor belt boundary by training a complete image segmentation model, which is convenient for controlling the overall offset of the conveyor belt; and by extracting the two center lines of the curve image covering the conveyor belt boundary, and then using the coordinate transformation relationship to determine the offset distance of the conveyor belt, real-time and accurate detection is achieved, and the safety of the detection is also guaranteed.

[0058] In a specific embodiment of the present invention, the fully trained image segmentation model is trained based on the improved YOLACT network.

[0059] It should be noted that the improved idea of ​​the improved YOLACT network is based on the traditional YOLACT network. The main improvements are the backbone network part, local optimization processing and feature pyramid part, and the recognition of conveyor belt feature fusion is enhanced. Considering the operating environment and detection requirements of the belt conveyor, the YOLACT algorithm is selected and improved. Combined with the characteristics of conveyor belt segmentation, the ResNet backbone feature extraction network of the original algorithm is replaced with the ResNeXt backbone feature extraction network. A bottom-up fusion FPN structure is added to the feature pyramid part to enhance the positioning ability of the underlying edge information and enhance the semantic information; three 3×3 convolution kernels are superimposed to replace the 7×7 convolution kernel in the YOLACT network, increasing the network depth while reducing the calculation parameters; using void convolution to enhance the spatial information of the feature layer, ultimately achieving an improvement in the detection accuracy of the YOLACT algorithm.

[0060] See also Figure 3 , Figure 3 A flowchart of an image segmentation model training process provided by one embodiment of the present invention. The image segmentation model training process includes:

[0061] Step S301: collecting historical transport images of the conveyor belt, annotating the historical transport images, and constructing a training set and a test set based on the annotated images;

[0062] Step S302: Input the training set into the backbone network of the improved YOLACT network to extract initial feature information;

[0063] Step S303: combining the backbone network and the improved FPN feature pyramid structure, fusing the initial feature information based on a preset fusion method to obtain effective feature information;

[0064] Step S304: jointly calculating the first effective feature information and the second effective feature information in the effective feature information, and outputting a first covering conveyor belt boundary curve image;

[0065] Step S305: Calculate the loss using the first covered conveyor belt boundary curve image and the second covered conveyor belt boundary curve image in the validation set. If the loss function meets the preset loss condition, the image segmentation model training is complete.

[0066] Among them, the dataset consisting of a training set and a validation set can be generated by collecting transportation videos in real scenes. Specifically, the images in the dataset can be annotated in the Labelme tool to generate labels that can better cover the conveyor belt boundary curve image. After the annotation is completed, the corresponding json file is generated, and the image is placed in the coco folder and converted into the final dataset format.

[0067] The dataset is then split into a training set and a validation set at an 8:2 ratio, and the divided dataset is fed into a pre-trained image segmentation model. For details, see Figure 4 , Figure 4 A schematic diagram of the structure of an image segmentation model provided by one embodiment of the present invention.

[0068] In this example, images from the dataset were first fed into the YOLAT input. To better identify the entire conveyor belt, the input image resolution of the backbone network, ResNeXt-50, was set to 550×550. It is understood that the ResNeXt feature extraction network structure selected in this paper can improve image recognition accuracy without increasing network computational cost, demonstrating superior feature extraction performance at the same parameter complexity as ResNet. ResNeXt introduces a new dimension, "cardinality," based on depth and width. Experiments clearly show that when the depth and width of the backbone network reach saturation, increasing the cardinality significantly improves the network's image recognition performance. This improves the reading of conveyor belt edge features without significantly increasing computational complexity. The backbone network retains more semantic features. After the image is extracted through the original backbone network, initial feature information is obtained, which is convolved into five feature layers: C1, C2, C3, C4, and C5, containing different semantic and target information.

[0069] Then, YOLACT combines the backbone network and the improved FPN feature pyramid structure to fuse the initial feature information calculated by the backbone network, that is, different feature layers, and strengthens the semantic information of the bottom layers through top-down and bottom-up fusion to generate effective feature information P2, P3, P4, P5, P6 and P7.

[0070] It should be noted that the improved FPN feature pyramid structure is based on the original feature pyramid. In the original feature pyramid, the feature maps for feature fusion only include the first three feature layers: C3, C4, and C5. In the resulting multi-scale feature pyramid, P5 is a simple copy of C5, and P6 and P7 are the result of two stride=2 convolutions on P5. They are essentially convolution extensions of the backbone network. Only P3 and P4 are feature layers derived through feature fusion, which can reduce the detection accuracy of the network itself.

[0071] The improved FPN architecture in this embodiment adds a bottom-up fusion path to the top-down fusion path of the original FPN structure. The P3, P4, and P5 layers in the original FPN architecture correspond to the improved M3, M4, and M5 layers. The P6 and P7 layers in the original architecture are moved to the top of the bottom-up fusion path, and the C2 feature map extracted from the backbone network is used as the bottom feature map of the bottom-up fusion path.

[0072] Through the optimization of FPN, the object detection performance is significantly improved. The top-down and bottom-up fusion can ensure that both the upper and lower feature layers can learn targets of different sizes. That is, the shallower feature layers also need to learn to strengthen semantic information, and the deeper feature layers also need to learn small targets and edge information. The FPN layer can obtain better semantic information and stronger spatial information at the same time.

[0073] Furthermore, the first effective feature information, namely the P3, P4, P5, P6 and P7 feature layers, is input into the PredictionHead part and the Fast NMS part; at the same time, the second effective feature information, namely the P2 feature layer, is locally optimized. It can be understood that the hole convolution can provide a larger receptive field based on the 3×3 convolution. In order to reduce the number of parameters caused by the large image and ensure that the spatial information of the feature layer is not excessively lost, the hole convolution is used instead of the ordinary convolution kernel pooling process to enhance the spatial information of the feature layer. Then the optimized P2 feature layer is input into the Protonet part for calculation. The training results of the convolutional neural network are obtained by the final two branches, and the first covering conveyor belt boundary curve image is extracted.

[0074] Furthermore, the first conveyor belt boundary curve image is compared with the second conveyor belt boundary image region in the validation set to calculate the loss function. If the loss function meets the preset loss condition, such as when the loss function decreases below 0.5 and tends to a stable stage, the model training process is considered to be basically completed.

[0075] In a specific embodiment of the present invention, the backbone network and the improved FPN feature pyramid structure are combined to fuse the initial feature information based on a preset fusion method to obtain effective feature information, including:

[0076] The initial feature information is fused based on a top-down fusion method to obtain intermediate feature information;

[0077] The first initial feature information in the initial feature information is used as the basis for bottom-up fusion, and is combined with the intermediate feature information for bottom-up fusion to obtain effective feature information.

[0078] For details, please refer to Figure 5 , Figure 5 A diagram of the FPN calculation process provided by one embodiment of the present invention.

[0079] The top-down fusion path calculation process corresponds to steps M4 and M5. For example, to obtain the fused feature layer Mi, first upsample the Mi+1 layer and expand it by two times to obtain Qi+1. A 1×1 convolution is performed on Ci to adjust the number of channels to the same as Qi+1, obtaining Ui. The upsampled and expanded feature layer Qi+1 is then added to the adjusted feature layer Ui to obtain the fused feature layer Mi.

[0080] Specifically, the bottom-up fusion path calculation process corresponds to steps P3, P4, and P5. Unlike traditional bidirectional fusion, this part removes feature layer M2. Instead, the C2 feature layer is directly adjusted through a 1×1 convolution to adjust the number of channels and then horizontally connected to P2 to serve as the basis for upward fusion. The other parts are calculated as follows. For example, to obtain the upward fused feature layer Pi+1, a 3×3 convolution with a stride of 2 is performed on the Pi layer to obtain the feature layer Yi+1. The feature layer Mi+1 obtained by the left fusion is added pixel by pixel to the feature layer Yi+1 to obtain the feature layer Xi+1. Finally, a convolution with a kernel size of 3×3 and a stride of 1 is performed on the feature layer Xi+1 to obtain the corresponding fused feature layer Pi+1.

[0081] In a specific embodiment of the present invention, the first center line of the conveyor belt boundary curve image is the conveyor belt machine center line;

[0082] Determining a second center line of the conveyor belt boundary curve image includes:

[0083] An image coordinate system is established with the center of the image covering the conveyor belt boundary curve as the origin and the first center line as the first direction coordinate axis;

[0084] The width of the conveyor belt covering the conveyor belt boundary curve image in the second direction of the image coordinate system is determined, and the center point coordinate set of the conveyor belt width is used as the second center line.

[0085] It is understandable that in order to facilitate the understanding of the two center lines, the image coordinate system can be established with the center of the image covering the conveyor belt boundary curve as the origin and the first center line, i.e., the machine center line, as the first direction (vertical) coordinate axis, wherein the horizontal direction is U Axis, vertical direction is V axis, then the coordinates of the first center line are ( 0 , v 1), the coordinates of the second center line are ( u 2 ， v 2).

[0086] Therefore, the horizontal conveyor belt image width is: ,in, hw for v Value equal to v i The number of pixels in the horizontal direction, u w0 is the horizontal coordinate corresponding to the leftmost side of the image, u wn The horizontal coordinate corresponding to the rightmost side of the image.

[0087] Therefore, the horizontal coordinate of the conveyor belt centerline image u for:

[0088] ,in, U bf Mark the horizontal positive boundary value of the target area for YOLACT.

[0089] It can be understood that after extracting the image covering the conveyor belt boundary curve, the second center line of the image is extracted. The degree of conveyor belt deviation can be judged by the second direction of the conveyor belt center line and the machine center line in the image coordinate system, that is, the horizontal direction.

[0090] In a specific embodiment of the present invention, calculating the pixel distance between the first center line and the second center line includes:

[0091] The pixel coordinates of the first center line and the second center line in the image coordinate system are respectively determined, and the pixel distance between the first center line and the second center line is determined according to the pixel coordinates.

[0092] It can be understood that since the image coordinate system uses the machine center line as the vertical coordinate axis, we can focus on the pixel coordinates of the conveyor belt center line in the image coordinate system. Specifically, we can determine the pixel distance between the first center line and the second center line based on the horizontal coordinate value.

[0093] In a specific embodiment of the present invention, the preset coordinate transformation relationship includes a transformation relationship among an image coordinate system, a camera coordinate system, and a world coordinate system.

[0094] It is understandable that the improved YOLACT algorithm can well calibrate the conveyor belt image and the conveyor belt boundary. In order to obtain the actual eccentricity distance and degree of deviation of the conveyor belt, the conveyor belt position needs to be quantified. The quantization process mainly converts the image, camera and real world coordinates to obtain the actual physical position corresponding to each pixel position of the segmented conveyor belt. For details, please refer to Figure 6 , Figure 6 A conversion diagram between camera coordinates and world coordinates provided by an embodiment of the present invention.

[0095] In a specific embodiment of the present invention, determining the conveyor belt offset distance corresponding to the pixel distance based on a preset coordinate transformation relationship includes:

[0096] The conveyor belt offset distance corresponding to the pixel distance is determined based on a proportional relationship included in a preset coordinate conversion relationship, wherein the proportional relationship is determined by the height of the camera from the conveyor belt, camera parameters, pixel width, and pixel length.

[0097] in, Figure 6 It mainly includes three coordinate systems, namely the image coordinate system UO 1 V ,by O 2 is the camera coordinate system with the origin, the world coordinate system XO 3 Y It can be understood that the points on the world coordinate system that are imaged by the camera optical axis are proportional to the points on the image coordinate system. Based on this proportional relationship, the offset distance corresponding to the pixel distance can be calculated.

[0098] Specifically, the pixel point on the image where the camera lens center is Q 1 and its point in world coordinates M It is an important medium for constructing proportional relationships, and can be solved by deducing geometric relationships. PQ The length of . Among them, the offset distance PQ is the pixel distance P 1 Q 1 is calculated through proportional relationship.

[0099] It is understandable that in the image coordinate system V The axis coincides with the center line of the machine, in world coordinates Y The axis coincides with the center line of the machine. Therefore, after the center line extraction is completed, the coordinates of each point on the center line in the image coordinate system are (u i ，v i ) , Q 1 is a point on the center line of the conveyor belt in the image coordinate system, and its coordinates are (u Q ，v Q ) , behind P 1 Q 1 is a point on the center line of the conveyor belt in the image coordinates Q 1 relative to the offset distance from the center line of the machine, here PQ The length is a point on the belt centerline in the world coordinate system Q The point on the center line of the corresponding machine P The horizontal offset distance has the same physical meaning as F(u Q ，v Q ) Same, both are offset distances.

[0100] Here to solve PQ It is to demonstrate how image coordinates are converted into real world coordinates, that is F ( u i ，v i ) corresponds to the center line (u i ，v i ) The actual deviation distance of the coordinate. F ( u i ，v i ) is the actual deviation distance of each point on the conveyor belt centerline relative to the machine centerline, and the average deviation is calculated. μ, The actual average deviation distance can be obtained.

[0101] Combine Figure 6 , PQ The specific solution process includes:

[0102] 、

[0103] 、

[0104] 、

[0105] 、

[0106] 、

[0107] 、

[0108] .

[0109] in, H is the height between the camera and the conveyor belt plane, O 3 M The world coordinate point corresponding to the image center coordinates and the camera is Y The distance on the axis, O 1( u center , v center ) is the image coordinate of the lens center point, P 1( u 1, v 1) and Q 1( u 2 , v 2) is the image coordinate of the measured pixel point, P 1 Q 1 is a variable, v 1 and v 2 equal, x pix is the actual pixel length, y piy is the actual pixel width, f is the camera focal length, α for HO 3 M In-plane O 2 points horizontally and O 1 M The angle between β for P 1 P and Y The angle between the axes, γ for P 1 P and O 1 M The angle between them.

[0110] Therefore, the true average deviation of the conveyor belt centerline μ for: ,in, v 0 and v n The target area segmented v The value range of F ( u i , v i ) corresponds to the center line (u i ,v i ) The actual deviation distance of the coordinate.

[0111] In order to better implement the conveyor belt deviation visual detection method in the embodiment of the present invention, based on the conveyor belt deviation visual detection method, please refer to Figure 7 , Figure 7 This is a schematic structural diagram of an embodiment of a conveyor belt deviation visual detection device provided by the present invention. The embodiment of the present invention provides a conveyor belt deviation visual detection device 700, comprising:

[0112] An acquisition module 701 is used to acquire real-time transport images of the conveyor belt;

[0113] Extraction module 702, used to input the real-time transport image of the conveyor belt into the trained image segmentation model to extract the image covering the conveyor belt boundary curve;

[0114] A calculation module 703 is used to determine a first center line and a second center line of the image covering the conveyor belt boundary curve, and calculate a pixel distance between the first center line and the second center line;

[0115] The determination module 704 is configured to determine the conveyor belt offset distance corresponding to the pixel distance based on a preset coordinate conversion relationship.

[0116] It should be noted here that the device 700 provided in the above embodiment can implement the technical solutions described in the above method embodiments. The specific implementation principles of the above modules or units can be found in the corresponding contents in the above method embodiments, which will not be repeated here.

[0117] Based on the above-mentioned method for visually detecting conveyor belt deviation, an embodiment of the present invention also provides an electronic device, including: a processor and a memory, and a computer program stored in the memory and executable on the processor; when the processor executes the computer program, the steps in the method for visually detecting conveyor belt deviation in the above-mentioned embodiments are implemented.

[0118] Figure 8 8 shows a schematic diagram of the structure of an electronic device 800 suitable for implementing an embodiment of the present invention. The electronic devices in the embodiments of the present invention may include, but are not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), and in-vehicle terminals (e.g., in-vehicle navigation terminals), as well as fixed terminals such as digital TVs and desktop computers. Figure 8 The electronic device shown is only an example and should not limit the functions and scope of use of the embodiments of the present invention.

[0119] The electronic device includes: a memory and a processor, wherein the processor here may be referred to as a processing device 801 hereinafter, and the memory may include at least one of a read-only memory (ROM) 802, a random access memory (RAM) 803, and a storage device 808 hereinafter, as specifically shown below:

[0120] like Figure 8As shown, electronic device 800 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 801, which can perform various appropriate actions and processes based on programs stored in a read-only memory (ROM) 802 or programs loaded from a storage device 808 into a random access memory (RAM) 803. RAM 803 also stores various programs and data required for the operation of electronic device 800. Processing device 801, ROM 802, and RAM 803 are interconnected via a bus 804. An input / output (I / O) interface 805 is also connected to bus 804.

[0121] Typically, the following devices may be connected to the I / O interface 805: an input device 806 including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 807 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 808 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 809. The communication device 809 may allow the electronic device 800 to communicate with other devices wirelessly or by wire to exchange data. Figure 8 The electronic device 800 is shown with various devices, but it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed instead.

[0122] In particular, according to an embodiment of the present invention, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present invention includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program includes program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network via the communication device 809, or installed from the storage device 808, or installed from the ROM 802. When the computer program is executed by the processing device 801, the above-mentioned functions defined in the method of the embodiment of the present invention are performed.

[0123] Based on the above-mentioned method for visually detecting conveyor belt deviation, an embodiment of the present invention also provides a computer-readable storage medium, which stores one or more programs. The one or more programs can be executed by one or more processors to implement the steps in the method for visually detecting conveyor belt deviation as in the above-mentioned embodiments.

[0124] Those skilled in the art will appreciate that all or part of the process steps of the above-described embodiments can be implemented by instructing related hardware through a computer program, and the program can be stored in a computer-readable storage medium, such as a magnetic disk, an optical disk, a read-only memory, or a random access memory.

[0125] The above description is only a preferred specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by any technician familiar with this technical field within the technical scope disclosed by the present invention should be covered by the scope of protection of the present invention.

Claims

1. A method for visually detecting conveyor belt deviation, characterized in that: include: Get real-time transport images of conveyor belts; Inputting the real-time transport image of the conveyor belt into a well-trained image segmentation model to extract an image of a curve covering the conveyor belt boundary; Determining a first center line and a second center line of the covered conveyor belt boundary curve image, and calculating a pixel distance between the first center line and the second center line; Based on a preset coordinate conversion relationship, determining a conveyor belt offset distance corresponding to the pixel distance; The fully trained image segmentation model is trained based on the improved YOLACT network. The training process of the fully trained image segmentation model includes: Collecting historical transport images of the conveyor belt, annotating the historical transport images, and constructing a training set and a test set based on the annotated images; Inputting the training set into the backbone network in the improved YOLACT network to extract initial feature information; Combining the backbone network and the improved FPN feature pyramid structure, the initial feature information is fused based on a preset fusion method to obtain effective feature information; jointly calculating the first effective feature information and the second effective feature information in the effective feature information, and outputting a first covering conveyor belt boundary curve image; Loss calculation is performed using the first covered conveyor belt boundary curve image and the second covered conveyor belt boundary curve image in the validation set. If the loss function meets a preset loss condition, the image segmentation model is fully trained. The combination of the backbone network and the improved FPN feature pyramid structure, fusing the initial feature information based on a preset fusion method to obtain effective feature information, includes: The initial feature information is fused based on a top-down fusion method to obtain intermediate feature information; Using the first initial feature information in the initial feature information as a basis for bottom-up fusion, and combining it with the intermediate feature information to perform bottom-up fusion to obtain the effective feature information; The first center line of the conveyor belt boundary curve image is the conveyor belt machine center line; Determining a second center line of the covered conveyor belt boundary curve image includes: An image coordinate system is established with the center of the covered conveyor belt boundary curve image as the origin and the first center line as the first direction coordinate axis; The conveyor belt width of the covered conveyor belt boundary curve image in the second direction of the image coordinate system is determined, and the center point coordinate set of the conveyor belt width is used as the second center line.

2. The conveyor belt deviation visual detection method according to claim 1 is characterized in that: The calculating the pixel distance between the first center line and the second center line includes: Pixel coordinates of the first center line and the second center line in the image coordinate system are respectively determined, and a pixel distance between the first center line and the second center line is determined according to the pixel coordinates.

3. The conveyor belt deviation visual detection method according to claim 2 is characterized in that: The preset coordinate conversion relationship includes the conversion relationship between the image coordinate system, the camera coordinate system and the world coordinate system.

4. The method for visually detecting conveyor belt deviation according to claim 3, characterized in that: The determining of the conveyor belt offset distance corresponding to the pixel distance based on a preset coordinate conversion relationship includes: The conveyor belt offset distance corresponding to the pixel distance is determined based on a proportional relationship included in the preset coordinate conversion relationship, wherein the proportional relationship is determined by the height of the camera from the conveyor belt, camera parameters, pixel width and pixel length.

5. A visual detection device for conveyor belt deviation, characterized in that: include: An acquisition module is used to acquire real-time transport images of the conveyor belt; An extraction module is used to input the real-time transport image of the conveyor belt into a well-trained image segmentation model to extract an image covering the conveyor belt boundary curve; a calculation module, configured to determine a first center line and a second center line of the image covering the conveyor belt boundary curve, and calculate a pixel distance between the first center line and the second center line; A determination module, configured to determine a conveyor belt offset distance corresponding to the pixel distance based on a preset coordinate conversion relationship; The fully trained image segmentation model is trained based on the improved YOLACT network. The training process of the fully trained image segmentation model includes: Collecting historical transport images of the conveyor belt, annotating the historical transport images, and constructing a training set and a test set based on the annotated images; Inputting the training set into the backbone network in the improved YOLACT network to extract initial feature information; Combining the backbone network and the improved FPN feature pyramid structure, the initial feature information is fused based on a preset fusion method to obtain effective feature information; jointly calculating the first effective feature information and the second effective feature information in the effective feature information, and outputting a first covering conveyor belt boundary curve image; Loss calculation is performed using the first covered conveyor belt boundary curve image and the second covered conveyor belt boundary curve image in the validation set. If the loss function meets a preset loss condition, the image segmentation model is fully trained. The combination of the backbone network and the improved FPN feature pyramid structure, fusing the initial feature information based on a preset fusion method to obtain effective feature information, includes: The initial feature information is fused based on a top-down fusion method to obtain intermediate feature information; Using the first initial feature information in the initial feature information as a basis for bottom-up fusion, and combining it with the intermediate feature information to perform bottom-up fusion to obtain the effective feature information; The first center line of the conveyor belt boundary curve image is the conveyor belt machine center line; Determining a second center line of the covered conveyor belt boundary curve image includes: An image coordinate system is established with the center of the covered conveyor belt boundary curve image as the origin and the first center line as the first direction coordinate axis; The conveyor belt width of the covered conveyor belt boundary curve image in the second direction of the image coordinate system is determined, and the center point coordinate set of the conveyor belt width is used as the second center line.

6. An electronic device, characterized in that: It includes a memory and a processor, wherein the memory is used to store programs; the processor is coupled to the memory and is used to execute the programs stored in the memory to implement the steps in the conveyor belt deviation visual detection method according to any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that Used to store computer-readable programs or instructions, which, when executed by a processor, can implement the steps of the conveyor belt deviation visual detection method described in any one of claims 1 to 4 above.

Images