A vision-based method for detecting the state of trough conveyors

Abstract

The present invention discloses a method for detecting the state of a trough conveyor based on vision. A detection point is selected in a trough conveyor working scene, and a visual system is set at the monitoring point to obtain video stream data; the obtained video stream data is subjected to frame extraction processing to obtain corresponding frame extraction image data; the image processing is subjected to visual processing to detect and obtain the state data of the trough conveyor and judge the current working state of the trough conveyor based on the state data. The advantages of the present invention are: by installing cameras at different points to obtain video data and developing corresponding monitoring algorithms, the state of the chain bucket conveyor can be monitored, and the deviation, overflow, skew, etc. can be quickly and accurately automatically identified, thereby providing a basis for timely alarm and troubleshooting.

Description

Technical Field

[0001] The invention relates to the field of intelligent monitoring of industrial transportation equipment, and in particular to a vision-based trough conveyor state detection method. Background Art

[0002] Since most automated production equipment is in a long-term working state, some operating failures are inevitable in the production operation of mechanical equipment. In the field of automated production, if the equipment failure is not discovered in time, at the very least, there will be problems with the quality of the products produced or errors in the production process, or at the worst, there will be production safety accidents such as damage to the production equipment. Therefore, the failure status of automatic planning production equipment needs to be monitored and detected.

[0003] Taking the automated transportation equipment in the field of cement production and transportation as an example, the traditional way is to troubleshoot faults through manual inspection. However, manual inspection is limited by manpower and inspection cycle, and it is impossible to achieve real-time monitoring. It is also impossible to accurately and timely detect faults when they occur. Therefore, traditional manual fault monitoring can no longer meet the company's needs for production efficiency.

[0004] Trough conveyor is a commonly used equipment in the field of industrial automation. In the field of cement automation production, the specific structure of trough conveyor is called chain bucket conveyor or inclined zipper conveyor. Chain bucket conveyor is a special equipment used by large cement plants to transport cement clinker. It is also a kind of transportation equipment commonly used in the field of industrial production. During the transportation process, deviation, overflow, roller skewness and other situations may occur. The existing technology of manual monitoring of its transportation project failure cannot meet the requirements of timely preparation. Summary of the invention

[0005] The purpose of the present invention is to overcome the shortcomings of the prior art and provide a method for detecting the state of a chain bucket conveyor based on vision, which uses a visual method to automatically detect the chain bucket conveyor to timely and accurately detect faults such as deviation and overflow, roller skew, etc.

[0006] In order to achieve the above-mentioned purpose, the technical solution adopted by the present invention is: a vision-based trough conveyor state detection method, selecting detection points in the trough conveyor working scene, and setting a visual system at the monitoring point to obtain video stream data; performing frame extraction processing on the obtained video stream data to obtain corresponding frame-extracted image data; performing visual processing on the image processing to detect and obtain the trough conveyor state data and judge the current trough conveyor working state based on the state data.

[0007] The collected frame-by-frame image data is used to detect the edge features of the skirt plate using a segmentation framework. The edge detection is performed using the edge mask images of the skirt plates on both sides. The coordinates of the edge points on both sides of the skirt plate are recorded. The center line and its coordinate information are calculated based on the fitting calculation of the edge points on both sides of the skirt plate. The deviation status of the trough conveyor is determined using the calculated center line and its coordinate information.

[0008] Establish an image coordinate system, process the frame-sampling image based on the image coordinate system to obtain the calculated center line and its coordinate information; obtain the center line and its coordinates in the standard state calculated by the frame-sampling image data under the normal state of the trough conveyor;

[0009] Determine whether the vehicle is running off the track and the direction of the running off based on the slope K of the fitted calculated center line;

[0010] The degree of deviation is determined by the distance between the calculated center line and the standard center line.

[0011] When the slope K is equal to 0, there is no deviation; when the slope K is greater than 0, the vehicle deviates to the right, and when the slope K is less than 0, the vehicle deviates to the left; the average distance between the center line and all points on the standard center line is calculated, and the severity of the deviation is determined based on the average distance and the set distance threshold.

[0012] The frame-sampling image is segmented and detected, and the image is divided into the area where the skirt panels on both sides are located and the area where the material is located. The sum of the areas where the skirt panels are located and the area where the material is located is calculated; then the ratio of the sum of the areas occupied by the left and right skirt panels is calculated respectively, and the overflow status is determined based on the ratio.

[0013] After segmentation and detection of the frame-sampling image, the mask list values ​​of the corresponding areas of the skirt board and the material are obtained. According to the pixel values ​​in the mask list, the contour coordinates of each area are extracted; the left and right skirt boards are determined according to the horizontal coordinate values ​​of the contour coordinates; based on the left and right skirt board segmentation, the image is divided into three parts: the left skirt board, the right skirt board and the material area, and then the area sum of the three areas is calculated and the corresponding proportion of the left and right skirt board areas is calculated.

[0014] When the proportion of the left skirt panel and the proportion of the right skirt panel are both greater than the set proportion threshold a, it is judged that no overflow has occurred; otherwise, when the proportion of the left skirt panel is less than the set threshold b and the proportion of the right skirt panel is greater than the set threshold a, it is judged that overflow may occur on the left skirt panel; when the proportion of the right skirt panel is less than the set threshold a and the proportion of the left skirt panel is greater than the set threshold b, it is judged that overflow may occur on the right skirt panel.

[0015] The video stream data collected at the roller is used to obtain the corresponding frame image of the roller, and the image segmentation and edge detection methods are applied to the frame image to obtain the contour of the roller. The circumscribed circle of the corresponding roller contour is fitted using the detected contour, and the area of ​​the circumscribed circle is calculated. The size change of the circumscribed circle is recorded, and whether the roller is skewed is determined based on the size change of the circumscribed circle area.

[0016] The maximum circumscribed circle of the fitted roller profile includes:

[0017] Get the points included in the roller contour, calculate the convex hull of this set of points, that is, the minimum convex polygon surrounding all the points; fit a minimum circle to surround the entire convex hull to get the circumscribed circle, and get the center coordinates and radius of the circle.

[0018] The advantages of the present invention are: by installing cameras at different points to obtain video data and developing corresponding monitoring algorithms, the status of the chain bucket conveyor can be monitored, and deviation, overflow, skewness and the like can be quickly and accurately automatically identified, thereby providing a basis for timely alarm and troubleshooting; this solution uses visual methods for detection, and the hardware cost only requires the installation of a visual camera at a set position and a corresponding data transmission system to transmit the visual image to the server for analysis and processing. The implementation is simple and reliable, and the detection can be close to real-time detection, which provides a basis for real-time alarm. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The following is a brief description of the contents expressed in the drawings of the present invention and the symbols in the drawings:

[0020] Figure 1 is a flow chart of the detection method of the present invention;

[0021] Figure 2 This is a schematic diagram of the deviation detection of the present invention;

[0022] Figure 3 It is the material segmentation diagram of the present invention;

[0023] Figure 4 This is a diagram of the skirt panel segmentation of the present invention. DETAILED DESCRIPTION

[0024] The specific implementation of the present invention will be further explained in detail below by describing the optimal embodiment with reference to the accompanying drawings.

[0025] The present invention uses visual detection technology to realize intelligent monitoring of diagonal zippers. By installing cameras at different points to obtain video data, the corresponding monitoring algorithm is developed to monitor the state of the diagonal zipper machine, and when deviation, overflow, skewness, etc. occur, an alarm is triggered to notify the on-site personnel.

[0026] Technical route: The technical route of the present invention is as follows Figure 1 As shown in the figure, real-time data streams at different points are obtained, frame data are taken at different times according to the detection requirements, the data are scaled according to a fixed size, and sent to the model for reasoning. According to different detection scenarios, corresponding post-processing is performed to obtain data such as roller area, material and skirt ratio, skirt center line, etc. The state of the oblique zipper machine can be measured by the change of data, which can achieve the purpose of real-time monitoring and finally obtain the detection result, that is, whether there is deviation, overflow, roller skew, etc. In these cases, the alarm value can be uploaded to the application platform, and the application platform will perform corresponding alarm execution according to the uploaded alarm information.

[0027] The present invention mainly adopts a visual segmentation algorithm framework, which can automatically extract multi-level features in the image, including edges, textures, shapes, etc. through a multi-layer network structure, and can well separate features such as rollers, materials, skirts, etc.; and can achieve real-time or near real-time processing speed, better process the details and global information in the image, and improve the accuracy and comprehensiveness of segmentation.

[0028] 1. Deviation detection

[0029] In the working scene of the trough conveyor, detection points are selected, and a visual system is set up at the monitoring points to obtain video stream data; deviation points are pre-selected, and a visual system is arranged at the points, that is, a camera is set up to collect the video stream data required for the deviation points, which is used for subsequent data analysis and obtaining the corresponding deviation status data; the obtained video stream data is subjected to frame extraction processing to obtain corresponding frame-extracted image data, and frame extraction is to extract a frame of image from the video stream according to time or other requirements for the video stream data, so as to facilitate image processing; the image obtained by frame extraction is subjected to visual processing to detect and obtain the status data of the trough conveyor and judge the current working status of the trough conveyor based on the status data.

[0030] For deviation detection, the collected frame-by-frame image data is used to detect the edge features of the skirt plate using a segmentation framework, and edge detection is performed using the edge mask images of the skirt plates on both sides. The coordinates of the edge points on both sides of the skirt plate are recorded, and the center line and its coordinate information are calculated based on the fitting calculation of the edge points on both sides of the skirt plate. The deviation status of the trough conveyor is judged using the calculated center line and its coordinate information.

[0031] An image coordinate system is established, and the frame-drawing image is processed based on the image coordinate system to obtain the calculated center line and its coordinate information; the center line and its coordinates under the standard state are calculated by obtaining the frame-drawing image data under the normal state of the trough conveyor; the image coordinate system is the pixel coordinate system, which means that the acquired image is cropped according to a fixed size or not cropped directly with the upper left corner fixed point of the original image as the origin, and the upper left corner fixed point is an x-axis along the horizontal direction and the y-axis along the vertical direction to establish the image coordinate system. In this coordinate system, each pixel point corresponds to a coordinate value; by calculating the identified edge points on both sides under this coordinate system, a calculated center line can be fitted. When there is no deviation under normal circumstances, a standard center line can also be calculated. By comparing the difference between the two center lines, it is determined whether there is deviation. Taking the slope of the standard center line as 0 as an example, when the calculated center line is not 0, there is a deviation. As for the degree of deviation, it can be calculated based on the average distance between the two center lines.

[0032] The slope K of the fitted calculated center line is used to determine whether the vehicle is deviating and the direction of the deviation. The degree of deviation is determined by the distance between the calculated center line and the standard center line. When the slope K is equal to 0, there is no deviation. When the slope K is greater than 0, the vehicle deviates to the right, and when the slope K is less than 0, the vehicle deviates to the left. The average distance between all points on the center line and the standard center line is calculated, and the severity of the deviation is determined based on the average distance and the set distance threshold.

[0033] The distance between two center points is obtained by fitting a number of edge points on both sides, which has a specific number of points on the center line. The sum of the two points can be obtained by taking the average.

[0034] The edge features of the skirt are detected using the segmentation framework. The edge detection is performed through the edge mask map of the skirts on both sides. The coordinates of the edge points on both sides are recorded. Through interpolation sampling, fifty coordinate points are taken on each side. Then, the center point of the coordinate points on both sides is calculated. The specific calculation method is

[0035] For example, the left edge coordinate point set is [(x L1 ,y 1 ),(x L2 ,y 2 ),(x L3 ,y 3 )................(x Ln ,y n )], the right edge coordinate point set is [(x R1 ,y 1 ),(x R2 ,y 2 ),(x R3 ,y 3 )................(xRn ,y n )], calculate the center point coordinates of the skirt plate as:

[0036]

[0037] By fitting a straight line through the coordinate points, the slope k of the straight line can be obtained.

[0038] At the same time, based on the professional experience of the on-site personnel, a center line is calibrated under normal circumstances, and the slope size, direction and average distance between the two straight lines are compared to comprehensively analyze whether the diagonal zipper is deviating. Figure 2 shown.

[0039] 1. Under normal circumstances, the skirt board does not deviate, and the center line at this time is used as the reference center line.

[0040] 2. When the skirt plate deviates to the right, the slope of the fitted center line is greater than zero, and the larger the absolute value of the slope, the more obvious the deviation. At the same time, the average distance between the two straight lines can also measure the degree of deviation.

[0041] 3. When the skirt is offset to the left, the slope of the fitted centerline is less than zero, and the greater the absolute value of the slope, the more obvious the deviation. At the same time, the average distance between the two straight lines can also measure the degree of deviation. The average distance is calculated based on the size of the pixel value, and the range of the average distance threshold value can be calibrated and set according to the actual situation.

[0042]

[0043]

[0044] 2. Overflow Detection

[0045] Select the material overflow point, set the corresponding camera to collect video stream data, then perform data extraction to obtain the extracted frame image, perform segmentation detection on the extracted frame image, divide the image into the area where the skirt panels on both sides are located and the area where the materials are located, calculate the area sum of the skirt panel area and the material area; then calculate the ratio of the area occupied by the left and right skirt panel areas respectively, and judge the overflow status according to the ratio.

[0046] After segmentation and detection of the frame-sampling image, the mask list values ​​of the corresponding areas of the skirt board and the material are obtained. According to the pixel values ​​in the mask list, the contour coordinates of each area are extracted; the left and right skirt boards are determined according to the horizontal coordinate values ​​of the contour coordinates; based on the left and right skirt board segmentation, the image is divided into three parts: the left skirt board, the right skirt board and the material area, and then the area sum of the three areas is calculated and the corresponding proportion of the left and right skirt board areas is calculated.

[0047] When the proportion of the left skirt board and the proportion of the right skirt board are both greater than the set proportion threshold a, it is judged that no overflow occurs; otherwise, when the proportion of the left skirt board is less than the set threshold b and the proportion of the right skirt board is greater than the set threshold a, it is judged that the left skirt board may have overflow; when the proportion of the right skirt board is less than the set threshold a and the proportion of the left skirt board is greater than the set threshold b, it is judged that the right skirt board may have overflow. The value range of ab can be calibrated and set according to actual needs.

[0048] The instance segmentation model performs pixel-by-pixel classification. For an original image, it is sent to the model for segmentation detection. Through post-processing of the model results, the skirt panels and materials on both sides can be segmented. The segmentation results are as follows Figure 3 , 4 As shown, Figure 3 The white blank area in the middle is the divided material area. Figure 4 The white blank area in the middle is the area of ​​the left and right skirt panels. By comparing the relative proportion of the left and right skirt panels to the material, it is possible to monitor whether there will be overflow on the skirt panels on both sides.

[0049] First, extract and process the detection model output to obtain the mask value of the corresponding area. The mask list is shown below.

[0050] Mask=[[0,0,0........0

[0051] 0,1,0........1

[0052] 0,0,0........0

[0053] 0,0,0........0

[0054] 0,1,0........1

[0055] 0,0,0........0]]

[0056] Secondly, according to the pixel values ​​in the list, the contour coordinates of each area are extracted, wherein the left and right skirt panels are determined according to the horizontal coordinate values ​​of the contour coordinates. It can be seen that the horizontal coordinate of the right skirt panel is much larger than that of the left skirt panel.

[0057] Finally, by comparing the divided areas of the three parts, it is possible to qualitatively analyze whether there will be overflow of the left and right skirt plates during the material transportation process of the oblique zipper machine.

[0058] The percentage of spillage

[0059] Left ratio Right proportion Overflow >10％ >10％ normal >10％ <5％ The right side skirt may have overflow <5％ >10％ The left side skirt may have overflow

[0060] 3. Roller skew detection

[0061] A roller position is selected, and a camera is set at the position to collect video stream data with the roller status. The video stream data is subjected to frame extraction processing to obtain a frame extraction image with a roller image. The frame extraction image corresponding to the roller is obtained for the video stream data collected at the roller. The outline of the roller is obtained by applying image segmentation and edge detection methods to the frame extraction image. The circumscribed circle of the corresponding roller outline is fitted using the detected outline, the area of ​​the circumscribed circle is calculated, the size change of the circumscribed circle is recorded, and whether the roller is skewed is determined based on the size change of the circumscribed circle area.

[0062] Fitting the maximum circumscribed circle of the roller contour includes: obtaining the points included in the roller contour, calculating the convex hull of this set of points, that is, the minimum convex polygon surrounding all the points; fitting a minimum circle to surround the entire convex hull to obtain the circumscribed circle, and obtaining the coordinates of the center and the radius of the circle.

[0063] The circumscribed circle of the roller is fitted using the results of the segmentation model + edge detection. By observing the change in the size of the circumscribed circle, it can be reflected whether the roller is offset.

[0064] For the segmentation of the roller, considering the greater influence of background factors, the corresponding circular shape of the roller is fitted using the maximum contour during edge detection.

[0065] 1. The input parameter is an array or list containing a set of points, that is, the points contained in the maximum contour.

[0066] 2. Calculate the convex hull of this set of points, that is, the smallest convex polygon that encloses all points.

[0067] 3. After obtaining the convex hull, the function will try to fit a minimum circle to enclose the entire convex hull. This circle is called the minimum circumscribed circle.

[0068] 4. The output result usually includes the coordinates of the center of the circle and the radius. This information can be used to draw the minimum circumscribed circle and calculate the area information of the drawn circle. By comparing the area change of the fitted circle of the roller, it can be determined whether the roller is skewed. When skew occurs, the area of ​​the fitted circle of the roller will become smaller to a certain extent. When there is no skew, the area of ​​the circumscribed circle obtained by fitting is fixed, and the area under normal circumstances is used as the standard value. If it is skewed, its area is always smaller than the area under normal circumstances; if in the actual monitoring process, the area of ​​the circumscribed circle calculated by analysis is smaller than the normal circumscribed circle area, it is considered to be in a skewed state, and the smaller the area, the more serious the skew. Therefore, the area of ​​the circumscribed circle can be fitted by the above method to determine whether there is roller skew.

[0069] Obviously, the specific implementation of the present invention is not limited to the above-mentioned methods. As long as various non-substantial improvements are made using the method concept and technical solution of the present invention, they are all within the protection scope of the present invention.

Claims

1. A method for detecting the state of a trough conveyor based on vision, characterized in that: A monitoring point is selected in the working scene of the trough conveyor, and a visual system is set at the monitoring point to obtain video stream data; the obtained video stream data is subjected to frame extraction processing to obtain corresponding frame extraction image data; the image processing is subjected to visual processing to detect and obtain the state data of the trough conveyor and judge the current working state of the trough conveyor based on the state data; the state data of the trough conveyor includes roller area, material and skirt board ratio, and skirt board centerline data; the current working state of the trough conveyor is judged according to the state data, including whether the trough conveyor is in a deviation state, an overflow state, and a roller skew state; The collected frame-sampling image data is used to detect the edge features of the skirt plate using a segmentation framework, and edge detection is performed using the edge mask images of the skirt plates on both sides. The coordinates of the edge points on both sides of the skirt plate are recorded, and the center line and its coordinate information are calculated based on the fitting calculation of the edge points on both sides of the skirt plate. The deviation state of the trough conveyor is determined using the calculated center line and its coordinate information. Establish an image coordinate system, process the frame-sampling image based on the image coordinate system to obtain the calculated center line and its coordinate information; obtain the center line and its coordinates in the standard state calculated by the frame-sampling image data under the normal state of the trough conveyor; Determine whether the vehicle is running off the track and the direction of the running off based on the slope K of the fitted calculated center line; The degree of deviation is determined by the distance between the calculated center line and the standard center line; When the slope K is equal to 0, there is no deviation; when the slope K is greater than 0, the vehicle deviates to the right, and when the slope K is less than 0, the vehicle deviates to the left; the average distance between the center line and all points on the standard center line is calculated, and the severity of the deviation is determined based on the average distance and the set distance threshold; Perform segmentation detection on the frame image, divide the image into the area where the skirts on both sides are located and the area where the material is located, calculate the area sum of the skirt area and the material area; then calculate the ratio of the area sum of the left and right skirt areas respectively, and judge the overflow status according to the ratio; After segmenting and detecting the frame-sampling image, the mask list values ​​of the corresponding areas of the skirt board and the material are obtained. According to the pixel values ​​in the mask list, the contour coordinates of each area are extracted; the left and right skirt boards are determined according to the horizontal coordinate values ​​of the contour coordinates; based on the left and right skirt board segmentation, the image is divided into three parts: the left skirt board, the right skirt board and the material area, and then the area sum of the three areas is calculated and the corresponding proportion of the left and right skirt board areas is calculated; When the proportion of the left skirt board and the proportion of the right skirt board are both greater than the set proportion threshold a, it is judged that no overflow occurs; otherwise, when the proportion of the left skirt board is less than the set threshold b and the proportion of the right skirt board is greater than the set threshold a, it is judged that overflow occurs on the left skirt board; when the proportion of the right skirt board is less than the set threshold a and the proportion of the left skirt board is greater than the set threshold b, it is judged that overflow occurs on the right skirt board; The frame-sampling image corresponding to the roller is obtained from the video stream data collected at the roller, and the image segmentation and edge detection methods are applied to the frame-sampling image to obtain the contour of the roller. The circumscribed circle of the corresponding roller contour is fitted using the detected contour, the area of ​​the circumscribed circle is calculated, the size change of the circumscribed circle is recorded, and whether the roller is skewed is determined based on the size change of the circumscribed circle area. The maximum circumscribed circle of the fitted roller profile includes: Get the points included in the roller contour, calculate the convex hull of this set of points, that is, the minimum convex polygon surrounding all the points; fit a minimum circle to surround the entire convex hull to get the circumscribed circle, and get the center coordinates and radius of the circle.

Images