A method for determining the production status of tobacco conveyor belts by fusing depth images and color images

Abstract

The present invention discloses a method for determining the production status of a tobacco conveyor belt by fusing depth images and color images. After acquiring the depth image and the color image, the method detects the color image and the depth image respectively to determine whether there are tobacco leaves on the conveyor belt, the height change of the tobacco leaves, the movement status of the conveyor belt, etc. Then, the depth detection result and the color image detection result are fused to determine the current production status of the conveyor belt. The present invention can monitor the production process of the conveyor belt in real time, thereby realizing timely alarm when there is a fault, improving production efficiency, improving production technology, and ensuring the stability of tobacco products.

Description

Technical Field

[0001] This invention belongs to the field of machine vision technology, specifically a method for determining the production status of a tobacco conveyor belt that combines depth image data and color image data. Background Technology

[0002] In the tobacco production process of the cigarette industry, conveyor belts may malfunction due to prolonged operation, resulting in phenomena such as excessive speed, excessive slowness, slippage, and deviation. Additionally, equipment failure may cause excessive accumulation of tobacco leaves on the conveyor belt, leading to blockages. Uneven feeding may also cause conveyor belt interruptions. To prevent these issues during production, operators are often scheduled to conduct regular inspections. However, the high temperature and humidity in the tobacco storage workshop make regular inspections unsuitable. If staff fail to detect malfunctions promptly, production progress will be affected. Summary of the Invention

[0003] This invention addresses the shortcomings of existing technologies by proposing a method for determining the production status of conveyor belts that integrates depth data and color images. This method aims to monitor the production process of conveyor belts in real time and continuously, thereby enabling timely alarms in case of faults, improving production efficiency, optimizing production processes, and ensuring the stability of tobacco products.

[0004] To achieve the above-mentioned objectives, the present invention adopts the following technical solution:

[0005] The present invention provides a method for determining the production status of a tobacco conveyor belt by fusing depth and color images, characterized by the following steps:

[0006] Step 1: Use a camera to capture depth and color images on the conveyor belt, with the camera's imaging plane parallel to the conveyor belt; the vertical direction of the depth and color images coincides with the direction of movement of the conveyor belt; the direction of movement of the conveyor belt is the y-direction of the image; the x-direction of the image is perpendicular to the direction of movement of the conveyor belt.

[0007] Step 2: Acquire a depth image T when the conveyor belt is stationary and there are no tobacco leaves. null And calculate T null The sum of all depth values ​​in ST null ;

[0008] Step 3: Acquire a color image I1 of the conveyor belt in operation, and let the width and height of I1 be W and H, respectively;

[0009] Step 4: Use a deep learning model to detect I1 to determine whether there are tobacco leaves on the conveyor belt. If there are tobacco leaves, proceed to step 5; otherwise, return to step 3 to continue collecting data.

[0010] Step Five: Collect the color image I2 on the next conveyor belt, as well as two consecutive depth images T1 and T2, and calculate the sum ST1 of all depth values in T1 and the sum ST2 of all depth values in T2;

[0011] Step Six: Based on the color image I2, use the Lucas-Kanade algorithm to calculate the velocity V(x i , y i ) = {v i , v i,x} of any i-th pixel point p in the optical flow field of the color image I1, where (x i,y , y i ) is the position coordinate of the i-th pixel point p in the color image I1, and v i and v i are the velocity values of the i-th pixel point p i,x in the horizontal and vertical directions respectively; The velocity set V of the color image I1 is composed of the velocities of all pixel points in the color image I1, and i ∈ [1, W×H];

[0012] Step Seven: Use Equation (1) to calculate the average value of the x direction of the velocity set V :

[0013] (1)

[0014] Step Eight: Use Equation (2) to calculate the average value of the y direction of the velocity set V :

[0015] (2)

[0016] Step Nine: If <o:p> > th_vx, then set the identifier s b = 1 and give an alarm to indicate that the conveyor belt is running off track, where th_vx represents the velocity threshold of the optical flow field in the x direction; otherwise, execute Step Ten;

[0017] Step Ten: If <th_min, then set the identifier s b = 2, indicating that the conveyor belt is in a stationary state, where th_min represents the minimum velocity threshold, otherwise, execute Step Eleven;

[0018] Step Eleven: If <00001: <th_low, then set the identifier s b = 3, indicating that the conveyor belt is suspected of running too slowly, where th_low represents the velocity too slow threshold; otherwise, execute Step Twelve;

[0019] ​​Step Twelve: If >th_max, then set the identifier s b =4 and give an alarm to indicate that the conveyor belt speed is too fast. Here, th_max represents the threshold for excessive speed. Otherwise, set the identifier s b =5, indicating that the conveyor belt speed is normal;

[0020] Step Thirteen: If s b =3, then assign counter + 1 to counter. Otherwise, set the count value counter = 0;

[0021] Step Fourteen: If counter>th_num, then set s b =6 and give an alarm to indicate that the conveyor belt speed is too slow. Here, th_num is the cumulative frame number threshold;

[0022] Step Fifteen: Calculate the first depth difference ST 1= ST1 - ST null , calculate the second depth difference ST 2= ST2 - ST null ; Calculate the third depth difference ST = ST1 - ST2;

[0023] Step Sixteen: If ST1 < th_tf_min and ST2 < th_tf_min, then set the depth identifier s m =1, indicating that there is no tobacco leaf on the conveyor belt. Otherwise, execute Step Seventeen. Here, th_tf_min is the minimum height threshold of the tobacco leaf;

[0024] Step Seventeen: If ST > th_var holds, then set the depth identifier s m =2, indicating that the conveyor belt is transporting the head or tail of the material. Otherwise, execute Step Eighteen. Here, th_var is the threshold for the height change of the tobacco leaf;

[0025] Step Eighteen: If ST1 > th_s or ST2 > th_s, then set the depth identifier s m =3, indicating that there is too much tobacco leaf on the conveyor belt. Otherwise, set the depth identifier s m =4, indicating that the height of the tobacco leaf on the conveyor belt is normal. Here, th_s is the maximum height threshold of the tobacco leaf;

[0026] Step Nineteen: If s mIf the value is 2, then assign counter_tf + 1 to counter_tf; otherwise, set the depth count value counter_tf = 0.

[0027] Step 20: If counter_tf > th_tf_num, then let the depth identifier s m =5, and an alarm is triggered indicating an abnormality in the material head or tail of the conveyor belt. Here, th_tf_num is the threshold for the cumulative number of depth data frames.

[0028] Step 21: If s b =5 and s m =3, and an alarm is triggered indicating that the height of the tobacco leaves on the conveyor belt is abnormal during the production process.

[0029] The present invention provides an electronic device, comprising a memory and a processor, wherein the memory is used to store a program that supports the processor in executing the tobacco conveyor production status determination method, and the processor is configured to execute the program stored in the memory.

[0030] The present invention discloses a computer-readable storage medium, wherein a computer program is stored on the computer-readable storage medium, and the computer program is executed by a processor to perform the steps of the tobacco conveyor production status determination method.

[0031] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0032] 1. The depth model of this invention can accurately detect the state of tobacco leaves on the conveyor belt and has strong generalization ability.

[0033] 2. This invention integrates depth and color images to detect the state of the conveyor belt, resulting in accurate and stable results.

[0034] 3. This invention improves equipment reliability and reduces production accidents caused by equipment failure.

[0035] 4. This invention provides an intelligent upgrade to the production status of conveyor belts, reducing labor intensity. Detailed Implementation

[0036] In this embodiment, a method for determining the production status of a tobacco conveyor belt by fusing depth images and color images includes the following steps:

[0037] Step 1: Install a depth camera and a color camera at the locations on the conveyor belt to be inspected. For more effective inspection, ensure that both cameras are installed in the same location to capture depth and color images of the conveyor belt. The camera's imaging plane should be parallel to the conveyor belt; the vertical directions of the depth and color images should coincide with the direction of the conveyor belt's movement; the direction of the conveyor belt's movement should be the y-direction of the image; and the x-direction of the image should be perpendicular to the direction of the conveyor belt's movement.

[0038] Step 2: To obtain an online detection benchmark, acquire a depth image T when the conveyor belt is stationary and there are no tobacco leaves. null And calculate T null The sum of all depth values ​​in ST null ;

[0039] Step 3: Acquire a color image I1 of the conveyor belt in operation, and let the width and height of I1 be W and H, respectively;

[0040] Step 4: Use a deep learning model to detect I1 to determine whether there are tobacco leaves on the conveyor belt. If there are tobacco leaves, proceed to step 5; otherwise, return to step 3 to continue collecting data.

[0041] Step 5: Acquire the color image I2 and two consecutive depth images T1 and T2 on the next conveyor belt, and calculate the sum of all depth values ​​in T1 ST1 and the sum of all depth values ​​in T2 ST2; the role of color image I2 is to work with color image I1 to obtain the optical flow field information of I1.

[0042] Step 6: Based on color image I2, use the Lucas-Kanade algorithm to calculate the optical flow field of any i-th pixel p in color image I1. i velocity V(x) i , y i )={v i,x , v i,y}, where (x i , y i Let p be the i-th pixel in the color image I1. i Position coordinates, v i,x and v i,y p, which is the i-th pixel. i The velocity values ​​are in the horizontal and vertical directions; the velocity set V of color image I1 is composed of the velocities of all pixels in the color image I1, i∈[1,W×H]; in this velocity set, the value in the y-direction reflects the state of the conveyor belt in the normal direction of movement, and the value in the x-direction reflects the state of the conveyor belt perpendicular to the direction of movement. Under normal circumstances, there should be no movement in the x-direction unless the conveyor belt deviates from its designated path.

[0043] Step 7: Calculate the average value of the x-direction of the velocity set V using Equation (1). :

[0044] (1)

[0045] Step 8: Calculate the average value of the y-direction of the velocity set V using Equation (2). :

[0046] (2)

[0047] Step 9: If > th_vx, then set the identifier s b = 1 and give an alarm to indicate that the conveyor belt is running off track. Here, th_vx represents the optical flow field velocity threshold in the x-direction; otherwise, execute Step 10. Using the average value to determine whether the conveyor belt is running off track can suppress the influence of noise.

[0048] Step 10: If < th_min, then set the identifier s b = 2, indicating that the conveyor belt is in a stationary state. Here, th_min represents the minimum velocity threshold; otherwise, execute Step 11.

[0049] Step 11: If < th_low, then set the identifier s b = 3, indicating that the conveyor belt is suspected of running too slowly. Here, th_low represents the velocity too slow threshold; otherwise, execute Step 12.

[0050] Step 12: If > th_max, then set the identifier s b = 4 and give an alarm to indicate that the conveyor belt is running too fast. Here, th_max represents the velocity too fast threshold; otherwise, set the identifier s b = 5, indicating that the conveyor belt is running at a normal speed.

[0051] Step 13: If s b = 3, then assign counter + 1 to counter; otherwise, set the count value counter = 0.

[0052] Step 14: If counter > th_num, then set s b = 6 and give an alarm to indicate that the conveyor belt is running too slowly. Here, th_num is the cumulative frame number threshold; only when the conveyor belt continuously runs too slowly and reaches a certain threshold will it be determined that the conveyor belt is running too slowly, and there may be a slipping phenomenon. This is because when the conveyor belt starts or stops, the conveyor belt speed may be lower than the normal speed.

[0053] Step 15: Calculate the first depth difference ST 1= ST1 - ST null , calculate the second depth difference ST 2= ST2 - ST null ; calculate the third depth difference ST = ST1 - ST2;

[0054] Step 16: If ST1 < th_tf_min and ST2 < th_tf_min, then set the depth identifier s m = 1, indicating that there is no tobacco leaf on the conveyor belt. Otherwise, execute Step 17, where th_tf_min is the minimum height threshold of the tobacco leaf; the data in the depth image is insensitive to a very small amount of tobacco leaf on the conveyor belt, so this situation needs to be filtered out. In production, this situation will occur at the head and tail of the material.

[0055] Step 17: If ST > th_var holds, then set the depth identifier s m = 2, indicating that the conveyor belt is transporting the head or tail of the material. Otherwise, execute Step 18, where th_var is the height change threshold of the tobacco leaf.

[0056] Step 18: If ST1 > th_s or ST2 > th_s, then set the depth identifier s m = 3, indicating that there is too much tobacco leaf on the conveyor belt. Otherwise, set the depth identifier s m = 4, indicating that the height of the tobacco leaf on the conveyor belt is normal, where th_s is the maximum height threshold of the tobacco leaf;

[0057] Step 19: If s m = 2, then assign counter_tf + 1 to counter_tf. Otherwise, set the depth count value counter_tf = 0;

[0058] Step 20: If counter_tf > th_tf_num, then set the depth identifier s m = 5 and alarm to indicate that the head or tail of the material on the conveyor belt is being transported abnormally, where th_tf_num is the threshold of the cumulative number of frames of depth data.

[0059] Step 21: If s b = 5 and s m=3, and an alarm will be triggered indicating an abnormal height of tobacco leaves on the conveyor belt during production. If the conveyor belt speed is normal, but there are too many tobacco leaves, it may cause blockages in other equipment stages, so an alarm needs to be triggered in this case.

[0060] In this embodiment, an electronic device includes a memory and a processor. The memory stores a program that supports the processor in executing the above-described method, and the processor is configured to execute the program stored in the memory.

[0061] In this embodiment, a computer-readable storage medium stores a computer program, which is executed by a processor to perform the steps of the above method.

Claims

1. A method for determining the production status of a tobacco conveyor belt by fusing depth images and color images, characterized in that: The following steps are included: Step 1: Use a camera to collect a depth image and a color image on the conveyor belt, and the imaging plane of the camera is parallel to the conveyor belt; the vertical direction of the depth image and the color image coincides with the moving direction of the conveyor belt; the moving direction of the conveyor belt is taken as the y direction of the image; and the x direction of the image is perpendicular to the moving direction of the conveyor belt; Step 2: Collect a depth image T when the conveyor belt is stationary and without tobacco leaves null , and calculate T null The sum of all depth values ​​in ST null ; Step 3: Collect a color image I1 of the conveyor belt in working state, and let the width and height of I1 be W and H respectively; Step 4: Use the deep learning model to detect I1 to determine whether there are tobacco leaves on the conveyor belt. If there are tobacco leaves, execute step 5; otherwise, return to step 3 to continue collecting; Step 5: collect the next color image I2 on the conveyor belt and two consecutive depth images T1 and T2, and calculate the sum ST1 of all depth values ​​in T1 and the sum ST2 of all depth values ​​in T2; Step 6: Based on the color image I2, the Lucas-Kanade algorithm is used to calculate the value of any i-th pixel point p in the optical flow field of the color image I1. i The speed V(x i , y i )={v i,x , v i,y }, where (x i , y i ) is the i-th pixel p in the color image I1 i The position coordinates, v i,x and v i,y are the i-th pixel p i The speed values ​​in the horizontal and vertical directions; the speed set V of the color image I1 is composed of the speeds of all pixels in the color image I1, i∈[1,W×H]; Step 7: Use formula (1) to calculate the average value of the velocity set V in the x direction : (1) Step 8: Use formula (2) to calculate the average value of the velocity set V in the y direction : (2) Step 9: If >th_vx, then let identifier s b =1, and an alarm is given to indicate that the conveyor belt is deviating, where th_vx represents the optical flow field velocity threshold in the x direction; otherwise, execute step 10; Step Ten: If <th_min, then set the identifier s b = 2, indicating that the conveyor belt is in a stationary state, where th_min represents the speed minimum threshold; otherwise, execute Step Eleven; Step Eleven: If <th_low, then set the identifier s b = 3, indicating that the conveyor belt is suspected of running too slowly, where th_low represents the slow speed threshold; otherwise, execute Step Twelve; Step 12: If >th_max, then let identifier s b =4, and an alarm is given to indicate that the conveyor belt speed is too high, where th_max represents the speed threshold. Otherwise, let the identifier s b =5, indicating that the conveyor belt speed is normal; Step 13: If s b =3, then assign counter+1 to counter, otherwise, set the count value counter=0; Step 14: If counter>th_num, then let s b =6, and an alarm is given to indicate that the conveyor belt speed is too slow, where th_num is the cumulative frame number threshold; Step 15: Calculate the first depth difference ST 1= ST1-ST null , calculate the second depth difference ST 2= ST2-ST null ; Calculate the third depth difference ST = ST1-ST2; Step Sixteen: If ST1 < th_tf_min and ST2 < th_tf_min, then set the depth identifier s m = 1, indicating that there is no tobacco leaf on the conveyor belt; otherwise, execute Step Seventeen, where th_tf_min is the minimum height threshold of the tobacco leaf. Step 17: If If ST>th_var holds, let the depth identifier s m =2, indicating that the conveyor belt is conveying the head or tail of the material, otherwise, execute step 18, where th_var is the threshold value of tobacco leaf height change; Step 18: If ST1>th_s or ST2>th_s, then let the depth identifier s m =3, indicating that there are too many tobacco leaves on the conveyor belt, otherwise, let the depth mark s m =4, indicating that the height of tobacco leaves on the conveyor belt is normal, where th_s is the maximum height threshold of tobacco leaves; Step 19: If s m =2, assign counter_tf+1 to counter_tf, otherwise, set the depth count value counter_tf=0; Step 20: If counter_tf>th_tf_num, then let the depth identifier s m =5, and an alarm is given to indicate that the conveyor belt is feeding abnormally at the head or tail of the material, where th_tf_num is the threshold of the accumulated frames of depth data; Step 21: If s b =5 and s m =3, and an alarm will be given to indicate that the height of tobacco leaves on the conveyor belt is abnormal during the production process.

2. An electronic device, comprising a memory and a processor, characterized in that: The memory is used to store a program that supports the processor to execute the method for determining the production status of a tobacco conveyor belt as described in claim 1, and the processor is configured to execute the program stored in the memory.

3. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by the processor, the steps of the method for determining the production status of a tobacco conveyor belt according to claim 1 are executed.

Images