Leaf storage cabinet production status determination method based on fusion of depth image and color image

Abstract

The present invention discloses a method for determining the production status of a leaf storage cabinet by fusing depth images and color images. After collecting the depth image and the color image, the method detects the color image and the depth image respectively to determine whether there are tobacco leaves on the conveyor belt of the leaf storage cabinet, the direction of movement of the conveyor belt, whether tobacco leaves are falling on the conveyor belt, and the change in the amount of tobacco leaves in the leaf storage cabinet. Then, the depth detection result and the color image detection result are fused to determine the current production status of the leaf storage cabinet. The present invention can monitor the production process of the leaf storage cabinet in real time, so as to realize timely alarm when there is a fault, improve production efficiency, improve production process, and ensure the stability of tobacco products.

Description

Technical Field

[0001] The present invention belongs to the technical field of machine vision, and in particular to a method for determining the production status of a leaf storage cabinet by fusing a depth image and a color image. Background Art

[0002] In the tobacco production of the cigarette industry, the leaf storage cabinet, as a device for tobacco storage and fermentation, may have driving failure, motor failure, and feeder failure. Furthermore, these equipment failures will lead to failure of feeding in the leaf storage cabinet. In the production process, in order to prevent the above phenomenon from happening, operators are often arranged to conduct regular inspections. However, in the leaf storage workshop, the temperature is high and the humidity is high, which is not suitable for regular inspections. If the staff cannot find the fault in time, it will affect the production progress. Summary of the invention

[0003] The present invention aims to solve the deficiencies of the above-mentioned prior art and proposes a method for determining the production status of a leaf storage cabinet by integrating depth data and color images, so as to monitor the production process of the leaf storage cabinet in real time and continuously, thereby realizing timely alarm in case of faults, improving production efficiency, and improving production processes to ensure the stability of tobacco products.

[0004] In order to achieve the above-mentioned purpose, the present invention adopts the following technical scheme:

[0005] The method for determining the production status of a leaf storage cabinet by fusing depth image data and color image data of the present invention is characterized in that it comprises the following steps:

[0006] Step 1: Install a camera for collecting depth image data and color image data at one end of the leaf storage cabinet, and the two cameras are installed at the same position;

[0007] Step 2: When there is no production and when there is no tobacco leaf in the leaf storage cabinet, collect a depth image T null ;

[0008] Step 3: Depth image T null Manually mark the regions of interest in the

[0009] Step 4: Calculate the depth image T null The sum of all depth values ​​in the region of interest is denoted as ST null ;

[0010] Step 5: During the production process, two depth image data T1 and T2 are collected at a certain interval, and the sum of all depth values ​​in the region of interest in T1 and T2 is calculated respectively, which are recorded as ST1 and ST2;

[0011] Step 6: Calculate the first depth difference ST= ST1-ST2, second depth difference ST1= ST1-ST null and the third depth difference ST2= ST2-ST null ;

[0012] Step 7: If ST1 <th null and ST2 <th null If both are true, it means that the leaf storage cabinet is unloaded. Otherwise, execute step eight, where th null is the no-load depth threshold;

[0013] Step 8: If ST <-th t If it is established, it means that the state of the leaf storage cabinet is feeding, otherwise, execute step nine, where th t is the threshold value of tobacco leaf depth change;

[0014] Step 9: If ST >th t If it is established, it means that the state of the leaf storage cabinet is discharging, otherwise, it means that the state of the leaf storage cabinet is storing;

[0015] Step 10: continuously collect two color images I1 and I2 of the leaf storage cabinet;

[0016] Step 11: Use deep learning to detect the vehicle position P1={x1, y1, w1, h1} in I1 and the vehicle position P2={x2, y2, w2, h2} in I2, where x1 and y1 are the horizontal and vertical center coordinates of the vehicle in I1, w1 is the width of the vehicle in I1, h1 is the height of the vehicle in I1, x2 and y2 are the horizontal and vertical center coordinates of the vehicle in I2, w2 is the width of the vehicle in I2, and h2 is the height of the vehicle in I2;

[0017] Step 12: Calculate the width change ratio of the vehicle using formula (1) :

[0018] (1)

[0019] In formula (1), min means the minimum value operation;

[0020] Step 13: Calculate the height change ratio of the vehicle using formula (2) :

[0021] (2)

[0022] Step 14: If <-th_var and <-th_var, it means that the vehicle is moving away from the camera, that is, from near to far; otherwise, execute step 15; where th_var is the threshold for judging the change of vehicle width;

[0023] Step 15: If >th_var and >th_var, it means the vehicle is moving towards the camera, that is, from far to near; otherwise, it means the vehicle is stationary;

[0024] Step 16: Use a deep learning model to detect the material dropping state of the crane in I1. If it is detected that the crane in I1 is dropping tobacco leaves, it means that the material dropping state of the crane in I1 is feeding; otherwise, it means that the material dropping state of the crane in I1 is not feeding;

[0025] Step 17: Use a deep learning model to detect the stock state of the traveling vehicle in I1. If tobacco leaves are detected on the traveling belt in I1, it means that there is material in the traveling vehicle in I1; otherwise, it means that there is no material in the traveling vehicle in I1.

[0026] Step 18: If the state of the leaf storage cabinet is empty and the material dropping state of the crane is feeding, it means that the production state of the leaf storage cabinet is production start; otherwise, execute step 19;

[0027] Step 19: If the status of the leaf storage cabinet is discharging and the material dropping status of the crane is feeding, it means that the production status of the leaf storage cabinet is a crane failure. Otherwise, execute step 20;

[0028] Step 20: If the status of the leaf storage cabinet is discharging and the material dropping status of the crane is not feeding, it means that the production status of the leaf storage cabinet is normal. Otherwise, execute step 21;

[0029] Step 21: If the state of the leaf storage cabinet is material storage and the material dropping state of the crane is material feeding, it means that the production state of the leaf storage cabinet is a crane failure. Otherwise, execute step 22;

[0030] Step 22: If the status of the leaf storage cabinet is material storage and the material dropping status of the crane is no material feeding, it means that the production status of the leaf storage cabinet is normal, otherwise, execute step 23;

[0031] Step 23: If the state of the leaf storage cabinet is feeding, and the material dropping state of the crane is feeding, and the movement direction of the crane is from far to near, it means that the production state of the leaf storage cabinet is normal, otherwise, execute step 24;

[0032] Step 23: If the state of the leaf storage cabinet is feeding, and the material dropping state of the crane is feeding, and the movement direction of the crane is from near to far, it means that the production state of the leaf storage cabinet is a crane failure, otherwise, execute step 24;

[0033] Step 24: If the state of the leaf storage cabinet is feeding, and the material dropping state of the crane is not feeding, and the movement direction of the crane is from near to far, and the material storage state of the crane is not out of material, it means that the production state of the leaf storage cabinet is a crane failure, otherwise, execute step 25;

[0034] Step 25: If the status of the leaf storage cabinet is feeding, and the material dropping status of the crane is not feeding, and the movement direction of the crane is from near to far, and the material storage status of the crane is material, it means that the production status of the leaf storage cabinet is normal, otherwise, execute step 26;

[0035] Step 26: If the status of the leaf storage cabinet is feeding, and the material dropping status of the crane is not feeding, and the movement direction of the crane is from far to near, and the material storage status of the crane is no material, then it means that the production status of the leaf storage cabinet is the end of production, otherwise, it means that the production status of the leaf storage cabinet is a crane failure.

[0036] An electronic device of the present invention includes a memory and a processor, wherein the memory is used to store a program that supports the processor to execute the method for determining the production status of a leaf storage cabinet, and the processor is configured to execute the program stored in the memory.

[0037] The present invention provides a computer-readable storage medium, wherein a computer program is stored on the computer-readable storage medium, and the computer program executes the steps of the method for determining the production status of a leaf storage cabinet when the computer program is run by a processor.

[0038] Compared with the prior art, the present invention has the following beneficial effects:

[0039] 1. The present invention adopts a decision tree to complete the production process equipment detection, with clear logic and fast detection speed.

[0040] 2. The depth model detection of the present invention has accurate position.

[0041] 3. The present invention uses depth data to determine the feeding state of the leaf storage cabinet, and the result has good stability.

[0042] 4. Improved equipment reliability and reduced production accidents caused by equipment failure.

[0043] 5. The production status of leaf storage cabinets is upgraded to be intelligent, reducing labor intensity. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 The figure is a logic diagram of the method of the present invention. DETAILED DESCRIPTION

[0045] In this embodiment, a method for determining the production status of a leaf storage cabinet by fusing depth image data and color image data includes the following steps:

[0046] Step 1: Install a camera for collecting depth image data and color image data at one end of the leaf storage cabinet. The two cameras are required to be installed at the same position.

[0047] Step 2: When production is not in progress and there is no tobacco leaf in the leaf storage cabinet, collect a depth image T null ; This image is used as a reference for calculating depth;

[0048] Step 3: In the depth image, manually mark the range that needs to be processed, that is, the region of interest of the depth image data. In order to improve the accuracy of the data, the region of interest needs to be concentrated inside the leaf storage cabinet.

[0049] Step 4: Calculate the depth image T null The sum of all depth values ​​in the region of interest is denoted as ST null .

[0050] Step 5: During the production process, collect two depth image data T1 and T2 at intervals of more than 1 minute. Calculate the sum of all depth values ​​in the region of interest in T1 and T2, respectively, and record them as ST1 and ST2.

[0051] Step 6: Calculate the first depth difference ST=ST1-ST2, the second depth difference ST1= ST1-ST null and the third depth difference ST2=ST2-ST null .

[0052] Step 7: If ST1 <th null and ST2 <th null If both are true, it means that the leaf storage cabinet is unloaded. Otherwise, execute step eight, where th null is the no-load depth threshold.

[0053] Step 8: If ST <-th t If it is established, it means that the leaf storage cabinet is in the feeding state. Otherwise, execute step nine, where th t is the threshold value of tobacco leaf depth change;

[0054] Step 9: If ST >th tIf it is established, it means that the state of the leaf storage cabinet is discharging, otherwise, it means that the state of the leaf storage cabinet is storing;

[0055] Step 10: Continuously collect two leaf storage cabinet color images I1 and I2.

[0056] Step 11: Use deep learning to detect the vehicle position P1={x1, y1, w1, h1} in I1 and the vehicle position P2={x2, y2, w2, h2} in I2, where x1 and y1 are the horizontal and vertical center coordinates of the vehicle in image I1, w1 is the width of the vehicle in image I1, h1 is the height of the vehicle in image I1, x2 and y2 are the horizontal and vertical center coordinates of the vehicle in image I2, w2 is the width of the vehicle in image I2, and h2 is the height of the vehicle in image I2;

[0057] Step 12: Calculate the width change ratio of the vehicle using formula (1) :

[0058] (1)

[0059] In formula (1), min represents the minimum value operation.

[0060] Step 13: Calculate the height change ratio of the vehicle using formula (2) :

[0061] (2)

[0062] Step 14: If <-th_var and <-th_var, it means that the vehicle is moving away from the camera, that is, from near to far; otherwise, execute step 15; wherein th_var is the threshold for judging the change of vehicle width.

[0063] Step 15: If >th_var and >th_var, it means the vehicle is moving towards the camera, that is, from far to near; otherwise, it means the vehicle is stationary;

[0064] Step 16: Use a deep learning model to detect the dropping state of the crane in image I1. If it is detected that the crane in image I1 is dropping tobacco leaves, it means that the dropping state of the crane in I1 is feeding; otherwise, it means that the dropping state of the crane in I1 is not feeding.

[0065] Step 17: Use the deep learning model to detect the stock status of the traveling vehicle in I1. If tobacco leaves are detected on the traveling belt in I1, it means that there is material on the traveling vehicle, otherwise it means that there is no material;

[0066] Step 18: If the leaf storage cabinet is in the empty state and the crane is in the feeding state, it means that the production state is the start of production; otherwise, go to step 19; Figure 1 The production status determination decision tree is fully displayed.

[0067] Step 19: If the leaf storage cabinet is in the discharging state and the crane is in the feeding state, it means that the production state is a crane failure. Otherwise, go to step 20.

[0068] Step 20: If the leaf storage cabinet is in the state of discharging and the crane is in the state of not feeding, it means that the production state is normal. Otherwise, go to step 21;

[0069] Step 21: If the leaf storage cabinet is in the storage state and the crane material dropping state is in the feeding state, it means that the production state is a crane failure. Otherwise, execute step 22;

[0070] Step 22: If the leaf storage cabinet is in the state of stocking and the crane is in the state of not feeding, it means that the production state is normal, otherwise, execute step 23;

[0071] Step 23: If the leaf storage cabinet is in the feeding state, and the crane is in the feeding state, and the crane is moving from far to near, it means that the production state is normal, otherwise, execute step 24;

[0072] Step 23: If the leaf storage cabinet is in the feeding state, and the crane is in the feeding state, and the crane is moving from near to far, it means that the production state is a crane failure, otherwise, execute step 24;

[0073] Step 24: If the leaf storage cabinet is in feeding state, and the crane material dropping state is not in feeding state, and the crane movement direction is from near to far, and the crane material storage state is not out of material, it means that the production state is a crane failure, otherwise, execute step 25;

[0074] Step 25: If the leaf storage cabinet is in the feeding state, and the crane material dropping state is not in the feeding state, and the crane movement direction is from near to far, and the crane material storage state has material, it means that the production state is normal, otherwise, execute step 26;

[0075] Step 26: If the leaf storage cabinet is in feeding state, and the crane dropping state is not in feeding state, and the crane moving direction is from far to near, and the crane storage state is no material, it means that the production state is over, otherwise, it means that the production state is crane failure.

[0076] In this embodiment, an electronic device includes a memory and a processor, wherein the memory is used to store a program that supports the processor to execute the above method, and the processor is configured to execute the program stored in the memory.

[0077] In this embodiment, a computer-readable storage medium stores a computer program on the computer-readable storage medium, and the computer program executes the steps of the above method when executed by a processor.

Claims

1. A method for determining the production status of leaf storage cabinets by fusing depth image data and color image data, characterized in that: The steps include: Step 1: Install a camera for collecting depth image data and color image data at one end of the leaf storage cabinet, and the two cameras are installed at the same position; Step 2: When there is no production and when there is no tobacco leaf in the leaf storage cabinet, collect a depth image T null ; Step 3: Depth image T null Manually mark the regions of interest in the Step 4: Calculate the depth image T null The sum of all depth values ​​in the region of interest is denoted as ST null ; Step 5: During the production process, two depth image data T1 and T2 are collected at a certain interval, and the sum of all depth values ​​in the region of interest in T1 and T2 is calculated respectively, which are recorded as ST1 and ST2; Step 6: Calculate the first depth difference ST= ST1-ST2, second depth difference ST1= ST1-ST null and the third depth difference ST2= ST2-ST null ; Step 7: If ST1 <th null and ST2 <th null If both are true, it means that the leaf storage cabinet is unloaded. Otherwise, execute step eight, where th null is the no-load depth threshold; Step 8: If ST <-th t If it is established, it means that the state of the leaf storage cabinet is feeding, otherwise, execute step nine, where th t is the threshold value of tobacco leaf depth change; Step 9: If ST >th t If it is established, it means that the state of the leaf storage cabinet is discharging, otherwise, it means that the state of the leaf storage cabinet is storing; Step 10: continuously collect two color images I1 and I2 of the leaf storage cabinet; Step 11: Use deep learning to detect the vehicle position P1={x1, y1, w1, h1} in I1 and the vehicle position P2={x2, y2, w2, h2} in I2, where x1 and y1 are the horizontal and vertical center coordinates of the vehicle in I1, w1 is the width of the vehicle in I1, h1 is the height of the vehicle in I1, x2 and y2 are the horizontal and vertical center coordinates of the vehicle in I2, w2 is the width of the vehicle in I2, and h2 is the height of the vehicle in I2; Step 12: Calculate the width change ratio of the vehicle using formula (1) : (1) In formula (1), min means the minimum value operation; Step 13: Calculate the height change ratio of the vehicle using formula (2) : (2) Step 14: If <-th_var and <-th_var, it means that the vehicle is moving away from the camera, that is, from near to far; otherwise, execute step 15; where th_var is the threshold for judging the change of vehicle width; Step 15: If >th_var and >th_var, it means the vehicle is moving towards the camera, that is, from far to near; otherwise, it means the vehicle is stationary; Step 16: Use a deep learning model to detect the material dropping state of the crane in I1. If it is detected that the crane in I1 is dropping tobacco leaves, it means that the material dropping state of the crane in I1 is feeding; otherwise, it means that the material dropping state of the crane in I1 is not feeding; Step 17: Use a deep learning model to detect the stock state of the traveling vehicle in I1. If tobacco leaves are detected on the traveling belt in I1, it means that there is material in the traveling vehicle in I1; otherwise, it means that there is no material in the traveling vehicle in I1. Step 18: If the state of the leaf storage cabinet is empty and the material dropping state of the crane is feeding, it means that the production state of the leaf storage cabinet is production start; otherwise, execute step 19; Step 19: If the status of the leaf storage cabinet is discharging and the material dropping status of the crane is feeding, it means that the production status of the leaf storage cabinet is a crane failure. Otherwise, execute step 20; Step 20: If the status of the leaf storage cabinet is discharging and the material dropping status of the crane is not feeding, it means that the production status of the leaf storage cabinet is normal. Otherwise, execute step 21; Step 21: If the state of the leaf storage cabinet is material storage and the material dropping state of the crane is material feeding, it means that the production state of the leaf storage cabinet is a crane failure. Otherwise, execute step 22; Step 22: If the status of the leaf storage cabinet is material storage and the material dropping status of the crane is no material feeding, it means that the production status of the leaf storage cabinet is normal, otherwise, execute step 23; Step 23: If the state of the leaf storage cabinet is feeding, and the material dropping state of the crane is feeding, and the movement direction of the crane is from far to near, it means that the production state of the leaf storage cabinet is normal, otherwise, execute step 24; Step 23: If the state of the leaf storage cabinet is feeding, and the material dropping state of the crane is feeding, and the movement direction of the crane is from near to far, it means that the production state of the leaf storage cabinet is a crane failure, otherwise, execute step 24; Step 24: If the state of the leaf storage cabinet is feeding, and the material dropping state of the crane is not feeding, and the movement direction of the crane is from near to far, and the material storage state of the crane is not out of material, it means that the production state of the leaf storage cabinet is a crane failure, otherwise, execute step 25; Step 25: If the status of the leaf storage cabinet is feeding, and the material dropping status of the crane is not feeding, and the movement direction of the crane is from near to far, and the material storage status of the crane is material, it means that the production status of the leaf storage cabinet is normal, otherwise, execute step 26; Step 26: If the status of the leaf storage cabinet is feeding, and the material dropping status of the crane is not feeding, and the movement direction of the crane is from far to near, and the material storage status of the crane is no material, then it means that the production status of the leaf storage cabinet is the end of production, otherwise, it means that the production status of the leaf storage cabinet is a crane failure.

2. An electronic device, comprising a memory and a processor, characterized in that: The memory is used to store a program that supports the processor to execute the method for determining the production status of the leaf storage cabinet according to claim 1, and the processor is configured to execute the program stored in the memory.

3. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by the processor, the steps of the method for determining the production status of the leaf storage cabinet according to claim 1 are executed.

Images