Cigarette box stack type anomaly detection method and device based on binocular vision
Through binocular vision technology, the problem of smoking box stacking abnormality detection is solved, and the existing technology cannot obtain the attitude and shape information of cigarette box and detect the stacking abnormality is realized, real-time detection of cigarette box position and posture and depth map comparison of cigarette box position and posture is realized. , improves the accuracy of palletization and system reliability.
Patent Information
- Application Number
- CN202410957968.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-17
- Publication Date
- 2024-11-22
AI Technical Summary
The existing palletizing robotic arms cannot obtain grabbing posture and shape information when detecting abnormalities of cigarette boxes, and cannot detect abnormalities in palletized cigarette boxes. The torque sensor is susceptible to interference from external environment, resulting in false alarms and misjudgments, which reduces the System reliability and accuracy.
The smoke box stacking abnormality detection method based on binocular vision is used to identify the category, position and posture of the cigarette box through stereoscopic visual reconstruction, and the depth map of the current stack is compared with the template to determine whether there are abnormalities.
Real-time detection of the position, posture and shape of the cigarette box is realized, the accuracy of palletization is improved, and abnormal conditions of the palletized cigarette box can be fully detected, which enhances the reliability and accuracy of the system.
Smart Images

Figure CN119006369A_ABST
Abstract
Description
Technical Field
[0001] The invention relates to the technical field of machine vision, and in particular to a method and a device for detecting cigarette box stack abnormality based on binocular vision. Background Art
[0002] Industrial palletizing robots, with their low cost and high efficiency, free people from heavy and monotonous palletizing work, improve production efficiency, reduce production costs, and shorten production cycles. The industrial robot palletizing system is generally composed of multiple subsystems. The electrical control subsystem controls the workflow and rhythm of the entire palletizing system. The material transmission subsystem transfers the material to the robot working area. The pallet conveying system transfers the pallet to the designated location. The robot arm then completes the cargo palletizing operation, and finally the forklift system or other transfer equipment transports the goods.
[0003] The palletizing robot arm has a high working frequency and a fast palletizing speed, but it is also prone to some problems, such as empty grabbing by the robot arm, bag switching causing palletizing abnormalities, and inconsistent warehouse quantities. However, the robot arm continues to perform the palletizing action according to the established procedures. If the on-site personnel do not handle it in time and urgently stop the palletizing robot, it may lead to serious consequences such as pallet collapse and even material damage.
[0004] The existing palletizing robot arm has a torque sensor installed on the gripping end or gripping fixture. This sensor can monitor the force and torque applied by the robot arm to the cigarette box. Under normal circumstances, when the robot arm grabs the cigarette box and palletizes it, the torque sensor will record a stable force and torque, which is caused by the weight of the cigarette box and the stable force applied by the robot arm. However, when an abnormal situation occurs, such as a package drop or empty grab, the torque sensor will record abnormal force and torque. For example, if the robot arm grabs the cigarette box, but suddenly loses weight during the lifting or moving process, the torque sensor will detect this sudden force reduction or weight loss. In addition, if the robot arm tilts or falls over during palletizing, the torque sensor will also record abnormal force and torque. By monitoring the data recorded by the torque sensor in real time and setting corresponding thresholds and algorithms to detect abnormal situations, the palletizing system can detect and handle abnormalities in a timely manner to ensure the stability and safety of the palletizing process.
[0005] However, the method of installing a torque sensor on the palletizing robot arm to detect whether the cigarette box is abnormal still has the following problems:
[0006] 1. The torque sensor is mainly used to detect the force applied by the robot arm when grasping and placing the cigarette box, but it cannot provide information about the grasping posture or shape. Therefore, for some abnormal situations, such as the cigarette box rotating or tilting during the grasping process, the torque sensor may not be able to detect this abnormality because the grasping weight has not changed.
[0007] 2. The torque sensor can only detect the situation of the cigarette boxes being grabbed and placed, but cannot detect the situation of the stacked cigarette boxes that have been palletized. If the stacked cigarette boxes collapse, the robot arm will not be able to detect it.
[0008] 3. Torque sensors are easily affected by external environmental factors, such as vibration, temperature changes or interference from stray forces, which may lead to false alarms or misjudgments of abnormal conditions, thereby reducing the reliability and accuracy of the system. Summary of the invention
[0009] To solve the above problems, the present application provides a method and device for detecting cigarette box stack anomalies based on binocular vision, which identifies the type, position and posture of the current cigarette box through stereoscopic vision reconstruction, and then compares the current stack depth map with the template to determine whether there is any abnormality in the stack.
[0010] The technical solution adopted by the present invention to solve the technical problem is:
[0011] The present invention provides a method for detecting abnormality of cigarette box stack type based on binocular vision, comprising the following steps:
[0012] Step S1: Create a depth map template
[0013] When each type of stack is stacked for the first time, a binocular camera is used to collect images of each stack after the stacking of each cigarette box is completed and stereoscopic vision reconstruction is performed to form a series of depth maps and establish a template;
[0014] Step S2: Smoke box detection and identification
[0015] During the palletizing process, binocular cameras continuously collect images and perform stereoscopic vision reconstruction to identify the type and position of the current cigarette box, estimate the posture of the image of the current cigarette box, and calculate the rotation angle and tilt angle of the current cigarette box to determine the direction of the current cigarette box in three-dimensional space.
[0016] Step S3: Anomaly detection and comparison
[0017] After the current cigarette box is stably placed, the depth map of the current stack is compared with the template in step S1, the difference between the current depth map and the template is calculated, and it is determined whether the difference between the current depth map and the template exceeds a preset threshold;
[0018] Step S4: Exception handling and recording
[0019] If the difference between the current depth map and the template exceeds a preset threshold, the system will determine that an abnormal situation has occurred, and immediately enter the abnormality handling process, and record the time, location and specific description of each abnormal situation.
[0020] As an improvement of the above solution, the stereoscopic vision reconstruction includes the following steps:
[0021] The images captured by the binocular camera are first preprocessed, and then the stereo matching algorithm is used to calculate the disparity of corresponding pixels in the image. Finally, the depth information of the cigarette box is calculated through the disparity and the calibration parameters of the binocular camera to generate a three-dimensional point cloud or depth map.
[0022] As an improvement to the above solution, step S2: cigarette box detection and identification
[0023] During the formal palletizing process, binocular cameras are used to continuously collect images and perform stereoscopic vision reconstruction. The images are segmented to separate the cigarette boxes from the background, and features are extracted from the segmented cigarette box areas. The extracted features are detected and classified using machine learning algorithms or deep learning models to identify the category and position of the current cigarette box, and the posture of the image of the current cigarette box is estimated. The rotation angle and tilt angle of the current cigarette box are calculated using the posture estimation algorithm to determine the direction of the current cigarette box in three-dimensional space.
[0024] As an improvement of the above solution, in step S2, if the rotation angle or tilt angle of the current cigarette box is abnormal, the palletizing operation is stopped, and an abnormal alarm is issued to remind the on-site staff to adjust the current cigarette box, and then continue the palletizing operation after the adjustment.
[0025] As an improvement of the above solution, in step S3, the comparison method between the depth map and the template is a structural similarity index, a mean square error or a peak signal-to-noise ratio.
[0026] As an improvement of the above solution, in step S4, the abnormality handling process includes stopping the palletizing operation and triggering an alarm to notify the operator.
[0027] As an improvement of the above solution, the exception handling process also includes adjusting the posture of the robot arm, repositioning the cigarette box or performing automatic corrective actions.
[0028] The present invention also provides a device for detecting abnormality of cigarette box stack type based on binocular vision, which is applied to any of the above-mentioned methods for detecting abnormality of cigarette box stack type based on binocular vision, comprising:
[0029] A robotic arm for grabbing cigarette boxes and stacking them;
[0030] A binocular camera, located above the stack, for collecting images;
[0031] A stack type analyzer, which is connected to the binocular camera and is used to identify the type and position of the cigarette box and calculate the rotation angle and tilt angle of the cigarette box;
[0032] An audible and visual alarm, which is in communication connection with the stack type analyzer and is used to sound an alarm when the stack is abnormal;
[0033] A controller is respectively connected to the mechanical arm and the stack type analyzer for controlling the operation of the mechanical arm.
[0034] As an improvement of the above-mentioned solution, the device for detecting abnormality of cigarette box stack type based on binocular vision further includes a display for displaying the stack image and the analysis result of the stack type analyzer.
[0035] The beneficial effects brought by the present invention are:
[0036] 1. The binocular camera can provide rich visual information and obtain the three-dimensional depth information of the stack through stereoscopic vision reconstruction, including the position, posture, shape, color, etc. of each cigarette box during palletizing. It can detect whether there is any abnormality in the current cigarette box, such as cigarette box position offset, tilt, package drop, etc., to ensure real-time detection of cigarette boxes during palletizing and improve the accuracy of palletizing;
[0037] 2. By comparing the current stack depth map with the template, it is possible to more comprehensively detect abnormal stack conditions, such as the collapse or tilt of stacked cigarette boxes. BRIEF DESCRIPTION OF THE DRAWINGS
[0038] The present invention will be further described below in conjunction with the accompanying drawings and specific embodiments.
[0039] Attached Figure 1 This is a flow chart of a method for detecting cigarette box stack abnormalities based on binocular vision in the present application. DETAILED DESCRIPTION
[0040] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0041] In the description of the present invention, it should be understood that the terms "upper side", "lower side", "upper end", "two ends", "width", "height", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside" and the like indicate orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the referred device or element must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as a limitation on the present invention.
[0042] In the present invention, unless otherwise clearly specified and limited, the terms "set", "install", "connect", "connect", "connection", "communication" and the like should be understood in a broad sense, for example, it can be a fixed connection or a detachable connection; it can be a mechanical connection; it can be directly connected or indirectly connected through an intermediate medium. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.
[0043] Example 1
[0044] This embodiment provides a method for detecting abnormality of cigarette box stack type based on binocular vision, comprising the following steps:
[0045] Step S1: Create a depth map template
[0046] When stacking each type of stack for the first time, the binocular camera collects images of each stack after stacking and performs stereoscopic vision reconstruction to form a series of depth maps and use them to build templates. For example, if the complete stack contains 30 cigarette boxes, 30 depth maps need to be collected and used to build templates.
[0047] Among them, the binocular camera can be flexibly configured and adjusted according to different palletizing task requirements to adapt to cigarette boxes of different sizes, shapes and colors.
[0048] Step S2: Smoke box detection and identification
[0049] During the formal palletizing process, binocular cameras are used to continuously collect images of the current cigarette box and perform stereoscopic vision reconstruction. At the same time, the stereoscopic images reconstructed by stereoscopic vision are segmented. The current cigarette box is separated from the background using segmentation algorithms such as segmentation based on color, texture or depth information, and various interference objects in the background, such as the robot body and bracket, are removed. The segmented current cigarette box area is subjected to feature extraction, including shape, color, texture and other features. The extracted features are detected and classified using machine learning algorithms or deep learning models to identify the category and position of the current cigarette box. The image of the current cigarette box is subjected to posture estimation using feature point matching, contour fitting or deep learning methods. The rotation angle and tilt angle of the current cigarette box are calculated through the posture estimation algorithm to determine the direction of the current cigarette box in three-dimensional space. If the rotation angle or tilt angle of the current cigarette box is abnormal, the robot arm stops running, and the abnormal result is displayed on the display, and an abnormal alarm is issued to remind the on-site staff to adjust the cigarette box. After the cigarette box is adjusted, the robot arm is restarted to continue the palletizing action.
[0050] Among them, the categories of cigarette boxes include ordinary cigarette boxes, special-shaped cigarette boxes, etc. Usually, the sizes of cigarette boxes of different categories are inconsistent.
[0051] Step S3: Stack anomaly detection and comparison
[0052] After recognizing that the current cigarette box is stably placed, the depth map of the current stack is compared with the template in step S1, the difference between the current depth map and the template is calculated, and it is determined whether the difference between the current depth map and the template exceeds a preset threshold. The depth map of the stack after the Nth cigarette box is placed is compared with the template of the stack after the Nth cigarette box is placed in step S1.
[0053] Step S4: Exception handling and recording
[0054] If the difference between the current depth map and the template exceeds the preset threshold, the system will determine that an abnormal situation has occurred and enter the abnormality handling process, while recording the time, location and specific description of each abnormal situation. Through the depth map template, the specific number of cigarette boxes placed when the abnormality occurred can be traced back, which helps to further analyze the cause of the abnormality and improve the stability of the palletizing system.
[0055] The stereo vision reconstruction includes the following steps: the images captured by the binocular camera are first preprocessed, including denoising and image correction, and then a stereo matching algorithm, such as region-based stereo matching, feature-based stereo matching or deep learning method, is used to calculate the disparity of corresponding pixel points in the image. Finally, the depth information of the cigarette box is calculated through the disparity and the calibration parameters of the binocular camera to generate a three-dimensional point cloud or depth map.
[0056] In step S3, the comparison method between the depth map and the template may be a structural similarity (SSIM) index, a mean square error (MSE) or a peak signal-to-noise ratio (PSNR), any one of which may be used.
[0057] In step S4, the exception handling process includes but is not limited to:
[0058] (1) Stop palletizing operation: stop executing palletizing related programs and suspend the palletizing action;
[0059] (2) Trigger an alarm to notify the operator: The alarm is sounded by an audible and visual alarm to notify the operator to take appropriate measures;
[0060] (3) Adjusting the posture of the robotic arm: Adjusting the posture of the robotic arm to avoid damaging the current stack. For example, the robotic arm can be raised to avoid crushing the cigarette box.
[0061] (4) Automatic correction action: When it is recognized that the rotation angle or tilt angle of the current cigarette box does not meet the standard, the posture of the current cigarette box is automatically corrected to meet the standard.
[0062] Example 2
[0063] This embodiment provides a device for detecting abnormality of cigarette box stack type based on binocular vision, which is applied to the method for detecting abnormality of cigarette box stack type based on binocular vision described in any one of the above-mentioned embodiments 1, including:
[0064] A robotic arm for grabbing cigarette boxes and stacking them;
[0065] A binocular camera is located above the stacks, installed on the ceiling 5 meters above each stack, and captures a 3D image of the stack vertically downward, and uses a binocular parallax stereo matching algorithm to calculate the stack height. When stacking cigarette boxes, the binocular camera also captures the image of the current cigarette box.
[0066] A stack type analyzer (embedded industrial computer) is connected to the binocular camera. The binocular camera uploads the collected stack images to the stack type analyzer, which identifies the type, position and posture of the cigarette box and performs stack depth map and template comparison analysis.
[0067] The sound and light alarm is connected to the stack type analyzer for communication. If the analysis result of the stack type analyzer is that the current cigarette box is abnormal or the completed stack is abnormal, the sound and light alarm is triggered to sound an alarm and notify the operator to take corresponding treatment measures.
[0068] The controller is respectively connected to the robot arm and the pallet type analyzer for communication. During palletizing, the controller controls the robot arm to grab the cigarette box and perform palletizing. If the analysis result of the pallet type analyzer is that the current cigarette box is abnormal or the completed pallet is abnormal, the controller adjusts the posture of the robot arm to adapt to the current abnormal state.
[0069] The display, binocular camera and stack type analyzer are respectively connected to the display for communication. The display is used to display the images collected by the binocular camera and the analysis results of the stack type analyzer so as to detect the situation of the stack in real time.
[0070] It should be noted that the above is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, it is still possible for those skilled in the art to modify the technical solutions described in the aforementioned embodiments or to make equivalent substitutions for some of the technical features therein. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.
Claims
1. A method for detecting abnormality of cigarette box stacks based on binocular vision, characterized in that: The following steps are involved: Step S1: Create a depth map template When each type of stack is stacked for the first time, a binocular camera is used to collect images of each stack after the stacking of each cigarette box is completed and stereoscopic vision reconstruction is performed to form a series of depth maps and establish a template; Step S2: Smoke box detection and identification During the palletizing process, binocular cameras continuously collect images and perform stereoscopic vision reconstruction to identify the type and position of the current cigarette box, estimate the posture of the image of the current cigarette box, and calculate the rotation angle and tilt angle of the current cigarette box to determine the direction of the current cigarette box in three-dimensional space. Step S3: Anomaly detection and comparison After the current cigarette box is stably placed, the depth map of the current stack is compared with the template in step S1, the difference between the current depth map and the template is calculated, and it is determined whether the difference between the current depth map and the template exceeds a preset threshold; Step S4: Exception handling and recording If the difference between the current depth map and the template exceeds a preset threshold, the system will determine that an abnormal situation has occurred, and immediately enter the abnormality handling process, and record the time, location and specific description of each abnormal situation.
2. The method for detecting abnormality of cigarette box stack type based on binocular vision according to claim 1, characterized in that: The stereoscopic vision reconstruction comprises the following steps: The images captured by the binocular camera are first preprocessed, and then the stereo matching algorithm is used to calculate the disparity of corresponding pixels in the image. Finally, the depth information of the cigarette box is calculated through the disparity and the calibration parameters of the binocular camera to generate a three-dimensional point cloud or depth map.
3. The method for detecting abnormality of cigarette box stack type based on binocular vision according to claim 1, characterized in that: Step S2: Smoke box detection and identification During the formal palletizing process, binocular cameras are used to continuously collect images and perform stereoscopic vision reconstruction. The images are segmented to separate the cigarette boxes from the background, and features are extracted from the segmented cigarette box areas. The extracted features are detected and classified using machine learning algorithms or deep learning models to identify the category and position of the current cigarette box, and the posture of the image of the current cigarette box is estimated. The rotation angle and tilt angle of the current cigarette box are calculated using the posture estimation algorithm to determine the direction of the current cigarette box in three-dimensional space.
4. A binocular vision-based cigarette box stack abnormality detection method according to claim 1 or 3, characterized in that: In step S2, if the current rotation angle or tilt angle of the cigarette box is abnormal, the palletizing operation is stopped, and an abnormal alarm is issued to remind the on-site staff to adjust the current cigarette box. The palletizing operation can be continued after the adjustment.
5. The method for detecting abnormality of cigarette box stack type based on binocular vision according to claim 1, characterized in that: In step S3, the depth map and the template are compared by a structural similarity index, a mean square error, or a peak signal-to-noise ratio.
6. The method for detecting abnormality of cigarette box stack type based on binocular vision according to claim 1, characterized in that: In step S4, the abnormality handling process includes stopping the palletizing operation and triggering an alarm to notify the operator.
7. The method for detecting abnormality of cigarette box stack type based on binocular vision according to claim 1, characterized in that: The exception handling process also includes adjusting the posture of the robot arm, repositioning the smoke box, or performing automatic corrective actions.
8. A device for detecting abnormality of cigarette box stack type based on binocular vision, the method being applied to the method for detecting abnormality of cigarette box stack type based on binocular vision as claimed in any one of claims 1 to 6, characterized in that: include A robotic arm for grabbing cigarette boxes and stacking them; A binocular camera, located above the stack, for collecting images; A stack type analyzer connected to the binocular camera for identifying the type, position and posture of the cigarette box and calculating the difference between the stack depth map and the template; An audible and visual alarm, which is in communication connection with the stack type analyzer and is used to sound an alarm when a smoke box or a stack is abnormal; A controller is respectively connected to the mechanical arm and the stack type analyzer for controlling the operation of the mechanical arm.
9. The binocular vision-based cigarette box stack abnormality detection device according to claim 8, characterized in that: The device for detecting abnormality of cigarette box stack type based on binocular vision also includes a display for displaying images of the cigarette box and the stack as well as the analysis result of the stack type analyzer.
Citation Information
Cited By
Corrugated board stacking defect detection method and system based on machine vision
CN121186087A
