A carton stacking control system based on machine vision

Abstract

The application relates to the technical field of carton stacking control, and particularly discloses a carton stacking control system based on machine vision, which comprises an image acquisition module, a height calculation module, a compensation amount generation module and a motion control module. The actual height value is calculated through the acquired top surface image, and the height deviation compensation amount is obtained by comparing the actual height value with the theoretical height value. Finally, the Z-axis target position of the stacking manipulator is corrected through the height deviation compensation amount. Meanwhile, the system is combined with a multi-dimensional compensation mechanism such as point cloud partition screening based on confidence weight, vibration soft compensation, moisture evaporation dynamic repair and X-Y plane deformation fine adjustment, so that the adaptive and collaborative correction of complex factors such as mechanical disturbance, environmental humidity and non-uniform deformation of the carton in the stacking process can be realized, and the positioning accuracy and stacking stability of multi-layer stacking are significantly improved.

Description

Technical Field

[0001] This invention relates to the field of carton palletizing control technology, specifically a carton palletizing control system based on machine vision. Background Technology

[0002] Palletizing is the final critical process before paper product packaging leaves the factory. In the packaging and printing industry, cardboard boxes differ from standard plastic turnover boxes. Their base material is corrugated cardboard, and their rigidity depends on their internal structure. They are also often treated with printing surface treatments such as lamination, varnishing, and hot stamping, resulting in significant changes in their physical properties. In current production, technicians typically use general palletizing control systems designed for bagged materials or rigid boxes. The core logic of these systems is to execute a pick-and-place cycle based on fixed coordinates preset by the teach pendant. This direct transfer of technology exposes the following subtle flaws that have not yet been recognized by the industry when dealing with the special object of operation: cardboard boxes. Existing general palletizing control systems only use the theoretical design height of the carton to calculate the target position of the end effector when controlling the robot arm to place the top layer of cartons. However, corrugated cartons are prone to deformation under pressure. Under the continuous pressure of multiple stacks, the lower layers of cartons (especially those with softer flutes like B-flute or E-flute) will experience localized collapses that are difficult to detect with the naked eye. This results in the actual top height of the carton being significantly lower than the theoretical value stored in the system. This change in the actual height of the stack caused by the deformation of the cartons under pressure occurs in real time and cannot be preset. When the system drives the robot arm to descend according to the theoretical height, the actual relative distance between the end effector and the surface of the carton has already deviated. If the deviation is positive (the actual height is lower than the theoretical value), the robot arm will press down excessively, directly crushing the corners of the carton and damaging the contents (such as wine bottles or electronic products). If the operator raises the set height to avoid crushing, the deviation will be negative, and the suction cup will not be able to effectively contact the surface of the carton, resulting in suction failure and the carton falling from a height.

[0003] The root cause of the above problems is that the existing general control system is essentially an open-loop position control based on an ideal geometric model. When the surface height of the palletized object changes during the palletizing process, the system cannot actively sense and adjust the downward position. Therefore, how to solve the problem of inaccurate downward position of the robot arm due to the deformation and non-uniform expansion of the carton under pressure is a problem that needs to be solved in this field. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention provides a machine vision-based carton palletizing control system that can effectively solve the problems mentioned in the background section.

[0005] To achieve the above objectives, the present invention provides the following technical solution: a machine vision-based carton palletizing control system, comprising an image acquisition module for capturing an image of the top surface of the carton already palletized below when the end effector of the palletizing robot moves directly above the carton to be palletized; a height calculation module for calculating the current actual height of the carton already palletized below based on the top surface image; a compensation amount generation module for comparing the current actual height with a pre-stored theoretical height to generate a height deviation compensation amount; and a motion control module for correcting the Z-axis downward target position of the palletizing robot when performing the placement action based on the height deviation compensation amount.

[0006] As a further solution, when the height calculation module re-includes the 3D spatial point cloud coordinates of the edge region that are not marked as collapse anomalies into the effective sampling point set, it calculates the overall confidence level of the edge region point cloud with a negative correlation according to the ratio as the first confidence level weight, and subtracts the attenuation amount determined by the ratio from the preset full confidence level. The final result is then assigned as the second confidence level weight to the 3D spatial point cloud coordinates of the central region, and the first confidence level weight is less than the second confidence level weight. The arithmetic mean is recalculated using a weighted average method, that is, the first confidence level weight and the second confidence level weight are used as the weighting coefficients of the corresponding 3D spatial point cloud coordinate Z-axis coordinate values. After recalculating the arithmetic mean using the weighted average method, the height calculation module also obtains the mechanical vibration frequency and vibration amplitude of the palletizing robot at the current shooting time, and determines whether the mechanical vibration frequency is within the preset carton resonance frequency range. If so, the arithmetic mean is weighted with the vibration confidence level weight determined by the negative correlation according to the vibration amplitude to obtain the current actual height value after vibration compensation.

[0007] As a further solution, after fusion and stitching, the height calculation module also determines whether the complete top surface image meets the preset residual reflective spot conditions. If it does, the pixel regions that meet the preset residual reflective spot conditions are identified as residual reflective spots. The preset residual reflective spot conditions indicate whether the grayscale value in the complete top surface image exceeds the preset speckle threshold and whether the area of ​​the connected region is less than the preset minimum spot area. If the residual reflective spots exist, the height calculation module extracts the gradient magnitude and gradient direction of the corresponding pixel neighborhood in the preprocessed image as gradient information, and uses the gradient information of the pixel neighborhood to determine the residual reflective spots. The reflective spots are repaired by pixel value interpolation and filling. The repaired complete top surface image is used for edge detection and three-dimensional spatial point cloud coordinate transformation. When the residual reflective spots are repaired by pixel value interpolation and filling using the gradient information of the pixel neighborhood, the height calculation module also obtains the time interval between the top surface image captured by the image acquisition module and the secondary top surface image. Based on the time interval and the preset moisture evaporation rate of the cardboard box surface, the change in gray value of the cardboard box surface is estimated, and the change in gray value is superimposed as a compensation factor to the interpolated pixel value to correct the inconsistency in gray value of image fusion caused by the time difference.

[0008] As a further solution, the compensation amount generation module also acquires the flute type information of the carton to be stacked, and looks up the compensation coefficient corresponding to the flute type information from a preset mapping table of carton flute type and compensation coefficient. The height deviation compensation amount is multiplied by the compensation coefficient to obtain the corrected height deviation compensation amount. The compensation amount generation module is also used to identify the surface printing pattern of the carton already stacked below based on the top surface image, extract the key feature points of the surface printing pattern, compare the distribution density of the key feature points with a preset standard density map to calculate the non-uniform expansion rate of the top surface of the carton in the horizontal and vertical planes, and generate a fine-tuning vector in the horizontal direction based on the non-uniform expansion rate. The fine-tuning vector and the corrected height deviation compensation amount are sent together to the motion control module to compensate for the interlayer alignment deviation caused by the non-uniform deformation of the carton already stacked below.

[0009] Compared with the prior art, the embodiments of the present invention have at least the following advantages or beneficial effects: (1) This invention provides a carton palletizing control system based on machine vision. The system has a progressive and relatively complete data processing and optimization mechanism designed in the height calculation module. Specifically, it includes key steps such as dividing the top contour area, determining the deviation between the Z-axis coordinate value and the arithmetic mean, and obtaining the confidence weight. The above steps help to filter out the influence of interference factors such as edge collapse and local reflection on height measurement step by step. At the same time, it can improve the confidence and robustness of the data that finally acts on the Z-axis downward compensation amount, and ensure the bonding accuracy between palletizing layers.

[0010] (2) After recalculating the arithmetic mean using a weighted average method, the present invention actively measures the mechanical vibration frequency and vibration amplitude of the palletizing robot at the current shooting moment, and corrects the actual height value in a soft compensation manner. This transforms the unavoidable mechanical vibration from a purely harmful interference source into a quantifiable and compensable measurement parameter. Especially when the vibration frequency falls into the resonance range of the carton, it effectively suppresses the potential damage of vibration amplification effect to the height measurement accuracy, significantly improves the system measurement stability under dynamic working conditions, and this soft compensation method does not require the addition of an independent physical damping platform for the image acquisition module, which can improve the data acquisition reliability of the palletizing robot under continuous high-speed operation.

[0011] (3) When performing pixel value interpolation and filling repair, this invention also introduces the dynamic characteristic of moisture evaporation in the carton into the visual image repair. That is, the time interval and moisture evaporation rate are estimated, and the estimation result is superimposed on the interpolated pixel value as a compensation factor. Existing technologies generally ignore the influence of time difference on image fusion. This solution accurately quantifies and compensates for this physical change to solve the long-standing visual detection problem in wet boxes or humid environments. It realizes dynamic perception and adaptive correction of the carton surface state over time and ensures the consistency of grayscale reference when fusing multiple frames of images. It fundamentally improves the environmental adaptability of visual measurement in high humidity environments. The repair algorithm is executed by the height calculation module, so that when the image acquisition module shoots a carton with high light reflection, it does not need to rely on high-cost polarized light sources or multiple angle-changing supplementary lighting equipment, thereby enhancing the hardware adaptability of the palletizing robot end vision system to complex lighting environments.

[0012] (4) This invention extends the height compensation problem of the Z-axis to the deformation compensation of the XY (horizontal and vertical) plane. That is, by extracting the key feature points of the natural visual speckle of the printed pattern on the surface of the stacked carton, and generating a fine-tuning vector based on the calculated non-uniform expansion rate, the non-uniform expansion of the carton is quantified and used to guide the fine-tuning of the stacking posture. This can solve the problem of misalignment between carton layers caused by temperature and humidity. The linear control mode that relies only on single height compensation in the traditional solution is upgraded to three-dimensional adaptive compensation in three-dimensional space, realizing the leap from passive adaptation to active deformation prediction and posture correction. This significantly improves the overall stability and neatness of the stacking pattern of multi-layer stacking. This three-dimensional adaptive compensation strategy is implemented by directly using the same frame of top surface image obtained by the image acquisition module and drives the motion control module to perform multi-axis linkage fine-tuning. There is no need to configure additional equipment for lateral deformation detection, which fully explores the hardware potential of the multi-degree-of-freedom motion of the palletizing robot. Attached Figure Description

[0013] The present invention will be further described with reference to the accompanying drawings, but the embodiments in the drawings do not constitute any limitation on the present invention. For those skilled in the art, other drawings can be obtained based on the following drawings without creative effort.

[0014] Figure 1 This is a schematic diagram of the system module connections of the present invention. Detailed Implementation

[0015] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0016] First embodiment, see reference Figure 1 The diagram shown is a schematic of the system module connection of the present invention. An embodiment of the present invention provides a technical solution: a carton palletizing control system based on machine vision, including an image acquisition module, a height calculation module, a compensation amount generation module, a motion control module, and a carton control database.

[0017] The image acquisition module is connected to the height calculation module, the height calculation module is connected to the compensation amount generation module, the compensation amount generation module is connected to the motion control module, and the image acquisition module, height calculation module, compensation amount generation module, and motion control module are all connected to the carton control database.

[0018] Specifically, the carton control database stores all preset values ​​of the above modules, including preset deviation thresholds, preset ratio thresholds, preset high gloss thresholds, preset reflectivity ratio thresholds, preset minimum spot area, theoretical height values, preset mapping tables of carton flute types and compensation coefficients, preset standard density diagrams, preset carton resonant frequency ranges, and carton surface moisture evaporation rates, etc. Based on the application of this embodiment in the carton palletizing scenario, the carton control database here can be a real-time process parameter database integrated into a factory-level Manufacturing Execution System (MES). This embodiment uses the MES process parameter database as an example to explain the detailed data processing and storage methods, as follows: The MES process parameter database interacts with each module via the OPC UA protocol. During the initialization phase, each module reads static parameters such as preset thresholds from the database and caches them in local memory. Dynamic data generated during operation, such as the calculated current actual height value and height deviation compensation amount, are written to the corresponding data table in the database in real time to form a complete traceability record of the palletizing process. The data interaction methods for other values ​​or between modules are similar to the read / write mechanism of the MES process parameter database, which is also based on the OPC UA protocol for real-time access. This example will not explain them in detail, and the value magnitude is related to the actual carton flute type and stacking layer number, which are not specifically limited in this example.

[0019] Preferably, the aforementioned carton control database and the palletizing robot controller are connected via industrial Ethernet (Ethernet / IP). The industrial camera of the image acquisition module is connected to the image processing industrial computer via the GigE Vision interface. The industrial computer runs a height calculation module and a compensation amount generation module. The industrial computer then sends the compensation amount to the motion control module (PLC) via the Ethernet / IP bus. The PLC ultimately controls the movement of each servo axis of the palletizing robot and the suction cup action of the end effector.

[0020] The image acquisition module is used to capture an image of the top surface of the already stacked cartons below when the end effector of the palletizing robot moves directly above the cartons to be palletized.

[0021] It should be explained that the aforementioned end effector refers to a multi-functional suction cup gripper installed at the end of the wrist flange of the palletizing robot. The imaging device is a high-resolution industrial CMOS camera integrated on the end effector bracket with the lens mounted vertically downwards. The final top image obtained is a two-dimensional grayscale image containing complete top information of the palletized cartons below.

[0022] When the end effector moves the carton to be stacked to a position directly above the already stacked carton, its Z-axis height is at the starting pre-alignment point of the placement action. At this time, taking a picture of the top surface of the already stacked carton below can directly and accurately measure the actual height of the target stack layer to be placed, providing real-time basis for subsequent precise downward compensation.

[0023] The height calculation module is used to calculate the current actual height of the stacked cartons below based on the top surface image.

[0024] Specifically, after the height calculation module acquires the top surface image, it performs grayscale conversion and filtering on the top surface image to obtain a preprocessed image, and performs edge detection on the preprocessed image to extract the top surface contour region of the stacked cartons below. The pixels in the top surface contour region are converted into three-dimensional spatial point cloud coordinates, and the current actual height value of the stacked cartons below is calculated based on the Z-axis coordinate value in the three-dimensional spatial point cloud coordinates.

[0025] It should be clarified that the aforementioned grayscale conversion and filtering specifically refer to performing weighted average grayscale conversion and Gaussian filtering denoising on the top image to obtain the preprocessed image; edge detection refers to using the Canny operator to extract edges from the preprocessed image, and the internal region enclosed by the final closed maximum outer contour is the top contour region of the stacked cardboard boxes below; the aforementioned conversion of pixels within the top contour region into three-dimensional spatial point cloud coordinates refers to combining pre-calibrated camera intrinsic parameters and hand-eye relationship matrix to convert the two-dimensional pixel coordinates (u, v) within the contour region into three-dimensional spatial point cloud coordinates (X, Y, Z) with reference to the robot's base coordinate system. For example, the camera intrinsic parameter matrix K includes the focal length (f... x f y ) and optical center coordinates (c x c y The hand-eye relationship matrix includes a rotation matrix R and a translation vector T. For a pixel (u, v) = (320, 240) within the top contour region, its corresponding depth value, obtained by laser structured light synchronization, is d = 850 mm. Therefore, the three-dimensional coordinates of this pixel in the camera coordinate system are (X... c =[d×(uc x )] / f x Y c =[d×(vc y )] / fy,Z c =d), and then use the hand-eye transformation matrix [R|T] to transform the point (X) in the camera coordinate system. c Y c Z c The coordinates are converted to the robot's base coordinate system to obtain the final (X, Y, Z) coordinate values. Finally, the Z-axis coordinate values ​​in the three-dimensional point cloud coordinates are averaged to obtain the current actual height of the stacked cartons below.

[0026] Specifically, before calculating the current actual height of the stacked cartons below based on the Z-axis coordinates in the three-dimensional point cloud coordinates, the top surface contour area is divided into a central area and an edge area. The three-dimensional point cloud coordinates in the central area are extracted as valid sampling points, and the three-dimensional point cloud coordinates in the edge area are removed. The arithmetic mean of the Z-axis coordinates of all valid sampling points is calculated, and the arithmetic mean is used as the current actual height of the stacked cartons below.

[0027] It should be explained that the above-mentioned top surface contour area is divided into a central area and an edge area. The division method involves taking the geometric center of the top surface contour area as the origin, and defining the inner area enclosed by reducing the length and width of the contour area to a preset ratio (e.g., 80%) of the original size as the central area, and defining the remaining part of the contour area as the edge area.

[0028] Furthermore, the height calculation module also calculates the deviation between the Z-axis coordinate value of each three-dimensional spatial point cloud coordinate within the edge region and the arithmetic mean. The deviation refers to the absolute value of the difference between the Z-axis coordinate value of a certain three-dimensional spatial point cloud coordinate within the edge region and the arithmetic mean. If the deviation between the Z-axis coordinate value of a certain three-dimensional spatial point cloud coordinate and the arithmetic mean exceeds a preset deviation threshold, then the three-dimensional spatial point cloud is marked as a collapse anomaly point, and the proportion of the number of collapse anomaly points to the total number of three-dimensional spatial point clouds within the edge region is counted. If the proportion exceeds a preset proportion threshold, the height calculation module further calculates the deviation between the Z-axis coordinate value of each three-dimensional spatial point cloud coordinate within the edge region and the arithmetic mean. When calculating the actual height value, the three-dimensional spatial point cloud coordinates corresponding to the collapse anomaly points are removed to avoid introducing Z-axis error components. That is, before calculating the height deviation compensation amount, the Z-axis coordinate values ​​corresponding to the collapse anomaly points are removed, and only the three-dimensional spatial point cloud coordinates of the areas that do not constitute collapse anomalies are used in the subsequent calculation. The height deviation compensation amount after removal is sent to the motion control module. At the same time, the height calculation module generates a carton collapse early warning signal and sends it to the upper monitoring system. This system refers to the SCADA system responsible for monitoring the entire line. It can highlight the abnormal collapse of the cartons in the stack on the display board to remind the operator to pay attention to the abnormal collapse of the cartons in the stack so that it can be checked and dealt with in a timely manner.

[0029] Specifically, if the ratio does not exceed the preset ratio threshold, the height calculation module will re-include the three-dimensional spatial point cloud coordinates of the edge area that are not marked as collapse anomalies into the set of valid sampling points, and recalculate the arithmetic mean of the Z-axis coordinate values ​​of all valid sampling points.

[0030] Furthermore, when the height calculation module re-includes the 3D spatial point cloud coordinates of the edge region that were not marked as collapse anomalies into the effective sampling point set, the overall confidence of the edge region point cloud is calculated based on the ratio with a negative correlation as the first confidence weight, i.e., the first confidence weight = 1 - the ratio of the number of collapse anomalies in the edge region to the total number of edge region point clouds; and after subtracting the attenuation amount determined by the ratio from the preset full confidence (usually 1), the final result is assigned as the second confidence weight to the 3D spatial point cloud coordinates of the central region, i.e., the second confidence weight = 1 - (ratio × attenuation coefficient), where the attenuation coefficient is an empirical constant, specifically determined through regression analysis of a large amount of on-site palletizing experimental data. For example, in the palletizing scenario of B-flute cartons with softer flutes and high surface reflectivity, the attenuation coefficient can be set to 0.3; and the first confidence weight is less than the second confidence weight.

[0031] Assuming the proportion of collapse anomalies in the edge region is 30%, the first confidence weight is 0.7; with an attenuation coefficient of 0.3, the second confidence weight is 1 - (0.3 × 0.3) = 0.91. This example is for illustration only and needs to be adjusted according to the actual situation to ensure that the first confidence weight is less than the second confidence weight.

[0032] The arithmetic mean is recalculated using a weighted average method, that is, the first confidence weight and the second confidence weight are used as weighting coefficients for the corresponding three-dimensional spatial point cloud Z-axis coordinate values, and this is defined as the arithmetic mean. ; Among them W center_i Z represents the second confidence weight of the i-th 3D point cloud coordinates in the central region. center_i W represents the Z-axis coordinate of the i-th 3D point cloud in the central region. edge_j Z represents the first confidence weight of the j-th valid point in the edge region. edge_j Let be the Z-axis coordinate of the j-th valid point in the edge region, i be the coordinate number of the three-dimensional spatial point cloud of the central region, i = 1, 2, 3, ..., m, m be the total number of three-dimensional spatial point cloud coordinates in the central region, and j be the valid point number of the edge region, j = 1, 2, 3, ..., n, n be the total number of valid points in the edge region.

[0033] After recalculating the arithmetic mean using a weighted average method, the height calculation module also acquires the mechanical vibration frequency and amplitude of the palletizing robot at the current shooting moment. This is achieved through real-time acquisition using a three-axis accelerometer mounted on the robot's base or upper arm. For example, the three-axis accelerometer continuously acquires the robot's acceleration time-domain signals in the X, Y, and Z directions at a sampling frequency of 1000Hz. After FFT transformation, the current dominant frequency is extracted as the mechanical vibration frequency. Simultaneously, the acceleration amplitude corresponding to the dominant frequency is integrally divided twice to obtain the displacement amplitude, which is then used as the vibration amplitude. The module then determines whether the mechanical vibration frequency is within a preset carton resonance frequency range. If so, the arithmetic mean is weighted by a vibration confidence weight determined based on a negative correlation with the vibration amplitude to obtain the current actual height value after vibration compensation, specifically H. final =Z avg ×[1-(A / A max ], where H final Z represents the current actual height after vibration compensation. avg The weighted average is the arithmetic mean of the Z-axis, where A is the real-time vibration amplitude. max The maximum allowable vibration amplitude of the system is determined by the mechanical structure design parameters of the palletizing robot and the maximum allowable steady-state vibration displacement amplitude of the servo drive system. It can be obtained by consulting the equipment technical specifications or by on-site modal testing and calibration.

[0034] Specifically, the preset carton resonance frequency range is not a pre-calibrated fixed value, but is dynamically identified and updated in real time by the height calculation module during the palletizing process. The specific identification process does not affect the current Z-axis downward calculation and control process, including the following steps: During steady-state operation of the robotic arm, the height calculation module continuously acquires the time-domain acceleration signals of the triaxial accelerometers in the Z-axis direction. It also extracts signal segments from the no-action gaps after the current placement action is completed (i.e., the gaps between the robotic arm hovering and waiting for the next carton or moving to the gripping position) to avoid interference from the impact and vibration of the palletizing action itself. Power spectral density (PSD) analysis is performed on the extracted Z-axis acceleration signal segments, extracting signals within a preset analysis frequency band (e.g., 5Hz-100Hz). By calculating the coherence function values ​​of the signals from the two accelerometers at the base and end effector of the robotic arm, peak frequencies with low coherence (coherence function value less than 0.3) are identified as the carton stack. The response frequency, denoted as fr, is centered on the real-time identified fr and fluctuates by a preset percentage (e.g., ±10%) to form the current dynamically updated carton resonance frequency range [fr×(1-10%), fr×(1+10%)], which is used in the judgment of the next placement action in the same layer or the next layer. This dynamic recognition process is executed as a low-priority background task within the CPU idle time slice where the image acquisition module and motion control module work together, ensuring that it does not occupy the real-time resources of the main computing and control process. When it is executed for the first time, the safe default range [15Hz, 30Hz] is used for the initial judgment until the first dynamic recognition result is updated.

[0035] In this embodiment, the purpose of obtaining the mechanical vibration frequency and vibration amplitude is that when the vibration frequency of the robot itself approaches the natural frequency of the cardboard stack and causes resonance, the visually measured Z-axis point cloud data will generate micron-level high-frequency fluctuations. These fluctuations are not the actual height changes of the cardboard surface. Therefore, it is necessary to reduce the weight or compensate the measured values ​​according to the vibration intensity in order to restore the true static height.

[0036] Specifically, before performing edge detection on the preprocessed image, the height calculation module also traverses the pixel matrix of the preprocessed image, sequentially reads the gray value G(u,v) stored at each pixel coordinate position (u,v), the gray value ranges from 0 to 255, and determines whether the gray value of each pixel in the preprocessed image exceeds a preset highlight threshold. If so, the connected region whose gray value exceeds the preset highlight threshold is determined as a highlight reflection region. A connected region refers to an image region composed of pixels whose gray values ​​all exceed the preset highlight threshold and are adjacent to each other.

[0037] If the high-brightness reflection area is detected, the height calculation module performs pixel value suppression processing on the high-brightness reflection area, that is, compresses the grayscale value of each pixel in the area to below a preset high-brightness threshold, for example, uniformly assigning a value of 95% of the preset high-brightness threshold; in order to reduce the interference of the high-brightness reflection area on edge detection, and then performs edge detection on the image.

[0038] Specifically, if the height calculation module detects that the area of ​​the high-gloss reflection region exceeds a preset reflectivity threshold, the height calculation module sends a retake command to the image acquisition module. Based on the distribution and size of the high-gloss reflection region, it calculates the minimum offset required for the second shot to avoid the current high-gloss reflection region as a preset distance. For example, if the high-gloss reflection region is located on the left edge of the top surface of the carton, its width is 1 / 4 of the top surface width, and its size is a continuous high-gloss strip 100 mm long and 50 mm wide, the calculated preset distance is a 55 mm rightward shift to ensure that the camera's second shot completely avoids this high-gloss reflection region. The motion control module controls the palletizing robot to move the end effector horizontally a preset distance before taking another shot to acquire a secondary top surface image.

[0039] The height calculation module merges and stitches the pixel regions outside the highlight reflection areas in the top surface image and the secondary top surface image. That is, it fills the highlight reflection area of ​​the first image with the pixels corresponding to the new positions in the secondary top surface image, and performs weighted smooth transition processing at the stitching seam to obtain a complete top surface image after highlight suppression, and performs edge detection based on the complete top surface image.

[0040] The above-mentioned processing method of secondary shooting and fusion stitching through horizontal offset effectively eliminates the interference of large-area high light reflection on the extraction of the top surface outline of the carton, avoids outline breakage or loss caused by reflection, and ultimately ensures that a complete and accurate top surface outline area can be obtained under various lighting conditions.

[0041] Furthermore, after the height calculation module performs fusion and stitching, it also determines whether the complete top surface image meets the preset residual reflective spot conditions. If it does, the pixel area that meets the preset residual reflective spot conditions is determined as a residual reflective spot.

[0042] The preset residual reflective spot condition means that the gray value in the complete top surface image exceeds the preset highlight threshold, and the area of ​​the connected region is smaller than the preset minimum spot area.

[0043] If the residual reflective spots exist, the height calculation module extracts the gradient magnitude and gradient direction of the corresponding pixel neighborhood in the preprocessed image as gradient information, uses the gradient information of the pixel neighborhood to perform pixel value interpolation and filling repair on the residual reflective spots, and uses the repaired complete top surface image for edge detection and three-dimensional spatial point cloud coordinate transformation.

[0044] It needs to be explained that the gradient magnitude and gradient direction of a pixel neighborhood represent the magnitude and direction of the rate of change of gray values ​​in the horizontal and vertical directions within the neighborhood of the pixel, respectively. Furthermore, the specific process of pixel value interpolation and filling repair starts from the pixels on the boundary of the residual reflective spot and gradually interpolates inwards along its gradient direction. The new gray value of each pixel to be filled is obtained by weighted averaging of the gray values ​​of known pixels in its neighborhood according to their gradient direction, i.e., I(p) = [∑{q∈N(p)}w(p,q)×I(q)] / [∑{q∈N(p)}w(p,q)], where p is the value to be filled. Let q be the q-th known pixel in the known pixel set N(p) in the neighborhood of the pixel to be filled. ∑{q∈N(p)} represents the summation of all known pixels q in the neighborhood set N(p) of the pixel to be filled. I(p) is the new gray value of the pixel to be filled. N(p) represents the known pixel set in the neighborhood of the pixel to be filled. I(q) is the gray value of the known pixel q in the neighborhood. w(p,q) is the weight, which is equal to the absolute value of the cosine of the angle between the line connecting points p and q and the gradient direction of point q, to ​​ensure that the known pixels closer to the gradient direction contribute more to the interpolation.

[0045] When using the gradient information of the pixel neighborhood to perform pixel value interpolation and filling repair on the residual reflective spots, the height calculation module also obtains the time interval between the top surface image captured by the image acquisition module and the secondary top surface image. Based on the time interval and the preset moisture evaporation rate of the cardboard box surface, it estimates the amount of gray value change on the cardboard box surface caused by moisture evaporation, and uses the amount of gray value change as a compensation factor to be superimposed on the interpolated pixel value to correct the image fusion gray value inconsistency caused by the time difference.

[0046] It should be noted that the aforementioned change in grayscale value of the cardboard box surface is actually the product of the evaporation rate of moisture on the cardboard box surface and the time interval; the aforementioned superposition refers to directly adding the change in grayscale value of the cardboard box surface to the calculated interpolated pixel value.

[0047] In the above embodiments, due to the time difference between the two shots, the evaporation of trace amounts of moisture on the surface of the cardboard box (especially the varnished or coated surface) may cause slight deviations in the overall grayscale of the images before and after the shooting in a high-temperature and dry environment. The amount of grayscale value change on the cardboard box surface is estimated by calculating the time interval and the interpolated pixel values ​​are compensated to ensure that the texture of the fused complete top surface image is consistent, so as to avoid edge detection errors caused by the grayscale differences of the image itself.

[0048] The compensation amount generation module is used to compare the current actual height value with the pre-stored theoretical height value, that is, the result of dividing the current actual height value by the theoretical height value, and use it as the height deviation compensation amount.

[0049] Specifically, the theoretical height value refers to the theoretical cumulative height from the palletizing reference surface (such as the pallet surface or the ground) to the top surface of the Nth layer of cartons currently being stacked. This value is calculated by multiplying the number of currently completed palletizing layers (N-1 layers) by the pre-stored theoretical design height of a single carton.

[0050] The motion control module is used to correct the Z-axis downward target position of the palletizing robot when performing the placement action based on the height deviation compensation amount.

[0051] It should be explained that the above correction is obtained by multiplying the original Z-axis downward target position coordinate value set by the teach pendant by the height deviation compensation amount. In addition, the Z-axis downward target position refers to the coordinate point that the end effector of the palletizing robot needs to reach when it moves downward in the vertical direction during the placement action.

[0052] In the above overall embodiment, by introducing a machine vision-based real-time height measurement and compensation mechanism, the traditional open-loop control mode can be completely changed, enabling the palletizing robot's downward movement to dynamically adapt to the actual height changes of the carton stack, fundamentally solving the problem of product crushing or adsorption failure caused by the carton being compressed and collapsed.

[0053] The first embodiment described above achieves dynamic compensation of the Z-axis height through visual feedback, effectively solving the problem of inaccurate downward displacement caused by the collapse of the carton under pressure. To further improve the compensation accuracy and adapt to the differences in deformation characteristics of different corrugated carton types and the interlayer misalignment caused by the non-uniform expansion of the carton surface, the present invention provides a second embodiment as follows: In the second embodiment, while keeping the other contents unchanged, there is a special case where different corrugation types of cartons lead to differences in their pressure-bearing deformation characteristics. That is, high-strength ACB corrugated cartons and low-strength BE corrugated cartons have drastically different deformations under the same pressure. This can lead to significant positioning deviations when using a uniform compensation strategy, such as insufficient compensation for soft-corrugated cartons or excessive compensation for hard-corrugated cartons. Therefore, adaptive compensation based on the corrugation type and non-uniform surface deformation of the cartons is also required. Specifically, the compensation amount generation module also obtains the corrugation type information of the cartons to be stacked, including the corrugation type (such as A-flute, B-flute, C-flute, E-flute or a combination thereof) of the current batch of cartons read from the MES production work order. It also searches for the compensation coefficient corresponding to the corrugation type information from a preset mapping table of carton corrugation type and compensation coefficient, and multiplies the height deviation compensation amount by the compensation coefficient to obtain the corrected height deviation compensation amount.

[0054] The compensation amount generation module is further used to identify the surface printing pattern of the stacked cartons below based on the top surface image through template matching and feature point detection, and extract key feature points of the surface printing pattern. For example, if the surface printing pattern is a crosshair mark for positioning, then the key feature point is the center intersection of the crosshairs. Therefore, its corresponding distribution density is represented by the number of identifiable crosshair center intersections per unit area. The distribution density of the key feature points is compared with a preset standard density map to calculate the non-uniform expansion rate of the top surface of the cartons in the XY plane, that is, non-uniform expansion rate = measured key feature point spacing / corresponding key feature point in the standard density map. The standard spacing minus 1, the distribution density reflects the number of identifiable key feature points per unit area. The higher the density, the more sufficient the key feature point spacing can be used for calculation. In specific calculations, first, the measured distribution density map and the standard density map are registered. The region corresponding to the current measured key feature point location is found in the standard density map. The pre-stored standard feature point spacing value within this region is extracted as the denominator. Then, the actual physical distance (calculated using a scale) between the pixel distance between two corresponding key feature points in the measured image is used as the numerator. Dividing by this numerator and subtracting 1 gives the non-uniform expansion rate of the local region. If the distribution density of a sub-region is lower than the preset minimum density threshold, then... This sub-region is marked as an unreliable region and does not participate in the expansion rate calculation. A horizontal fine-tuning vector is generated based on the non-uniform expansion rate. Specifically, the direction of the fine-tuning vector is aligned with the direction of the principal strain with the maximum expansion. Its magnitude is proportional to the product of the non-uniform expansion rate in that direction and the size of the carton's top surface. For example, if the calculated non-uniform expansion rate of the carton's top surface in the X direction is 2% and in the Y direction is 0.5%, then the principal strain direction is the X-axis, where the expansion is maximum. The direction of the fine-tuning vector is perpendicular to the X-axis, i.e., along the Y-axis. Assuming the size of the carton's top surface in the Y-axis direction is 400mm, and the scaling factor is 0.5, then the fine-tuning vector... The magnitude of the adjustment vector is 0.5 × 2% × 400mm = 4mm; therefore, the final fine-tuning vector is (0, 4mm), indicating that the end effector needs to translate 4mm in the positive Y-axis direction to compensate for the alignment deviation caused by non-uniform expansion; the fine-tuning vector and the corrected height deviation compensation amount are sent to the motion control module so that the palletizing robot can adaptively fine-tune the placement posture of the current carton to be palletized in the horizontal and vertical planes while the Z-axis is probing downwards. Here, fine-tuning means that the end effector translates in the horizontal plane along the direction and magnitude of the fine-tuning vector to compensate for the interlayer alignment deviation caused by the non-uniform deformation of the carton already palletized below.

[0055] Specifically, the preset standard density map is not ideal sample box data collected offline, but is generated online and dynamically updated by the height calculation module at the initial stage of the palletizing operation based on the real-time image of the current pallet position. Its specific generation and application process does not affect the normal palletizing cycle time, and specifically includes the following steps: After the motion control module completes the placement of the first layer of cartons in the current stack, and in a stable state where the cartons are not yet under load, the image acquisition module is triggered to capture an image of the top surface of the first layer of cartons according to a standard procedure. The height calculation module performs grayscale conversion, filtering, and highlight suppression on the top surface image of the first layer to ensure consistency with the image preprocessing algorithm in subsequent processing. Then, it identifies and extracts key feature points of the surface printing pattern in the image, calculates and records the precise position (u, v) of each key feature point in the image coordinate system and the Euclidean distance between adjacent feature points, and generates an initial standard density map before deformation that perfectly matches the batch of cartons, the current lighting, and the shooting posture. The initial standard density map and the data of each key feature point in it are compared with the unique identifier of the current stack (such as the pallet number). After being bound with RFID codes, the data is stored in the carton control database and serves as the sole comparison benchmark for calculating the non-uniform expansion rate of all subsequent layers of cartons in that stacking position. When the stacking height exceeds the preset number of layers (e.g., more than 5 layers), or when there is a significant change in ambient light (determined by a sudden change in the factory lighting switch or the average grayscale value of the image), the system can automatically trigger a benchmark update process: After all stacking operations in the current stacking position are completed, before stacking begins in the new stacking position, an image of the top surface of the first layer of cartons in the new stacking position is taken, and the distribution of key feature points at this time is calculated as a new standard density map to replace the old benchmark. This update only applies to the new stacking position and does not retrospectively correct the completed old stacking position to compensate for the overall benchmark shift of the lower layer of cartons caused by continuous pressure, while ensuring that each layer in the same stacking position uses a consistent comparison benchmark.

[0056] The second embodiment, based on solving the problem of carton height collapse, further considers the influence of the flute structure on the compression amount and the non-uniform expansion caused by the printing surface treatment. By comprehensively compensating for height and horizontal direction, it can achieve higher precision interlayer alignment and effectively improve the stability and overall quality of high stacking.

[0057] The above description is merely an example and illustration of the structure of the present invention. Those skilled in the art can make various modifications or additions to the specific embodiments described, or use similar methods to replace them, as long as they do not deviate from the structure of the invention or exceed the scope defined by the present invention, they should all fall within the protection scope of the present invention.

Claims

1. A carton palletizing control system based on machine vision, characterized in that, include: The image acquisition module is used to capture an image of the top surface of the already stacked cartons below when the end effector of the palletizing robot moves directly above the cartons to be palletized. The height calculation module is used to calculate the current actual height of the stacked cartons below based on the top surface image; The compensation amount generation module is used to compare the current actual height value with the pre-stored theoretical height value to generate a height deviation compensation amount; The motion control module is used to correct the Z-axis downward target position of the palletizing robot when performing the placement action based on the height deviation compensation amount.

2. The carton palletizing control system based on machine vision according to claim 1, characterized in that: After the height calculation module acquires the top surface image, it performs grayscale conversion and filtering on the top surface image to obtain a preprocessed image. Then, it performs edge detection on the preprocessed image to extract the top surface contour region of the stacked cartons below. The pixels in the top surface contour region are converted into three-dimensional spatial point cloud coordinates. The current actual height value of the stacked cartons below is calculated based on the Z-axis coordinate value in the three-dimensional spatial point cloud coordinates.

3. The carton palletizing control system based on machine vision according to claim 2, characterized in that: Before calculating the current actual height of the stacked cartons below based on the Z-axis coordinate value in the three-dimensional point cloud coordinates, the top surface contour area is divided into a central area and an edge area. The three-dimensional point cloud coordinates in the central area are extracted as valid sampling points, and the three-dimensional point cloud coordinates in the edge area are removed. The arithmetic mean of the Z-axis coordinate values ​​of all valid sampling points is calculated, and the arithmetic mean is used as the current actual height of the stacked cartons below.

4. The carton palletizing control system based on machine vision according to claim 3, characterized in that: The height calculation module also calculates the deviation between the Z-axis coordinate value of each three-dimensional spatial point cloud coordinate in the edge region and the arithmetic mean. If the deviation between the Z-axis coordinate value of a certain three-dimensional spatial point cloud coordinate and the arithmetic mean exceeds a preset deviation threshold, the three-dimensional spatial point cloud is marked as a collapse anomaly point. The module also counts the proportion of the number of collapse anomalies to the total number of three-dimensional spatial point clouds in the edge region. If the proportion exceeds a preset proportion threshold, the height calculation module removes the three-dimensional spatial point cloud coordinates corresponding to the collapse anomaly point when calculating the current actual height value, and sends the height deviation compensation amount after removal to the motion control module. At the same time, it generates a cardboard box collapse warning signal and sends it to the upper monitoring system.

5. The carton palletizing control system based on machine vision according to claim 4, characterized in that: If the ratio does not exceed the preset ratio threshold, the height calculation module will re-include the three-dimensional spatial point cloud coordinates of the edge area that are not marked as collapse anomalies into the set of valid sampling points, and recalculate the arithmetic mean of the Z-axis coordinate values ​​of all valid sampling points.

6. The carton palletizing control system based on machine vision according to claim 5, characterized in that: When the height calculation module re-includes the three-dimensional spatial point cloud coordinates of the edge region that were not marked as collapse anomalies into the set of effective sampling points, it calculates the overall confidence of the edge region point cloud with a negative correlation according to the ratio as the first confidence weight, and subtracts the attenuation amount determined by the ratio from the preset full confidence, and assigns the final result as the second confidence weight to the three-dimensional spatial point cloud coordinates of the central region, and the first confidence weight is less than the second confidence weight; The arithmetic mean is recalculated using a weighted average method, that is, the first confidence weight and the second confidence weight are used as the weighting coefficients for the corresponding three-dimensional spatial point cloud coordinate Z-axis coordinate values; After recalculating the arithmetic mean using a weighted average method, the height calculation module also obtains the mechanical vibration frequency and vibration amplitude of the palletizing robot at the current shooting moment, determines whether the mechanical vibration frequency is within the preset carton resonance frequency range, and if so, weights the arithmetic mean with the vibration confidence weight determined based on the vibration amplitude with a negative correlation to obtain the current actual height value after vibration compensation.

7. The carton palletizing control system based on machine vision according to claim 2, characterized in that: Before performing edge detection on the preprocessed image, the height calculation module also acquires and determines whether the gray value of each pixel in the preprocessed image exceeds a preset specular threshold. If so, the connected region whose gray value exceeds the preset specular threshold is determined as a specular reflection region. If the high-brightness reflection area is detected, the height calculation module performs pixel value suppression processing on the high-brightness reflection area to reduce the interference of the high-brightness reflection area on edge detection, and then performs edge detection on the image.

8. The carton palletizing control system based on machine vision according to claim 7, characterized in that: If the height calculation module detects that the area of ​​the high-light reflection region accounts for more than the area of ​​the top surface contour region, the height calculation module sends a retake command to the image acquisition module. Based on the distribution position and size of the high-light reflection region, the module calculates the minimum offset required to avoid the current high-light reflection region in the second shot as a preset distance. The module then controls the palletizing robot to move the end effector horizontally by the preset distance and take another shot to obtain a second top surface image. The height calculation module merges and stitches the pixel regions outside the highlight reflection areas in the top surface image and the secondary top surface image to obtain a complete top surface image after highlight suppression, and performs edge detection based on the complete top surface image.

9. The carton palletizing control system based on machine vision according to claim 8, characterized in that: After the height calculation module performs fusion and stitching, it also determines whether the complete top surface image meets the preset residual reflective spot conditions. If it does, the pixel area that meets the preset residual reflective spot conditions is determined as a residual reflective spot. The preset residual reflective spot condition means that the gray value in the complete top surface image exceeds the preset highlight threshold and the area of ​​the connected region is smaller than the preset minimum spot area. If the residual reflective spots exist, the height calculation module extracts the gradient magnitude and gradient direction of the corresponding pixel neighborhood in the preprocessed image as gradient information, uses the gradient information of the pixel neighborhood to perform pixel value interpolation and filling repair on the residual reflective spots, and uses the repaired complete top surface image for edge detection and three-dimensional spatial point cloud coordinate transformation. When using the gradient information of the pixel neighborhood to perform pixel value interpolation and filling repair on the residual reflective spots, the height calculation module also obtains the time interval between the top surface image captured by the image acquisition module and the secondary top surface image. Based on the time interval and the preset moisture evaporation rate of the cardboard box surface, the module estimates the amount of gray value change on the cardboard box surface and uses the amount of gray value change as a compensation factor to be superimposed on the interpolated pixel value to correct the image fusion gray value inconsistency caused by the time difference.

10. The carton palletizing control system based on machine vision according to claim 1, characterized in that: The compensation amount generation module also obtains the flute type information of the carton to be stacked, and looks up the compensation coefficient corresponding to the flute type information from the preset mapping table of carton flute type and compensation coefficient. The height deviation compensation amount is multiplied by the compensation coefficient to obtain the corrected height deviation compensation amount. The compensation amount generation module is also used to identify the surface printing pattern of the stacked cartons below based on the top surface image, extract the key feature points of the surface printing pattern, compare the distribution density of the key feature points with a preset standard density map to calculate the non-uniform expansion rate of the top surface of the cartons in the XY plane, and generate a horizontal fine-tuning vector based on the non-uniform expansion rate. The fine-tuning vector and the corrected height deviation compensation amount are sent together to the motion control module to compensate for the interlayer alignment deviation caused by the non-uniform deformation of the stacked cartons below.

Images