An automated paper cup tracking, sorting, and conveying system based on vision detection

Abstract

This invention discloses an automatic tracking, sorting, and conveying system for paper cups based on vision detection, belonging to the field of industrial automation technology. Addressing the problem that existing technologies use fixed-speed models for position prediction, which cannot compensate for conveyor belt speed fluctuations and the inertial slippage of lightweight paper cups leading to high grasping failure rates, this invention uses dual-source observation of encoders and visual optical flow to obtain the instantaneous speed of the conveyor belt and the actual speed of the paper cups. It establishes a state estimation model including position, speed, and sliding displacement, employs an adaptive filtering algorithm to estimate the actual motion state of the paper cups in real time, dynamically adjusts the grasping sequence based on the sliding displacement, and iteratively optimizes the estimation parameters using historical prediction errors. This effectively compensates for speed fluctuations and sliding displacements, significantly reduces position prediction errors, and improves the grasping success rate, making it suitable for automatic sorting operations on paper cup production lines.

Description

Technical Field

[0001] This invention relates to the field of industrial automation technology, and more specifically, to an automatic tracking, sorting and conveying system for paper cups based on vision detection. Background Technology

[0002] In the finished product sorting section of the paper cup manufacturing industry, vision-based robotic automated sorting systems have been widely used. Existing technology typically uses industrial cameras to capture images of paper cups on a conveyor belt, extracts the planar coordinates of the cups using image processing algorithms, records the acquisition time, and then performs position prediction calculations using a fixed conveyor belt speed and a fixed system delay time. This yields the theoretical time and predicted coordinates for the cup to reach the robot's grasping position. Finally, the coordinate information is transmitted to the robot controller in a preset data format to execute the grasping action. Such systems can achieve basic sorting functions under ideal conditions where the conveyor belt runs stably and there is no relative slippage between the cups and the conveyor belt. Their position prediction models typically assume a constant and known conveyor belt speed, a fixed system processing delay, and no relative motion between the cups and the conveyor belt.

[0003] The existing technology has the following shortcomings: In actual operation, conveyor belts are affected by factors such as load changes and mechanical wear, resulting in fluctuations between the actual operating speed and the nominal speed. Fixed speed models cannot reflect these real-time changes. Paper cups, as lightweight and thin-walled containers, typically weigh only a few grams. During the start-up, stopping, or speed change of the conveyor belt, they will experience slight slippage relative to the conveyor belt due to inertia. Existing position prediction models do not consider this slippage effect. The total delay time from image acquisition to robot grasping varies dynamically with factors such as the number of paper cups in the image and image complexity. Fixed delay compensation methods cannot adapt to this variation. These shortcomings lead to a large deviation between the predicted grasping position and the actual position, exceeding the grasping tolerance range of the robot's suction cup, resulting in a high grasping failure rate.

[0004] To address the above problems, this invention proposes a solution. Summary of the Invention

[0005] In order to overcome the above-mentioned defects of the prior art, embodiments of the present invention provide a visual detection-based automatic tracking, sorting and conveying system for paper cups to solve the problems mentioned in the background art.

[0006] To achieve the above objectives, the present invention provides the following technical solution: An automatic tracking, sorting and conveying system for paper cups based on vision detection includes: a reference coordinate acquisition module, a dual-source velocity observation module, a state delay compensation module and a prediction closed-loop correction module, with signal connections between the modules; The reference coordinate acquisition module: by defining the camera coordinate system and triggering at equal time intervals, it acquires images of paper cups and extracts planar physical coordinates, synchronously records the acquisition time, and uses a linear model to calculate and predict the grab position based on the preset running speed of the conveyor belt and the fixed system delay. It identifies individual paper cups by combining the acquisition time and the horizontal coordinates, encapsulates the coordinate data, and outputs it. Dual-source speed observation module: The encoder pulse counting differential calculation of the roller angular displacement, combined with the roller diameter to convert the instantaneous speed of the conveyor belt, visual feature point corner detection and optical flow tracking, convert pixel displacement into physical displacement, calculate the actual speed of the paper cup, compare the dual-source speed difference to obtain the sliding speed, numerical integration to accumulate the sliding displacement, timestamp alignment and interpolation of encoder data and visual speed data, and output synchronous observation sequence. State delay compensation module: Defines a state vector including position, velocity, and sliding displacement, constructs a state transition matrix and an observation matrix, uses adaptive filtering for recursive estimation, and in the prediction stage, recursively estimates the state and uncertainty based on the kinematic model. It dynamically adjusts the process uncertainty measurement matrix based on the conveyor belt acceleration amplitude, dynamically adjusts the observation uncertainty measurement matrix based on image sharpness and contrast scores, and dynamically estimates the system processing delay time based on the number of paper cups and edge complexity in the image. Predictive closed-loop correction module: Based on the state estimation results, the difference between the instantaneous speed and sliding speed of the conveyor belt is integrated over the estimation period to obtain the cumulative displacement. The predicted grasping position is obtained by superimposing the acquisition time position. The grasping time is estimated based on the grasping distance and dynamic delay. The module determines whether to delay and wait to restore stable contact based on the sliding displacement amplitude. Historical prediction errors are recorded, and the covariance and variance are statistically analyzed. The process and observation uncertainty measurement matrix are iteratively updated with the learning rate. The trigger time for taking pictures is calculated in reverse. The grasping command is encapsulated and optimized before being output.

[0007] In a preferred embodiment, the reference coordinate acquisition module includes the following steps: A user coordinate system is established with the upper left corner of the camera's field of view as the origin, the first axis along the direction of the conveyor belt's movement, and the second axis perpendicular to the direction of the conveyor belt's movement. The calibrated distance between the camera's field of view and the robot's grasping area is stored. The camera is triggered to take pictures at a preset cycle and the acquisition time is recorded synchronously. The acquired images are preprocessed by grayscale conversion, filtering, and contrast enhancement. The outline of the paper cup is extracted by edge detection and converted into physical coordinates. Based on the preset operating speed of the conveyor belt and the delay of the fixed system, a linear model is used to calculate the theoretical time for the paper cup to move to the edge of the camera's field of view, the theoretical time to reach the grasping position, and the predicted lateral coordinates. Individual paper cups are identified by a combination of the acquisition time and the horizontal coordinate, and the coordinate data is packaged and output in a preset format that includes the total number of targets, the type of paper cup, the vertical coordinate, the horizontal coordinate, and the deflection angle.

[0008] In a preferred embodiment, the dual-source velocity observation module includes the following steps: The encoder pulse count is read in the first sampling period, the angular displacement is calculated differentially and converted into the instantaneous speed of the conveyor belt; the pixel displacement of feature points is obtained through corner detection and optical flow tracking between consecutive image frames, and after being converted into physical displacement, the mean and variance of the actual speed of the paper cup are calculated. The sliding speed is obtained by subtracting the instantaneous speed of the conveyor belt from the actual speed of the paper cup. The sliding displacement is accumulated by numerical integration. After interpolating and aligning the encoder and visual speed data, a synchronous observation sequence is output.

[0009] In a preferred embodiment, the state delay compensation module includes the following steps: Define a state vector for each paper cup, including position, velocity, and sliding displacement, construct a state transition matrix and an observation matrix, and use adaptive filtering to recursively estimate the state; The uncertainty measurement matrix of the process is dynamically adjusted based on the acceleration amplitude of the conveyor belt, the uncertainty measurement matrix of the observation is dynamically adjusted based on the image sharpness and contrast score, and the system processing delay time is dynamically estimated based on the number of paper cups and the edge complexity.

[0010] In a preferred embodiment, the prediction closed-loop correction module includes the following steps: The cumulative displacement is obtained by integrating the difference between the instantaneous speed and sliding speed of the conveyor belt estimated over the estimation period. The position estimate at the acquisition time is superimposed to predict the gripping position. The gripping time is calculated based on the gripping distance and dynamic delay estimate. When the estimated sliding displacement exceeds the threshold, the delay is waited to restore stable contact. Record historical prediction errors and calculate covariance and variance. Iterate and update the process and observation uncertainty measurement matrix with learning rate, reverse calculate the trigger time for taking pictures, and encapsulate the predicted capture position, capture time and deflection angle as capture command output.

[0011] The technical effects and advantages of the present invention, a vision-based automatic paper cup tracking, sorting and conveying system, are as follows: This invention, through a dynamic rate sensing and sliding feature recognition module, accurately matches the real-time state of conveyor belt speed fluctuations and the relative sliding of paper cups, completely solving the position prediction deviation problem caused by fixed parameter presets in existing systems. Based on the existing parallel robotic arm vision sorting architecture, it modifies the existing platform without requiring large-scale modifications; only the acquisition unit and algorithm logic need to be added, significantly reducing modification and deployment costs and adapting to the upgrade needs of most existing paper cup sorting production lines. Through a prediction deviation closed-loop iterative optimization module, a deviation sample library is built and algorithm parameters are dynamically adjusted to achieve continuous self-optimization of the prediction model. This allows for long-term adaptation to changes in operating conditions within the scenario, maintaining system stability and accuracy and reducing the frequency of subsequent maintenance. It effectively eliminates problems such as conveyor line jamming and classification errors, improving the sorting efficiency of high-speed soft paper cups, meeting the sorting needs of large-scale production scenarios, and significantly reducing the time and economic costs of manual re-inspection and downtime maintenance. Attached Figure Description

[0012] Figure 1 This is a schematic diagram of a paper cup automatic tracking, sorting and conveying system module based on vision detection according to the present invention. Detailed Implementation

[0013] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0014] Example Please see Figure 1 As shown, this invention discloses an automatic paper cup tracking, sorting and conveying system based on visual detection, including: a reference coordinate acquisition module, a dual-source speed observation module, a state delay compensation module and a prediction closed-loop correction module, with signal connections between each module; The reference coordinate acquisition module: by defining the camera coordinate system and triggering at equal time intervals, it acquires images of paper cups and extracts planar physical coordinates, synchronously records the acquisition time, and uses a linear model to calculate and predict the grab position based on the preset running speed of the conveyor belt and the fixed system delay. It identifies individual paper cups by combining the acquisition time and the horizontal coordinates, encapsulates the coordinate data, and outputs it. Dual-source speed observation module: The encoder pulse counting differential calculation of the roller angular displacement, combined with the roller diameter to convert the instantaneous speed of the conveyor belt, visual feature point corner detection and optical flow tracking, convert pixel displacement into physical displacement, calculate the actual speed of the paper cup, compare the dual-source speed difference to obtain the sliding speed, numerical integration to accumulate the sliding displacement, timestamp alignment and interpolation of encoder data and visual speed data, and output synchronous observation sequence. State delay compensation module: Defines a state vector including position, velocity, and sliding displacement, constructs a state transition matrix and an observation matrix, uses adaptive filtering for recursive estimation, and in the prediction stage, recursively estimates the state and uncertainty based on the kinematic model. It dynamically adjusts the process uncertainty measurement matrix based on the conveyor belt acceleration amplitude, dynamically adjusts the observation uncertainty measurement matrix based on image sharpness and contrast scores, and dynamically estimates the system processing delay time based on the number of paper cups and edge complexity in the image. Predictive closed-loop correction module: Based on the state estimation results, the difference between the instantaneous speed and sliding speed of the conveyor belt is integrated over the estimation period to obtain the cumulative displacement. The predicted grasping position is obtained by superimposing the acquisition time position. The grasping time is estimated based on the grasping distance and dynamic delay. The module determines whether to delay and wait to restore stable contact based on the sliding displacement amplitude. Historical prediction errors are recorded, and the covariance and variance are statistically analyzed. The process and observation uncertainty measurement matrix are iteratively updated with the learning rate. The trigger time for taking pictures is calculated in reverse. The grasping command is encapsulated and optimized before being output.

[0015] In the reference coordinate acquisition module, images of paper cups are acquired and planar physical coordinates are extracted through camera coordinate system definition and time-interval triggering. The acquisition time is recorded synchronously. Based on the preset running speed of the conveyor belt and fixed system delay, a linear model is used to calculate and predict the grab position. Individual paper cups are identified by a combination of acquisition time and lateral coordinates. The coordinate data is encapsulated and output. Specific content includes: The establishment of a position prediction benchmark and the acquisition of coordinate information based on a fixed velocity model establish the inheritance relationship with existing visual detection technologies, clarify the workflow of traditional position prediction methods and their implicit assumptions, and provide a clear reference benchmark for the improvement of subsequent steps. This module specifically includes camera field of view coordinate system definition, paper cup image acquisition triggering, planar coordinate information extraction, acquisition time recording, fixed velocity position prediction calculation, coordinate identification method application, and data communication format encapsulation. Define the camera field of view coordinate system, fix an industrial camera above the conveyor belt, make the camera optical axis perpendicular to the surface of the conveyor belt, and cover the effective working area of ​​the conveyor belt with the camera field of view. Establish a camera user coordinate system. The origin of the coordinate system is set at the upper left corner of the camera's field of view. The first axis of the coordinate system extends along the direction of the conveyor belt's movement, and the second axis of the coordinate system is perpendicular to the direction of the conveyor belt's movement and parallel to the width direction of the conveyor belt. The camera's field of view and the robot's operating area are arranged sequentially in the direction of the conveyor belt's movement, with a preset distance interval between them. This distance interval is determined by measuring the calibration plate during the system calibration phase and stored in the system memory. The system triggers the image acquisition of paper cups by initiating the image acquisition process when the paper cup moves into the camera's field of view along the conveyor belt. The triggering mechanism adopts an equal time interval triggering method, that is, the camera is triggered to take pictures at preset collection cycles. At each triggering moment, the current system timestamp is recorded synchronously, and this timestamp serves as a reference for the paper cup collection time. The accuracy of the acquisition time is determined by the resolution of the system clock. In this embodiment, a system clock with millisecond-level accuracy is used. After extracting planar coordinate information, the image data acquired by the camera is transmitted to the image processing unit. The system first performs preprocessing operations on the image, including grayscale conversion, noise filtering, contrast enhancement and other processing steps. After preprocessing, an edge detection algorithm is used to extract the contour information of the paper cup in the image, and the position of the paper cup in the image is determined by contour analysis; The image pixel coordinates are converted into physical coordinates in the camera user coordinate system to obtain the horizontal and vertical coordinate values ​​of the paper cup. The horizontal coordinate value represents the position of the paper cup along the direction of movement of the conveyor belt, and the vertical coordinate value represents the position of the paper cup in the width direction of the conveyor belt; The transformation parameters used in the coordinate transformation process are determined by measurement using a calibration plate during the system calibration phase. The acquisition time is recorded. While acquiring the plane coordinate information of the paper cup, the acquisition time corresponding to the coordinate information is recorded. The acquisition time is based on the system clock and is recorded using a unified timestamp format. The data collection time and coordinate information are linked together to form the initial state information of the paper cup; A combination of acquisition time and horizontal coordinate is used as the unique identifier for the paper cup. This identification method can distinguish multiple individual paper cups in the same field of view, avoiding duplicate or missed captures in subsequent processing. A fixed-velocity position prediction calculation is performed. Based on the collected paper cup coordinate information and the collection time, a fixed-velocity model is used to predict the time and position when the paper cup will reach the robot's grasping position. The prediction calculation is based on the following assumptions: the conveyor belt runs at a constant speed and equals the preset speed value; there is no relative slippage between the paper cup and the conveyor belt; and the total delay time from image acquisition to robot grasping is a fixed value. The prediction calculation first calculates the theoretical time required for the paper cup to move to the camera's field of view based on the distance between the current longitudinal coordinate of the paper cup and the camera's field of view boundary, combined with the preset running speed of the conveyor belt. Add the theoretical time to the preset system delay time to obtain the theoretical time when the paper cup reaches the robot's grasping position; Based on the preset operating speed of the conveyor belt and the theoretical time interval, calculate the predicted lateral position of the paper cup at the moment of grasping; The coordinate identification method is applied, and each paper cup is uniquely identified by a combination of the collection time and the horizontal coordinate. When there are multiple paper cups at the same collection time, they are sorted and distinguished according to the horizontal coordinate values ​​of each paper cup. This identification method ensures that the status information of the same paper cup at different times can be accurately associated during subsequent tracking, avoiding tracking confusion caused by changes in the number of paper cups or overlapping positions; The data communication format is encapsulated, and the recognition results are encapsulated according to the preset data communication format, which includes field information such as total number of targets, paper cup type, vertical coordinate, horizontal coordinate and deflection angle; The encapsulated data is transmitted to the robot controller through the communication interface, and the robot controller plans the grasping path based on the received coordinate information; The initial predicted position output has a prediction error, which mainly comes from three aspects: first, there is a fluctuation deviation between the actual running speed of the conveyor belt and the preset running speed; second, the lightweight paper cups experience slight slippage relative to the conveyor belt due to inertia during the start-up, stop, or speed change of the conveyor belt; and third, there is a dynamic change between the actual processing delay of the system and the preset delay time. The existence of these errors is clearly recorded, providing a clear direction for improvement and a verification benchmark for subsequent steps.

[0016] In the dual-source speed observation module, encoder pulse counting differential calculation of roller angular displacement, combined with roller diameter conversion to conveyor belt instantaneous speed, visual feature point corner detection and optical flow tracking to convert pixel displacement into physical displacement, calculate the actual speed of the paper cup, compare the dual-source speed difference to obtain the sliding speed, numerical integration to accumulate the sliding displacement, timestamp alignment and interpolation of encoder data and visual speed data, and output a synchronous observation sequence. Specific content includes: The encoder and visual optical flow dual-source velocity observation and preliminary estimation of sliding displacement are based on the coordinate acquisition established by the reference coordinate acquisition module. An encoder pulse counting acquisition module and a visual optical flow velocity calculation module are added. The instantaneous running speed of the conveyor belt and the actual moving speed of the paper cup are obtained through dual-source observation. Then, the sliding speed and sliding displacement of the paper cup relative to the conveyor belt are estimated in advance, providing observation input data for the state estimation of the state delay compensation module. Specifically, this includes encoder data acquisition and processing, instantaneous speed calculation of conveyor belt, visual feature point detection and tracking, actual speed calculation of paper cup, sliding speed estimation, sliding displacement integral calculation, and multi-source data timestamp alignment. Encoder data acquisition and processing are performed by installing a rotary encoder on the coaxial position of the conveyor belt drive roller. This encoder can output a pulse signal corresponding to the rotation angle of the roller in real time. The encoder pulse counting acquisition module reads the current pulse count value of the encoder at intervals of the first sampling period. This sampling period is much smaller than the camera image acquisition period, ensuring that it can capture rapid changes in the conveyor belt speed. Each pulse count value read is bound to the corresponding system timestamp and stored to form an encoder pulse count time sequence; The pulse counting time series is differentially processed to calculate the change in pulse count between adjacent sampling times. This change reflects the rotation angle change of the roller within the sampling period. The instantaneous speed of the conveyor belt is calculated based on the change in encoder pulse count. The calculation process first converts the pulse count change into the angular displacement of the drum, and this conversion relationship is determined by the encoder's resolution parameter; The angular displacement is converted into the linear displacement of the conveyor belt based on the diameter parameter of the roller, and this conversion relationship is determined by the geometry of the roller. The line position is removed at sampling period intervals to obtain the average running speed of the conveyor belt within that sampling period, which is used as an estimate of the instantaneous speed of the conveyor belt. The speed values ​​calculated from multiple consecutive sampling periods are smoothed to eliminate the influence of measurement noise, resulting in a smoothed instantaneous speed time series of the conveyor belt. Visual feature point detection and tracking are performed, and the sparse feature point tracking algorithm is used to calculate the moving speed of the paper cup between consecutive image frames captured by the camera; In the first frame of the image, feature points suitable for tracking are detected. The feature point detection adopts the corner detection method, and the location with significant gray-scale changes in the image is selected as the candidate feature points. Then, the final set of feature points for tracking is selected according to the preset feature point quantity requirements. Each feature point in the feature point set records its pixel coordinate position in the image; In subsequent image frames, an optical flow tracing algorithm is used to track the corresponding position of each feature point in the new frame. The optical flow tracing algorithm is based on the assumptions of constant brightness and spatial continuity, and calculates the displacement vector of the feature points through iterative optimization. The tracking results are validated, and feature points that fail to track or have excessive tracking errors are removed. The feature points that are tracked effectively are retained for velocity calculation. Calculate the actual speed of the paper cup by calculating the actual speed of the paper cup based on the displacement vector of the effectively tracked feature points. The calculation process first converts the pixel displacement of each effective feature point between two consecutive frames into physical displacement. This conversion relationship is determined by the camera's calibration parameters, including parameters such as camera mounting height, lens focal length, and pixel size. Remove the physical bits by the time interval between two frames to obtain the velocity vector corresponding to each feature point; Statistical processing is performed on the velocity vectors of all valid feature points, and the average value of the velocity vectors is calculated as an estimate of the actual moving speed of the paper cup. Simultaneously, the variance of the velocity vector is calculated. This variance reflects the reliability of the velocity estimate; the smaller the variance, the more reliable the velocity estimate. Time series analysis is performed on velocity values ​​calculated across multiple consecutive frames to detect velocity trends and abnormal fluctuations. Sliding speed estimation is performed by calculating the sliding speed of the paper cup relative to the conveyor belt based on the instantaneous speed of the conveyor belt and the actual speed of the paper cup. The sliding speed is equal to the actual speed of the paper cup minus the instantaneous speed of the conveyor belt. When the sliding speed is positive, it means that the paper cup moves faster than the conveyor belt, and the paper cup slides forward relative to the conveyor belt. When the sliding speed is negative, it means that the paper cup moves slower than the conveyor belt, and the paper cup slides backward relative to the conveyor belt. The absolute value of the sliding speed reflects the intensity of the sliding. When the sliding speed is close to zero, it means that there is basically no relative sliding between the paper cup and the conveyor belt. Statistical analysis is performed on the time series of sliding speeds to calculate the mean, variance, and extreme values ​​of the sliding speeds. These statistics are used to evaluate the overall characteristics of the sliding phenomenon. Perform sliding displacement integral calculation, and calculate the cumulative sliding displacement of the paper cup relative to the conveyor belt based on the sliding speed time series; The integral calculation adopts the numerical integration method, which discretizes the sliding velocity time series into multiple sampling points. The sliding velocity in each sampling period is multiplied by the sampling period time interval to obtain the displacement increment in that period. Then, all displacement increments are accumulated to obtain the total sliding displacement from the start time to the current time. The integral result is drift corrected. When the paper cup and the conveyor belt are detected to re-establish stable contact, the sliding displacement is reset to zero or close to zero to avoid the estimated value of sliding displacement from continuously increasing due to integral drift. The sign of the sliding displacement indicates the direction of sliding, and the absolute value indicates the cumulative distance of sliding. To perform timestamp alignment of multi-source data, since the encoder data acquisition cycle, camera image acquisition cycle, and optical flow calculation cycle are different, it is necessary to unify the data from different sources onto a common time base. Interpolation methods are used to align data with different sampling rates onto a unified time grid. Linear interpolation is used for encoder data, while preserving previous values ​​or linear interpolation methods are used for visual velocity data. The aligned data have the same timestamp, which facilitates data fusion processing in subsequent steps. Record the timestamp precision and source identifier for each data point to ensure that the original source of the data can be traced in subsequent processing; The output instantaneous speed sequence of the conveyor belt, the actual speed sequence of the paper cup, the sliding speed sequence, and the estimated sliding displacement will serve as the observation input data for the state estimation module of the state delay compensation module. Compared with the fixed speed assumption adopted by the reference coordinate acquisition module, this module obtains speed information through actual measurement, which can reflect the real fluctuation of the conveyor belt speed and the actual degree of paper cup sliding, providing a reliable data foundation for accurate position prediction.

[0017] In the state delay compensation module, a state vector containing position, velocity, and sliding displacement is defined, and a state transition matrix and observation matrix are constructed. Adaptive filtering and recursive estimation are used. In the prediction stage, the state and uncertainty are recursively estimated based on the kinematic model. The process uncertainty measurement matrix is ​​dynamically adjusted based on the conveyor belt acceleration amplitude, and the observation uncertainty measurement matrix is ​​dynamically adjusted based on image sharpness and contrast scores. Simultaneously, the system processing delay time is dynamically estimated based on the number of paper cups and edge complexity in the image. Specific content includes: Based on the dual-source velocity observation data output by the dual-source velocity observation module, a state estimation model is established. The true position, true velocity, and sliding displacement state of the paper cup are estimated in real time through an adaptive filtering algorithm. At the same time, the system processing delay time is dynamically estimated, replacing the fixed velocity model and fixed delay compensation method used in the reference coordinate acquisition module. This module specifically includes state vector definition, state transition model establishment, observation model establishment, dynamic adjustment of process uncertainty measurement matrix, dynamic adjustment of observation uncertainty measurement matrix, state recursive estimation calculation, and dynamic delay time estimation. Define a state vector containing three components: the first component represents the position coordinates of the paper cup in the direction of the conveyor belt, the second component represents the velocity of the paper cup in that direction, and the third component represents the sliding displacement of the paper cup relative to the conveyor belt. The unit of position coordinates is length unit, the unit of velocity is length unit divided by time unit, and the unit of sliding displacement is length unit; The state estimation vector is updated once in each estimation period, and the updated state estimate is used as the initial state for the next period. Each tracked paper cup maintains an independent state estimation vector to ensure that the states of each target do not interfere with each other during multi-target tracking; Establish a state transition model to describe the evolution of the state estimation vector over time. This model is based on kinematic principles and assumes that between two adjacent estimation periods, the change in the position of the paper cup is equal to the velocity multiplied by the time interval, the change in velocity is equal to the acceleration multiplied by the time interval, and the change in sliding displacement follows a decay law. The state transition model is represented in matrix form, where the state transition matrix describes the coupling relationship between state components, and the control input matrix describes the influence of external acceleration input on the state. The acceleration input is calculated from the difference of the instantaneous speed of the conveyor belt, reflecting the acceleration or deceleration state of the conveyor belt; The sliding attenuation coefficient represents the proportion of natural attenuation of sliding displacement over time. This coefficient is determined based on the frictional characteristics between the paper cup and the conveyor belt. Establish an observation model to describe the relationship between the observation data and the state estimation vector; The observation data includes two components: the first component is the instantaneous speed observation value of the conveyor belt calculated by the encoder, and the second component is the actual speed observation value of the paper cup calculated by the visual optical flow. The observation model is represented in matrix form, where the observation matrix describes how the state components are mapped to the observation space. The conveyor belt speed observation mainly reflects the velocity component in the state vector, while the actual speed observation of the paper cup reflects the sum of the velocity component and the sliding velocity component. The observation model assumes that the observation data equals the linear transformation of the state vector plus the observation noise, which reflects the measurement error and the deviation caused by model simplification. The process uncertainty measurement matrix is ​​dynamically adjusted. The process uncertainty measurement matrix is ​​defined, which describes the degree of uncertainty of the state transition model. The process uncertainty measurement matrix is ​​a diagonal matrix, with the diagonal elements corresponding to the degree of uncertainty of the three state components: position, velocity, and sliding displacement. The process uncertainty metric matrix is ​​dynamically adjusted based on the magnitude of the conveyor belt acceleration. When the absolute value of the acceleration is large, it indicates that the conveyor belt is in a variable speed state, and the uncertainty of the paper cup sliding increases. In this case, the value of the process uncertainty metric matrix is ​​increased. When the absolute value of the acceleration is small, it indicates that the conveyor belt is in a uniform speed state, and the sliding is relatively stable. In this case, the value of the process uncertainty metric matrix is ​​decreased. The dynamic adjustment of the process uncertainty metric matrix enables the state estimator to adapt to different operating conditions, placing more trust in observed data under variable speed conditions and more trust in model predictions under constant speed conditions. Dynamically adjust the observation uncertainty metric matrix and define the observation uncertainty metric matrix, which describes the reliability of the observation data; The observation uncertainty measurement matrix is ​​a diagonal matrix, with the diagonal elements corresponding to the degree of uncertainty of the encoder velocity observation and the visual velocity observation, respectively. The uncertainty of visual speed observations is dynamically adjusted based on the image quality score, which is obtained by comprehensively calculating indicators such as image sharpness, contrast, and illumination uniformity. A higher image quality score indicates that the visual observation data is reliable, thus reducing the corresponding degree of uncertainty. When the image quality score is low, it indicates that the visual observation data may be affected by blurring, reflection, or occlusion, increasing the corresponding degree of uncertainty; Dynamically adjusting the observation uncertainty metric matrix enables the state estimator to adaptively allocate weights based on the quality of the observation data, making full use of visual information when visual observations are reliable, and relying more on encoder information when visual observations are unreliable. A recursive state estimation calculation is performed, and a recursive estimation algorithm is used for state estimation, which includes two stages: prediction and update. During the forecasting phase, the state forecast for the current period is calculated based on the state estimate and state transition model of the previous period, and the forecast uncertainty is calculated based on the process uncertainty measurement matrix. In the update phase, the observed predicted value is calculated based on the observation model. The difference between the actual observed value and the observed predicted value is taken as the information. The state update weight coefficient is calculated based on the information and the prediction uncertainty. Then, the state predicted value is corrected by multiplying the state update weight coefficient by the information to obtain the final state estimate. Simultaneously, the uncertainty measure of the state estimate is updated. The state estimate includes the position estimate, velocity estimate, and sliding displacement estimate. These estimates will be used to predict the position prediction calculation of the closed-loop correction module. Perform dynamic delay time estimation and establish a dynamic delay time estimation model, which describes the total delay time from the image acquisition time to the robot's grasping execution time. Total latency includes multiple components such as image processing time, data transmission time, robot controller computation time, and robot response time; The image processing time is dynamically estimated based on the number of paper cups in the image and the image complexity. The more paper cups there are and the more complex the image edges, the longer the processing time will be. Establish an empirical relationship between latency and the number of paper cups and image complexity by statistically analyzing historical data, and update the parameters of this empirical relationship regularly; The dynamic delay time estimate will be used to predict the capture time calculation of the closed-loop correction module, replacing the fixed delay time used in the reference coordinate acquisition module. The position estimate, velocity estimate, sliding displacement estimate, and dynamic delay time estimate output by this module will serve as input data for the prediction closed-loop correction module to perform position prediction and grasping timing optimization. Compared with the fixed model of the reference coordinate acquisition module, this module can track the real motion state of the paper cup in real time through adaptive state estimation and adapt to the processing time changes under different working conditions through dynamic delay estimation, providing reliable state information for accurate grasping position prediction.

[0018] In the prediction closed-loop correction module, based on the state estimation results, the difference between the instantaneous speed and sliding speed of the conveyor belt is integrated over the estimation period to obtain the cumulative displacement. This cumulative displacement is then superimposed with the acquisition time position to obtain the predicted grasping position. The grasping time is calculated based on the grasping distance and dynamic delay estimation. The magnitude of the sliding displacement is used to determine whether to delay and wait to restore stable contact. Historical prediction errors are recorded, and the covariance and variance are statistically analyzed. The process and observation uncertainty measurement matrix is ​​iteratively updated using the learning rate. The trigger time for taking a picture is calculated in reverse, and the optimized grasping command is encapsulated and output. Specific content includes: The module for iterative optimization of position prediction and dynamic adjustment of grasping timing is based on the state estimation results output by the state delay compensation module. It calculates the accurate predicted grasping position, dynamically adjusts the grasping timing strategy according to the magnitude of the sliding displacement, iteratively optimizes the state estimation parameters using historical prediction errors, forms a closed-loop optimization system, and finally outputs the optimized grasping command to the robot controller. This module specifically includes cumulative displacement prediction calculation, capture time calculation, sliding stability waiting strategy, historical prediction error statistics, process uncertainty metric matrix iterative update, observation uncertainty metric matrix iterative update, trigger image capture time calculation, and capture command output. The cumulative displacement prediction calculation is performed. Based on the position estimate, velocity estimate and sliding displacement estimate output by the state delay compensation module, the cumulative displacement of the paper cup from the image acquisition time to the predicted grasping time is calculated. The cumulative displacement calculation uses an integral method, which accumulates the difference between the velocity estimate and the sliding velocity estimate over time to obtain the actual displacement of the paper cup relative to the ground. Specifically, starting from the acquisition time, the system calculates the displacement increment within each step, with the estimation period as the step size. The displacement increment is equal to the velocity estimate minus the sliding velocity estimate multiplied by the step time interval. Then, the displacement increments of all steps are summed to obtain the total cumulative displacement. The predicted grab position is obtained by adding the estimated position at the time of acquisition to the total cumulative displacement. This cumulative displacement prediction method takes into account velocity fluctuations and sliding compensation, and is more accurate than the linear prediction method used in the reference coordinate acquisition module. The grasping time is calculated. Based on the predicted grasping position, the theoretical time when the paper cup arrives at the robot's grasping area is calculated. The calculation process first determines the distance between the camera's field of view boundary and the robot's grasping point. This distance is measured and determined during the system calibration phase. Calculate the theoretical time required for the paper cup to move that distance based on the current speed estimate, and add the theoretical time to the dynamic delay time estimate and the robot response time to obtain the predicted grasping time; The predicted grasping moment indicates the target moment when the robot should perform the grasping action. It is necessary to ensure that the robot's end effector can accurately reach the predicted grasping position at that moment. A sliding stability waiting strategy is implemented, and the waiting strategy is triggered based on the magnitude of the estimated sliding displacement. When the absolute value of the estimated sliding displacement is greater than the preset sliding threshold, it indicates that there is obvious relative sliding between the paper cup and the conveyor belt. Immediately grabbing the cup at this time may result in a gripping position deviation or the paper cup tipping over. Trigger the sliding stability waiting strategy, delay the grabbing moment by a waiting time interval. The waiting time interval is proportional to the magnitude of the sliding displacement estimate. The larger the sliding displacement, the longer the waiting time, giving the paper cup enough time to re-establish stable contact with the conveyor belt. During the waiting period, the change in the sliding displacement estimate is continuously monitored. When the sliding displacement estimate decays to below the sliding threshold, the paper cup is considered to have returned to a stable state and the grasping action can be performed. The sliding stability waiting strategy effectively avoids failures caused by grasping during the sliding of the paper cup; Perform historical prediction error statistics, record the actual results of each grasping operation, including the actual grasping position of the paper cup and the predicted grasping position, and calculate the deviation between the two as the prediction error; Maintain a historical prediction error window, which stores the prediction error data of the most recent crawling operations; When the window is full, newly entered error data replaces the oldest error data, keeping the window size constant. Statistical analysis is performed on the data in the historical prediction error window to calculate the mean, variance, and covariance matrix of the error. These statistics reflect the long-term performance of the state estimation model and are used to guide the iterative optimization of the model parameters. The process uncertainty measurement matrix is ​​iteratively updated based on the statistical results of historical prediction errors. The update rule is: the new process uncertainty metric matrix is ​​equal to the old process uncertainty metric matrix plus the learning rate multiplied by the covariance matrix of the historical prediction error; When the historical prediction error is large, the value of the covariance matrix increases, which leads to an increase in the process uncertainty measurement matrix. In subsequent estimations, the state estimator will reduce its confidence in the model prediction and increase its dependence on the observation data. When the historical prediction error is small, the process uncertainty measurement matrix changes little and maintains the original estimation characteristics; The uncertainty metric matrix of the iterative update process enables the state estimator to automatically adjust based on long-term performance and adapt to slow changes in operating conditions. The observation uncertainty metric matrix is ​​iteratively updated based on the statistical results of historical prediction errors. The update rule is: the new observation uncertainty metric matrix is ​​equal to the old observation uncertainty metric matrix plus a function of the learning rate multiplied by the historical prediction error variance; When the prediction error variance of a certain observation channel is large, it indicates that the data quality of that observation channel may be degraded. The system increases the corresponding observation uncertainty metric and reduces the weight of that observation channel in state updates. Iteratively updating the observation uncertainty metric matrix enables the state estimator to adaptively adjust the weight allocation based on the long-term performance of each observation channel, thereby improving the robustness of the state estimation. The system calculates the trigger time for taking a picture by back-calculating the time when the camera should be triggered to take a picture, based on the predicted grasping time, the estimated dynamic delay time, and the robot's response time. The trigger time for taking a picture is equal to the predicted capture time minus the estimated dynamic delay time minus the robot response time; Image acquisition is initiated at the moment the photo is triggered, ensuring that image processing and robot preparation are completed by the time the paper cup reaches the grasping position; Dynamically adjusting the trigger time for taking a picture can compensate for changes in the delay time, ensuring the timing accuracy of the capture action; Output the grasping command, encapsulate the optimized grasping position coordinates, grasping time and paper cup deflection angle according to the preset data communication format, and generate the grasping command; The grabbing command includes fields such as total number of targets, paper cup type, vertical coordinate, horizontal coordinate, and deflection angle; The encapsulated grasping command is transmitted to the robot controller through the communication interface. The robot controller plans the motion trajectory of the end effector according to the command and arrives at the designated position at the specified time to perform the grasping action. The data format in the grasping instructions is consistent with the format used by the reference coordinate acquisition module to ensure compatibility with existing robot control systems; This module forms a complete closed-loop optimization system through cumulative displacement prediction, sliding stabilization waiting, historical error statistics, and parameter iterative updates. Compared with the open-loop prediction of the reference coordinate acquisition module, this module can continuously correct the estimated parameters based on the actual grasping results, so that the system performance gradually improves with running time. After optimization by this module, the position prediction error is significantly reduced, the grasping failure rate is greatly reduced, and the system can adapt to complex working conditions such as conveyor belt speed fluctuations and paper cup micro-slippage, meeting the actual needs of automatic tracking, sorting and conveying of paper cups.

[0019] The above formulas are all dimensionless calculations. The formulas are derived from software simulations based on a large amount of collected data to obtain the most recent real-world results. The preset parameters in the formulas are set by those skilled in the art according to the actual situation.

[0020] The above embodiments can be implemented, in whole or in part, by software, hardware, firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, in the form of a computer program product.

[0021] Those skilled in the art will recognize that the modules and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and inventive constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0022] In addition, the functional modules in the various embodiments of this application can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module.

[0023] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0024] In conclusion, the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A vision-based automatic tracking, sorting, and conveying system for paper cups, characterized in that, It includes a reference coordinate acquisition module, a dual-source velocity observation module, a state delay compensation module, and a prediction closed-loop correction module, with signal connections between each module; The reference coordinate acquisition module: by defining the camera coordinate system and triggering at equal time intervals, it acquires images of paper cups and extracts planar physical coordinates, synchronously records the acquisition time, and uses a linear model to calculate and predict the grab position based on the preset running speed of the conveyor belt and the fixed system delay. It identifies individual paper cups by combining the acquisition time and the horizontal coordinates, encapsulates the coordinate data, and outputs it. Dual-source speed observation module: The encoder pulse counting differential calculation of the roller angular displacement, combined with the roller diameter to convert the instantaneous speed of the conveyor belt, visual feature point corner detection and optical flow tracking, convert pixel displacement into physical displacement, calculate the actual speed of the paper cup, compare the dual-source speed difference to obtain the sliding speed, numerical integration to accumulate the sliding displacement, timestamp alignment and interpolation of encoder data and visual speed data, and output synchronous observation sequence. State delay compensation module: Defines a state vector including position, velocity, and sliding displacement, constructs a state transition matrix and an observation matrix, uses adaptive filtering for recursive estimation, and in the prediction stage, recursively estimates the state and uncertainty based on the kinematic model. It dynamically adjusts the process uncertainty measurement matrix based on the conveyor belt acceleration amplitude, dynamically adjusts the observation uncertainty measurement matrix based on image sharpness and contrast scores, and dynamically estimates the system processing delay time based on the number of paper cups and edge complexity in the image. Predictive closed-loop correction module: Based on the state estimation results, the difference between the instantaneous speed and sliding speed of the conveyor belt is integrated over the estimation period to obtain the cumulative displacement. The predicted grasping position is obtained by superimposing the acquisition time position. The grasping time is estimated based on the grasping distance and dynamic delay. The module determines whether to delay and wait to restore stable contact based on the sliding displacement amplitude. Historical prediction errors are recorded, and the covariance and variance are statistically analyzed. The process and observation uncertainty measurement matrix are iteratively updated with the learning rate. The trigger time for taking pictures is calculated in reverse. The grasping command is encapsulated and optimized before being output.

2. The paper cup automatic tracking, sorting and conveying system based on vision detection according to claim 1, characterized in that, The reference coordinate acquisition module establishes a user coordinate system with the upper left corner of the camera's field of view as the origin, the first axis along the direction of the conveyor belt's movement, and the second axis perpendicular to the direction of the conveyor belt's movement, and stores the calibration distance between the camera's field of view and the robot's grasping area.

3. The paper cup automatic tracking, sorting and conveying system based on vision detection according to claim 2, characterized in that, The reference coordinate acquisition module triggers the camera to take pictures at a preset cycle and records the acquisition time synchronously. The acquired images are preprocessed by grayscale conversion, filtering, and contrast enhancement. The paper cup outline is extracted by edge detection and converted into physical coordinates.

4. The paper cup automatic tracking, sorting and conveying system based on vision detection according to claim 2, characterized in that, The reference coordinate acquisition module uses a linear model to calculate the theoretical time for the paper cup to move to the camera's field of view boundary, the theoretical time to reach the grasping position, and the predicted lateral coordinates, based on the preset running speed of the conveyor belt and the fixed system delay.

5. The paper cup automatic tracking, sorting and conveying system based on vision detection according to claim 2, characterized in that, The reference coordinate acquisition module identifies individual paper cups by combining the acquisition time with the horizontal coordinate, and encapsulates the coordinate data output in a preset format that includes the total number of targets, paper cup type, vertical coordinate, horizontal coordinate, and deflection angle.

6. The paper cup automatic tracking, sorting and conveying system based on vision detection according to claim 1, characterized in that, The dual-source speed observation module reads the encoder pulse count in the first sampling period, calculates the angular displacement differentially, and converts it into the instantaneous speed of the conveyor belt; between consecutive image frames, it obtains the pixel displacement of feature points through corner detection and optical flow tracking, converts it into physical displacement, and then calculates the mean and variance of the actual speed of the paper cup.

7. The paper cup automatic tracking, sorting and conveying system based on vision detection according to claim 6, characterized in that, The dual-source velocity observation module subtracts the instantaneous speed of the conveyor belt from the actual speed of the paper cup to obtain the sliding speed. It accumulates the sliding displacement through numerical integration and outputs a synchronous observation sequence after interpolating and aligning the encoder and visual velocity data.

8. The paper cup automatic tracking, sorting and conveying system based on vision detection according to claim 1, characterized in that, The state delay compensation module defines a state vector for each paper cup, including position, velocity, and sliding displacement, constructs a state transition matrix and an observation matrix, and uses adaptive filtering to recursively estimate the state. It dynamically adjusts the process uncertainty measurement matrix based on the conveyor belt acceleration amplitude, dynamically adjusts the observation uncertainty measurement matrix based on image clarity and contrast scores, and dynamically estimates the system processing delay time based on the number of paper cups and edge complexity.

9. The paper cup automatic tracking, sorting and conveying system based on vision detection according to claim 1, characterized in that, The prediction closed-loop correction module integrates the difference between the instantaneous speed and sliding speed of the conveyor belt estimated by the state period to obtain the cumulative displacement, and superimposes the position estimate at the acquisition time to predict the grasping position; it calculates the grasping time based on the grasping distance and dynamic delay estimate, and delays and waits to restore stable contact when the sliding displacement estimate exceeds the threshold.

10. A visual detection-based automatic tracking, sorting, and conveying system for paper cups according to claim 9, characterized in that, The prediction closed-loop correction module records historical prediction errors and calculates covariance and variance. It iteratively updates the process and observation uncertainty measurement matrix with the learning rate, reversely calculates the trigger time for taking pictures, and encapsulates the predicted capture position, capture time, and deflection angle as capture command output.

Images