A foreign object recognition and conveyor coordinated foreign object pickup method and system

Abstract

This invention discloses a method and system for foreign object pickup in collaboration with a conveyor belt, relating to the field of intelligent control technology. The method includes: acquiring an image above the conveyor belt and performing target detection to generate target data; converting pixel positions into belt surface coordinates and fusing encoder belt speed and visual belt speed to obtain belt speed estimation; calculating the arrival time distribution of the target data and a preset pickup baseline based on the target data, belt speed estimation, and detection time, and performing feasibility determination under opportunity constraints; selecting execution objects under minimum action interval and resource mutual exclusion constraints, generating merged actions, and outputting a scheduling result set; performing total delay compensation and trigger time calculation on the actions in the scheduling result set, and obtaining execution feedback to adaptively update decision thresholds and scheduling parameters online. The method of this invention optimizes the execution order and, combined with delay compensation and online parameter updates, achieves self-correction and continuous optimization of the pickup action.

Description

Technical Field

[0001] This invention relates to the field of intelligent control technology, and in particular to a method and system for picking up foreign objects in coordination with a conveyor belt. Background Technology

[0002] With the rapid development of automated sorting and intelligent manufacturing systems, foreign object detection and automatic rejection technologies based on visual recognition have gradually become an important part of the intelligent upgrading of production lines. Currently, industries such as industrial production, food processing, recycled resource sorting, and packaging transportation commonly use high-speed conveyor belts to transport materials, achieving real-time detection of foreign objects through industrial cameras or multimodal sensors installed above the belt. Existing foreign object recognition systems are mostly based on deep learning visual algorithms (such as convolutional neural network object detection frameworks) to identify and locate abnormal targets in images, and then remove them using robotic arms, pneumatic nozzles, or electromagnetic adsorption devices. However, these systems often rely on image recognition as their core component, lacking deep coordination with the conveyor belt's motion. Their picking decisions often depend solely on static detection results, failing to fully consider the time delay, speed fluctuations, and execution cycle limitations during the material's movement on the belt surface. As production cycles increase, simple recognition-action serial logic cannot handle the timing matching problems in high-speed dynamic scenarios, easily leading to missed rejections, incorrect rejections, or out-of-synchronization picking actions, thus limiting the system's sorting accuracy and operational stability.

[0003] Most existing methods assume a constant conveyor belt speed or fixed delay, neglecting dynamic factors such as encoder slippage, drive fluctuations, image acquisition delays, and actuator activation times, resulting in a non-negligible time deviation between recognition and execution. Secondly, some schemes only apply a single threshold filter to recognition confidence, lacking a dynamic order-taking mechanism based on time margin and uncertainty, leading to frequent triggering of low-reliability targets and wasted motion resources. Thirdly, existing scheduling mechanisms generally employ static sorting or fixed-cycle triggering, failing to introduce opportunity-constrained optimization models based on arrival time distribution and success probability, and lacking dynamic coordination strategies for issues such as temporal conflicts, resource mutual exclusion, and jet merging among multiple targets. Finally, traditional picking systems often use offline parameter setting methods, lacking online adaptive optimization based on real-time execution feedback, making it difficult for the system to maintain long-term stability under environmental disturbances or cycle time changes. Summary of the Invention

[0004] In view of the aforementioned existing problems, the present invention is proposed.

[0005] Therefore, this invention provides a foreign object picking method that combines foreign object identification with conveyor belt coordination to solve the problems of weak temporal coupling, unstable action scheduling, and lack of closed-loop adaptation in existing foreign object identification systems.

[0006] To solve the above-mentioned technical problems, the present invention provides the following technical solution:

[0007] In a first aspect, the present invention provides a foreign object picking method in coordination with a conveyor belt for foreign object identification, which includes: acquiring an image above the conveyor belt, performing target detection to generate target data, and filtering low-reliability targets based on a dynamic order acceptance threshold;

[0008] The pixel position is converted into a strip surface coordinate through the calibrated pixel-to-strip surface mapping, and the strip speed is estimated by fusing the encoder strip speed and the visual strip speed.

[0009] Based on target data, belt speed estimation, and detection time, the arrival time distribution of target data and preset picking baseline is calculated, and the feasibility of opportunity constraints is determined.

[0010] Within the rolling time window, the queue targets are sorted according to the target scoring function, the execution target is selected under the minimum action interval and resource mutual exclusion constraints, and the merged action is generated for horizontally adjacent targets that can share the same time coverage, and the scheduling result set is output.

[0011] The system performs total delay compensation on the actions in the scheduling result set, performs timing correction based on the uncertainty of the arrival time distribution, and obtains execution feedback to perform online adaptive updates of decision thresholds and scheduling parameters.

[0012] As a preferred embodiment of the foreign object identification and conveyor belt coordinated foreign object pickup method of the present invention, wherein: the target detection generating target data includes acquiring the current frame image at a set sampling period under a unified time axis, wherein, This indicates the acquisition timestamp of the current frame image; preprocessing the current frame image yields the image frame used as the detection input, while retaining the timestamp. constant;

[0013] The preprocessed image frames are input into the object detection network to obtain a candidate object set, where each candidate object's data... This includes the detection timestamp when the target was detected and the pixel position of the target in the image coordinate system. Detect the confidence level of the network output. Target category labels Dimensional rejectability features obtained from the estimation of the size, shape, and depth of the detection frame. ; Indicates the target index;

[0014] A dynamic confidence threshold function is set based on the approximate remaining time for each candidate target. Applying the CDC rule, if the conditions are met, the current candidate target is considered to have passed the dynamic order acceptance filter; otherwise, it is marked as not accepting orders for this period. For each target that passes the CDC filter... Generate an initial data object.

[0015] As a preferred embodiment of the foreign object identification and conveyor belt coordinated foreign object pickup method of the present invention, wherein: the pre-calibrated pixel-to-surface mapping includes, for each obtained target, defining the target object as... Offline calibration yields pixel-band homography matrix. This describes the projection relationship between the camera plane and the conveyor belt plane.

[0016] During operation, two types of belt speed information are acquired, and a belt speed sequence is obtained from the encoder. Band velocity is estimated from the optical flow of textures or feature points in an image. ; Indicates the current time;

[0017] For each target Using homography matrix pixel coordinates Projected onto the belt plane of the conveyor belt coordinate system, the physical coordinates of the target in the belt plane coordinate system are obtained through homogeneous normalization.

[0018] As a preferred embodiment of the foreign object identification and conveyor belt collaborative foreign object pickup method of the present invention, wherein: the belt speed estimation obtained by fusing encoder belt speed and visual belt speed includes, pixel uncertainty through The propagation extends to the belt space, creating positional uncertainty; within a time window near the detection moment, the belt speed is obtained from the encoder. Calculating band velocity from image optical flow feature drift Define the estimated values ​​corresponding to the two belt speeds as follows: and ;

[0019] Analyze the uncertainty of the two estimates and assign variances to them as consistency measures; when the absolute value of the difference between the two estimates... If the value is less than the preset consistency threshold, the belt speed estimates from the two sources are considered to be consistent, and the belt speed estimate is obtained by weighted fusion based on accuracy. .

[0020] As a preferred embodiment of the foreign object identification and conveyor belt coordinated foreign object pickup method of the present invention, wherein: the sorting of queue targets according to the target scoring function includes, at the current time Open length is A rolling time window; for the target set determined by chance constraints. Perform scheduling and merging;

[0021] Each goal For each pick-up order, the time center value of the target arriving at the pick-up line is included. Uncertainty variance of target arrival time The horizontal coordinate of the target in the conveyor belt coordinate system ; Target category label Confidence level obtained from image recognition ; Removable features characterizing target size or quality ; Calculate the success probability of the target being successfully eliminated within a given action time window. ;

[0022] For each target Calculate the overall score After the scoring is completed, press From largest to smallest Sort the data to obtain the sorted queue. .

[0023] As a preferred embodiment of the foreign object identification and conveyor belt coordinated foreign object pickup method of the present invention, wherein: the total delay compensation and trigger time calculation for the action of the scheduling result set includes obtaining the sorting queue Then, set the minimum action interval based on the cycle time capability of the actuator. Scheduling is performed as a constraint; during scheduling, from Starting with the highest-scoring target, examine each target in descending order of score. If the target is currently being prepared to be added to the scheduling set With any selected target in the scheduling set Time center value difference All are not less than Then the current target will be added to the execution set of the current round. Otherwise, skip the current target and hold it until the next rolling cycle before attempting to schedule it again.

[0024] The initial execution set that satisfies the minimum action interval constraint is obtained through screening. For sets Any pair of targets If the lateral distance of the targets is sufficient to cover them all within the duration of a single jet pulse, then the jet action is combined into a single jet motion; for targets that meet the merging condition... Construct a new merger target Reference arrival time A weighted average is performed, with the weights representing the success probability of the target. The merged pulse width is the maximum value plus a safety margin, based on the pulse width required for a single target, ensuring that the jet time window covers the arrival time distribution of both targets. After merging, the original two targets... From the set Remove from the middle and insert a new merge target. The updated execution set is obtained. Each element represents an actual execution action, and the output is a set of scheduling results. .

[0025] As a preferred embodiment of the foreign object identification and conveyor belt coordinated foreign object pickup method of the present invention, wherein: the online adaptive update of the decision threshold and scheduling parameters includes, setting the scheduling result set As input for delay compensation and trigger control steps, the recognition results are coordinated with the conveyor belt cycle time and execution resources;

[0026] Utilizing delayed components and arrival time distribution parameters The set is calculated. The final trigger time of each action is combined with the action template and action parameters to form an actual control command, which drives the picking device to complete the removal of foreign objects;

[0027] To implement feedback and time residual Based on this, the delay parameters, success probability threshold, dynamic order acceptance threshold, minimum action interval, and speed fusion weight are updated online adaptively.

[0028] Secondly, the present invention provides a foreign object identification and conveyor belt coordinated foreign object pickup system, including a perception generation module, which acquires images and detects targets, performs first-round filtering according to the dynamic order acceptance threshold CDC, and forms candidate tickets;

[0029] The spatiotemporal collaborative estimation module integrates encoder and visual speed measurement, calculates arrival time distribution, and obtains success probability based on action time window to determine whether it can enter the queue.

[0030] The rolling scheduling module sorts feasible targets by score and selects targets under the minimum action interval and resource constraints.

[0031] The delay compensation trigger module calculates the total lead time and obtains the trigger time, issues commands and collects feedback, and updates the threshold, delay component and pulse width fusion weight online to form a closed-loop optimization.

[0032] Thirdly, the present invention provides a computer device including a memory and a processor, wherein the memory stores a computer program, wherein: when the computer program is executed by the processor, it implements any step of the foreign object pickup method in coordination with a conveyor belt as described in the first aspect of the present invention.

[0033] Fourthly, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein: when the computer program is executed by a processor, it implements any step of the foreign object pickup method for foreign object identification and conveyor belt coordination as described in the first aspect of the present invention.

[0034] The beneficial effects of this invention are as follows: By establishing a unified collaborative control algorithm between foreign object identification and conveyor belt movement, real-time matching of identification results with dynamic belt speed and execution cycle time is achieved, significantly improving the timing accuracy and sorting stability of the picking system. Employing vision-encoder dual-source belt speed fusion and arrival time distribution prediction effectively suppresses timing deviations caused by encoder slippage and conveyor belt speed fluctuations. The introduction of a dynamic order acceptance threshold and opportunity constraint judgment mechanism enables adaptive selection of reliable targets based on identification confidence and time margin, reducing invalid triggers and wasted actions. The rolling time window scheduling and merging strategy solves the cycle time conflict problem of multiple targets arriving simultaneously, achieving optimal allocation of execution resources. Combined with delay compensation and online adaptive update mechanisms, delay parameters and scheduling thresholds can be automatically corrected based on execution feedback, maintaining high identification accuracy and action synchronization during long-term operation. Therefore, this invention can achieve highly reliable, low-error automatic foreign object picking in high-speed conveying scenarios, possessing strong engineering practicality and promotional value. Attached Figure Description

[0035] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0036] Figure 1 A flowchart of a foreign object pickup method that integrates foreign object identification with a conveyor belt. Detailed Implementation

[0037] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0038] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0039] Secondly, the term "an embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in different places in this specification does not necessarily refer to the same embodiment, nor is it a single or selective embodiment that is mutually exclusive with other embodiments.

[0040] Reference Figure 1 This is one embodiment of the present invention, which provides a foreign object pickup method that combines foreign object identification with a conveyor belt, including the following steps:

[0041] S1: Acquire the image above the conveyor belt, perform target detection to generate target data, and filter low-reliability targets based on the dynamic order acceptance threshold.

[0042] Furthermore, under a unified timeline, the current frame image is acquired at a set sampling period, denoted as... ;in, This is the timestamp for capturing this frame of the image. Perform necessary preprocessing (denoising, brightness normalization, ROI cropping) to obtain image frames suitable for detection input, while preserving the timestamps. This remains unchanged; all subsequent test results will be associated with this timestamp.

[0043] Object detection and original candidate generation, preprocessing image frames Input the object detection network to obtain a set of candidate objects:

[0044]

[0045] Each candidate target At least including:

[0046]

[0047] The parameters mean: This indicates the detection timestamp corresponding to when the target was detected, usually associated with the frame time. Same or with a fixed offset; This indicates the pixel position of the target in the image coordinate system (which can be taken as the center of the detection box or the centroid). This indicates the confidence level of the detection network output; Indicate the category label of the target (e.g., foreign object type A / B / metal, etc.); This represents the size / rejectability feature estimated from the size, shape, or depth of the detection box. This represents pixel-level uncertainty, which can be obtained from the regression variance of the bounding box returned by the network or empirical statistics, and is used for subsequent error propagation.

[0048] At this point, only the target set of the pixel domain is obtained, and it has not yet been associated with the conveyor belt coordinates or belt speed.

[0049] A rough estimate of remaining time or direction distance (for CDC) is required to achieve dynamic order acceptance threshold CDC.

[0050] Let the rough estimate of the belt speed at the current moment be... (Based on the belt speed estimate or preset nominal speed from the last update), the approximate location of the baseline in the image is picked as follows: Or the corresponding one-dimensional directed pixel coordinates A coarse-grained directional pixel distance can be defined for each target:

[0051]

[0052] Or directly use the previously known pixels. The surface mapping yields a coarse zone-direction distance. In simple cases, the following can be used:

[0053]

[0054] or:

[0055]

[0056] Will The approximate time interval between the target's movement from the detection position to the pickup line is considered as the input variable for the CDC curve.

[0057] Dynamic order acceptance threshold (CDC) and candidate filtering are used to avoid accepting orders with low confidence only when the target is about to be reached, based on a rough remaining time. Set a dynamic confidence threshold function:

[0058]

[0059] in, Let be a monotonically increasing function, such that the shorter the remaining time, the higher the required minimum confidence level. For example, when The target is relatively large (still upstream), requiring only a moderate level of confidence to accept the order; when If the target is very small (close to the pickup line), then it is required that... Orders are only accepted when the time is close to 1 to reduce false triggering at close intervals.

[0060] For each candidate target di, apply the CDC rule:

[0061]

[0062] If the above conditions are met, the candidate target is considered to have passed the dynamic order-acceptance filtering and is retained for subsequent steps; otherwise, it is discarded or marked as not accepting orders in this cycle. The retained set is denoted as:

[0063]

[0064] For each target filtered by CDC, an initial data object is generated, the content of which is:

[0065]

[0066] It should be noted that at this point, the Ticket does not yet contain information such as surface coordinates, speed estimation, and arrival time distribution parameters; these will be gradually added in subsequent steps. However, through S1, target screening and order acceptance control have already been completed in the pixel domain. Only targets worth processing in terms of confidence level and approximate remaining time are passed into the coordination and scheduling process; high-risk candidates with low confidence levels and close to the arrival line are directly filtered out to avoid consuming subsequent scheduling and execution resources. Finally, the output of step one is the initial Ticket set for the current frame.

[0067] S2: Convert the pixel position into belt coordinates through the calibrated pixel-belt-surface mapping, and obtain the belt speed estimate by fusing the encoder belt speed and the visual belt speed.

[0068] Furthermore, using the target data output by S1 as input, an object containing spatial location and motion state is created for each foreign object target for subsequent arrival time calculation.

[0069] Specifically, for each target obtained Let the target be... Its initial attributes include: detection time Pixel coordinates Identification confidence level Category tags Size or shape characteristics and the pixel-level uncertainty given by the detection network. .

[0070] First, based on the pre-calibrated pixel-to-area mapping matrix , pixel coordinates Mapping the coordinates of the belt surface to the conveyor belt coordinate system yields the belt-direction coordinates of the target on the belt surface. and horizontal coordinates In this process, pixel uncertainty The position uncertainty in the direction of the band is synchronously converted through the mapping relationship, and then used to estimate the variance of the target's directional distance error. .

[0071] At the same time, at the target detection time Within a nearby time window, obtain conveyor belt speed estimates from the encoder side. The visual band velocity estimate is calculated using the optical flow or feature point displacement of adjacent frames. Within this time window, representative values ​​are extracted for both the encoder belt speed and the vision belt speed. and And configure the corresponding uncertainty variance for each. and .

[0072] Then, the belt speed difference is calculated:

[0073]

[0074] when When the speed is less than a preset consistency threshold, the two belt speed estimates are considered to be basically consistent at that moment, and the reciprocals of their respective variances can be used as weights. and By performing weighted fusion, the belt velocity estimate at the time of the target is obtained. And calculate the corresponding belt speed variance. .

[0075] when When the threshold is exceeded, based on historical reliability and self-test results, the weight of the mismatched data source is reduced, or the bandwidth of the more reliable side is temporarily used as the threshold. The state is then recorded in the target object for subsequent anomaly detection and parameter correction.

[0076] After the above processing, the initial target object It is expanded to include the geometric position of the belt surface and the fused belt velocity information.

[0077] It should be noted that the pixel positions of foreign objects obtained from image recognition are converted into their actual physical positions in the conveyor belt coordinate system, and the instantaneous speed of the conveyor belt is estimated by combining multi-source information. A mapping relationship between pixel coordinates and belt surface coordinates is established through camera calibration, ensuring that the target's position in the image plane accurately corresponds to the belt direction and lateral position on the conveyor belt surface. Simultaneously, mechanical speed data of the conveyor belt is obtained from the encoder, and the visual speed is estimated through optical flow or feature point displacement in continuous images. These two estimates are then compared for consistency and weighted fusion to form a more stable belt speed estimation result.

[0078] S3: Based on the target data, belt speed estimation and detection time, calculate the arrival time distribution of the target data and the preset picking baseline, and determine the feasibility of the opportunity constraint.

[0079] Furthermore, by utilizing the coordinates of the target on the surface... Belt speed estimation and its variance Combined with the detection time Calculate the center value of the arrival time of the target relative to the preset picking baseline. And summarize the various error sources into arrival time variance. This allows for the use of parameterized time distributions to describe the arrival time of each target. Specifically, the belt-direction coordinates of the pickup baseline are set in the conveyor belt coordinate system as follows: For each target First, calculate the directional distance between the target's current directional position and the picking baseline:

[0080]

[0081] The belt speed is approximately constant over a short period of time. Under these conditions, from the detection time The nominal time required for the target to reach the pickup baseline can be expressed as the belt distance divided by the belt speed estimate, i.e.:

[0082]

[0083] Adding this nominal time increment to the detection time yields the center value of the target's arrival time. ,Right now:

[0084]

[0085] so, This gives the estimated time center position of the target arriving at the pickup line under the current bandwidth conditions. Considering the uncertainties in pixel position, bandwidth mapping, bandwidth estimation, and execution link delay, the target's arrival time is treated as a random variable. Under the first-order approximation, the uncertainty of the arrival time can be decomposed into three main sources:

[0086] First, there is pixel-level uncertainty. Through the mapping matrix The variance of the zone distance transmitted to the zone surface direction is denoted as . Second, the belt speed variance obtained by dual-source fusion of belt speeds. The time error caused by the acquisition, communication and execution links; and the variance of delay jitter caused by the acquisition, communication and execution links. .

[0087] By using the error propagation relationship, these three types of variance are summed according to their contributions to form the total variance of the target arrival time. ,Right now: yes , and The combined quantities are used to characterize the overall uncertainty level of the target's arrival time. Based on this, the target... The arrival time is represented by a distribution with a central value and a variance parameter, denoted as:

[0088]

[0089] in, From the distance Belt speed estimation With the detection time jointly determined, It is obtained by combining distance error, belt speed error and delay jitter.

[0090] It should be noted that the time distribution will then be used to calculate the success probability by overlapping it with the action time window. Through the above processing, the object of S2 is further extended to an object carrying arrival time distribution parameters. Based on the target's spatial position on the conveyor belt, belt speed, and detection time, the arrival time of the target at the pickup position is predicted, and the uncertainty of this time is quantified. Using the belt-direction distance from the target to the pickup line and the belt speed estimate, the expected arrival time of the target is calculated. Since there are errors in image measurement, belt speed variation, and system delay, the system models the arrival time as a distribution with a central value and variance, and forms the total uncertainty through error propagation analysis.

[0091] S4: Within the rolling time window, sort the queue targets according to the target scoring function, select the execution object under the minimum action interval and resource mutual exclusion constraints, generate merge actions for horizontally adjacent targets that can share the same time, and output the scheduling result set.

[0092] The set of objectives determined by chance constraints:

[0093]

[0094] Scheduling and merging are performed. Here, each target... For each work order to be picked up, at least the following should be recorded internally: This indicates the center time value of the target's arrival at the pickup line; This represents the variance of the uncertainty in the time of arrival of the target. This represents the horizontal coordinate of the target in the conveyor belt coordinate system; Category labels indicating the target; This represents the confidence level obtained from image recognition; This indicates a removable characteristic that represents the size or quality of a target. This represents the success probability of the target being successfully removed within a given action time window, calculated in step three.

[0095] Based on this, firstly for each target Calculate a comprehensive score This is used to measure the value of prioritizing the execution of this objective within the current time window. The scoring function can be written as:

[0096]

[0097] in, It is a category The corresponding priority score; It is based on the remaining time to arrive. The time urgency function for calculation, for example, can be taken as... The shorter the remaining time, the higher the urgency. This is used to reflect the cullingability of the target under the current actuator (the better the size and mass fit, the higher the score). It is the probability of success obtained from the opportunity constraint assessment; It is the confidence level of image recognition; These are the weighting coefficients for each item.

[0098] After scoring, press From largest to smallest Sort the data to obtain the sorted queue. In obtaining the sorted queue Then, a minimum action interval is set based on the cycle time capability of the actuator. This means that the time interval between any two triggered actions must not be less than [a certain value]. During scheduling, from Starting with the highest-scoring target, examine each target in descending order of score. If the target is currently being prepared to be added to the scheduling set With any selected target in the scheduling set Time center value difference:

[0099]

[0100] All are not less than That is, satisfying:

[0101]

[0102] Then add the target to the execution set of the current round. Otherwise, to avoid excessively dense actions causing execution conflicts, the target is skipped and reserved for scheduling in the next rolling cycle. This round of filtering yields an initial set of executions that satisfies the minimum action interval constraint. .

[0103] After obtaining the set considering the cycle time constraint Then, for targets applicable to the jet terminal, further judgment on merging actions is performed to improve the utilization efficiency of a single action. For ensemble... Any pair of targets If their lateral distance satisfies:

[0104]

[0105] That is, the two targets are sufficiently close in the horizontal direction, and the difference in their center arrival times is:

[0106]

[0107] Not exceeding the preset time merging threshold If, either the target pair can be covered by both targets within the duration of a single jet pulse, then the pair is considered to meet the merging condition of lateral proximity + temporal overlap and can be merged into a single jet action. For target pairs that meet the merging condition... Construct a new merger target Its reference arrival time They can be arranged according to their respective weights. Calculate a weighted average, using the respective success probabilities or scores as weights:

[0108]

[0109] Merged pulse width Then the required pulse width for a single target can be achieved. Take the maximum value and add a safety margin. ,Right now:

[0110]

[0111] This ensures that the jet timing window covers the arrival time distribution of both targets. After merging, the original two targets... From the set Remove from the middle and insert a new merge target. The updated execution set is obtained. Each element represents an actual action, which can be a single-target action or a combined action.

[0112] The formula for the output scheduling result set is expressed as:

[0113]

[0114] It should be noted that within the rolling time window, a comprehensive score is calculated for all feasible targets, taking into account factors such as category importance, time urgency, target elimination capability, and success probability. Then, the optimal target sequence is selected based on the minimum action interval of the execution device and resource constraints. For targets that are spatially and temporally adjacent, the system further determines whether they can be merged into a single action to improve efficiency and reduce energy consumption. The final scheduling output generates action templates and preliminary execution parameters for each target or merged target, providing a planning basis for subsequent delay compensation and trigger control.

[0115] Each execution unit It should include at least: the identifier of the target or merged target, and the center value of the reference arrival time after scheduling. The corresponding arrival time variance Selected action template type And the preliminary motion parameters corresponding to this template. For example, the start time of the jet pulse and the base pulse width. This scheduling result set... This will directly serve as input for subsequent delay compensation and trigger control steps, thereby achieving coordination between the recognition results, conveyor belt cycle time, and execution resources. Based on arrival time prediction and success probability assessment, multiple targets are dynamically sorted and scheduled to achieve optimal pickup under limited execution resources and cycle time constraints.

[0116] S5: Perform total delay compensation on the actions of the scheduling result set, perform timing correction based on the uncertainty of arrival time distribution, and obtain execution feedback to perform online adaptive updates of decision thresholds and scheduling parameters.

[0117] Based on the scheduling result set output by S4 For each action to be executed, total delay compensation and trigger time are calculated. Then, based on execution feedback, the delay parameters and decision thresholds are updated online adaptively to form a closed loop.

[0118] Input data and delay parameters, for each execution unit obtained in S4 ,Include:

[0119]

[0120] in: This indicates the center value of the arrival time of the target or merged target corresponding to this action; This indicates the variance of the arrival time corresponding to the action; Indicates the type of the selected motion template (e.g., jet, electromagnetic, gripping, etc.); This indicates the preliminary motion parameters (such as pulse width and target pose) obtained in step four based on the template.

[0121] Maintaining four types of delay components: image acquisition and recognition processing delay Communication or network transmission delay The delay in the arrival of control commands at the execution driver. Terminal onset time This is related to the motion template, such as the jet rise time and the gripper closing time.

[0122] For a certain execution unit The corresponding total lead time is denoted as:

[0123]

[0124] Delay compensation and uncertainty correction at trigger time, for the set Each execution unit in First, based on the total lead time Calculate the nominal trigger time:

[0125]

[0126] That is, using the predicted arrival time center value Based on the baseline, move forward and backward. The arrival time is used to determine the theoretically correct time when control commands should be issued. However, the arrival time is uncertain. A variance-dependent correction function can be introduced. This is used to slightly advance the trigger when the variance is large in order to increase the coverage probability. For example, set:

[0127]

[0128] in For a monotonically increasing function, the larger the variance, The larger the value, the more likely the eventual trigger time can be written as:

[0129]

[0130] And guarantee:

[0131]

[0132] For all of It also needs to be compared with the minimum action interval determined in step four. Maintain consistency to avoid overly concentrated triggering commands that could cause resource congestion.

[0133] Ultimately, , With the calculated Combined into the actual control commands issued:

[0134] When the clock reaches At that time, the corresponding [issue] was issued. The actuator performs actions such as jetting, electromagnetic adsorption, or mechanical gripping. During the completion of these actions, feedback information is collected from relevant sensor channels, including but not limited to: position / limit switch signals; pressure or flow sensor signals; magnetic flux or current feedback signals; and secondary visual confirmation signals. Based on this feedback, an execution result label is defined for each action.

[0135]

[0136] in This indicates that the target was successfully removed (or merged) in this operation. This indicates that the action was unsuccessful. Simultaneously, record execution-related time information, such as the moment the sensor confirms success. (If measurable), this provides data for subsequent online delay identification. 4. Online identification and update of delay components: Based on the difference between the trigger time and the actual success time, online identification of delay components is performed. For this action... Residuals can be defined:

[0137]

[0138] Alternatively, the deviation between the predicted arrival and the actual confirmation can be defined based on the specific implementation. This residual can be used to recursively update the total lead time or individual components, for example, using an exponential sliding motion.

[0139]

[0140] in For a smaller learning rate. Furthermore, the updates can be broken down into... On each component, and according to different templates or categories A set of delay parameters is maintained independently, so that different types of actions have their own optimal lead time.

[0141] The execution results were collected over a period of time. After incorporating relevant statistics, some decision parameters are adaptively adjusted based on success rate and false triggering. For example, when the actual success rate of a certain type of target is significantly lower than expected, the success probability threshold for that category is increased. Or appropriately increase the trigger advance. With pulse width adjustment, enhance action coverage; when the proportion of false triggers (actions executed but targets not detected) increases, appropriately increase the confidence requirement in the dynamic order acceptance threshold curve CDC, or shorten the pulse width to reduce coverage of non-targets in time proximity; when statistical analysis reveals arrival time residuals... When the existence bias is positive or negative, adjust the reference value of the delay component or the band speed fusion weight to make the new It more closely approximates the optimal trigger point in reality; when the beat utilization is low or execution is piling up, the minimum action interval can be finely adjusted. This increases or decreases throughput. These adaptive updates can be abstracted as parameter vectors. Update:

[0142]

[0143] in Include , CDC curve parameters Speed ​​fusion weights, etc. To update the gain matrix, This is the amount of correction based on statistical indicators.

[0144] It should be noted that after S4 completes action scheduling, it obtains the predicted arrival time and corresponding execution template for each target or merged target. Due to multiple stages from recognition to execution, including image acquisition delay, communication delay, control response time, and end-point activation delay, the overall delay causes a deviation between command triggering and the actual arrival time of the target. Therefore, a total lead time is calculated for each action, and the predicted arrival time is adjusted forward to obtain the optimal triggering time, ensuring the actuator starts the instant the target arrives at the pickup position, guaranteeing timing accuracy. During actual operation, execution feedback is acquired through sensors or secondary vision signals to determine the success or failure of each action, and the deviation between the predicted arrival time and the actual execution confirmation time is statistically analyzed. Based on these residuals, the system performs online identification and dynamic correction of various delay components, gradually approximating the system's true response characteristics. Simultaneously, control parameters such as the decision threshold, action interval, and speed fusion weight are adaptively adjusted based on the success rate and false trigger ratio, achieving closed-loop optimization between recognition, scheduling, and execution.

[0145] This embodiment also provides a foreign object pickup system that integrates foreign object identification with a conveyor belt, including:

[0146] The perception generation module acquires images and performs target detection, performs initial filtering based on the dynamic order acceptance threshold CDC, and generates candidate tickets.

[0147] The spatiotemporal collaborative estimation module integrates encoder and vision speed measurement, calculates arrival time distribution, and obtains success probability based on action time window to determine whether it can enter the queue.

[0148] The rolling scheduling module sorts feasible targets by score and selects targets under the conditions of minimum action interval and resource constraints.

[0149] The delay compensation trigger module calculates the total lead time and obtains the trigger time, issues commands and collects feedback, and updates the threshold, delay component and pulse width fusion weight online to form a closed-loop optimization.

[0150] This embodiment also provides a computer device applicable to the foreign object pickup method that combines foreign object identification with a conveyor belt, comprising: a memory and a processor; the memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions to implement the foreign object pickup method that combines foreign object identification with a conveyor belt as proposed in the above embodiment.

[0151] The computer device can be a terminal, comprising a processor, memory, communication interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, carrier networks, NFC (Near Field Communication), or other technologies. The display screen can be an LCD screen or an e-ink screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad on the computer device's casing, or an external keyboard, touchpad, or mouse.

[0152] This embodiment also provides a storage medium storing a computer program, which, when executed by a processor, implements the foreign object pickup method for foreign object identification and conveyor belt coordination as proposed in the above embodiments. The storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read Only Memory (EPROM), Programmable Red-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0153] In summary, this invention achieves the following: acquiring images above the conveyor belt, performing target detection to generate target data, and filtering low-reliability targets based on a dynamic order acceptance threshold; converting pixel positions into belt surface coordinates through a calibrated pixel-to-belt surface mapping, and fusing encoder belt speed and visual belt speed to obtain belt speed estimation; calculating the arrival time distribution of target data and a preset picking baseline based on target data, belt speed estimation, and detection time, and performing feasibility determination under opportunity constraints; within a rolling time window, sorting queue targets according to a target scoring function, selecting execution objects under minimum action interval and resource mutual exclusion constraints, and generating merged actions for horizontally adjacent targets that can share the same time coverage, outputting a scheduling result set; performing total delay compensation on the actions in the scheduling result set, performing time sequence correction based on the uncertainty of the arrival time distribution, and obtaining execution feedback to adaptively update the decision threshold and scheduling parameters online.

[0154] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A method for foreign object pickup in coordination with conveyor belt for foreign object identification, characterized in that: This includes acquiring images above the conveyor belt, performing target detection to generate target data, and filtering low-reliability targets based on dynamic order acceptance thresholds; The pixel position is converted into a strip surface coordinate through the calibrated pixel-to-strip surface mapping, and the strip speed is estimated by fusing the encoder strip speed and the visual strip speed. Based on target data, belt speed estimation, and detection time, the arrival time distribution of target data and preset picking baseline is calculated, and the feasibility of opportunity constraints is determined. Within the rolling time window, the queue targets are sorted according to the target scoring function, the execution target is selected under the minimum action interval and resource mutual exclusion constraints, and the merged action is generated for horizontally adjacent targets that can share the same time coverage, and the scheduling result set is output. The total delay compensation is performed on the actions of the scheduling result set, the timing correction is performed in combination with the uncertainty of the arrival time distribution, and the execution feedback is obtained to make online adaptive updates to the decision threshold and scheduling parameters. The step of generating target data through target detection includes acquiring the current frame image at a set sampling period along a unified time axis, wherein... This indicates the acquisition timestamp of the current frame image; preprocessing the current frame image yields the image frame used as the detection input, while retaining the timestamp. constant; The preprocessed image frames are input into the object detection network to obtain a candidate object set, where each candidate object's data... This includes the detection timestamp when the target was detected and the pixel position of the target in the image coordinate system. Detect the confidence level of the network output. Target category labels Dimensional rejectability features obtained from the estimation of the size, shape, and depth of the detection frame. ; Indicates the target index; A dynamic confidence threshold function is set based on the approximate remaining time for each candidate target. Applying the CDC rule, if the conditions are met, the current candidate target is considered to have passed the dynamic order acceptance filter; otherwise, it is marked as not accepting orders for this period. For each target that passes the CDC filter... Generate an initial data object.

2. The foreign object pickup method in collaboration with a conveyor belt for foreign object identification as described in claim 1, characterized in that: The calibrated pixel-to-area mapping includes, for each obtained target, defining the target object as... Offline calibration yields pixel-band homography matrix. This describes the projection relationship between the camera plane and the conveyor belt plane. During operation, two types of belt speed information are acquired, and a belt speed sequence is obtained from the encoder. Band velocity is estimated from the optical flow of textures or feature points in an image. ; Indicates the current time; For each target Using homography matrix pixel coordinates Projected onto the belt plane of the conveyor belt coordinate system, the physical coordinates of the target in the belt plane coordinate system are obtained through homogeneous normalization.

3. The foreign object pickup method in collaboration with a conveyor belt for foreign object identification as described in claim 2, characterized in that: The bandwidth estimation obtained by combining the fusion encoder bandwidth and the visual bandwidth includes, through pixel uncertainty... The propagation extends to the belt space, creating positional uncertainty; within a time window near the detection moment, the belt speed is obtained from the encoder. Calculating band velocity from image optical flow feature drift Define the estimated values ​​corresponding to the two belt speeds as follows: and ; Analyze the uncertainty of the two estimates and assign variances to them as consistency measures; when the absolute value of the difference between the two estimates... If the value is less than the preset consistency threshold, the belt speed estimates from the two sources are considered to be consistent, and the belt speed estimate is obtained by weighted fusion based on accuracy. .

4. The foreign object pickup method in collaboration with conveyor belt for foreign object identification as described in claim 3, characterized in that: The sorting of queue targets based on the target scoring function includes, at the current time... Open length is A rolling time window; for the target set determined by chance constraints. Perform scheduling and merging; Indicates the number of individuals in the target set; Each goal For each pick-up order, the time center value of the target arriving at the pick-up line is included. Uncertainty variance of target arrival time The target's lateral coordinates in the conveyor belt coordinate system, the target's category label, the confidence level obtained from image recognition, and the removability features characterizing the target's size or quality; Calculate the success probability of successfully removing the target within a given action time window; For each target Calculate the overall score After scoring, press From largest to smallest Sort the data to obtain the sorted queue. .

5. The foreign object pickup method in collaboration with a conveyor belt for foreign object identification as described in claim 4, characterized in that: The actions of performing total delay compensation and trigger time calculation on the scheduling result set include obtaining the sorted queue. Then, set the minimum action interval based on the cycle time capability of the actuator. Scheduling is performed as a constraint; during scheduling, from Starting with the highest-scoring target, examine each target in descending order of score. ; If the target is currently being added to the scheduling set With any selected target in the scheduling set Time center value difference All are not less than Then the current target will be added to the execution set of the current round. Otherwise, skip the current target and hold it until the next rolling cycle before attempting to schedule it again. The initial execution set that satisfies the minimum action interval constraint is obtained through screening. For sets Any pair of targets If the lateral distance of the targets is sufficient to cover them all within the duration of a single jet pulse, then the jet action is combined into a single jet motion; for targets that meet the merging condition... Construct a new merger target Reference arrival time A weighted average is performed, with the weights representing the success probability of the target. The merged pulse width is the maximum value plus a safety margin, based on the pulse width required for a single target, ensuring that the jet time window covers the arrival time distribution of both targets. After merging, the original two targets... From the set Remove from the middle and insert a new merge target. The updated execution set is obtained. Each element represents an actual execution action, and the output is a set of scheduling results. .

6. The foreign object pickup method in collaboration with conveyor belt for foreign object identification as described in claim 5, characterized in that: The online adaptive update of the decision threshold and scheduling parameters includes updating the scheduling result set. As input for delay compensation and trigger control steps, the recognition results are coordinated with the conveyor belt cycle time and execution resources; Utilizing delayed components and arrival time distribution parameters The set is calculated. The final trigger time of each action is combined with the action template and action parameters to form the actual control command, driving the pickup device to complete the foreign object removal; among which, Represents a set The index of the target corresponding to the action; To execute feedback and time residual Based on this, the delay parameters, success probability threshold, dynamic order acceptance threshold, minimum action interval, and speed fusion weight are updated online adaptively.

7. A foreign object pickup system that integrates foreign object identification and conveyor belt coordination, based on the foreign object pickup method that integrates foreign object identification and conveyor belt coordination according to any one of claims 1 to 6, characterized in that: include, The perception generation module acquires images and performs target detection, then performs the first round of filtering based on the dynamic order acceptance threshold CDC to form candidate tickets; The spatiotemporal collaborative estimation module integrates encoder and visual speed measurement, calculates arrival time distribution, and obtains success probability based on action time window to determine whether it can enter the queue. The rolling scheduling module sorts feasible targets by score and selects targets under the minimum action interval and resource constraints. The delay compensation trigger module calculates the total lead time and obtains the trigger time, issues commands and collects feedback, and updates the threshold, delay component and pulse width fusion weight online to form a closed-loop optimization.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, it implements the steps of the foreign object identification and conveyor belt coordinated foreign object pickup method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by the processor, it implements the steps of the foreign object identification and conveyor belt coordinated foreign object pickup method according to any one of claims 1 to 6.

Images