A parcel image recognition method for a sorting robot

Abstract

The present application relates to the technical field of intelligent logistics sorting and machine vision identification, and particularly relates to a parcel image identification method for a sorting robot, which acquires a parcel image sequence and parcel processing associated information, determines a parcel label area, position information and a parcel label face direction state at a target image acquisition time, determines an execution prediction result in combination with a posture evolution quantity and execution prediction information, further determines a flip axis and a direction control action, and determines a final direction control action or triggers a dynamic control action after visual confirmation before execution, the present application predicts a label face direction state through an image sequence, a posture evolution quantity and execution prediction information, determines a direction control action in combination with a direction state obsolescence, a flip axis credible difference, a code reading direction and an end trajectory, and performs visual confirmation before execution, thereby improving the matching, reliability and real-time performance of the identification result and the robot action.

Description

Technical Field

[0001] This invention relates to the field of intelligent logistics sorting and machine vision recognition technology, and more specifically, to a package image recognition method for sorting robots. Background Technology

[0002] As logistics sorting operations become more automated and faster, sorting robots are increasingly involved in processes such as package identification, gripping, flipping, releasing, and sorting. In these scenarios, packages typically move continuously along a conveyor line. Image acquisition devices photograph the passing packages and use the information from the shipping labels, barcodes, or other tags in the images to identify them, providing a basis for the sorting robot to perform corresponding actions.

[0003] In conventional package recognition, the label area and its orientation are typically determined based on an image of the package captured at a specific moment, and the robot's end effector is then used to select either a gripping or flipping action. This method can meet basic sorting requirements when the package posture is relatively stable, the label area is sufficiently exposed, and the distance between the image acquisition position and the execution position is relatively short. However, in actual transport processes, packages vary significantly in shape, size, material, and placement posture. Packages may also slip, deflect, or slightly tumble due to factors such as conveyor belt vibration, contact with adjacent packages, and changes in surface friction. Consequently, the orientation of the package label surface obtained at the moment of image acquisition may deviate from the orientation when the sorting robot actually reaches the package's execution position.

[0004] Meanwhile, there is usually a time interval between the planned motion trajectory of the sorting robot's end effector and its actual contact with the package. If the flip axis, gripping side, or release action is determined directly based solely on early image recognition results, it may be difficult to fully reflect the posture evolution of the package within this time interval. Especially in situations where the label area is close to the edge of the package, the gripping side is adjacent to the label area, and the preset reading direction has high requirements for the label face orientation, the reliability of the flip axis selection depends not only on the current label face orientation, but also on the reliability of the orientation state after propagation over time, the interference of the end effector motion trajectory, and the actual visual state before the action is executed.

[0005] Therefore, this application proposes a package image recognition method for sorting robots. Summary of the Invention

[0006] In view of the shortcomings of the existing technology, the purpose of this invention is to provide a package image recognition method for sorting robots.

[0007] To achieve the above objectives, the present invention provides the following technical solution: A method for package image recognition for sorting robots includes: S1, acquire the package image sequence and package processing association information of the target package; wherein, the package processing association information includes image acquisition time information, preset code reading direction and execution prediction information, and the execution prediction information includes the arrival prediction information of the target package and the end motion trajectory information of the end effector of the sorting robot; S2, based on the package image sequence and image acquisition time information, determine the target image acquisition time, and determine the package label area, the position information of the package label area, and the orientation state of the package label surface at the target image acquisition time; S3, determine the attitude evolution of the target package based on the package image sequence; based on the package label orientation state, attitude evolution, and execution prediction information at the time of target image acquisition, determine the package execution position, the expected execution time of the sorting robot, the temporal confidence region of the orientation state, the predicted orientation state of the package label, and the orientation state staleness of the target package. S4. Based on the predicted orientation state of the package label, the age of the orientation state, the location information of the package label area, the preset reading direction, and the end motion trajectory information, determine the target candidate flip axis, the pre-selected direction adjustment action, and the flip axis confidence difference. S5, based on the location information of the package label area and the end motion trajectory information, performs pre-action visual confirmation on the target package before the end effector of the sorting robot reaches the package execution position, and determines the final direction control action or triggers dynamic control action based on the pre-action visual confirmation.

[0008] In one embodiment, the orientation state of the package label surface includes the normal direction of the package label bearing surface where the package label area is located, the main axis direction of the target package, the position information of the package label area, the orientation relationship of the package label bearing surface relative to the preset reading direction, and the confidence level of the orientation state observation.

[0009] In one embodiment, the orientation state temporal confidence domain is the temporal confidence domain formed by the propagation of the orientation state of the package label surface from the target image acquisition time to the expected execution time of the sorting robot. The orientation state temporal confidence domain includes the predicted orientation state of the package label surface at the expected execution time of the sorting robot, the orientation state propagation uncertainty, and the effective envelope of the orientation state.

[0010] In one embodiment, the orientation state staleness is determined as follows: the orientation state propagation deviation is determined based on the orientation state of the package label surface at the time of target image acquisition and the predicted orientation state of the package label surface; the temporal confidence domain expansion is determined based on the orientation state propagation uncertainty and the effective envelope of the orientation state; the attitude evolution residual factor is determined based on the attitude evolution amount; the temporal interval decay factor is determined based on the temporal interval between the time of target image acquisition and the expected execution time of the sorting robot; and the orientation state staleness is determined based on the orientation state propagation deviation, the temporal confidence domain expansion, the attitude evolution residual factor, and the temporal interval decay factor.

[0011] In one embodiment, a set of candidate direction control actions and a set of candidate flip axes are generated based on the predicted orientation state of the package label, the age of the orientation state, the location information of the package label area, the preset reading direction, and the end motion trajectory information. The reliable distribution of candidate flip axes is determined based on the set of candidate direction control actions and the set of candidate flip axes. Based on the reliable distribution of candidate flip axes, a target candidate flip axis is determined, a pre-selected direction control action is determined from the set of candidate direction control actions, and the reliability difference of the flip axis is determined.

[0012] In one embodiment, the candidate orientation control action set includes a candidate flipping action, a candidate clamping side selection action corresponding to the candidate flipping action, and a candidate release action, wherein the candidate flipping action corresponds to different candidate flipping axes in the candidate flipping axis set.

[0013] In one embodiment, the candidate flip axis confidence distribution is formed as follows: for each candidate flip axis, an axial motion evaluation vector is constructed based on the image frames in the wrapped image sequence, and the confidence level of the candidate flip axis in the corresponding image frame is determined according to the axial motion evaluation vector; for adjacent image frames in the wrapped image sequence, the confidence level of the same candidate flip axis in the adjacent image frames is obtained respectively, and the confidence difference of coaxial adjacent frames is determined according to the difference of the confidence level of the candidate flip axis in the adjacent image frames; and a candidate flip axis confidence distribution is formed based on the confidence levels of multiple candidate flip axes and the confidence difference of coaxial adjacent frames.

[0014] In one embodiment, the axial motion evaluation vector includes a code reading orientation benefit factor, a grasping reachability margin factor, a tag occlusion penalty factor, a trajectory interference penalty factor, a temporal confidence domain sensitivity factor, and a staleness penalty factor.

[0015] In one embodiment, in S5, based on the location information of the package label area and the end motion trajectory information, a pre-action confirmation image is collected before the end effector of the sorting robot reaches the package execution position. The orientation state of the wrapped label surface before the action is determined based on the pre-action confirmation image, and the orientation state of the wrapped label surface before the action is compared with the predicted orientation state of the wrapped label surface to obtain the orientation deviation before the action. The orientation state temporal confidence domain is corrected based on the orientation deviation before the action, and the orientation state staleness and flip axis confidence error are updated according to the corrected orientation state temporal confidence domain to obtain the updated orientation state staleness and the updated flip axis confidence error.

[0016] In one embodiment, the final direction control action or the triggering of a dynamic control action is determined based on the updated orientation state staleness, the updated flip axis confidence error, the orientation deviation before the action, and the preset verification conditions. When determining the final direction control action, generate the flip axis selection command, clamping side selection command, or release command corresponding to the final direction control action.

[0017] Compared with the prior art, the present invention has the following beneficial effects: This invention obtains the package image sequence, image acquisition time information, preset code reading direction, and execution prediction information of the target package to determine the package label area, the position information of the package label area, and the orientation state of the package label surface at the target image acquisition time. It also combines the attitude evolution quantity to predict the orientation state of the package label surface at the package execution position. This can reduce the orientation judgment deviation caused by changes in package attitude between the image acquisition time and the expected execution time of the sorting robot, and improve the correspondence between the orientation control basis and the actual execution state of the package. Based on the predicted label orientation state, orientation state age, label area location information, preset reading direction, and end-effector trajectory information, the target candidate flip axis, pre-selected direction control action, and flip axis confidence difference are determined. The label orientation requirement, label area location constraint, and end-effector motion conditions are incorporated into the flip axis selection process, which helps to reduce the risk of the label area being obscured by the clamping, the flip action being unfavorable for reading, and the end-effector trajectory interfering with the process, and improves the adaptability of the pre-selected direction control action. Before the end effector of the sorting robot reaches the package execution position, a pre-action confirmation image is acquired. Based on the pre-action direction deviation between the pre-action package label orientation state and the predicted package label orientation state, the orientation state staleness and flip axis confidence error are updated, and the final orientation adjustment action or dynamic adjustment action is determined accordingly. This is beneficial for adapting to the slippage, deflection or posture change of the target package during the transportation process, and improves the real-time consistency between the package label recognition result and the flip axis selection, clamping side selection or release operation. Attached Figure Description

[0018] Figure 1This is a schematic diagram of the overall process of a package image recognition method for sorting robots according to the present invention; Figure 2 This is a schematic diagram of the process of determining the target candidate flip axis, the pre-selected direction adjustment action, and the flip axis confidence difference in this invention. Figure 3 This is a schematic diagram illustrating the formation process of the axial motion evaluation vector, candidate flip axis credibility, coaxial adjacent frame credibility difference, and candidate flip axis credibility distribution in this invention. Figure 4 This is a schematic diagram illustrating the process of determining the pre-action visual confirmation, pre-action directional deviation, final directional adjustment action, or triggered dynamic adjustment action in this invention. Detailed Implementation

[0019] Reference Figure 1 A method for package image recognition for sorting robots, comprising: S1. Acquire the package image sequence and package processing association information of the target package. The package processing association information includes image acquisition time information, preset reading direction, and execution prediction information. The execution prediction information includes the target package arrival prediction information and the end effector trajectory information of the sorting robot. The package image sequence records the appearance changes, label area exposure, and posture change trends of the target package within the image acquisition range. The image acquisition time information establishes a time reference for each image frame in the package image sequence, facilitating the differentiation of the acquisition sequence of different image frames. The preset reading direction defines the spatial orientation required for reading package labels in the sorting scenario. The target package arrival prediction information reflects the arrival position and arrival time changes of the target package during transport. The end effector trajectory information of the sorting robot reflects the movement path of the sorting robot's end effector when approaching, gripping, flipping, or releasing the target package.

[0020] By simultaneously acquiring visual information, time information, code reading direction information, and execution prediction information, the image recognition process of the target package establishes a correspondence with the timing of the sorting robot's actions, its movement path, and the orientation of the code reading, thus avoiding making directional judgments based solely on a single frame image.

[0021] After the target package enters the image acquisition area of ​​the sorting conveyor line, image acquisition devices located on the side and above the conveyor line continuously acquire a sequence of package images and simultaneously read the image acquisition time information matching each image frame. Simultaneously, based on the barcode reader installation angle, barcode window position, and label reading requirements at the sorting site, a preset barcode reading direction is obtained. Furthermore, based on the encoder feedback from the conveyor line, the target package's current position on the conveyor line, the conveying speed, and the current motion state of the sorting robot's end effector, execution prediction information is generated. This execution prediction information includes the target package arrival prediction information and the end effector trajectory information of the sorting robot. The target package arrival prediction information records the predicted position and time of the target package's arrival in the sorting robot's executable area, while the end effector trajectory information records the spatial motion path of the sorting robot's end effector when approaching the target package, avoiding adjacent packages, clamping the target package, or adjusting the target package's direction.

[0022] For example, when a target package moves along the conveyor line at a predetermined speed and passes through the shared field of view of the top and side cameras, the top camera continuously acquires image frames of the target package's upper surface, and the side camera acquires image frames of the target package's side profile. Each image frame carries image acquisition time information under a unified clock. The sorting control equipment calculates the time when the target package enters the gripping area based on the conveyor line encoder pulses, and forms the end-effector trajectory information of the sorting robot's end effector based on the path taken by the sorting robot's end effector to avoid the conveyor line's obstruction, approach the target package's edge, and complete the gripping action. Through the above processing, the package image sequence, image acquisition time information, preset code reading direction, target package arrival prediction information, and end-effector trajectory information of the sorting robot's end effector are obtained under the same time reference, ensuring that the visual state, conveying state, and robot execution state of the target package have a consistent data basis.

[0023] Establish a transport line coordinate system under a unified time reference , No. Coordinate system of image acquisition device Sorting robot base coordinate system End effector coordinate system and the target package coordinate system . No. The intrinsic parameter matrix of the image acquisition device is denoted as... , No. Coordinate system of image acquisition device To the coordinate system of the conveyor line The rotation matrix and translation vector are denoted as follows: and .

[0024] No. Frame by the first Image captured by an image acquisition device. Original timestamps of image frames. Calibrated time offset Correction to uniform time The formula is:

[0025] The timestamps of the conveyor encoder and robot controller are aligned to a unified time base according to their respective calibration offsets. The time synchronization error between the image frame, the conveyor encoder, and the robot controller does not exceed half of the smaller of the robot control cycle and the image acquisition cycle; image frames exceeding this range are not included in the selection of the target image acquisition time.

[0026] The robot's end effector trajectory in the sorting robot base coordinate system The coordinates are obtained below and converted to the conveyor line coordinate system according to the equipment calibration relationship. Participating in candidate clamping side and trajectory interference judgment; end effector coordinate system The coordinate system is used to describe the contact area of ​​the grippers.

[0027] Packaging label area pixels In depth Or, under the constraint of the label bearing surface, convert to the coordinates of the image acquisition device. and the coordinates of the conveyor line The formula is:

[0028] in, and For pixel coordinates, This is either a depth value or a scale value obtained from the equation of the label bearing surface. For the pixel at the th Coordinate system of image acquisition device The spatial coordinates below, For the corresponding points in the conveyor line coordinate system The spatial coordinates below.

[0029] S2, based on the package image sequence and image acquisition time information, determine the target image acquisition time, and determine the package label area, its position information, and its orientation state at that time. The target image acquisition time is the image acquisition time in the package image sequence that matches the target package label recognition quality, attitude stability, or prediction information. The package label area is the image area on the target package carrying label recognition information; its position information records the correspondence between the area and its location in image coordinates, package surface position, or spatial position. The package label orientation state describes the directional relationship between the package label bearing surface where the label area is located and the preset reading direction and the target package's attitude.

[0030] This step transforms the visual observations in the package image sequence into a time-referenced package label area, the location information of the package label area, and the orientation state of the package label face, so that the label position and label orientation of the target package at the time of target image acquisition can be expressed in a definite manner.

[0031] Based on the image acquisition time information corresponding to each image frame in the package image sequence, the label visibility, image clarity, motion blur, package edge integrity, and label texture recognizability of the target package at different acquisition times are comprehensively evaluated. The acquisition time with relatively complete label visibility, minimal change in the target package posture, and matching with the target package's motion state is selected as the target image acquisition time. In the image frame corresponding to the target image acquisition time, the outer contour, edge direction, label texture boundary, and barcode or label character area of ​​the target package are identified to determine the package label area on the target package. The position information of the package label area is determined based on the image coordinates, the outer contour coordinates of the target package, and the calibration relationship of the image acquisition device. Furthermore, by combining the edge orientation, brightness continuity, displacement relationship of the label area in adjacent image frames, and the outline of the target package, the normal direction of the label bearing surface is obtained by fitting the label bearing surface. Based on the long side, wide side, and height directions of the target package at the time of target image acquisition, the main axis direction of the target package is determined. The normal direction of the label bearing surface is compared with the preset reading direction to obtain the orientation relationship of the label bearing surface relative to the preset reading direction. The confidence level of the orientation state observation is determined by combining the position information of the label area, the boundary integrity of the label bearing surface, the label texture clarity, the image acquisition angle deviation, and the orientation consistency between adjacent image frames. A valid image frame is an image frame in which the time synchronization error meets the requirements, the outer contour of the target package is identifiable, the package label area is locatable, and the pixel coordinates are successfully converted.

[0032] For the first Frame-based image quality scoring The formula is:

[0033] in, To ensure the completeness of the label. For label texture clarity, To ensure the integrity of the package edges, To ensure directional consistency between adjacent frames, To determine the degree of motion blur, and The values ​​range from 0 to 1; and These are image quality weights, all of which are not less than 0, and .

[0034] Select the image frame with the highest image quality score from the valid image frames corresponding to the same target package, and denot the frame number as follows: The time of image acquisition is recorded as The orientation state of the package label at the moment of target image acquisition is denoted as... It includes the center point of the target package in the coordinate system of the conveyor line. The lower position The center point of the package label area is in the coordinate system of the conveyor line. The lower position The unit normal direction of the label bearing surface Target package main axis unit direction Target package coordinate system Relative to the conveyor line coordinate system attitude rotation matrix and direction state observation confidence .

[0035] The unit normal direction of the label bearing surface is taken from the inside of the target package to the outside of the label; the positive direction of the principal axis of the target package is according to the target package coordinate system. The right hand is confirmed.

[0036] Confidence of orientation state observation The value is obtained by normalizing the label exposure completeness, label texture clarity, wrapping edge integrity, directional consistency of adjacent frames, and motion blur degree of the image frame corresponding to the target image acquisition time, with a value range of 0 to 1.

[0037] For example, when a target package passes the top image acquisition device and the side image acquisition device on the conveyor line, if the boundary of the label area in the top image frame is complete, the barcode texture is clear, and the long side direction of the target package is stable at a certain image acquisition moment, then the image acquisition moment is determined as the target image acquisition moment. At this moment, the label area is determined as the package label area, and its position on the upper surface of the target package, which is biased towards the left front side, is recorded. Based on the normal direction fitted to the upper surface of the target package, the long side direction of the target package, and the angle between the normal direction and the preset reading direction, the package label surface orientation state of the target package at the target image acquisition moment is formed.

[0038] S3. Determine the attitude evolution of the target package based on the package image sequence; based on the package label orientation state, attitude evolution, and execution prediction information at the time of target image acquisition, determine the package execution position, the expected execution time of the sorting robot, the temporal confidence domain of the orientation state, the predicted orientation state of the package label, and the orientation state staleness of the target package.

[0039] The attitude evolution quantity reflects the orientation changes, tumbling changes, or translational changes of the target package within the time range covered by the package image sequence. The orientation state of the package label at the moment of target image acquisition reflects the observed state at that moment. Combining the attitude evolution quantity and execution prediction information, a direction state prediction result close to the expected execution time of the sorting robot is formed. The package execution position corresponds to the spatial position of the target package receiving the orientation control action of the sorting robot's end effector. The expected execution time of the sorting robot corresponds to the time node when the end effector of the sorting robot and the target package have an execution relationship. The temporal confidence domain of the orientation state describes the confidence range of the package label orientation state after it propagates from the moment of target image acquisition to the expected execution time of the sorting robot. The predicted orientation state of the package label provides an estimate of the orientation state near the expected execution time of the sorting robot. The orientation state staleness reflects the degree of reliability decay of the observation information at the moment of target image acquisition due to time intervals and attitude changes.

[0040] Based on the changes in the outer contour, edges, label area, and main axis direction of the target package in consecutive image frames, the attitude evolution of the target package is determined. The attitude evolution includes the angular change, translational change, offset of the main axis direction, and normal change of the label bearing surface between adjacent image frames. The image acquisition time information is used to calculate the time interval between adjacent image frames to obtain the temporal continuity of the target package's attitude change. Sorting robot expected execution time The spatial position of the target package at that moment is taken as the package execution position, which is the earliest time when the target package enters the robot's executable area and the end effector can reach the candidate contact position.

[0041] For consecutive image frames and Calculate the target wrapping posture separately and central location Inter-frame rotation Inter-frame translation amount Translation speed and angular velocity satisfy:

[0042]

[0043] in, Rotation matrix The corresponding Lie algebra matrix, This is a mapping from an antisymmetric matrix to a three-dimensional vector.

[0044] Before the target image acquisition time The velocity-weighted average of the effective frames is used as the predicted velocity. and predicted angular velocity The formula is:

[0045] in, The number of valid frames used in the prediction. Take 3 to 6; if there are fewer than 3 valid frames, take the actual number of valid frames. For frame weights, and .

[0046] make The sorting robot's estimated execution time The predicted position, predicted attitude, predicted label bearing surface normal direction, predicted package main axis direction, and predicted label area center position are as follows:

[0047]

[0048] in, For the reason The constructed antisymmetric matrix, For matrix exponents, and This is the predicted amount for the expected execution time.

[0049] For example, when the target package rolls slightly on the conveyor line, the long side of the package deflects sequentially in three consecutive frames of images, and the package label area gradually moves from the left side of the upper surface to the center of the image. At this time, based on the offset of the package's main axis and the change in the normal direction of the package label bearing surface, the attitude evolution of the target package after the target image acquisition time can be obtained. Based on the package label orientation state, posture evolution, and execution prediction information corresponding to the target image acquisition time, the package label orientation state is propagated from the target image acquisition time to the sorting robot's expected execution time.

[0050] The predicted package label orientation state at the expected execution time is denoted as It includes predicting the center point of the label region. Predict the normal direction of the label bearing surface. And predict the main axis direction of the package The uncertainty in directional state propagation is denoted as... , Initial observation uncertainty at the time of target image acquisition Jacobian matrix propagated via state and propagation noise The formula is:

[0051] in, For predicting the state Status of target image acquisition time The first-order partial derivative matrix; It consists of the detection variance of the label area position, the detection variance of the label bearing surface normal, and the detection variance of the package main axis direction; It consists of conveyor line speed error, inter-frame attitude fluctuation, time synchronization error, and target package edge occlusion error.

[0052] Take confidence level , for to The numerical values ​​in the direction state time series confidence region Defined as:

[0053] in, For direction state variables, for Dimensions For degrees of freedom Confidence level The corresponding chi-square quantile.

[0054] Directional status aging The value ranges from 0 to 1. Directional state propagation deviation factor. The prediction offset is determined by normalization based on the label bearing surface normal direction, wrapping principal axis direction, and label area center position between the target image acquisition time and the expected execution time; the temporal confidence region expansion factor is also included. in accordance with Compared to The degree of covariance expansion is normalized to determine the attitude evolution residual factor. Determined based on the normalization of fluctuations in effective intra-frame translational velocity and angular velocity relative to predicted velocity and angular velocity; temporal interval attenuation factor. in accordance with The time reference for transporting packages from the visual acquisition center to the package execution location is normalized and determined.

[0055] Directional status aging satisfy:

[0056] in, and The weights for merging age are all not less than 0, and ; This means that the input value is limited to between 0 and 1, with values ​​less than 0 counted as 0 and values ​​greater than 1 counted as 1.

[0057] For example, if the target image is acquired at the time corresponding to the fifteenth frame, and the sorting robot is expected to execute the sorting at 0.4 seconds after that acquisition time, if the target package's attitude evolution during this time period shows that the normal direction of the package label bearing surface is deflected relatively small, the uncertainty of the orientation state propagation is in a low range, and the effective envelope of the orientation state has not expanded significantly, then the orientation state aging is low. If the target package experiences edge collisions or abrupt attitude changes during transportation, causing an increase in the difference between the predicted orientation state of the package label surface and the orientation state of the package label surface corresponding to the acquisition time of the target image, and the attitude evolution residual factor increases, then the orientation state aging will increase accordingly.

[0058] Reference Figure 2 S4. Based on the predicted orientation state of the package label, the age of the orientation state, the location information of the package label area, the preset reading direction, and the end motion trajectory information, determine the target candidate flip axis, the pre-selected direction adjustment action, and the flip axis confidence difference.

[0059] The preset reading direction imposes reading constraints on the orientation of the package label's surface, while the end effector's motion trajectory information forms motion constraints on the reachable path, contact direction, and avoidance relationships of the sorting robot's end effector. The predicted package label surface orientation state, orientation state aging, and package label area position information together reflect whether the current prediction result is suitable for direct orientation adjustment. The target candidate flip axis is the flip axis more suitable for adjusting the target package's orientation from the candidate flip axis set. The pre-selected orientation adjustment action is the orientation adjustment scheme formed before visual confirmation. The flip axis confidence difference reflects the confidence advantage of the target candidate flip axis relative to other candidate flip axes. This step combines the package label area recognition result, orientation state prediction result, and end effector motion conditions to form the flip axis selection and orientation adjustment action selection results related to robot action, and retains the flip axis confidence difference as a judgment basis for pre-action review.

[0060] Based on the predicted orientation state of the package label, the age of the orientation state, the location information of the package label area, the preset reading direction, and the end effector trajectory information, a set of candidate orientation control actions and a set of candidate flip axes are generated to address the possible orientation adjustment needs of the target package at the package execution position. The set of candidate orientation control actions includes candidate flip actions, candidate gripping side selection actions corresponding to the candidate flip actions, and candidate release actions. The candidate flip actions correspond to different candidate flip axes in the set of candidate flip axes. Each candidate flip axis is set in combination with the orientation relationship of the package label bearing surface relative to the preset reading direction, the offset position of the package label area on the surface of the target package, the spatial path of the sorting robot's end effector approaching the target package, and the accessibility of the gripping side. For example, when the predicted state of the package label orientation shows that the package label bearing surface is facing the side of the conveyor line and is not conducive to reading the preset code reading direction, the candidate flip axis set includes candidate flip axes that flip along the long side of the target package, candidate flip axes that flip along the short side of the target package, and release corresponding selection that maintains the current posture. The candidate clamping side selection action avoids the label area based on the position information of the package label area to reduce clamping obstruction. Candidate flip axis in target package coordinate system As defined in [the document], projection line segments in the image coordinate system are not used as the criterion for coaxiality determination. Candidate flip axes include the target wrapping coordinate system. The lower long axis short side axis and height axis .

[0061] No. Frame 1 The candidate flip axes are in the conveyor line coordinate system The downward direction is:

[0062] in, , For the first Frame-based target wrap coordinate system Relative to the conveyor line coordinate system The attitude rotation matrix. The same candidate flip axis refers to having the same... Candidate flip axes with numbers.

[0063] The candidate orientation control action consists of a candidate flip axis, a candidate flip angle, and a candidate gripping side. The candidate gripping side is determined based on tag avoidance, robot reachability, and gripper contact area. Action combinations that satisfy tag avoidance and robot reachability constraints are included in the evaluation; action combinations with tag overlap or robot inaccessibility are not included in the evaluation.

[0064] The signified candidate flip angle is denoted as , Pick or Its positive or negative direction is determined by the direction of the end-effector push. When the same candidate flip axis has opposite executable push directions, they are evaluated as different action combinations.

[0065] The normal direction of the label bearing surface after the candidate flipping action is completed is denoted as The preset reading direction is in the coordinate system of the conveyor line. The unit vector under is denoted as The rotation operator is denoted as Preset reading direction The positive direction is taken as the direction when the tag's outer normal is consistent with the reader's optimal reading direction.

[0066]

[0067] in, For the first Normal direction of the tag carrying surface under the frame. For around the unit axis Rotation angle Rotation operator.

[0068] Reference Figure 3 For each candidate flip axis, the axial motion evaluation content is calculated based on the image frames in the wrapped image sequence. The candidate flip axis in the first The frame-based axial motion evaluation includes the read orientation benefit factor. Extracting reachability margin factors Tag occlusion penalty factor Trajectory interference penalty factor Time series confidence region sensitivity factor and orientation status aging .

[0069] Each factor satisfies:

[0070]

[0071]

[0072]

[0073] in, The normal direction of the label bearing surface after the candidate flipping action is completed. To preset the reading direction in the conveyor line coordinate system The unit vector below, This represents the minimum reachable margin from the end-effector trajectory to the candidate gripping side. To minimize the safety margin, This is the projection of the gripper contact area. This is the area for package labels. and Calculate the intersection of areas on the same package surface or within the same image plane. This is the minimum distance between the candidate end trajectory and the conveyor line boundary, adjacent packages, and robot restricted areas. This is the safe distance threshold.

[0074] Time series confidence region sensitivity factor The benefit based on the read orientation corresponding to the candidate flip axis is in the first... The maximum change in the directional state timing confidence domain during frame propagation to the expected execution time, and then normalized to a sensitivity threshold. Obtained through normalization. The maximum variation does not exceed... Normalize proportionally, exceeding Hours are counted as 1; The larger the value, the more sensitive the candidate flip axis is to directional uncertainty.

[0075] Candidate flip axis confidence satisfy:

[0076] in, and All are not less than 0, and .

[0077] When the same candidate flip axis corresponds to multiple candidate flip angles, candidate gripping sides, or end effector pushing directions, calculate separately for each action combination that satisfies the tag avoidance and robot reachability constraints. Take the largest value As the candidate flip axis in the The reliability of the frame; the candidate flip angle, candidate clamping side and end push direction corresponding to the value are recorded together with the candidate flip axis.

[0078] Coaxial adjacent frame reliability difference For the first Candidate flip axes in the effective frame set The absolute value of the confidence difference between adjacent valid frames sorted by time, without assuming the original frame numbers are consecutive. Overall credibility of candidate flip axes satisfy:

[0079] in, The set of valid frames for evaluation. For the number of valid frames, for The smallest frame number in the sequence. For frame weights, and ; The weight for penalizing fluctuations between adjacent frames. When At that time, the fluctuation penalty term for adjacent frames is set to 0.

[0080] The set of candidate flip axes that were not excluded It consists of candidate flip axes for at least one candidate gripping side that satisfies tag avoidance and robot reachability constraints. The angle between the predicted tag bearing surface normal direction and the preset code reading direction is no greater than [value missing]. And the directional state of antiquity Not greater than At that time, the candidate release action serves as a directional control action. The value is empty and the angle between the predicted tag bearing surface normal direction and the preset code reading direction is no greater than 100°. At that time, the target direction adjustment action selects candidate release actions; It is empty and the included angle is greater than At that time, dynamic adjustment actions are triggered.

[0081] Candidate Flip Axis Confidential Distribution satisfy:

[0082] in, For reliable distributed temperature coefficients, Number the candidate flip axes. middle, The candidate flip axis corresponding to the maximum value is determined as the target candidate flip axis.

[0083] Reversal axis reliability The difference between the confidence distribution value of the target candidate flip axis and the highest confidence distribution value among the remaining candidate flip axes. The larger the value, the more obvious the credible advantage of the target candidate flip axis compared to the other candidate flip axes; When there is only one candidate flip axis, Take 1.

[0084] Reference Figure 4 S5, based on the location information of the package label area and the end effector's motion trajectory information, before the sorting robot's end effector reaches the package execution position, performs pre-action visual confirmation on the target package, and determines the final orientation adjustment action or triggers a dynamic adjustment action based on the pre-action visual confirmation. The pre-action visual confirmation occurs before the sorting robot's end effector reaches the package execution position, leaving room and time for adjusting or canceling the target package's orientation adjustment action. The pre-action visual confirmation is based on the location information of the package label area and the end effector's motion trajectory information, avoiding reliance solely on the package label orientation state at the time of target image acquisition or unconfirmed predicted package label orientation state.

[0085] Pre-action visual verification reviews the actual state of the target package before execution, checking whether the predicted package label orientation, orientation aging, and flip axis confidence error meet preset verification conditions. When the pre-action visual verification result matches the prediction result, the final orientation adjustment action is determined; when the orientation deviation, aging, or flip axis confidence error does not meet the execution reliability requirements, dynamic adjustment action is triggered to reduce the risk of accidental flipping, accidental clamping, or accidental release.

[0086] Based on the location information of the package label area and the end effector's motion trajectory information, before the sorting robot's end effector reaches the package execution position, a pre-action visual confirmation moment is selected on the approach path corresponding to the end effector's motion trajectory information, and a pre-action confirmation image is acquired at this moment. The pre-action visual confirmation moment is located before the sorting robot's end effector enters the gripping contact range, and the pre-action confirmation image covers the package label area of ​​the target package, the boundary of the package label bearing surface, the outer contour of the target package, and the relative position of the end effector approaching the target package. Based on the pre-action confirmation image, the actual position of the package label area, the normal direction of the package label bearing surface, and the main axis direction of the target package are re-identified to determine the pre-action package label surface orientation state. This pre-action package label surface orientation state is then compared with the predicted package label surface orientation state to obtain the pre-action orientation deviation. This pre-action orientation deviation includes the deviation of the normal direction of the package label bearing surface, the deviation of the main axis direction, the deviation of the package label area position, and the orientation difference of the package label bearing surface relative to the preset reading direction. For example, if the image confirming the action before the target package enters the sorting robot's execution area on the conveyor line shows that the package label area has shifted from the predicted position on the upper surface to a position closer to the front edge, and the normal direction of the package label bearing surface has an additional deflection relative to the preset reading direction, then the direction deviation before the action is obtained. The moment of visual confirmation before the action is recorded as The estimated execution time of the sorting robot is recorded as The minimum time for replanning is denoted as Minimum time for replanning The time consumed is determined by the time spent on image acquisition, image recognition, robot trajectory replanning, and control command issuance before the action is confirmed.

[0087] The pre-action confirmation image obtains the orientation state of the label-wrapped surface before the action, including the position of the center point of the label-wrapped area before the action. Before the action, wrap the label bearing surface in the unit normal direction. Before the action, wrap the main axis unit direction. and pre-action observation confidence .

[0088] Confirm the image corresponding to the action and According to the aforementioned posture, the propagation relationship is... spread to Afterwards, it participates in the calculation of pre-action directional deviation and state fusion; the quantity after propagation is still recorded as... and .

[0089] Pre-movement directional deviation satisfy:

[0090] in, and The bias fusion weights are all not less than 0, and ; The length of the diagonal of the circumscribed cuboid of the target is given.

[0091] Pre-action observation fusion coefficient satisfy:

[0092] in, To prevent constants with zero denominators, take .

[0093] The corrected center point of the predicted label area, the corrected normal direction of the predicted label bearing surface, and the corrected principal axis direction of the predicted package are as follows:

[0094]

[0095] The observation uncertainty corresponding to the pre-action confirmation image is denoted as . . The variance is composed of the detection variance of the label region position in the pre-action confirmed image, the detection variance of the label bearing surface normal, and the detection variance of the wrapping principal axis direction, and is calculated according to the pre-action observation confidence level. Scaling When decrease Increase.

[0096] Confirm the observation uncertainty corresponding to the image before taking action. The relationship of propagation according to the same posture is... spread to Afterwards, it participates in the calculation of the propagation uncertainty of the corrected directional state; the propagated quantity is still denoted as... .

[0097] The deviation expansion coefficient is denoted as The identity matrix with the same dimension as the uncertainty of the direction state propagation is denoted as Corrected direction state propagation uncertainty satisfy:

[0098] The corrected directional state temporal confidence region is determined in the same form as the aforementioned directional state temporal confidence region, wherein the state center is determined by the center point of the corrected label region. Normal direction of the label bearing surface and the main axis direction of the package Composition, propagation uncertainty is composed of Replace with Confidence level Maintain consistency.

[0099] Updated direction status obsolescence And the updated flip axis credibility difference satisfy:

[0100] in, The age index is updated. The confidence difference is used to update the coefficients. and All values ​​are between 0 and 1.

[0101] Preset review criteria include an obsolescence threshold. Confidence difference threshold Directional deviation threshold Minimum time for replanning Orientation deviation threshold It is calibrated based on the allowable deviation of the reading posture, the allowable deviation of the clamping side selection, and the minimum safe distance between the jaw contact area and the tag area.

[0102] when Not greater than , Not less than and Not greater than At that time, confirm the current directional adjustment action. Greater than , Less than or Greater than And the remaining time Not less than When this happens, a dynamic control action is triggered. Greater than , Less than or Greater than And the remaining time Less than If the target package continues to be conveyed and meets the mechanical safety constraints, the flipping is canceled and a release command is output; if the target package enters the robot's safety boundary, the end effector is kept in a safe position.

[0103] The dynamic control actions include at least one of the following: recalculating the candidate flip axis confidence distribution, switching the candidate clamping side, reducing the end effector approach speed, canceling the flip and outputting a release command, and maintaining the current end position and acquiring a new confirmation image.

[0104] For example, if the pre-action confirmation image confirms that the target candidate flip axis still maintains the highest confidence level, and the package label bearing surface can approach the preset reading direction after being flipped by the target candidate flip axis, then a flip axis selection instruction and a clamping side selection instruction are generated; if the pre-action confirmation image shows that the target package has laterally slipped, causing the original candidate clamping side to approach the package label area and pose a risk of obstruction, then a dynamic adjustment action is triggered to select a new clamping side or pause the flipping and generate a release instruction.

[0105] In this embodiment, the weights of the same type in the same fusion rule are equal when there are no historical samples; when there are historical samples, the weights are set according to the degree of difference between the correctly executed samples and the incorrectly executed samples on the corresponding factors.

[0106] Confidence level The value is set to 0.90 to 0.99, with 0.95 used when historical samples are missing. The effective number of frames M for prediction is set to 3 to 6, with 4 used when historical samples are missing. (The text also mentions a reliable distribution temperature coefficient.) The value ranges from 0.05 to 0.30, with 0.15 used when historical samples are lacking. This is the weight for the adjacent frame fluctuation penalty. Values ​​between 0 and 1, with 0.30 used when historical samples are unavailable. Bias expansion coefficient. obsolescence update coefficient And confidence difference update coefficient All values ​​are between 0 and 1, with a value of 0.50 when historical samples are unavailable. Sensitivity normalization threshold. The threshold for obsolescence is 0.10 when historical samples are missing. Use a value between 0.35 and 0.60, and 0.45 if historical samples are unavailable. Confidence difference threshold. Use values ​​between 0.10 and 0.25, and 0.15 if historical samples are unavailable.

[0107] Minimum safety margin The safety distance threshold is obtained by adding the upper limit of the gripper positioning error and the gripping safety margin. Take the mechanical safety distance; allowable deviation of the tag normal angle in the reading direction. Calibration is based on the reader's installation angle and allowable reading range; directional deviation threshold. The calibration is based on the allowable deviation of the reading posture, the allowable deviation of the clamping side selection, and the minimum safe distance between the gripper contact area and the tag area.

[0108] The above parameters are stored in the sorting control equipment and are updated according to the barcode reader installation angle, gripper specifications, conveying speed and robot control cycle.

[0109] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A package image recognition method for sorting robots, characterized in that, include: S1, acquire the package image sequence and package processing association information of the target package; wherein, the package processing association information includes image acquisition time information, preset code reading direction and execution prediction information, and the execution prediction information includes the arrival prediction information of the target package and the end motion trajectory information of the end effector of the sorting robot; S2, based on the package image sequence and image acquisition time information, determine the target image acquisition time, and determine the package label area, the position information of the package label area, and the orientation state of the package label surface at the target image acquisition time; S3, determine the attitude evolution of the target package based on the package image sequence; based on the package label orientation state, attitude evolution, and execution prediction information at the time of target image acquisition, determine the package execution position, the expected execution time of the sorting robot, the temporal confidence region of the orientation state, the predicted orientation state of the package label, and the orientation state staleness of the target package. S4. Based on the predicted orientation state of the package label, the age of the orientation state, the location information of the package label area, the preset reading direction, and the end motion trajectory information, determine the target candidate flip axis, the pre-selected direction adjustment action, and the flip axis confidence difference. S5, based on the location information of the package label area and the end motion trajectory information, performs pre-action visual confirmation on the target package before the end effector of the sorting robot reaches the package execution position, and determines the final direction control action or triggers dynamic control action based on the pre-action visual confirmation.

2. The package image recognition method for sorting robots according to claim 1, characterized in that, The orientation status of the package label includes the normal direction of the package label bearing surface where the package label area is located, the main axis direction of the target package, the position information of the package label area, the orientation relationship of the package label bearing surface relative to the preset reading direction, and the confidence level of the orientation status observation.

3. The package image recognition method for sorting robots according to claim 2, characterized in that, The orientation state temporal confidence domain is the temporal confidence domain formed by the propagation of the orientation state of the package label surface from the target image acquisition time to the expected execution time of the sorting robot. The orientation state temporal confidence domain includes the predicted orientation state of the package label surface at the expected execution time of the sorting robot, the orientation state propagation uncertainty, and the effective envelope of the orientation state.

4. The package image recognition method for sorting robots according to claim 3, characterized in that, The staleness of the orientation state is determined as follows: the orientation state propagation deviation is determined based on the orientation state of the package label surface at the time of target image acquisition and the predicted orientation state of the package label surface; the temporal confidence domain expansion is determined based on the orientation state propagation uncertainty and the effective envelope of the orientation state; the attitude evolution residual factor is determined based on the attitude evolution amount; and the temporal interval decay factor is determined based on the temporal interval between the time of target image acquisition and the expected execution time of the sorting robot. The aging of orientation states is determined based on orientation state propagation bias, temporal confidence domain expansion, attitude evolution residual factor, and temporal interval decay factor.

5. A package image recognition method for sorting robots according to claim 1, characterized in that, Based on the predicted orientation state of the package label, the age of the orientation state, the location information of the package label area, the preset reading direction, and the end motion trajectory information, a set of candidate orientation control actions and a set of candidate flip axes are generated, and the reliable distribution of candidate flip axes is determined based on the set of candidate orientation control actions and the set of candidate flip axes. Based on the confidence distribution of candidate flip axes, the target candidate flip axis is determined, the pre-selected direction control action is determined from the set of candidate direction control actions, and the confidence difference of the flip axis is determined.

6. A package image recognition method for sorting robots according to claim 5, characterized in that, The candidate direction control action set includes candidate flipping actions, candidate clamping side selection actions corresponding to the candidate flipping actions, and candidate release actions. The candidate flipping actions correspond to different candidate flipping axes in the candidate flipping axis set.

7. A package image recognition method for sorting robots according to claim 5, characterized in that, The candidate flip axis confidence distribution is formed as follows: for each candidate flip axis, an axial motion evaluation vector is constructed based on the image frames in the wrapped image sequence, and the confidence of the candidate flip axis in the corresponding image frame is determined according to the axial motion evaluation vector. For adjacent image frames in the packaged image sequence, the confidence level of the candidate flip axis under the same candidate flip axis in the adjacent image frames is obtained respectively, and the confidence level difference of coaxial adjacent frames is determined based on the difference in the confidence level of the candidate flip axis under the adjacent image frames. A candidate flip axis confidence distribution is formed based on the confidence levels of multiple candidate flip axes and the confidence differences between coaxial adjacent frames.

8. A package image recognition method for sorting robots according to claim 7, characterized in that, The axial motion evaluation vector includes the code reading orientation benefit factor, the grasping reachability margin factor, the tag occlusion penalty factor, the trajectory interference penalty factor, the temporal confidence domain sensitivity factor, and the obsolescence penalty factor.

9. A package image recognition method for sorting robots according to any one of claims 1-8, characterized in that, In S5, based on the location information of the package label area and the end motion trajectory information, a pre-action confirmation image is collected before the sorting robot's end effector reaches the package execution position. The orientation state of the wrapped label surface before the action is determined based on the pre-action confirmation image, and the orientation state of the wrapped label surface before the action is compared with the predicted orientation state of the wrapped label surface to obtain the orientation deviation before the action. The orientation state temporal confidence domain is corrected based on the orientation deviation before the action, and the orientation state staleness and flip axis confidence error are updated according to the corrected orientation state temporal confidence domain to obtain the updated orientation state staleness and the updated flip axis confidence error.

10. A package image recognition method for sorting robots according to claim 9, characterized in that, Based on the updated orientation status staleness, the updated flip axis confidence error, the orientation deviation before the action, and the preset verification conditions, determine the final orientation control action or trigger the dynamic control action. When determining the final direction control action, generate the flip axis selection command, clamping side selection command, or release command corresponding to the final direction control action.

Images