A robot target occlusion state recognition method based on deep learning

Abstract

The application discloses a robot target occlusion state recognition method based on deep learning, and particularly relates to the technical field of robot vision recognition, and is used for solving the problem that the occlusion state recognition is unstable when the target is occluded in the existing robot grabbing and there is lack of retest write basis. The color depth image of the current grabbing area is collected, the target category, the approaching direction of the clamping jaw, the opening width of the clamping jaw and the last round of occlusion state version are read, the current grabbing recognition frame is generated, the current occlusion state is locked in combination with the target visible mask, the candidate complete contour, the adjacent object boundary, the depth mutation boundary and the occupation relationship, and the retest recognition label is generated, so that the effect of continuously associating and judging the target visible condition, the occlusion boundary, the clamping jaw approaching blocking condition and the historical occlusion state is achieved, and the reliability of the occlusion state selection before the robot grabbing is improved.

Description

Technical Field

[0001] This invention relates to the field of robot vision recognition technology, specifically a deep learning-based method for recognizing the occlusion state of a target being grasped by a robot. Background Technology

[0002] With the increasing application of industrial robots in scenarios such as bin loading, pallet sorting, and conveyor belt grasping, visual recognition methods based on color images, depth images, or color depth images have been widely used to determine the position, contour, and grasping posture of target workpieces. Existing robot grasping systems typically first acquire images of the grasping area using a vision camera, then use image segmentation, object detection, or deep learning recognition models to identify the target area, and provide the robot control system with grasping criteria based on the identified target position. In scenarios where objects are regularly arranged or their complete shape is clearly visible, the above methods can meet conventional grasping and recognition needs.

[0003] However, in actual production environments where bins are stacked haphazardly, adjacent workpieces overlap, and targets are partially occluded, the visible area of ​​the target surface, the boundaries of neighboring objects, and the boundaries of abrupt depth changes often coexist. Relying solely on the target detection box, the segmented target region, or the results of a single identification attempt makes it difficult to distinguish between different situations, such as incomplete target edges, coverage by neighboring objects, interference from abrupt depth changes, and obstructed access to the gripper. Existing methods focus more on whether the target is identified and its approximate location, lacking a unified process for determining occlusion that correlates the target's visibility, candidate complete outline, boundaries of neighboring objects, boundaries of abrupt depth changes, and gripping constraints such as the gripper's approach direction and gripper opening width. This can easily lead to situations where the target is identified but the actual contact surface is occluded, or the gripper is still considered graspable despite obstacles in the access channel.

[0004] Furthermore, existing grasping and recognition processes typically use the recognition result of the current image frame as a one-time judgment basis. Feedback information such as subsequent retest images, gripper closure, and contact trigger markers lacks a stable version connection with the next round of occlusion status. When the previous round of recognition results cannot be continuously correlated with subsequent changes in the field, the system struggles to determine whether occlusion has lessened, whether retesting is still necessary, or whether the original occlusion state should be maintained, leading to frequent fluctuations in the occlusion state during continuous grasping. Therefore, existing technologies still suffer from unstable occlusion state recognition when the target is occluded and lack sufficient data for retesting and data feedback. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides a deep learning-based method for recognizing the occlusion state of a robot grasping target, thereby solving the problems mentioned in the background section.

[0006] To achieve the above objectives, the present invention provides the following technical solution: a deep learning-based method for recognizing the occlusion state of a robot grasping target, comprising: S1. Acquire a color depth image of the current grasping area, read the target category, gripper approach direction, gripper opening width, and previous occlusion state version, and generate the current grasping recognition frame. S2. Input the current captured and recognized frame into the deep learning recognition model to generate target visible mask, candidate complete contour, neighboring object boundary, and depth abrupt boundary; S3. Based on the difference between the target's visible mask and the candidate complete outline, and combined with the boundaries of nearby objects and depth abrupt change boundaries, determine the suspected occlusion boundaries. S4. Generate a gripper approach channel according to the gripper approach direction and gripper opening width, and calculate the occupancy relationship between the suspected occupancy boundary and the gripper approach channel and the target contact surface. S5. Based on the target visibility ratio formed by the target visibility mask and candidate complete contours, suspected occlusion boundaries, occupancy relationships, and the previous round of occlusion status version, lock the current occlusion status and generate a retest recognition label. S6. Collect the retested color depth image after execution, read the gripper closure amount and contact trigger mark, and write down the next round of occlusion status version based on the changes in the visible area of ​​the target and the changes in the suspected occlusion boundary.

[0007] Furthermore, S1 includes:

[0008] The current grasping area is determined according to the fixed image area within the camera calibration coordinate system, and the effective boundary of this round is defined by the batch number of this round of tasks, the station positioning signal, and the stopping position of the material box;

[0009] Color depth images are formed by pairing color images and depth images acquired at the same time, and aligned to the same image coordinate scale according to the camera calibration version;

[0010] The frame-by-frame record of the current capture and recognition frame includes the color image address, depth image address, target category, gripper approach direction, gripper opening width, and the previous round of occlusion state version.

[0011] Furthermore, S2 includes:

[0012] The deep learning recognition model is formed by solidifying color depth images of similar grasping regions, target category records, and manually labeled records. The manually labeled records correspond to the target visible mask, candidate complete contour, boundaries of neighboring objects, and depth abrupt boundary.

[0013] The model version is bound to the target category table, camera calibration version, and model parameter version. The model parameter version records the proximity statistical range, depth verification range, depth mutation threshold, and target category shape constraints.

[0014] Furthermore, S3 includes:

[0015] The visual computing unit checks that the target visible mask, candidate complete contour, boundary of neighboring objects and boundary of depth abrupt change are all bound to the same current capture and recognition frame version number, model version number and model parameter version number;

[0016] Using the outer edge range of the target candidate corresponding to the complete outline of the candidate as the outer boundary, and the area occupied by the target visible mask as the seen area, a difference region is formed that is not covered by the target visible mask and is connected to the outer edge of the target visible mask.

[0017] Furthermore, S3 also includes:

[0018] The visual computing unit selects the boundaries of neighboring objects within the neighbor statistics range recorded in the model parameter version located at the outer edge of the difference region, and selects the depth abrupt change boundaries within the depth verification range recorded in the model parameter version located at the outer edge of the difference region.

[0019] When the same continuous boundary segment has both neighboring object boundary relationships and depth abrupt boundary relationships, a suspected occlusion boundary is formed.

[0020] Furthermore, S4 includes:

[0021] The visual computing unit organizes the approach direction of the gripper, the opening width of the gripper, and the suspected occlusion boundary under the same current grasping and recognition frame version number into the same coordinate system.

[0022] The channel centerline is determined based on the approach direction of the grippers, and the channel width is determined based on the gripper opening width and the gripper parameter version corresponding to the current target category.

[0023] The outer edge region of the target that falls within the projection range of the inner contact surface of the two fingers of the gripper in the candidate complete contour is determined as the target contact surface.

[0024] Furthermore, S4 also includes:

[0025] The visual computing unit aligns the suspected occlusion boundary with the gripper approach channel and the target contact surface to the same coordinate scale.

[0026] Using the range of the gripper approaching the channel and the range of the target contact surface as the statistical range, record the continuous boundary segments that the suspected occupancy boundary falls into each statistical range, and form the occupied length, occupied area and associated suspected occupancy boundary number.

[0027] Furthermore, S5 includes:

[0028] The target visibility ratio is the percentage of the target visibility mask coverage area relative to the outer edge range of the corresponding target candidate in the complete candidate contour;

[0029] The visual computing unit checks the target visibility ratio, suspected occlusion boundaries, occupancy relationships, and the previous round of occlusion status, all of which are bound to the same current capture and recognition frame version number.

[0030] According to the occupancy boundary of the target contact surface, the occupancy boundary of the gripper approach channel, the existence conditions of the suspected occupancy boundary, and the unoccupancy confirmation boundary recorded in the occupancy status rule version, the current occupancy status is locked, and a retest identification label is generated.

[0031] Furthermore, S6 includes:

[0032] After the completion signal arrives and the gripper stop signal is generated, the vision controller acquires a remeasured color depth image and reads the gripper closure amount and contact trigger mark.

[0033] The visual computing unit compares the target visible area before execution with the target visible area after retesting, corresponding to the target visible mask, to form the change in the target visible area;

[0034] Compare the suspected occlusion boundary before execution with the remeasured suspected occlusion boundary obtained by forming the same suspected occlusion boundary caliber to form the change of suspected occlusion boundary;

[0035] Write the next round of occlusion state version according to the occlusion state writeback rule version.

[0036] Compared with the prior art, the present invention has the following beneficial effects: 1. By acquiring color depth images of the current grasping area, the target category, gripper approach direction, gripper opening width, and previous occlusion state version are read to generate the current grasping recognition frame. Based on the target visibility mask, candidate complete contours, adjacent object boundaries, and depth abrupt boundary, suspected occlusion boundaries are determined. Then, the occupancy relationship between the suspected occupancy boundaries and the gripper approach channel and target contact surface is combined to lock the current occupancy state and generate a retest recognition label. This achieves the effect of continuously associating and judging the target visibility, occlusion boundaries, gripper approach obstruction, and historical occupancy states, avoiding the problem of unstable occupancy state judgment caused by relying solely on single-frame image independent recognition, and improving the reliability of the robot's selection of subsequent processing states such as direct grasping, edge retesting, channel retesting, or contact surface retesting before grasping. 2. By collecting and retesting color depth images, gripper closure amount, and contact trigger markers after execution, and writing down the next round of occlusion status version based on changes in the target's visible area and suspected occlusion boundary changes, the system achieves a closed-loop connection between the current round of visual recognition results, execution feedback results, and the next round of occlusion status version. This allows the occlusion status to be continuously updated and traced with the retest results, avoiding the problem that the previous round of occlusion judgment could not be corrected for changes in the field after execution, and improving the inheritability and version consistency of occlusion recognition results in continuous grasping operations. Attached Figure Description

[0037] Figure 1 A schematic diagram of the overall process for a robot to identify the occlusion state of a target being grasped;

[0038] Figure 2 This is a schematic diagram illustrating the generation of the currently captured and identified frames and the aggregation of field sources;

[0039] Figure 3 A schematic diagram illustrating the formation of differential regions and the determination of suspected occlusion boundaries;

[0040] Figure 4 A schematic diagram illustrating the generation and version binding of deep learning recognition model results;

[0041] Figure 5 A schematic diagram is created to show the approach channel of the gripper, the contact surface with the target, and the occupancy relationship.

[0042] Figure 6 A schematic diagram illustrating the current occlusion state locking and retesting of the identification label generation;

[0043] Figure 7 This diagram illustrates the data collection, change comparison, and write-back of the next round of occlusion status. Detailed Implementation

[0044] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0045] Example: Combined with Appendix Figures 1-7 This embodiment provides a deep learning-based method for recognizing the occlusion state of a target during robot grasping, including:

[0046] S1. Acquire a color depth image of the current grasping area, read the target category, gripper approach direction, gripper opening width, and previous occlusion state version, and generate the current grasping recognition frame. The specific implementation is as follows:

[0047] When the gripping station enters the current recognition cycle, the vision controller captures an image of the current gripping area. The current gripping area is determined according to the fixed image area in the camera calibration coordinate system, and the effective boundary of this cycle is defined by the task batch number, station positioning signal, and material bin dwell position. This area corresponds to the upper surface of the material bin, the bearing surface of the loading pallet, or the dwell area of ​​the sorting conveyor belt that the robot end gripper can observe before the candidate gripping in this cycle. The execution subject is the vision controller, and the image source is a color depth camera installed above or to the side of the gripping station. The acquisition interval starts after the station positioning signal arrives and ends when the camera completes the submission of the color image and depth image at the same acquisition time.

[0048] A color depth image is formed by pairing a color image and a depth image. The color image records the surface texture, color, and outline of the target within the grasping area, while the depth image records the distance from the camera to measurable points within the grasping area, in millimeters. The vision controller aligns the color image and depth image to the same image coordinate scale according to the currently active camera calibration version. First, it removes depth points marked as invalid by the camera. Then, it performs consistency processing on individual isolated transition points according to the surrounding continuous depth records. Finally, it retains the continuous depth changes at the target edge and does not fill in the occluded areas as valid depth.

[0049] If the color image is missing, the depth image is missing, the camera calibration version is missing, or the time difference between the acquisition of the color image and the depth image exceeds the submission window, stop generating the current capture and recognition frame, and set the image record of this round to a pending verification state.

[0050] After completing the color depth image acquisition, the vision controller reads the target category, which is the target workpiece category name or category number specified in this round of grasping task, with the priority source being the grasping task sheet for this round. If the target category is not recorded in the grasping task sheet for this round, the current station's changeover confirmation record is read. If the grasping task sheet for this round and the current station's changeover confirmation record are inconsistent, the record for this round is set to a pending record status, and the target category is not temporarily rewritten based on the current color depth image.

[0051] Then, the approach direction of the gripper is read. The approach direction of the gripper is the approach axis direction recorded in the workstation coordinate system when the gripper approaches the target in this round. It is derived from the candidate approach posture in the robot grasping task and is written into the record in this round along with the reading time and direction reference. In this step, this field is only used as a frame field of the current grasping recognition frame and is not used to generate the grasping path in this step.

[0052] Next, the gripper opening width is read. The gripper opening width is the set distance between the inner contact surfaces of the two grippers in this round before closing, in millimeters, and is derived from the current task setting value of the gripper controller. This value also needs to be associated with the gripper parameter version corresponding to the current target category. If the value is empty, exceeds the allowed range recorded in the gripper parameter version, or the reading time is earlier than the effective time of this round of task batch, the record of this round is set to pending recording, and the gripper opening width of the previous batch is not used as a substitute.

[0053] Then read the previous round of occlusion state version. The previous round of occlusion state version is the occlusion state record version formed after the previous round of recognition for the same grasping area and the same target category. It comes from the occlusion state record area and is associated according to the grasping area identifier, target category and task batch number. If there are multiple associatable versions, read the version that was formed most recently and has not been invalidated. If there are no associatable versions, only write the initial version tag. The initial version tag includes the initial version number and the tag that the previous round of occlusion state was not formed. Do not fill in the previous round state name.

[0054] After the above information is organized in the order of color depth image, target category, gripper approach direction, gripper opening width, and previous occlusion state version, the vision controller generates the current grasping and recognition frame. The current grasping and recognition frame is a frame record of the current occlusion state recognition. The image itself is stored in the image buffer area, and the frame record is stored in the workstation task record area. The frame record includes the current grasping and recognition frame version number, color image address, depth image address, target category, gripper approach direction, gripper opening width, previous occlusion state version, grasping area identifier, task batch number, acquisition time, camera calibration version, and field source marker.

[0055] The target category, gripper approach direction, gripper opening width, and previous occlusion state version must not be automatically filled in; if any of the above fields are missing, the current record is set to pending completion.

[0056] Once the current capture and recognition frame is generated, it is submitted to the next stage for invocation. Only one current capture and recognition frame is formed for the same task batch number, the same capture area identifier, and the same acquisition time. When submitted repeatedly, only the existing version is resent, no new version is generated, and the submitted version is not overwritten. When communication is briefly interrupted, the vision controller caches the current capture and recognition frames in the order of acquisition time, and submits them in the original order after recovery.

[0057] This step applies to robot gripping production lines equipped with fixed-station cameras that can read gripper task parameters and have target category task records. When personnel, order, or customer information is involved, only the task batch number, equipment identifier, and field source marker are retained, and the operator's identity plaintext and sensitive fields of the production order are not recorded.

[0058] During on-site inspection, no less than 50 currently captured and recognized frames are extracted from the continuous production batches. The consistency rates of color image and depth image pairing, target category source, gripper opening width reading, and previous round occlusion state version association are checked. If any of these values ​​are lower than the on-site set threshold, the submission of new frames to the next stage is paused. The camera calibration version, task list category, and gripper parameter version are also checked.

[0059] Preferably, in the automotive connector bin loading station, the color image resolution can be set to 1280×720, the depth distance range can be set to 200 mm to 1200 mm, the time difference between color image and depth image acquisition does not exceed 20 milliseconds, the gripper opening width can be set to 45 mm, the allowable reading deviation does not exceed 0.5 mm, and the time from the arrival of the station positioning signal to the completion of submission of the current grasping and recognition frame does not exceed 200 milliseconds; for example, if the target category of a certain batch is metal buckle, the gripper approach direction is from top to bottom and slightly forward in the station coordinate system, the previous round of occlusion state version is version 37, and the vision controller forms version 38 of the current grasping and recognition frame in one cycle and submits it to the next stage.

[0060] As an alternative approach, the color depth image can be formed by aligning separately installed color and depth cameras with the same camera calibration version. The gripper opening width can be read by the gripper controller's real-time position feedback, as long as the final current grasping and recognition frame still contains the color depth image, target category, gripper approach direction, gripper opening width, and the previous round of occlusion state version, and the source of each field can be traced.

[0061] S2. Input the current captured and recognized frame into the deep learning recognition model to generate the target visibility mask, candidate complete contour, neighboring object boundaries, and depth abrupt boundary. Specifically, this is implemented as follows:

[0062] After the vision controller obtains the current grasping and recognition frame, it sends the frame to the currently active deep learning recognition model. The deep learning recognition model is deployed in the vision computing unit of the grasping station. Before production, it is solidified by color depth images of the same grasping area, target category records, and manual annotation records. The manual annotation records correspond to at least four types of results: target visibility mask, candidate complete contour, neighboring object boundary, and depth abrupt boundary. This allows the same model version to generate the above four results in groups under the same current grasping and recognition frame. The model version is bound to the target category table, camera calibration version, and model parameter version. The model parameter version records the neighbor statistical range, depth abrupt threshold, target category shape constraint, and result submission deadline. Only one active model version and one active model parameter version are called under the same current grasping and recognition frame.

[0063] The execution period begins when the current capture and recognition frame is received and ends when the target visible mask, candidate complete contour, neighboring object boundary, and depth abrupt boundary are recorded with each frame. This step generates the above four results based only on the color depth image, target category, gripper approach direction, gripper opening width, and the previous round of occlusion state version in the current capture and recognition frame. Suspected occlusion boundaries are not determined in this step.

[0064] Before the current captured and recognized frame enters the deep learning recognition model, the visual computing unit first checks whether the current captured and recognized frame version number, color image address, depth image address, target category, camera calibration version, previous occlusion state version, model version, and model parameter version are complete. Then, it reads the color image and depth image according to the fixed image coordinates of the current captured and recognized frame. During this reading process, it only performs consistency maintenance on the image size, image coordinates, and effective depth point markings according to the camera calibration version. It does not supplement the target category, gripper approach direction, gripper opening width, previous occlusion state version, or fill unmeasurable occlusion areas as the target surface.

[0065] If the current captured and recognized frame field is missing, the image address cannot be read, the model version does not cover the current target category, the model version does not match the camera calibration version, or the model parameter version is not effective, the visual computing unit stops generating the four results for this round, marks the current captured and recognized frame as pending verification, and retains the reason for non-recognition, the current captured and recognized frame version number, the model version number, and the model parameter version number.

[0066] After receiving the current capture and recognition frame, the deep learning recognition model first generates a target visibility mask. The target visibility mask is the set of pixel coordinates of the target surface that is identified as the target category in the current color depth image and can be directly observed from the camera's perspective. The recording aperture is the pixel position and the corresponding effective depth position in the image coordinate system. Only pixels that simultaneously meet the target category record and the effective depth correspondence are accepted. Invalid depth points of the camera, isolated points with specular highlights, fixed edges of the hopper, fixed edges of the pallet, and images of the gripper body are not accepted.

[0067] Subsequently, candidate complete contours are generated. The candidate complete contours are target outer edge candidate records formed by the deep learning recognition model based on the category shape constraints corresponding to the current target category, the outer edge of the target's visible mask, the color depth image, and the previous round of occlusion state version. The record scope is the set of closed or unclosed line segments corresponding to the outer edge of the same target in the current view. The candidate complete contours do not change the target category, do not replace the target's visible mask, and do not determine the occlusion reason in this step. When the currently visible part is insufficient to support a closed outer edge, the deep learning recognition model retains the set of unclosed line segments and the incomplete contour markers, and does not force the completion of a closed contour.

[0068] Regenerate the boundaries of neighboring objects; the boundaries of neighboring objects are the outer edge segments of non-target objects located outside the outer edge of the target visible mask in the current color depth image, falling within the neighbor statistical range recorded in the model parameter version, and identified as non-target objects. The recording scope is the boundary pixel chain and corresponding depth position in the image coordinate system. When forming, only the outer edges of non-target objects that have a spatial adjacent relationship with the current target visible mask are recorded. The images of the bin wall, pallet fixed edge, and gripper body are not included in the boundaries of neighboring objects. When there are multiple adjacent non-target objects, the visual computing unit records each boundary segment according to the spatial connectivity relationship within the same current grasping and recognition frame.

[0069] Finally, a depth mutation boundary is generated. The depth mutation boundary is a continuous boundary record formed after the distance change between adjacent effective measurement points in the current depth image reaches the depth mutation threshold recorded in the model parameter version. The record size is the boundary pixel chain, the effective depth interval on both sides of the boundary and the formation time, and the depth unit is millimeters. When forming the depth mutation boundary, the visual computing unit first excludes invalid depth points of the camera, and then retains the continuous change formed by the height difference of the object, the stacked contact edge and the occlusion leading edge, without generating an independent boundary for a single noise point.

[0070] The target visible mask, candidate complete contour, neighboring object boundary, and depth abrupt boundary must be grouped and bound according to the current capture and recognition frame version number and stored in the visual recognition result recording area. The recorded content includes the current capture and recognition frame version number, model version number, model parameter version number, target visible mask address, candidate complete contour address, neighboring object boundary address, depth abrupt boundary address, formation time, and field source marker. If any item cannot be formed, a complete recognition result group is not generated, the previous frame result is not used to fill in the missing items, and the current capture and recognition frame is transferred to the pending verification stage.

[0071] Only one set of recognition results is generated for the same current capture and recognition frame version number, the same model version number, and the same model parameter version number. When submitting repeatedly, only the existing record is returned, no new record is generated, and the record already submitted to the next stage is not overwritten.

[0072] After the current step is completed, the next step calls the above 4 recognition results by the current capture and recognition frame version number; if the visual computing unit occupies resources for a short time, it will be queued and executed according to the time when the current capture and recognition frame is formed. Records that exceed the submission time limit of this workstation will not be overwritten by subsequent records, but will be transferred to the pending verification along with the original current capture and recognition frame version number.

[0073] This step applies to visual recognition devices with color depth images and target category task records in scenarios such as disordered bin sorting, pallet loading, and conveyor belt paused grasping. When production batch information is involved, only the current grasping and recognition frame version number, task batch number, and device identifier are retained, and the operator's identity is not recorded in plaintext.

[0074] During on-site spot checks, no fewer than 50 sets of recognition results are randomly selected from continuous production batches. The checks are made to verify whether the visible mask of the target only covers the visible target surface, whether the candidate complete contour corresponds to the same target category, whether the boundary of adjacent objects excludes the fixed edge of the hopper, and whether the boundary of depth abrupt change excludes isolated noise points. If any of these items are lower than the on-site set threshold, the model version is suspended from submitting results at this workstation, and the model version, model parameter version, camera calibration version, and the source of the currently captured recognition frame are checked.

[0075] Preferably, in the automotive connector bin loading station, the resolution of the color image corresponding to the current grasping and recognition frame can be set to 1280×720, the depth distance range can be set to 200 mm to 1200 mm, the depth abrupt change threshold can be set to 8 mm, the proximity statistics range can be set to 30 pixels beyond the outer edge of the target visible mask, and the formation time of a single set of recognition results does not exceed 150 milliseconds; for example, if the version number of a certain current grasping and recognition frame is version 38, the target category is metal buckle, the model version is metal buckle bin recognition version 6, the model parameter version is version 6 parameters, the visual computing unit generates 1 target visible mask, 1 candidate complete contour, 3 segments of the neighboring object boundary, and 5 segments of the depth abrupt change boundary, and binds the 4 results with the current grasping and recognition frame version 38.

[0076] As an alternative approach, deep learning recognition models can be deployed on edge computing devices at workstations or on production line vision servers. Target visibility masks, candidate complete contours, boundaries of neighboring objects, and depth abrupt boundary can be stored as pixel maps, contour point sets, or boundary line segment files, as long as all four results are generated from the current captured recognition frame using the same model version and the same model parameter version, and can be traced through the version number of the current captured recognition frame.

[0077] S3. Based on the difference region between the target's visible mask and the candidate complete outline, combined with the boundaries of nearby objects and depth abrupt change boundaries, determine the suspected occlusion boundaries. Specifically, this is implemented as follows:

[0078] After the visual computing unit obtains the target visible mask, candidate complete contour, neighboring object boundary, and depth abrupt boundary of the group records under the same current capture and recognition frame version number, it first checks whether the four records all come from the same current capture and recognition frame version number, the same model version number, and the same model parameter version number. This action occurs within the visual computing unit of the capture station, and the effective period starts from the completion of the four recognition results in the previous step and ends when the suspected occlusion boundary record is formed in this step.

[0079] The target visible mask uses the set of target visible surface pixel coordinates formed in the previous step, and the candidate complete contour uses the same target outer edge candidate record formed in the previous step. Both are based on the image coordinate system of the current capture and recognition frame. If any of the target visible mask address, candidate complete contour address, neighboring object boundary address, or depth change boundary address cannot be read, or if the version numbers of the four records are inconsistent, the visual computing unit stops determining the suspected occlusion boundary, transfers the current capture and recognition frame to the pending verification stage, and does not use the previous frame record to complete the record.

[0080] This step first organizes the differences between the target visible mask and the candidate complete contour. The difference region is the pixel region within the outer edge of the target candidate corresponding to the candidate complete contour that is not covered by the target visible mask and is connected to the outer edge of the target visible mask. Its physical meaning is the part that is within the outer edge of the target candidate under the current view but is not directly observed as the target surface. The recording aperture is the set of region pixels and the corresponding depth validity mark in the image coordinate system.

[0081] When the candidate complete contour is a closed line segment, the outer edge range of the target candidate is determined according to the area enclosed by the closed line segment; when the candidate complete contour is a set of non-closed line segments, the outer edge range of the target candidate is determined according to the target category shape constraint and the order of the endpoints of the non-closed line segments recorded in the current model parameter version. The determined outer edge range of the target candidate is only used as the boundary for sorting out the difference area and is not used as the filling content of the target visible mask.

[0082] When forming the difference region, the visual computing unit first aligns the target visible mask and the candidate complete contour to the same image coordinate scale. Then, it takes the target candidate outer edge range corresponding to the candidate complete contour as the outer boundary and the area occupied by the target visible mask as the seen area, and retains the uncovered part that connects to the outer edge of the target visible mask. Areas that fall completely outside the target candidate outer edge range are not included in the difference region. Small holes inside that are completely surrounded by the target visible mask and do not touch the outer edge of the target visible mask are not included in the difference region. Isolated fragments with an area lower than the minimum region threshold recorded in the model parameter version are not included in the difference region.

[0083] If the target visible mask is empty, the candidate complete contour is empty, or there is no spatial correspondence between the candidate complete contour and the target visible mask, the visual computing unit will not continue to generate suspected occlusion boundaries and will set the current record to pending verification. If the four recognition results cannot be read, the version numbers of the four recognition results are inconsistent, the target visible mask is empty, the candidate complete contour is empty, or there is no spatial correspondence between the candidate complete contour and the target visible mask, the current captured recognition frame will be transferred to pending verification. In addition to the above situations, if no valid suspected occlusion boundary segment is formed, a suspected occlusion boundary empty record will be generated, and the current captured recognition frame will not be transferred to pending verification.

[0084] Subsequently, the selection is performed by combining the boundaries of nearby objects. The boundaries of nearby objects are recorded using the outer edge line segments of non-target objects formed in the previous step. Their physical meaning is the outline of non-target objects near the outer edge of the visible mask of the target in the current grasping area. The recording caliber is the boundary pixel chain and the corresponding depth position. The visual computing unit only selects the boundaries of nearby objects within the neighboring statistical range that fall into the outer edge of the difference area. The neighboring statistical range is recorded by the current model parameter version and is represented by pixel distance or millimeter distance after workstation calibration.

[0085] The outer edge of the difference region is a continuous boundary segment in the difference region facing the boundary of the adjacent object, excluding the boundary on the side that faces completely into the visible mask of the target.

[0086] Even if the images of the bin wall, pallet fixed edge, and gripper body fall within the neighboring statistical range, they will not be included in this step as the boundary of neighboring objects. When the boundary of a neighboring object crosses multiple differential regions, candidate associated segments are recorded separately according to the corresponding positions of the same boundary pixel chain and the outer edges of each differential region, and they are not merged into a single boundary.

[0087] Then, the depth mutation boundary is used for verification. The depth mutation boundary adopts the continuous depth change boundary formed in the previous step. Its physical meaning is the spatial height boundary formed after the change in the distance between adjacent effective measurement points in the current depth image reaches the threshold recorded in the current model parameter version. The depth unit is millimeters. The visual computing unit only uses the depth mutation boundary segment within the depth verification range that falls into the outer edge of the difference region. The depth verification range is recorded by the current model parameter version and is characterized by pixel distance or millimeter distance after station calibration. There must be effective depth records on both sides of this boundary segment.

[0088] Break edges composed of invalid depth points from the camera, abrupt changes formed by a single noise point, and depth abrupt change boundaries that do not fall within the depth verification range are not included in the determination of suspected occlusion boundaries.

[0089] After completing the above processing, the visual computing unit determines the suspected occlusion boundary. The suspected occlusion boundary is the boundary segment record formed when the same continuous boundary segment in the outer edge of the difference region simultaneously contains the boundary segment of the neighboring object that falls into the neighboring statistical range and the boundary segment of the depth change that falls into the depth verification range. The record includes the boundary pixel chain in the current image coordinate system, the difference region number, the associated neighboring object boundary number, the associated depth change boundary number, the formation time, and the model parameter version number.

[0090] If the outer edge of the difference region only has a boundary segment of a neighboring object but no corresponding boundary segment of a depth change, or only has a boundary segment of a depth change but no corresponding boundary segment of a neighboring object, then no valid suspected occlusion boundary segment is formed, and a suspected occlusion boundary empty record is generated under the same current capture and recognition frame version number; the suspected occlusion boundary empty record only indicates that no suspected occlusion boundary was confirmed in this round and is not considered as a pending verification state; if there are multiple continuous boundary segments that meet the conditions in the same difference region, the visual computing unit forms multiple suspected occlusion boundaries according to the spatial connectivity relationship, and keeps the number of each segment independent.

[0091] Suspected occlusion boundary records are stored in the visual recognition result recording area and are bound to the current capture and recognition frame version number. The minimum record content includes the current capture and recognition frame version number, the address of the difference region, the address of the suspected occlusion boundary, the address of the boundary of the associated neighboring object, the address of the boundary of the associated depth change, the model parameter version number, and the formation time. When an empty record of a suspected occlusion boundary is generated, the address of the suspected occlusion boundary is written to the address of the empty record, the address of the boundary of the associated neighboring object and the address of the boundary of the associated depth change are written to the "no valid association" flag, and the address of the difference region still retains the address of the corresponding difference region record. Only one set of suspected occlusion boundary records is generated for the same current capture and recognition frame version number and the same model parameter version number. When submitted repeatedly, the existing record is returned and the submitted record is not overwritten.

[0092] This step is applicable to the identification process where the target is partially exposed and the outer edge of the target may be covered by nearby objects in scenarios such as disordered sorting of bins, pallet loading, and conveyor belt stoppage grabbing. When the current target is completely invisible, this step does not generate a suspected occlusion boundary, but instead transfers the current grabbing and identification frame to the pending verification stage.

[0093] During on-site spot checks, no fewer than 50 sets of suspected occlusion boundary records can be randomly selected from continuous production batches. Check whether the difference area comes from the same frame difference between the target visible mask and the candidate complete outline, whether the boundaries of adjacent objects exclude fixed tooling edges, whether the depth change boundary has effective depth support, and whether the suspected occlusion boundary is associated with both the boundary of adjacent objects and the depth change boundary. If any of these values ​​are below the on-site set threshold, the submission of suspected occlusion boundary records for this model parameter version at this workstation will be suspended, and the source of the currently captured and identified frame and the versions of the four identification results will be checked back.

[0094] Preferably, in the loading station of the automotive connector bin, the minimum threshold of the difference region can be set to 80 pixels, the neighbor statistics range can be set to 12 pixels beyond the outer edge of the difference region, the depth verification range can be set to 15 pixels beyond the outer edge of the difference region, the depth change threshold can be set to 8 millimeters, and the determination time of a single suspected occlusion boundary does not exceed 80 milliseconds; for example, when the current capture and recognition frame version number is 38, the visual computing unit obtains two difference regions from the target visible mask of the metal buckle and the candidate complete contour. One difference region has both the outer edge boundary of the nut and the depth change boundary at its outer edge, forming a suspected occlusion boundary segment. The other difference region only has a depth change boundary and no neighboring object boundary. This difference region does not form an effective suspected occlusion boundary segment, and only the difference region record is retained.

[0095] As an alternative approach, suspected occlusion boundaries can be stored using boundary pixel chains, polyline point sets, or boundary band files with width; the proximity statistical range and depth verification range can also be recorded using millimeter distances after workstation calibration, as long as the suspected occlusion boundaries are still determined by the difference area between the target visible mask and the candidate complete contour, combined with the boundaries of neighboring objects and depth abrupt boundary, and can be traced through the current capture and recognition frame version number.

[0096] S4. Generate a gripper approach channel according to the gripper approach direction and gripper opening width, and calculate the occupancy relationship between the suspected occupant boundary and the gripper approach channel and the target contact surface. The specific implementation is as follows:

[0097] After obtaining the gripper approach direction, gripper opening width, and suspected occlusion boundary under the same current grasping and recognition frame version number, the visual computing unit organizes the gripper approach direction, gripper opening width, and suspected occlusion boundary into the same coordinate system according to the fixed camera calibration version of the current grasping and recognition frame, and generates a gripper approach channel. This action occurs within the visual computing unit of the grasping station, and the effective period starts from the completion of the recording of the suspected occlusion boundary until the gripper approach channel and occupancy relationship are recorded with each frame.

[0098] Only one coordinate system can be used within the same current grasping and recognition frame. When using image coordinates, the gripper approach channel, suspected occlusion boundary, and target contact surface are all recorded in pixel positions. When using point cloud coordinates, the gripper approach channel, suspected occlusion boundary, and target contact surface are all recorded in millimeter positions. The two systems should not be mixed in the same set of records. When using point cloud coordinates, the boundary points involved in the conversion in the suspected occlusion boundary, gripper approach channel, and target contact surface must all have valid depth records. If the boundary points involved in the conversion lack valid depth records, the visual computing unit will switch to image coordinates to form the record for this step. If the gripper approach channel and occupancy relationship still cannot be formed under image coordinates, the current grasping and recognition frame will be transferred to the pending verification stage.

[0099] The approach direction of the gripper follows the approach axis direction in the workstation coordinate system recorded in the current gripping recognition frame when the gripper approaches the target. Its source is the candidate approach posture in the robot gripping task. The vision computing unit only reads this direction as the direction basis of the gripper approach channel and does not generate a gripping path in this step. The gripper opening width follows the set distance between the inner contact surfaces of the two fingers of the gripper before closing, recorded in the current gripping recognition frame. The unit is millimeters. The vision computing unit converts this width to the coordinate aperture used in this step according to the camera calibration version. The conversion result is only effective within the version number of the current gripping recognition frame.

[0100] If the gripper approach direction is missing, multiple candidate values ​​exist for the gripper approach direction but no unique value is locked in the current grasping and recognition frame, the gripper opening width is missing, the camera calibration version is missing, or the suspected occlusion boundary record cannot be read, the visual computing unit stops generating the gripper approach channel and transfers the current grasping and recognition frame to the pending verification stage. The previous gripper approach direction, the previous batch of gripper opening width, or the previous frame's suspected occlusion boundary are not used to fill in the missing values. If the suspected occlusion boundary record can be read but the number of valid suspected occlusion boundary segments is 0, the visual computing unit still generates the gripper approach channel according to the gripper approach direction and gripper opening width, and records the length and area occupied by the suspected occlusion boundary on the gripper approach channel and the target contact surface as 0. If the suspected occlusion boundary record cannot be read or the version number of the suspected occlusion boundary record is inconsistent with the version number of the current grasping and recognition frame, the current grasping and recognition frame is transferred to the pending verification stage.

[0101] This step generates a gripper approach channel according to the gripper approach direction and gripper opening width. The gripper approach channel is a strip-shaped area swept by the inner contact surface of the two fingers and the safety reserved range on both sides when the gripper approaches the target along the gripper approach direction. The physical meaning corresponds to the spatial range that the gripper may pass through before entering the vicinity of the target in this round. The recording caliber includes the channel centerline, channel width, channel front end position, channel rear end position, channel side boundaries and formation time.

[0102] The channel centerline is determined in this step by projection in the coordinate aperture according to the approach direction of the gripper; the channel width is determined by the gripper opening width and the reserved width on both sides as recorded in the current gripper parameter version; the channel front end position is determined by the outer edge of the candidate complete contour facing the approach direction of the gripper; the channel rear end position is determined by extending the channel front end position in the opposite direction of the approach direction of the gripper to the channel length recorded in the current model parameter version.

[0103] When the candidate complete contour is a set of non-closed line segments, the front end position of the channel adopts the target candidate outer edge range determined by the target category shape constraint in the current model parameter version, and this range is not written into the target visibility mask.

[0104] Then, the occupancy relationship between the suspected occupancy boundary and the gripper approach channel is calculated. The suspected occupancy boundary uses the boundary pixel chain, polyline point set or boundary band record formed in the previous step. The occupancy relationship is the spatial occupancy record formed after the suspected occupancy boundary falls into the range of the gripper approach channel. The record includes the suspected occupancy boundary number, the position of the gripper approach channel occupied, the occupancy length, the occupancy area and the time of occupancy formation.

[0105] The visual computing unit aligns the suspected occlusion boundary and the gripper approach channel to the same coordinate system used in this step. Then, it uses the area enclosed by the two sides, front end, and rear end of the gripper approach channel as the statistical range and records the continuous boundary segments of the suspected occlusion boundary that fall within this statistical range. Only the portion of the continuous boundary segment that falls within the gripper approach channel range is counted in the occupancy relationship; boundary segments that are completely outside the gripper approach channel are not counted. The occupancy length is recorded according to the length of the continuous boundary segment that falls within the gripper approach channel range, and the occupancy area is recorded according to the overlapping area of ​​the boundary band and the gripper approach channel. When the suspected occlusion boundary is recorded as a single line without width, the occupancy area is calculated based on the boundary band width recorded in the current model parameter version and is not temporarily specified by the on-site operator.

[0106] Next, the occupancy relationship between the suspected occupancy boundary and the target contact surface is calculated. The target contact surface is the outer edge area of ​​the target that falls within the projection range of the inner contact surface of the two fingers of the gripper in the candidate complete contour. Its physical meaning corresponds to the surface range that is expected to come into contact with the target when the gripper closes in this round. The visual computing unit determines the target contact surface record according to the gripper approach direction, gripper opening width, gripper fingertip contact surface width recorded in the current gripper parameter version, and candidate complete contour. The record caliber includes the target contact surface position, contact surface length, contact surface width, and the current grasping recognition frame version number.

[0107] During the calculation, the visual computing unit aligns the suspected occlusion boundary and the target contact surface to the same coordinate system used in this step, records the continuous boundary segments of the suspected occlusion boundary that fall within the target contact surface, and forms its occupied length, occupied area, and associated suspected occlusion boundary number on the target contact surface; when the suspected occlusion boundary is located outside the target contact surface, it is not included in the occupancy relationship of the target contact surface.

[0108] Upon completion, the gripper approach channel and occupancy relationship are stored in the visual recognition result recording area and bound to the current grasping and recognition frame version number. The minimum record content includes the current grasping and recognition frame version number, gripper approach direction, gripper opening width, gripper approach channel address, target contact surface address, suspected occlusion boundary address, occupancy relationship of the suspected occupancy boundary to the gripper approach channel, occupancy relationship of the suspected occupancy boundary to the target contact surface, camera calibration version, gripper parameter version, model parameter version, and formation time. Only one set of gripper approach channel and occupancy relationship records is formed for the same current grasping and recognition frame version number, the same gripper approach direction, the same gripper opening width, the same gripper parameter version, and the same model parameter version. When submitting repeatedly, the existing record is returned, no new record is generated, and the submitted record is not overwritten.

[0109] This step applies to scenarios where parallel grippers and two-finger grippers are used to grasp parts from hoppers, palletized workpieces, and workpieces that are stopped on a conveyor belt. If the gripper opening width exceeds the range recorded in the gripper parameter version corresponding to the current target category, or if the suspected occupancy boundary record cannot be read, this step will not generate an occupancy relationship, and the current grasping and recognition frame will be transferred to pending verification. If the suspected occupancy boundary record can be read but the number of valid suspected occupancy boundary segments is 0, this step will generate an occupancy relationship record with both the occupancy length and occupancy area being 0.

[0110] During on-site spot checks, no fewer than 50 sets of records can be randomly selected from continuous production batches to verify whether the gripper approach channel is formed according to the gripper approach direction and gripper opening width, whether the target contact surface is located in the outer edge area of ​​the target within the projection range of the inner contact surface of the two gripper fingers in the candidate complete contour, and whether the occupancy relationship of the suspected occupancy boundary to the gripper approach channel and the target contact surface can be traced back to the same suspected occupancy boundary number; if any of these items are lower than the on-site set threshold, the submission of occupancy relationship records at this workstation will be suspended, and the camera calibration version, gripper opening width source, gripper parameter version, model parameter version, and suspected occupancy boundary records will be checked.

[0111] Preferably, in the loading station of the automotive connector bin, the gripper opening width can be set to 45 mm, the channel length can be set to extend 60 mm from the front end of the channel along the gripper approach direction, the reserved width on both sides can be set to 3 mm each, the gripper fingertip contact surface width can be set to 6 mm, the boundary band width recorded in the model parameter version can be set to 2 mm, and the formation time of a single occupancy relationship does not exceed 60 milliseconds; for example, if the current gripping recognition frame version number is 38, the gripper approach direction is from top to bottom and slightly forward in the station coordinate system, the visual computing unit generates a gripper approach channel based on the 45 mm gripper opening width, and a segment of the suspected occupancy boundary enters the front end of the gripper approach channel by 18 mm and falls into the target contact surface by 4 mm, forming the occupancy relationship of the suspected occupancy boundary on the gripper approach channel and the target contact surface.

[0112] As an alternative approach, the gripper approach channel can be stored in a strip region file in image coordinates or a spatial channel point set in point cloud coordinates, and the target contact surface can be stored in a target outer edge line segment or contact surface region file. As long as the gripper approach channel is still generated according to the gripper approach direction and gripper opening width, and the occupancy relationship is still formed by the entry range of the suspected occlusion boundary to the gripper approach channel and the target contact surface, and can be traced through the current grab recognition frame version number.

[0113] S5. Based on the target visibility ratio formed by the target visibility mask and candidate complete contours, suspected occlusion boundaries, occupancy relationships, and the previous round of occlusion state version, lock the current occlusion state and generate a retest recognition label. Specifically, the implementation is as follows:

[0114] After obtaining the target visibility ratio, suspected occlusion boundaries, occupancy relationships, and the previous occlusion status version under the same current capture and recognition frame version number, the visual computing unit locks the current occlusion status according to the currently effective occlusion status rule version and generates a retest recognition label. This action occurs within the visual computing unit of the capture station, and its duration starts from the completion of the occupancy relationship recording and ends when the current occupancy status and the retest recognition label are recorded with the frame.

[0115] The target visibility ratio is obtained by organizing the target visibility mask and the candidate complete contour under the same coordinate system. The recording caliber is the percentage of the target visibility mask coverage area to the target candidate outer edge range corresponding to the candidate complete contour, and the unit is percentage. When the candidate complete contour is a set of non-closed line segments, the target candidate outer edge range adopts the range determined by the target category shape constraint in the current model parameter version. This range is only used as the calculation boundary of the target visibility ratio and is not added to the target visibility mask.

[0116] When the target visibility mask is empty, the candidate complete outline is empty, or the target candidate outer edge range cannot be formed, the target visibility ratio is not generated, and the current captured and identified frame is transferred to pending verification.

[0117] The suspected occlusion boundary uses the boundary segment record formed in the previous step. The record includes the suspected occlusion boundary number, the number of the difference region to which it belongs, the boundary number of the associated neighboring object, and the boundary number of the associated depth change boundary. The occupancy relationship uses the occupancy relationship of the suspected occupancy boundary to the gripper approach channel and the target contact surface formed in the previous step. The record includes the corresponding suspected occupancy boundary number, the name of the occupied object, the occupied length, and the occupied area. The previous round of occupancy state version comes from the current capture and recognition frame. The record includes the previous round version number, the previous round state name, and the formation time. If it is an initial version marker, it is only used as the starting point for the absence of a previous round state and the previous round state name is not presumed.

[0118] The visual computing unit first checks whether the target visibility ratio, suspected occlusion boundaries, occupancy relationships, and the previous round of occlusion state version are all bound to the same current capture and recognition frame version number. If any record cannot be read, the version number is inconsistent, or the target visibility ratio has not been formed, the current occupancy state is stopped from being locked, the current capture and recognition frame is transferred to pending verification, and the previous frame state is not used as a substitute. When the occupancy relationship record can be read and the occupancy length and occupancy area are both 0, it is used as a valid occupancy relationship record to participate in the current occupancy state locking.

[0119] The current occlusion status is locked according to the status order in the occlusion status rule version. The occlusion status rule version records the target visibility ratio boundary, the existence condition of the suspected occlusion boundary, the gripper approach channel occupied boundary, the target contact surface occupied boundary, the boundary buffer range, and the state continuation condition of the previous round. Only one effective occlusion status rule version is used for the same current capture and recognition frame. Wherein, reaching the boundary means greater than or equal to the boundary, and below the boundary means less than the boundary. The boundary buffer range is the interval where the corresponding explicit boundary is not reached but the difference from the explicit boundary falls within the range recorded in the occlusion status rule version. When the target contact surface occupied boundary, the gripper approach channel occupied boundary, or the no-occlusion confirmation boundary has been reached, it does not enter the boundary buffer range. In this step, the existence of a suspected occlusion boundary means that the suspected occlusion boundary record can be read and the number of valid suspected occlusion boundary segments is greater than 0. The absence of a suspected occlusion boundary means that the suspected occlusion boundary record can be read and the number of valid suspected occlusion boundary segments is 0. This situation corresponds to the empty suspected occlusion boundary record formed in the previous step, which is not the same as the suspected occlusion boundary record not being readable.

[0120] The visual computing unit first checks the occupancy relationship between the suspected occupancy boundary and the target contact surface. If the length or area occupied by the target contact surface reaches the target contact surface occupancy boundary as recorded in the occupancy state rule version, the target contact surface is locked as occupancy. If the target contact surface is not locked as occupancy, and the length or area occupied by the suspected occupancy boundary to the gripper approach channel reaches the gripper approach channel occupancy boundary as recorded in the occupancy state rule version, the gripper approach channel is locked as occupancy. If neither of the above two states is locked, and the suspected occupancy boundary exists, the target contact surface occupancy relationship does not reach the target contact surface occupancy boundary, the gripper approach channel occupancy relationship does not reach the gripper approach channel occupancy boundary, and the target visibility ratio is lower than the no-occupancy confirmation boundary, the edge occupancy but channel open state is locked. If neither of the above three states is locked, and the suspected occupancy boundary does not exist, and the target visibility ratio reaches the no-occupancy confirmation boundary, the no-occupancy confirmable state is locked.

[0121] When the visible proportion of the target is within the boundary buffer range recorded in the occlusion state rule version, the visual computing unit reads the previous round of occlusion state version; when the previous round state name belongs to the current adjacent state range and the previous round of occlusion state version can be traced through the current captured and recognized frame, the previous round state name is continued as the current occlusion state; when the previous round of occlusion state version is the initial version mark, the previous round state name cannot be traced, or the previous round state name is inconsistent with the adjacent state range corresponding to the current record, it is locked as the occlusion source pending retesting state.

[0122] In any of the remaining cases where none of the aforementioned locking conditions are met, the source of the occlusion will be locked and await retesting.

[0123] The above state order is a unique locking order. The state that is satisfied first becomes the current occlusion state, and subsequent states will not be judged again. The previous occlusion state version only participates in the continuation judgment within the boundary buffer range and cannot change the current occlusion state when the occupied boundary is clearly reached or the unoccupied confirmation boundary is clearly reached.

[0124] Once the current occlusion state is locked, the visual computing unit generates a retest identification label. The retest identification label is a record of the retest requirements formed with the current occlusion state. The record includes the retest identification label name, the corresponding current occlusion state, the associated suspected occlusion boundary number, the associated occupancy relationship number, the occlusion state rule version, and the time of formation.

[0125] When the current occlusion state is unobstructed and verifiable, a label requiring no retesting is generated; when the current occlusion state is edge-occluded but the channel is open, an edge retesting label is generated; when the current occlusion state is target contact surface occlusion, a contact surface retesting label is generated; when the current occlusion state is gripper approaching the channel and occluding it, a channel retesting label is generated; when the current occlusion state is occlusion source awaiting retesting, a source retesting label is generated. Each current occlusion state corresponds to only one retesting identification label and is not selected temporarily by on-site operators.

[0126] The current occlusion status and retest identification label are stored in the visual recognition result recording area and are bound to the current capture and recognition frame version number. The minimum record content includes the current capture and recognition frame version number, target visibility ratio, suspected occlusion boundary address, occupancy relationship address, previous round occlusion status version, current occlusion status, retest identification label, occlusion status rule version, and formation time. The same current capture and recognition frame version number and the same occlusion status rule version will only form one set of current occlusion status and retest identification label. When submitting repeatedly, the existing record will be returned, no new record will be generated, and the submitted record will not be overwritten.

[0127] This step applies to scenarios such as disordered sorting of bins, pallet loading, and conveyor belt stoppage grabbing where the target has at least a visible mask and an occupancy relationship has been formed; when the current target is completely invisible, the occlusion status rule version is not in effect, or the previous occlusion status version cannot be traced, the current grabbing and recognition frame is transferred to pending verification.

[0128] During on-site spot checks, at least 50 sets of records can be randomly selected from continuous production batches to verify whether the target visibility ratio is formed by the target visibility mask and candidate complete outline in the same frame, whether the suspected occupancy boundary comes from the same frame record, whether the occupancy relationship corresponds to the gripper approach channel and the target contact surface, whether the current occupancy status is locked according to the status order in the occupancy status rule version, and whether the retest identification label uniquely corresponds to the current occupancy status. If any item is lower than the on-site set threshold, the current workstation is suspended from submitting the current occupancy status record, and the occupancy status rule version, occupancy relationship record, and the previous round of occupancy status version are checked.

[0129] Preferably, in the automotive connector bin loading station, the unobstructed confirmation boundary can be set to 85%, and the boundary buffer range can be set to a range lower than the unobstructed confirmation boundary with a difference of no more than 5 percentage points. The target contact surface occupancy length reaches 4 mm to lock the target contact surface occupancy state, and the gripper approach channel occupancy length reaches 10 mm to lock the gripper approach channel occupancy state. The formation time of a single set of current occupancy state and retest identification tag does not exceed 50 milliseconds. For example, when the current grasping and identification frame version number is 38, the target visibility ratio is 68%, there is one suspected occupancy boundary segment, the suspected occupancy boundary occupies 18 mm of the gripper approach channel and 4 mm of the target contact surface, and the previous round of occupancy state version is 37. The visual computing unit locks the target contact surface occupancy state according to the occupancy state rule version and generates a contact surface retest tag.

[0130] As an alternative approach, the current occlusion status can be stored using a status name file, status number record, or status table record. The retest identification label can be stored using a label name or label number. As long as the current occlusion status is still locked based on the target visibility ratio, suspected occlusion boundary, occupancy relationship, and the previous round of occlusion status version, the retest identification label will still be generated with the current occlusion status and can be traced through the version number of the currently captured identification frame.

[0131] S6. Acquire the retested color depth image after execution, read the gripper closure amount and contact trigger mark, and write down the next round of occlusion status version based on changes in the target's visible area and suspected occlusion boundary changes. The specific implementation is as follows:

[0132] After the robot grasping execution system completes this round of execution and issues an execution completion signal, the vision controller acquires a re-measured color depth image at the same grasping station. The acquisition interval starts from the arrival of the execution completion signal and the formation of the gripper stop signal, and ends when the re-measured color image and re-measured depth image are submitted in pairs at the same acquisition time. The re-measured color depth image uses the current grasping area and camera calibration version corresponding to the current grasping recognition frame. The vision controller first aligns the re-measured color image and re-measured depth image to the same coordinate scale, and then removes depth points marked as invalid by the camera and single isolated jump points. It does not fill in the occluded areas that are still unmeasurable after execution as the target surface.

[0133] If the retested color image is missing, the retested depth image is missing, the correspondence between the retest acquisition time and the execution completion signal cannot be confirmed, or the camera calibration version is inconsistent with the current captured and recognized frame, the next round of occlusion state version write-back will not be performed, and the current captured and recognized frame will be transferred to pending verification.

[0134] The gripper closure amount is then read. The gripper closure amount is the closure displacement of the inner contact surface of the two gripper fingers relative to the opening width of the gripper in this round after execution. The unit is millimeters. It is derived from the closure position record of the gripper controller before and after the execution completion signal is formed. The vision controller reads this value according to the current gripper parameter version and only records the closure displacement associated with the current grasping and recognition frame version number. It does not use the gripper closure amount of the previous batch to replace it.

[0135] Then read the contact trigger mark. The contact trigger mark is a binary mark formed by the contact detection element of the gripper fingertip, the change record of the gripper current, or the contact confirmation record of the gripper controller during the execution process. It is recorded as whether it has been triggered or not, and associated with the trigger time. When there are multiple contact confirmation sources in the same execution process, the contact confirmation source recorded in the current gripper parameter version shall prevail. If the sources are inconsistent, the current grasping and identification frame shall be transferred to the pending verification, and shall not be temporarily selected by the on-site operator.

[0136] After completing the acquisition of the retested color depth image, gripper closure amount, and contact trigger mark, the visual computing unit rewrites the occlusion state version for the next round based on the changes in the target visible area and the suspected occlusion boundary. The changes in the target visible area are obtained by comparing the target visible area corresponding to the target visible mask before execution with the retested target visible area re-formed in the retested color depth image according to the same model version and the same model parameter version, under the same coordinate caliber. The caliber recorded includes the newly added visible area, the reduced visible area, and the unchanged area. The unit is pixel area or square millimeter after station calibration. Only one unit caliber is used within the same current grasping and recognition frame.

[0137] Changes in suspected occlusion boundaries are obtained by comparing the suspected occlusion boundaries before execution with the re-formed suspected occlusion boundaries in the re-tested color depth image, which are formed using the same suspected occlusion boundary caliber. Only re-tested suspected occlusion boundaries that can establish a correspondence based on the current capture and recognition frame version number, difference region number, and boundary spatial position participate in boundary shortening, boundary lengthening, boundary position movement, and boundary disappearance records. Re-tested suspected occlusion boundaries that cannot establish a correspondence are only used as traces to be verified and do not rewrite the suspected occlusion boundaries before execution. The visible area of ​​the re-tested target cannot be formed using the same model version and the same model parameter version, or the re-tested suspected occlusion boundary records... If the same suspected occlusion boundary cannot be formed, the visual computing unit will not write back the occlusion state version in the next round, but will transfer the current captured and recognized frame to the pending verification stage, and retain the address of the retested color depth image, the gripper closure amount, the contact trigger mark, and the reason for not writing back; if the retested suspected occlusion boundary record can be formed but the number of valid suspected occlusion boundary segments is 0, and the number of valid suspected occlusion boundary segments in the previous suspected occlusion boundary record is greater than 0, it will be treated as if the retested suspected occlusion boundary has disappeared; if the number of valid suspected occlusion boundary segments in both the previous suspected occlusion boundary record and the retested suspected occlusion boundary record is 0, it will be treated as if the suspected occlusion boundary has not changed.

[0138] During the write-back process, the visual computing unit generates the next occlusion state version according to the currently effective occlusion state write-back rule version. The occlusion state write-back rule version records the change boundary of the target's visible area, the change boundary of the suspected occlusion boundary, the gripper closure amount boundary, the contact trigger mark participation condition, and the version submission boundary. Only one effective occlusion state write-back rule version is used for the same currently captured and recognized frame. Among them, reaching the boundary means greater than or equal to the boundary, and below the boundary means less than the boundary.

[0139] The visual computing unit first checks for changes in the target's visible area and changes in suspected occlusion boundaries. If the target's visible area increases to the boundary of the visible area change, and the suspected occlusion boundary disappears relative to the previous suspected occlusion boundary, and the target's visible area reaches the no-occlusion confirmation boundary recorded in the occlusion state write-back rule version, then the next round of occlusion state version writes the no-occlusion confirmation state. If the target's visible area increases to the boundary of the visible area change, and the suspected occlusion boundary shortens relative to the previous suspected occlusion boundary but still exists, then the next round of occlusion state version writes "edge occlusion but channel open". If the aforementioned conditions are not met, and the target visible area decreases to the target visible area change boundary, or the suspected occlusion boundary extends to the suspected occlusion boundary change boundary, the next round of occlusion status version will write the occlusion source pending retest status; if the aforementioned conditions are not met, and the target visible area does not increase, the suspected occlusion boundary does not shorten, the gripper closure amount is lower than the gripper closure amount boundary and a triggered contact trigger mark is formed, the next round of occlusion status version will maintain the current occlusion status and write the contact obstruction basis; if none of the aforementioned conditions are met, the next round of occlusion status version will write the occlusion source pending retest status.

[0140] The above write-back judgment is executed in the order described above. The rule that is satisfied first is used as the basis for writing the next round of occlusion state version, and subsequent rules will not be rewritten.

[0141] The next round of occlusion status version is an incremental version record that does not overwrite the previous round of occlusion status version or the current occlusion status record. The minimum record content includes the current capture and recognition frame version number, the address of the retested color image, the address of the retested depth image, the gripper closure amount, the contact trigger mark, the change of the target visible area, the change of the suspected occlusion boundary, the current occlusion status, the next round status name, the previous occlusion status version, the occlusion status write-back rule version, the next round occlusion status version number, and the formation time. Only one next round occlusion status version is generated for the same current capture and recognition frame version number, the same retest acquisition time, and the same occlusion status write-back rule version. When submitting repeatedly, the existing version is returned, no new version is generated, and the submitted version is not overwritten.

[0142] This step is applicable to scenarios of disordered sorting of bins, pallet loading, and conveyor belt paused gripping where the current occlusion state and retest identification tags have already been formed. The trajectory and force control process of the robot's actions are not rewritten in this step. This step only collects the retest color depth image, gripper closure amount, and contact trigger mark after execution, and completes the next round of occlusion state version write-back.

[0143] During on-site spot checks, at least 50 records of the next round of occlusion status versions can be randomly selected from continuous production batches to verify whether the retested color depth image corresponds to the current grasping area, whether the gripper closure amount comes from the same execution process, whether the contact trigger mark comes from the contact confirmation source recorded in the current gripper parameter version, and whether the changes in the target visible area and suspected occlusion boundary changes can be traced back to the records before and after execution. If any of these items are lower than the on-site set threshold, the submission of the next round of occlusion status versions at this workstation will be suspended, and the retest acquisition records, gripper controller records, and occlusion status write-back rule versions will be checked.

[0144] Preferably, in the automotive connector bin loading station, the re-measured color depth image can be acquired within 100 to 300 milliseconds after the execution completion signal. The resolution of the re-measured color image can be set to 1280×720, the re-measured depth distance range can be set to 200 mm to 1200 mm, the target visible area change boundary can be set to an area change of 1200 pixels, the suspected occlusion boundary change boundary can be set to a length change of 15 mm, the gripper closure amount boundary can be set to 35 mm, and the next round of occlusion state version formation time does not exceed 120 milliseconds. For example, if the current grasping and recognition frame version number is 38, the current occlusion state is the target contact surface occlusion state, after execution, the re-measured color depth image shows that the target visible area increases by 1600 pixels, the suspected occlusion boundary shortens by 22 mm, the gripper closure amount is 38 mm, the contact trigger mark is triggered, and the visual computing unit forms version 39 of the next round of occlusion state according to the occlusion state write-back rule version, and writes the next round state name into the edge occlusion but channel open state.

[0145] As an alternative approach, the retested color depth image can be formed by aligning the split color camera and depth camera with the same camera calibration version. The gripper closure amount can be formed by the gripper encoder position record or the gripper controller position feedback. The contact trigger mark can be formed by the gripper fingertip contact switch or the gripper current change record. As long as the next round of occlusion state version is still written back based on the changes in the target visible area and the suspected occlusion boundary changes, and can be traced through the current capture and identification frame version number.

[0146] In the operating scenario shown in this embodiment: the robot gripping production line is a material loading station for automotive connector bins. The bins contain a mixture of metal clips, nuts, and washers. A two-finger parallel gripper is installed at the end of the robot. A color depth camera is installed above the gripping station. The vision computing unit and the gripper controller are both connected to the same task recording area of ​​the station.

[0147] Once the material bin reaches the positioning position and generates a workstation positioning signal, the vision controller acquires a color depth image in the current gripping area. The color image resolution is set to 1280×720, and the depth distance range is set to 200 mm to 1200 mm. The time difference between the color image and the depth image acquisition is controlled within 20 milliseconds. The target category recorded in the current gripping task sheet is metal buckle. The approach direction of the gripper given by the robot gripping task is from top to bottom and slightly forward in the workstation coordinate system. The gripper opening width set by the gripper controller for the current task is 45 mm. The most recent non-obliterated occlusion state version for the same gripping area and the same target category in the occlusion state recording area is version 37. Based on this, the vision controller generates version 38 of the current gripping recognition frame and writes the color image address, depth image address, target category, gripper approach direction, gripper opening width, previous occlusion state version, camera calibration version, and task batch number into the workstation task recording area.

[0148] Subsequently, the visual computing unit calls the 6th version of the deep learning recognition model and the 6th version of the model parameters for the metal buckle bin recognition. It generates target visibility mask, candidate complete contour, neighboring object boundary, and depth abrupt boundary for the current captured recognition frame of version 38. Among them, the target visibility mask corresponds to the set of surface pixels that can be directly observed by the metal buckle from the camera's perspective, the candidate complete contour corresponds to the candidate record of the target outer edge of the metal buckle from the current perspective, the neighboring object boundary records the non-target object outer edge line segments of nuts and washers near the target outer edge, and the depth abrupt boundary records the continuous height change boundary between stacked parts in the bin.

[0149] The visual computing unit further aligns the target visible mask with the candidate complete outline to obtain two difference regions. The outer edge of the first difference region has both the outer edge boundary of the nut and the depth abrupt boundary, forming a suspected occlusion boundary. The second difference region only has the depth abrupt boundary and no adjacent object boundary, so it does not form an effective suspected occlusion boundary segment and only retains the difference region record.

[0150] Subsequently, the visual computing unit generates a gripper approach channel according to the gripper approach direction and the 45 mm gripper opening width. The channel length is set to extend 60 mm in the opposite direction from the front end of the channel along the gripper approach direction, with a reserved width of 3 mm on each side, and the gripper fingertip contact surface width is set to 6 mm. Calculated using the same coordinate system, the suspected occlusion boundary segment enters the front end of the gripper approach channel by 18 mm and falls into the target contact surface by 4 mm, thus forming the occupancy relationship between the suspected occupancy boundary and the gripper approach channel and the target contact surface.

[0151] The visual computing unit continues to obtain the target visibility ratio of 68% based on the target visibility mask and candidate complete contours, and locks the current occlusion state by combining the suspected occlusion boundary, occupancy relationship and the previous occlusion state version 37. Since the length occupied by the suspected occlusion boundary on the target contact surface reaches the 4 mm target contact surface occupancy boundary recorded in the occlusion state rule version, the visual computing unit locks the current occlusion state corresponding to the current capture and recognition frame of version 38 as the target contact surface occlusion state, and generates a contact surface retest label.

[0152] The robot grasping execution system then completes a gripper approach and closing action according to the predetermined grasping task of this round. After the execution completion signal and gripper stop signal are formed, the vision controller acquires a retest color depth image within 100 to 300 milliseconds. At the same time, it reads the gripper closing amount of 38 mm from the gripper controller and reads the contact trigger mark as triggered.

[0153] The visual computing unit compares the target's visible area before execution with the target's visible area after retesting under the same coordinate system, and finds that the target's visible area has increased by 1600 pixels; then it compares the suspected occlusion boundary before execution with the suspected occlusion boundary after retesting, and finds that the suspected occlusion boundary has shortened by 22 mm but still exists.

[0154] In the occlusion state write-back rule version, the target visible area change boundary is set to an area change of 1200 pixels, the suspected occlusion boundary change boundary is set to a length change of 15 mm, and the gripper closure amount boundary is set to 35 mm. Since the target visible area increases to reach the change boundary, and the retested suspected occlusion boundary is shortened compared to the previous suspected occlusion boundary but still exists, the visual computing unit forms version 39 of the next round of occlusion state, writes the next round state name into the edge occlusion but channel open state, and records version 38 of the current grasping and recognition frame version number, the retested color image address, the retested depth image address, the gripper closure amount, the contact trigger mark, the target visible area change, the suspected occlusion boundary change, the previous occlusion state version, the occlusion state write-back rule version, and the formation time.

[0155] In this implementation case, if the number of valid suspected occlusion boundary segments after retesting is 0 and the visible area of ​​the retested target reaches the unoccluded confirmation boundary, then the next round of occlusion status version is written to the unoccluded confirmation status; if the retested image is missing, the gripper closure amount cannot be read, the contact trigger mark source is inconsistent, or the visible area of ​​the retested target cannot be formed according to the same model version and the same model parameter version, then the next round of occlusion status version is not written back, and the current captured recognition frame is transferred to pending verification.

[0156] All calculations involved in the embodiments are dimensionless numerical calculations, and the preset parameters and thresholds in the calculations are set by those skilled in the art according to the actual situation.

[0157] It should be noted that this invention can be deployed on the device itself to realize embedded applications, or it can run on a PC or other terminal with a user interface, thereby meeting various hardware environments and usage requirements.

[0158] The above embodiments can be implemented, in whole or in part, by software, hardware, firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wireless or wired transmission; wired transmission methods include optical fiber, twisted pair, coaxial cable, etc.; wireless transmission includes infrared, microwave, etc. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center containing one or more sets of available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium. A semiconductor medium can be a solid-state drive.

[0159] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and modules described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0160] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or modules may be electrical, mechanical, or other forms.

[0161] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules; they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0162] In addition, the functional modules in the various embodiments of this application can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module.

[0163] If the aforementioned functions are implemented as software functional modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0164] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0165] In conclusion, the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A deep learning-based method for recognizing the occlusion state of a target during robot grasping, characterized in that, include: S1. Acquire a color depth image of the current grasping area, read the target category, gripper approach direction, gripper opening width, and previous occlusion state version, and generate the current grasping recognition frame. S2. Input the current captured and recognized frame into the deep learning recognition model to generate target visible mask, candidate complete contour, neighboring object boundary, and depth abrupt boundary; S3. Based on the difference between the target's visible mask and the candidate complete outline, and combined with the boundaries of nearby objects and depth abrupt change boundaries, determine the suspected occlusion boundaries. S4. Generate a gripper approach channel according to the gripper approach direction and gripper opening width, and calculate the occupancy relationship between the suspected occupancy boundary and the gripper approach channel and the target contact surface. S5. Based on the target visibility ratio formed by the target visibility mask and candidate complete contours, suspected occlusion boundaries, occupancy relationships, and the previous round of occlusion status version, lock the current occlusion status and generate a retest recognition label. S6. Collect the retested color depth image after execution, read the gripper closure amount and contact trigger mark, and write down the next round of occlusion status version based on the changes in the visible area of ​​the target and the changes in the suspected occlusion boundary.

2. The method for recognizing the occlusion state of a robot grasping target based on deep learning according to claim 1, characterized in that, S1 includes: The current grasping area is determined according to the fixed image area within the camera calibration coordinate system, and the effective boundary of this round is defined by the batch number of this round of tasks, the station positioning signal, and the stopping position of the material box; Color depth images are formed by pairing color images and depth images acquired at the same time, and aligned to the same image coordinate scale according to the camera calibration version; The frame-by-frame record of the current capture and recognition frame includes the color image address, depth image address, target category, gripper approach direction, gripper opening width, and the previous round of occlusion state version.

3. The method for recognizing the occlusion state of a robot grasping target based on deep learning according to claim 1, characterized in that, S2 include: The deep learning recognition model is formed by solidifying color depth images of similar grasping regions, target category records, and manually labeled records. The manually labeled records correspond to the target visible mask, candidate complete contour, boundaries of neighboring objects, and depth abrupt boundary. The model version is bound to the target category table, camera calibration version, and model parameter version. The model parameter version records the proximity statistical range, depth verification range, depth mutation threshold, and target category shape constraints.

4. The method for recognizing the occlusion state of a robot grasping target based on deep learning according to claim 1, characterized in that, S3 include: The visual computing unit checks that the target visible mask, candidate complete contour, boundary of neighboring objects and boundary of depth abrupt change are all bound to the same current capture and recognition frame version number, model version number and model parameter version number; Using the outer edge range of the target candidate corresponding to the complete outline of the candidate as the outer boundary, and the area occupied by the target visible mask as the seen area, a difference region is formed that is not covered by the target visible mask and is connected to the outer edge of the target visible mask.

5. The method for recognizing the occlusion state of a robot grasping target based on deep learning according to claim 4, characterized in that, S3 also includes: The visual computing unit selects the boundaries of neighboring objects within the neighbor statistics range recorded in the model parameter version located at the outer edge of the difference region, and selects the depth abrupt change boundaries within the depth verification range recorded in the model parameter version located at the outer edge of the difference region. When the same continuous boundary segment has both neighboring object boundary relationships and depth abrupt boundary relationships, a suspected occlusion boundary is formed.

6. The method for recognizing the occlusion state of a robot grasping target based on deep learning according to claim 1, characterized in that, S4 include: The visual computing unit organizes the approach direction of the gripper, the opening width of the gripper, and the suspected occlusion boundary under the same current grasping and recognition frame version number into the same coordinate system. The channel centerline is determined based on the approach direction of the grippers, and the channel width is determined based on the gripper opening width and the gripper parameter version corresponding to the current target category. The outer edge region of the target that falls within the projection range of the inner contact surface of the two fingers of the gripper in the candidate complete contour is determined as the target contact surface.

7. The method for recognizing the occlusion state of a robot grasping target based on deep learning according to claim 6, characterized in that, S4 also includes: The visual computing unit aligns the suspected occlusion boundary with the gripper approach channel and the target contact surface to the same coordinate scale. Using the range of the gripper approaching the channel and the range of the target contact surface as the statistical range, record the continuous boundary segments that the suspected occupancy boundary falls into each statistical range, and form the occupied length, occupied area and associated suspected occupancy boundary number.

8. The method for recognizing the occlusion state of a robot grasping target based on deep learning according to claim 1, characterized in that, S5 include: The target visibility ratio is the percentage of the target visibility mask coverage area relative to the outer edge range of the corresponding target candidate in the complete candidate contour; The visual computing unit checks the target visibility ratio, suspected occlusion boundaries, occupancy relationships, and the previous round of occlusion status, all of which are bound to the same current capture and recognition frame version number. According to the occupancy boundary of the target contact surface, the occupancy boundary of the gripper approach channel, the existence conditions of the suspected occupancy boundary, and the unoccupancy confirmation boundary recorded in the occupancy status rule version, the current occupancy status is locked, and a retest identification label is generated.

9. The method for recognizing the occlusion state of a robot grasping target based on deep learning according to claim 1, characterized in that, S6 include: After the completion signal arrives and the gripper stop signal is generated, the vision controller acquires a remeasured color depth image and reads the gripper closure amount and contact trigger mark. The visual computing unit compares the target visible area before execution with the target visible area after retesting, corresponding to the target visible mask, to form the change in the target visible area; Compare the suspected occlusion boundary before execution with the remeasured suspected occlusion boundary obtained by forming the same suspected occlusion boundary caliber to form the change of suspected occlusion boundary; Write the next round of occlusion state version according to the occlusion state writeback rule version.

Images