A robot vision servo control method based on depth point cloud

Abstract

This invention discloses a robot visual servo control method based on depth point clouds, belonging to the field of intelligent control technology. The method includes: statistically analyzing the local point density, depth continuity, occlusion ratio, and missing trend of a target point cloud, and attaching these parameters to the corresponding point cloud region to form a target point cloud package; constructing a set of point cloud confidence servo error output terms based on a viewpoint maintenance index, and performing dynamic weighting in conjunction with local point cloud region quality markers to generate point cloud confidence servo error data and active viewpoint adjustment commands; performing robot visual servo control based on the currently executed control package, and outputting the target alignment state and end effector positioning state when the point cloud confidence servo error data meets preset joint discrimination conditions. This invention, by generating viewpoint maintenance indices, continuously evaluates the coverage changes, boundary changes, and center position changes of the target object region in the current field of view, achieving a stable representation of continuous observation maintenance.

Description

Technical Field

[0001] This invention relates to the field of intelligent control technology, and in particular to a robot vision servo control method based on depth point clouds. Background Technology

[0002] With the continuous development of depth camera imaging accuracy and 3D point cloud processing technology, robot vision servo control based on depth point clouds has gradually become an important research direction in the fields of intelligent manufacturing and automated assembly. Related technologies model the 3D spatial distribution, posture information and geometric features of the target object, and combine them with the pose information of the robot end effector to achieve closed-loop collaborative processing from environmental perception to motion control. This further promotes the transformation of vision servoing from traditional 2D image feature-driven to 3D spatial structure perception and multi-source information fusion-driven. At the same time, based on continuous observation state analysis and field of view coverage evaluation mechanisms, it enhances the adaptability of operations in complex scenarios.

[0003] Existing technologies struggle to balance stable observation of the target area and unified representation of multiple types of deviations within the same control cycle when deep point clouds exhibit local point density unevenness, depth continuity fluctuations, and dynamic changes in occlusion and missing data. This leads to inconsistent transmission of positional deviations, attitude deviations, and region alignment deviations during the coupling process. Furthermore, it results in a lack of effective coordination between observation attitude adjustment and end-effector control, thereby affecting the continuous coverage of the target area and the stability of alignment determination during visual servoing, and reducing the continuity and consistency of the overall control process. Summary of the Invention

[0004] In view of the aforementioned existing problems, the present invention is proposed.

[0005] Therefore, this invention provides a robot visual servo control method based on depth point clouds to solve the problem of insufficient visual servo alignment stability under the influence of occlusion and observation disturbances in depth point clouds.

[0006] To solve the above-mentioned technical problems, the present invention provides the following technical solution: This invention provides a robot visual servo control method based on depth point clouds, comprising: statistically analyzing the local point density, depth continuity, occlusion ratio, and missing trend of a target point cloud, and attaching it to the corresponding point cloud region to form a target point cloud package; performing target segmentation and geometric region analysis on the target point cloud package, extracting spatial state information and geometric region information, constructing target state description data, and generating a viewpoint maintenance index; constructing a set of point cloud confidence servo error output terms based on the viewpoint maintenance index, and performing dynamic weighting in conjunction with local point cloud region quality markers to generate point cloud confidence servo error data and active viewpoint adjustment instructions; calculating the end-effector approximation control quantity based on the point cloud confidence servo error data, executing the active viewpoint adjustment instructions, calculating the observation posture correction quantity, and fusing them into the current execution control package; performing robot visual servo control based on the current execution control package, and outputting the target alignment state and end-effector positioning state when the point cloud confidence servo error data meets the preset joint discrimination conditions.

[0007] As a preferred embodiment of the robot vision servo control method based on depth point clouds described in this invention, the specific steps for forming the target point cloud package are as follows: Acquire depth point clouds, record the current acquisition sequence information to generate acquisition time sequence identifiers, and perform point cloud preprocessing to obtain the target point cloud; The target point cloud is divided into local point cloud regions, region identifiers are generated, and the local point density, depth continuity, occlusion ratio and missing trend of the local point cloud regions are statistically analyzed to form the target point cloud package.

[0008] As a preferred embodiment of the robot vision servo control method based on depth point cloud described in this invention, the specific steps for constructing the target state description data are as follows: The local point cloud regions are quality-labeled based on the target point cloud package and aggregated according to spatial adjacency. Local point cloud regions with abnormal missing trends are filtered out to obtain the target object point cloud set. Spatial state information and geometric region information are extracted from the point cloud set of the target object, and then associated and encapsulated with region identifiers and acquisition time sequence identifiers to form target state description data.

[0009] As a preferred embodiment of the robot vision servo control method based on depth point clouds described in this invention, the specific steps for generating the viewpoint maintenance index are as follows: Based on the geometric region information in the target state description data, combined with the observation direction, observation distance and field of view coverage of the depth camera, the coverage change, boundary change and center position change of the target object region in the current field of view are analyzed to obtain the current observation state data; Based on the current observation data, assess the current observation posture's ability to maintain continuous observation of the target area and generate a viewpoint maintenance index.

[0010] As a preferred embodiment of the robot vision servo control method based on depth point clouds described in this invention, the specific steps for constructing the point cloud confidence servo error output term set are as follows: The input fusion layer receives spatial state information, geometric region information, viewpoint maintenance index, robot current end effector pose, and local point cloud region quality markers, and collects error information according to region identifier and acquisition time sequence identifier to form a unified error input set. The unified error input set is input into the state alignment layer for position and attitude mapping, and the geometric region information is matched to form the original deviation set; The confidence modulation layer combines local point cloud region quality markers to pre-adjust the participation intensity of the original bias set, and the view compensation layer combines view maintenance index to perform continuous compensation and orientation correction to form a compensated bias set. The output fusion layer correlates and fuses the deviation items in the compensation deviation set, and outputs a set of point cloud confidence servo error output items.

[0011] As a preferred embodiment of the robot vision servo control method based on depth point clouds described in this invention, the specific process of generating point cloud confidence servo error data and active viewpoint adjustment commands is as follows: The deviation items, region identifiers, acquisition time sequence identifiers, and local point cloud region quality markers are extracted from the point cloud confidence servo error output item set and matched and integrated to form a weighted input deviation set. Based on the weighted input bias set combined with local point density, depth continuity, occlusion ratio and missing trend, confidence redistribution and influence weight allocation are performed on each bias item to form point cloud confidence servo error data; Based on the attitude deviation and regional alignment deviation after participation intensity adjustment and influence weight allocation, and combined with the viewpoint maintenance index, the current observation state is analyzed for observation offset and the viewpoint adjustment level is classified, and an active viewpoint adjustment command is formed.

[0012] As a preferred embodiment of the robot vision servo control method based on depth point clouds described in this invention, the specific process of calculating the end-effector approximation control quantity is as follows: Based on the deviation items in the point cloud confidence servo error data, the data is merged and organized in combination with the region identifier and the acquisition time sequence identifier to form the end control input set; Based on the end-effector control input set and combined with the robot's current end-effector pose and end-effector operation direction, the execution direction mapping and coordination of each deviation term are performed to form the end-effector approximation control quantity.

[0013] As a preferred embodiment of the robot vision servo control method based on depth point clouds described in this invention, the specific process for calculating the observed posture correction is as follows: Read the active viewpoint adjustment command, and combine it with the area identifier, acquisition time sequence identifier, viewpoint maintenance index and current observation status to perform alignment, merging and correlation matching to form the observation correction input set; Based on the observation correction input set, the active viewpoint adjustment command execution is mapped and coordinated to form the observation attitude correction quantity.

[0014] As a preferred embodiment of the robot vision servo control method based on depth point clouds described in this invention, the fusion into the current execution control package is specifically as follows: Read the end-point approximation control quantity and the observed attitude correction quantity, align, merge and integrate them to form a control fusion input set; Based on the control fusion input set, the timing, direction and rhythm of the end-point approximation control quantity and the observed attitude correction quantity are coordinated and associated with the current execution control package.

[0015] As a preferred embodiment of the robot vision servo control method based on depth point clouds described in this invention, the specific process of outputting the target alignment state and the end effector positioning state is as follows: Based on point cloud confidence servo error data, viewpoint maintenance index, current execution control package and current observation status data, the deviation term is continuously tracked and status judgment is performed to form a status discrimination input set; Based on the state discrimination input set, the point cloud confidence servo error data, view maintenance index and current observation state data are jointly discriminated, and the target alignment state and end position state are output when the joint discrimination conditions are met.

[0016] The beneficial effects of this invention are as follows: By generating a viewpoint maintenance index, the coverage changes, boundary changes, and center position changes of the target object region in the current field of view are continuously evaluated, achieving a stable representation of the continuous observation maintenance status and providing a reliable basis for active viewpoint adjustment commands and state discrimination input sets; by constructing a set of point cloud confidence servo error output terms, the target state description data, local point cloud region quality markers, and the robot's current end effector pose are uniformly collected, compensated, and fused, realizing the orderly generation of point cloud confidence servo error data, which is used to support the coordinated formation of end-effector approximation control quantities, observation posture correction quantities, and current execution control packages, thereby improving the alignment stability and positioning reliability of the robot's visual servo control. Attached Figure Description

[0017] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a flowchart of a robot vision servo control method based on depth point clouds.

[0019] Figure 2 A flowchart for constructing the set of point cloud confidence servo error output terms.

[0020] Figure 3 A flowchart for generating commands to adjust the active viewpoint.

[0021] Figure 4 This is a flowchart for determining the current execution control packet. Detailed Implementation

[0022] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0023] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0024] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in different places in this specification does not necessarily refer to the same embodiment, nor is it a single or selective embodiment that is mutually exclusive with other embodiments.

[0025] Reference Figures 1-4 This is one embodiment of the present invention, which provides a robot vision servo control method based on depth point clouds, including the following steps: S1: Statistically analyze the local point density, depth continuity, occlusion ratio, and missing trend of the target point cloud, and attach it to the corresponding point cloud region to form a target point cloud package.

[0026] S1.1: Acquire depth point cloud, record the current acquisition sequence information to generate acquisition timing identifier, and perform point cloud preprocessing to obtain target point cloud.

[0027] Specifically, a depth camera is used to collect depth point clouds of the current work scene, and the sequence information of this collection is synchronously written into the current point cloud data as a collection time sequence identifier; point cloud preprocessing is performed on the depth point cloud to remove irrelevant background and abnormal noise points, and retain the main point cloud of the work area where the target object is located to obtain the target point cloud.

[0028] It should be noted that the acquisition time sequence identifier refers to the time sequence marking information that represents the acquisition sequence of the current depth point cloud.

[0029] S1.2: Divide the target point cloud into local point cloud regions, generate region identifiers, and statistically analyze the local point density, depth continuity, occlusion ratio, and missing trend of the local point cloud regions to form the target point cloud package.

[0030] Specifically, the target point cloud is partitioned according to its spatial distribution and local connectivity, dividing it into multiple independent local point cloud regions and assigning each region a corresponding region identifier. The distribution of points within each local point cloud region is used to characterize local point density; the depth changes between adjacent points are used to characterize depth continuity; the proportion of occluded parts in the current region is used to characterize the occlusion ratio; and the missing trend is extracted by combining the point cloud changes of local point cloud regions over continuous acquisition cycles. The region identifier, local point density, depth continuity, occlusion ratio, and missing trend are then correlated and encapsulated to form a target point cloud package.

[0031] Among them, region identification refers to the identification marking information that distinguishes each local point cloud region.

[0032] S2: Perform target segmentation and geometric region analysis on the target point cloud packet, extract spatial state information and geometric region information, construct target state description data, and generate view maintenance index.

[0033] S2.1: Based on the target point cloud package, the local point cloud regions are quality-marked and aggregated according to spatial adjacency. Local point cloud regions with abnormal missing trends are filtered out to obtain the target object point cloud set.

[0034] Specifically, based on the region identifiers in the target point cloud package and the local point density, depth continuity, occlusion ratio, and missing trend of each local point cloud region, the local point density, depth continuity, occlusion ratio, and missing trend of each local point cloud region are compared with preset quality judgment criteria item by item, and each local point cloud region is marked as a reliable region or an abnormal region according to the comparison results. In particular, the number of points and the area coverage of the local point cloud region in the current control cycle are compared with the number of points and the area coverage of the corresponding local point cloud region in the previous control cycle. When the number of points continuously decreases and the area coverage continuously shrinks, and the decrease and shrinkage reach the preset missing trend judgment threshold, the local point cloud region is judged as a missing trend abnormal region. Local point cloud regions that are close to each other and continuously distributed are aggregated according to spatial adjacency, and the local point cloud regions judged as missing trend abnormal are removed, retaining the point cloud regions that jointly represent the main outline of the target object, to obtain the target object point cloud set.

[0035] The missing trend discrimination threshold is preset based on the allowable decrease in the number of points in a local point cloud region and the shrinkage of the region coverage area under continuous control cycle.

[0036] Based on the reference point cloud region of the target object under normal observation conditions, the local point density, depth continuity, occlusion ratio and missing trend are statistically analyzed, and the lower limit of local point density, the lower limit of depth continuity, the upper limit of occlusion ratio and the upper limit of missing trend are set as quality judgment criteria.

[0037] Spatial adjacency refers to the proximity and boundary contact relationships between different local point cloud regions in three-dimensional space.

[0038] S2.2: Extract spatial state information and geometric region information from the point cloud set of the target object, and associate and encapsulate them with region identifier and acquisition time sequence identifier to form target state description data.

[0039] Specifically, based on the point cloud set of the target object, the position distribution, orientation and spatial contour information representing the current three-dimensional state of the target object are extracted to form spatial state information; then, geometric region information is extracted around the contour boundary, area range and task-related position of the target object surface to represent the key operation area of ​​the target object and its spatial distribution; the spatial state information, geometric region information and corresponding region identifiers and acquisition time sequence identifiers are associated and encapsulated to form target state description data.

[0040] Among them, the spatial state information describes the current overall spatial position, orientation, and contour distribution of the target object, providing a reference for the positional and orientation correspondence between the robot end effector and the target object; the geometric region information describes the operation area and distribution of the target object's surface, providing a reference for viewpoint maintenance, region alignment, and active viewpoint adjustment.

[0041] The task action position refers to the target alignment position, contact position, or operation position on the surface of the target object that corresponds to the current robot operation task.

[0042] S2.3: Based on the geometric region information in the target state description data, combined with the observation direction, observation distance and field of view coverage of the depth camera, analyze the coverage changes, boundary changes and center position changes of the target object region in the current field of view to obtain the current observation state data.

[0043] Specifically, the current observation direction is obtained based on the orientation of the depth camera relative to the robot's end effector in the current control cycle, combined with the current pose of the robot's end effector; the current observation distance is obtained based on the interval between the position of the depth camera in the current control cycle and the spatial distribution of the target object region; and the current field of view coverage is obtained based on the observable range of the target object region formed by the depth camera in the current observation direction and at the current observation distance.

[0044] Based on the geometric region information in the target state description data, the spatial distribution location and boundary range of the target object region under the current control cycle are determined. Combining the observation direction, observation distance, and the field of view coverage of the depth camera, the spatial distribution location and boundary range of the target object region are mapped onto the projection plane corresponding to the current field of view, obtaining the projection range, projection boundary, and projection center position of the target object region in the current field of view. The projection range of the target object region in the current field of view is compared with the original region range of the target object region to obtain the coverage change. The projection boundary of the target object region retained in the current field of view is compared with the original boundary range to obtain the boundary change. The current projection center position of the target object region is compared with the center position of the field of view to obtain the center position change. The coverage change, boundary change, and center position change are summarized and organized to form the current observation state data.

[0045] S2.4: Based on the current observation status data, evaluate the current observation attitude for maintaining continuous observation of the target area and generate a viewpoint maintenance index.

[0046] Specifically, the current observation status data includes coverage changes, boundary changes, and center position changes. Combined with the observation status of the corresponding target area under adjacent control cycles, continuous checks are performed on whether coverage changes continue to converge, whether boundary changes remain continuous, and whether center position changes are within the preset allowable range of center position offset. The checks on coverage changes, boundary changes, and center position changes are summarized to generate the observation maintenance status of the current observation attitude, and view maintenance indicators are generated based on the observation maintenance status.

[0047] It should be noted that, based on the center position of the depth camera's field of view, the initial center position of the target object region, and the allowable center offset tolerance during visual servoing, the allowable offset range of the target object region's center relative to the center of the field of view is preset as the allowable range of center position offset.

[0048] The viewpoint maintenance index describes the degree to which the current observation posture maintains continuous observation of the target area, and serves as the basis for generating active viewpoint adjustment commands and executing viewpoint compensation.

[0049] The viewpoint maintenance index is calculated using the following formula: ; In the formula, Indicates the current control cycle Maintain indicators from the perspective of the following Indicates the current control cycle Below, the effective coverage area of ​​the target object region within the current field of view. Indicates the target object region in the reference state. The following is the reference coverage area. Indicates the current control cycle The length of the preserved boundary of the target object region in the current field of view. Indicates the target object region in the reference state. The reference boundary length below, Indicates the current control cycle The offset distance between the center of the target area and the center of the field of view. This indicates the upper limit of the allowed range for center position offset. distance.

[0050] Among them, the reference coverage area is the area value corresponding to the original area of ​​the target object, the reference boundary length is the boundary length value corresponding to the original boundary of the target object, and the effective coverage area is the area value corresponding to the part of the target object's projection range that overlaps with the original area range in the current field of view.

[0051] S3: Based on the viewpoint maintenance index, construct a set of point cloud confidence servo error output items, and perform dynamic weighting in combination with local point cloud region quality labels to generate point cloud confidence servo error data and active viewpoint adjustment instructions.

[0052] S3.1: Receive spatial state information, geometric region information, viewpoint maintenance index, robot current end effector pose, and local point cloud region quality markers through the input fusion layer, and collect error information according to region identifier and acquisition time sequence identifier to form a unified error input set.

[0053] Specifically, the input fusion layer merges spatial state information, geometric region information, view maintenance index, robot current end effector pose, and local point cloud region quality markers. Based on region identifiers, spatial state information, geometric region information, and local point cloud region quality markers belonging to the same target object region are matched. Based on acquisition time sequence identifiers, spatial state information, geometric region information, view maintenance index, and robot current end effector pose under the same acquisition cycle are temporally aligned. The spatial state information, geometric region information, view maintenance index, robot current end effector pose, and local point cloud region quality markers after region matching and temporal alignment are uniformly merged to form a unified error input set.

[0054] S3.2: Input the unified error input set into the state alignment layer for position and attitude mapping, and match the geometric region information to form the original deviation set.

[0055] Specifically, the unified error input set is fed into the state alignment layer. The spatial position distribution and orientation in the target state description data are compared one by one with the actual position and orientation of the robot's current end effector to obtain the position deviation and orientation deviation. The area range, boundary position and task-related position representing the operation area in the geometric region information are compared and matched with the current operation direction and action position of the end effector to obtain the alignment deviation. The position deviation, orientation deviation and alignment deviation are summarized to form the original deviation set.

[0056] It should be noted that the original deviation set represents the positional deviation, attitude deviation, and region alignment deviation between the target object and the robot's current end effector, and is the initial deviation input for the point cloud confidence servo error output item set.

[0057] S3.3: The confidence modulation layer combines local point cloud region quality markers to pre-adjust the participation intensity of the original deviation set, and the view compensation layer combines view maintenance index to perform continuous compensation and direction correction to form a compensated deviation set.

[0058] Specifically, the original bias set is fed into the confidence modulation layer, and the participation intensity of each bias term is pre-adjusted by combining the local point density, depth continuity, occlusion ratio, and missing trend in the local point cloud region quality label, so as to distinguish the participation degree of different bias terms when entering the subsequent fusion processing; the bias terms corresponding to the region with stable quality state are set to high participation intensity, and the bias terms corresponding to the region with obvious occlusion or abnormal missing trend are set to low participation intensity; the bias terms after the participation intensity pre-adjustment are fed into the view compensation layer, and the bias fluctuation caused by the change of observation attitude is continuously compensated and the direction is corrected by combining the view maintenance index, forming a compensated bias set.

[0059] It should be noted that the deviation items include position deviation, attitude deviation, and area alignment deviation.

[0060] S3.4: The deviation items in the compensation deviation set are correlated and fused by the output fusion layer to output a set of point cloud confidence servo error output items.

[0061] Specifically, the compensation deviation set is sent to the output fusion layer. Position deviations, attitude deviations, and region alignment deviations belonging to the same target object region are grouped into the same deviation group according to the region identifier. The deviation groups under the same control cycle are time-aligned according to the acquisition time sequence identifier. The participation intensity corresponding to each deviation item is used as the fusion weight, and the compensation state corresponding to each deviation item is used as the correction basis. The position deviations, attitude deviations, and region alignment deviations in the same deviation group are weighted and synthesized to form the servo error output items corresponding to the target object region. The servo error output items corresponding to each target object region are uniformly summarized according to the current control cycle to obtain the point cloud confidence servo error output item set.

[0062] S3.5: Extract deviation items, region identifiers, acquisition timing identifiers, and local point cloud region quality markers from the point cloud confidence servo error output item set, and match and integrate them to form a weighted input deviation set; Specifically, position deviation, attitude deviation, and region alignment deviation are extracted from the point cloud confidence servo error output item set. At the same time, the region identifier, acquisition time sequence identifier, and local point cloud region quality mark corresponding to each deviation item are extracted. Based on the region identifier, deviation items belonging to the same target object region are matched with the corresponding quality mark. Then, based on the acquisition time sequence identifier, related deviation items under the same acquisition cycle are merged to form a weighted input deviation set.

[0063] S3.6: Based on the weighted input bias set combined with local point density, depth continuity, occlusion ratio and missing trend, confidence level redistribution and influence weight allocation are performed on each bias item to form point cloud confidence servo error data.

[0064] Specifically, the reliability of each deviation item is distinguished based on its local point density, depth continuity, occlusion ratio, and missing trend in the weighted input deviation set. Local point density and depth continuity are mapped to reliability redistribution, and occlusion ratio and missing trend are mapped to influence weight allocation. Position deviation, pose deviation, and region alignment deviation are assigned corresponding participation strength and influence weights. After reliability redistribution and influence weight allocation, each deviation item is uniformly organized according to region identifier and acquisition time sequence identifier to form point cloud confidence servo error data.

[0065] S3.7: Based on the attitude deviation and area alignment deviation after participation intensity adjustment and influence weight allocation, and combined with the viewpoint maintenance index, the current observation state is analyzed for observation offset and the viewpoint adjustment level is classified to form an active viewpoint adjustment command.

[0066] Specifically, attitude deviation and region alignment deviation are screened out from the deviation items after the participation intensity adjustment and influence weight allocation are completed. Combined with the viewpoint maintenance index, the observation direction offset, observation distance offset, and position offset of the target object region in the current field of view are extracted in the current observation state. Based on the observation direction offset, observation distance offset, and position offset, the viewpoint adjustment direction, adjustment range, and adjustment priority to be executed are classified. The viewpoint adjustment direction, adjustment range, and adjustment priority are associated with the corresponding region identifier and acquisition time sequence identifier to form an active viewpoint adjustment command.

[0067] The required viewing angle adjustments are categorized by direction, magnitude, and priority. For example, when the viewing angle maintenance index is lower than the preset observation maintenance requirement, or when the boundary of the target object area is close to leaving the current field of view boundary, or when the task-related location is close to the edge of the field of view, it is classified as a first-level priority, indicating that it should be executed first.

[0068] If the viewing angle maintenance index is not lower than the preset observation maintenance requirement, but the position offset continues to increase or the boundary preservation degree continues to decrease within two or more consecutive control cycles, it is classified as a secondary priority, indicating that it will be executed subsequently.

[0069] When the viewing angle maintenance index meets the preset observation maintenance requirements, and the target object area remains stably within the target coverage field of view, with only a slight offset correction requirement, it is classified as a three-level priority, indicating that the execution will be postponed.

[0070] S4: Based on point cloud confidence servo error data, calculate the end-point approximation control quantity, execute active viewpoint adjustment commands, calculate the observed attitude correction quantity, and fuse them into the current execution control package.

[0071] S4.1: Based on the deviation items in the point cloud confidence servo error data, the data is merged and organized in combination with the region identifier and the acquisition timing identifier to form the end control input set.

[0072] Specifically, position deviation, attitude deviation, and region alignment deviation are extracted from point cloud confidence servo error data, and the region identifier and acquisition timing identifier corresponding to each deviation item are extracted simultaneously. Based on the region identifier, each deviation item belonging to the same target object region is classified into the same control category. Based on the acquisition timing identifier, related deviation items under the same acquisition cycle are merged in time to form the end control input set.

[0073] S4.2: Based on the end-effector control input set and combined with the robot's current end-effector pose and end-effector operation direction, the execution direction mapping and coordination of each deviation term is performed to form the end-effector approximation control quantity.

[0074] Specifically, based on the geometric region information in the target state description data and the current pose of the robot's end effector, the current orientation of the end effector relative to the target object's operating region is extracted as the end effector's operating direction. Combined with the robot's current end effector pose, the position deviation, attitude deviation, and region alignment deviation in the end effector control input set are respectively mapped to the end effector's approximation direction, attitude adjustment direction, and operating region fitting direction. The approximation direction corresponding to the position deviation, the attitude adjustment direction corresponding to the attitude deviation, and the operating region fitting direction corresponding to the region alignment deviation are coordinated to maintain consistent connection between position approximation, attitude correction, and region alignment under the same control cycle. The coordinated position approximation direction, attitude adjustment direction, and operating region fitting direction are summarized to form the end effector approximation control quantity.

[0075] Among them, the end-effector approximation control variable describes the comprehensive control requirements of the robot end effector when it approaches the target object, corrects its posture, and fits into the region. It is the foundation of end-effector motion control in the current execution control package.

[0076] S4.3: Read the active viewpoint adjustment command, and combine it with the area identifier, acquisition time sequence identifier, viewpoint maintenance index and current observation status to perform alignment, merging and correlation matching to form the observation correction input set.

[0077] Specifically, the perspective adjustment direction, adjustment range, and adjustment priority are extracted from the active perspective adjustment command, and the corresponding area identifier and acquisition time sequence identifier are extracted simultaneously. Based on the area identifier, the active perspective adjustment commands belonging to the same target object area are matched with the perspective maintenance index and the current observation status. Based on the acquisition time sequence identifier, the perspective adjustment direction, adjustment range, adjustment priority, perspective maintenance index, and current observation status under the same acquisition cycle are merged in time sequence to form the observation correction input set.

[0078] S4.4: Based on the observation correction input set, the active viewpoint adjustment command is mapped and coordinated to form the observation attitude correction quantity.

[0079] Specifically, based on the adjustment direction, adjustment magnitude, and adjustment priority in the observation correction input set, the active viewpoint adjustment command is mapped to the current depth camera's observation attitude adjustment requirements, clarifying the observation direction and degree of correction required under the current control cycle, and generating candidate observation attitude correction items; combined with the viewpoint maintenance index and the current observation status, the sequential and connecting relationships between the candidate observation attitude correction items are coordinated to generate observation attitude coordination correction items; and the observation attitude coordination correction items are summarized to form the observation attitude correction amount.

[0080] It should be noted that the observation attitude coordination correction term represents the various observation attitude correction contents after sequential and sequential coordination under the same control cycle. The viewpoint adjustment requirements corresponding to different target object areas are kept consistent in execution order and adjustment rhythm, providing an intermediate basis for generating stable observation attitude correction quantities.

[0081] S4.5: Read the end-approach control quantity and observed attitude correction quantity, align, merge and integrate them to form a control fusion input set.

[0082] Specifically, the end-point approach control and observation attitude correction quantities are aggregated into the same control cycle, and aligned and merged according to the corresponding area identifier and acquisition timing identifier to clarify the end-point motion requirements and observation attitude adjustment requirements corresponding to the same target area. The aligned and merged end-point approach control and observation attitude correction quantities are then connected and integrated to ensure that the end-point motion and observation attitude adjustment are consistent in execution order and coordination, forming a control fusion input set.

[0083] S4.6: Based on the control fusion input set, the timing, direction and rhythm of the end-point approximation control quantity and the observed attitude correction quantity are coordinated and associated with the current execution control package.

[0084] Specifically, focusing on the end-approach control quantity and the observation attitude correction quantity in the control fusion input set, the timing coordination of the execution relationship of the end-approach control quantity and the observation attitude correction quantity under the same control cycle is performed, the direction coordination of the coordination relationship between the end-move direction and the observation attitude adjustment direction is performed, and the rhythm coordination of the coordination relationship between the end-approach speed change and the observation attitude adjustment speed is performed. The end-approach control quantity and the observation attitude correction quantity after completing the timing coordination, direction coordination and rhythm coordination are uniformly associated to form the current execution control package.

[0085] Among them, timing coordination is to obtain the execution order of the terminal approach control quantity and the observation attitude correction quantity in the same control cycle based on the adjustment priority, adjustment magnitude and current observation status of the observation attitude correction quantity; direction coordination is to suppress, delay or compensate for the directional components that would cause the target object area to deviate from the current field of view based on whether there is a cooperative or conflicting relationship between the terminal motion direction and the observation attitude adjustment direction; rhythm coordination is to synchronize the terminal approach velocity and the observation attitude correction rhythm based on the matching relationship between the terminal approach velocity change and the observation attitude adjustment speed, so as to keep the target object area continuously and stably within the current field of view.

[0086] S5: Perform robot vision servo control based on the current execution control package. When the point cloud confidence servo error data meets the preset joint discrimination conditions, output the target alignment status and end effector positioning status.

[0087] S5.1: Based on the point cloud confidence servo error data, view maintenance index, current execution control package, and current observation status data, continuously track the deviation term and perform state judgment to form a state discrimination input set.

[0088] Specifically, following the continuous control cycle, position deviation, attitude deviation, and region alignment deviation are extracted from the point cloud confidence servo error data. Combined with the viewpoint maintenance index, the current execution control package, and the current observation state data for the corresponding cycle, each deviation item is compared before and after according to the acquisition time sequence identifier to obtain the change trajectory of each deviation item under the continuous control cycle. The change trajectory of each deviation item is then checked against the end-approach state corresponding to the current execution control package, the observation attitude adjustment state, and the target object region observation state corresponding to the current observation state data. Based on preset state judgment rules, it is determined whether the deviation continues to converge, whether the observation continues to be maintained, and whether the control response is effective, resulting in the deviation convergence state, observation maintenance state, and control response state. These states are then summarized to form a state discrimination input set.

[0089] It should be noted that the change trajectory refers to the sequential record of the numerical changes of the same deviation item under continuous control cycles.

[0090] S5.2: Based on the state discrimination input set, perform joint discrimination on point cloud confidence servo error data, view maintenance index and current observation state data, and output target alignment state and end position state when the joint discrimination conditions are met.

[0091] Specifically, based on the deviation convergence state, observation hold state, and control response state in the input set, the system verifies whether the position deviation in the point cloud confidence servo error data is continuously within a preset position convergence range, forming a position deviation verification state; it verifies whether the attitude deviation is continuously within a preset attitude convergence range, forming an attitude deviation verification state; it verifies whether the region alignment deviation is continuously within a preset region alignment convergence range, forming a region alignment deviation verification state; and it verifies whether the viewpoint maintenance index continuously meets the preset observation hold requirements, forming a viewpoint maintenance verification state.

[0092] The system verifies whether the current observation status data continuously indicates that the target object area is within the target coverage field of view, forming an observation status verification status. The system summarizes the position deviation verification status, attitude deviation verification status, area alignment deviation verification status, viewpoint maintenance verification status, and observation status verification status to generate a target alignment discrimination status and an end-point positioning discrimination status. When the target alignment discrimination status and the end-point positioning discrimination status simultaneously meet the joint discrimination conditions, the system outputs the target alignment status and the end-point positioning status.

[0093] The position convergence range is preset based on the allowable spatial deviation of the theoretical alignment position between the robot end effector and the target object. It can be combined with the geometric dimensions of the target object, assembly clearance, gripping tolerance or contact tolerance, and the allowable offset interval between the current position of the end effector and the target position is used as the preset position convergence range.

[0094] The attitude convergence range is preset based on the allowable attitude deviation between the end effector and the target object. It can be combined with the surface normal requirements of the target object, the tool contact angle requirements, or the assembly orientation requirements to use the allowable deflection range of the current attitude of the end effector relative to the target attitude as the preset attitude convergence range.

[0095] The region alignment convergence range is preset based on the allowable alignment error between the key operation area of ​​the target object and the end effect area. It can be combined with the target area boundary size, center offset tolerance and task-related action position deviation, and the region center offset range, boundary fitting deviation range or key position alignment deviation range as the preset region alignment convergence range.

[0096] The observation maintenance requirements are preset based on the stable observation needs of the depth camera in visual servoing of the target area. They can be combined with the coverage of the target area in the field of view, the degree of boundary preservation, the tolerance of center position offset, and the observation continuity requirements under continuous control cycle. The minimum maintenance standard that meets the continuous observation is used as the preset observation maintenance requirements.

[0097] The joint discrimination condition is pre-set based on the combination of conditions that simultaneously meet the mission completion requirements of position alignment, attitude alignment, area alignment, and observation maintenance. All the position deviation verification status, attitude deviation verification status, area alignment deviation verification status, viewpoint maintenance verification status, and observation status verification status must reach the corresponding pass standard as the joint discrimination condition. The joint discrimination condition includes position deviation, attitude deviation, area alignment deviation, and viewpoint maintenance index.

[0098] In summary, this invention achieves stable characterization of continuous observation by generating viewpoint maintenance indices to continuously evaluate the coverage, boundary, and center position changes of the target object region in the current field of view, providing a reliable basis for active viewpoint adjustment commands and state discrimination input sets. Furthermore, by constructing a set of point cloud confidence servo error output terms, it unifies, compensates for, and fuses target state description data, local point cloud region quality markers, and the robot's current end effector pose, achieving the orderly generation of point cloud confidence servo error data. This data supports the coordinated formation of end-effector approximation control quantities, observation posture correction quantities, and the current execution control package, improving the alignment stability and positioning reliability of the robot's visual servo control.

[0099] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A method for robot visual servoing based on depth point cloud, characterized in that, include: The local point density, depth continuity, occlusion ratio, and missing trend of the target point cloud are statistically analyzed and attached to the corresponding point cloud region to form a target point cloud package. Target segmentation and geometric region analysis are performed on the target point cloud packet, and spatial state information and geometric region information are extracted to construct target state description data and generate view maintenance index; Based on the viewpoint maintenance index, a set of point cloud confidence servo error output items is constructed, and dynamic weighting is performed in combination with local point cloud region quality labels to generate point cloud confidence servo error data and active viewpoint adjustment instructions. Based on point cloud confidence servo error data, the end-effector approximation control quantity is calculated, and the active viewpoint adjustment command is executed to calculate the observed attitude correction quantity and fuse them into the current execution control package; The robot performs visual servo control based on the current execution control package. When the point cloud confidence servo error data meets the preset joint discrimination conditions, the target alignment status and end effector positioning status are output.

2. The deep point cloud based robot vision servo control method of claim 1, wherein, The specific steps for forming the target point cloud packet are as follows: Acquire depth point clouds, record the current acquisition sequence information to generate acquisition time sequence identifiers, and perform point cloud preprocessing to obtain the target point cloud; The target point cloud is divided into local point cloud regions, region identifiers are generated, and the local point density, depth continuity, occlusion ratio and missing trend of the local point cloud regions are statistically analyzed to form the target point cloud package.

3. The deep point cloud based robot vision servo control method of claim 2, wherein, The specific steps for constructing the target state description data are as follows: The local point cloud regions are quality-labeled based on the target point cloud package and aggregated according to spatial adjacency. Local point cloud regions with abnormal missing trends are filtered out to obtain the target object point cloud set. Spatial state information and geometric region information are extracted from the point cloud set of the target object, and associated and encapsulated with region identifiers and acquisition time sequence identifiers to form target state description data.

4. The robot vision servo control method based on depth point clouds as described in claim 1 or 3, characterized in that, The specific steps for generating the viewpoint maintenance index are as follows: Based on the geometric region information in the target state description data, combined with the observation direction, observation distance and field of view coverage of the depth camera, the coverage change, boundary change and center position change of the target object region in the current field of view are analyzed to obtain the current observation state data; Based on the current observation data, assess the current observation posture's ability to maintain continuous observation of the target area and generate a viewpoint maintenance index.

5. The robot vision servo control method based on depth point cloud as described in claim 1, characterized in that, The specific steps for constructing the point cloud confidence servo error output term set are as follows: The input fusion layer receives spatial state information, geometric region information, viewpoint maintenance index, robot current end effector pose, and local point cloud region quality markers, and collects error information according to region identifier and acquisition time sequence identifier to form a unified error input set. The unified error input set is input into the state alignment layer for position and attitude mapping, and the geometric region information is matched to form the original deviation set; The confidence modulation layer combines local point cloud region quality markers to pre-adjust the participation intensity of the original bias set, and the view compensation layer combines view maintenance index to perform continuous compensation and orientation correction to form a compensated bias set. The output fusion layer correlates and fuses the deviation items in the compensation deviation set, and outputs a set of point cloud confidence servo error output items.

6. The robot vision servo control method based on depth point clouds as described in claim 5, characterized in that, The specific process for generating point cloud confidence servo error data and active viewpoint adjustment commands is as follows: The deviation items, region identifiers, acquisition time sequence identifiers, and local point cloud region quality markers are extracted from the point cloud confidence servo error output item set and matched and integrated to form a weighted input deviation set. Based on the weighted input bias set combined with local point density, depth continuity, occlusion ratio and missing trend, confidence redistribution and influence weight allocation are performed on each bias item to form point cloud confidence servo error data; Based on the attitude deviation and regional alignment deviation after participation intensity adjustment and influence weight allocation, and combined with the viewpoint maintenance index, the current observation state is analyzed for observation offset and the viewpoint adjustment level is classified, and an active viewpoint adjustment command is formed.

7. The robot vision servo control method based on depth point clouds as described in claim 1, characterized in that, The specific process for calculating the terminal approximation control quantity is as follows: Based on the deviation items in the point cloud confidence servo error data, the data is merged and organized in combination with the region identifier and the acquisition time sequence identifier to form the end control input set; Based on the end-effector control input set and combined with the robot's current end-effector pose and end-effector operation direction, the execution direction mapping and coordination of each deviation term are performed to form the end-effector approximation control quantity.

8. The robot vision servo control method based on depth point cloud as described in claim 1, characterized in that, The specific process for calculating the observed attitude correction is as follows: Read the active viewpoint adjustment command, and combine it with the area identifier, acquisition time sequence identifier, viewpoint maintenance index and current observation status to perform alignment, merging and correlation matching to form the observation correction input set; Based on the observation correction input set, the active viewpoint adjustment command execution is mapped and coordinated to form the observation attitude correction quantity.

9. The robot vision servo control method based on depth point clouds as described in claim 7 or 8, characterized in that, The fusion is the current execution control package, and the specific process is as follows: Read the end-point approximation control quantity and the observed attitude correction quantity, align, merge and integrate them to form a control fusion input set; Based on the control fusion input set, the timing, direction and rhythm of the end-point approximation control quantity and the observed attitude correction quantity are coordinated and associated with the current execution control package.

10. The robot vision servo control method based on depth point clouds as described in claim 1, characterized in that, The specific process for outputting the target alignment state and the end-effector in-place state is as follows: Based on point cloud confidence servo error data, viewpoint maintenance index, current execution control package and current observation status data, the deviation term is continuously tracked and status judgment is performed to form a status discrimination input set; Based on the state discrimination input set, the point cloud confidence servo error data, view maintenance index and current observation state data are jointly discriminated, and the target alignment state and end position state are output when the joint discrimination conditions are met.

Images