A visual servoing-based robot pose closed-loop guidance method

Abstract

The application relates to the technical field of mechanical control, and particularly discloses a mechanical arm pose closed-loop guiding method based on visual servoing, which comprises the following steps: acquiring historical grabbing data of successful multiple times of grabbing target objects by a mechanical arm, real-time pose data of the mechanical arm, real-time spatial position of the target object and target pose data; determining object visual data and angle distortion mapping data of the target object; determining a mapping coordinate set of each time of successfully grabbing the target object by the mechanical arm, and constructing a collaborative correlation model; collecting real-time spatial images of the target object, determining judgment labels, predicting grabbing values and real-time pose guiding data. The method can realize full-process closed-loop intelligent guiding of the pose of the mechanical arm, solves the problems of target recognition distortion caused by visual angle distortion and low success rate of grabbing in a complex scene, improves grabbing precision and scene robustness under multiple visual angles, reduces on-site calibration cost, realizes intelligent reuse of historical grabbing experience, and avoids invalid grabbing actions.

Description

Technical Field

[0001] This invention relates to the field of mechanical control technology, and in particular to a closed-loop guidance method for the pose of a robotic arm based on visual servoing. Background Technology

[0002] Early robotic arm grasping operations relied entirely on manual teaching and programming, enabling only repetitive grasping at fixed points without visual feedback. This resulted in poor adaptability to scenarios with changing target positions or viewpoints. Subsequent open-loop guidance technologies based on fixed vision required calibration boards for camera and robotic arm alignment, limiting operation to preset fixed viewpoints. Perspective distortion caused by viewpoint changes directly led to a sharp drop in target positioning accuracy, making it unsuitable for complex multi-view grasping scenarios. Later visual servoing technologies primarily used raw image features for pose calculation, failing to fully leverage and reuse historical successful grasping experience. The lack of a pre-judgment mechanism for grasping performance resulted in significant fluctuations in grasping success rates under different working conditions, failing to meet the high-precision, high-robustness grasping requirements of industrial scenarios.

[0003] Therefore, this invention proposes a closed-loop guidance method for robot arm pose based on visual servoing. Summary of the Invention

[0004] This invention provides a visual servoing-based closed-loop guidance method for robotic arm pose. By acquiring historical grasping data of multiple successful grasps of a target object by the robotic arm, real-time pose data of the robotic arm, real-time spatial position of the target object, and target pose data, it determines the allowable angle range and the optimal straight-line distance for each allowable relative angle within the allowable angle range. It also determines the object's visual data and the target object's angular distortion mapping data, and determines the target bounding box, mapped boundary image, and mapped coordinate set for each successful grasp of the target object by the robotic arm. Furthermore, it constructs a collaborative association model, using a first visual sensor to acquire real-time spatial images of the target object, and determines the judgment label, predicted grasping value, and real-time pose guidance data. This method enables intelligent closed-loop guidance of the robotic arm's pose throughout the entire process, solving the problems of target recognition distortion caused by viewpoint distortion and low grasping success rate in complex scenes. It improves grasping accuracy and scene robustness under multiple views, reduces on-site calibration costs, enables intelligent reuse of historical grasping experience, and effectively avoids the risk of executing invalid grasping actions.

[0005] This invention provides a visual servoing-based closed-loop guidance method for robotic arm pose, comprising: S1: Obtain historical grasping data of the robotic arm successfully grasping the target object multiple times, obtain the real-time pose data of the robotic arm, and obtain the real-time spatial position and pose data of the target object. S2: Based on the historical grasping data of all successful grasping of the target object by the robotic arm, determine the allowable angle range and the optimal straight-line distance for each allowable relative angle within the allowable angle range, and determine the object visual data and the angular distortion mapping data of the target object; S3: Based on the historical grasping data of all successful grasping of target objects by the robotic arm, object visual data, and target object angle distortion mapping data, determine the target bounding box, mapped boundary image, and mapped coordinate set of each successful grasping of target objects by the robotic arm, and construct a collaborative association model. S4: Based on the real-time spatial image of the target object acquired by the first vision sensor, and based on the real-time spatial image, angle distortion mapping data, the optimal straight-line distance of all allowed relative angles within the allowed angle range, and the collaborative association model, determine the judgment label, predict the grasp value, and obtain real-time pose guidance data.

[0006] Preferably, a visual servoing-based robotic arm pose closed-loop guidance method acquires historical grasping data of the robotic arm successfully grasping a target object multiple times, acquires real-time pose data of the robotic arm, and acquires real-time spatial position and pose data of the target object, including: Acquire historical grasping data of the robotic arm successfully grasping the target object multiple times. The historical grasping data includes: historical end images collected based on the second vision sensor, historical straight-line distance between the second vision sensor and the target object, historical relative angle, and grasping evaluation value. Acquire real-time pose data of the robotic arm, and acquire real-time spatial position and pose data of the target object.

[0007] Preferably, a visual servoing-based robotic arm pose closed-loop guidance method, based on historical grasping data of all successful grasping of target objects by the robotic arm, determines the allowable angle range and the optimal straight-line distance for each allowable relative angle within the allowable angle range, including: Based on the historical relative angles from all successful grasping data of the robotic arm, a set of relative angles is determined. Extract the historical straight-line distance corresponding to the highest crawling evaluation value among all historical crawling data for each historical relative angle in the relative angle set, and use it as the optimal straight-line distance for the historical relative angle; Based on the historical relative angles in the historical capture data of all successful captures of the target object, the allowable angle range is determined, which includes multiple allowable relative angles; Based on the optimal straight-line distance of all historical relative angles in the relative angle set, the allowable relative angle corresponding to each historical relative angle within the allowable angle range is assigned a value, and interpolation is performed on each allowable relative angle within the allowable angle range other than all historical relative angles to determine the optimal straight-line distance of each allowable relative angle within the allowable angle range.

[0008] Preferably, a visual servoing-based robotic arm pose closed-loop guidance method determines object visual data and target object angular distortion mapping data, including: The robotic arm is controlled to move to each allowable relative angle within the allowable angle range and the optimal straight-line distance of the allowable relative angle, and angular visual images of the allowable relative angles are acquired based on a second vision sensor; The allowable relative angle with a value of 0 within the allowable angle range is defined as the standard relative angle, and the visual image of the angle corresponding to the standard relative angle is defined as the standard visual image. Based on the angular visual images of all permissible relative angles within the permissible angle range, determine the object visual data of the target object; The standard visual image of the standard relative angle and the angular visual images of all allowed relative angles in the object visual data are input into the perspective transformation model to determine the angular distortion mapping data of the target object. The angular distortion mapping data includes the perspective distortion vector of each allowed relative angle within the allowed angle range.

[0009] Preferably, a visual servoing-based robotic arm pose closed-loop guidance method, based on historical grasping data of all successful grasping of target objects by the robotic arm, object visual data, and angular distortion mapping data of the target object, determines the target bounding box, mapped boundary image, and mapped coordinate set for each successful grasping of the target object by the robotic arm, and constructs a collaborative association model, including: The historical end image in the historical grasping data of each successful grasping of the target object by the robotic arm is preprocessed, and the target bounding box of the target object in the historical end image is determined by the preprocessed historical end image. Based on the perspective distortion vector of the allowed relative angle corresponding to the historical relative angle of each successful grasp of the target object by the robotic arm in the angle distortion mapping data, the historical end image and target bounding box of the second successful grasp of the target object are mapped to determine the mapped boundary image of the target object for each successful grasp of the robotic arm. Feature point detection and effective point filtering are performed on the mapping boundary image of the target object successfully grasped by the robotic arm each time to determine the mapping feature data of the target object successfully grasped by the robotic arm each time. The mapping feature data includes multiple mapping feature points and the target mapping coordinates of each mapping feature point. Based on the historical end image and target bounding box of each successful grasp of the target object by the robotic arm, the percentage of images of the target object successfully grasped each time is calculated; Feature point detection and effective point filtering are performed on the standard visual image of the target object at a standard relative angle in the object visual data to determine the standard feature data of the target object. The standard feature data includes multiple standard feature points of the standard visual image and the standard coordinates of each standard feature point. Based on the principle of spatial nearest neighbor, feature point matching is performed on the standard feature data of the target object and the mapping feature data of the target object for each successful capture to determine the set of mapping coordinates of the target object for each successful capture. Based on the target bounding box, mapped boundary image, and mapped coordinate set of each successfully grasped target object by the robotic arm, the mapping density value of the target object for each successful grasp is calculated. Based on the set of mapped coordinates of all successful grasps of the target object by the robotic arm, calculate the feature distribution value of each successful grasp of the target object by the robotic arm; The collaborative association model is constructed by taking the image proportion, mapping density, and feature distribution values ​​of all successfully captured target objects as inputs and taking the capture evaluation values ​​from the historical capture data of all successfully captured target objects as outputs.

[0010] Preferably, a visual servoing-based robotic arm pose closed-loop guidance method, based on real-time spatial images of the target object acquired by a first visual sensor, and based on the real-time spatial images, angle distortion mapping data, the optimal straight-line distance of all allowable relative angles within the allowable angle range, and a collaborative association model, determines the judgment label, predicted grasping value, and real-time pose guidance data, including: Real-time spatial images of the target object are acquired based on the first visual sensor, and the real-time acquisition angle is determined based on the position of the first visual sensor and the position of the target object. The real-time spatial image is preprocessed, and the target detection is performed on the preprocessed real-time spatial image to determine the spatial bounding box based on the real-time acquisition angle. Determine whether the real-time acquired angle falls within the allowable angle range of the angle distortion mapping data, and determine the judgment label, which includes "belongs to" and "does not belong to". Based on the judgment label, real-time pose data, target pose data, real-time acquisition angle, angle distortion mapping data, the optimal straight-line distance of all allowed relative angles within the allowed angle range, and the collaborative association model, the real-time pose guidance data is determined.

[0011] Preferably, a visual servoing-based robotic arm pose closed-loop guidance method determines real-time pose guidance data based on judgment labels, real-time pose data, target pose data, real-time acquired angles, angle distortion mapping data, the optimal straight-line distance of all allowable relative angles within the allowable angle range, and a collaborative association model, including: If the label is determined to be a real-time spatial image, the spatial bounding box is mapped based on the perspective distortion vector of the allowed relative angle in the angle distortion mapping data corresponding to the real-time acquired angle, and the real-time mapped image is determined. Feature point detection and effective point filtering are performed on the real-time mapped image to determine the real-time mapping data, which includes multiple real-time feature points and the real-time mapped coordinates of each real-time feature point. Based on the spatial nearest neighbor principle, the standard feature data and real-time mapping data of the target object are mapped to determine the real-time mapping set; Based on real-time spatial images and spatial bounding boxes, the real-time proportion value is calculated. At the same time, based on spatial bounding boxes, real-time mapped images, and real-time mapping sets, the real-time density value is calculated. The real-time distribution value is calculated based on the real-time mapping set and the mapping coordinate set of all successful grasps of the target object by the robotic arm. Input the real-time percentage value, real-time density value, and real-time distribution value into the collaborative association model to determine the predicted capture value; If the predicted grasp value is greater than or equal to the preset grasp value threshold, the real-time pose guidance data is determined based on the real-time pose data and the target pose data.

[0012] Preferably, a visual servoing-based robotic arm pose closed-loop guidance method determines real-time pose guidance data based on judgment labels, real-time pose data, target pose data, real-time acquired angles, angle distortion mapping data, the optimal straight-line distance of all allowable relative angles within the allowable angle range, and a collaborative association model, including: If the label is determined to be non-belonging, or if the label is determined to be belonging and the predicted crawl value is less than the preset crawl value threshold; Based on each allowed relative angle within the allowed angle range, the optimal straight-line distance of the allowed relative angle, and the real-time spatial position of the target object, the target spatial position for each allowed relative angle within the allowed angle range is determined; Based on the target spatial position of each allowable relative angle within the allowable angle range and the real-time pose data of the robotic arm, preset guidance data for each allowable relative angle within the allowable angle range is determined. The preset guidance data includes a preset planned path and preset indirect pose guidance data. With the goal of minimizing the planned path and the pose adjustment, the system selects real-time relative angles and real-time indirect pose guidance data based on the preset indirect pose guidance data of all allowed relative angles within the allowable angle range. Based on real-time indirect pose guidance data, the robotic arm is controlled to move to the target spatial position corresponding to the allowable relative angle in real-time relative angle. Real-time end-effector images are acquired based on the second vision sensor, and real-time precise pose guidance data is determined. Based on real-time indirect pose guidance data and real-time precise pose guidance data, the real-time pose guidance data is determined.

[0013] The beneficial effects of this invention compared to existing technologies are as follows: By acquiring historical grasping data of multiple successful grasping of target objects by the robotic arm, real-time pose data of the robotic arm, real-time spatial position of the target object, and target pose data, the allowable angle range and the optimal straight-line distance for each allowable relative angle within the allowable angle range are determined. Furthermore, the visual data of the object and the angular distortion mapping data of the target object are determined. The target bounding box, mapped boundary image, and mapped coordinate set for each successful grasping of the target object by the robotic arm are determined, and a collaborative association model is constructed. Based on real-time spatial images of the target object acquired by the first visual sensor, the judgment label, predicted grasping value, and real-time pose guidance data are determined. This enables closed-loop intelligent guidance of the robotic arm's pose throughout the entire process, solving the problems of target recognition distortion caused by perspective distortion and low grasping success rate in complex scenes. It improves grasping accuracy and scene robustness under multiple perspectives, reduces on-site calibration costs, realizes intelligent reuse of historical grasping experience, and effectively avoids the risk of executing invalid grasping actions.

[0014] Other features and advantages of the invention will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention may be realized and obtained by means of the structures particularly pointed out in this application.

[0015] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0016] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings: Figure 1 This is a flowchart of a visual servoing-based robotic arm pose closed-loop guidance method according to an embodiment of the present invention. Detailed Implementation

[0017] The preferred embodiments of the present invention will be described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit the present invention. Example 1:

[0018] This invention provides a visual servoing-based closed-loop guidance method for robotic arm pose positioning, referencing... Figure 1 ,include: S1: Obtain historical grasping data of the robotic arm successfully grasping the target object multiple times, obtain the real-time pose data of the robotic arm, and obtain the real-time spatial position and pose data of the target object. S2: Based on the historical grasping data of all successful grasping of the target object by the robotic arm, determine the allowable angle range and the optimal straight-line distance for each allowable relative angle within the allowable angle range, and determine the object visual data and the angular distortion mapping data of the target object; S3: Based on the historical grasping data of all successful grasping of target objects by the robotic arm, object visual data, and target object angle distortion mapping data, determine the target bounding box, mapped boundary image, and mapped coordinate set of each successful grasping of target objects by the robotic arm, and construct a collaborative association model. S4: Based on the real-time spatial image of the target object acquired by the first vision sensor, and based on the real-time spatial image, angle distortion mapping data, the optimal straight-line distance of all allowed relative angles within the allowed angle range, and the collaborative association model, determine the judgment label, predict the grasp value, and obtain real-time pose guidance data.

[0019] In this embodiment, the robotic arm is a multi-degree-of-freedom automated execution device used to perform target object grasping and handling operations. Visual servoing refers to a control method that controls the movement of the robotic arm through image data collected by a visual sensor. Real-time pose data refers to the spatial position and attitude angle data of the robotic arm's end effector at the current moment, which is acquired in real time through the encoder and pose sensor built into the robotic arm. Target object refers to the object to be grasped by the robotic arm. Real-time spatial position refers to the spatial coordinate data of the target object in the current work scene. Target pose data refers to the standard pose data of the end effector required for the robotic arm to stably grasp the target object.

[0020] In this embodiment, the allowable angle range refers to the relative angle interval between the vision sensor and the target object that allows the robotic arm to stably identify and successfully grasp the target object. It is the angular boundary that ensures the effectiveness of visual recognition and the success rate of grasping. The allowable relative angle refers to each discrete relative angle value between the optical axis of the vision sensor and the reference plane of the target object within the allowable angle range. The optimal straight-line distance refers to the optimal straight-line distance between the robotic arm's end effector and the target object for each allowable relative angle, where visual recognition distortion is minimized and the grasping success rate is highest. Object visual data refers to the basic visual recognition data of the target object, such as standard appearance features, size parameters, and texture features, extracted from historical grasping data. Angle distortion mapping data refers to a structured dataset characterizing the correspondence between the relative angle between the vision sensor and the target object and the degree of distortion in the target object image, clarifying the distortion rules and correction mapping relationships of the target object image under different relative angles.

[0021] In this embodiment, historical capture data, object visual data, and angle distortion mapping data of all successful captures are retrieved. First, the standard target bounding box of the target object is determined based on the object visual data. Then, for the relative angle corresponding to each successful capture, the distortion of the acquired target image is corrected by using the angle distortion mapping data to generate a mapped boundary image under the corresponding viewpoint. The relative angle, mapped boundary image features, and mapped coordinates in all historical capture data are calculated as input features, and the capture evaluation value of the corresponding capture case is used as the output target. Model training and parameter optimization are carried out to finally construct a collaborative association model.

[0022] In this embodiment, the first vision sensor, which has completed synchronous calibration, is activated to acquire real-time spatial images of the target object in the work scene. The acquired real-time spatial images are preprocessed and feature extracted. Then, the distortion of the real-time images is corrected by using angle distortion mapping data. The allowable relative angle and optimal straight-line distance corresponding to the current viewpoint are matched and determined. The corrected real-time proportion value, real-time density value, and real-time density value are then input into a pre-built collaborative association model. The model outputs the predicted grasping value corresponding to the current work state and determines the judgment label and real-time pose guidance data.

[0023] The beneficial effects of the above technology are as follows: By acquiring historical grasping data of multiple successful grasping of target objects by the robotic arm, real-time pose data of the robotic arm, real-time spatial position of the target object, and target pose data, the allowable angle range and the optimal straight-line distance for each allowable relative angle within the allowable angle range are determined. Furthermore, the object's visual data and the target object's angular distortion mapping data are determined. The target bounding box, mapped boundary image, and mapped coordinate set for each successful grasping of the target object by the robotic arm are determined, and a collaborative association model is constructed. Based on real-time spatial images of the target object acquired by the first visual sensor, the judgment label, predicted grasping value, and real-time pose guidance data are determined. This enables closed-loop intelligent guidance of the robotic arm's pose throughout the entire process, solving the problems of target recognition distortion caused by perspective distortion and low grasping success rate in complex scenes. It improves grasping accuracy and scene robustness under multiple perspectives, reduces on-site calibration costs, realizes intelligent reuse of historical grasping experience, and effectively avoids the risk of executing invalid grasping actions. Example 2:

[0024] Based on Example 1, a visual servoing-based robotic arm pose closed-loop guidance method acquires historical grasping data of the robotic arm successfully grasping a target object multiple times, acquires the real-time pose data of the robotic arm, and acquires the real-time spatial position and pose data of the target object, including: Acquire historical grasping data of the robotic arm successfully grasping the target object multiple times. The historical grasping data includes: historical end images collected based on the second vision sensor, historical straight-line distance between the second vision sensor and the target object, historical relative angle, and grasping evaluation value. Acquire real-time pose data of the robotic arm, and acquire real-time spatial position and pose data of the target object.

[0025] In this embodiment, the second vision sensor is a follow-up vision acquisition device mounted on the end effector of the robotic arm. It is rigidly connected to the end effector and moves synchronously with the robotic arm, enabling real-time acquisition of near-field images between the end effector and the target object. Historical grasping data comprises data from each successful grasp of the target object by the robotic arm. Historical end-effector images are end-effector images of past successful grasping operations acquired by the second vision sensor mounted on the end effector, directly extracted from the historical grasping data. Historical straight-line distance is the linear spatial distance between the second vision sensor and the target object's reference plane in each past successful grasping operation, calculated through visual ranging or robotic arm kinematics. Historical relative angle is the spatial angle between the optical axis of the second vision sensor corresponding to each successful grasping historical image and the target object's reference plane, characterizing the relative posture relationship between the end-effector view and the target object. The grasping evaluation value is a standardized numerical value quantifying the overall effect of each grasping operation, used to characterize the quality of the corresponding grasping case. First, the rigid installation and time synchronization calibration of the second vision sensor and the robotic arm end effector are completed to ensure that the sensor acquisition time is perfectly aligned with the robotic arm pose acquisition time. Then, the historical operation database of the robotic arm control system is retrieved, and all successful object grasping cases are selected. From the corresponding storage location of each successful case, the historical end effector image synchronously acquired by the second vision sensor, the historical straight-line distance recorded synchronously, the historical relative angle, and the grasping evaluation value generated after the corresponding grasping operation are extracted. All data of the same grasping case are aligned in time dimension and verified for integrity to form a complete historical grasping dataset.

[0026] In this embodiment, real-time pose data refers to the position coordinates and attitude angles of the robotic arm's end effector in the global spatial coordinate system of the work scene at the current moment. Position coordinates include spatial values ​​along the x, y, and z axes, and attitude angles include angle values ​​in the roll, pitch, and yaw dimensions. The real-time spatial position of the target object refers to the three-dimensional coordinate data of the target object in the global spatial coordinate system of the work scene at the current moment, representing the specific location of the target object in the scene. Target pose data refers to the standard pose data required for the robotic arm's end effector to complete stable and accurate grasping of the target object, including the optimal grasping position and grasping posture of the end effector, serving as the target pose reference for the robotic arm's grasping operation. The robotic arm's pose acquisition module, global positioning module, and synchronous clock system are activated to ensure complete time consistency across all data acquisition devices. Then, at a preset acquisition frequency, the real-time pose data of the robotic arm's end effector and the real-time spatial position of the target object in the scene are synchronously acquired. Simultaneously, pre-calibrated target grasping standard pose data are retrieved, or the target pose data is calculated based on the real-time acquired target object spatial posture.

[0027] The beneficial effects of the above technologies are: acquiring historical grasping data of the robotic arm successfully grasping the target object multiple times, acquiring the real-time pose data of the robotic arm, acquiring the real-time spatial position and pose data of the target object, which can provide data support for determining the allowable angle range and the optimal straight-line distance of each allowable relative angle within the allowable angle range. Example 3:

[0028] Based on Example 2, a visual servoing-based robotic arm pose closed-loop guidance method, based on historical grasping data of all successful grasping of target objects by the robotic arm, determines the allowable angle range and the optimal straight-line distance for each allowable relative angle within the allowable angle range, including: Based on the historical relative angles from all successful grasping data of the robotic arm, a set of relative angles is determined. Extract the historical straight-line distance corresponding to the highest crawling evaluation value among all historical crawling data for each historical relative angle in the relative angle set, and use it as the optimal straight-line distance for the historical relative angle; Based on the historical relative angles in the historical capture data of all successful captures of the target object, the allowable angle range is determined, which includes multiple allowable relative angles; Based on the optimal straight-line distance of all historical relative angles in the relative angle set, the allowable relative angle corresponding to each historical relative angle within the allowable angle range is assigned a value, and interpolation is performed on each allowable relative angle within the allowable angle range other than all historical relative angles to determine the optimal straight-line distance of each allowable relative angle within the allowable angle range.

[0029] In this embodiment, the relative angle set refers to the set of angles formed after deduplicating the historical relative angles corresponding to all historical successful capture cases. It completely covers all viewpoints that have been verified as successfully captureable. The process involves retrieving historical capture data for all previously successful captures of the target object, iterating through each historical capture data point, extracting the corresponding historical relative angle values ​​from each data point, and then deduplicating all extracted historical relative angles to remove duplicate angle values, thus forming a complete relative angle set.

[0030] In this embodiment, the optimal straight-line distance refers to the straight-line distance that achieves the best grasping effect for each historical relative angle, and is the optimal position reference for the robotic arm's grasping at that viewpoint. Each historical relative angle in the set of relative angles is retrieved. For each historical relative angle, all historical grasping cases corresponding to that angle are selected from the full set of historical grasping data. Then, from all the selected cases, the historical grasping data with the highest grasping evaluation value is found, and the corresponding historical straight-line distance is extracted from that data. Finally, this distance is determined as the optimal straight-line distance for that historical relative angle. This process is repeated to complete the calibration of the optimal straight-line distance for all historical relative angles in the set of relative angles.

[0031] In this embodiment, the allowable angle range refers to the relative angle interval within which the robotic arm can stably identify and successfully grasp target objects, verified by historical grasping data. It serves as the visual boundary to ensure the safety and success rate of grasping operations. The allowable relative angle refers to each angle value discretized within the allowable angle range using a fixed step size, covering all continuous angle points from the lower limit to the upper limit of the interval. Historical grasping data for all successful grasps of target objects is retrieved, and the historical relative angles corresponding to all cases are extracted. Numerical distribution statistics are performed on all historical relative angles to determine the minimum and maximum relative angle values ​​corresponding to all successful grasping cases. The minimum value is used as the lower limit of the interval, and the maximum value as the upper limit, defining the initial angle interval. Then, combining the robotic arm's motion limits and the effective boundary of visual recognition, the initial interval is boundary-corrected, eliminating invalid angles exceeding the robotic arm's motion range, ultimately determining the complete allowable angle range. Finally, the allowable angle range is discretized using a preset fixed angle step size to obtain multiple continuous allowable relative angles within the interval.

[0032] In this embodiment, the assignment process refers to directly assigning the calibrated optimal straight-line distance corresponding to the historical relative angle to the allowable relative angle within the allowable angle range that completely coincides with the historical angle. The interpolation process refers to calculating the optimal straight-line distance corresponding to the point location based on the optimal straight-line distance of adjacent historical relative angles for allowable relative angles without historical data within the allowable angle range, thereby completing the parameters for points without historical data. The calibrated optimal straight-line distances corresponding to all historical relative angles in the relative angle set are retrieved. Each allowable relative angle within the allowable angle range is traversed, and it is first determined whether the allowable relative angle exists in the relative angle set. If an overlapping historical relative angle exists... The optimal straight-line distance corresponding to the historical relative angle is directly assigned to the allowable relative angle to complete the parameter calibration of the point. If the allowable relative angle does not exist in the relative angle set, two historical relative angles with calibration values ​​that are adjacent to the point are selected. Based on the optimal straight-line distance and angle interval of the two adjacent points, the optimal straight-line distance corresponding to the allowable relative angle is calculated by interpolation algorithm to complete the parameter completion of the point. The parameter calibration of all allowable relative angles within the allowable angle range is completed iteratively, and finally the optimal straight-line distance corresponding to each allowable relative angle within the allowable angle range is determined.

[0033] The beneficial effects of the above technologies are as follows: Based on the historical grasping data of all successful grasping of target objects by the robotic arm, the allowable angle range and the optimal straight-line distance of each allowable relative angle within the allowable angle range can be determined. This can achieve full coverage of the optimal grasping parameters for all points within the allowable angle range, realize the standardized reuse of historical successful grasping experience across the entire range, and reduce on-site calibration costs. Example 4:

[0034] Based on Example 3, a visual servoing-based robotic arm pose closed-loop guidance method determines the object's visual data and the target object's angular distortion mapping data, including: The robotic arm is controlled to move to each allowable relative angle within the allowable angle range and the optimal straight-line distance of the allowable relative angle, and angular visual images of the allowable relative angles are acquired based on a second vision sensor; The allowable relative angle with a value of 0 within the allowable angle range is defined as the standard relative angle, and the visual image of the angle corresponding to the standard relative angle is defined as the standard visual image. Based on the angular visual images of all permissible relative angles within the permissible angle range, determine the object visual data of the target object; The standard visual image of the standard relative angle and the angular visual images of all allowed relative angles in the object visual data are input into the perspective transformation model to determine the angular distortion mapping data of the target object. The angular distortion mapping data includes the perspective distortion vector of each allowed relative angle within the allowed angle range.

[0035] In this embodiment, the angular visual image refers to the visual image containing the complete area of ​​the target object collected by the second visual sensor installed at the end of the robotic arm for each allowed relative angle.

[0036] In this embodiment, the angular visual image is a visual image containing the complete area of ​​the target object, acquired by the second vision sensor at each allowed relative angle and the optimal straight-line distance position. The pre-calibrated allowed angle range and the optimal straight-line distance corresponding to each allowed relative angle within the range are retrieved. Then, for each allowed relative angle, based on the fixed spatial position of the target object, the spatial coordinates and posture that the robotic arm's end effector needs to reach are calculated. The robotic arm is controlled to precisely move to that position, ensuring that the relative angle and distance between the second vision sensor and the target object perfectly match the current allowed relative angle and the corresponding optimal straight-line distance. The second vision sensor is then activated to acquire an image containing the complete area of ​​the target object from the current viewpoint, completing the angular visual image acquisition corresponding to that allowed relative angle. This process is repeated until all allowed relative angles within the allowed angle range have been acquired.

[0037] In this embodiment, the standard relative angle refers to the relative angle where the optical axis of the visual sensor is completely perpendicular to the reference plane of the target object, with a value of 0. At this angle, the target object exhibits no perspective distortion and serves as the reference angle for image comparison and distortion correction across all viewpoints. The standard visual image is the reference image of the target object acquired by the second visual sensor at the standard relative angle, showing no perspective distortion.

[0038] In this embodiment, angular visual images corresponding to all allowed relative angles within the allowed angle range are retrieved to form object visual data of the target object.

[0039] In this embodiment, the perspective transformation model is an algorithmic model used to realize the perspective projection transformation of images from different viewpoints and to quantify the distortion features of images from different viewpoints relative to a reference image. The perspective distortion vector is a feature vector that quantifies the distortion offset, scaling, rotation, and perspective deformation of the target image relative to the standard visual image at the corresponding allowed relative angle, and is the core parameter for image distortion correction. The standard visual image corresponding to the standard relative angle is extracted from the object visual data, and simultaneously, all angle visual images corresponding to the allowed relative angles within the allowed angle range are retrieved. The standard visual image and all angle visual images are then synchronously input into the pre-constructed perspective transformation model. The model first extracts feature points from the standard visual image and each angle visual image, establishes a one-to-one matching relationship for the feature points, and then calculates the perspective transformation matrix of each angle visual image relative to the standard visual image based on the matched feature point pairs. The perspective transformation matrix is ​​then decomposed to extract the perspective distortion features at the corresponding viewpoint, generating a perspective distortion vector corresponding to each allowed relative angle. Finally, all perspective distortion vectors corresponding to the allowed relative angles are integrated in angular order to form the angular distortion mapping data of the target object.

[0040] In this embodiment, the perspective transformation model is based on the imaging rules of three-dimensional spatial perspective projection. It establishes a pixel coordinate mapping relationship between two-dimensional image planes to realize perspective projection transformation and distortion reduction of images from different viewpoints. Its core logic is to obtain the perspective transformation matrix between the two image planes by matching the same feature points between the reference image and the image to be transformed. This matrix can completely characterize all the features of translation, rotation, scaling, shearing, and perspective distortion caused by changes in viewpoint. The perspective transformation matrix can project the image to be transformed from any viewpoint onto the viewpoint plane of the reference image, thus correcting perspective distortion. Simultaneously, by decomposing the parameters of the perspective transformation matrix, the perspective distortion feature vector of the image at the corresponding viewpoint can be quantified, providing a precise quantitative basis for image distortion correction at different viewpoints.

[0041] The beneficial effects of the above technologies are as follows: determining the visual data of the object and the angular distortion mapping data of the target object can improve the accuracy and robustness of target recognition under different perspectives, and provide a precise distortion correction benchmark for building collaborative association models and real-time grasping guidance. Example 5:

[0042] Based on Example 4, a visual servoing-based robotic arm pose closed-loop guidance method is proposed. This method uses historical grasping data from all successful object grasping attempts, object visual data, and target object angular distortion mapping data to determine the target bounding box, mapped boundary image, and mapped coordinate set for each successful object grasping attempt. A collaborative association model is then constructed, including: The historical end image in the historical grasping data of each successful grasping of the target object by the robotic arm is preprocessed, and the target bounding box of the target object in the historical end image is determined by the preprocessed historical end image. Based on the perspective distortion vector of the allowed relative angle corresponding to the historical relative angle of each successful grasp of the target object by the robotic arm in the angle distortion mapping data, the historical end image and target bounding box of the second successful grasp of the target object are mapped to determine the mapped boundary image of the target object for each successful grasp of the robotic arm. Feature point detection and effective point filtering are performed on the mapping boundary image of the target object successfully grasped by the robotic arm each time to determine the mapping feature data of the target object successfully grasped by the robotic arm each time. The mapping feature data includes multiple mapping feature points and the target mapping coordinates of each mapping feature point. Based on the historical end image and target bounding box of each successful grasp of the target object by the robotic arm, the percentage of images of the target object successfully grasped each time is calculated; Feature point detection and effective point filtering are performed on the standard visual image of the target object at a standard relative angle in the object visual data to determine the standard feature data of the target object. The standard feature data includes multiple standard feature points of the standard visual image and the standard coordinates of each standard feature point. Based on the principle of spatial nearest neighbor, feature point matching is performed on the standard feature data of the target object and the mapping feature data of the target object for each successful capture to determine the set of mapping coordinates of the target object for each successful capture. Based on the target bounding box, mapped boundary image, and mapped coordinate set of each successfully grasped target object by the robotic arm, the mapping density value of the target object for each successful grasp is calculated. Based on the set of mapped coordinates of all successful grasps of the target object by the robotic arm, calculate the feature distribution value of each successful grasp of the target object by the robotic arm; The collaborative association model is constructed by taking the image proportion, mapping density, and feature distribution values ​​of all successfully captured target objects as inputs and taking the capture evaluation values ​​from the historical capture data of all successfully captured target objects as outputs.

[0043] In this embodiment, preprocessing refers to the standardization process of denoising, grayscale conversion, and contrast enhancement on the historical end image to eliminate image noise and illumination interference and improve the accuracy of target detection. Target detection refers to the process of identifying and locating the target object region in the image using a target detection algorithm. The target bounding box is a rectangular box that completely selects the target object region in the historical end image, containing the pixel coordinates and size information of the target object in the image. All historical capture data of successful target object captures are retrieved, and the corresponding historical end image is extracted for each data point. Each historical end image is preprocessed sequentially to eliminate interference factors such as illumination changes and image noise. The preprocessed image is then input into a pre-trained target detection model to identify and locate the target object region in the image, outputting a rectangular box that completely selects the target object region, and finally determining the target bounding box corresponding to each successful capture.

[0044] In this embodiment, the mapped boundary image refers to the target object boundary image that, after distortion correction, is aligned with the viewpoint of a standard visual image. The historical relative angle corresponding to each successful capture is retrieved. Then, from the pre-constructed angle distortion mapping data, the allowable relative angle corresponding to that historical relative angle is matched, and the corresponding perspective distortion vector is extracted. Based on this perspective distortion vector, a perspective transformation is performed on the corresponding captured historical end image and the target bounding box to correct the perspective distortion caused by the viewpoint, aligning the image and bounding box to the reference viewpoint of the standard relative angle, ultimately generating the mapped boundary image corresponding to each successful capture.

[0045] In this embodiment, feature point detection refers to the process of extracting key points representing core features such as the contour and texture of a target object from an image using a feature detection algorithm. Effective point filtering refers to the process of removing falsely detected, redundant, and repetitive feature points located in the background region, and retaining stable and effective feature points on the target object. Mapped feature data is a structured dataset that integrates effective feature points and their coordinates. Mapped feature points are the effective feature key points retained after filtering. Target mapped coordinates refer to the pixel coordinates of each mapped feature point in the mapped boundary image. The mapped boundary image corresponding to each successful capture is retrieved, and feature point detection is performed on the image using a feature detection algorithm to extract all initial feature points. Then, effective point filtering is performed on the initial feature points, removing invalid feature points that are redundant, repetitive, and unstable in the background region, and retaining stable and effective feature points located within the target object region as mapped feature points. Simultaneously, the pixel coordinates of each mapped feature point in the image are extracted as target mapped coordinates. Finally, all mapped feature points and their corresponding target mapped coordinates are integrated to form the mapped feature data corresponding to each successful capture.

[0046] In this embodiment, based on the historical end image and target bounding box of each successful grasp of the target object by the robotic arm, the percentage of images of the target object successfully grasped each time is calculated. The calculation formula is expressed as follows: ; in, This represents the area of ​​the historical end image of the robotic arm when it successfully grasps the target object for the i-th time. This represents the area of ​​the bounding box of the target in the historical end image of the i-th successful grasp of the target object by the robotic arm.

[0047] In this embodiment, the standard feature data is a benchmark structured dataset that integrates effective feature points and their coordinates from a standard visual image. Standard feature points are the effective core feature points extracted from the standard visual image. Standard coordinates are the pixel coordinates of each standard feature point in the standard visual image. From the object visual data of the target object, the standard visual image corresponding to the standard relative angle is extracted. Feature point detection is performed on the standard visual image using a feature detection algorithm to extract all initial standard feature points in the image. Then, the initial standard feature points are filtered for effectiveness, eliminating invalid and interfering feature points, and retaining the stable and effective core feature points on the target object as standard feature points. Simultaneously, the pixel coordinates of each standard feature point in the standard visual image are extracted as standard coordinates. Finally, all standard feature points and their corresponding standard coordinates are integrated to form the standard feature data of the target object.

[0048] In this embodiment, the spatial nearest neighbor principle refers to the matching rule in the image pixel coordinate system that determines the mapping feature point that is spatially closest to the standard feature point as the corresponding matching point of the standard feature point. The standard feature data of the target object and the mapping feature data corresponding to each successful capture are retrieved. Each standard feature point in the standard feature data is traversed, and based on the spatial nearest neighbor principle, the mapping feature point that is spatially closest to the standard feature point in the corresponding captured mapping feature data is found, establishing a one-to-one correspondence between the two feature points. Then, the target mapping coordinates of all matched mapping feature points are extracted and arranged and integrated according to the fixed order of the standard feature points, ultimately forming the set of mapping coordinates corresponding to each successful capture.

[0049] In this embodiment, the set of mapped coordinates of the target object successfully grasped by the robotic arm for the i-th time is represented as: ; Let represent the standard coordinates of the 1st, jth, and N2th standard feature points in the standard feature data, respectively. This represents the mapped coordinates based on the 1st, jth, and N2th standard feature points in the mapped feature data of the robotic arm's i-th successful grasp of the target object. If no corresponding mapped coordinates are found for the j-th standard feature point in the standard feature data, then... .

[0050] In this embodiment, the mapping density value of the target object is calculated for each successful grasping of the target object by the robotic arm, based on the target bounding box, the mapped boundary image, and the set of mapped coordinates. ; in, This represents the mapping density value of the target object successfully grasped by the robotic arm for the i-th time. This represents the area of ​​the mapped boundary image of the target object successfully grasped by the robotic arm in the i-th instance. This represents the first indicator function based on the j-th standard feature point in the mapped feature data of the i-th successful grasping of the target object by the robotic arm.

[0051] In this embodiment, This represents the spatial density of the target object successfully grasped by the robotic arm for the i-th time.

[0052] In this embodiment, , where represents the rate of change of density of the target object when the robotic arm successfully grasps it for the i-th time.

[0053] In this embodiment, based on the set of mapped coordinates of all successful grasps of the target object by the robotic arm, the feature distribution value of each successful grasp of the target object by the robotic arm is calculated. The calculation formula is expressed as follows: ; in, N represents the feature distribution value of the robotic arm successfully grasping the target object for the i-th time, and N3 represents the number of times the robotic arm successfully grasps the target object.

[0054] In this embodiment, the collaborative association model is a machine learning model that establishes a nonlinear mapping relationship between multi-dimensional visual features and grasping effect evaluation values. It serves as the core decision-making vehicle for guiding the robotic arm's grasping pose and predicting the grasping effect. The model collects the image proportion values, mapping density values, and feature distribution values ​​corresponding to all successful grasps of the target object to form the input feature dataset for model training. Simultaneously, it retrieves the grasping evaluation values ​​corresponding to all successful grasps to form the output label dataset for model training. The input feature dataset and output label dataset are then divided into training and validation sets. A suitable machine learning algorithm is selected to build the model framework. The training set data is used to complete the model training and fitting, while the validation set data is used to complete the model's accuracy verification and parameter optimization, ultimately constructing a complete collaborative association model.

[0055] In this embodiment, the collaborative association model is a supervised machine learning model for robotic arm visual servo grasping scenarios. It uses the standardized distortion-corrected image proportion, mapping density, and feature distribution values ​​from historical successful grasping cases as input variables, and the grasping evaluation value obtained by quantifying the actual operation effect of the corresponding cases as the output label. An adapted machine learning algorithm fits the nonlinear mapping relationship between multi-dimensional visual features and the overall grasping effect. During training, the model automatically learns the influence weights of different feature dimensions on the grasping effect, deeply explores the intrinsic correlation between image proportion features, density features, distribution, and grasping success rate and grasping stability. The model is fitted and trained using a pre-defined training set, and its accuracy is verified and hyperparameters are optimized using a validation set. After eliminating the risk of overfitting, a stable inference model is formed. The trained model can accurately output a grasping effect prediction score under the current grasping conditions based on the three homogeneous visual features input in real time, providing a quantitative intelligent judgment basis for the robotic arm's pose adjustment and grasping action decision-making.

[0056] The beneficial effects of the above technologies are as follows: Based on the historical grasping data of all successful grasping of target objects by the robotic arm, the visual data of the object, and the angular distortion mapping data of the target object, the target bounding box, the mapped boundary image, and the set of mapped coordinates of the target object successfully grasped by the robotic arm each time can be determined, and a collaborative correlation model can be constructed. This can achieve a deep correlation between visual features and grasping effect, improve the model's prediction accuracy and grasping robustness, and transform historical successful grasping experience into a quantifiable basis for intelligent decision-making. Example 6:

[0057] Based on Example 1, a visual servoing-based robotic arm pose closed-loop guidance method is proposed. This method uses a first visual sensor to acquire real-time spatial images of the target object. Based on the real-time spatial images, angle distortion mapping data, the optimal straight-line distance of all allowable relative angles within the allowable angle range, and a collaborative correlation model, it determines the judgment label, predicted grasping value, and real-time pose guidance data, including: Real-time spatial images of the target object are acquired based on the first visual sensor, and the real-time acquisition angle is determined based on the position of the first visual sensor and the position of the target object. The real-time spatial image is preprocessed, and the target detection is performed on the preprocessed real-time spatial image to determine the spatial bounding box based on the real-time acquisition angle. Determine whether the real-time acquired angle falls within the allowable angle range of the angle distortion mapping data, and determine the judgment label, which includes "belongs to" and "does not belong to". Based on the judgment label, real-time pose data, target pose data, real-time acquisition angle, angle distortion mapping data, the optimal straight-line distance of all allowed relative angles within the allowed angle range, and the collaborative association model, the real-time pose guidance data is determined.

[0058] In this embodiment, the first vision sensor is a vision acquisition device used to collect images of target objects within the work scene. It is pre-calibrated spatially and synchronized with the robotic arm control system, and can be installed at a fixed location within the work scene. The real-time spatial image is the work scene image containing the target object, acquired by the first vision sensor at the current moment. The real-time acquisition angle is the relative spatial angle between the optical axis of the first vision sensor and the reference plane of the target object at the current moment, representing the relative posture relationship between the current acquisition viewpoint and the target object. The first vision sensor, after calibration and synchronization, is activated, and images within the work scene are acquired, resulting in a real-time spatial image containing the target object. Simultaneously, the real-time spatial position of the first vision sensor and the real-time spatial position of the target object are obtained through the spatial positioning module. Based on these two spatial positions, the relative angle between them is calculated, ultimately determining the real-time acquisition angle.

[0059] In this embodiment, the real-time spatial image acquired by the first vision sensor is retrieved, and the image is standardized and preprocessed to eliminate various interference factors. Then, the preprocessed real-time spatial image is input into the pre-trained target detection model to identify and locate the target object region in the image, output a rectangular box that selects the complete region of the target object, and finally determine the spatial bounding box corresponding to the current real-time acquisition angle.

[0060] In this embodiment, the judgment label is a binary classification label used to identify whether the current real-time acquisition angle is within the allowed angle range, divided into two categories: "belongs" and "does not belong". Pre-constructed angle distortion mapping data is retrieved, the allowed angle range is extracted, and then the currently calculated real-time acquisition angle is obtained. The value of the real-time acquisition angle is compared with the upper and lower limits of the allowed angle range to determine whether the real-time acquisition angle falls within the allowed angle range. If the real-time acquisition angle is not within the allowed angle range, the corresponding judgment label is directly determined as "does not belong"; if the real-time acquisition angle is within the allowed angle range, the corresponding judgment label is determined as "belongs".

[0061] The beneficial effects of the above technologies are as follows: Based on the real-time spatial image of the target object acquired by the first vision sensor, and based on the real-time spatial image, angle distortion mapping data, the optimal straight-line distance of all allowable relative angles within the allowable angle range, and the collaborative correlation model, the judgment label, the predicted grasping value, and the real-time pose guidance data can be determined, which can improve the success rate and real-time performance of grasping operations under multiple perspectives, avoid the operational risks of invalid perspectives from the source, and realize the real-time intelligent reuse of historical grasping experience. Example 7:

[0062] Based on Example 6, a visual servoing-based robotic arm pose closed-loop guidance method determines real-time pose guidance data based on judgment labels, real-time pose data, target pose data, real-time acquired angles, angle distortion mapping data, the optimal straight-line distance of all allowable relative angles within the allowable angle range, and a collaborative association model. This includes: If the label is determined to be a real-time spatial image, the spatial bounding box is mapped based on the perspective distortion vector of the allowed relative angle in the angle distortion mapping data corresponding to the real-time acquired angle, and the real-time mapped image is determined. Feature point detection and effective point filtering are performed on the real-time mapped image to determine the real-time mapping data, which includes multiple real-time feature points and the real-time mapped coordinates of each real-time feature point. Based on the spatial nearest neighbor principle, the standard feature data and real-time mapping data of the target object are mapped to determine the real-time mapping set; Based on real-time spatial images and spatial bounding boxes, the real-time proportion value is calculated. At the same time, based on spatial bounding boxes, real-time mapped images, and real-time mapping sets, the real-time density value is calculated. The real-time distribution value is calculated based on the real-time mapping set and the mapping coordinate set of all successful grasps of the target object by the robotic arm. Input the real-time percentage value, real-time density value, and real-time distribution value into the collaborative association model to determine the predicted capture value; If the predicted grasp value is greater than or equal to the preset grasp value threshold, the real-time pose guidance data is determined based on the real-time pose data and the target pose data.

[0063] In this embodiment, the mapping process involves performing perspective transformation and distortion correction on the original image and bounding box based on perspective distortion vectors. The real-time mapped image is a standardized image aligned with the viewpoint of a standard visual image after distortion correction. The current judgment label is obtained. When the judgment label indicates a "belongs" state, pre-constructed angle distortion mapping data is retrieved, and the allowable relative angle corresponding to the current real-time acquisition angle is matched. The perspective distortion vector corresponding to this allowable relative angle is extracted. Then, based on this perspective distortion vector, perspective transformation is performed on the current real-time spatial image and spatial bounding box to correct the perspective distortion caused by the current viewpoint, aligning the image and bounding box to the reference viewpoint of the standard relative angle, ultimately generating the real-time mapped image.

[0064] In this embodiment, the real-time mapping data is a structured dataset that integrates effective feature points and their corresponding coordinates. Real-time feature points are the effective core feature points on the target object that have been filtered and retained. Real-time mapping coordinates are the pixel coordinates of each real-time feature point in the real-time mapping image. The generated real-time mapping image is retrieved, and feature point detection is performed on the image using a feature detection algorithm to extract all initial feature points. Then, the initial feature points are filtered for effectiveness, removing redundant and unstable feature points located in the background region, and retaining stable and effective feature points located within the target object region as real-time feature points. Simultaneously, the pixel coordinates of each real-time feature point in the real-time mapping image are extracted as real-time mapping coordinates. Finally, all real-time feature points and their corresponding real-time mapping coordinates are integrated to form the real-time mapping data.

[0065] In this embodiment, the standard feature data of the pre-constructed target object and the currently generated real-time mapping data are retrieved. Each standard feature point in the standard feature data is traversed. Based on the principle of spatial nearest neighbor, the real-time feature point that is closest to the pixel space of the standard feature point is found in the real-time mapping data, and a one-to-one correspondence between the two feature points is established. Then, the real-time mapping coordinates of all matched real-time feature points are extracted and arranged and integrated in a fixed order according to the standard feature points to finally form a real-time mapping set.

[0066] In this embodiment, the calculation logic is the same as that for calculating the real-time percentage value of the image proportion of each successfully captured target object, and the calculation logic is the same as that for calculating the real-time density value of each successfully captured target object.

[0067] In this embodiment, based on the real-time mapping set and the mapping coordinate set of all successful grasps of the target object by the robotic arm, the real-time distribution value is calculated, and the calculation formula is expressed as follows: ; in, Indicates real-time distribution values. This represents the second indicator function based on the j-th standard feature point in the real-time mapping set. This represents the mapping coordinates based on the j-th standard feature point in the real-time mapping set. This represents the real-time mapping set and the set of mapping coordinates of all successfully captured target objects, based on the standard deviation of the j-th standard feature point.

[0068] In this embodiment, the predicted grasping value is the model output. It is a comprehensive score of the grasping effect achievable under the current viewpoint and grasping posture. The higher the value, the better the grasping success rate and grasping stability. First, the real-time proportion value, real-time density value, and real-time distribution value are calculated, and the three real-time feature values ​​are integrated according to the input format required by the model training. Then, the integrated feature data is input into the pre-trained collaborative association model. The model completes the calculation and analysis based on the pre-learned mapping relationship between features and grasping effect, and outputs the predicted grasping effect score corresponding to the current grasping conditions, thus determining the predicted grasping value.

[0069] In this embodiment, the preset grasping value threshold is a pre-set minimum grasping effect score threshold that ensures the success rate of the grasping operation. Only when the predicted grasping value reaches or exceeds this threshold can a high success rate of the grasping action be guaranteed. Real-time pose guidance data consists of real-time pose correction parameters and motion trajectory data used to control the movement adjustment of the robotic arm. It can be directly input into the robotic arm control system to achieve closed-loop guidance. The predicted grasping value output by the model and the preset grasping value threshold are obtained, and the predicted grasping value is compared with the preset grasping value threshold. When the predicted grasping value is greater than or equal to the preset grasping value threshold, it is determined that the current grasping conditions can achieve a stable grasping with a high success rate. Then, the real-time pose data and target pose data of the robotic arm are retrieved, the deviation between the real-time pose and the target pose is calculated, and the pose correction parameters and optimal motion trajectory of the robotic arm end effector are calculated based on the deviation, finally generating real-time pose guidance data.

[0070] The beneficial effects of the above technologies are as follows: Based on the judgment label, real-time pose data, target pose data, real-time acquisition angle, angle distortion mapping data, the optimal straight-line distance of all allowable relative angles within the allowable angle range, and the collaborative association model, real-time pose guidance data can be determined, which can generate accurate pose guidance data, improve the accuracy and robustness of grasping operations under complex perspectives, realize the real-time closed-loop reuse of historical successful grasping experience, and avoid the execution risk of low success rate grasping actions from the source. Example 8:

[0071] Based on Example 6, a visual servoing-based robotic arm pose closed-loop guidance method determines real-time pose guidance data based on judgment labels, real-time pose data, target pose data, real-time acquired angles, angle distortion mapping data, the optimal straight-line distance of all allowable relative angles within the allowable angle range, and a collaborative association model. This includes: If the label is determined to be non-belonging, or if the label is determined to be belonging and the predicted crawl value is less than the preset crawl value threshold; Based on each allowed relative angle within the allowed angle range, the optimal straight-line distance of the allowed relative angle, and the real-time spatial position of the target object, the target spatial position for each allowed relative angle within the allowed angle range is determined; Based on the target spatial position of each allowable relative angle within the allowable angle range and the real-time pose data of the robotic arm, preset guidance data for each allowable relative angle within the allowable angle range is determined. The preset guidance data includes a preset planned path and preset indirect pose guidance data. With the goal of minimizing the planned path and the pose adjustment, the system selects real-time relative angles and real-time indirect pose guidance data based on the preset indirect pose guidance data of all allowed relative angles within the allowable angle range. Based on real-time indirect pose guidance data, the robotic arm is controlled to move to the target spatial position corresponding to the allowable relative angle in real-time relative angle. Real-time end-effector images are acquired based on the second vision sensor, and real-time precise pose guidance data is determined. Based on real-time indirect pose guidance data and real-time precise pose guidance data, the real-time pose guidance data is determined.

[0072] In this embodiment, "not belonging" means the current viewpoint is outside the historically verified safe grasping range. "Belonging" and the predicted grasping value is less than the preset grasping value threshold means that although the current viewpoint is within the safe range, the predicted grasping effect cannot meet the success rate requirements. The triggering logic for this condition is that as long as either of the above two scenarios is met, the subsequent robotic arm pose adjustment process will be initiated.

[0073] In this embodiment, the target spatial position is the spatial coordinate that the robotic arm's end-effector vision sensor needs to reach at each allowed relative angle to achieve the optimal grasping effect. First, a pre-calibrated range of allowed angles and the optimal straight-line distance corresponding to each allowed relative angle within that range are retrieved. Then, the real-time spatial position of the target object is obtained. Using the real-time spatial position of the target object as a reference, combined with the viewing direction of each allowed relative angle and the optimal straight-line distance, the three-dimensional spatial coordinates that the robotic arm's end-effector sensor needs to reach are calculated, ultimately determining the target spatial position corresponding to each allowed relative angle.

[0074] In this embodiment, the preset guidance data is a complete set of motion control data that guides the robotic arm from its current pose to the corresponding target spatial position for each allowed relative angle. The preset planned path is the collision-free optimal motion trajectory of the robotic arm from its current pose to the target spatial position. The preset indirect pose guidance data consists of segmented pose adjustment parameters that control the robotic arm's movement along the planned path. By retrieving the target spatial position corresponding to each allowed relative angle and the robotic arm's real-time pose data, and combining the obstacle distribution of the work scene with the robotic arm's motion constraints, a collision-free motion path is planned for each target spatial position, generating the preset planned path. Simultaneously, based on the planned path, the segmented pose adjustment parameters during the robotic arm's movement are calculated to generate the preset indirect pose guidance data. Finally, these are integrated to form the preset guidance data corresponding to each allowed relative angle.

[0075] In this embodiment, the minimum planned path refers to the shortest trajectory length of the robotic arm from its current pose to the target spatial position. Minimum pose adjustment refers to the minimum number of joint degrees of freedom that need to be adjusted during the robotic arm's movement, resulting in the smallest amplitude of pose change. The optimization objective is to select the scheme with the highest motion efficiency and the lowest robotic arm wear from all candidate guidance schemes. The real-time relative angle is the optimal allowable relative angle finally selected, representing the target viewpoint the robotic arm needs to reach after adjustment. Real-time indirect pose guidance data is the pose adjustment and motion control data corresponding to the optimal scheme finally selected. The dual optimization objectives of minimum planned path and minimum pose adjustment are clearly defined, and corresponding weight coefficients are set for the two optimization objectives. Then, the preset indirect pose guidance data corresponding to all allowable relative angles within the allowable angle range are traversed. The path length and pose adjustment amplitude of each candidate scheme are calculated. Based on the dual optimization objectives, a comprehensive score is calculated for each scheme. The allowable relative angle corresponding to the scheme with the highest comprehensive score is selected as the real-time relative angle, and the preset indirect pose guidance data corresponding to this scheme is also selected as the real-time indirect pose guidance data.

[0076] In this embodiment, the real-time end-effector image is a close-range real-time image of the target object captured by the second sensor after the robotic arm moves to the target spatial position. Real-time precise pose guidance data is calculated based on the close-range end-effector image and is used to correct minor pose deviations in the robotic arm's end effector. The real-time indirect pose guidance data is input into the robotic arm control system, which controls the robotic arm to move along the planned path to the target spatial position corresponding to the real-time relative angle, completing coarse positioning guidance. Then, the second vision sensor is activated to acquire a real-time end-effector image from the current end-effector's perspective. The real-time end-effector image is preprocessed for target detection and feature extraction, and the minor pose deviation between the robotic arm's end effector and the target object's grasping point is calculated. Based on the deviation, high-precision pose correction parameters are calculated, ultimately determining the real-time precise pose guidance data.

[0077] In this embodiment, real-time pose guidance data is the complete closed-loop motion control data that controls the robotic arm to complete the grasping operation from the current initial pose to the final optimal grasping pose. It serves as the final execution basis for the robotic arm's grasping operation. The coarse motion path and large-range pose adjustment parameters from the real-time indirect pose guidance data are retrieved, and the end-effector micro-pose correction parameters from the real-time precise pose guidance data are retrieved. These two sets of data are then fused temporally and integrated with the parameters. First, a large-range coarse guidance motion is executed, followed by precise end-effector pose correction, ultimately forming complete, staged real-time pose guidance data.

[0078] The beneficial effects of the above technologies are as follows: Based on the judgment label, real-time pose data, target pose data, real-time acquisition angle, angle distortion mapping data, the optimal straight-line distance of all allowed relative angles within the allowed angle range, and the collaborative association model, real-time pose guidance data can be determined, which can form a complete closed-loop pose adjustment, solve the problems of no response strategy in invalid scenarios, large motion loss, and insufficient adjustment accuracy, significantly improve the robustness and efficiency of grasping operations in complex scenarios, ensure the success rate of grasping in all scenarios, and realize the full-scenario reuse of historical best grasping experience.

[0079] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A closed-loop guidance method for robot arm pose based on visual servoing, characterized in that, include: S1: Obtain historical grasping data of the robotic arm successfully grasping the target object multiple times, obtain the real-time pose data of the robotic arm, and obtain the real-time spatial position and pose data of the target object. S2: Based on the historical grasping data of all successful grasping of the target object by the robotic arm, determine the allowable angle range and the optimal straight-line distance for each allowable relative angle within the allowable angle range, and determine the object visual data and the angular distortion mapping data of the target object; S3: Based on the historical grasping data of all successful grasping of target objects by the robotic arm, object visual data, and target object angle distortion mapping data, determine the target bounding box, mapped boundary image, and mapped coordinate set of each successful grasping of target objects by the robotic arm, and construct a collaborative association model. S4: Based on the real-time spatial image of the target object acquired by the first vision sensor, and based on the real-time spatial image, angle distortion mapping data, the optimal straight-line distance of all allowed relative angles within the allowed angle range, and the collaborative association model, determine the judgment label, predict the grasp value, and the real-time pose guidance data. Based on historical grasping data from all successful object grasping attempts by the robotic arm, the allowable angle range and the optimal straight-line distance for each allowable relative angle within the allowable angle range are determined, including: Based on the historical relative angles from all successful grasping data of the robotic arm, a set of relative angles is determined. Extract the historical straight-line distance corresponding to the highest crawling evaluation value among all historical crawling data for each historical relative angle in the relative angle set, and use it as the optimal straight-line distance for the historical relative angle; Based on the historical relative angles in the historical capture data of all successful captures of the target object, the allowable angle range is determined, which includes multiple allowable relative angles; Based on the optimal straight-line distance of all historical relative angles in the relative angle set, the allowable relative angle corresponding to each historical relative angle within the allowable angle range is assigned a value, and interpolation is performed on each allowable relative angle within the allowable angle range other than all historical relative angles to determine the optimal straight-line distance of each allowable relative angle within the allowable angle range.

2. The visual servoing-based robotic arm pose closed-loop guidance method according to claim 1, characterized in that, Acquire historical grasping data of the robotic arm successfully grasping the target object multiple times, acquire the real-time pose data of the robotic arm, and acquire the real-time spatial position and pose data of the target object, including: Acquire historical grasping data of the robotic arm successfully grasping the target object multiple times. The historical grasping data includes: historical end images collected based on the second vision sensor, historical straight-line distance between the second vision sensor and the target object, historical relative angle, and grasping evaluation value. Acquire real-time pose data of the robotic arm, and acquire real-time spatial position and pose data of the target object.

3. The visual servoing-based robotic arm pose closed-loop guidance method according to claim 1, characterized in that, Determine the object's visual data and the target object's angular distortion mapping data, including: The robotic arm is controlled to move to each allowable relative angle within the allowable angle range and the optimal straight-line distance of the allowable relative angle, and angular visual images of the allowable relative angles are acquired based on a second vision sensor; The allowable relative angle with a value of 0 within the allowable angle range is defined as the standard relative angle, and the visual image of the angle corresponding to the standard relative angle is defined as the standard visual image. Based on the angular visual images of all permissible relative angles within the permissible angle range, determine the object visual data of the target object; The standard visual image of the standard relative angle and the angular visual images of all allowed relative angles in the object visual data are input into the perspective transformation model to determine the angular distortion mapping data of the target object. The angular distortion mapping data includes the perspective distortion vector of each allowed relative angle within the allowed angle range.

4. The visual servoing-based robotic arm pose closed-loop guidance method according to claim 3, characterized in that, Based on the historical grasping data of all successful object grasping attempts by the robotic arm, object visual data, and angular distortion mapping data of the object, the target bounding box, mapped boundary image, and set of mapped coordinates for each successful object grasping attempt by the robotic arm are determined. A collaborative association model is then constructed, including: The historical end image in the historical grasping data of each successful grasping of the target object by the robotic arm is preprocessed, and the target bounding box of the target object in the historical end image is determined by the preprocessed historical end image. Based on the perspective distortion vector of the allowed relative angle corresponding to the historical relative angle of each successful grasp of the target object by the robotic arm in the angle distortion mapping data, the historical end image and target bounding box of the second successful grasp of the target object are mapped to determine the mapped boundary image of the target object for each successful grasp of the robotic arm. Feature point detection and effective point filtering are performed on the mapping boundary image of the target object successfully grasped by the robotic arm each time to determine the mapping feature data of the target object successfully grasped by the robotic arm each time. The mapping feature data includes multiple mapping feature points and the target mapping coordinates of each mapping feature point. Based on the historical end image and target bounding box of each successful grasp of the target object by the robotic arm, the percentage of images of the target object successfully grasped each time is calculated; Feature point detection and effective point filtering are performed on the standard visual image of the target object at a standard relative angle in the object visual data to determine the standard feature data of the target object. The standard feature data includes multiple standard feature points of the standard visual image and the standard coordinates of each standard feature point. Based on the principle of spatial nearest neighbor, feature point matching is performed on the standard feature data of the target object and the mapping feature data of the target object for each successful capture to determine the set of mapping coordinates of the target object for each successful capture. Based on the target bounding box, mapped boundary image, and mapped coordinate set of each successfully grasped target object by the robotic arm, the mapping density value of the target object for each successful grasp is calculated. Based on the set of mapped coordinates of all successful grasps of the target object by the robotic arm, calculate the feature distribution value of each successful grasp of the target object by the robotic arm; The collaborative association model is constructed by taking the image proportion, mapping density, and feature distribution values ​​of all successfully captured target objects as inputs and taking the capture evaluation values ​​from the historical capture data of all successfully captured target objects as outputs.

5. The visual servoing-based robotic arm pose closed-loop guidance method according to claim 1, characterized in that, Based on real-time spatial images of the target object acquired using a first-vision sensor, and based on these real-time spatial images, angular distortion mapping data, the optimal straight-line distances of all permissible relative angles within the allowed angular range, and a collaborative correlation model, the system determines the judgment label, predicts the grasp value, and provides real-time pose guidance data, including: Real-time spatial images of the target object are acquired based on the first visual sensor, and the real-time acquisition angle is determined based on the position of the first visual sensor and the position of the target object. The real-time spatial image is preprocessed, and the target detection is performed on the preprocessed real-time spatial image to determine the spatial bounding box based on the real-time acquisition angle. Determine whether the real-time acquired angle falls within the allowable angle range of the angle distortion mapping data, and determine the judgment label, which includes "belongs to" and "does not belong to". Based on the judgment label, real-time pose data, target pose data, real-time acquisition angle, angle distortion mapping data, the optimal straight-line distance of all allowed relative angles within the allowed angle range, and the collaborative association model, the real-time pose guidance data is determined.

6. The visual servoing-based robotic arm pose closed-loop guidance method according to claim 5, characterized in that, Based on the judgment label, real-time pose data, target pose data, real-time acquired angles, angle distortion mapping data, the optimal straight-line distance of all allowable relative angles within the allowable angle range, and the collaborative association model, real-time pose guidance data is determined, including: If the label is determined to be a real-time spatial image, the spatial bounding box is mapped based on the perspective distortion vector of the allowed relative angle in the angle distortion mapping data corresponding to the real-time acquired angle, and the real-time mapped image is determined. Feature point detection and effective point filtering are performed on the real-time mapped image to determine the real-time mapping data, which includes multiple real-time feature points and the real-time mapped coordinates of each real-time feature point. Based on the spatial nearest neighbor principle, the standard feature data and real-time mapping data of the target object are mapped to determine the real-time mapping set; Based on real-time spatial images and spatial bounding boxes, the real-time proportion value is calculated. At the same time, based on spatial bounding boxes, real-time mapped images, and real-time mapping sets, the real-time density value is calculated. The real-time distribution value is calculated based on the real-time mapping set and the mapping coordinate set of all successful grasps of the target object by the robotic arm. Input the real-time percentage value, real-time density value, and real-time distribution value into the collaborative association model to determine the predicted capture value; If the predicted grasp value is greater than or equal to the preset grasp value threshold, the real-time pose guidance data is determined based on the real-time pose data and the target pose data.

7. The visual servoing-based robotic arm pose closed-loop guidance method according to claim 5, characterized in that, Based on the judgment label, real-time pose data, target pose data, real-time acquired angles, angle distortion mapping data, the optimal straight-line distance of all allowable relative angles within the allowable angle range, and the collaborative association model, real-time pose guidance data is determined, including: If the label is determined to be non-belonging, or if the label is determined to be belonging and the predicted crawl value is less than the preset crawl value threshold; Based on each allowed relative angle within the allowed angle range, the optimal straight-line distance of the allowed relative angle, and the real-time spatial position of the target object, the target spatial position for each allowed relative angle within the allowed angle range is determined; Based on the target spatial position of each allowable relative angle within the allowable angle range and the real-time pose data of the robotic arm, preset guidance data for each allowable relative angle within the allowable angle range is determined. The preset guidance data includes a preset planned path and preset indirect pose guidance data. With the goal of minimizing the planned path and the pose adjustment, the system selects real-time relative angles and real-time indirect pose guidance data based on the preset indirect pose guidance data of all allowed relative angles within the allowable angle range. Based on real-time indirect pose guidance data, the robotic arm is controlled to move to the target spatial position corresponding to the allowable relative angle in real-time relative angle. Real-time end-effector images are acquired based on the second vision sensor, and real-time precise pose guidance data is determined. Based on real-time indirect pose guidance data and real-time precise pose guidance data, the real-time pose guidance data is determined.

Images