Composite robot positioning method and system based on global and local distributed visual collaboration

Abstract

The invention discloses a composite robot positioning method based on global and local distributed visual collaboration. The method comprises the following steps: acquiring pose information of a target object and a carrying mechanism in a global coordinate system by using a global camera; the pre-observation pose of the mechanical arm is reversely calculated based on the pose information and the view field parameters of the local camera; a collision-free path is planned through a pre-established global and local distributed visual cooperative positioning model based on the pre-observed pose and in combination with the environmental safety envelope obtained by the global camera, the mechanical arm is controlled to move to the pre-observed pose, and field-of-view handover of the visual system is completed; a local camera is used for collecting an image of a target object, and a visual servo control method is adopted to guide the tail end of the mechanical arm to correct position deviation in real time until the tail end is positioned to a target operation position. The invention discloses a composite robot positioning method and system based on global and local distributed visual collaboration, which improve the operation rhythm and the positioning robustness and realize the compatibility of a large field of view and high precision.

Description

Technical Field

[0001] This invention relates to the field of visual positioning technology, specifically to a composite robot positioning method and system based on global and local distributed visual collaboration. Background Technology

[0002] In smart manufacturing and flexible logistics scenarios, mobile manipulators are widely used because they combine the mobility of handling mechanisms (such as AGVs) with the operational capabilities of robotic arms. Visual positioning technology is a core technology for achieving precise grasping or docking.

[0003] Current composite robot visual localization methods mainly fall into two categories: single end-eye vision (Eye-in-Hand) and single global vision (Eye-to-Hand). However, both of these methods have significant drawbacks in practical applications.

[0004] Single-end vision involves mounting a camera at the end of the robotic arm. This method offers high precision but has a limited field of view (FOV). Since the navigation and positioning accuracy of the chassis of a handling mechanism (such as an AGV) is typically on the centimeter level, when there is a deviation in the stopping position of the handling mechanism (such as an AGV), the target object often goes out of the field of view of the end-view camera. This not only greatly reduces operational efficiency but also increases the risk of the robotic arm colliding with its surroundings.

[0005] Single global vision relies solely on external fixed cameras or global cameras on the vehicle body of a transport mechanism (such as an AGV) for positioning. This method offers a wide field of view and can quickly lock onto targets, but as the camera moves further away from the target, the pixel density decreases, and the positioning accuracy (typically at the centimeter level) cannot meet the requirements of precision assembly or grasping (which typically requires millimeter or even sub-millimeter level).

[0006] Furthermore, existing vision positioning systems are often fragmented, with global vision used only for navigation of handling mechanisms (such as AGVs) and local vision used only for robotic arm servoing. They lack a mechanism for "field of view decoupling and data collaboration," and cannot effectively and smoothly transition the advantages of a large global field of view to the advantages of high local precision. This makes it easy for the system to experience positioning loss or motion lag during the transition between "large-scale approach" and "small-scale precision operation."

[0007] Therefore, there is an urgent need to develop a distributed visual positioning method that can coordinate a large global field of view with high local precision at the end point to resolve the contradiction between the field of view and positioning accuracy, and to achieve efficient, collision-free, and high-precision operation of composite robots. Summary of the Invention

[0008] This invention overcomes the shortcomings of the prior art and provides a composite robot localization method and system based on global and local distributed vision collaboration, which improves the cycle time and localization robustness, and achieves compatibility between a large field of view and high precision.

[0009] To achieve the above objectives, the technical solution adopted by this invention is: a composite robot localization method based on global and local distributed visual collaboration, comprising the following steps: Step S1: Use the global camera to acquire information about the work scene, as well as the pose information of the target object and the handling mechanism in the global coordinate system, and construct an environmental safety envelope; Step S2: Based on the pose information and the field of view parameters of the local camera, reverse calculate the pre-observation pose of the robotic arm. The pre-observation pose includes the end-effector pose of the robotic arm that places the target object within the optimal imaging field of view of the local camera. Step S3: Based on the pre-observed pose and combined with the environmental safety envelope obtained by the global camera, a collision-free path is planned through a pre-established global and local distributed visual cooperative localization model. Based on the collision-free path, the robotic arm is controlled to move to the pre-observed pose to complete the field of view handover of the vision system. Step S4: Use a local camera to acquire images of the target object, and use a visual servo control method to guide the end effector of the robotic arm to correct the position deviation in real time until it is positioned at the target working position.

[0010] In a preferred embodiment of the present invention, the pre-observation pose calculation method in step S2 includes: Establish a local observation coordinate system with the pose information of the target object as the origin; Based on the optimal observation distance d of the local camera opt With optical axis direction constraints, calculate the ideal pose of the end camera in the local camera in the observation coordinate system; Through coordinate transformation chain The ideal pose is converted into the end flange pose in the robot arm base coordinate system. , as the pre-observed pose; Let be the pose transformation matrix from the base coordinate system to the target object coordinate system Obj; It is the inverse of the transformation matrix from the end effector coordinate system to the target object coordinate system.

[0011] In a preferred embodiment of the present invention, the field of view inverse calculation includes: setting the ideal observation distance of the end camera. and ideal observation angle; The calculation of the pre-observed pose includes: the pose information of the target object. Based on this, translate along the target normal direction by d opt To obtain the theoretical pose that the robotic arm's end effector should reach. ; ; This is the coarse pose transformation matrix from the base coordinate system to the target coordinate system O; Let be the offset transformation matrix from the target coordinate system O to the end effector coordinate system E; Inverse kinematics solution and optimization include: for Perform inverse kinematics calculation for the robot; if multiple solutions exist, use a weighted function to select the solution with the minimum joint movement and no singular states. Obstacle avoidance planning includes: combining the environmental point cloud provided by the global camera, and using the Rapid Expanding Random Tree (RRT) algorithm to plan a path from the current pose to... A collision-free path.

[0012] In a preferred embodiment of the present invention, the visual servoing control method in step S4 includes: Construct the image feature Jacobian matrix. A mapping relationship between image feature error and robotic arm end-effector speed is established based on the image feature Jacobian matrix. By using proportional-integral-derivative control laws or adaptive gain control laws, speed control commands for the robotic arm are generated in real time to correct positioning errors caused by docking deviations of the handling mechanism, uneven ground, and global camera calibration residuals.

[0013] In a preferred embodiment of the present invention, the abnormal handling logic for field of view handover is as follows: when the robotic arm reaches the pre-observation pose, it detects whether the local camera has successfully identified the target object features; if the identification is successful, it switches to step S4 to perform local visual servoing; if the identification fails, it keeps the robotic arm base stationary and controls the end effector to perform a spiral search or zoom scan in a preset neighborhood of the pre-observation pose until the target features are captured.

[0014] In a preferred embodiment of the present invention, step S1 further includes: performing system calibration and initialization during the system deployment phase; The system deployment includes building a composite robot localization system based on global and local distributed vision collaboration; The deployment-based system establishes the coordinate transformation chain of the entire system through system calibration and initialization. System calibration and initialization include: Determine the pose matrix of the global camera relative to the world coordinate system W. It is used to achieve global camera calibration; Determine the pose matrix of the local camera relative to the end flange E of the robotic arm. It is used to achieve hand-eye calibration; Determine the rigidity transformation relationship of the robotic arm base B relative to the center A of the handling mechanism. It is used to achieve joint calibration of robot and handling mechanism.

[0015] In a preferred embodiment of the present invention, the acquisition of pose information includes global visual perception and coarse localization; when the transport mechanism navigates to the vicinity of the work point and stops, the global vision system is triggered, including the following steps; The global camera captures a panoramic image containing the target object; Use deep learning object detection algorithms or QR code recognition algorithms to identify target object regions; The PnP algorithm is used to calculate the pose of the target object's coordinate system O relative to the global camera Cg. The transport organization uploads its real-time odometer data. ; Transform the target pose to the robot base coordinate system B to obtain the target pose. And as pose information: ; in, Let W be the pose transformation matrix from the world coordinate system W to the center A of the transport mechanism; The pose transformation matrix from the robot arm base B to the center A of the handling mechanism; From the world coordinate system W to the global camera The pose transformation matrix; From the target object coordinate system O to the global camera The pose transformation matrix; In a preferred embodiment of the present invention, the field of view handover of the visual system in step S3 includes the following steps: The robot controller drives the robotic arm to move rapidly along the planned path to the pre-observed pose; Images are acquired by the end camera and preprocessed using adaptive threshold segmentation and distortion correction algorithms; Identify feature points (such as QR code corner points) in an image and calculate the precise pose of the target relative to the end-point camera. ; Implement pose calculation: Then it enters the closed-loop control stage, defining the current image feature s and the desired feature. error Using the image Jacobian matrix Calculate the camera velocity spinor in To control the gain, The pseudo-inverse of the Jacobian matrix is ​​used to implement visual servoing. The robotic arm according to v c Perform real-time fine-tuning until the error e converges to the preset threshold (e.g., pixel error < 2 pixels, corresponding to spatial error < 0.5 mm); complete the fine-tuning approximation.

[0016] In a preferred embodiment of the present invention, applied to dual-arm cooperative positioning, the method includes the following steps: Two local cameras controlling the arms observe the visual features of the target object from different positions. The observation data from two local cameras are mapped to the same world coordinate system to construct a multi-view joint optimization equation; The six-DOF pose of the target object is solved by minimizing the total reprojection error, thereby eliminating the ambiguity of pose solution under a single local viewpoint.

[0017] In a preferred embodiment of the present invention, a high-precision positioning system for a composite robot based on global and local distributed vision collaboration is provided. This system employs a composite robot positioning method based on global and local distributed vision collaboration and is applied to a composite robot system comprising a transport mechanism chassis, a robotic arm, a local camera mounted at the end of the robotic arm, and a global camera mounted in the external environment. The global camera is fixedly mounted on a bracket outside the work area, and its field of view covers the area where the transport mechanism stops and the area where the target object is located. A robotic arm is installed in the work area, and the robotic arm is mounted on a transport mechanism trolley. A local camera is installed at the end of the robotic arm; the transport mechanism is an intelligent transport robot AGV.

[0018] In a preferred embodiment of the present invention, applied to dual-arm cooperative positioning, the method includes the following steps: Two local cameras controlling the arms observe the visual features of the target object from different positions. The observation data from two local cameras are mapped to the same world coordinate system to construct a multi-view joint optimization equation; The six-DOF pose of the target object is solved by minimizing the total reprojection error, thereby eliminating the ambiguity of pose solution under a single local viewpoint.

[0019] In a preferred embodiment of the present invention, a high-precision positioning system for a composite robot based on global and local distributed vision collaboration is provided. This system employs a composite robot positioning method based on global and local distributed vision collaboration and is applied to a composite robot system comprising a transport mechanism chassis, a robotic arm, a local camera mounted at the end of the robotic arm, and a global camera mounted in the external environment. The global camera is fixedly mounted on a bracket outside the work area, and its field of view covers the area where the transport mechanism stops and the area where the target object is located. A robotic arm is installed in the work area, and the robotic arm is mounted on a transport mechanism trolley. A local camera is installed at the end of the robotic arm.

[0020] This invention addresses the deficiencies in the technical background, and the beneficial technical effects of this invention are: A composite robot localization method and system based on global and local distributed vision collaboration is proposed. By constructing a hierarchical localization strategy, the system uses an external global camera for "coarse localization and field of view prediction" to guide the end-effector camera of the robotic arm directly to the "optimal observation pose". Subsequently, it switches to local vision for "fine localization", thereby eliminating the blind search process, significantly improving the operation cycle and localization robustness, and achieving compatibility between a large field of view and high precision.

[0021] 1. Utilize global vision guidance to eliminate blind searching at the end point, achieve "one-step" observation pose, and improve the operation cycle time.

[0022] 2. The system has a wide field of view compensation capability, which can reduce the docking deviation of the handling mechanism (such as AGV), reduce errors, and significantly reduce the dependence on high-precision navigation chassis and hardware costs.

[0023] 3. A dynamic anti-collision envelope is constructed based on the global field of view to monitor environmental obstacles in real time and ensure the safety of the robotic arm during rapid movement.

[0024] 4. Through multi-scale visual complementarity of global and local perspectives, it effectively overcomes the interference of lighting changes, occlusion, and large scenes on a single visual system, exhibiting strong environmental adaptability and robustness. Attached Figure Description

[0025] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0026] Figure 1 This is a flowchart illustrating a composite robot localization method based on global and local distributed vision collaboration in a preferred embodiment of the present invention. Figure 2 This is a schematic diagram of a composite robot localization system based on global and local distributed vision collaboration in a preferred embodiment of the present invention. Figure 3 This is a schematic diagram of the construction of an environmental safety envelope for a composite robot localization system based on global and local distributed visual collaboration, as shown in a preferred embodiment of the present invention. Figure 4 This is a diagram illustrating the collision-free path planning effect of a composite robot localization system based on global and local distributed vision collaboration in a preferred embodiment of the present invention. Figure 5 This is a comparison of the CLAHE algorithm before and after processing in a preferred embodiment of the present invention for a composite robot localization system based on global and local distributed vision collaboration (original image). Figure 6This is a comparison of the CLAHE algorithm before and after processing in a preferred embodiment of the present invention for a composite robot localization system based on global and local distributed visual collaboration (image after CLAHE processing). The components include: 1. Global camera; 2. Local camera; 3. AGV (Automated Guided Vehicle); 4. Robotic arm. Detailed Implementation

[0027] The present invention will now be described in further detail with reference to the accompanying drawings and embodiments. These drawings are simplified schematic diagrams, which are only used to illustrate the basic structure of the present invention and therefore only show the components relevant to the present invention.

[0028] It should be noted that if directional indicators (such as up, down, bottom, top, etc.) are involved in the embodiments of the present invention, these directional indicators are only used to explain the relative positional relationship and movement of the components in a specific posture. If the specific posture changes, the directional indicators will also change accordingly. The terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, features defined with "first" and "second" may explicitly or implicitly include one or more of that feature. Unless otherwise explicitly specified and limited, the terms "set," "connected," and "linked" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal connection of two components. For those skilled in the art, the specific meaning of the above terms in the present invention can be understood according to the specific circumstances.

[0029] Example 1, as Figures 1-2 As shown, a composite robot localization method based on global and local distributed vision collaboration includes the following steps: Step S1: Use global camera 1 to acquire the work scene information, as well as the pose information of the target object and the handling mechanism in the global coordinate system, and construct the environmental safety envelope.

[0030] Step S1 also includes: performing system calibration and initialization during the system deployment phase; wherein, system deployment includes building a composite robot localization system based on global and local distributed vision collaboration; based on the deployed system, a coordinate transformation chain for the entire system is established through system calibration and initialization, and system calibration and initialization includes: determining the pose matrix of global camera 1 relative to the world coordinate system W. This is used to calibrate the global camera 1 and determine the pose matrix of the local camera 2 relative to the end flange E of the robotic arm 4. Used for hand-eye calibration; determining the rigidity transformation relationship of the robotic arm base B relative to the center A of the handling mechanism. It is used to achieve joint calibration of robot and handling mechanism.

[0031] Furthermore, the handling mechanism employs an AGV (Automated Guided Vehicle). A unified world coordinate system is constructed through hand-eye calibration and multi-camera extrinsic parameter calibration. Global camera 1 is responsible for monitoring the relative relationship between the "AGV base" and the "target object" in the world coordinate system. This provides a unified mathematical benchmark for collaborative positioning, enabling the conversion between camera data from different perspectives and compensating for the initial deviation of the robotic arm 4 base caused by the AGV's docking. Specifically, the coarse positioning calculation involves global camera 1 identifying the target object and calculating its approximate pose information in the world coordinate system. .

[0032] Step S2: Based on the pose information and the field-of-view parameters of the local camera 2, the pre-observation pose of the robotic arm 4 is calculated in reverse. The pre-observation pose includes the end-effector pose of the robotic arm 4 that positions the target object within the optimal imaging field of view of the local camera 2. The inverse field-of-view calculation is performed based on the intrinsic parameter matrix K of the end-effector camera and the optimal observation distance d. opt (Ensuring clear features and collision-free distance), and combining the target's pose information, a "visual cone" is constructed inversely. Pre-observation pose: Within the "visual cone," select the optimal pose (without singularities) obtained from the inverse kinematics solution of the robotic arm as the "pre-observation pose." Function: This step serves as a bridge connecting the global and local levels.

[0033] The pre-observation pose calculation method in step S2 includes: establishing a local observation coordinate system with the pose information of the target object as the origin; and calculating the optimal observation distance d of the local camera 2. opt Given the optical axis direction constraint, calculate the ideal pose of the end camera in local camera 2 in the observation coordinate system; and use the coordinate transformation chain... The ideal pose is converted into the end flange pose in the coordinate system of the four bases of the robotic arm, which is used as the pre-observed pose.

[0034] Furthermore, the inverse field of view calculation includes: setting the ideal observation distance d for the end camera. opt And the ideal observation angle (optical axis perpendicular to the target surface). Further, the calculation of the pre-observation pose includes: the pose information of the target object. Based on this, translate along the target normal direction by d opt The theoretical pose that the end effector of the robotic arm 4 should reach is obtained. ; Furthermore, the inverse kinematics solution and optimization include: for Perform inverse kinematics calculation for the robot. If multiple solutions exist, use a weighted function to select the solution with the minimum joint movement and no singular states. Further, obstacle avoidance planning includes: combining the environmental point cloud provided by global camera 1, using an improved RRT (Fast Expanding Random Tree) algorithm to plan a path from the current pose to... A collision-free path.

[0035] In practice, the acquisition of environmental point clouds and the construction of the security envelope include the following sub-steps: Step S1.1, Environment Point Cloud Acquisition: Global camera 1 uses an RGB-D depth camera (Realsense D435). First, acquire the scene's depth map D(u, v) and color map I(u, v). Based on the camera's intrinsic parameter matrix K, use the formula... , Convert pixel coordinates to 3D point cloud data P in the camera coordinate system cloud scale: The scaling factor for the depth values. z: Z-axis coordinates in the camera coordinate system. Camera principal point coordinates; The camera's focal length (in pixels) in the X / Y directions.

[0036] Step S1.2, Point cloud preprocessing: Voxel grid filter is applied to the original point cloud to reduce the amount of data, and statistical outlier removal algorithm (SOR) is used to remove outliers caused by lighting or measurement noise to obtain a clean environmental point cloud.

[0037] Step S1.3, Construct an environmental security envelope (Occupancy Map): Discretize the environmental point cloud using an octree map structure or a grid map.

[0038] Mark the voxel containing the obstacle as "occupied".

[0039] Envelope expansion: Considering the physical dimensions and motion safety margin of the robotic arm 4, with each "occupied" voxel as the center and the maximum link radius R of the robotic arm 4 as the radius, the expansion is calculated. arm and safe buffer distance sum The obstacle area is spatially expanded using a radius of 1.

[0040] The expanded region is called the "environmental safety envelope." In subsequent path planning, any path point falling within this envelope is considered a collision risk point. The environmental safety envelope construction system is as follows: Figure 1 , Figure 2 As shown in Figure 3. The 3D point cloud and the dilated safety envelope are shown in Figure 3.

[0041] Furthermore, the acquisition of pose information includes global visual perception and coarse localization. When the AGV navigates to the vicinity of the work point and stops, the global vision system is triggered, including the following steps: global camera 1 captures a panoramic image containing the target object; the target object region is identified using a deep learning object detection algorithm or a QR code recognition algorithm; and the pose of the target object coordinate system O relative to the global camera 1's Cg is calculated using the PnP algorithm. AGV uploads its real-time odometer data. Transform the target pose to the robot base coordinate system B to obtain the target pose. And as pose information: .

[0042] Step S3: Based on the pre-observed pose and combined with the environmental safety envelope obtained by global camera 1, a collision-free path is planned through a pre-established global and local distributed visual cooperative localization model.

[0043] Among them, based on the environmental safety envelope generated by global camera 1, an improved RRT algorithm is used to plan the path: Step S3.1, sampling, includes: randomly sampling in joint space or Cartesian space to obtain sampling points. .

[0044] Step S3.2, growth, includes: finding the nearest node in the tree. Growth step size in the direction of sampling points Get a new node .

[0045] Step S3.3, collision detection, includes: detecting connections. and The line segment is checked for intersection with the "environmental safety envelope". If it intersects (i.e., a collision occurs), the node is discarded; otherwise, it is added to the path tree. Path smoothing: After generating the initial path, B-spline interpolation is used to smooth the path inflections, generating the final continuous collision-free trajectory. Based on the collision-free path, the robotic arm 4 is controlled to move to the pre-observed pose, completing the field of view handover of the vision system. The collision-free path planning effect diagram is shown below. Figure 4 As shown.

[0046] In step S3, the field of view handover of the vision system includes the following steps: the robot controller drives the robotic arm 4 to move rapidly along the planned path to the pre-observed pose; the end-effector camera acquires images, and the images are preprocessed using adaptive threshold segmentation and distortion correction algorithms; feature points (such as QR code corner points) in the image are identified, and the precise pose of the target relative to the end-effector camera is calculated. ; Achieve pose calculation and enter the closed-loop control stage, defining the current image features s and the desired features. error Using the image Jacobian matrix Calculate the camera velocity spinor in To control the gain, Using the pseudo-inverse of the Jacobian matrix, visual servo control is achieved. Robotic arm 4 operates based on v... c Perform real-time fine-tuning until the error e converges to the preset threshold (e.g., pixel error < 2 pixels, corresponding to spatial error < 0.5 mm); complete the fine-tuning approximation.

[0047] Furthermore, the anomaly handling logic for field-of-view transition is as follows: After the robotic arm 4 reaches the pre-observation pose, it checks whether the local camera 2 has successfully identified the target object features; if the identification is successful, it switches to step S4 for local visual servoing; if the identification fails, it keeps the base of the robotic arm 4 stationary and controls the end effector to perform a helical search or zoom scan within a preset neighborhood of the pre-observation pose until the target features are captured. Furthermore, in this embodiment, the identification method used to identify the target object features is the YOVOv8 identification algorithm.

[0048] Furthermore, in the dynamic field-of-view switching strategy, after the AGV stops, the system combines the environmental point cloud information provided by global camera 1 to plan the obstacle avoidance path for robotic arm 4 to move from its current posture to the "pre-observed pose". Handover determination: When robotic arm 4 reaches the pre-observed pose, the system checks whether the end-effector camera has successfully recognized the target QR code. If recognition is successful, control switches from the "global guidance module" to the "local servo module"; if not recognized (in extreme occlusion situations), a search strategy based on global vision assistance is initiated. Purpose: To ensure a smooth transition in the positioning process and prevent control logic conflicts.

[0049] Step S4: Use the local camera 2 to acquire images of the target object, and use the visual servo control method to guide the end of the robotic arm 4 to correct the position deviation in real time until it is positioned at the target working position.

[0050] The visual servo control method in step S4 includes: constructing an image feature Jacobian matrix, and establishing a mapping relationship between image feature error and the end effector speed of the robotic arm 4 based on the image feature Jacobian matrix; specifically, the mapping relationship model is as follows: Among them, image feature error (Current feature point coordinates - Desired feature point coordinates). L s Let Jacobian matrix be the image's Jacobian matrix for feature points on the image plane. Its corresponding L s The format is: ; Where Z is the depth value of the feature point.

[0051] The proportional-integral-derivative (PID) control law or adaptive gain control law is used. Specifically, to correct errors caused by AGV docking deviation (leading to initial position offset) and uneven ground (leading to camera optical axis tilt), the following control law is used to generate the end effector speed command for the robotic arm 4: ; The proportional term is For adaptive gain, a larger value is taken when the error e is large to quickly approach the target value, and a smaller value is taken when the error is small. To prevent overshoot oscillation.

[0052] Integral term Used to eliminate steady-state errors. Since AGV docking may cause a slight tilt angle in the base, this will produce a constant gravity component or visual bias. The integral term can accumulate this small error and generate a reverse compensation control quantity to ensure that the final positioning error converges to zero.

[0053] Furthermore, in weighted visual servoing based on local vision, image enhancement is performed: In the local recognition stage, an adaptive histogram equalization (CLAHE) algorithm is introduced to process local images to address interference from near-field metallic reflections or shadows. The comparison before and after CLAHE processing is shown below. Figure 5 , Figure 6 As shown, traditional global histogram equalization, while enhancing dark details, excessively amplifies background noise. CLAHE divides the image into multiple 8×8 sub-blocks, calculates the histogram for each sub-block, and sets a clipping limit (2.0-4.0 in this embodiment) to truncate the histogram, redistributing the pixels of the truncated portion. Finally, bilinear interpolation is used to eliminate artificial artifacts at the sub-block boundaries. Visual servoing: After the end-effector takes over control, an image feature Jacobian matrix is ​​constructed using image-based servoing (IBVS), and the end-effector velocity spinor is calculated in real time. Binocular collaboration (for dual arms): For large objects, the two end-effectors of the left and right arms simultaneously observe different reference points on the object, and jointly solve the 6D pose of the object by optimizing the reprojection error. Function: Corrects AGV docking errors, long-distance measurement errors of global camera 1, and kinematic errors of the robotic arm 4 itself, achieving final precise grasping.

[0054] Example 2 applies the composite robot localization method based on global and local distributed vision collaboration from Example 1 to dual-arm collaborative localization, including the following steps: Two local cameras 2, controlling the two arms, observe the visual features of the target object at different positions. The observation data from the two local cameras 2 are mapped to the same world coordinate system to construct a multi-view joint optimization equation; The six-DOF pose of the target object is solved by minimizing the total reprojection error, thereby eliminating the ambiguity of pose solution under a single local viewpoint.

[0055] Specifically, the task allocation is as follows: the system controls the left and right arms to move to the pre-observed poses at both ends of the workpiece.

[0056] Specifically, the joint solution is as follows: the cameras at the ends of the left and right arms simultaneously observe two different identification codes on the workpiece.

[0057] Specifically, the constraint optimization involves constructing multi-point constraint equations and solving for the uniquely determined pose of the workpiece by minimizing the binocular reprojection error function E. ;where P i p represents the three-dimensional coordinates of the feature point. i For the observed pixel coordinates, where That is .

[0058] Example 3: A high-precision positioning system for a composite robot based on global and local distributed visual collaboration. This system employs the global and local distributed visual collaboration-based composite robot positioning method from Example 1 or Example 2, applied to a composite robot system comprising an AGV mobile chassis, a robotic arm 4, a local camera 2 mounted at the end of the robotic arm 4, and a global camera 1 mounted in the external environment. The system includes: the global camera 1 fixedly mounted on a bracket surrounding the work area, its field of view covering the AGV docking area and the area where the target object is located; a robotic arm 4 is positioned in the work area and mounted on the AGV trolley 3, with the local camera 2 mounted at its end. Further, in this example, the global camera 1 is a binocular depth camera; the local camera 2 is an RGB-D camera; and the visual identifiers affixed to the AGV and the target object are AprilTags, QR codes, or natural feature points. However, this is not the only option; in other examples, other existing models can be selected based on actual application requirements.

[0059] Working principle: like Figure 1 , Figure 2As shown, this invention discloses a composite robot localization method and system based on global and local distributed vision collaboration. By constructing a hierarchical localization strategy, it utilizes an external global camera 1 for "coarse localization and field-of-view prediction," guiding the end-effector camera of the robotic arm 4 directly to the "optimal observation pose." Subsequently, it switches to local vision for "fine localization," thereby eliminating the blind search process, significantly improving operational cycle time and localization robustness, and achieving compatibility between a large field of view and high precision. Global vision guidance eliminates blind end-effector search, achieving "one-step" observation pose and improving operational cycle time. The system possesses a large field-of-view compensation capability, reducing AGV docking deviations and errors, and significantly reducing reliance on high-precision navigation chassis and hardware costs. A dynamic anti-collision envelope is constructed based on the global field of view, monitoring environmental obstacles in real time to ensure the safety of the robotic arm 4 during rapid movement. Through multi-scale visual complementarity between global and local vision, it effectively overcomes interference from lighting changes, occlusion, and large scenes on a single vision system, exhibiting strong environmental adaptability and robustness.

[0060] This invention employs a distributed hardware architecture combining an external global camera 1 and a local camera 2 at the end of a robotic arm 4. The first stage (global field of view perception and constraint): The external global camera 1 captures information about the work scene, the AGV pose, and the approximate position of the target object, calculating a "pre-observation area" for the robotic arm 4 that covers the target object. The second stage (field of view guidance and path planning): Based on global visual feedback, the robotic arm 4 is controlled to move rapidly to the "pre-observation pose," ensuring the target object falls within the center of the end-effector's field of view (FOV), and a collision-free path is planned from the current pose to the pre-observation pose. The third stage (dynamic field of view switching): The robotic arm 4 is controlled to move rapidly to the pre-observation pose, ensuring the target object is within the center of the end-effector's FOV, completing a smooth switch of control from the global coordinate system to the local coordinate system. The fourth stage (local visual servoing): The end-effector is activated, and image-based visual servoing (IBVS) is performed using close-range high-resolution images to eliminate AGV docking errors, base tilt errors caused by uneven ground, and calibration errors of the global camera 1, achieving millimeter-level final positioning.

[0061] The above specific embodiments are specific support for the concept proposed in this invention, and should not be used to limit the scope of protection of this invention. Any equivalent changes or modifications made on the basis of this technical solution in accordance with the technical concept proposed in this invention shall still fall within the scope of protection of this invention.

Claims

1. A composite robot localization method based on global and local distributed visual collaboration, characterized in that, Includes the following steps: Step S1: Use the global camera to acquire information about the work scene, as well as the pose information of the target object and the handling mechanism in the global coordinate system, and construct an environmental safety envelope; Step S2: Based on the pose information and the field of view parameters of the local camera, reverse calculate the pre-observation pose of the robotic arm. The pre-observation pose includes the end-effector pose of the robotic arm that places the target object within the optimal imaging field of view of the local camera. Step S3: Based on the pre-observed pose and combined with the environmental safety envelope obtained by the global camera, a collision-free path is planned through a pre-established global and local distributed visual cooperative localization model. Based on the collision-free path, the robotic arm is controlled to move to the pre-observed pose to complete the field of view handover of the vision system. Step S4: Use a local camera to acquire images of the target object, and use a visual servo control method to guide the end effector of the robotic arm to correct the position deviation in real time until it is positioned at the target working position.

2. The composite robot localization method based on global and local distributed vision collaboration according to claim 1, characterized in that: The method for calculating the pre-observed pose in step S2 includes: Establish a local observation coordinate system with the pose information of the target object as the origin; Based on the optimal observation distance d of the local camera opt With optical axis direction constraints, calculate the ideal pose of the end camera in the local camera in the observation coordinate system; Through coordinate transformation chain Transform the ideal pose into the end flange pose in the robot arm base coordinate system. , as the pre-observed pose; Let be the pose transformation matrix from the base coordinate system to the target object coordinate system Obj; It is the inverse of the transformation matrix from the end effector coordinate system to the target object coordinate system.

3. The composite robot localization method based on global and local distributed vision collaboration according to claim 2, characterized in that: Field of view derivation includes: setting the ideal observation distance d for the end camera. opt and ideal observation angle; The calculation of the pre-observed pose includes: the pose information of the target object. Based on this, translate along the direction of the target normal. To obtain the theoretical pose that the robotic arm's end effector should reach. ; ; This is the coarse pose transformation matrix from the base coordinate system to the target coordinate system O; Let be the offset transformation matrix from the target coordinate system O to the end effector coordinate system E; Inverse kinematics solution and optimization include: for Perform inverse kinematics calculation for the robot; if multiple solutions exist, use a weighted function to select the solution with the minimum joint movement and no singular states. Obstacle avoidance planning includes: combining the environmental point cloud provided by the global camera, and using a fast expanding random tree algorithm to plan an obstacle avoidance path from the current pose to... A collision-free path.

4. The composite robot localization method based on global and local distributed vision collaboration according to claim 2, characterized in that: The visual servo control method in step S4 includes: Construct the image feature Jacobian matrix. A mapping relationship between image feature error and robotic arm end-effector speed is established based on the image feature Jacobian matrix. By using proportional-integral-derivative control laws or adaptive gain control laws, speed control commands for the robotic arm are generated in real time to correct positioning errors caused by docking deviations of the handling mechanism, uneven ground, and global camera calibration residuals.

5. The composite robot localization method based on global and local distributed vision collaboration according to claim 4, characterized in that: Anomaly handling logic for field of view handover: After the robotic arm reaches the pre-observed pose, it checks whether the local camera has successfully identified the features of the target object; If the recognition is successful, proceed to step S4 for local visual servoing; If recognition fails, keep the robotic arm base stationary and control the end effector to perform a spiral search or zoom scan within a preset neighborhood of the pre-observed pose until the target feature is captured.

6. The composite robot localization method based on global and local distributed vision collaboration according to claim 5, characterized in that: Step S1 also includes: performing system calibration and initialization during the system deployment phase; The system deployment includes building a composite robot localization system based on global and local distributed vision collaboration; The deployment-based system establishes the coordinate transformation chain of the entire system through system calibration and initialization. System calibration and initialization include: Determine the pose matrix of the global camera relative to the world coordinate system W. It is used to achieve global camera calibration; Determine the pose matrix of the local camera relative to the end flange E of the robotic arm. It is used to achieve hand-eye calibration; Determine the rigidity transformation relationship of the robotic arm base B relative to the center A of the handling mechanism. It is used to achieve joint calibration of robot and handling mechanism.

7. The composite robot localization method based on global and local distributed vision collaboration according to claim 6, characterized in that: Acquiring pose information includes global visual perception and coarse localization; when the handling mechanism navigates to the vicinity of the work point and stops, the global vision system is triggered. Includes the following steps; The global camera captures a panoramic image containing the target object; Use deep learning object detection algorithms or QR code recognition algorithms to identify target object regions; The PnP algorithm is used to calculate the pose of the target object's coordinate system O relative to the global camera Cg. The transport organization uploads its real-time odometer data. ; Transform the target pose to the robot base coordinate system B to obtain the target pose. And as pose information: ; in, Let W be the pose transformation matrix from the world coordinate system W to the center A of the transport mechanism; The pose transformation matrix from the robot arm base B to the center A of the handling mechanism; The pose transformation matrix from the world coordinate system W to the global camera Cg; Let be the pose transformation matrix from the target object coordinate system O to the global camera Cg.

8. The composite robot localization method based on global and local distributed vision collaboration according to claim 7, characterized in that: The field of view handover of the vision system in step S3 includes the following steps: The robot controller drives the robotic arm to move rapidly along the planned path to the pre-observed pose; Images are acquired by the end camera and preprocessed using adaptive threshold segmentation and distortion correction algorithms; Identify feature points in the image and calculate the precise pose of the target relative to the end-effector camera. ; Implement pose calculation: Then it enters the closed-loop control stage, defining the current image feature s and the desired feature. error Using the image Jacobian matrix Calculate the camera velocity spinor in To control the gain, The pseudo-inverse of the Jacobian matrix is ​​used to achieve visual servo control; The robotic arm according to v c Perform real-time fine-tuning until the error e converges to the preset threshold; complete the fine-tuning approximation.

9. The composite robot localization method based on global and local distributed vision collaboration according to claim 8, characterized in that: The application to dual-arm cooperative positioning includes the following steps: Two local cameras controlling the arms observe the visual features of the target object from different positions. The observation data from two local cameras are mapped to the same world coordinate system to construct a multi-view joint optimization equation; The six-DOF pose of the target object is solved by minimizing the total reprojection error, thereby eliminating the ambiguity of pose solution under a single local viewpoint.

10. A high-precision positioning system for a composite robot based on global and local distributed vision collaboration, characterized in that: The composite robot localization method based on global and local distributed vision collaboration, as described in any one of claims 1-9, is applied to a composite robot system comprising a transport mechanism chassis, a robotic arm, a local camera mounted at the end of the robotic arm, and a global camera mounted in the external environment, comprising: The global camera is fixedly mounted on a bracket outside the work area, and its field of view covers the area where the transport mechanism stops and the area where the target object is located. A robotic arm is installed in the work area, and the robotic arm is mounted on a transport mechanism trolley. A local camera is installed at the end of the robotic arm; the transport mechanism is an intelligent transport robot AGV.

Images