A collaborative control method and system for a mobile robotic arm

Abstract

This invention relates to the field of intelligent control technology for robotic arms, and more particularly to a collaborative control method and system for a mobile robotic arm. The method involves oriented an RGB camera mounted on the end effector of the robotic arm toward the target object relative to the end effector. Based on the image captured by the RGB camera, the imaging region of the target object is extracted. Occluded areas within the imaging region are identified, and areas outside the occluded areas are designated as unoccluded areas. A comprehensive perception quality function is calculated based on the occlusion rate, target integrity, and the clarity of the image of the target object's imaging region. Based on the magnitude of the comprehensive perception quality function, it is determined whether to adjust the robotic arm chassis pose or the robotic arm joint vectors. If no adjustment is needed, the recognition result of the target object is obtained based on the image captured by the current RGB camera. If adjustment is required, the recognition result of the target object is obtained based on the image captured by the adjusted RGB camera. This invention effectively improves the observation accuracy and system robustness in complex agricultural scenarios.

Description

Technical Field

[0001] This invention relates to the field of intelligent control technology for robotic arms, and in particular to a collaborative control method and system for a mobile robotic arm. Background Technology

[0002] With the continuous development of smart agriculture, precision agriculture, and agricultural robotics, the demand for automated and intelligent sensing of crop growth status, maturity, pests and diseases, fruit quantity, and phenotypic characteristics continues to grow. Traditional manual inspection methods suffer from high labor intensity, low efficiency, strong subjectivity, and poor consistency, making it difficult to meet the actual needs of modern agriculture for high-throughput, high-precision, and low-cost monitoring. Therefore, robotic arm systems have been introduced into this field.

[0003] Current robotic arm systems typically employ two control methods. The first is a fixed robotic arm sensing scheme, where the robotic arm is fixed to a ground base, work platform, or stationary support, and a camera at the end of the robotic arm observes and identifies the target crop. This type of scheme has a simple structure and relatively mature control, suitable for regular, localized, and small-scale scenarios. However, this type of scheme has limited workspace; when the target is beyond the reach of the robotic arm, the system cannot continue its observation task. When the target is obstructed by obstacles such as branches and leaves, although the robotic arm can change its posture within a certain range, the fixed base limits the actual new perspectives it can acquire, making it difficult to truly bypass obstructions and complete large-scale, multi-point sensing tasks in the field. The second is a serial operation scheme using a mobile platform and a robotic arm. The robotic arm is mounted on a mobile platform, which first moves to the vicinity of the target, then stops, and the robotic arm performs localized sensing or operation. This "move to position—stop—robotic arm operation" approach expands the system's operating range to some extent. Therefore, mobile robotic arm systems combining mobile platforms and robotic arms, due to their simultaneous large-scale mobility and localized fine-tuning capabilities, are gradually becoming an important technological route for agricultural intelligent equipment. These systems typically integrate multiple sensors such as LiDAR, depth cameras, and RGB cameras on a chassis. They enable cross-regional movement via the chassis and local perspective adjustment and target observation via a robotic arm. They have promising applications in orchard inspection, fruit identification, crop phenotyping, and pest and disease detection.

[0004] Current robotic arm systems mostly focus on motion safety, adjusting their trajectory only when the planned path collides with or is about to collide with obstacles. However, in agricultural scenarios, robotic arm systems require automated and intelligent perception of crop growth status, maturity, pests and diseases, fruit quantity, and phenotypic characteristics. But in agricultural environments, crops grow densely, branches and leaves intertwine, and fruits often overlap or partially occlude, leading to widespread problems such as incomplete target outlines, localized image blurring, and distorted depth information. This is especially true in orchards, greenhouses, and open fields, where target crops are frequently partially obscured by leaves, branches, support rods, adjacent fruits, or environmental components. Existing serial operation solutions using mobile platforms and robotic arms address occlusion issues by expanding occlusion samples, training more robust target detection networks, or using prior models to reconstruct and complete occluded areas.

[0005] However, current solutions to the occlusion problem in robotic arm systems are essentially aimed at maximizing detection capabilities under existing observation conditions. Relying solely on post-compensation methods such as image algorithm optimization, sample expansion, and region reconstruction makes it difficult to proactively reduce the impact of occlusion on target recognition and localization at the observation pose level. Instead, they rely on model fitting and prior inference to fill in missing information, rather than real-world environmental perception. This inevitably leads to recognition bias and spatial positioning errors, resulting in decreased target positioning accuracy and significant deviations in grasping pose estimation. Consequently, problems such as robotic arm operation interference, misidentification of fruit, missed identification, or operation failure are highly likely to occur, making it difficult to adapt to the stable operation requirements in complex field occlusion environments. Summary of the Invention

[0006] Therefore, the technical problem to be solved by the present invention is to overcome the shortcomings of existing robotic arm systems when facing occlusion problems, such as passive response to occlusion, poor detection accuracy, and difficulty in adapting to the stable operation requirements in complex field occlusion environments.

[0007] To address the aforementioned technical problems, this invention provides a collaborative control method for a mobile robotic arm, comprising: Based on the current pose of the robotic arm chassis and the joint vectors of the robotic arm, the RGB camera mounted on the end of the robotic arm is oriented towards the target relative to the end of the robotic arm. Based on the image acquired by the RGB camera, the imaging area of ​​the target is extracted. Identify the occluded area in the imaging region of the target to be tested, and take the area outside the occluded area as the unoccluded area; The area of ​​the imaging region of the target to be measured is taken as the total projected area of ​​the target; The ratio of the area of ​​the occluded region to the total area of ​​the target projection is used as the occlusion rate. The ratio of the area of ​​the unobstructed region to the reference area of ​​the target is used as the target integrity. The comprehensive sensing quality function value is calculated based on the occlusion rate, target integrity, and image sharpness of the target imaging area. Based on the magnitude of the comprehensive perception quality function, it is determined whether to adjust the pose of the robotic arm chassis or the joint vectors of the robotic arm. If no adjustment is needed, the recognition result of the target under test is obtained based on the image acquired by the current RGB camera. If adjustment is needed, the recognition result of the target under test is obtained based on the image acquired by the adjusted RGB camera.

[0008] Preferably, the method for obtaining the sharpness of the image of the target imaging region includes: The image of the target imaging region is processed by the Laplacian operator, and the pixel variance of the processed image is calculated as the image sharpness.

[0009] Preferably, the method for calculating the comprehensive sensing quality function value based on occlusion rate, target integrity, and image sharpness of the target imaging area includes: The sharpness of the image of the target imaging area is normalized. Based on the occlusion rate, target integrity, and the sharpness of the normalized image of the target region, the comprehensive sensing quality function value is calculated using the following formula: , in, To comprehensively perceive the quality function, As the first weighting coefficient, This is the second weighting coefficient. This is the third weighting coefficient. It is the fourth weighting coefficient. This is the fifth weighting coefficient. , For the completeness of the target, To determine the sharpness of the normalized image, For occlusion rate, This represents an exponential function with the natural constant as its base. The distance between the center coordinates of the RGB camera mounted on the end effector of the robotic arm and the center of the target being measured. For reference observation distance, This is the distance attenuation coefficient. The angle between the optical axis of the RGB camera and the normal of the surface of the target object is denoted as .

[0010] Preferably, the method for determining whether to adjust the robot arm chassis pose or robot arm joint vectors based on the magnitude of the comprehensive sensing quality function includes: If the overall sensing quality function is greater than or equal to the set sensing quality threshold, and the target to be measured is located within the workspace of the robotic arm, then only the joint vector of the robotic arm is adjusted. If the overall perceived quality function is less than the set perceived quality threshold, or if the target to be measured exceeds the working space of the robotic arm, then the pose of the robotic arm chassis and the joint vectors of the robotic arm will be adjusted.

[0011] Preferably, the method for adjusting the pose of the robotic arm chassis and the joint vectors of the robotic arm includes: Random sampling is performed within the joint space of the robotic arm to construct multiple sets of robotic arm joint vectors; Based on the Jacobian matrix of the mobile robotic arm system under each set of robotic arm joint vectors, the normalized operability corresponding to each set of robotic arm joint vectors is calculated sequentially. Based on the normalized operability corresponding to each group of robotic arm joint vectors, the joint vectors of each group of robotic arms are filtered to obtain the target robotic arm joint vector. Construct an inverse capability graph using the inverse pose of the manipulator base relative to the end effector and the normalized operability corresponding to all target manipulator joint vectors; Based on the inverse pose of the manipulator base relative to the end effector that matches the pose of the target under test in the world coordinate system in the inverse capability diagram, a set of candidate chassis poses is obtained; each candidate chassis pose in the set of candidate chassis poses corresponds to one or more target manipulator joint vectors; Each candidate chassis pose and its corresponding target robot joint vector are used as a candidate combination; The optimal candidate combination is determined based on the normalized operability, target reachability margin, geometric observation fit, normalized chassis cost, and chassis safety margin corresponding to each candidate combination. With the goal of achieving the optimal candidate combination, the pose of the robotic arm chassis and the joint vectors of the robotic arm are adjusted.

[0012] Preferably, the method for filtering the group of robotic arm joint vectors based on the normalized operability corresponding to each group of robotic arm joint vectors to obtain the target robotic arm joint vector includes: Perform self-collision detection on each group of robotic arm joint vectors and remove robotic arm joint vectors that have self-collision. For the remaining robotic arm joint vectors after filtering at each location, sort them in descending order according to normalized operability, and select the previously set number of robotic arm joint vectors as the target robotic arm joint vectors.

[0013] Preferably, the method for determining the optimal candidate combination based on the normalized operability, target reachability margin, geometric observation fit, normalized chassis cost, and chassis safety margin corresponding to each candidate combination includes: Based on the normalized operability, target reachability margin, geometric observation fit, normalized chassis cost, and chassis safety margin corresponding to each candidate combination, the comprehensive score of each candidate combination is calculated using the following formula: , in, For the first The overall score of each candidate combination For candidate combination index, It is the sixth weighting coefficient. It is the seventh weighting coefficient. This is the eighth weighting coefficient. It is the ninth weighting coefficient. This is the tenth weighting coefficient. , For the first The normalized operability corresponding to each candidate combination For the first The target achievable margin corresponding to each candidate combination For the first Geometric observation fit of each candidate combination For the first The normalized chassis cost corresponding to each candidate combination For the first Chassis safety margin corresponding to each candidate combination; The candidate combination with the highest overall score is selected as the optimal candidate combination.

[0014] Preferably, the formula for calculating the target attainability margin for each candidate combination is: , in, For the first The target achievable margin corresponding to each candidate combination For the first Under each candidate combination, the distance from the robotic arm base to the center of the target being measured is... To set the ideal observation distance for the robotic arm, , These represent the minimum and maximum observable distances of the robotic arm, respectively. To prevent extremely small positive numbers with a denominator of zero; The formula for calculating the geometric observation fit for each candidate combination is: , in, For the first Geometric observation fit of each candidate combination For the first Under the candidate combinations, the angle between the optical axis direction of the RGB camera and the direction from the center of the RGB camera to the center of the target being measured. The maximum allowable angle; The process of normalizing the chassis cost for each candidate combination includes: After normalizing the chassis cost for each candidate combination, the normalized chassis cost for each candidate combination is obtained; the formula for calculating the chassis cost for each candidate combination is as follows: , in, For the first The chassis cost corresponding to each candidate combination. Let x be the current x-coordinate of the robotic arm chassis. Let y be the current y-coordinate of the robotic arm chassis. For the first The chassis x-coordinates corresponding to each candidate combination For the first The chassis y-coordinates corresponding to each candidate combination; The formula for calculating the chassis safety margin for each candidate combination is as follows: , in, For the first Chassis safety margin corresponding to each candidate combination This indicates taking the minimum value. For the first The minimum distance between the candidate chassis pose and its nearest obstacle in each candidate combination. To establish a safe distance.

[0015] Preferably, the identification results of the target under test under the pose of the robot arm chassis and the joint vectors of the robot arm are obtained; Based on the recognition results of the target under the pose of the robotic arm chassis and the joint vectors of the robotic arm in each group, the final recognition result of the target is obtained.

[0016] The present invention also provides a collaborative control system for a mobile robotic arm, comprising: The imaging region acquisition module is used to, based on the current pose of the robotic arm chassis and the joint vectors of the robotic arm, orient the RGB camera mounted on the end of the robotic arm toward the target relative to the end of the robotic arm, and extract the imaging region of the target based on the image acquired by the RGB camera. The region recognition module is used to identify the occluded area in the imaging region of the target under test, and to regard the area outside the occluded area as the unoccluded area. The total area acquisition module is used to obtain the area of ​​the imaging region of the target under test as the total projected area of ​​the target; The occlusion rate calculation module is used to calculate the occlusion rate as the ratio of the area of ​​the occluded region to the total area of ​​the target projection. The target integrity calculation module is used to determine the target integrity by the ratio of the area of ​​the unobstructed region to the reference area of ​​the target under test. The function calculation module is used to calculate the comprehensive sensing quality function value based on the occlusion rate, target integrity, and the sharpness of the image of the imaging area of ​​the target under test; The recognition module is used to determine whether to adjust the pose of the robotic arm chassis or the joint vector of the robotic arm based on the magnitude of the comprehensive perception quality function. If no adjustment is needed, the recognition result of the target under test is obtained based on the image acquired by the current RGB camera; if adjustment is needed, the recognition result of the target under test is obtained based on the image acquired by the adjusted RGB camera.

[0017] Compared with the prior art, the above-described technical solution of the present invention has the following advantages: The present invention discloses a collaborative control method and system for a mobile robotic arm. This invention extracts the imaging region of the target in an image, identifies occlusion regions, calculates the target occlusion rate and target integrity, and constructs a comprehensive perception quality function based on image clarity to quantitatively evaluate the imaging quality from the current observation perspective. Simultaneously, based on the comprehensive perception quality, it adaptively determines whether to optimize the robotic arm chassis pose and joint vectors. By adjusting the observation pose, it actively avoids occlusion interference, reduces the impact of occlusion, edge loss, and poor local perspectives on depth estimation and target localization, and completes the target contour missing from a single perspective. This optimizes imaging conditions from the observation source, acquires high-quality target images, and achieves accurate target recognition.

[0018] Furthermore, to address the need to change the chassis observation position, existing methods often employ online search or online optimization to find a suitable pose. This approach is prone to problems such as high computational cost, slow solution speed, and susceptibility to local optima when dealing with complex scenarios, numerous constraints, and the requirement for rapid response. This invention employs an offline mapping combined with online rapid querying strategy for pose solving. It pre-samples the target region within the robot arm joint space, generating multiple sets of robot arm joint vectors. Normalized operability is calculated using the Jacobian matrix corresponding to each configuration. Based on operability, high-quality target robot arm joint vectors are selected, and the inverse pose of the corresponding robot arm base relative to the end effector is matched to complete the offline construction of the inverse capability map. During online operation, only the inverse pose of the robot arm base relative to the end effector needs to be matched based on the position of the target object. A set of candidate chassis poses is quickly derived, and a collaborative candidate combination of chassis pose and robot arm joint vectors is constructed. The optimal combination is selected by combining the normalized comprehensive perception quality, operability, and chassis movement distance of each candidate combination. This ultimately achieves rapid collaborative adjustment of chassis pose and robot arm joint vectors, effectively avoiding the massive computational problems of online iterative optimization, reducing the risk of getting trapped in local optima during continuous online optimization, and quickly obtaining a better comprehensive evaluation observation combination within the discrete candidate space, meeting the real-time response requirements of the operation. Attached Figure Description

[0019] To make the content of this invention easier to understand, the invention will be further described in detail below with reference to specific embodiments and accompanying drawings, wherein:

[0020] Figure 1 This is an isometric view of a mobile robotic arm system.

[0021] Figure 2 This is a flowchart illustrating a collaborative control method for a mobile robotic arm according to the present invention.

[0022] Figure 3 This is a flowchart illustrating the steps of establishing unified kinematic modeling and multi-sensor joint calibration in this invention.

[0023] Figure 4 This is a flowchart illustrating the environmental perception and map construction steps of the present invention.

[0024] Figure 5 This is a flowchart illustrating the target recognition, occlusion detection, and perception quality assessment steps of the present invention.

[0025] Figure 6 This is a flowchart illustrating the steps for constructing the robotic arm capability diagram and inverse capability diagram of the present invention.

[0026] Figure 7 This is a schematic diagram of the vehicle-arm cooperative control decision-making process based on perceived quality according to the present invention.

[0027] Figure 8 This is a flowchart illustrating the optimal observation point and collaborative control execution steps of the present invention.

[0028] The following are the labeling symbols in the instruction manual's attached diagrams: 1. Chassis; 2. Depth point cloud camera; 3. Multi-line LiDAR; 4. RGB camera; 5. Robotic arm; 6. Control and computing unit. Detailed Implementation

[0029] The present invention will be further described below with reference to the accompanying drawings and specific embodiments, so that those skilled in the art can better understand and implement the present invention. However, the embodiments described are not intended to limit the present invention.

[0030] like Figure 1 As shown, Figure 1 This is an isometric view of the mobile robotic arm system.

[0031] refer to Figure 2 This embodiment provides a collaborative control method for a mobile robotic arm, including: like Figure 3 As shown, Figure 3 This is a flowchart illustrating the steps of establishing unified kinematic modeling and multi-sensor joint calibration in this invention.

[0032] The method of this invention is based on a unified coordinate system. The coordinate system and parameters of each component of the mobile robotic arm system have been calibrated using conventional calibration methods, as detailed below: This invention establishes a world coordinate system {W}, a mobile platform coordinate system {B}, a robotic arm base coordinate system {H}, and an end effector coordinate system {E}. It calibrates the kinematic parameters of the Ackerman mobile platform, calibrates the DH parameters of the robotic arm 5, and performs joint calibration on the depth point cloud camera 2, the multi-line LiDAR 3, and the RGB camera 4, thereby establishing a unified kinematic model and a system Jacobian matrix.

[0033] Establish the kinematic model of the Ackerman chassis, and the robot arm chassis in position 1. Defined as: , in, Let x be the x-coordinate of the chassis center in the world coordinate system. Let y be the plane position of the chassis center in the world coordinate system. The heading angle of chassis 1, This indicates transpose.

[0034] Ackermann chassis kinematics model control input Defined as: , in, Let the linear velocity of chassis 1 be _____. This is the equivalent steering angle of the front wheels.

[0035] The kinematic model of the Ackermann chassis is: , in, , , These represent the rates of change of the chassis center's lateral position, longitudinal position, and heading angle in the world coordinate system, respectively. This refers to the chassis wheelbase.

[0036] Robotic arm joint variables for: , in, For the first robotic arm The motion variables of a joint are as follows: when the joint is a rotational joint, the corresponding motion variable is the joint angle; when the joint is a translational joint, the corresponding motion variable is the joint displacement. This represents the total degrees of freedom of the robotic arm. In this invention, .

[0037] A forward kinematic model of robotic arm 5 is established using DH parameters, and the pose of the end effector in the robotic arm base coordinate system is determined. for: , in, The pose of the end effector in the coordinate system of the robot arm base. For the first robotic arm The homogeneous transformation matrix corresponding to each joint. Index for robotic arm joints.

[0038] The pose of the end effector in the world coordinate system satisfies: , in, This represents the pose of the end effector in the world coordinate system. For the installation and transformation of the robotic arm base relative to chassis 1, Let be the pose transformation matrix of the mobile platform in the world coordinate system. This represents the pose of the end effector in the coordinate system of the robot arm base.

[0039] Joint calibration of depth point cloud camera 2, multi-line lidar 3 and end-effector RGB camera 4: Based on geometric relationships, obtain the installation relationships of the sensors to the world coordinate system, chassis coordinate system and end-effector coordinate system, and obtain the pose of each sensor in the world coordinate system.

[0040] The pose of sensor S in the world coordinate system can be expressed as: , in, Let S be the pose of sensor S in the world coordinate system. Let be the pose transformation matrix of the mobile platform in the world coordinate system. Let S be the pose of sensor S in the coordinate system of the moving platform.

[0041] The pose of the mobile platform and the joint angles of the robotic arm are uniformly represented as a system state vector, and the mobile robotic arm system has a unified state vector. It can be represented as: , Mobile robotic arm system velocity vector for: , The superscript · indicates the differentiation operation.

[0042] The following relationship exists between the end effector speed and the mobile robotic arm system speed: , in, For the end effector speed, The Jacobian matrix of the mobile robotic arm system. Let be the velocity vector of the mobile robotic arm system.

[0043] Step S1: Under the current pose of the robotic arm chassis and the joint variables of the robotic arm, make the RGB camera 4 mounted on the end of the robotic arm 5 face the target to be tested in the direction relative to the end of the robotic arm, and extract the imaging area of ​​the target to be tested based on the image acquired by the RGB camera 4. In this embodiment, specifically, the method for obtaining the direction of the target relative to the end effector of the robotic arm includes: Based on the point cloud map of the spatial region where the target is located, the orientation of the target relative to the end effector of the robotic arm is obtained.

[0044] like Figure 4 As shown, Figure 4 This is a flowchart illustrating the environmental perception and map construction steps of the present invention.

[0045] Methods for constructing point cloud maps of the spatial region where the target to be tested is located include: The environmental point cloud of the spatial region where the target is located is acquired using a multi-line lidar 3, and a two-dimensional occupancy grid map is constructed using a simultaneous localization and mapping (SLAM) algorithm. , in, It is a two-dimensional occupancy raster map. These represent the horizontal and vertical coordinates of the raster cell within the two-dimensional raster map, respectively. This is the set of grid cell state values: 0 indicates that the grid area is a free and passable area, 0.5 indicates that the grid area is an unknown area, and 1 indicates that the grid area is occupied by obstacles.

[0046] Acquire local 3D point clouds using depth point cloud camera 2: , in, It is a local 3D point cloud collection. For the first A local 3D point cloud, For the first Spatial coordinates of a local 3D point cloud For the first Color information of a local 3D point cloud.

[0047] The environmental point cloud and local 3D point cloud of the spatial region containing the target are fused together under the world coordinate system. The formula is as follows: , in, This indicates that the environmental point cloud of the spatial region where the target is located is transformed to the world coordinate system. This indicates the transformation of a local 3D point cloud to the world coordinate system. This is a collection of environmental point clouds representing the spatial region where the target is located. This is the fused point cloud set.

[0048] Based on the constructed 2D occupancy raster map, the fused point cloud set is filtered: Redundant point clouds that exceed the coverage of the raster map are removed, and only point clouds whose planar projection falls within the range of the raster map are retained, thus obtaining a fused point cloud map of the spatial region where the target is located.

[0049] After voxel filtering downsampling of the fused point cloud map of the spatial region where the target is located, the voxel size is set to reduce the point cloud density while preserving geometric features. Then, ground segmentation is performed to separate the ground point cloud from the non-ground point cloud. The regional point cloud of the target (such as crops) is extracted, the target is segmented, and individual target point cloud clusters are identified. Features are extracted for each target point cloud cluster, including center position, bounding box, volume estimation, etc.

[0050] Extract the target point cloud cluster from the fused point cloud map of the spatial region where the target is located, and calculate the position coordinates of the target's center point in the world coordinate system based on the target point cloud cluster. Combine the chassis pose, robotic arm installation transformation, and the robotic arm's forward kinematics model to solve for the end effector's position coordinates in the world coordinate system. Using the end effector's position coordinates in the world coordinate system as the starting point and the target's center point's position coordinates in the world coordinate system as the ending point, construct a direction vector, and normalize the direction vector to obtain the spatial direction of the target relative to the robotic arm's end effector. In this embodiment, the coordinates of the two-dimensional center pixel of the target in the image acquired by the RGB camera 4 are assumed to be... Depth value Camera internal parameters are Then the three-dimensional position of the target to be measured can be back-projected as: , in, , , These represent the x, y, and z coordinates of the target object in the end-camera coordinate system, respectively. , These are the horizontal coordinates and vertical coordinates of the camera's principal image point, respectively. , These are the camera's horizontal equivalent focal length and vertical equivalent focal length, respectively.

[0051] The pose of the target in the end-camera coordinate system is given by the homogeneous transformation matrix. Description, homogeneous transformation matrix for: , in, It is a translation vector. , The rotation matrix is ​​constructed by taking the normal to the surface of the target as the main axis and the line of sight from the center of the target to the optical center of the camera as the observation axis, thus forming a local orthogonal coordinate system of the target.

[0052] The pose of the target in the world coordinate system {W} is: , in, Let the target be in the world coordinate system as its pose. This represents the pose transformation matrix of the end effector in the world coordinate system. The pose of the target object in the end-camera coordinate system.

[0053] like Figure 5 As shown, Figure 5 This is a flowchart illustrating the target recognition, occlusion detection, and perception quality assessment steps of the present invention.

[0054] In this embodiment, a deep learning semantic segmentation network is used to perform pixel-level segmentation on the image acquired by the RGB camera 4 to obtain a mask of the imaging region of the target to be tested. The imaging region of the target under test is extracted using a mask of the imaging region of the target under test.

[0055] Step S2: Identify the occluded area in the imaging region of the target to be tested, and take the area outside the occluded area as the unoccluded area; In this embodiment, non-target occluders (such as leaves, branches, supporting structures, etc.) in the image acquired by RGB camera 4 are segmented to obtain an occluder mask; By combining depth map data, depth information is compared pixel by pixel only within the imaging area of ​​the target to be tested. For pixels that are simultaneously within the mask of the occluder, the front-to-back relationship is determined along the same line of sight of the camera: if the depth value corresponding to the pixel in the target area is greater than the depth value of the pixel in the corresponding occluder, the pixel is determined to be the occluded pixel of the target; otherwise, it is determined to be an unoccluded visible pixel. The region consisting of all the pixels of the target that are occluded is the occluded region, and the mask for the occluded region is... The area of ​​the occluded region is the total number of pixels of all targets that are occluded. The region consisting of all unobstructed visible pixels is the unobstructed region, and the area of ​​the unobstructed region is the total number of all unobstructed visible pixels.

[0056] Step S3: Use the area of ​​the imaging region of the target to be measured as the total projected area of ​​the target; The area of ​​the imaging region of the target under test is equal to the total number of pixels in the imaging region of the target under test, and its calculation formula is as follows: , in, The total projected area of ​​the target Indicates the x-axis index. Indicates the y-axis index. This is a binary mask for the imaging region of the target to be tested.

[0057] Step S4: The ratio of the area of ​​the occluded region to the total area of ​​the target projection is used as the occlusion rate; The formula for calculating the area of ​​the obscured region is: , in, The area of ​​the obscured region. It is a binary mask for the occluded area.

[0058] Occlusion rate The calculation formula is: , in, , This indicates that there is no obvious obstruction. This indicates severe obstruction.

[0059] Step S5: The ratio of the area of ​​the unobstructed region to the reference area of ​​the target to be measured is used as the target integrity. The formula for calculating target completeness is: , in, For the completeness of the target, The area of ​​the unobstructed area. The reference area of ​​the target to be measured. .

[0060] Step S6: Calculate the comprehensive sensing quality function value based on the occlusion rate, target integrity, and image sharpness of the target imaging area; In this embodiment, specifically, the method for obtaining the sharpness of the image of the target imaging region includes: The image of the target imaging region is processed using the Laplacian operator, and the pixel variance of the processed image is calculated as the sharpness of the target imaging region image. The formula is as follows: , in, The sharpness of the image of the target imaging area is considered. The result of processing the Laplace operator. This indicates the calculation of variance. The larger the value, the clearer the image.

[0061] In this embodiment, preferably, the method for calculating the comprehensive sensing quality function value based on occlusion rate, target integrity, and image sharpness of the target imaging area includes: The sharpness of the image of the target imaging area is normalized. Based on the occlusion rate, target integrity, and the sharpness of the normalized image of the target region, the comprehensive sensing quality function value is calculated using the following formula: , in, To comprehensively perceive the quality function, As the first weighting coefficient, This is the second weighting coefficient. This is the third weighting coefficient. It is the fourth weighting coefficient. This is the fifth weighting coefficient. , For the completeness of the target, The sharpness of the normalized image of the target region is given by [reference to image quality]. For occlusion rate, This represents an exponential function with the natural constant as its base. The distance between the center coordinates of the RGB camera 4 mounted on the end effector of the robotic arm and the center of the target being measured. For reference observation distance, This is the distance attenuation coefficient. The angle between the optical axis of the RGB camera and the normal of the surface of the target object is denoted as .

[0062] Step S7: Based on the magnitude of the comprehensive perception quality function, determine whether to adjust the pose of the robotic arm chassis or the joint variables of the robotic arm. If no adjustment is needed, obtain the recognition result of the target under test based on the image acquired by the current RGB camera 4. If adjustment is needed, obtain the recognition result of the target under test based on the image acquired by the adjusted RGB camera 4.

[0063] In this embodiment, specifically, the method for determining whether to adjust the robot arm chassis pose or robot arm joint variables based on the magnitude of the comprehensive sensing quality function includes: If the overall perceived quality function is greater than or equal to the set perceived quality threshold If the target to be measured is located within the workspace of the robotic arm, then only the joint variables of the robotic arm are adjusted; If the overall perceived quality function is less than the set perceived quality threshold If the target being measured exceeds the working space of the robotic arm, the pose of the robotic arm chassis and the joint variables of the robotic arm will be adjusted.

[0064] when If the overall perception quality meets the requirements, it is determined that the overall perception quality is insufficient; otherwise, it is determined that the overall perception quality is insufficient.

[0065] This invention switches modes based on two conditions: whether the target to be measured is located within the workspace of the robotic arm and whether the current overall perception quality is greater than or equal to a set perception quality threshold.

[0066] If the overall perceived quality function is greater than or equal to the set perceived quality threshold If the target to be measured is located within the workspace of the robotic arm, then the robotic arm independent perception mode is executed: the robotic arm 5 adjusts the posture of the RGB camera mounted at the end of the robotic arm within a local range without moving the chassis 1, and searches for the optimal RGB camera observation posture within the reachable space of the robotic arm; the RGB camera posture is determined by the joint variables of the robotic arm.

[0067] If the target is not within the current workspace of the robotic arm, the platform-dominated mode is executed. In this case, the chassis 1 will first perform overall pose adjustment to bring the target into the reachable space of the robotic arm, and then the robotic arm 5 will perform local observation optimization.

[0068] If the target is within the workspace but < The platform-arm coupling mode is executed. That is, although the target under test is in the reachable space of the robotic arm, the current view quality is insufficient, indicating that simple local adjustment of the robotic arm may not be enough to eliminate occlusion, and joint optimization of chassis 1 and robotic arm 5 is required.

[0069] This invention can proactively adjust the observation position and attitude when it detects insufficient target integrity, excessive occlusion rate, or decreased image clarity, thereby more effectively mitigating the impact of occlusion and improving target visibility and recognition accuracy. When the target is already within the robotic arm's workspace and the observation quality is high, this invention prioritizes the robotic arm's independent sensing mode, reducing unnecessary movement of the chassis 1 and improving operational efficiency. When the target is beyond the robotic arm's reach or the observation quality is insufficient, it switches to a vehicle-arm collaborative mode to improve the overall system reachability and observation capability. This strategy achieves a balance between efficiency and capability.

[0070] Existing observation pose search methods typically rely on online iterative solutions, resulting in poor real-time performance and difficulty in adapting to complex agricultural scenarios. To address this issue, this invention constructs a robotic arm capability graph and inverse capability graph offline, enabling rapid reverse lookup of candidate chassis observation poses from the target pose. This method avoids the high computational cost of traditional online continuous optimization search, significantly improving online planning efficiency and real-time response capabilities.

[0071] like Figure 6 , Figure 7 As shown, Figure 6 This is a flowchart illustrating the steps involved in constructing the capability graph and inverse capability graph of the robotic arm according to the present invention. Figure 7 This is a schematic diagram of the vehicle-arm cooperative control decision-making process based on perceived quality according to the present invention.

[0072] In this embodiment, preferably, the method for adjusting the pose of the robotic arm chassis and the joint variables of the robotic arm includes: Random sampling is carried out within the joint space of the robotic arm, and multiple sets of robotic arm joint variables are constructed based on the joint angles of each set of samples. The joint variables of the robotic arm are: , in, For the first Group the joint variables of the robotic arm. For the index of the robotic arm joint variables, For the first In the group of robotic arm joint variables, the 5th robotic arm Motion variables of each joint This represents the total number of joint variables in the robotic arm.

[0073] In this embodiment, preferably, the number of sampling times The sampling range for each joint is the physical limit range of that joint, and the sampling method can be uniform random sampling.

[0074] Based on the Jacobian matrix of the mobile robotic arm system under each group of robotic arm joint variables, the normalized operability corresponding to each group of robotic arm joint variables is calculated sequentially. The formula for calculating the operability corresponding to the joint variables of a robotic arm is: , in, For the first The operability corresponding to the joint variables of the robotic arm For the first Jacobian matrix of a mobile robotic arm system under grouped joint variables. Represents a determinant.

[0075] Maneuverability describes the ability of a robotic arm to make fine adjustments and precise movements near its end effector position. The larger the value, the greater the flexibility in the vicinity of that posture.

[0076] Normalize the operability corresponding to the joint variables of the robotic arm to obtain the normalized operability corresponding to the joint variables of the robotic arm. The formula is as follows: , in, For the first Normalized operability corresponding to the joint variables of the robotic arm It is a function for maximizing the value.

[0077] This invention constructs a capability map (CM) based on the pose of the robotic arm base relative to the end effector and the normalized maneuverability corresponding to all robotic arm joint variables. The formula is as follows: , in, For the first The pose of the robotic arm base relative to the end effector corresponds to the joint variables of the robotic arm. For the first Normalized operability corresponding to the joint variables of the robotic arm This represents the total number of joint variables in the robotic arm. Index for the joint variables of the robotic arm.

[0078] Based on the normalized operability corresponding to each group of robotic arm joint variables, the target robotic arm joint variables are obtained by filtering the group of robotic arm joint variables, including: Perform self-collision detection on the joint variables of each group of robotic arms and remove the joint variables of robotic arms that have self-collision. In this embodiment, self-collision detection and environmental safety boundary filtering are performed on each group of robotic arm joint variables. If a collision is detected, the sample is discarded. The remaining robotic arm joint variables after filtering constitute the valid sample set. Recorded as: .

[0079] For the remaining robotic arm joint variables after filtering at each location, sort them in descending order according to normalized operability, and select the previously set number of robotic arm joint variables as the target robotic arm joint variables.

[0080] An inverse capability graph (ICM) is constructed using the inverse pose of the manipulator base relative to the end effector corresponding to all target manipulator joint variables and the normalized maneuverability. The formula is as follows:

[0081] in, For the first The inverse pose of the robot arm base relative to the end effector corresponds to the joint variables of the target robot arm. For the first Normalized operability corresponding to the joint variables of the target robotic arm Index the joint variables of the target robotic arm. To set the quantity, that is, the total number of joint variables of the target robotic arm. , , To retain the proportion.

[0082] Based on the inverse pose of the robotic arm base relative to the end effector, which matches the pose of the target under test in the world coordinate system in the inverse capability map, a set of candidate chassis poses is obtained, including: Based on the pose of the target in the world coordinate system Based on the constraints of observation distance and observation angle (with the camera optical axis as close as possible to the normal of the target surface), calculate the desired pose of the end effector; The desired pose of the end effector is the end effector pose that matches the pose of the target under test in the world coordinate system. Using the desired pose of the end effector as the query condition, the candidate pose set of the corresponding robot arm base in the world coordinate system is retrieved from the inverse capability graph. , in, This represents the candidate pose of the robot arm base in the world coordinate system, corresponding to the desired pose of the end effector. This represents the inverse pose of the robotic arm base relative to the end effector, corresponding to the desired pose of the end effector. This represents the desired pose of the end effector.

[0083] Since the robotic arm base {H} is rigidly connected and fixedly mounted on the mobile chassis {B}, based on the installation transformation matrix between the two... The candidate poses of the robot arm base in the world coordinate system corresponding to the desired pose of the end effector can be used to determine the candidate chassis pose, as shown in the formula: , in, for The pose transformation matrix of the corresponding mobile platform in the world coordinate system.

[0084] Will The candidate chassis pose is obtained by converting the pose into two-dimensional plane coordinates and heading angle.

[0085] In the candidate pose set of the robot arm base in the world coordinate system corresponding to the desired pose of the end effector, each candidate pose of the robot arm base in the world coordinate system corresponds to a unique candidate chassis pose, and the candidate chassis pose set is obtained by summing them up. .

[0086] in, For the first One candidate chassis position. For the first The x-coordinate of the plane position of the chassis center in the world coordinate system corresponding to each candidate chassis pose. For the first The y-coordinate of the plane position of the chassis center in the world coordinate system corresponding to each candidate chassis pose. For the first The heading angle of the chassis corresponding to each candidate chassis pose. Indicates transpose. For candidate chassis pose indexes This represents the total number of candidate chassis poses. In this embodiment, to accelerate the querying of all end effector poses that match the location of the target under test in the inverse capability graph, a spatial index, such as an octree or hash index, is established on the inverse capability graph (ICM).

[0087] Each candidate chassis pose and its corresponding target robotic arm joint variables are used as a candidate combination; The Inverse Capability Map (ICM) pre-stores multiple sets of sample records. Each record synchronously stores a set of target manipulator joint vectors, the inverse pose of the manipulator base relative to the end effector, and the normalized operability. When searching the ICM with the same desired end effector pose, multiple different sample records can be matched. Different sample records have different target manipulator joint vectors and inverse poses. After kinematic pose calculation and mapping, multiple records can be solved to obtain the same candidate chassis pose. In the candidate chassis pose set, each candidate chassis pose corresponds to the target manipulator joint vector corresponding to one or more sample records in the ICM. Each candidate chassis pose and its native corresponding target manipulator joint vector in the ICM are used as a candidate combination.

[0088] The optimal candidate combination is determined based on the comprehensive perception quality function value, target reachability margin, geometric observation fit, normalized chassis cost, and chassis safety margin corresponding to each candidate combination. Candidate poses must meet the following requirements: chassis does not collide with obstacles, path is passable, target can enter the robotic arm's workspace, robotic arm posture meets joint limits, robotic arm does not self-collision, and end-effector roll and pitch angles are within tolerance.

[0089] With the goal of achieving the optimal candidate combination, the pose of the robotic arm chassis and the joint variables of the robotic arm are adjusted.

[0090] In this embodiment, preferably, the method for determining the optimal candidate combination based on the comprehensive sensing quality function value corresponding to each candidate combination, the normalized operability, and the moving distance from the current chassis pose to the candidate chassis pose in each candidate combination includes: Based on the normalized operability, the normalized integrated sensing quality function value, and the distance traveled from the current chassis pose to the candidate chassis pose in each candidate combination for each candidate combination, the comprehensive score for each candidate combination is calculated using the following formula: , in, For the first The overall score of each candidate combination For candidate combination index, It is the sixth weighting coefficient. It is the seventh weighting coefficient. This is the eighth weighting coefficient. It is the ninth weighting coefficient. This is the tenth weighting coefficient. , For the first The normalized operability corresponding to each candidate combination For the first The target reachability margin corresponding to each candidate combination is used to evaluate whether the target is within the target working area of ​​the robotic arm. For the first The geometric observation fit of each candidate combination is used to determine whether the RGB camera 4 corresponding to the candidate combination is roughly facing the target and whether the observation distance is reasonable. For the first The normalized chassis cost corresponding to each candidate combination is used to constrain the spatial distance between the candidate chassis pose and the current chassis pose, thus avoiding distant candidate points. For the first The chassis safety margin corresponding to each candidate combination is used to evaluate the feasibility and safety of moving chassis 1 from its current position to the candidate chassis pose. The candidate combination with the highest overall score is selected as the optimal candidate combination.

[0091] Preferably, the formula for calculating the target attainability margin for each candidate combination is: , in, For the first The target achievable margin corresponding to each candidate combination For the first Under each candidate combination, the distance from the robotic arm base to the center of the target being measured is... To set the ideal observation distance for the robotic arm, , These represent the minimum and maximum observable distances of the robotic arm, respectively. To prevent extremely small positive numbers with a denominator of zero; The formula for calculating the geometric observation fit for each candidate combination is: , in, For the first Geometric observation fit of each candidate combination For the first Under the candidate combinations, the angle between the optical axis direction of the RGB camera and the direction from the center of the RGB camera to the center of the target being measured. The maximum allowable angle; The process of normalizing the chassis cost for each candidate combination includes: After normalizing the chassis cost for each candidate combination, the normalized chassis cost for each candidate combination is obtained; the formula for calculating the chassis cost for each candidate combination is as follows: , in, For the first The chassis cost corresponding to each candidate combination. Let x be the current x-coordinate of the robotic arm chassis. Let y be the current y-coordinate of the robotic arm chassis. For the first The chassis x-coordinates corresponding to each candidate combination For the first The chassis y-coordinates corresponding to each candidate combination; The formula for calculating the chassis safety margin for each candidate combination is as follows: , in, For the first Chassis safety margin corresponding to each candidate combination This indicates taking the minimum value. For the first The minimum distance between the candidate chassis pose and its nearest obstacle in each candidate combination. To establish a safe distance.

[0092] The candidate combination with the highest overall score is selected as the optimal candidate combination.

[0093] When the chassis observation position needs to be changed, existing methods often use online search or online optimization to find a suitable pose. This approach is prone to problems such as high computational cost, slow solution speed, and getting trapped in local optima when the scene is complex, has many constraints, and requires rapid response. It is difficult to meet the requirements of real-time applications in agricultural fields. This invention achieves rapid reverse lookup of candidate chassis observation poses from the target pose by constructing the robot arm's capability graph and inverse capability graph offline. This method avoids the high computational cost of traditional online continuous optimization search, significantly improving online planning efficiency and real-time response capability.

[0094] In mobile robotic arm systems, existing vehicle-arm cooperative control methods typically employ task weighting and zero-space projection. While these methods are effective in scenarios with fewer constraints, in agricultural settings, the system is often simultaneously affected by factors such as chassis steering constraints, robotic arm joint limit constraints, robotic arm self-collision constraints, environmental obstacle collision constraints, and real-time requirements. Traditional weighted control methods struggle to guarantee that safety tasks always take priority; zero-space projection methods are prone to numerical instability and solution difficulties when handling inequality constraints. Therefore, current technology still lacks a truly suitable control scheme for complex agricultural scenarios that can unify safety constraints, perception tasks, and vehicle-arm cooperation.

[0095] like Figure 8 As shown, Figure 8 This is a flowchart illustrating the optimal observation point and collaborative control execution steps of the present invention.

[0096] In this embodiment, the method for adjusting the pose of the robotic arm chassis and the joint vectors of the robotic arm with the optimal candidate combination as the objective includes: With the goal of achieving the optimal candidate combination, the HQP hierarchical quadratic programming framework is used to adjust the pose of the robotic arm chassis and the joint vectors of the robotic arm.

[0097] To simultaneously satisfy safety constraints, end-point observation tasks, and chassis path tracking tasks, this invention employs a hierarchical quadratic programming (HQP) control framework to coordinate control tasks according to priority. The first layer comprises safety constraint tasks, including joint position limit constraints, joint velocity limit constraints, self-collision constraints, and environmental virtual wall constraints. These safety constraints are transformed into control barrier functions (CBF) in the form of: , in, for The derivative with respect to time represents the rate of change of the safety margin, where s represents the system state. It is a positive gain coefficient. The barrier function represents the safety margin of the current state of the mobile robotic arm system relative to the safety constraint boundary.

[0098] The second layer involves the robotic arm's end-effector operations, including end-effector trajectory tracking control to minimize end-effector velocity errors. , in, The Jacobian matrix at the end effector of the robotic arm. For the desired speed of the end effector, For the joint speed of the robotic arm, This represents the L2 norm.

[0099] The third layer is the mobile platform path tracking task: the platform expects the path to be a Dobbins curve from the current position to the target pose. Considering the minimum turning radius constraint of the Ackermann chassis, the platform speed error is minimized. , in, Jacobian matrix for mobile platforms Expected speed for mobile platforms.

[0100] The fourth layer is for optimization tasks: including maximizing the operability of the robotic arm and optimizing joint configuration to make the joint angles close to the ideal value and avoid joint limits.

[0101] The HQP problem is solved hierarchically using a QP solver. Each layer uses the solution from the previous layer as an equality constraint to solve the optimization problem of the current layer and obtain the optimal joint velocity. And chassis control input. The chassis control inputs are sent to the underlying servo controller for execution. During movement, the HQP solution is updated in real time to achieve dynamic feedback control. Safety constraint violations are monitored in real time; if an impending collision or joint over-limit is detected, an emergency stop or replanning is immediately triggered.

[0102] When the robotic arm moves from the current robotic arm joint vector to the target robotic arm joint vector, an obstacle avoidance planning algorithm based on random sampling is used.

[0103] Get the current starting joint vector of the robotic arm and target robotic arm joint vector In addition to environmental obstacle information, a robotic arm configuration space C-space is constructed, and obstacles are mapped as infeasible regions in the C-space. .

[0104] Initialize the RRT tree T to Given a root node, randomly sample a point in the C-space and find the node in the tree T that is closest to the sampled point. ,from Expand the step size in the direction of the sampling point to obtain a new node. Detection from arrive Is the path consistent with Collision, if no collision, then Add tree T. Periodically try to connect tree T with... Connection, if from arrive If the path has no collisions, then the path has been successfully found; The planned path is smoothed and post-processed, and a smooth trajectory is generated using cubic spline interpolation. During the movement of the robotic arm, environmental changes are monitored in real time. If a new obstacle appears, local replanning is triggered, and a new obstacle avoidance path is planned from the current position to the target.

[0105] In this embodiment, preferably, the identification results of the target under test under the pose of the robot arm chassis and the joint vector of the robot arm are obtained; Based on the recognition results of the target under the pose of the robotic arm chassis and the joint vectors of the robotic arm in each group, the final recognition result of the target is obtained.

[0106] To improve recognition reliability in complex occlusion scenarios, this invention supports multi-viewpoint observation and recording of perception data from each observation point, establishing a multi-viewpoint observation sequence: , in, This is a multi-viewpoint observation sequence. For the first Images captured by an RGB camera under the pose of the robotic arm chassis and the joint vectors of the robotic arm. For the first Point cloud of the robotic arm chassis pose and joint vectors. For the first The combined perceived mass function value of the robotic arm chassis pose and the robotic arm joint vectors. For the first The identification category of the target under test based on the pose of the robotic arm chassis and the joint vectors of the robotic arm. For the first The confidence level of target recognition under the pose of the robotic arm chassis and the joint vectors of the robotic arm. The number of combinations of the robot arm chassis pose and the robot arm joint vectors.

[0107] In this embodiment, optionally, a Bayesian fusion method is used to fuse the identification results of the target under the robot arm chassis pose and the robot arm joint vectors in each group: , in, For the target category, For in category The following observations The likelihood probability, For category The prior probability.

[0108] In this embodiment, optionally, a deep learning fusion method is used to construct a multi-view fusion network: features are extracted from the images finally acquired by the RGB camera under the pose of the robotic arm chassis and the joint vector of the robotic arm in each group, the features are weighted and fused according to the perceptual quality, and the fused features are input into the classifier to obtain the final recognition result.

[0109] In this embodiment, the ICP algorithm or feature-based registration method is used to unify the multi-view point clouds into the target coordinate system and construct a complete three-dimensional model of the target to be measured.

[0110] The final recognition result is output, including the target category, 3D location, confidence level, and complete point cloud model. If the perception quality still does not meet the requirements, the next observation point planning is triggered until the requirements are met or the maximum number of observations is reached.

[0111] This invention addresses the problems of traditional mobile robotic arm systems in agricultural scenarios where crops are obstructed, making it difficult to actively obtain high-quality observation perspectives, lacking real-time collaborative decision-making between the chassis 1 and the robotic arm 5, and exhibiting unstable control under multiple constraints. It provides a collaborative control method and system for mobile robotic arms dealing with obstructed crops. By establishing a unified kinematic model, multi-sensor joint calibration relationships, robotic arm capability graphs and inverse capability graphs, a perception quality evaluation function, a control mode switching mechanism based on perception quality, an optimal observation point evaluation function, and a hierarchical quadratic programming control framework, dynamic collaboration between the mobile platform and the robotic arm 5 is achieved. This enables the system to actively bypass obstructions, improving target recognition accuracy, 3D positioning accuracy, and overall perception efficiency in complex agricultural scenarios.

[0112] This second embodiment provides a collaborative control system for a mobile robotic arm, including: The imaging region acquisition module is used to make the RGB camera 4 mounted on the end of the robotic arm face the target relative to the end of the robotic arm under the current pose of the robotic arm chassis and the joint variables of the robotic arm, and extract the imaging region of the target based on the image acquired by the RGB camera 4. The region recognition module is used to identify the occluded area in the imaging region of the target under test, and to regard the area outside the occluded area as the unoccluded area. The total area acquisition module is used to obtain the area of ​​the imaging region of the target under test as the total projected area of ​​the target; The occlusion rate calculation module is used to calculate the occlusion rate as the ratio of the area of ​​the occluded region to the total area of ​​the target projection. The target integrity calculation module is used to determine the target integrity by the ratio of the area of ​​the unobstructed region to the reference area of ​​the target under test. The function calculation module is used to calculate the comprehensive sensing quality function value based on the occlusion rate, target integrity, and the sharpness of the image of the imaging area of ​​the target under test; The recognition module is used to determine whether to adjust the pose of the robotic arm chassis or the joint variables of the robotic arm based on the magnitude of the comprehensive perception quality function. If no adjustment is needed, the recognition result of the target under test is obtained based on the image acquired by the current RGB camera; if adjustment is needed, the recognition result of the target under test is obtained based on the image acquired by the adjusted RGB camera.

[0113] This third embodiment provides a mobile robotic arm system employing the above-described cooperative control method, comprising: Chassis 1, specifically the Ackerman mobile chassis, is used to carry robotic arm 5 and various sensors, and provides global mobility; The depth point cloud camera 2 is mounted on the chassis 1 to acquire local 3D point cloud and depth information to assist in target localization and occlusion detection; Multi-line LiDAR 3, mounted on chassis 1, is used for environmental perception, positioning and navigation, and map building; An RGB camera 4, mounted at the end of the robotic arm 5, is used to acquire images and perform target recognition, segmentation, and occlusion analysis. Robotic arm 5, mounted on chassis 1, is used for local fine operations and observation of attitude adjustment; The control computing unit 6 is used to execute the above-mentioned collaborative control method for the mobile robotic arm.

[0114] Based on Embodiment 3, this Embodiment 4 is applied to the automatic identification and positioning of fruits obscured by branches and leaves in an orchard. The mobile robotic arm system includes a mobile chassis 1 with an Ackerman steering structure, two multi-degree-of-freedom robotic arms 5 mounted on the chassis 1, a depth point cloud camera 2 mounted on the chassis 1, a multi-line LiDAR 3 mounted on the chassis 1, two RGB cameras 4 mounted at the ends of the robotic arms 5, and a control and computing unit 6 mounted on an onboard industrial control computer.

[0115] After the mobile robotic arm system is activated, it first collects point cloud data of the orchard environment using a multi-line LiDAR 3, and then uses a simultaneous localization and mapping algorithm to construct a map of the orchard's inter-row passages to determine the current position of the chassis 1 and the distribution of surrounding obstacles. Simultaneously, a depth point cloud camera 2 collects local point cloud data of the plant area in front, acquiring spatial structure information near the target. All sensor data is converted to a unified coordinate system based on pre-completed multi-sensor calibration results.

[0116] Once the mobile robotic arm system detects a suspected fruit target in the local point cloud and image, it controls the robotic arm 5 to turn the end effector RGB camera 4 towards the target fruit to acquire an image from the current viewpoint. A deep learning object detection network identifies the target fruit region, obtaining its bounding box, category information, and confidence score. Combined with the depth image, the target pixel coordinates are converted into a three-dimensional spatial position, determining the target fruit's location in the world coordinate system.

[0117] The mobile robotic arm system analyzes the occlusion situation from the current viewpoint. It identifies occlusion objects such as leaves and branches in the image through semantic segmentation, determines occlusion edges through depth boundary analysis, and uses ray tracing to determine the occlusion relationship between the target fruit and the camera, calculating the occlusion ratio of the target's projected area. Simultaneously, it calculates indicators such as target integrity, image sharpness, and observation distance, comprehensively obtaining the current perceived quality value Q.

[0118] If the current comprehensive perception quality function value reaches a preset threshold, and the target fruit is within the current robotic arm's workspace, the mobile robotic arm system enters an independent perception mode. The robotic arm adjusts the end-effector camera's orientation only within a localized area to position the fruit in the center of the camera's field of view, further improving image clarity and observation completeness. After local optimization, the target fruit and its 3D position are output.

[0119] If the current integrated perception quality function value is lower than a preset threshold, or if the target fruit is not within the robotic arm's workspace, the system enters a vehicle-arm collaborative adjustment mode. The control calculation unit 6 calls the inverse capability map (ICM) based on the target fruit's 3D position to quickly retrieve multiple candidate chassis poses. Then, collision detection, feasibility screening, and comprehensive evaluation are performed on these candidate poses. Evaluation factors include the robotic arm's operability, target reachability margin, geometric observation fit, chassis safety margin, and chassis movement cost under that pose. Finally, the candidate pose with the highest score is selected as the new optimal observation point.

[0120] After determining the optimal observation point, the mobile robotic arm system plans a feasible path that satisfies the steering constraints for the Ackerman chassis, and simultaneously plans a collision-free path for robotic arm 5 from the current configuration to the target observation configuration. During execution, a hierarchical control framework using the HQP control method is employed, prioritizing safety constraints such as robotic arm joint limits, self-collision avoidance, and chassis obstacle avoidance, followed by the completion of the robotic arm end-effector observation task and the chassis path tracking task.

[0121] Once chassis 1 and robotic arm 5 reach the new observation position, the mobile robotic arm system again acquires target images and point clouds, repeating the target recognition, occlusion detection, and perception quality assessment processes. If the perception quality meets the requirements at this point, the current result is fused with the recognition result from the previous observation point to output the final fruit recognition result, spatial location, and confidence level. If the requirements are still not met, the system continues to plan the next observation point until the maximum number of observations is reached.

[0122] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0123] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0124] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0125] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0126] Obviously, the above embodiments are merely illustrative examples for clear explanation and are not intended to limit the implementation. Those skilled in the art will recognize that other variations or modifications can be made based on the above description. It is neither necessary nor possible to exhaustively list all possible implementations here. However, obvious variations or modifications derived therefrom are still within the scope of protection of this invention.

Claims

1. A collaborative control method for a mobile robotic arm, characterized in that, include: Based on the current pose of the robotic arm chassis and the joint vectors of the robotic arm, the RGB camera mounted on the end of the robotic arm is oriented towards the target relative to the end of the robotic arm. Based on the image acquired by the RGB camera, the imaging area of ​​the target is extracted. Identify the occluded area in the imaging region of the target to be tested, and take the area outside the occluded area as the unoccluded area; The area of ​​the imaging region of the target to be measured is taken as the total projected area of ​​the target; The ratio of the area of ​​the occluded region to the total area of ​​the target projection is used as the occlusion rate. The ratio of the area of ​​the unobstructed region to the reference area of ​​the target is used as the target integrity. The comprehensive sensing quality function value is calculated based on the occlusion rate, target integrity, and image sharpness of the target imaging area. Based on the magnitude of the comprehensive perception quality function, it is determined whether to adjust the pose of the robotic arm chassis or the joint vector of the robotic arm. If no adjustment is needed, the recognition result of the target to be tested is obtained based on the image acquired by the current RGB camera. If adjustments are needed, the recognition result of the target under test is obtained based on the image captured by the adjusted RGB camera.

2. The collaborative control method for a mobile robotic arm according to claim 1, characterized in that, Methods for obtaining the sharpness of the image of the target imaging area include: The image of the target imaging region is processed by the Laplacian operator, and the pixel variance of the processed image is calculated as the image sharpness.

3. The collaborative control method for a mobile robotic arm according to claim 1, characterized in that, Methods for calculating the comprehensive sensing quality function value based on occlusion rate, target integrity, and image sharpness of the target imaging region include: The sharpness of the image of the target imaging area is normalized. Based on the occlusion rate, target integrity, and the sharpness of the normalized image of the target region, the comprehensive sensing quality function value is calculated using the following formula: ， in, To comprehensively perceive the quality function, As the first weighting coefficient, This is the second weighting coefficient. This is the third weighting coefficient. It is the fourth weighting coefficient. This is the fifth weighting coefficient. , For the completeness of the target, To determine the sharpness of the normalized image, For occlusion rate, This represents an exponential function with the natural constant as its base. The distance between the center coordinates of the RGB camera mounted on the end effector of the robotic arm and the center of the target being measured. For reference observation distance, This is the distance attenuation coefficient. The angle between the optical axis of the RGB camera and the normal of the surface of the target object is denoted as .

4. The collaborative control method for a mobile robotic arm according to claim 1, characterized in that, Methods for determining whether to adjust the robot arm chassis pose or robot arm joint vectors based on the magnitude of the comprehensive sensing quality function include: If the overall sensing quality function is greater than or equal to the set sensing quality threshold, and the target to be measured is located within the workspace of the robotic arm, then only the joint vector of the robotic arm is adjusted. If the overall perceived quality function is less than the set perceived quality threshold, or if the target to be measured exceeds the working space of the robotic arm, then the pose of the robotic arm chassis and the joint vectors of the robotic arm will be adjusted.

5. The collaborative control method for a mobile robotic arm according to claim 1, characterized in that, The method for adjusting the pose of the robotic arm chassis and the joint vectors of the robotic arm includes: Random sampling is performed within the joint space of the robotic arm to construct multiple sets of robotic arm joint vectors; Based on the Jacobian matrix of the mobile robotic arm system under each set of robotic arm joint vectors, the normalized operability corresponding to each set of robotic arm joint vectors is calculated sequentially. Based on the normalized operability corresponding to each group of robotic arm joint vectors, the joint vectors of each group of robotic arms are filtered to obtain the target robotic arm joint vector. Construct an inverse capability graph using the inverse pose of the manipulator base relative to the end effector and the normalized operability corresponding to all target manipulator joint vectors; Based on the inverse pose of the manipulator base relative to the end effector that matches the pose of the target under test in the world coordinate system in the inverse capability diagram, a set of candidate chassis poses is obtained; each candidate chassis pose in the set of candidate chassis poses corresponds to one or more target manipulator joint vectors; Each candidate chassis pose and its corresponding target robot joint vector are used as a candidate combination; The optimal candidate combination is determined based on the normalized operability, target reachability margin, geometric observation fit, normalized chassis cost, and chassis safety margin corresponding to each candidate combination. With the goal of achieving the optimal candidate combination, the pose of the robotic arm chassis and the joint vectors of the robotic arm are adjusted.

6. The collaborative control method for a mobile robotic arm according to claim 5, characterized in that, The method for filtering the joint vectors of each group of robotic arms based on the normalized operability corresponding to each group of robotic arm joint vectors to obtain the target robotic arm joint vector includes: Perform self-collision detection on each group of robotic arm joint vectors and remove robotic arm joint vectors that have self-collision. For the remaining robotic arm joint vectors after filtering at each location, sort them in descending order according to normalized operability, and select the previously set number of robotic arm joint vectors as the target robotic arm joint vectors.

7. The collaborative control method for a mobile robotic arm according to claim 5, characterized in that, The method for determining the optimal candidate combination based on the normalized operability, target reachability margin, geometric observation fit, normalized chassis cost, and chassis safety margin corresponding to each candidate combination includes: Based on the normalized operability, target reachability margin, geometric observation fit, normalized chassis cost, and chassis safety margin corresponding to each candidate combination, the comprehensive score of each candidate combination is calculated using the following formula: ， in, For the first The overall score of each candidate combination For candidate combination index, It is the sixth weighting coefficient. It is the seventh weighting coefficient. This is the eighth weighting coefficient. It is the ninth weighting coefficient. This is the tenth weighting coefficient. , For the first The normalized operability corresponding to each candidate combination For the first The target achievable margin corresponding to each candidate combination For the first Geometric observation fit of each candidate combination For the first The normalized chassis cost corresponding to each candidate combination For the first Chassis safety margin corresponding to each candidate combination; The candidate combination with the highest overall score is selected as the optimal candidate combination.

8. The collaborative control method for a mobile robotic arm according to claim 5, characterized in that, The formula for calculating the target reachability margin for each candidate combination is: ， in, For the first The target achievable margin corresponding to each candidate combination For the first Under each candidate combination, the distance from the robotic arm base to the center of the target being measured is... To set the ideal observation distance for the robotic arm, , These represent the minimum and maximum observable distances of the robotic arm, respectively. To prevent extremely small positive numbers with a denominator of zero, For candidate combination index; The formula for calculating the geometric observation fit for each candidate combination is: ， in, For the first Geometric observation fit of each candidate combination For the first Under the candidate combinations, the angle between the optical axis direction of the RGB camera and the direction from the center of the RGB camera to the center of the target being measured. The maximum allowable angle; The process of normalizing the chassis cost for each candidate combination includes: After normalizing the chassis cost for each candidate combination, the normalized chassis cost for each candidate combination is obtained; the formula for calculating the chassis cost for each candidate combination is as follows: ， in, For the first The chassis cost corresponding to each candidate combination. Let x be the current x-coordinate of the robotic arm chassis. Let y be the current y-coordinate of the robotic arm chassis. For the first The chassis x-coordinates corresponding to each candidate combination For the first The chassis y-coordinates corresponding to each candidate combination; The formula for calculating the chassis safety margin for each candidate combination is as follows: ， in, For the first Chassis safety margin corresponding to each candidate combination This indicates taking the minimum value. For the first The minimum distance between the candidate chassis pose and its nearest obstacle in each candidate combination. To establish a safe distance.

9. The collaborative control method for a mobile robotic arm according to claim 1, characterized in that, Obtain the recognition results of the target under the pose of the robotic arm chassis and the joint vectors of the robotic arm for each group; Based on the recognition results of the target under the pose of the robotic arm chassis and the joint vectors of the robotic arm in each group, the final recognition result of the target is obtained.

10. A collaborative control system for a mobile robotic arm, characterized in that, include: The imaging region acquisition module is used to make the RGB camera mounted on the end of the robotic arm face the target relative to the end of the robotic arm under the current pose of the robotic arm chassis and the joint vector of the robotic arm, and extract the imaging region of the target based on the image acquired by the RGB camera. The region recognition module is used to identify the occluded area in the imaging region of the target under test, and to regard the area outside the occluded area as the unoccluded area. The total area acquisition module is used to obtain the area of ​​the imaging region of the target under test as the total projected area of ​​the target; The occlusion rate calculation module is used to calculate the occlusion rate as the ratio of the area of ​​the occluded region to the total area of ​​the target projection. The target integrity calculation module is used to determine the target integrity by the ratio of the area of ​​the unobstructed region to the reference area of ​​the target under test. The function calculation module is used to calculate the comprehensive sensing quality function value based on the occlusion rate, target integrity, and the sharpness of the image of the imaging area of ​​the target under test; The recognition module is used to determine whether to adjust the pose of the robotic arm chassis or the joint vector of the robotic arm based on the magnitude of the comprehensive perception quality function. If no adjustment is needed, the recognition result of the target to be tested is obtained based on the image captured by the current RGB camera. If adjustments are needed, the recognition result of the target under test is obtained based on the image captured by the adjusted RGB camera.

Images