A VLA-based collaborative planning method and device for mobile robotic arms
By using a VLA-based operability map planning method, the problems of insufficient extension of the robotic arm, obstructed vision, or increased collision risk caused by poor docking position of mobile robots were solved. This method enabled collaborative planning between the robotic arm and the chassis, improving operational efficiency and success rate.
Patent Information
- Application Number
- CN202611299551.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-08-26
- Publication Date
- 2026-11-17
AI Technical Summary
In existing technologies, when mobile robots perform tasks such as grasping and manipulating target objects, poor docking positions often result in insufficient extension of the robotic arm, obstructed vision, or increased risk of collisions, affecting work efficiency and success rate.
By constructing an operability map based on the VLA model and combining accessibility, visibility, grasping stability and collision risk indicators, the optimal docking position and navigation path are planned, realizing the collaborative planning of the robotic arm and chassis, reducing collision risk and improving operation efficiency.
This enabled the robotic arm to be positioned in one go and start working immediately, reducing the risk of collisions, improving work efficiency and success rate, and avoiding the problem of repeated posture adjustments caused by the separation of navigation and operation.
Smart Images

Figure FT_1 
Figure FT_2 
Figure FT_3
Abstract
Description
Technical Field
[0001] This invention relates to the field of artificial intelligence technology, and in particular to a collaborative planning method and apparatus for a mobile robotic arm based on VLA. Background Technology
[0002] Currently, mobile robots are increasingly used to perform tasks such as grasping, moving, and placing target objects. Existing technologies typically involve first planning the path based on the target object's location, and then activating the robotic arm to grasp or manipulate the object once the robot reaches the target area. However, this method cannot predict whether the chassis's docking position will meet the robotic arm's operational needs. This leads to problems in actual operations, such as insufficient arm extension, obstructed vision, or increased collision risk due to poor docking positions. Repeated adjustments to the robot's posture are necessary to complete the task, severely impacting operational efficiency and success rate.
[0003] Therefore, there is an urgent need to provide a new technical solution to address the above problems. Summary of the Invention
[0004] This invention provides a VLA-based collaborative planning method and device for mobile robotic arms, which can reduce collision risks and improve operational efficiency and success rate.
[0005] In a first aspect, embodiments of the present invention provide a VLA-based collaborative planning method for a mobile robotic arm, comprising: Based on the operation task of the target object, the availability information for the robotic arm to perform the operation task is obtained using the VLA model. The availability information includes the optimal operation area, the optimal approach direction of the robotic arm end effector, and the spatial constraints during the operation. Based on the three-dimensional spatial information of the target object and the availability information, an operability map is constructed. The basic operability indicators for each location in the operability map include accessibility indicators, visibility indicators, grasping stability indicators, and collision risk indicators. Based on the aforementioned basic operational indicators and spatial constraints, several candidate docking positions are determined around the target object. Each candidate docking position includes planar position information and chassis orientation information of the robotic arm. The comprehensive operational score of each candidate docking station is calculated based on its basic operational metrics, its matching degree with the optimal operational area and the optimal approach direction, and the navigation path cost of moving from the current position to the candidate docking station. Based on the comprehensive operation score of each candidate docking position and the navigation path cost, a comprehensive cost function is constructed. The optimal docking position and optimal navigation path that make the comprehensive cost function optimal are determined. The chassis of the robotic arm is controlled to move along the optimal navigation path to the optimal docking position, and then the robotic arm performs the operation task.
[0006] Secondly, embodiments of the present invention also provide a VLA-based mobile robotic arm collaborative planning device, comprising: The available information acquisition module, based on the operation task of the target object, uses the VLA model to obtain the availability information for the robotic arm to perform the operation task. The availability information includes the optimal operation area, the optimal approach direction of the robotic arm end effector, and the spatial constraints during the operation. The map building module, connected to the available information acquisition module, constructs an operable map based on the three-dimensional spatial information of the target object and the available information. The basic operational indicators for each location in the operable map include accessibility indicators, visibility indicators, grasping stability indicators, and collision risk indicators. The candidate location determination module is connected to the map construction module. Based on the basic operation indicators and spatial constraints, it determines several candidate docking positions around the target object. Each candidate docking position includes planar position information and chassis orientation information of the robotic arm. The comprehensive scoring determination module, connected to the candidate location determination module, calculates the comprehensive operational score of each candidate docking station based on its basic operational indicators, its matching degree with the optimal operational area and the optimal approach direction, and the navigation path cost of moving from the current location to the candidate docking station. The optimal path planning module, connected to the comprehensive scoring determination module, constructs a comprehensive cost function based on the comprehensive operation score of each candidate docking position and the navigation path cost. It then determines the optimal docking position and optimal navigation path that optimizes the comprehensive cost function. This allows the robot arm's chassis to move along the optimal navigation path to the optimal docking position, after which the robot arm performs the operation task.
[0007] Thirdly, embodiments of the present invention also provide an electronic device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, it implements the method described in any embodiment of the present invention.
[0008] Fourthly, embodiments of the present invention also provide a computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to perform the methods described in any embodiment of the present invention.
[0009] This invention provides a VLA-based collaborative planning method and apparatus for a mobile robotic arm. It utilizes a VLA model to parse operational task information into availability information that constrains the robotic arm's execution of tasks. Based on this availability information, an operability map is constructed. Multiple basic operational indicators are used to comprehensively evaluate each location on the map, providing a global quantitative basis for docking position selection. Spatial constraints enable the navigation planning process to predict whether the chassis docking position meets the robotic arm's operational requirements, preventing problems such as insufficient robotic arm extension, obstructed vision, or increased collision risk due to poor docking positions. By constructing a comprehensive cost function that includes navigation path cost, comprehensive operational score, and spatial constraint cost, the optimal docking position and optimal navigation path are solved simultaneously, achieving collaborative planning of docking position and path selection. This avoids the repeated pose adjustments caused by the separation of navigation and operation in traditional serial architectures, allowing the robot to dock in place immediately and begin operation, reducing collision risk and improving operational efficiency and success rate. Attached Figure Description
[0010] To more clearly illustrate the technical solutions in the embodiments of the present invention or related technologies, the drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0011] Figure 1 This is a flowchart of the VLA-based collaborative planning method for mobile robotic arms provided in an embodiment of the present invention; Figure 2 This is a structural diagram of the VLA-based mobile robotic arm collaborative planning device provided in an embodiment of the present invention; Figure 3 This is a block diagram of the collaborative planning process of a mobile robotic arm based on VLA provided in an embodiment of the present invention; Figure 4 This is a block diagram of the operable map construction process provided in the embodiments of the present invention; Figure 5 This is a schematic diagram of semantic parsing and availability prediction of the VLA model task provided in an embodiment of the present invention; Figure 6 This is a diagram of the navigation and robotic arm collaborative optimization framework provided in an embodiment of the present invention. Detailed Implementation
[0012] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are some embodiments of the present invention, but not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.
[0013] Please refer to Figure 1 This invention provides a VLA-based collaborative planning method for mobile robotic arms, comprising: Step 100: Based on the operation task of the target object, obtain the availability information of the robotic arm to perform the operation task using the VLA model; This availability information includes the optimal operating area, the optimal approach direction of the robotic arm end effector, and the spatial constraints during operation. Step 102: Construct an operability map based on the three-dimensional spatial information and availability information of the target object; The basic operational metrics for each location in this operability map include accessibility metrics, visibility metrics, capture stability metrics, and collision risk metrics. Step 104: Based on the basic operational indicators and spatial constraints, determine several candidate docking positions around the target object; Each candidate docking position includes planar position information and chassis orientation information of the robotic arm; Step 106: Calculate the comprehensive operational score of each candidate docking station based on its basic operational metrics, its matching degree with the best operational area and the best approach direction, and the navigation path cost of moving from the current location to the candidate docking station. Step 108: Construct a comprehensive cost function based on the comprehensive operation score and navigation path cost of each candidate docking position, determine the optimal docking position and optimal navigation path that make the comprehensive cost function optimal, so that the chassis of the robotic arm moves along the optimal navigation path to the optimal docking position, and the robotic arm performs the operation task.
[0014] This embodiment provides a VLA-based collaborative planning method for mobile robotic arms. By integrating the semantic information of the operation task into navigation decisions, the operational requirements of the robotic arm are incorporated as constraints into the path search process during the navigation planning stage, achieving collaborative optimization of the mobile chassis and robotic arm operations. First, based on the operation task of the target object, the availability information for completing the target task is inferred through a vision-language-action model. The availability information includes the optimal operating area, the optimal approach direction of the robotic arm's end effector, and the spatial constraints during the operation.
[0015] like Figure 3 and Figure 6 As shown, an operability map is constructed based on the 3D spatial information and availability information of the target object. The operability map is built upon a traditional geometric map, overlaying a feasibility assessment layer oriented towards the operational task. Each location in the operability map is assigned basic operational indicators, including reachability, visibility, grasping stability, and collision risk. These four indicators together constitute the physical benchmark for evaluating whether any location in the environment is suitable for operation, extending discrete object attributes into a continuous spatial operational potential field, providing a global quantitative basis for docking site selection.
[0016] Within the continuous space represented by the operability map, infeasible areas (such as restricted areas) are first eliminated based on spatial constraints. Then, within the remaining areas, a set of candidate docking positions is generated by sampling based on the distribution characteristics of basic operational indicators. Each candidate docking position contains two-dimensional planar coordinates and a chassis orientation angle. The chassis orientation determines the initial posture of the robotic arm base and affects the workspace coverage.
[0017] Based on the fundamental operational metrics of each candidate docking station, its matching degree with the optimal operational area and optimal approach direction, and the navigation path cost of moving from the current location to the candidate docking station, a comprehensive operational score is calculated for that candidate docking station. The comprehensive operational score is a multi-dimensional quantitative evaluation of a single candidate docking station, considering both the fundamental physical metrics provided by the map and the degree of matching with availability information; the smaller the deviation between the actual docking posture and the ideal posture, the higher the score. Simultaneously, navigation path cost is taken into account, reflecting the trade-off between arrival cost and operational quality.
[0018] A comprehensive cost function is constructed based on the overall operational score, navigation path cost, and spatial constraints of each candidate docking station. The optimization objective is to minimize this comprehensive cost function. Combinations of paths and destinations are explored simultaneously, attracting areas with high overall operational scores during the path search process and automatically avoiding geometrically reachable but operationally difficult suboptimal areas. Ultimately, the optimal docking station and optimal navigation path are output. By jointly solving navigation and operation within the same optimization framework, the chassis can be positioned in one go and immediately put into operation, fundamentally avoiding the problem of repeated pose adjustments caused by the separation of navigation and operation.
[0019] In one embodiment of the present invention, based on the operation task of the target object, the availability information for the robotic arm to perform the operation task is obtained using a VLA model, including: The natural language instructions for the operation task and the current scene image are input into the trained VLA model to obtain the optimal operation area, the optimal approach direction of the robotic arm end effector, and the spatial constraints during the operation process. Among them, the optimal operating area is the area on the surface of the target object that can be grasped or manipulated, the optimal approach direction is the angular range of the approach direction of the robotic arm end relative to the target object, the angular range includes the yaw angle component and the pitch angle component, and the spatial constraints include the prohibited areas that the robotic arm and chassis need to avoid during the operation and the operation sequence constraints corresponding to each sub-task in the multi-step operation task.
[0020] In this embodiment, the robot first acquires RGB images, depth images, and point cloud data of the current environment using an RGBD camera and LiDAR. It then runs a SLAM algorithm (such as ORB-SLAM3 or RTAB-Map) to construct a geometric map of the environment. Simultaneously, it utilizes an open-vocabulary object detection model (such as Grounding DINO or OV-DETR) and a semantic segmentation model to identify object categories, bounding boxes, and point cloud masks in the scene. The detection results are then associated with their corresponding 3D locations on the map, forming a 3D semantic map containing object category, size, center position, and orientation. If a prior map already exists, only the changed areas are incrementally updated. This 3D semantic map provides the foundation for determining the spatial location of target objects, enabling the VLA model to associate target nouns in natural language commands with specific objects in the scene.
[0021] Building upon this foundation, the VLA (Vision-Language-Action) model acts as a bridge between high-level semantic understanding and low-level spatial planning. Specifically, VLA models (such as RT-2 or PaLM-E) receive multimodal data as input: RGB or depth images of the current scene, and natural language commands from the user (e.g., "grab the glass on the workbench to apply adhesive"). Internally, the model uses a cross-modal alignment mechanism to correlate verbs (e.g., "grab") and nouns (e.g., "apply adhesive to the glass") with pixel-level features in the image, yielding the output availability information.
[0022] Define the task semantic representation of VLA output as follows: Where g is the target grasping region; d is the optimal approach direction of the end effector; c is the task constraint region; and o is the operation sequence constraint. This is achieved through a semantic-spatial mapping function: M represents a three-dimensional semantic map. For semantic space mapping functions, generate spatial constraints. The parameters in the spatial constraint domain correspond to the capture feasible domain, the approach direction constraint domain, the forbidden region, and the operation sequence constraint domain, respectively.
[0023] Specifically, the optimal operating area is the region on the surface of the target object that can be grasped or manipulated. This area is quantified as a geometric representation in three-dimensional space. For example, for the target of "glass adhesive application," the optimal operating area might be identified as the set of curved point clouds containing the adhesive applicator handle, or a three-dimensional bounding box enclosing the handle. The optimal approach direction is the angular range of the robotic arm's end effector relative to the target object, including yaw and pitch components, and is a conical space or angular range allowing a certain degree of freedom. For example, for the adhesive applicator handle, the yaw angle might be acceptable within ±30°, while the pitch angle is limited to ±10° in the horizontal direction. Spatial constraints include prohibited areas that the robotic arm and chassis must avoid during operation, as well as the operation sequence constraints corresponding to each subtask in a multi-step operation task. Prohibited areas correspond to fragile items, high-temperature components, or areas of human activity, and are marked as high-value or impassable areas on the map. Operation sequence constraints describe the temporal logic of task execution, such as: opening the cabinet door before grasping the internal items.
[0024] In one embodiment of the present invention, the reachability index is used to characterize whether each position is within the kinematic reachable space of the robotic arm; The visibility metric is used to characterize whether the line of sight between each location and the optimal operating area is obstructed by obstacles; Grasping stability metrics are used to characterize the expected success rate of a robotic arm performing a grasping or manipulation at each position. The collision risk index is used to characterize the probability of collision between the robotic arm and chassis and environmental obstacles when operating at each position.
[0025] In this embodiment, the reachability index is used to characterize whether each position is within the kinematic reachable space of the robotic arm. In specific calculations, a quick query can be performed using a pre-generated point cloud of the robotic arm's workspace, or the Inverse Kinematics (IK) solver can be invoked in real-time for candidate docking positions. If at least one set of joint angle solutions allows the end effector to reach the optimal operating area without exceeding joint limits, the reachability index for that position is assigned a value of 1 or a higher score; otherwise, it is 0, i.e., the value range is [0, 1].
[0026] The visibility metric characterizes whether the line of sight between each position and the optimal operating area is obstructed by obstacles. It is typically calculated using a ray casting algorithm, which involves casting a virtual ray from the camera or end effector position corresponding to the candidate docking position towards the center or key point of the optimal operating area, and counting the number of environmental obstacles the ray passes through or the depth of penetration. If there is no obstruction, the visibility score is full; if there is partial obstruction, the score decreases linearly according to the obstruction ratio, with a value ranging from [0, 1]. This metric directly determines whether the robot can "see" the object being operated on and is a prerequisite for successful visual servoing.
[0027] The gripping stability metric characterizes the expected success rate of a robotic arm performing gripping or manipulation at each position. This metric goes beyond simple geometric contact judgment, incorporating mechanical analysis and empirical knowledge. For example, it can calculate the ability to resist external force disturbances in the current approach direction based on force closure theory, or use a pre-trained gripping quality assessment network to predict the probability of successful gripping based on the object's material, shape, and contact surface friction coefficient. For the optimal operating area recommended by the VLA model, since it inherently contains a semantic prior of "easy to grip," additional confidence weighting can be applied when calculating this metric, with a value ranging from [0, 1].
[0028] The collision risk index is used to characterize the probability of collision between the robotic arm and chassis and environmental obstacles when operating at each position. To balance computational efficiency and safety, signed distance field (SDF) or voxelized dilatation detection methods are typically used. The minimum Euclidean distance from the robotic arm links and chassis shell to the nearest environmental obstacle is usually calculated. When this distance is less than a safety threshold, the collision risk index approaches 1. The collision risk index ranges from [0, 1].
[0029] By quantifying the above four dimensions, the originally vague concept of operability is transformed into a precise numerical matrix, providing a solid data foundation for subsequent comprehensive scoring and joint optimization, and ensuring that the planning results not only conform to human task intentions but also meet the physical operation limits of robots.
[0030] In one embodiment of the present invention, based on basic operational indicators and spatial constraints, several candidate docking positions around the target object are determined, including: Based on basic operational metrics and the historical operational success rate of each location, an operability score is generated. Locations within the area surrounding the target object that meet spatial constraints and have an operability score greater than the score threshold are identified as candidate docking spots, and a set of candidate docking spots is generated. The candidate docking station set is as follows: Each parking space is , These are the position coordinates of the location in a two-dimensional plane coordinate system, used to characterize planar position information. The chassis orientation angle is used to characterize chassis orientation information.
[0031] In this embodiment, several candidate parking positions around the target object are determined based on basic operational indicators and spatial constraints. Each candidate parking position includes planar position information and chassis orientation information. Specifically, positions within the area surrounding the target object that have an operability score greater than the score threshold and do not fall into restricted areas are identified as candidate parking positions, generating a set of candidate parking positions.
[0032] Specifically, to improve search efficiency and ensure solution quality, this embodiment employs a non-uniform sampling strategy when generating the candidate docking station set. High-density sampling is used in regions where the operability score is above a preset threshold to finely search for potential optimal solutions. In regions with moderate or low scores, low-density sampling or only retaining local extrema is used to avoid wasting computational resources in obviously suboptimal regions.
[0033] Spatial constraints act as hard filters, directly eliminating positions falling into restricted areas and excluding them from subsequent scoring calculations and path planning. This dual mechanism of soft scoring guidance and hard constraint filtering ensures both the physical legitimacy of the candidate set and a sufficient number of high-quality candidate solutions, laying the foundation for subsequent optimization of the comprehensive cost function.
[0034] In one embodiment of the present invention, the operability score is represented by the following formula: For operability scoring, , As an accessibility indicator, As a visibility indicator, To capture stability metrics, As a collision risk indicator, , , and These are the weighting coefficients. This represents the historical success rate. This represents the historical experience weighting coefficient.
[0035] In this embodiment, as Figure 4 As shown, the operability score is obtained by weighting the basic operational indicators and adding historical experience bias terms, i.e. . For historical bias, This represents the historical success rate. This represents the historical experience weighting coefficient. This is not static prior knowledge, but rather dynamic memory acquired by the system (i.e., the mobile robotic arm planning system implementing this method) through incremental learning during long-term operation. When the robot successfully completes the grasping task in a certain pose, the system stores this pose and its neighborhood... The value is adjusted upwards by a preset step size; conversely, if a failure or collision occurs, it is adjusted downwards accordingly. This mechanism gives the operability map an adaptive capability that "gets smarter with use," handling uncertainties such as changes in lighting, slight object displacement, or sensor noise. It can serve as a robust prior bias, guiding the planner to prioritize areas that have been proven safe and effective in the past, reducing over-reliance on new environmental perceptions and improving the overall stability of the system.
[0036] The weighting coefficients in this embodiment , , and The system can dynamically adjust its weighting based on the type of current task. For example, when performing precision assembly or fragile item handling tasks, the system automatically increases the weight of the handling stability index and the penalty weight of the collision risk index. Conversely, when performing large-scale inspections or rough handling tasks, the weights of the accessibility and visibility indices are appropriately increased. This task-oriented dynamic weighting strategy allows the same operability map to flexibly adapt to diverse operational needs without requiring a new map for each task.
[0037] It should be noted that the historical bias can be maintained as an additional historical experience layer in the operability map, used to store the success rate and frequency of operations at or near that location in previous successful tasks. The final operability map includes: location coordinates, target object ID, operability score, historical success rate, and last update time.
[0038] In one embodiment of the present invention, the comprehensive operational score is represented by the following formula: For comprehensive operational scoring, For navigation path cost, The accessibility score is obtained by weighting accessibility indicators. This is a crawling quality score weighted by crawling stability metrics. The collision risk score is obtained by weighting collision risk indicators. A semantic preference score is used to characterize the degree of matching between the current position and the optimal operating region and the optimal proximity direction. , , , and These are the weighting coefficients.
[0039] In this embodiment, a comprehensive operational score is calculated for each candidate parking space in the candidate parking space set, i.e. .in, The navigation path cost is the cost of moving the robot from its current position to the candidate docking position. This cost is calculated using a global path search algorithm (such as A). The algorithm is pre-computed on the occupied grid map. The navigation path cost is measured by path length or estimated travel time. The accessibility score is a weighted score based on accessibility indicators, which reflects whether the robotic arm base can physically reach the target area when it is in the docking position. The score is a weighted score based on the crawling stability index. This score is closely related to the optimal operating area recommended by the VLA model. The closer the actual crawling point is to the optimal operating area, the higher the score. The collision risk score is a weighted score based on collision risk indicators, used to quantify the probability of collision between the robotic arm and chassis and environmental obstacles when operating at the candidate parking position. The semantic preference score is used to characterize the degree of matching between the current position and the optimal operating area and the optimal approach direction. The calculation of the semantic preference score includes the following steps: calculate the actual approach direction of the robotic arm end effector relative to the target object based on the planar position of the candidate docking position and the chassis orientation, and compare the actual approach direction with the optimal approach direction output by the VLA model. The smaller the angle deviation, the higher the semantic preference score.
[0040] , , , and Using the weighted coefficients, the above five factors are weighted and summed to unify arrival cost, physical feasibility, operational stability, safety, and semantic matching within a quantitative framework. This allows for a balance between arrival cost and operational quality, preventing a decrease in overall task efficiency due to selecting a docking station that is locally optimal but too far away. It ensures basic kinematic feasibility, guarantees the probability of operational success, safeguards operational safety, and makes the planning more aligned with human operational intentions. This multi-dimensional evaluation mechanism ensures that candidate docking stations achieve a reasonable balance between local operational quality and global task efficiency.
[0041] In one embodiment of the present invention, the comprehensive cost function is expressed by the following formula: For the comprehensive cost function, For navigation path cost, The overall operational score for candidate parking positions. As a result of spatial constraints, , and These are the weighting coefficients; The cost of spatial constraints is expressed by the following formula: This represents the angular deviation between the actual approach direction and the optimal approach direction corresponding to the candidate parking position. The degree of intrusion of the candidate parking space into the restricted area. This is an indication value for a violation of the operation sequence constraint by the candidate docking position. , and These are the weighting coefficients.
[0042] In this embodiment, the minimum value of the comprehensive cost function corresponds to the global optimal solution. Wherein, The cost of the navigation path is used as a cost item for global path planning. The overall operational score for candidate docking positions serves as a quality reward for the terminal status (the higher the score, the lower the overall cost). The spatial constraint cost is used to transform the hard semantic constraints output by the VLA model into a penalty term. , and These are the weighting coefficients.
[0043] Specifically, the spatial constraint cost is expressed by the following formula: This represents the angular deviation between the actual approach direction and the optimal approach direction corresponding to the candidate parking position. When this deviation exceeds the allowable range given by the VLA model, The value of will increase, which will lead to an increase in the value of the space constraint cost. The degree of intrusion of the candidate docking position into the restricted area can be calculated by taking the minimum penetration depth between the robotic arm or chassis envelope and the geometric boundary of the restricted area. If no intrusion occurs, the value is zero. This is an indicator value for a candidate docking position violating the operation sequence constraint. For example, in a "open the door first, then retrieve the item" task, if the current candidate docking position is in the "retrieve item" stage but the door has not yet been opened, then... It is assigned a large penalty constant, and takes the value of zero if the order is valid. , and These are the weighting coefficients.
[0044] By constructing the aforementioned comprehensive cost function, deep collaborative optimization of navigation and operation is achieved. For example, in a certain grasping task, there exists a candidate docking point A. A has an extremely high reachability index and the shortest navigation path, but this location is located on the edge of the prohibited area marked by the VLA model, and the approach direction deviates from the recommended optimal approach direction by 60 degrees. In traditional methods, this location might be selected due to geometric feasibility, leading to collisions or grasping failures during actual operation. However, in the joint optimization model of this application, although... Smaller and Higher, but due to and The existence of spatial constraints comes at a cost. This will increase significantly, thus affecting the overall cost function. The optimizer automatically discards this "pseudo-optimal solution" because it ranks higher than other candidate positions, and instead selects another docking position B that is slightly further away but fully meets the semantic constraints and safety requirements. This mechanism ensures that the planning result is not only mathematically optimal, but also truly executable and safe in the physical world and in the task logic, solving the technical problem of the separation between geometric planning and semantic understanding in existing technologies.
[0045] Furthermore, the optimal parking position obtained from the planning is used as the navigation target point, and the mobile chassis is controlled to move to the parking position along the optimal navigation path. Upon arrival, based on the joint trajectory solved by the inverse kinematics of the robotic arm, the robotic arm is controlled to perform tasks such as grasping, assembly, or inspection. During execution, the relative pose information between the end effector and the environmental target is acquired in real time through a hand-eye camera to monitor whether there is a deviation between the actual state and the planned expectation. If the operation is hindered due to factors such as environmental changes, positioning errors, or slight displacement of objects, an online local replanning mechanism is triggered. The system regenerates candidate parking positions in a local area near the current parking position and quickly updates the comprehensive operation score and local path, enabling the robot to adaptively adjust its operation strategy without returning to the global planning, thereby ensuring the robustness and success rate of task execution. This process integrates planning and execution into a closed loop, enabling the robot to have dynamic response capabilities when facing uncertainties in the real environment.
[0046] According to another embodiment, the present invention provides a VLA-based mobile robotic arm collaborative planning device. Figure 2 A schematic block diagram of a collaborative planning device for a mobile robotic arm based on VLA is shown. It is understood that this device can be implemented using any device, equipment, platform, or cluster of devices with computing and processing capabilities. Figure 2 As shown, the device includes: an information acquisition module 200, a map construction module 202, a candidate location determination module 204, a comprehensive score determination module 206, and an optimal route planning module 208. The main functions of each component are as follows: The available information acquisition module 200 obtains availability information for the robotic arm to perform the operation task based on the target object using a VLA model. The availability information includes the optimal operation area, the optimal approach direction of the robotic arm end effector, and the spatial constraints during the operation. The map construction module 202 is connected to the available information acquisition module. Based on the three-dimensional spatial information of the target object and the availability information, it constructs an operable map. The basic operational indicators for each location in the operable map include accessibility indicators, visibility indicators, grasping stability indicators, and collision risk indicators. The candidate location determination module 204 is connected to the map construction module. Based on the basic operation indicators and spatial constraints, it determines several candidate docking positions around the target object. Each candidate docking position includes planar position information and chassis orientation information of the robotic arm. The comprehensive scoring determination module 206 is connected to the candidate position determination module. Based on the basic operation indicators of each candidate docking station, the matching degree with the best operation area and the best approach direction, and the navigation path cost of moving from the current position to the candidate docking station, it calculates the comprehensive operation score of the candidate docking station. The optimal path planning module 208 is connected to the comprehensive scoring determination module. Based on the comprehensive operation score of each candidate docking position and the navigation path cost, a comprehensive cost function is constructed. The optimal docking position and the optimal navigation path that make the comprehensive cost function optimal are determined so that the chassis of the robotic arm moves along the optimal navigation path to the optimal docking position and then the robotic arm performs the operation task.
[0047] In this embodiment of the specification, the information acquisition module 200 can be used to execute step 100 in the above method embodiment, the map construction module 202 can be used to execute step 102 in the above method embodiment, the candidate location determination module 204 can be used to execute step 104 in the above method embodiment, the comprehensive score determination module 206 can be used to execute step 106 in the above method embodiment, and the optimal path planning module 208 can be used to execute step 108 in the above method embodiment.
[0048] In one embodiment of the present invention, the available information acquisition module is configured to perform the following operations: The natural language instructions for the operation task and the current scene image are input into the trained VLA model to obtain the optimal operation area, the optimal approach direction of the robotic arm end effector, and the spatial constraints during the operation process. The optimal operating area is the area on the surface of the target object that can be grasped or manipulated. The optimal approach direction is the angular range of the approach direction of the robotic arm end relative to the target object. The angular range includes yaw angle components and pitch angle components. The spatial constraints include prohibited areas that the robotic arm and chassis must avoid during operation, as well as the operation sequence constraints corresponding to each sub-task in the multi-step operation task.
[0049] In one embodiment of the present invention, the reachability index is used to characterize whether each position is within the kinematic reachable space of the robotic arm; The visibility index is used to characterize whether the line of sight between each location and the optimal operating area is obstructed by obstacles; The grasping stability index is used to characterize the expected success rate of the robotic arm when performing grasping or operation at each position; The collision risk index is used to characterize the probability of collision between the robotic arm and chassis and environmental obstacles when operating at each position.
[0050] In one embodiment of the present invention, the candidate position determination module is configured to perform the following operations: Based on the aforementioned basic operational metrics and the historical operational success rate of each location, an operability score is generated. The locations within the area surrounding the target object that meet the spatial constraints and have an operability score greater than the score threshold are identified as candidate docking positions, and a set of candidate docking positions is generated. The candidate docking station set is as follows: Each parking space is , These are the position coordinates of the location in a two-dimensional plane coordinate system, used to characterize planar position information. The chassis orientation angle is used to characterize chassis orientation information.
[0051] In one embodiment of the present invention, the operability score is represented by the following formula: For operability scoring, , As an accessibility indicator, As a visibility indicator, To capture stability metrics, As a collision risk indicator, , , and These are the weighting coefficients. This represents the historical success rate. This represents the historical experience weighting coefficient.
[0052] In one embodiment of the present invention, the comprehensive operational score is represented by the following formula: For comprehensive operational scoring, For navigation path cost, The accessibility score is obtained by weighting accessibility indicators. This is a crawling quality score weighted by crawling stability metrics. The collision risk score is obtained by weighting collision risk indicators. A semantic preference score is assigned to characterize the degree of matching between the current position and the optimal operating region and the optimal proximity direction. , , , and These are the weighting coefficients.
[0053] In one embodiment of the present invention, the comprehensive cost function is expressed by the following formula: For the comprehensive cost function, For navigation path cost, The overall operational score for candidate parking positions. As a result of spatial constraints, , and These are the weighting coefficients; The spatial constraint cost is expressed by the following formula: This represents the angular deviation between the actual approach direction corresponding to the candidate parking position and the optimal approach direction. The degree of intrusion of the candidate parking space into the restricted area. This is an indication value for the violation of the operation sequence constraint by the candidate docking position. , and These are the weighting coefficients.
[0054] According to another embodiment, an electronic device is also provided, including a memory and a processor, wherein executable code is stored in the memory, and when the processor executes the executable code, it implements a combination... Figure 1 The method.
[0055] The various embodiments in this invention are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the device embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.
[0056] Those skilled in the art will recognize that, in one or more of the examples above, the functions described in this invention can be implemented using hardware, software, firmware, or any combination thereof. When implemented in software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or code on a computer-readable medium.
[0057] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made on the basis of the technical solution of the present invention should be included within the scope of protection of the present invention.
Claims
1. A VLA-based mobile manipulator collaborative planning method, characterized in that, include: Based on the operation task of the target object, the availability information for the robotic arm to perform the operation task is obtained using the VLA model. The availability information includes the optimal operation area, the optimal approach direction of the robotic arm end effector, and the spatial constraints during the operation. Based on the three-dimensional spatial information of the target object and the availability information, an operability map is constructed. The basic operability indicators for each location in the operability map include accessibility indicators, visibility indicators, grasping stability indicators, and collision risk indicators. Based on the aforementioned basic operational indicators and spatial constraints, several candidate docking positions are determined around the target object. Each candidate docking position includes planar position information and chassis orientation information of the robotic arm. The comprehensive operational score of each candidate docking station is calculated based on its basic operational metrics, its matching degree with the optimal operational area and the optimal approach direction, and the navigation path cost of moving from the current position to the candidate docking station. Based on the comprehensive operation score of each candidate docking position and the navigation path cost, a comprehensive cost function is constructed. The optimal docking position and optimal navigation path that make the comprehensive cost function optimal are determined. The chassis of the robotic arm is controlled to move along the optimal navigation path to the optimal docking position, and then the robotic arm performs the operation task.
2. The method of claim 1, wherein, The target object-based operation task utilizes a VLA model to obtain availability information for the robotic arm to perform the operation task, including: The natural language instructions for the operation task and the current scene image are input into the trained VLA model to obtain the optimal operation area, the optimal approach direction of the robotic arm end effector, and the spatial constraints during the operation process. The optimal operating area is the area on the surface of the target object that can be grasped or manipulated. The optimal approach direction is the angular range of the approach direction of the robotic arm end relative to the target object. The angular range includes yaw angle components and pitch angle components. The spatial constraints include prohibited areas that the robotic arm and chassis must avoid during operation, as well as the operation sequence constraints corresponding to each sub-task in the multi-step operation task.
3. The method of claim 1, wherein, The accessibility index is used to characterize whether each position is within the kinematic reach space of the robotic arm; The visibility index is used to characterize whether the line of sight between each location and the optimal operating area is obstructed by obstacles; The grasping stability index is used to characterize the expected success rate of the robotic arm when performing grasping or operation at each position; The collision risk index is used to characterize the probability of collision between the robotic arm and chassis and environmental obstacles when operating at each position.
4. The method of claim 3, wherein, The process of determining several candidate docking positions around the target object based on the aforementioned basic operational indicators and spatial constraints includes: Based on the aforementioned basic operational metrics and the historical operational success rate of each location, an operability score is generated. The locations within the area surrounding the target object that meet the spatial constraints and have an operability score greater than the score threshold are identified as candidate docking positions, and a set of candidate docking positions is generated. The candidate parking position set is Each parking position is , is a position coordinate of the position in a two-dimensional plane coordinate system, used to represent plane position information, is an orientation angle of the chassis, used to represent chassis orientation information.
5. The method of claim 4, wherein, The operability score is expressed by the following formula: is an operability score, , is an accessibility indicator, is a visibility indicator, is a grasping stability indicator, is a collision risk indicator, , , and is a weight coefficient, is a historical operation success rate, is a historical experience weight coefficient.
6. The method of claim 1, wherein, The comprehensive operational score is expressed by the following formula: is a comprehensive operation score, is a navigation path cost, is an accessibility score weighted based on an accessibility indicator, is a grasping quality score weighted based on a grasping stability indicator, is a collision risk score weighted based on a collision risk indicator, is a semantic preference score, used to represent the matching degree between the current position and the best operation region and the best approaching direction, , , , and is a weight coefficient.
7. The method of claim 6, wherein, The comprehensive cost function is expressed by the following formula: is a comprehensive cost function, is a navigation path cost, is a comprehensive operation score for a candidate stop position, is a space constraint cost, , and are weight coefficients; The spatial constraint cost is expressed by the following formula: an angle deviation between an actual approach direction corresponding to the candidate stop position and the best approach direction, an intrusion degree of the candidate stop position into the no-entry area, an indication value of violation of the operation sequence constraint by the candidate stop position, 、 and are weight coefficients.
8. A VLA-based collaborative planning device for a mobile robotic arm, characterized in that, include: The available information acquisition module, based on the operation task of the target object, uses the VLA model to obtain the availability information for the robotic arm to perform the operation task. The availability information includes the optimal operation area, the optimal approach direction of the robotic arm end effector, and the spatial constraints during the operation. The map building module, connected to the available information acquisition module, constructs an operable map based on the three-dimensional spatial information of the target object and the available information. The basic operational indicators for each location in the operable map include accessibility indicators, visibility indicators, grasping stability indicators, and collision risk indicators. The candidate location determination module is connected to the map construction module. Based on the basic operation indicators and spatial constraints, it determines several candidate docking positions around the target object. Each candidate docking position includes planar position information and chassis orientation information of the robotic arm. The comprehensive scoring determination module, connected to the candidate location determination module, calculates the comprehensive operational score of each candidate docking station based on its basic operational indicators, its matching degree with the optimal operational area and the optimal approach direction, and the navigation path cost of moving from the current location to the candidate docking station. The optimal path planning module, connected to the comprehensive scoring determination module, constructs a comprehensive cost function based on the comprehensive operation score of each candidate docking position and the navigation path cost. It then determines the optimal docking position and optimal navigation path that optimizes the comprehensive cost function. This allows the robot arm's chassis to move along the optimal navigation path to the optimal docking position, after which the robot arm performs the operation task.
9. An electronic device, characterized in that, It includes a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the method as described in any one of claims 1-7.
10. A computer-readable storage medium, characterized in that, It stores a computer program that, when executed in a computer, causes the computer to perform the method described in any one of claims 1-7.
