Path planning methods, path planning devices, and robots for robots

Abstract

This application discloses a path planning method, a path planning device, and a robot. The method includes: acquiring a visual image of the robot's forward direction; determining a contact risk assessment result of the target object output by the VLM model based on the visual image and a VLM model; if the contact risk assessment result indicates that the robot cannot directly contact the target object, determining a first path for the robot and controlling the robot to move along the first path; if the contact risk assessment result indicates that the robot can directly contact the target object, determining a second path for the robot and controlling the robot to move along the second path. This approach enables the robot to distinguish between contactable and inaccessible objects based on semantic risk assessment, actively traversing soft obstacles when contactable, and avoiding deadlock problems caused by absolute obstacle avoidance.

Description

Technical Field

[0001] This application relates to the field of robot path planning, and more particularly to a robot path planning method, path planning device, and robot. Background Technology

[0002] As service robots and humanoid robots gradually enter unstructured living and working environments such as homes, nursing homes, supermarkets, and warehouses, robot motion planning faces challenges that are drastically different from those in laboratories or factories. In traditional structured industrial environments, the robot's workspace is precisely modeled, and the position, shape, and material of all obstacles are pre-defined. The motion planner can calculate a completely collision-free trajectory based on the accurate geometric model.

[0003] However, traditional robot motion planning algorithms treat all objects in the environment as insurmountable obstacles with infinite rigidity. When a robot needs to traverse a narrow space partially obscured by soft obstacles such as curtains, door curtains, or low shrubs, the planner may be unable to find a collision-free path and get stuck in a deadlock, making it impossible for the robot to pass through. Summary of the Invention

[0004] This application addresses some shortcomings by providing a robot path planning method, a path planning device, and a robot. This solution enables the robot to perform semantic risk assessment of objects based on a vision-language model (VLM), intelligently distinguishing between accessible soft obstacles and inaccessible dangerous objects. Furthermore, it actively traverses low-risk areas instead of detouring or getting stuck, effectively overcoming the robot freezing problem caused by traditional absolute obstacle avoidance logic. Simultaneously, this solution deeply integrates semantic understanding and motion planning, significantly improving the safety and task completion efficiency of robot movement in unstructured environments.

[0005] In a first aspect, a path planning method for a robot is provided. The method includes: acquiring a visual image of the robot's forward direction, the visual image including a target object; determining a contact risk assessment result of the target object output by the VLM model based on the visual image and a VLM model, the contact risk assessment result indicating the risk of the robot directly contacting the target object; if the contact risk assessment result indicates that the robot cannot directly contact the target object, determining a first path for the robot and controlling the robot to move along the first path, the first path avoiding the target object; if the contact risk assessment result indicates that the robot can directly contact the target object, determining a three-dimensional risk field of the three-dimensional scene in which the robot is located, the three-dimensional scene including the target object, the three-dimensional risk field indicating the contact risk at different locations in the three-dimensional scene; and determining a second path for the robot based on the three-dimensional risk field and controlling the robot to move along the second path, the second path passing through the target object.

[0006] This approach eliminates the perception of all objects as insurmountable rigid obstacles. Instead, the robot autonomously decides whether contact is acceptable based on a risk score output from the VLM model. Consequently, it can proactively plan paths through low-risk soft obstacles such as curtains and door curtains, completely resolving the deadlock problem caused by traditional planners failing to find collision-free paths. Furthermore, this approach directly integrates risk assessment results into the path cost function, aligning the robot's movement behavior with human intuition. For example, it strictly avoids high-value, fragile objects while allowing gentle contact with soft, harmless objects, significantly improving mobility and task completion rates in unstructured environments.

[0007] In conjunction with the first aspect, in a possible implementation of the first aspect, determining the second path of the robot based on the three-dimensional risk field includes: acquiring multiple candidate paths of the robot, and determining a contact risk score for each sampling point on the multiple candidate paths based on the three-dimensional risk field; acquiring the task intent of the robot; determining the collision cost of each spatial region in the three-dimensional scene based on the contact risk score of each sampling point on the multiple candidate paths and the task intent, wherein the collision cost is used to indicate the severity of a collision when the robot collides in a spatial region in the three-dimensional scene; and determining the second path from the multiple candidate paths based on the collision cost information, wherein the second path includes spatial regions in the three-dimensional scene where the collision cost is less than a preset cost threshold.

[0008] This approach enables robots to dynamically adjust the collision cost weights of different spatial regions based on high-level task semantics (such as rapid passage or careful operation), allowing path selection in the same environment to adapt to different tasks. For example, when the instruction is rapid passage, the collision cost in low-risk areas is intentionally reduced to encourage passage through soft obstacles; when the instruction is precise operation, even low-risk objects are assigned higher obstacle avoidance weights. This achieves intention-driven flexible motion planning, significantly improving flexibility and safety in multi-task scenarios.

[0009] In conjunction with the first aspect, in a possible implementation of the first aspect, determining the second path of the robot based on the three-dimensional risk field further includes: determining a contact risk score for each sampling point on the second path based on the three-dimensional risk field; and determining the expected stiffness parameter and expected contact intention for each time step of the second path based on the contact risk score for each sampling point on the second path, wherein the expected stiffness parameter is used to indicate the expected stiffness of the robot's end effector in each direction of Cartesian space at each time step.

[0010] This approach allows the motion planner to output not only the geometric pose trajectory but also the expected stiffness parameters and contact intentions at each time step along the trajectory, achieving unified planning of the trajectory and stiffness map. Furthermore, it fundamentally solves the problem of the separation between path planning and impedance control in traditional solutions. The planner clearly understands which segments will experience expected contact and presets compliant parameters, and which segments require precise positioning and preset stiffness parameters. This enables the robot to smoothly and continuously perform actions such as gently opening curtains and precisely grasping fragile objects in a single motion.

[0011] In conjunction with the first aspect, in a possible implementation of the first aspect, determining the expected stiffness parameter for each time step of the second path based on the contact risk score of each sampling point on the second path includes: obtaining the minimum stiffness and maximum stiffness on the second path; determining the expected stiffness parameter for each time step of the second path based on the contact risk score of the current sampling point, the minimum stiffness, and the maximum stiffness, wherein the expected stiffness parameter is the sum of the minimum stiffness and the product of the contact risk score and the stiffness difference, and the stiffness difference is the difference between the maximum stiffness and the minimum stiffness.

[0012] This scheme ensures that the stiffness parameter changes smoothly along the trajectory, preventing abrupt jitter. Furthermore, by adjusting the specific values ​​of the minimum and maximum stiffness, it can be easily adapted to different robotic arm hardware and task requirements, demonstrating strong engineering practicality.

[0013] In conjunction with the first aspect, in a possible implementation of the first aspect, controlling the robot to move along the second path includes: adjusting the stiffness and damping parameters of the robot according to the trajectory changes of the second path during the movement of the robot.

[0014] This approach eliminates the globally fixed parameters of the impedance controller, allowing them to be dynamically adjusted in real time as the robot moves along its trajectory. For example, it automatically reduces stiffness to achieve compliant pushing when traversing soft obstacles, and automatically increases stiffness to ensure positioning accuracy when approaching high-value objects. This dynamic adaptive impedance control enables the robot to handle continuously changing behaviors from extremely compliant to extremely rigid with the same control framework, avoiding insufficient accuracy or collision damage caused by fixed parameters.

[0015] In conjunction with the first aspect, in a possible implementation of the first aspect, the contact risk assessment result includes the maximum accessible force of the target object. Based on the trajectory changes of the second path, the stiffness and damping parameters of the robot are adjusted, including: acquiring the real-time contact force during the robot's movement along the second path; adjusting the stiffness and damping parameters of the robot based on a comparison between the maximum accessible force and the real-time contact force, wherein when the real-time contact force is greater than the product of the maximum accessible force and a preset coefficient, the stiffness of the robot is increased, and the preset coefficient is a positive number less than 1; when the real-time contact force is greater than the maximum accessible force, the impedance offset is increased in the opposite direction of the real-time contact force.

[0016] This scheme introduces a closed-loop feedback mechanism for contact force and a dual protection mechanism. When the actual contact force exceeds a certain level of the maximum contact force estimated by the VLM (Vibration Modeling Library), the local stiffness is automatically increased to limit the contact depth. When the contact force exceeds the upper limit, protective yielding is immediately implemented, triggering a VLM reassessment. This mechanism effectively prevents damage to objects or robots caused by VLM assessment deviations. Simultaneously, it utilizes actual contact data to calibrate the risk scoring model online, forming a continuously optimizing learning loop, greatly improving the system's robustness in unstructured environments.

[0017] In conjunction with the first aspect, in a possible implementation of the first aspect, the three-dimensional risk field of the three-dimensional scene in which the robot is located is determined based on the contact risk assessment result, including: obtaining the center position of the target object and the risk diffusion radius related to the size of the target object; and using a Gaussian diffusion model to diffuse the contact risk assessment result of the target object to the occupied area of ​​the target object and its neighborhood based on the center position, the risk diffusion radius and the contact risk assessment result, thereby obtaining the three-dimensional risk field.

[0018] This scheme enables the continuous risk field representation to perform fine-grained risk assessment for each sampling point on the path, avoiding abrupt changes at object boundaries. It also supports the superposition and fusion of risk fields from multiple objects, providing the planner with a rich, geometrically aligned semantic cost map.

[0019] In conjunction with the first aspect or the above-described implementation of the first aspect, in another possible implementation of the first aspect, the contact risk assessment result includes at least one of the following: object category of the target object, material inference, risk score, maximum contactable force, and contact direction constraint.

[0020] In conjunction with the first aspect or the above-described implementation of the first aspect, in another possible implementation of the first aspect, the safety constraints of the robot include at least one of joint limiting, self-collision avoidance, and human area avoidance.

[0021] This scheme ensures that safety constraints are always executed with the highest priority through zero-space projection, regardless of stiffness adjustments. This, in turn, guarantees that compliant control will never override fundamental safety constraints, fundamentally protecting the robot's physical and human-robot interaction safety under all circumstances.

[0022] Secondly, a path planning device for a robot is provided, comprising: a data acquisition module for acquiring a visual image of the robot in its forward direction, the visual image including a target object; a processing module for determining, based on the visual image and a visual-language-action (VLM) model, a contact risk assessment result of the target object output by the VLM model, the contact risk assessment result indicating the risk of the robot directly contacting the target object; specifically, the processing module for determining a first path for the robot and controlling the robot to move along the first path, the first path avoiding the target object, when the contact risk assessment result indicates that the robot cannot directly contact the target object; specifically, the processing module for determining a three-dimensional risk field of the three-dimensional scene in which the robot is located, the three-dimensional scene including the target object, the three-dimensional risk field indicating the contact risk at different locations in the three-dimensional scene, the processing module for determining a second path for the robot and controlling the robot to move along the second path, the second path passing through the target object, based on the three-dimensional risk field, when the contact risk assessment result indicates that the robot can directly contact the target object; and specifically, the processing module for determining a second path for the robot and controlling the robot to move along the second path, the second path passing through the target object, based on the three-dimensional risk field.

[0023] Thirdly, a robot is provided, comprising: at least one memory and at least one processor, the memory being coupled to the processor; the memory being used to store computer program code / instructions; and when the computer program code / instructions are executed by the processor, causing the robot to implement the control method as described in any one of the first aspects or any possible implementations of the first aspect.

[0024] Fourthly, a readable storage medium is provided that stores instructions which, when executed on an electronic device, cause the electronic device to implement the control method as described in any one of the first aspects or any possible implementations of the first aspect.

[0025] Fifthly, a computer program product is provided, comprising: computer instructions that, when executed on an electronic device, cause the electronic device to implement the control method as described in any one of the first aspects or any possible implementation thereof. Attached Figure Description

[0026] Figure 1 A schematic diagram illustrating a robot path planning technique is shown below;

[0027] Figure 2 A schematic flowchart of the robot path planning method provided in an embodiment of this application is shown;

[0028] Figure 3 A flowchart of a path planning method for a robot provided in an embodiment of this application is shown;

[0029] Figure 4 This illustration shows a path planning method based on a three-dimensional risk field, as provided in an embodiment of this application.

[0030] Figure 5 A schematic diagram illustrating the experimental results of robot path planning-based motion provided in an embodiment of this application is shown.

[0031] Figure 6 This illustration shows a schematic diagram of the module composition of a path planning device provided in an embodiment of this application;

[0032] Figure 7 A block diagram of the robot provided in an embodiment of this application is shown;

[0033] Figure 8 This illustration shows a structural schematic diagram of a system-on-chip (SoC) provided in an embodiment of this application. Detailed Implementation

[0034] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0035] First, the prior art and existing technical problems involved in the embodiments of this application are introduced.

[0036] As mentioned earlier, traditional motion planning algorithms treat all objects in the environment as insurmountable obstacles with infinite stiffness. Figure 1 A schematic diagram illustrating a robot path planning technique is shown below. (Example) Figure 1 As shown, when the robot needs to pass through... Figure 1 When a narrow space is partially obscured by soft obstacles such as curtains or door curtains, the robot's planner will determine that the soft obstacle is an impassable obstacle with infinite rigidity. Therefore, the robot's planner will choose to avoid the soft obstacle. Ultimately, in Figure 1 In the scenario shown, the robot will get stuck in a freezing robot problem because it cannot find a collision-free path, and will be completely unable to pass through soft obstacles such as curtains and door curtains.

[0037] Faced with the complexities of real-world physical environments, a robot's motion planning strategy must transcend... Figure 1 The diagram illustrates a simple obstacle avoidance logic. In everyday life, when navigating cluttered spaces, humans naturally determine their movement strategies based on a semantic understanding of object properties. For example, gently pushing aside a hanging curtain to cross a corridor, avoiding a glass full of hot water, or using physical contact to push open a spring-loaded door. This intelligent decision-making ability based on acceptable contact is essential for robots to operate efficiently in real-world environments.

[0038] Furthermore, existing geometry-based motion planners only process the three-dimensional geometric information of objects, such as point clouds and meshes, and completely fail to understand the semantic attributes of objects. For example, a motion planner cannot distinguish between a piece of glass (e.g., hard, fragile, high value) and a piece of foam board of the same size (e.g., soft, low value, accessible). Moreover, motion planners impose the same safety distance constraints on all objects, resulting in extremely low motion efficiency in cluttered environments.

[0039] To address the aforementioned issues, this application proposes a path planning method 100 for robots. Figure 2 A schematic flowchart of a robot path planning method 100 provided in an embodiment of this application is shown. Figure 2As shown, method 100 includes steps S110 to S150. In method 100, the robot is able to perform risk assessment of objects based on a VLM model, intelligently distinguishing between accessible soft obstacles and inaccessible dangerous objects. Consequently, the robot can actively traverse low-risk areas instead of detouring or getting stuck, effectively overcoming the robot freezing problem caused by traditional absolute obstacle avoidance logic. Simultaneously, method 100 deeply integrates semantic understanding and motion planning, significantly improving the safety and task completion efficiency of robot movement in unstructured environments.

[0040] Step S110: Obtain a visual image of the robot in the forward direction.

[0041] The visual image can be a two-dimensional color image acquired by the robot through a visual sensor, such as an image containing the three color channels of RGB. This visual image includes the target object, which can refer to obstacles or interactive objects that the robot may encounter along its movement path. For example, the target object could be a cotton curtain hanging in a corridor, a cardboard box on the floor, or a glass of water on a table.

[0042] For example, in a home corridor traversal scenario, the robot is equipped with a depth camera that can continuously capture color images at a frequency of no less than 15 frames per second. These color images serve as the robot's visual images in its forward direction. Subsequently, after instance segmentation, one or more target objects, such as curtains, glasses, or cardboard boxes, can be identified from these visual images.

[0043] Step S120: Based on the visual image and the VLM model, determine the contact risk assessment result of the target object output by the VLM model.

[0044] The VLM model can identify objects from images and infer their semantic attributes. For example, the VLM model can identify from an image "this is a fragile glass vase" or "this is a soft cotton curtain". Optionally, in embodiments of this application, the VLM model can be a multimodal large model, fine-tuned for material classification and risk assessment datasets, capable of inferring the category, material, fragility, and accessibility of objects from images.

[0045] In embodiments of this application, the contact risk assessment result can be used to indicate the risk of the robot directly contacting the target object. For example, the contact risk assessment result may include a risk score, maximum acceptable contact force, material inference, contact direction constraints, and risk semantic explanation text. For instance, the contact risk assessment result may refer to the comprehensive judgment output by the VLM model regarding whether an object is suitable for direct physical contact by the robot, with values ​​ranging from 0 to 1. Here, 0 indicates fully accessible, such as a hanging empty curtain; 1 indicates absolutely inaccessible, such as a glass cup. For another example, if the material inference of a cotton curtain is soft fabric, it could correspond to a risk score of 0.05, a maximum acceptable contact force of 15N, and a recommended side-push opening, with the risk explanation text "Soft fabric, safe to push sideways." For yet another example, if the material inference of a glass cup is fragile, it would output a risk score of 0.9, a maximum acceptable contact force of 2N, and the risk explanation text "Glass material is fragile, must be avoided."

[0046] Step S130: If the contact risk assessment result indicates that the robot cannot directly contact the target object, determine the first path of the robot and control the robot to move along the first path.

[0047] Since the robot cannot directly contact the target object, the first path avoids the target object. For example, the first path can refer to a collision-free trajectory from the robot's current position to the target point, with the trajectory maintaining a safe distance from the high-risk target object.

[0048] For example, in an embodiment of this application, when the risk score in the contact risk assessment result is higher than a preset threshold, such as 0.8, or the maximum acceptable contact force F is close to 0, the target object is determined to be untouchable. Then, the motion planner treats the untouchable target object as a rigid, impassable obstacle, and can generate a first path that completely avoids the target object. For example, for a glass with a risk score of 0.92, the motion planner will generate a detour path away from the glass, so that neither the robot end effector nor the robot body will have physical contact with the glass.

[0049] Step S140: If the contact risk assessment result indicates that the robot can directly contact the target object, determine the three-dimensional risk field of the three-dimensional scene in which the robot is located based on the contact risk assessment result.

[0050] The three-dimensional scene includes the target object, and the three-dimensional risk field is used to indicate the contact risk at different locations in the three-dimensional scene. For example, in an embodiment of this application, the three-dimensional risk field is a voxel grid covering the robot's workspace, where each voxel stores a contact risk score value between 0 and 1, and the resolution can be between 1 and 5 cm.

[0051] For example, in an embodiment of this application, for a reachable target object, the center position of the object and the risk diffusion radius related to the object's size can be read, and a Gaussian diffusion model is used to diffuse the risk score in the contact risk assessment result of the object to the object's occupied area and its neighborhood, forming a continuous three-dimensional risk field. The contact risk score is highest at the center of the target object, then decays outwards according to a Gaussian function, with the risk in the vacant area approaching 0.

[0052] Step S150: Based on the three-dimensional risk field, determine the second path of the robot and control the robot to move along the second path.

[0053] Unlike the first path described above, this second path passes through the target object. For example, in an embodiment of this application, the second path may refer to a motion trajectory that allows the robot to make predictable contact with the target object, such as a path that passes directly through a curtain. Compared to the traditional planner described above, which may lock up due to its inability to find a collision-free path, this second path passing through the target object means that the robot's robotic arm or body will make physical contact with the object during its movement, and the contact process is gentle and safe.

[0054] The method 100 proposed in this application improves interpretability and auditability by acquiring a visual image of the robot's forward direction and using a VLM model to determine the contact risk assessment result of the target object. Furthermore, it generates avoidance paths or a three-dimensional risk field and a second path containing active contact based on the accessibility, achieving intelligent differentiation between soft, accessible objects and rigid, dangerous objects. This enables the robot to autonomously decide, like a human, when to accept contact and when to strictly avoid it, fundamentally breaking the deadlock problem caused by traditional absolute obstacle avoidance and significantly improving motion efficiency and task completion rate in unstructured environments.

[0055] The following is about Figure 1 The feasible options for certain steps or other feasible options are described in detail.

[0056] As mentioned above, the embodiments of this application can use a Gaussian diffusion model to diffuse the contact risk assessment results of the target object to its occupied area and neighborhood based on the center position of the target object, the risk diffusion radius, and the contact risk assessment results, thereby obtaining a three-dimensional risk field. The risk diffusion radius can be positively correlated with the size of the target object; for example, a large cardboard box has a larger diffusion radius than a small vase, thus creating a gradually changing safety buffer zone around the object within the three-dimensional risk field.

[0057] This 3D risk field supports the superposition and fusion of risk fields from multiple objects. Specifically, the risk value at each spatial location is the maximum value or weighted sum of the contributions from all objects at that location. For example, in a corridor scene, a cotton curtain has a risk score of 0.08, and the risk value within its occupied area and a 10cm radius decreases from 0.08 to 0. A glass has a risk score of 0.92; its size is smaller than the cotton curtain, resulting in a smaller surrounding radius of about 5cm, but its internal risk value is close to 0.92. In the final generated 3D risk field, the curtain area is light green (low risk), the glass area is dark red (high risk), and the risk value in the middle of the empty corridor is close to 0. The risk values ​​in other areas can be the superposition of the risk values ​​within the cotton curtain's radius and the glass's radius.

[0058] Optionally, in step 140, this embodiment of the application can dynamically determine the collision cost by combining the contact risk score of the candidate paths and the task intent, thereby selecting the optimal path. Specifically, this embodiment of the application can obtain multiple candidate paths for the robot and determine the contact risk score of each sampling point on the multiple candidate paths based on the three-dimensional risk field. Then, the task intent of the robot can be obtained. For example, the task intent can refer to a high-level semantic goal parsed from user voice commands or task planning, such as "quickly cross the corridor to reach the kitchen" or "carefully approach the vase on the table," etc.

[0059] Then, this embodiment of the application can determine the collision cost of each spatial region in the 3D scene based on the contact risk score of each sampling point on the multiple candidate paths and the task intent. The collision cost indicates the severity of a collision that occurs when the robot collides with a spatial region in the 3D scene. Finally, this embodiment of the application can determine the second path from the multiple candidate paths based on the collision cost information. The second path includes spatial regions in the 3D scene where the collision cost is less than a preset cost threshold.

[0060] For example, in an embodiment of this application, the collision cost can be equal to the risk at the current location multiplied by the expected contact force. Therefore, for low-risk objects, the contact force cost approaches zero even if present, while for high-risk objects, any contact force incurs a significant cost. For instance, in the task of "going to the kitchen to get a water glass," the task intent is to navigate to the kitchen and retrieve the water glass. The robot's motion planner then generates multiple candidate paths, including a path through a curtain and other detour paths. When calculating the cost of multiple candidate paths, the path through the curtain has a low risk score and small contact force, resulting in a contact cost of only 0.08 × 0.5N = 0.04, far less than the additional travel cost of other detour paths. Therefore, the path through the curtain is selected as the second path.

[0061] Optionally, in embodiments of this application, the weights of collision costs can be adjusted using a cross-attention network. The task intent can dynamically adjust the collision cost weights. When the intent is to traverse quickly, the collision cost weights in low-risk areas are reduced to encourage traversing soft obstacles; when the intent is to operate cautiously, even low-risk objects are assigned higher obstacle avoidance weights.

[0062] Optionally, in step S150, the second path output by the planner not only includes the end-effector pose sequence, but may also include the expected stiffness parameters and expected contact intentions at each time step along the trajectory in the second path. This allows for the formation of a complete trajectory and stiffness map. Furthermore, when controlling the robot's movement, the impedance controller can adjust the compliance of the robotic arm in real time based on the expected stiffness at the current position, thereby enabling the safe pushing away and passing of soft obstacles.

[0063] Specifically, determining the robot's second path based on the three-dimensional risk field further includes: determining a contact risk score for each sampling point on the second path based on the three-dimensional risk field; and determining the expected stiffness parameter and expected contact intention for each time step of the second path based on the contact risk score for each sampling point on the second path. The expected stiffness parameter can be used to indicate the expected stiffness of the robot's end effector in various directions of Cartesian space at each time step.

[0064] For example, the anticipated contact intention can refer to the motion planner's prediction of whether physical contact will occur on a certain path segment. For instance, it might be marked as anticipated contact when passing through a curtain, and as no contact in an open area. The anticipated stiffness parameter can be a parameter that maps the risk score to a specific stiffness value, allowing the robot to automatically become compliant in contact segments and remain rigid in non-contact segments. For example, in the section of the second path mentioned above where the robot passes through the curtain, the contact risk score at the sampling point is 0.08. The planner marks this section as anticipated contact and calculates the anticipated stiffness to be approximately 120 N / m. In the section near the water cup, the risk score is 0.92, and it is marked as no contact or strict avoidance intention, with an anticipated stiffness of approximately 1850 N / m.

[0065] Optionally, embodiments of this application can calculate the expected stiffness for each time step using a linear mapping formula. Specifically, based on the contact risk score of each sampling point on the second path, the expected stiffness parameter for each time step of the second path is determined, including: obtaining the minimum stiffness and maximum stiffness on the second path; and determining the expected stiffness parameter for each time step of the second path based on the contact risk score of the current sampling point, the minimum stiffness, and the maximum stiffness. The expected stiffness parameter is the sum of the minimum stiffness and the product of the contact risk score and the stiffness difference, where the stiffness difference is the difference between the maximum stiffness and the minimum stiffness. This linear mapping ensures continuous change in stiffness along the path, preventing abrupt changes that could cause jitter. By adjusting the values ​​of the minimum and maximum stiffness, the hardware characteristics of different robotic arms can be adapted.

[0066] It should be understood that existing impedance control schemes typically use globally fixed stiffness and damping parameters. Consequently, when precision operations are required, the fixed low stiffness can lead to insufficient positioning accuracy; and in the event of an accidental collision with a hard object, the fixed low damping cannot provide sufficient braking force, potentially causing damage to the robot or the object. In other words, the impedance parameters cannot be adjusted in real time based on the physical and semantic properties of the current interactive object.

[0067] Optionally, to address the technical problem that a globally fixed impedance parameter cannot simultaneously satisfy both compliant passage and precise positioning, in embodiments of this application, the stiffness and damping parameters can be dynamically adjusted based on the trajectory changes of the second path during robot movement. Specifically, controlling the robot to move along the second path includes adjusting the robot's stiffness and damping parameters according to the trajectory changes of the second path during the robot's movement. It should be understood that stiffness and damping parameters are core parameters of the impedance controller. Stiffness determines the robot's ability to resist positional deviations caused by external forces, while damping determines its energy dissipation characteristics. For example, dynamic adjustment can include automatically reducing the robot's stiffness when passing through a curtain and automatically increasing its stiffness when approaching a fragile object, achieving smooth behavior switching within the same motion task.

[0068] Furthermore, by adjusting the virtual stiffness and virtual damper parameters of the robotic arm's end effector, the robot can exhibit controllable compliant behavior when physically contacting its environment. This addresses the problem that existing impedance control schemes cannot adaptively adjust based on the semantic attributes of the contacting object.

[0069] Optionally, to address the safety risk that VLM assessment deviations may cause contact forces to exceed the object's tolerance limits, embodiments of this application may also introduce a contact force feedback protection mechanism. Specifically, the contact risk assessment result includes the maximum accessible force of the target object. Based on the trajectory changes of the second path, the stiffness and damping parameters of the robot are adjusted, including: acquiring the real-time contact force during the robot's movement along the second path; and adjusting the robot's stiffness and damping parameters based on a comparison between the maximum accessible force and the real-time contact force. Specifically, when the real-time contact force is greater than the product of the maximum accessible force and a preset coefficient, the robot's stiffness is increased. The preset coefficient is a positive number less than 1. When the real-time contact force is greater than the maximum accessible force, the impedance offset is increased in the opposite direction of the real-time contact force.

[0070] For example, the maximum accessible force is the upper limit of the contact force that an object can withstand, as estimated by the VLM model. For instance, 20N for a curtain and 3N for a glass. The preset coefficient can be 0.8, meaning that when 80% of the upper limit is reached, an early warning is triggered, and the stiffness is increased to limit the contact depth. When the actual contact force exceeds the maximum accessible force, the system determines that an anomaly has occurred, immediately performs a protective retreat, increases the impedance offset in the opposite direction of the contact force to disengage the robot from contact, and triggers the VLM model to reassess the risk level of the object.

[0071] For example, the VLM model assesses the maximum accessible force of the door curtain as 15N. Subsequently, after the robot contacts the curtain, the real-time contact force increases to 12N (greater than 15N × 0.8 = 12N). The robot system then issues a warning, increasing the local stiffness from 100N / m to 300N / m. Then, when the contact force continues to rise to 16N, exceeding the 15N limit, the system immediately increases the impedance offset by 40mm in the opposite direction, and the robotic arm smoothly retreats to disengage from contact, triggering a reassessment of the VLM model. For instance, if the reassessment reveals a metal object hidden behind the curtain, the risk score can be increased from 0.08 to 0.5, and the subsequent motion planner will select an alternative path.

[0072] Optionally, the robot safety constraints in this embodiment may include at least one of joint limiting, self-collision avoidance, and human area avoidance. Regardless of the risk adjustment settings, the robot's basic safety constraints are always executed with the highest priority through null-space projection. For example, joint limiting refers to automatically injecting a repulsive torque away from the limit into the control torque when a joint angle approaches its hardware limit; self-collision avoidance refers to injecting an avoidance component when a potential collision is detected with the robot's own components; human area avoidance refers to applying the highest stiffness and maximum safe distance constraints to the human area identified by the VLM model, regardless of the risk score, to ensure human-robot safety. This multi-priority mechanism ensures that compliant control will never override the basic safety constraints.

[0073] Through the above scheme, this embodiment of the application achieves a smooth transition from gently opening a curtain to precisely grasping a fragile item in a single movement by mapping the risk score to a dynamically changing expected stiffness parameter along the trajectory and combining it with impedance control. Furthermore, by introducing a closed-loop feedback mechanism for contact force and a dual protection mechanism, physical damage caused by VLM model evaluation bias is effectively prevented, and the risk model is calibrated online using actual contact data, forming a continuously optimizing learning loop that greatly enhances the robustness and safety of the system.

[0074] The following describes a detailed embodiment of method 100 with reference to the accompanying drawings.

[0075] Example 1:

[0076] Figure 3 A flowchart of a robot path planning method according to an embodiment of this application is shown. The process includes steps S210 to S240.

[0077] First, we will introduce step S210, which involves environmental perception and risk assessment.

[0078] Step S210 includes three-dimensional environment perception (step S211), VLM risk inference (step S212), and three-dimensional risk matrix generation (step S213). The VLM model can not only identify object categories, but also infer the material properties, vulnerability scores, and acceptable contact forces of each object, providing semantic-level physical interaction guidance for subsequent motion planning.

[0079] Corresponding to step S110, in step S211, an RGB-D depth camera, LiDAR, joint encoder, and force / torque sensor can be installed on the robot's head or body. The RGB-D camera can continuously acquire color images and aligned depth maps at a frequency of at least 15fps, generating 3D point cloud data within the current field of view (i.e., the visual image in step S110), including XYZ coordinates and RGB colors. Furthermore, a 3D occupancy grid map can be constructed and continuously updated within the perception range to distinguish between observed occupied areas, vacant areas, and unknown areas. In addition, embodiments of this application can also perform instance segmentation of objects in the environment based on the depth map and point cloud, obtaining point cloud clusters and corresponding RGB image regions for each independent object.

[0080] Corresponding to step S120, in step S212, the present application embodiment can send the current RGB frame and the segmented object region into VLM for semantic reasoning, and then infer the vulnerability and danger score of each environmental object (that is, the target object in the aforementioned step S120) (that is, the contact risk assessment result in the aforementioned step S120).

[0081] For example, the VLM model is a targeted fine-tuned vision-language model based on a multimodal large model architecture, taking RGB images and / or point cloud data as input and outputting structured risk assessment results. For instance, the fine-tuning could be done using LoRA (low-rank adaptation) for material classification and risk assessment datasets. As another example, the fine-tuning dataset could contain real-world image-risk assessment annotation pairs, covering the following object categories. Among them, the risk scores for textiles such as curtains, door curtains, clothing, and towels range from 0.02 to 0.15; the risk scores for cardboard / foam products such as express delivery boxes, foam boards, and paper bags range from 0.05 to 0.20; the risk scores for plastic products such as plastic chairs and plastic boxes range from 0.20 to 0.50; the risk scores for fragile items such as glassware, ceramics, and electronic devices range from 0.80 to 0.98; the risk score for human and animal products is 1.0 (i.e., a fixed value, not dependent on model reasoning); and the risk scores for fixed structures such as walls, table legs, and pillars range from 0.70 to 0.90.

[0082] For example, the VLM model can output the following structured information for each detected object (i.e., the contact risk assessment result in step S120 above): object category name (e.g., cotton curtain, glass vase, metal table leg); material inference (e.g., soft fabric, rigid and fragile, rigid and strong); risk score in the range of 0 to 1, where 0 indicates that it is completely accessible, such as a hanging empty curtain, and 1 indicates that it is absolutely inaccessible, such as an open glass filled with hot water; maximum acceptable contact force F, i.e., the estimated maximum contact force that the object can withstand; contact direction constraint, i.e., the recommended contact direction (e.g., the curtain is suitable to be pushed open to the side, not to be pulled down); semantic explanation text, i.e., a sentence explaining the reason for the risk score, such as "This is a hanging sheer curtain, the material is soft and light, pushing it open to the side will not cause any damage, and it belongs to an object with extremely low risk of accessibility."

[0083] Optionally, in embodiments of this application, for human or animal objects in the environment, the VLM model can always output a risk score of 1.0 (i.e., absolutely no human contact) and a maximum acceptable contact force F of 0, thereby ensuring human safety.

[0084] Optionally, in some other embodiments of this application, the inference frequency of the VLM model can be 1~5 Hz, that is, the semantic evaluation is updated once every 200~1000 ms, and the inference latency is determined by the computing power of the robot's graphics processing unit (GPU).

[0085] Subsequently, if the VLM model output (i.e., the contact risk assessment result in step S120 above) indicates that the robot cannot directly contact a certain object, a first path for the robot is determined and the robot is controlled to move along the first path. This first path avoids the object. For example, if the VLM model output indicates that the robot needs to avoid a glass and a human body in the scene, the first path will avoid both the glass and the human body.

[0086] Corresponding to step S140, in step S213, if the result output by the VLM model (i.e., the contact risk assessment result in step S120 above) indicates that the robot can directly contact an object, this embodiment of the application can generate a three-dimensional risk matrix. Specifically, this embodiment of the application can extend the discrete object-level risk score output by the VLM model to a continuous risk field R(x, y, z) (range from 0 to 1) covering the entire three-dimensional workspace.

[0087] For example, for each object identified by the VLM model, this embodiment of the application can generate a continuous three-dimensional risk field R(x, y, z) (i.e., the three-dimensional risk field in the aforementioned step S140) by using the three-dimensional occupied area and risk score of each object and a Gaussian diffusion model to diffuse the risk value to that area and its neighborhood. For example, for each object o identified by the VLM, let its center position be... The risk score is The risk diffusion radius is (Positively correlated with the size of the object), thus the risk value at point x. The following formula from the Gaussian diffusion model can be used for calculation:

[0088]

[0089] In the free space, the risk value naturally decays towards 0, meaning free passage is allowed. Therefore, this embodiment can determine the second path in step S150 based on the three-dimensional risk field R(x, y, z) obtained above. Step two below will describe the process of determining the second path in detail.

[0090] Optionally, in embodiments of this application, the three-dimensional risk field R(x, y, z) can be stored in a voxel grid (e.g., with a resolution of 2 cm) and aligned with the three-dimensional topological map space. After each output update of the VLM model, the updated three-dimensional risk matrix can be recalculated and published.

[0091] For example, Figure 4 This illustration shows a path planning method based on a three-dimensional risk field, as provided in an embodiment of this application. Figure 4The distribution of the three-dimensional risk field R(x, y, z) generated by different objects in a home corridor environment is shown in three-dimensional space. Figure 4 As shown, the scene uses a narrow family corridor as the environment, including objects such as a robot, curtains, cube-shaped obstacles, a table with vases, and standing figures. The risk rating for the curtain area is also included. =0.05, indicating extremely low risk (green), meaning the area is safe to pass through; Risk score for cube obstacles. =0.10, indicating low risk (light green), with a slight risk of collision; the vase on the table has a risk score of 0.10. =0.95, high risk (red), indicating a fragile item that will be damaged upon approach; a heavily restricted area; risk score for the standing person. =1.0, which is absolutely untouchable (dark red), representing the highest priority safety constraint, and robots are strictly prohibited from passing through it.

[0092] Among them, such as Figure 4 As shown, different objects serve as risk sources, represented by translucent spheres of varying radii and colors, showcasing their three-dimensional risk fields R(x, y, z). Larger sphere radii and a more reddish color indicate higher risk levels. Furthermore, robots can... Figure 4 The three-dimensional risk field R(x, y, z) shown determines the second path for the robot, starting from the bottom left corner and targeting the person on the right. Wherein, as... Figure 4 As shown, the robot can be controlled to start from the bottom left corner and move flexibly through the extremely low-risk curtain area. Then, when passing near low-risk cube obstacles, it will automatically avoid them. However, between the high-risk vase and the person, the path will strictly avoid the high-risk area, ensuring that it does not enter dangerous areas while efficiently completing the task of approaching the target person.

[0093] The following describes the semantic alignment motion planning in step S220. Corresponding to step S150, in step S220, the embodiments of this application can utilize the aforementioned step 213 and... Figure 4 The three-dimensional risk field shown is incorporated into the cost function of motion planning, enabling the motion planner to generate motion trajectories that accept moderate contact (i.e., the second path in step S150 above). Furthermore, this motion trajectory includes not only the pose sequence but also the expected contact intention (i.e., whether contact is expected) and expected stiffness parameters at each time step, forming a structure as follows: Figure 4 The complete trajectory and stiffness map is shown.

[0094] In step S221, this application constructs the planning problem, that is, constructs an augmented motion planning cost function. (That is, the aforementioned collision cost), the function is shown in the following formula:

[0095]

[0096] In the formula, The cost for trajectory smoothness can be determined by minimizing joint acceleration and jerk. For traditional geometric collisions, strict obstacle avoidance constraints are retained for high-risk objects with a risk score r > 0.8; The cost of achieving the goal; The semantically perceived contact cost can be expressed as the product of the semantic risk at the current location and the expected contact force magnitude, and its mathematical expression can be:

[0097]

[0098] For low-risk objects ( Approximately 0), even if contact force is expected, the cost approaches 0 (i.e., contact is permissible); for high-risk objects ( Approximately 1), any anticipated contact force would incur a huge cost (i.e., strict obstacle avoidance is required).

[0099] In addition, w1, w2, w3, and w4 are cost weights. These weights can be automatically selected by the task instructions through VLM semantic parsing, or they can be manually adjusted by the operator. For example, in navigation tasks, the weight of w3 can be increased to more aggressively traverse soft obstacles; for precision operation tasks, the weight of w2 can be increased.

[0100] For example, in a navigation traversal task mode such as navigating a corridor to reach the kitchen, w1 is 1 (track smoothness), w2 is 10.0 (high-risk collision cost, only effective for objects with r > 0.8), w3 is 0.5 (low contact cost weight, allowing low-risk contact), and w4 is 5. As another example, in a precision manipulation task mode such as carefully picking up a vase from a table, w1 is 2, w2 is 50 (extremely high collision cost, strict obstacle avoidance for all objects), w3 is 20.0 (extremely high contact cost weight, avoiding even low-risk objects), and w4 is 3.

[0101] In step S222, this embodiment of the application can input the robot's current state and target state, the semantic risk sequence along the candidate path, and the task-level semantic intent feature vector into the cross-attention neural network module. For example, the semantic risk sequence along the candidate path can be the semantic risk values ​​of M sampling points on the candidate path { , , ..., The risk feature sequence is encoded, and the task intent vector can be generated based on the language encoder of the task instructions and the VLM model. Furthermore, the attention weights between the risk feature sequence output by the cross-attention neural network module and the task intent can be obtained, and the collision cost of each spatial region can be dynamically adjusted. For example, when the task intent is to pass quickly, the attention mechanism reduces the collision cost weight of low-risk regions, allowing the planner to choose the shortcut through the curtain; when the task intent is to operate cautiously, the collision cost is maintained at a higher level even when facing low-risk objects.

[0102] In step S223, the second path output by the motion planner is not only a traditional end-effector pose trajectory, but may also include the expected contact intention (i.e., whether contact is expected) and expected stiffness parameters at each time step along the trajectory, forming a path as follows: Figure 4 The complete trajectory and stiffness map are shown. In embodiments of this application, the risk score can be mapped to the desired stiffness parameters (including translational and rotational directions) and damping parameters of the robot's end effector (or whole-body joints) in various directions in Cartesian space, such that each time step is accompanied by both the desired stiffness vector and the damping parameters.

[0103] For example, the mathematical expression for the stiffness mapping of the desired stiffness vector k is:

[0104]

[0105] In the formula, With minimum stiffness (e.g., 50 N / m), corresponding to a fully accessible area (R=0), the robotic arm behaves like a soft spring; With maximum stiffness (e.g., 2000 N / m), corresponding to an absolutely inaccessible area (R=1), the robotic arm exhibits rigid positioning.

[0106] For example, the damping parameter D can be calculated by coupling the critical damping condition with the stiffness, as shown in the following equation:

[0107]

[0108] In the formula, For the damping ratio, such as 0.7-1.0, preferably 0.85 (that is, slightly underdamped, providing moderate dynamic response); Let be the equivalent mass of the robotic arm in the i-th direction.

[0109] The following describes the execution of dynamic impedance control in step S230. The stiffness and damping parameters of the impedance controller can continuously and dynamically change along the trajectory, allowing the robot's robotic arm to move as smoothly as a spring when traversing soft obstacles and as precisely as a rigid body during precision operations.

[0110] In the operational space impedance control in step S231, the impedance controller can determine the control torque based on the reference trajectory output during motion planning, the expected stiffness parameters, and the damping parameters. The calculation formula is shown below:

[0111]

[0112] In the formula, To control the torque; For the Jacobian matrix of the robotic arm; and Let these be the desired stiffness and damping matrices. and The current reference trajectory point and reference velocity; and The current actual end-effector pose and velocity; This is the gravity compensation torque.

[0113] In step S232, during the determination of the contact crossing behavior, the controller operates at a frequency of at least 200 Hz to ensure the real-time performance and stability of the impedance behavior. Specifically, when the robotic arm moves along a low-risk path segment and makes expected contact with a soft obstacle, low stiffness (e.g., 50 N / m) allows the robotic arm to gently push away the obstacle like a spring, applying almost no destructive force. Furthermore, the robotic arm continues to advance along the reference trajectory while smoothly deflecting, ultimately smoothly pushing aside the curtain to cross. When the robotic arm approaches a high-risk object for precision operations, high stiffness ensures sub-millimeter positioning accuracy. Moreover, safety constraints (such as self-collision avoidance and joint limiting) have the highest priority and are never overridden by stiffness adjustments.

[0114] In the stiffness transition smoothing of step S233, when the robotic arm moves along the trajectory, such as from a low-risk area to a high-risk area (or vice versa), the stiffness parameter... It can change continuously along the trajectory without abrupt changes. For example, such as Figure 4 As shown, different road sections correspond to different stiffness coefficients k. For example, in the extremely low-risk curtain area, k=50, the stiffness is the lowest, and the robot can pass flexibly; when passing next to a low-risk cube obstacle, k=500, the stiffness is increased, and the path will automatically avoid the obstacle; while between the high-risk vase and the figure, k=2000, the stiffness reaches the highest, and the path will strictly avoid the high-risk area.

[0115] For example, in the stiffness transition smoothing mechanism of this application embodiment, when the stiffness parameter changes continuously along the trajectory, if the rate of change of stiffness exceeds a threshold, an additional first-order low-pass filter is applied to ensure smooth transition. This process is shown in the following equation:

[0116]

[0117] In the formula, For example, 0.2 is the filter coefficient. Here, Δt represents the desired stiffness parameter and Δt represents the control period. This filtering process prevents robot arm jitter or torque abrupt changes caused by sudden stiffness fluctuations.

[0118] In the safety priority assurance step S234, safety constraints are always executed with the highest priority. For example, in joint position limiting, if the joint angle is close to the physical limit, the control torque is limited to a repulsive torque away from the limiting direction. As another example, in self-collision avoidance, if a self-collision risk is detected, a repulsive component is injected into the control torque. Furthermore, in absolute human body area avoidance, the space occupied by the human body is subject to the highest stiffness and maximum safe distance constraints, regardless of how the semantic risk system assesses it. In addition, safety constraints can also achieve priority hierarchical implementation through null-space projection.

[0119] The contact feedback and continuous optimization process in step S240 is described below.

[0120] In step S241, during contact force monitoring and protection, the robot's force sensor or torque sensor continuously monitors the actual contact force vector F at at least 500 Hz. Then, the actual contact force can be compared in real time with the maximum acceptable contact force F of the object predicted by the VLM model. max Where, if |F max If | > 0.8 * F, the system enters a "contact force warning" state, appropriately increasing the local stiffness to reduce further contact depth; if | F max | > F, the system performs a protective retreat, increasing the impedance offset in the opposite direction of the contact force, and simultaneously triggering a re-evaluation of the VLM model.

[0121] In the online calibration of the VLM risk score in step S242, this embodiment of the application can compare the force-displacement characteristics measured during the actual contact process with the VLM risk score. For example, if the VLM model rates an object as low risk ( =0.1), but actual contact reveals that the object's stiffness is much higher than expected (e.g., a metal rod is hidden behind a curtain), thus the object's risk score can be increased and the semantic risk matrix updated immediately. For example, if the VLM model rates an object as medium risk ( The initial risk score was 0.5, but actual contact feedback indicated the object was extremely soft, allowing for a downward adjustment of the risk score to improve subsequent passage efficiency. The calibrated risk score was then fed back to the VLM model as a correction factor, enhancing the accuracy of subsequent assessments.

[0122] In the decision log recording in step S243, the system automatically writes an audit log each time a contact event or critical path decision is made. For example, it can record the trigger time and robot state, the semantic evaluation results of the VLM model (including explanatory text), the path and stiffness parameters selected by the planner, the actual contact force and contact results, and whether protective actions were triggered.

[0123] Example 2:

[0124] Taking a service robot navigating a curtained hallway in a home environment as an example, the user can issue a voice command such as "Go to the kitchen and get the water glass from the table." The robot's natural language processing module can then parse the command and set the navigation target to the kitchen tabletop.

[0125] Then, corresponding to steps S110 and S211, the robot's RGB-D camera scans the corridor environment, and instance segmentation detects the target objects. For example, it detects a hanging cotton curtain (obscuring 80% of the corridor width) on the left side of the corridor, a cardboard box on the right side of the corridor, and a glass of water on the right side of the corridor table. Afterwards, corresponding to steps S120 and S212, the VLM model can output a semantic risk assessment for each object. For example, the cotton curtain corresponds to... =0.08 and F=20 N, the semantic risk interpretation includes "soft fabric, safe for side push", corresponding to the cardboard box. =0.12, F=40 N, semantic risk interpretation includes "lightweight cardboard box, safe to push open", corresponding to glass water cup. =0.92, F=3 N, the semantic risk interpretation includes "glass is fragile and must be avoided".

[0126] Then, corresponding to steps S140 and S213, this embodiment of the application can generate a three-dimensional risk field. Wherein, reference... Figure 4 As shown, the curtain and cardboard box area can be light green (low risk), while the glass cup area is dark red (high risk), with a gradually spreading risk field around it. Furthermore, the semantically aligned motion planner can navigate through the task pattern path. Specifically, if the robot identifies the curtain as low-risk, it can generate a path to directly pass through it. Then, the robotic arm moves along the planned trajectory. Upon reaching the curtain, the impedance controller stiffness decreases. Due to the relatively small contact force of approximately 0.5N, the robotic arm produces a compliant offset of about 4mm, smoothly pushing open the curtain to pass through. At this point, the audit log can record the VLM evaluation results, the rationale for the path selection, the actual contact force (0.5N < 20N upper limit), and the task success.

[0127] Based on the above embodiments Figure 5 A schematic diagram illustrating the experimental results of robot path planning-based motion provided in an embodiment of this application is shown. Compared to Figure 1 The robot deadlock shown is shown. Figure 5 The robot shown can traverse low-risk, accessible door curtain areas in compliant mode (k=50 N / m). =0.05), and it only takes 3.2 seconds to reach the target area, such as Figure 4 The locations of the humans shown.

[0128] Specifically, when the robot pushes open the curtain, the VLM model can evaluate the plastic curtain as... =0.08, F=15N. The planner then generates a traversal path with a preset stiffness of 100 N / m. When the robotic arm contacts the curtain, if the force sensor detects that the actual contact force rapidly increases to 12N (approaching 80% of the upper limit), a contact force warning state can be entered, and the local stiffness is automatically increased from 100N / m to 300N / m to reduce further contact depth. If the contact force continues to rise to 16N > the 15N upper limit, it is determined that the actual object stiffness exceeds the VLM's expectation, and a protective retreat is immediately executed, such as increasing the impedance offset by 40mm in the opposite direction of the contact force, allowing the robotic arm to smoothly retreat and disengage from contact. Simultaneously, a VLM reassessment can be triggered, i.e., if the VLM model analysis discovers a metal hook stuck behind the curtain, the risk score is increased. =0.45. The updated semantic risk matrix reconstruction allows the motion planner to automatically calculate detour paths to avoid the area. This event can be fully logged to the audit log, including the original assessment, actual force values, protective actions, and the revised assessment.

[0129] Based on the above scheme, this application embodiment uses a VLM model to perform semantic risk assessment on environmental objects, enabling the motion planner to distinguish between accessible soft obstacles and inaccessible dangerous objects. In scenarios involving soft obstructions such as curtains and door curtains, the planner can choose to actively traverse them instead of detouring or getting stuck, significantly improving motion efficiency and task completion rate in unstructured environments. Furthermore, this application embodiment incorporates motion trajectory and impedance control parameters into the same planning framework, outputting complete trajectory and stiffness maps. The planner understands which path segments will involve contact and presets compliant parameters, and which path segments require precise positioning and preset stiffness parameters, achieving a deep integration of path planning and force control.

[0130] Furthermore, the impedance controller parameters in this embodiment can continuously and dynamically change along the trajectory. In low-risk areas (k approximately 50 N / m), the robotic arm behaves like a soft spring, safely pushing over soft obstacles. In high-risk areas (k approximately 2000 N / m), the robotic arm maintains rigid positioning accuracy, with a smooth transition between these two extreme behaviors in the same movement. Additionally, this embodiment provides dual safety assurance through real-time contact force monitoring and comparison with VLM (Vehicle Modeling Library) estimates. When the actual contact force exceeds expectations, the system automatically performs protective retreat and triggers a VLM reassessment to prevent damage to the object due to VLM assessment deviations.

[0131] Finally, the VLM model in this embodiment provides a natural language explanation for each risk score (e.g., "Choosing to open the curtains is because they are soft fabric"), and the system automatically generates structured audit logs to meet the requirements of safety compliance and legal accountability, which is particularly important in the field of consumer-facing service robots. Furthermore, this embodiment incorporates task semantic intent into planning decisions through a cross-attention mechanism. Different task instructions (e.g., passing quickly and approaching cautiously) in the same environment automatically generate different contact strategies, improving the system's flexibility in multi-task scenarios.

[0132] It is understood that the specific functions and corresponding technical effects of the path planning device provided in this embodiment can be referred to the explanation of the above embodiment. Figure 6 This diagram illustrates the module composition of a path planning device 600 provided in an embodiment of this application. Specifically, the acquisition module 610 is used to acquire a visual image of the robot in the forward direction, the visual image including the target object; the processing module 620 is used to determine the contact risk assessment result of the target object output by the VLM model based on the visual image and the visual-language VLM model, the contact risk assessment result being used to indicate the risk of the robot directly contacting the target object; the processing module 620 is specifically used to determine a first path for the robot and control the robot to move along the first path when the contact risk assessment result indicates that the robot cannot directly contact the target object, the first path avoiding the target object; the processing module 620 is specifically used to determine a three-dimensional risk field of the three-dimensional scene in which the robot is located, the three-dimensional scene including the target object, based on the contact risk assessment result when the contact risk assessment result indicates that the robot can directly contact the target object, the three-dimensional risk field being used to indicate the contact risk at different locations in the three-dimensional scene; the processing module 620 is specifically used to determine a second path for the robot based on the three-dimensional risk field and control the robot to move along the second path, the second path passing through the target object.

[0133] This application also provides a computer program product that, when run on an electronic device, causes the electronic device to execute the technical solutions described in the above embodiments. Its implementation principle and technical effects are similar to those of the related embodiments described above, and will not be repeated here.

[0134] This application also provides a readable storage medium containing instructions that, when executed by an electronic device, cause the electronic device to perform the technical solutions described in the above embodiments. The implementation principle and technical effects are similar and will not be repeated here.

[0135] This application also provides a chip for executing instructions. When the chip is running, it executes the technical solutions described in the above embodiments. Its implementation principle and technical effects are similar and will not be repeated here.

[0136] The following describes the hardware module of the robot of this application, which can be used to implement the aforementioned method 100 and Figures 3 to 5 The execution process is shown.

[0137] Now for reference Figure 7 The diagram shows a block diagram of a robot 700 according to one embodiment of this application. The robot 700 may include one or more processors 701 coupled to a controller hub 703. In at least one embodiment, the controller hub 703 communicates with the processors 701 via a multi-branch bus such as a front side bus (FSB), a point-to-point interface such as a quick path interconnect (QPI), or a similar connection 710. The processors 701 execute instructions controlling general types of data processing operations. In one embodiment, the controller hub 703 includes, but is not limited to, a graphics memory controller hub (GMCH) (not shown) and an input / output hub (IOH) (which may be on a separate chip) (not shown), wherein the GMCH includes memory and a graphics controller and is coupled to the IOH.

[0138] Robot 700 may also include a first coprocessor 702 coupled to a controller hub 703 and a memory 704. Alternatively, one or both of the memory and the GMCH may be integrated within the processor, with the memory 704 and the first coprocessor 702 directly coupled to the processor 701 and the controller hub 703, which resides on a single chip with the IOH. The memory 704 may be, for example, dynamic random access memory (DRAM), phase change memory (PCM), or a combination of both. In one embodiment, the first coprocessor 702 is a dedicated processor, such as, for example, a high-throughput many integrated core (MIC) processor, a network or communication processor, a compression engine, a graphics processor, a general-purpose computing on GPU (GPGPU), or an embedded processor, etc. Optional properties of the first coprocessor 702 are indicated by dashed lines. Figure 7 middle.

[0139] As a computer-readable storage medium, memory 704 may include one or more tangible, non-transitory computer-readable media for storing data and / or instructions. For example, memory 704 may include any suitable non-volatile memory such as flash memory and / or any suitable non-volatile storage device such as one or more hard-disk drives (HDD(s)), one or more compact disc (CD) drives, and / or one or more digital versatile disc (DVD) drives.

[0140] In one embodiment, robot 700 may further include a network interface controller (NIC) 706. NIC 706 may include a transceiver for providing a radio interface to robot 700, enabling communication with any other suitable device, such as a front-end module, antenna, etc. In various embodiments, NIC 706 may be integrated with other components of robot 700. NIC 706 can implement the functionality of the communication unit in the above embodiments.

[0141] Robot 700 may further include input / output (I / O) devices 705. I / O 705 may include: a user interface designed to enable a user to interact with robot 700; an interface designed to enable peripheral components to interact with robot 700; and / or sensors designed to determine environmental conditions and / or position information related to robot 700.

[0142] It is worth noting that, Figure 7 This is merely an example. That is, although... Figure 7 The diagram shows that the robot 700 includes multiple devices such as a processor 701, a controller hub 703, and a memory 704. However, in practical applications, devices using the methods of this application may include only a portion of the devices in the robot 700. For example, it may include only the processor 701 and the NIC 706. Figure 7 The properties of the optional devices are shown in dashed lines. According to some embodiments of this application, the memory 704, which is a computer-readable storage medium, stores instructions that, when executed on a computer, cause the robot 700 to perform the methods according to the above embodiments. Specific details can be found in the methods of the above embodiments, and will not be repeated here.

[0143] Now for reference Figure 8 The diagram shown is a block diagram of a SoC 800 according to an embodiment of this application. Figure 8 In the diagram, similar components share the same reference numerals. Additionally, dashed boxes are an optional feature for more advanced SoCs. Figure 8 In this SoC 800, the following are included: an interconnect unit 850 coupled to an application processor 810; a system proxy unit 880; a bus controller unit 890; an integrated memory controller unit 840; one or more second coprocessors 820, which may include integrated graphics logic, an image processor, an audio processor, and a video processor; a static random access memory (SRAM) unit 830; and a direct memory access (DMA) unit 860. In one embodiment, the second coprocessor 820 includes a dedicated processor, such as, for example, a network or communication processor, a compression engine, a GPGPU, a high-throughput MIC processor, or an embedded processor.

[0144] The static random-access memory (SRAM) cell 830 may include one or more computer-readable media for storing data and / or instructions. The computer-readable storage medium may store instructions, specifically, temporary and permanent copies of those instructions. These instructions may include, when executed by at least one unit in the processor, causing the SoC 800 to perform the path planning method according to the above embodiments, as detailed in the methods described above, which will not be repeated here.

[0145] Various embodiments of the mechanisms disclosed in this application can be implemented in hardware, software, firmware, or combinations of these implementation methods. Embodiments of this application can be implemented as computer programs or program code executable on a programmable system, the programmable system including at least one processor, a storage system (including volatile and non-volatile memory and / or storage elements), at least one input device, and at least one output device.

[0146] Program code can be applied to input instructions to execute the functions described in this application and generate output information. The output information can be applied to one or more output devices in a known manner. For the purposes of this application, the processing system includes any system having a processor such as, for example, a digital signal processor (DSP), a microcontroller, an application-specific integrated circuit (ASIC), or a microprocessor.

[0147] The program code can be implemented using a high-level procedural language or an object-oriented programming language to communicate with the processing system. Assembly language or machine language can also be used when needed. In fact, the mechanisms described in this application are not limited to any particular programming language. In either case, the language can be a compiled language or an interpreted language.

[0148] In some cases, the disclosed embodiments may be implemented in hardware, firmware, software, or any combination thereof. The disclosed embodiments may also be implemented as instructions carried or stored thereon on one or more temporary or non-temporary machine-readable (e.g., computer-readable) storage media, which may be read and executed by one or more processors. For example, the instructions may be distributed via a network or through other computer-readable media. Therefore, machine-readable media may include any mechanism for storing or transmitting information in a machine-readable (e.g., computer-readable) form, including but not limited to floppy disks, optical disks, CD-ROMs, compact disc read-only memory (CD-ROMs), magneto-optical disks, read-only memory (ROM), random-access memory (RAM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), magnetic cards or optical cards, flash memory, or tangible machine-readable storage for transmitting information (e.g., carrier waves, infrared signals, digital signals, etc.) using the Internet in the form of electrical, optical, acoustic, or other forms of propagated signals. Therefore, machine-readable media includes any type of machine-readable medium suitable for storing or transmitting electronic instructions or information in a machine-readable (e.g., computer-readable) form.

[0149] In the accompanying drawings, some structural or methodological features may be shown in a specific arrangement and / or order. However, it should be understood that such a specific arrangement and / or order may not be necessary. Rather, in some embodiments, these features may be arranged in a manner and / or order different from that shown in the accompanying drawings. Furthermore, including structural or methodological features in a particular figure does not imply that such features are required in all embodiments, and in some embodiments, these features may be omitted or may be combined with other features.

[0150] It should be noted that all units / modules mentioned in the device embodiments of this application are logical units / modules. Physically, a logical unit / module can be a physical unit / module, a part of a physical unit / module, or a combination of multiple physical units / modules. The physical implementation of these logical units / modules themselves is not the most important factor; the combination of functions implemented by these logical units / modules is the key to solving the technical problems proposed in this application. Furthermore, to highlight the innovative aspects of this application, the above-described device embodiments of this application have not introduced units / modules that are not closely related to solving the technical problems proposed in this application. This does not mean that the above-described device embodiments do not contain other units / modules.

[0151] It should be noted that in the examples and description of this patent, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one" does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0152] Although this application has been illustrated and described with reference to certain preferred embodiments thereof, those skilled in the art should understand that various changes in form and detail may be made thereto without departing from the spirit and scope of this application.

Claims

1. A path planning method for a robot, characterized in that, Applications in robots, including: Acquire a visual image of the robot in the forward direction, the visual image including the target object; Based on the visual-language VLM model and the visual image, the contact risk assessment result of the target object output by the VLM model is determined, and the contact risk assessment result is used to indicate the risk of the robot directly contacting the target object; If the contact risk assessment result indicates that the robot cannot directly contact the target object, a first path for the robot is determined and the robot is controlled to move along the first path, which avoids the target object. If the contact risk assessment result indicates that the robot can directly contact the target object, a three-dimensional risk field is determined based on the contact risk assessment result. The three-dimensional scene includes the target object, and the three-dimensional risk field is used to indicate the contact risk at different locations in the three-dimensional scene. Based on the three-dimensional risk field, a second path for the robot is determined and the robot is controlled to move along the second path, which passes through the target object; Determining the robot's second path based on the three-dimensional risk field further includes: Based on the three-dimensional risk field, determine the contact risk score for each sampling point on the second path; Based on the contact risk score at each sampling point on the second path, the expected stiffness parameter and expected contact intention for each time step of the second path are determined. The expected stiffness parameter is used to indicate the expected stiffness of the robot's end effector in each direction of Cartesian space at each time step.

2. The method according to claim 1, characterized in that, Determining the robot's second path based on the three-dimensional risk field includes: Multiple candidate paths for the robot are obtained, and a contact risk score for each sampling point on the multiple candidate paths is determined based on the three-dimensional risk field. Obtain the robot's task intent; Based on the contact risk score of each sampling point on the multiple candidate paths and the task intent, the collision cost of each spatial region in the three-dimensional scene is determined. The collision cost is used to indicate the severity of the robot's collision in the spatial region of the three-dimensional scene. Based on the collision cost, the second path is determined from the plurality of candidate paths, wherein the second path includes a spatial region in the three-dimensional scene where the collision cost is less than a preset cost threshold.

3. The method according to claim 1, characterized in that, The step of determining the expected stiffness parameter for each time step of the second path based on the contact risk score of each sampling point on the second path includes: Obtain the minimum and maximum stiffness along the second path; Based on the contact risk score of the current sampling point, the minimum stiffness, and the maximum stiffness, the expected stiffness parameter for each time step of the second path is determined, wherein the expected stiffness parameter is the sum of the minimum stiffness and the product of the contact risk score and the stiffness difference corresponding to each time step, and the stiffness difference is the difference between the maximum stiffness and the minimum stiffness.

4. The method according to claim 1 or 2, characterized in that, Controlling the robot to move along the second path includes: During the movement of the robot, the stiffness and damping parameters of the robot are adjusted according to the trajectory changes of the second path.

5. The method according to claim 4, characterized in that, The contact risk assessment result includes the maximum contactable force of the target object, and the adjustment of the robot's stiffness and damping parameters based on the trajectory changes of the second path includes: The real-time contact force is obtained during the robot's movement along the second path; Based on the comparison between the maximum accessible force and the real-time contact force, the stiffness and damping parameters of the robot are adjusted. Specifically, when the real-time contact force is greater than the product of the maximum accessible force and a preset coefficient, the stiffness of the robot is increased. The preset coefficient is a positive number less than 1. When the real-time contact force is greater than the maximum accessible force, the impedance offset is increased in the opposite direction of the real-time contact force.

6. The method according to claim 1 or 2, characterized in that, The step of determining the three-dimensional risk field of the three-dimensional scene in which the robot is located based on the contact risk assessment results includes: Obtain the center position of the target object and the risk diffusion radius related to the size of the target object; Based on the central location, the risk diffusion radius, and the contact risk assessment results, a Gaussian diffusion model is used to diffuse the contact risk assessment results of the target object to the occupied area of ​​the target object and its neighborhood, thereby obtaining the three-dimensional risk field.

7. The method according to claim 1 or 2, characterized in that, The contact risk assessment results include at least one of the following: object category, material inference, risk score, maximum contact force, and contact direction constraint of the target object.

8. The method according to claim 1 or 2, characterized in that, The robot's safety constraints include at least one of joint limiting, self-collision avoidance, and human area avoidance.

9. A path planning device for a robot, characterized in that, include: The acquisition module is used to: acquire a visual image of the robot in the forward direction, wherein the visual image includes the target object; The processing module is configured to: determine the contact risk assessment result of the target object output by the visual-language VLM model and the visual image, the contact risk assessment result being used to indicate the risk of the robot directly contacting the target object; The processing module is specifically used to: when the contact risk assessment result indicates that the robot cannot directly contact the target object, determine a first path for the robot and control the robot to move along the first path, the first path avoiding the target object; The processing module is specifically used to: when the contact risk assessment result indicates that the robot can directly contact the target object, determine the three-dimensional risk field of the three-dimensional scene in which the robot is located based on the contact risk assessment result, wherein the three-dimensional scene includes the target object, and the three-dimensional risk field is used to indicate the contact risk at different locations in the three-dimensional scene; The processing module is specifically used to: determine the second path of the robot based on the three-dimensional risk field and control the robot to move along the second path, wherein the second path passes through the target object; The processing module is specifically used to: determine the contact risk score of each sampling point on the second path based on the three-dimensional risk field; Based on the contact risk score at each sampling point on the second path, the expected stiffness parameter and expected contact intention for each time step of the second path are determined. The expected stiffness parameter is used to indicate the expected stiffness of the robot's end effector in each direction of Cartesian space at each time step.

10. A robot, characterized in that, include: At least one memory and at least one processor, the memory being coupled to the processor; the memory being used to store computer program code / instructions; when the computer program code / instructions are executed by the processor, causing the robot to implement the path planning method as described in any one of claims 1 to 8.

11. A readable storage medium, characterized in that, The readable storage medium stores instructions that, when executed on an electronic device, cause the electronic device to perform the path planning method as described in any one of claims 1 to 8.

12. A computer program product, characterized in that, include: Computer instructions, when executed on an electronic device, cause the electronic device to perform the path planning method as described in any one of claims 1 to 8.

Images