Control method, device, apparatus and storage medium of robot device

Abstract

The application provides a control method and device of a robot device, equipment and a storage medium. The method comprises: acquiring distance information between a target object and the robot device, and risk partition information of the robot device, wherein the risk partition information comprises ranges of at least two risk partitions and risk levels corresponding to each risk partition; determining a target risk partition where the target object is located according to the distance information and the risk partition information; and performing operation control on the robot device according to the target risk partition. Thus, the robot device can be controlled according to different risk partitions where the target object is located, the degree of refinement of the operation control of the robot device is improved, unnecessary shutdown in a low-risk situation is reduced, and corresponding control measures are taken in time when the target object enters a higher risk partition, so that the operation safety and work efficiency of the robot device are taken into account.

Description

Technical Field

[0001] This application relates to the field of robot safety control technology, specifically to a control method, device, equipment, and storage medium for a robot device. Background Technology

[0002] With the development of robotics technology, robotic equipment is widely used in industrial manufacturing, warehousing and logistics, inspection services, collaborative operations, and other automation scenarios. During operation, robotic equipment may be in adjacent spaces with people, vehicles, workpieces, other robots, or obstacles. Therefore, it is necessary to detect target objects around the robotic equipment and control its operation based on the detection results to reduce the risk of collisions or interference between the target objects and the robotic equipment.

[0003] In related technologies, the distance between the target object and the robot is typically obtained, and the robot is controlled to decelerate or stop when this distance is less than a preset distance threshold. While this method can improve the operational safety of the robot to some extent, it relies primarily on a single distance threshold for control, making it difficult to distinguish the risk levels corresponding to different levels of proximity.

[0004] In practical applications, the level of risk varies at different distances when a target object approaches the robot. If the robot is stopped as soon as the target object enters the low-risk zone, it can cause unnecessary downtime and affect work efficiency; if the robot is only stopped when it is very close to the target object, it may result in untimely safety control. Summary of the Invention

[0005] This application provides a control method, device, equipment, and storage medium for robot equipment, which solves the problem in related technologies that robot equipment is mainly controlled based on a single distance threshold, resulting in coarse control granularity and difficulty in balancing operational safety and work efficiency.

[0006] Firstly, a method for controlling a robot device is provided, comprising:

[0007] Obtain distance information between the target object and the robot device, as well as risk zoning information of the robot device, wherein the risk zoning information includes the range of at least two risk zoning areas and the risk level corresponding to each risk zoning area;

[0008] Based on the distance information and the risk zoning information, the target risk zoning where the target object is located is determined;

[0009] The robot equipment is operated and controlled according to the target risk zone.

[0010] Optionally, obtaining the distance information between the target object and the robot device includes:

[0011] Obtain the obstacle point cloud corresponding to the target object;

[0012] Obtain the safety case corresponding to the robot device;

[0013] Determine the closest distance between the obstacle point cloud and the containment structure;

[0014] The nearest distance is used as the distance information between the target object and the robot device.

[0015] Optionally, obtaining the obstacle point cloud corresponding to the target object includes:

[0016] Obtain the environmental point cloud of the area surrounding the robot device;

[0017] The local region of interest corresponding to the target object is determined based on the environmental point cloud.

[0018] The point cloud of the target object is obtained by removing the point cloud of the robot device itself from the point cloud corresponding to the local region of interest.

[0019] The point cloud of the target object is used as the point cloud of the obstacle.

[0020] Optionally, obtaining the distance information between the target object and the robot device includes:

[0021] Obtain a depth map of the area surrounding the robot device;

[0022] Determine the depth distance information corresponding to the target object based on the depth map;

[0023] The depth distance information is used as the distance information between the target object and the robot device.

[0024] Optionally, determining the depth distance information corresponding to the target object based on the depth map includes:

[0025] Candidate image regions are determined based on the depth values ​​in the depth map;

[0026] The object corresponding to the candidate image region is determined as the target object;

[0027] Based on the depth information corresponding to the candidate image region, the depth distance information corresponding to the target object is determined.

[0028] Optionally, determining the candidate image region based on the depth value in the depth map includes:

[0029] Image regions that meet at least one of the following conditions are identified as candidate image regions:

[0030] The minimum depth in the image region is less than or equal to a first distance threshold;

[0031] The depth of the low percentile in the image region is less than or equal to the second distance threshold;

[0032] The number of pixels in the image region whose depth is less than or equal to the third distance threshold is greater than or equal to the number threshold;

[0033] The area of ​​a connected region in the image region whose depth is less than or equal to the fourth distance threshold is greater than or equal to the area threshold.

[0034] The rate of depth change in the image region is greater than or equal to a velocity threshold.

[0035] Optionally, after determining the initial risk zone where the target object is located based on the depth distance information and the risk zone information, the method further includes:

[0036] If the risk level corresponding to the initial risk partition meets the preset risk conditions, a local region of interest containing the target object is determined based on the depth map;

[0037] Based on the depth information corresponding to the local region of interest, a point cloud of the target object is generated.

[0038] Obtain the safety case corresponding to the robot device;

[0039] Determine the closest distance between the target object point cloud and the containment structure;

[0040] Based on the nearest distance and the risk zoning information, the target risk zoning where the target object is located is determined.

[0041] Optionally, determining the local region of interest containing the target object based on the depth map includes:

[0042] Based on the depth pixel distribution of the target object in the depth map, the candidate region corresponding to the target object is determined;

[0043] The candidate region is determined as the local region of interest; or, the candidate region is expanded according to at least one of the following: the position of the target object in the candidate region, the depth pixel distribution corresponding to the target object, and the distance between the target object and the boundary of the candidate region, to obtain the local region of interest.

[0044] Optionally, generating the target object point cloud corresponding to the target object based on the depth information corresponding to the local region of interest includes:

[0045] A 3D point cloud transformation is performed on the depth pixels within the local region of interest to obtain a local point cloud. Based on at least one of the geometric model of the robot device, the pose information of the robot device, and the safety shell corresponding to the robot device, the point cloud of the robot device itself is removed from the local point cloud to obtain the point cloud of the target object.

[0046] Optionally, the safety containment is a dynamic safety containment determined based on the current state of the robotic device.

[0047] Optionally, the dynamic containment structure is determined in the following ways:

[0048] Obtain the device model information and current status information of the robot device;

[0049] Based on the device model information and the current state information, the real-time spatial range of the robot device is determined;

[0050] The dynamic safety shell is determined based on the real-time spatial range and the preset safety margin.

[0051] Optionally, determining the closest distance between the obstacle point cloud and the containment, or determining the closest distance between the target object point cloud and the containment, includes:

[0052] Determine the distances from multiple points in the target point cloud to the containment structure, wherein the target point cloud is the obstacle point cloud or the target object point cloud;

[0053] The closest distance is determined based on the distances from the plurality of points to the containment vessel.

[0054] Optionally, determining the closest distance based on the distances from the plurality of points to the containment vessel includes:

[0055] A distance set is determined based on the distances from the plurality of points to the containment structure;

[0056] The minimum distance in the set of distances is determined as the nearest distance; or,

[0057] The distance set is subjected to abnormal distance removal and / or valid point filtering. Based on the minimum distance, the average distance of the nearest multiple points, the low percentile distance, the nearest valid cluster distance, or the distance confirmed after multiple consecutive frames in the filtered distance set, a distance characterization value is determined and the distance characterization value is used as the nearest distance.

[0058] Optionally, the at least two risk zones include at least two of the following: a warning zone, a deceleration zone, and a stop zone;

[0059] The risk level of the warning zone is lower than that of the deceleration zone, and the risk level of the deceleration zone is lower than that of the stop zone.

[0060] Optionally, the step of controlling the operation of the robot device according to the target risk zone includes:

[0061] If the target risk zone is a warning zone, the robot device is controlled to maintain its current operating state and output a warning message;

[0062] When the target risk zone is a deceleration zone, the original operating state of the robot equipment is adjusted to obtain an operating state that reduces the risk of collision.

[0063] If the target risk zone is a stop zone, control the robot to stop running, pause running, or maintain its current position.

[0064] Optionally, adjusting the original operating state of the robot device to obtain an operating state that reduces the risk of collision includes:

[0065] Obtain the orientation information of the target object relative to the robot device;

[0066] Based on the distance information and the orientation information, determine the direction and / or amount of operation adjustment;

[0067] The original operating state of the robot device is adjusted according to the operating adjustment direction and / or the operating adjustment amount to obtain the adjusted operating state.

[0068] Optionally, determining the running adjustment direction and / or running adjustment amount based on the distance information and the orientation information includes:

[0069] Based on the orientation information, determine the direction of the robot device to move away from the target object;

[0070] The operational adjustment amount related to the distance to the target object is determined based on the distance information, wherein the operational adjustment amount is negatively correlated with the distance value corresponding to the distance information.

[0071] Optionally, adjusting the original operating state of the robot device according to the operating adjustment direction and / or the operating adjustment amount includes:

[0072] Based on the said running adjustment direction and / or the said running adjustment amount, determine at least one of the following: trajectory correction amount, velocity correction amount, acceleration correction amount, end pose correction amount, joint position correction amount, and joint velocity correction amount;

[0073] The original operating state is adjusted based on at least one of the trajectory correction amount, the velocity correction amount, the acceleration correction amount, the end pose correction amount, the joint position correction amount, and the joint velocity correction amount.

[0074] Optionally, after obtaining the adjusted operating state, and before controlling the robot device to operate according to the operating state to be executed, the method further includes:

[0075] Determine whether the adjusted operating state meets the operating restrictions of the robot device;

[0076] If the operating restrictions are met, the adjusted operating state is determined as the pending operating state of the robot device, and the robot device is controlled to operate according to the pending operating state.

[0077] If the operating restrictions are not met, the operating speed of the robot device will be reduced, or the robot device will be controlled to stop operating.

[0078] Optionally, the method further includes:

[0079] When the target object is within the warning area, a warning message is output, which is used to indicate that the target object is in the warning area.

[0080] In a second aspect, a control device for a robot is provided, characterized in that it comprises:

[0081] The acquisition module is used to acquire distance information between the target object and the robot device, as well as risk zoning information of the robot device. The risk zoning information includes the range of at least two risk zoning areas and the risk level corresponding to each risk zoning area.

[0082] The determination module is used to determine the target risk zone where the target object is located based on the distance information and the risk zone information;

[0083] The control module is used to control the operation of the robot equipment according to the target risk zone.

[0084] Thirdly, a robotic device is provided, including a processor, a memory, and a program or instructions stored in the memory and executable on the processor, wherein the program or instructions, when executed by the processor, implement the steps of the method described in the first aspect.

[0085] Fourthly, embodiments of this application provide a readable storage medium storing a computer program, the computer program including program instructions, which, when executed by a computer, implement the steps of the method as described in the first aspect.

[0086] Fifthly, embodiments of this application provide a computer program product, including a computer program or instructions, wherein when the computer program or instructions are executed, they implement the steps of the method involved in the first aspect described above.

[0087] In this application, distance information between the target object and the robot device, as well as the robot device's risk zone information, are obtained. Based on these information, the target risk zone where the target object is located is determined, and then the robot device is controlled according to this risk zone. Since the risk zone information includes the range of at least two risk zones and the corresponding risk level for each zone, the robot device can implement different levels of operational control based on the different risk zones the target object is located in. Compared to directly controlling the robot device to decelerate or stop based solely on a single distance threshold, this application can distinguish the risk level corresponding to the target object at different distance ranges. This allows the robot device to employ lighter control measures in low-risk approach states and stronger control measures in high-risk approach states, thereby improving the precision of operational control. Attached Figure Description

[0088] Figure 1 This is a flowchart illustrating a control method for a robot device provided in an embodiment of this application;

[0089] Figure 2 This is a schematic diagram of the risk partitioning provided in the embodiments of this application;

[0090] Figure 3 This is one of the flowcharts illustrating a method for generating risk zones for a robotic arm, as provided in an embodiment of this application.

[0091] Figure 4 This is a second schematic flowchart of a method for generating risk zones for a robotic arm, provided in an embodiment of this application.

[0092] Figure 5 This is a schematic diagram of the structure of a control device for a robot device provided in an embodiment of this application;

[0093] Figure 6 This is a schematic diagram of the structure of the robot device provided in the embodiments of this application. Detailed Implementation

[0094] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0095] The term "comprising," and any variations thereof, used in the specification and claims of this application, is intended to cover a non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not necessarily limited to those explicitly listed, but may include other steps or units not explicitly listed or inherent to such process, method, product, or apparatus. Furthermore, the use of "and / or" in the specification and claims indicates at least one of the connected objects, such as A and / or B, indicating the inclusion of A alone, B alone, or both A and B.

[0096] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design that is described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design. Specifically, the use of the terms "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0097] See Figure 1 This application provides a control method for a robot device, the specific steps of which include:

[0098] Step 11: Obtain the distance information between the target object and the robot device, as well as the risk zoning information of the robot device, wherein the risk zoning information includes the range of at least two risk zoning areas and the risk level corresponding to each risk zoning area.

[0099] In one possible implementation, the robotic device includes at least one of a fixed robotic arm, a collaborative robotic arm, an industrial robotic arm, a mobile robot, a robotic arm mounted on a mobile chassis, a unibody intelligent robot, an automated mobile robot, a service robot, an inspection robot, a warehouse robot, and other robotic devices having motion actuators or capable of moving within a workspace.

[0100] In one possible implementation, the target object may include personnel, vehicles, other robots, workpieces, tools, obstacles, animals, or other objects entering the environment surrounding the robotic device. The target object can be an external object determined through image recognition, depth detection, point cloud segmentation, radar detection, target tracking, or multi-sensor fusion, or it can be a candidate obstacle determined based on depth maps, point clouds, or distance sensor data. That is, the target object in this application is not limited to a specific object that has already undergone category identification; as long as it may affect the operational safety of the robotic device, it can participate in subsequent risk zoning judgment as a target object.

[0101] In one possible implementation, distance information is used to represent the spatial proximity between a target object and a robotic device. This distance information can be the distance between the target object and the robot body, the robot end effector, the robot's moving parts, the containment vessel, a dynamic containment vessel, a preset reference point, or a preset reference surface. The distance information can be one-dimensional depth distance, two-dimensional planar distance, three-dimensional Euclidean distance, the closest distance from a point cloud to the containment vessel, the closest distance after statistical filtering, the distance confirmed after multiple consecutive frames, or other distance results that can characterize the proximity of the target object. Therefore, the distance information is not limited to the distance from the target object to the geometric center of the robotic device, nor is it limited to the distance value directly output by a single sensor.

[0102] In one possible implementation, risk zoning information is used to represent the division of areas around the robot device into different risk levels. Risk zoning information may include the extent of at least two risk zones and the corresponding risk level for each zone. The extent of a risk zone can be represented by a distance boundary, spatial region, 3D containment enclosure extension, bounding box, bounding sphere, capsule, mesh extension, voxel region, 2D image region, or 3D point cloud region. The risk level indicates the degree of risk of contact, interference, or collision between the target object and the robot device after the target object enters the corresponding risk zone. A higher risk level generally indicates that the target object is closer to the robot device, or that the robot device requires stronger safety control measures in its current operating state.

[0103] In one possible implementation, the risk zoning information can be pre-configured or dynamically determined based on the robot's model, load information, end-effector information, operating speed, braking capability, current direction of motion, current pose, environmental perception results, and / or user configuration. The risk zoning information can also be generated by the robot itself, or generated by a host computer, edge computing device, safety controller, or other external device and then sent to the robot. Therefore, this application does not limit the entity or method of generating the risk zoning information.

[0104] This approach links the actual spatial proximity between the target object and the robotic device with the risk zone information currently used by the robotic device, providing a basis for subsequent determination of the target object's risk zone and execution of tiered operational control. Compared to simply stopping the robotic device based on a fixed distance threshold, this application enables more granular control based on different risk levels, which is beneficial for balancing operational safety and work efficiency.

[0105] Step 12: Determine the target risk zone where the target object is located based on the distance information and the risk zone information.

[0106] In one possible implementation, distance information can be compared with the range of each risk zone in the risk zone information to determine the target risk zone where the target object is located. For example, when the risk zone information includes a warning zone, a deceleration zone, and a stop zone, the distance information between the target object and the robot device can be used to determine whether the target object is located outside the warning zone, deceleration zone, stop zone, or risk zone.

[0107] In one possible implementation, when risk zones are represented by distance boundaries, distance information can be compared with the boundaries of each risk zone. When the distance information falls within the distance range corresponding to a certain risk zone, that risk zone is identified as the target risk zone. When risk zones are represented by spatial regions, it can be determined whether the target object falls into the corresponding risk zone based on the location of the target object, the positional relationship between the point cloud or local region and the spatial region.

[0108] In one possible implementation, if a target object simultaneously meets the judgment criteria of multiple risk zones, the risk zone with the higher risk level can be selected as the target risk zone. If multiple target objects exist, the risk zone where each target object is located can be determined separately, and the final target risk zone used to control the robot device can be determined based on the highest risk level, closest distance, maximum approach speed, or target object type.

[0109] By implementing this method, distance detection results can be converted into clear risk zoning results, enabling robotic devices to execute different control strategies based on the different risk zones where the target object is located, thereby avoiding the use of the same treatment method for all approach situations.

[0110] Step 13: Control the operation of the robot equipment according to the target risk zone.

[0111] In one possible implementation, the robotic device can perform operational controls such as alerting, speed limiting, deceleration, trajectory adjustment, partial obstacle avoidance, pausing, stopping, maintaining the current position, or safety braking based on the risk level corresponding to the target risk zone. Different risk zones can correspond to different control intensities. When the risk level is low, an alert message can be output or a slight speed limit can be implemented; when the risk level is high, the speed can be reduced, acceleration limited, or the operating state adjusted; when the risk level further increases, the robotic device can be controlled to stop operation, pause operation, or maintain the current position.

[0112] In one possible implementation, operational control can be applied to the robot's end effector speed, joint speed, chassis speed, trajectory execution speed, acceleration, end effector pose, joint position, joint speed, or task execution state. Operational control can also be transmitted to the robot controller via a controller application programming interface (API), fieldbus, industrial Ethernet, robot operating system interface, digital signals, or a host computer control interface.

[0113] Implementing this method allows for tiered control of robotic equipment based on the risk zone where the target object is located. This enables the robotic equipment to first warn, then slow down or avoid the target object as it gradually approaches, and finally stop, thereby reducing the risk of collisions and minimizing unnecessary downtime.

[0114] In one possible implementation, obtaining the distance information between the target object and the robot device includes: obtaining an obstacle point cloud corresponding to the target object; obtaining a safety shell corresponding to the robot device; determining the closest distance between the obstacle point cloud and the safety shell; and using the closest distance as the distance information between the target object and the robot device.

[0115] Candidate obstacles can be external objects that may affect the safe operation of the robot, determined based on depth maps, point clouds, image recognition results, radar detection results, or multi-sensor fusion results. The obstacle point cloud represents the point distribution of candidate obstacles in three-dimensional space. The safety housing represents the safe reference range outside the robot body, moving parts, end effector, or load. This safety housing can be a fixed safety housing or a dynamic safety housing that changes with the robot's current pose, current speed, load, or end effector state.

[0116] The closest distance can be understood as the minimum spatial distance between the obstacle point cloud and the containment, or it can be the result of statistical processing of the closest distance. This closest distance is not limited to the distance from a single point to a single geometric center, but can be the distance from multiple points in the point cloud to the containment after statistical analysis of the minimum, average, median, lower percentile, effective clusters, or confirmed by multiple consecutive frames.

[0117] This method allows for the direct calculation of the spatial distance between candidate obstacles and the robot's safety housing, making the distance information more closely reflect the robot's actual shape and range of motion. Compared to simply calculating the distance between the target object and the robot's center point, this method more accurately reflects whether the target object is close to the robot's hazardous area.

[0118] In this embodiment, the obstacle point cloud can be a representation of the target object point cloud; the target object point cloud is used to represent the point cloud distribution of the target object in three-dimensional space; the candidate external obstacle point cloud is used to represent the external object point cloud that may affect the safe operation of the robot after removing the robot's own point cloud; the target point cloud is a point cloud used to calculate the distance with the containment structure, and it can be either the obstacle point cloud or the target object point cloud.

[0119] In one possible implementation, obtaining the obstacle point cloud corresponding to the target object includes: obtaining the environmental point cloud of the area surrounding the robot device; determining the local region of interest corresponding to the target object based on the environmental point cloud; removing the robot device's own point cloud from the point cloud corresponding to the local region of interest to obtain the target object point cloud; and using the target object point cloud as the obstacle point cloud.

[0120] The environmental point cloud represents the three-dimensional spatial distribution of object surfaces in the area surrounding the robot. It can be obtained by backprojection of a depth map or directly from LiDAR, structured light cameras, time-of-flight depth cameras, multi-view cameras, or multi-sensor fusion. The local region of interest (ROI) represents a local spatial region or corresponding image region in the environmental point cloud where candidate obstacles may exist. This ROI can be determined based on the nearest depth point, nearby connected regions, object detection boxes, the projection area along the robot's motion direction, the area near key links, or the results of multi-strategy fusion.

[0121] The robot's own point cloud is used to represent the point cloud corresponding to the robot body, links, joints, end effector, or load. This self-point cloud can be determined based on the robot model, current joint angles, current pose, sensor extrinsic parameters, body mask, or containment. By removing the robot's own point cloud from the point cloud corresponding to the local region of interest, misidentification of the robot's own structure as external obstacles can be avoided.

[0122] This approach first identifies local regions of interest (ROIs) from the environmental point cloud that may contain candidate obstacles, then removes the robot's own point cloud, thereby reducing the overhead of processing the entire point cloud and improving the accuracy of the target object's point cloud. This method can handle both the initial coarse-grained judgment at high frame rates and the subsequent fine-grained judgment process, while also reducing the risk of false detections caused by the robot's own structure.

[0123] In one possible implementation, obtaining the distance information between the target object and the robot device includes: obtaining a depth map of the area surrounding the robot device; determining the depth distance information corresponding to the target object based on the depth map; and using the depth distance information as the distance information between the target object and the robot device.

[0124] The depth map represents the depth value corresponding to each pixel in the image. The depth value reflects the distance of the corresponding environmental point relative to a depth sensor, robotic device, or unified coordinate system. The depth map can be obtained from red-green-blue depth cameras, structured light depth cameras, time-of-flight depth cameras, binocular cameras, multi-view cameras, monocular depth estimation algorithms, panoramic depth estimation algorithms, or point cloud projection. Depth distance information can be the minimum depth, average depth, low percentile depth, near-connected region depth, or depth result after multi-frame filtering of the image region containing the target object.

[0125] This approach allows for the rapid acquisition of distance information between the target object and the robot without immediately performing point cloud back projection and 3D safe house distance calculations. It is suitable for scenarios requiring high frame rate risk assessment, rapid alerts, or low computing power.

[0126] In one possible implementation, determining the depth distance information corresponding to the target object based on the depth map includes: determining a candidate image region based on the depth value in the depth map; determining the object corresponding to the candidate image region as the target object; and determining the depth distance information corresponding to the target object based on the depth information corresponding to the candidate image region.

[0127] Candidate image regions are used to represent areas in the depth map where nearby objects may exist. Candidate image regions can be determined based on depth values, depth variations, the number of nearby pixels, the area of ​​connected regions, object detection results, or image segmentation results. Objects in candidate image regions can be people, vehicles, other robots, workpieces, tools, or other obstacles, or they can be candidate targets identified as potentially affecting the safety of robotic equipment when the category is not yet clearly defined.

[0128] By implementing this method, candidate image regions that may contain target objects can be screened out in the depth map first, and then the distance information can be determined based on the depth information of the region. This avoids complex processing of the entire depth map and helps to improve the real-time performance of target object distance judgment.

[0129] In one possible implementation, determining candidate image regions based on depth values ​​in the depth map includes: identifying image regions that satisfy at least one of the following conditions as candidate image regions: the minimum depth in the image region is less than or equal to a first distance threshold; the low percentile depth in the image region is less than or equal to a second distance threshold; the number of pixels in the image region with depths less than or equal to a third distance threshold is greater than or equal to a number threshold; the area of ​​connected regions in the image region with depths less than or equal to a fourth distance threshold is greater than or equal to an area threshold; and the depth change rate in the image region is greater than or equal to a speed threshold.

[0130] Among these, minimum depth can be used to quickly detect the nearest target; low percentile depth can be used to reduce the impact of individual noise points on the judgment result; the number of nearby pixels and the area of ​​nearby connected regions can be used to determine whether the nearby target has a certain spatial scale; and the depth change rate can be used to determine whether the target object is rapidly approaching the robot. These conditions can be used individually or in combination. The first distance threshold, second distance threshold, third distance threshold, number threshold, area threshold, and speed threshold can be determined based on the robot model, operating speed, braking capability, sensor accuracy, and the safety level of the operating scenario.

[0131] By implementing this method, candidate image regions can be determined through various depth statistical conditions, which can quickly detect nearby targets and reduce the risk of false triggering caused by single-point noise, local depth anomalies, or false detections.

[0132] In one possible implementation, after determining the initial risk zone where the target object is located based on the depth distance information and the risk zone information, the method further includes: if the risk level corresponding to the initial risk zone meets a preset risk condition, determining a local region of interest containing the target object based on the depth map; generating a target object point cloud corresponding to the target object based on the depth information corresponding to the local region of interest; obtaining the safety shell corresponding to the robot device; determining the closest distance between the target object point cloud and the safety shell; and determining the target risk zone where the target object is located based on the closest distance and the risk zone information.

[0133] The preset risk conditions can include situations where the target risk zone is a deceleration zone, a stop zone, the latter half of a deceleration zone, or an area close to a stop zone; or situations where the approach speed of the target object is large, the depth confidence is insufficient, the target object area is large, or the target object is located in front of the robot's direction of movement. In other words, the preset risk conditions are used to indicate risk states where a coarse judgment based on the depth map alone may not be sufficient to meet the accuracy requirements of safety control, and further fine judgment using point clouds is required.

[0134] A local region of interest (ROI) represents a local image region or local spatial region containing a target object. The target object point cloud represents the point cloud distribution of the target object in 3D space. It can be obtained by backprojecting depth pixels within the ROI or by cropping a local point cloud directly output from the sensor. Updating the target risk partition based on the nearest distance between the target object point cloud and the containment structure further refines the risk partitions coarsely determined from the depth map into more accurate 3D spatial risk partitions.

[0135] Implementing this approach allows for rapid assessment using depth maps when the risk is low, while enabling precise assessment of local point clouds and calculation of safe zone distances when the risk is high. This approach ensures the accuracy of safety assessments while reducing the computational overhead of continuous point cloud computing.

[0136] In one possible implementation, determining the local region of interest containing the target object based on the depth map includes:

[0137] Based on the depth pixel distribution of the target object in the depth map, the candidate region corresponding to the target object is determined;

[0138] The candidate region is determined as the local region of interest; or, the candidate region is expanded according to at least one of the following: the position of the target object in the candidate region, the depth pixel distribution corresponding to the target object, and the distance between the target object and the boundary of the candidate region, to obtain the local region of interest.

[0139] Enlarging the candidate image region can be achieved by expanding it to a fixed pixel width, adaptively expanding it according to the depth and distance of the target object, expanding it according to the ratio of the bounding box of the connected region, expanding it along the direction of approach to the target object, expanding it along the direction of movement of the robot, or expanding it in conjunction with the sensor's field of view. By expanding the candidate image region, the complete outline of the target object can be covered as much as possible, avoiding the omission of the target object's edge point cloud due to the candidate region being too small.

[0140] By implementing this method, the local region of interest can be determined based on the specific situation of the candidate image region. This allows subsequent point cloud precision judgment to reduce the processing range while preserving the complete spatial information of the target object as much as possible, thereby improving the reliability of the nearest distance calculation.

[0141] In one possible implementation, generating a target object point cloud based on the depth information corresponding to the local region of interest includes: performing a 3D point cloud transformation on the depth pixels within the local region of interest to obtain a local point cloud; and removing the robot device's own point cloud from the local point cloud based on at least one of the robot device's geometric model, the robot device's pose information, and the robot device's corresponding safety shell to obtain the target object point cloud.

[0142] By implementing this method, local targets in the depth map can be converted into 3D point clouds, and the point cloud of the robot itself can be removed, thereby obtaining a more accurate point cloud of the target object, providing a reliable data foundation for subsequent calculation of the closest distance to the containment.

[0143] In one possible implementation, the containment is a dynamic containment determined based on the current state of the robotic device.

[0144] The current state can include at least one of the following: the robot's current joint angle, current joint velocity, end effector pose, end effector velocity, link pose, mobile chassis pose, current trajectory execution speed, load state, end effector tool state, or operating mode. A dynamic safety shell is used to represent the safe reference range of the robot in its current state, and it can be updated in real time as the robot's posture, velocity, load, or end effector changes.

[0145] Implementing this method allows the safety enclosure to match the real-time status of the robot equipment, avoiding insufficient safety protection due to an excessively small fixed safety range, and also avoiding unnecessary deceleration or shutdown due to an excessively large fixed safety range.

[0146] In one possible implementation, the dynamic containment is determined by: acquiring the device model information and current state information of the robot device; determining the real-time spatial range of the robot device based on the device model information and the current state information; and determining the dynamic containment based on the real-time spatial range and a preset safety margin.

[0147] The device model information may include at least one of the following: the robot's geometric model, link model, joint model, end-effector model, load model, unified robot description format model, bounding box model, capsule model, mesh model, voxel model, or signed distance field model. Current state information may include at least one of the following: current joint angle, joint velocity, end-effector pose, link pose, mobile chassis pose, load state, and operating mode. Real-time spatial extent represents the spatial extent that the robot actually occupies or may influence at the current moment.

[0148] In one possible implementation, the real-time position and orientation of the robot body, links, joints, end effector, or load in a unified coordinate system can be calculated based on device model information and current state information. A dynamic safety shell is then obtained by extending a safety distance outside this real-time spatial range. The extended safety distance can be determined based on the robot type, movement speed, load, end effector hazard attributes, sensor errors, and user configuration.

[0149] Implementing this method allows for the generation of a dynamic safety shell based on the robot equipment model and its current state. This safety shell can reflect the robot equipment's current true shape and motion risks, providing a more accurate spatial reference for judging the distance to the target object and controlling risk zones.

[0150] In one possible implementation, determining the closest distance between the obstacle point cloud and the containment, or determining the closest distance between the target object point cloud and the containment, includes: determining the distances from a plurality of points in the target point cloud to the containment, wherein the target point cloud is the obstacle point cloud or the target object point cloud; and determining the closest distance based on the distances from the plurality of points to the containment.

[0151] In this point cloud, multiple points can represent all points in the obstacle point cloud, or a subset of points after downsampling, filtering, clustering, confidence filtering, or effective region filtering. The distance from a point to the containment can be determined based on the containment's representation. For example, when the containment is represented using a bounding sphere, bounding box, or capsule, the shortest distance from a point to the corresponding geometric surface can be calculated; when the containment is represented using a mesh model, the shortest distance from a point to the mesh surface can be calculated; and when the containment is represented using a signed distance field, the distance value of a point in the distance field can be queried.

[0152] By implementing this method, the distance relationship between multiple spatial points in the point cloud and the containment can be fully utilized to determine the closest distance, avoiding inaccurate risk assessment caused by relying solely on a single detection point or distance in a single direction.

[0153] In one possible implementation, determining the nearest distance based on the distances from the plurality of points to the containment includes: determining a distance set based on the distances from the plurality of points to the containment; determining the minimum distance in the distance set as the nearest distance; or, performing abnormal distance elimination and / or valid point filtering on the distance set, and determining a distance characterization value based on the minimum distance in the filtered distance set, the average distance of the nearest plurality of points, the low percentile distance, the nearest valid cluster distance, or the distance confirmed after multiple consecutive frames, and using the distance characterization value as the nearest distance.

[0154] Among these methods, the minimum value can be used to conservatively assess the nearest risk; the average and median distances of the nearest multiple points can reduce the impact of a single noise point; the low percentile distance can reflect the near-range risk while balancing conservatism and stability; the nearest effective cluster distance can avoid misjudgments triggered by isolated noise points; and the distance confirmed after multiple consecutive frames can confirm the continuous approach of the target object through the time dimension. These methods can be used individually or in combination.

[0155] By implementing this approach, the appropriate nearest distance statistical method can be selected based on point cloud quality, sensor noise, and security level, thereby improving the stability and reliability of risk zoning judgment and reducing the risk of false triggering or missed triggering.

[0156] In one possible implementation, the at least two risk zones include at least two of a warning zone, a deceleration zone, and a stop zone; wherein the risk level of the warning zone is lower than the risk level of the deceleration zone, and the risk level of the deceleration zone is lower than the risk level of the stop zone.

[0157] The warning zone indicates an area where the target object is close to the robot but the risk level is relatively low; the deceleration zone indicates an area where the target object is getting closer to the robot and there is a higher risk of approach; and the stop zone indicates an area where the target object is close to the robot or there is a risk of collision. The warning zone, deceleration zone, and stop zone can be set in order of increasing distance from the robot, or they can be set around the dynamic safety housing, end effector, key links, the outer contour of the mobile chassis, or the predicted motion trajectory of the robot.

[0158] By implementing this approach, graded safety control can be achieved through risk zones with at least two different risk levels, enabling robotic devices to respond step by step based on the proximity of the target object, rather than stopping directly in all risk situations.

[0159] In one possible implementation, controlling the operation of the robot device according to the target risk zone includes: when the target risk zone is a warning zone, controlling the robot device to maintain its current operating state and / or outputting a warning message, the warning message being used to indicate that the target object is in the warning zone; when the target risk zone is a deceleration zone, adjusting the original operating state of the robot device to obtain an operating state that reduces the risk of collision; and when the target risk zone is a stop zone, controlling the robot device to stop operating, pause operating, or maintain its current position.

[0160] Optionally, the alert information may include audible and visual alarms, interface prompts, directional prompts, log recordings, remote monitoring prompts, risk area displays, or sending alert signals to the host computer. Adjusting the operating status may include reducing the speed multiplier, limiting the maximum linear speed, limiting joint speed, limiting acceleration, adjusting the trajectory, partially avoiding obstacles, or changing the end effector pose. Stopping operation, pausing operation, or maintaining the current position can be determined based on the robot's controller capabilities and the safety requirements of the current task.

[0161] By implementing this method, different control strategies can be adopted for the warning zone, deceleration zone, and stop zone. This maintains operational continuity when the risk is low, reduces operational risk when the risk increases, and stops or maintains a safe state in a timely manner when the risk is high, thus balancing safety and efficiency.

[0162] In one possible implementation, adjusting the original operating state of the robot device to obtain an operating state that reduces the risk of collision includes: acquiring the orientation information of the target object relative to the robot device; determining the operating adjustment direction and / or the operating adjustment amount based on the distance information and the orientation information; and adjusting the original operating state of the robot device based on the operating adjustment direction and / or the operating adjustment amount to obtain the adjusted operating state.

[0163] The orientation information indicates the position and direction of the target object relative to the robot device. This can include the target object being in front of, behind, to the left, to the right, above, below, near the end effector, near a critical link, or in front of the moving chassis's direction of travel. The movement adjustment direction indicates the direction the robot device needs to adjust to move away from the target object or reduce approach risk. The movement adjustment amount indicates the adjustment magnitude and can include speed adjustment, pose adjustment, trajectory correction, joint position correction, or joint speed correction.

[0164] By implementing this method, the robot can be oriented and adjusted based on the distance and orientation between the target object and the robot, enabling the robot to not only decelerate but also correct its movement in a direction that reduces risk, thereby improving its obstacle avoidance capability within the deceleration zone.

[0165] In one possible implementation, determining the operating adjustment direction and / or operating adjustment amount based on the distance information and the orientation information includes: determining the operating adjustment direction that moves the robot away from the target object based on the orientation information; and determining the operating adjustment amount related to the distance to the target object based on the distance information, wherein the operating adjustment amount is negatively correlated with the distance value corresponding to the distance information.

[0166] Specifically, when the target object is located on one side of the robot, the direction away from the target object can be determined as the direction of operation adjustment; when the target object is located in front of the robot's direction of movement, the robot's speed along that direction can be reduced or its movement path adjusted. The smaller the distance information, the closer the target object is to the robot, and the larger the corresponding operation adjustment amount can be; the larger the distance information, the lower the risk of the target object, and the smaller the corresponding operation adjustment amount can be. The operation adjustment amount can be determined using linear functions, piecewise functions, reciprocal functions, quadratic functions, or lookup tables.

[0167] Implementing this method allows the direction of operational adjustments to match the location of the target object, and the amount of operational adjustments to match the proximity of the target object, thereby achieving smoother and more targeted risk avoidance control.

[0168] In one possible implementation, adjusting the original operating state of the robot device according to the operating adjustment direction and / or the operating adjustment amount includes: determining at least one of trajectory correction amount, velocity correction amount, acceleration correction amount, end-effector pose correction amount, joint position correction amount, and joint velocity correction amount according to the operating adjustment direction and / or the operating adjustment amount; and adjusting the original operating state according to at least one of the trajectory correction amount, velocity correction amount, acceleration correction amount, end-effector pose correction amount, joint position correction amount, and joint velocity correction amount.

[0169] The original operating state can include the robot's current task trajectory, end effector speed, joint speed, chassis speed, end effector pose, joint position, or controller output commands. Trajectory corrections can be used to adjust the robot's subsequent motion path; speed corrections can be used to reduce or change the motion speed; acceleration corrections can be used to limit the amplitude of motion changes; end effector pose corrections can be used to adjust the position or attitude of the end effector; and joint position and speed corrections can be used to achieve obstacle avoidance control within joint space.

[0170] By implementing this method, appropriate adjustment objects can be selected according to the specific control interface and motion form of the robot equipment, so that the control method of this application can be adapted to various control architectures such as position control, speed control, trajectory control, joint control or host computer supervision control.

[0171] In one possible implementation, after obtaining the adjusted operating state, and before controlling the robot device to operate according to the operating state to be executed, the method further includes: determining whether the adjusted operating state meets the operating restrictions of the robot device; if the operating restrictions are met, determining the adjusted operating state as the operating state to be executed for the robot device, and controlling the robot device to operate according to the operating state to be executed; if the operating restrictions are not met, reducing the operating speed of the robot device, or controlling the robot device to stop operating.

[0172] The operational constraints may include at least one of the following: speed multiplier limit, maximum linear speed limit, joint speed limit, acceleration limit, joint limit, workspace limit, end effector posture constraint, load constraint, controller safety constraint, task path constraint, and constraint to avoid entering higher-risk zones. If the adjusted operating state does not meet the operational constraints, it indicates that the adjustment may cause the robot to exceed its own capabilities, safety boundaries, or task constraints. In this case, the operating speed can be reduced, or the robot can be stopped.

[0173] Implementing this method allows for further verification of the feasibility and safety of the adjustment results after the robot equipment has been adjusted for obstacle avoidance or operational status, thus avoiding the introduction of new operational risks due to obstacle avoidance control itself and improving the reliability of robot equipment operation control.

[0174] See Figure 2 This application provides an example of risk zoning. Figure 2 The example of forming warning zones, deceleration zones, and stopping zones around robot equipment is used for illustration.

[0175] In one possible implementation, the risk zoning information includes at least two of the following: a warning zone, a deceleration zone, and a stop zone. The warning zone, deceleration zone, and stop zone each correspond to a different risk level, with the risk level of the warning zone being lower than that of the deceleration zone, and the risk level of the deceleration zone being lower than that of the stop zone.

[0176] The warning zone can be used to indicate an area where a target object has approached the robot but the risk level is relatively low. When the target object enters the warning zone, the robot can maintain its current operating state and output a warning message. The warning message may include at least one of the following: audible and visual warnings, interface prompts, directional prompts, log recordings, remote monitoring prompts, or sending a warning signal to a host computer.

[0177] A deceleration zone can be used to indicate an area where a target object is approaching the robot and there is a high risk of approaching. When a target object enters the deceleration zone, the robot can adjust its operating state. The adjustment of the operating state may include at least one of the following: reducing the speed multiplier, limiting the end effector speed, limiting the joint speed, limiting the speed of the moving chassis, limiting acceleration, adjusting the motion trajectory, performing local obstacle avoidance, or adjusting the end effector pose.

[0178] A stop zone can be used to indicate an area where the target object is close to the robot or where there is a risk of collision. When the target object enters the stop zone, the robot can stop, pause, or maintain its current position. Optionally, the robot can also perform a safety brake, pause the current task, enter a safety hold state, or output a stop signal to the safety controller.

[0179] In one possible implementation, the warning zone, deceleration zone, and stop zone can be arranged in order of increasing distance from the robot. That is, the warning zone is relatively far from the robot to detect potential risks of approach in advance; the deceleration zone is located inside the warning zone to reduce the risk of robot movement as the target object gets closer; and the stop zone is located closer to the robot to stop or maintain a safe state when the target object approaches a more dangerous distance.

[0180] It should be noted that, Figure 2 The example of risk zones arranged in a ring around the robot is provided for illustrative purposes only and is not intended to limit the specific shape, number, or generation method of the risk zones. In practical applications, risk zones can be determined based on the robot's structural form, current pose, current speed, direction of motion, braking capability, end-effector shape, load status, work path, dynamic containment structure, and the location of target objects in the surrounding environment.

[0181] For example, risk zones can be one or more of the following shapes: circular, elliptical, polygonal, sector-shaped, annular, rectangular, bounding box, bounding sphere, capsule, mesh extension, voxel region, 3D containment shell extension region, or irregular spatial region. For fixed robotic arms, risk zones can be generated around the robotic arm base, key links, end effector, dynamic containment shell, or current work path; for mobile robots, risk zones can be generated based on the outer contour of the mobile chassis, travel direction, turning radius, and braking distance; for mobile robots equipped with robotic arms, risk zones can be generated by combining the mobile chassis's range of motion and the robotic arm's workspace.

[0182] thus, Figure 2The risk partitions shown are only for facilitating understanding of the spatial relationships between areas of different risk levels. This application does not limit the risk partitions to fixed annular areas, nor does it limit them to having the same center or regular boundaries. Implementing this approach allows for better adaptability of the risk partition information, enabling subsequent operational control based on the target risk partition of the target object to be applicable to different types of robotic equipment and different operational scenarios.

[0183] For ease of explanation, the following mainly uses robotic arms as an example to introduce the risk zoning generation and risk control process, but it should not be construed as a limitation on the type of robotic equipment.

[0184] See Figure 3 The risk partition generation and control process provided in this application embodiment may include the following steps.

[0185] Step 31: Acquire multi-source sensing information.

[0186] In one possible implementation, sensing modules installed on the robot arm body, robot arm base, robot arm peripheral workstations, the periphery of the robot body, or near the end effector collect sensing information about the environment around the robot arm.

[0187] For example, the perception module may include one or more visual sensors, depth sensors, or 3D perception sensors. For instance, the perception module may include one or more of the following: a fisheye camera, a wide-angle camera, a binocular red-green-blue (RGB) camera, a color-depth (RGB-D) camera, a structured light depth camera, a time-of-flight (TOF) depth camera, a multi-view camera, and a lidar system.

[0188] Optionally, the perception module may use a single depth camera, multiple depth cameras, binocular cameras, fisheye cameras, multi-fisheye panoramic systems, RGB cameras combined with monocular depth estimation algorithms, or combinations of the above schemes. This application does not limit this.

[0189] In one possible implementation, the perceived information includes one or more of the following: color image, grayscale image, fisheye image, wide-angle image, panoramic image, binocular image, depth map, disparity map, 3D point cloud, local point cloud, and panoramic view cloud. Specifically, the color image, grayscale image, fisheye image, wide-angle image, panoramic image, and binocular image are used to acquire two-dimensional visual information of the environment surrounding the robotic arm; the depth map and disparity map are used to acquire distance information of environmental targets relative to the sensors; and the 3D point cloud, local point cloud, and panoramic view cloud are used to represent the positional distribution of environmental targets in three-dimensional space.

[0190] Furthermore, the perceived information may carry one or more of the following: timestamp, sensor number, camera intrinsic parameters, distortion parameters, extrinsic parameters, confidence level, effective field of view identifier, depth validity identifier, and pixel coordinate information. If a multi-sensor system is used, the perceived information collected by each sensor can be synchronized through hardware triggering, software synchronization, or time window matching.

[0191] In one possible implementation, the confidence level can be directly output by the sensor, or it can be calculated based on one or more of the following: depth validity, image sharpness, parallax matching cost, point cloud density, reprojection error, multi-sensor observation consistency, and continuous frame stability.

[0192] Step 32: Preprocess the multi-source sensing information and unify the coordinate system.

[0193] In one possible implementation, the camera intrinsic parameters, distortion parameters, extrinsic parameters and mounting pose corresponding to each sensor are obtained, and the sensing information obtained in step 31 is preprocessed according to the sensor type.

[0194] Specifically, for images acquired by fisheye or wide-angle cameras, distortion correction, field of view unfolding, spherical projection, or panoramic stitching can be performed based on distortion parameters; for binocular images acquired by binocular color cameras, disparity maps can be calculated based on binocular matching, and depth maps or point clouds can be obtained from the disparity maps; for data acquired by RGB-D cameras, structured light depth cameras, or TOF depth cameras, the corresponding depth maps or point clouds can be obtained directly; for data acquired by multi-view cameras, depth maps or point clouds can be obtained based on multi-view matching or multi-view reconstruction.

[0195] Optionally, the depth map may come from a hardware depth camera, binocular matching, multi-view reconstruction, monocular depth estimation, panoramic depth estimation, or point cloud projection; this application does not limit this.

[0196] After obtaining a depth map or point cloud, the pixel coordinates in the depth map can be back-projected into 3D points in the sensor coordinate system based on the camera intrinsic parameters. Then, based on the sensor extrinsic parameters, the 3D points in the sensor coordinate system can be transformed to the robot arm base coordinate system, robot body coordinate system, world coordinate system, or unified control coordinate system. For point clouds directly output by the sensor, the point cloud can be transformed to the robot arm base coordinate system, robot body coordinate system, world coordinate system, or unified control coordinate system based on the corresponding extrinsic parameters of that sensor.

[0197] When multiple sensors are present, point clouds, depth information, or obstacle candidate regions obtained from different sensors can be transformed into the same coordinate system for fusion. During fusion, the timestamp, sensor number, confidence level, effective field of view identifier, and depth validity identifier corresponding to each sensing information can be retained. The multi-source sensing information can be filtered, merged, or weighted based on timestamp matching, confidence level weighting, spatial location consistency, or sensor priority.

[0198] In one possible implementation, when multiple sensors output conflicting information about depth, distance, or obstacle location in the same area, a more conservative distance result can be selected as the basis for subsequent risk assessment. Alternatively, the robotic arm speed can be reduced and the system can wait for confirmation from the next frame or multiple consecutive frames of data.

[0199] Therefore, images, depth maps, or point clouds collected by different types of sensors can be placed under a unified spatial reference with the real-time pose of the robotic arm, the robotic arm body mask, and the robotic arm dynamic safety shell, thus providing a data foundation for subsequent removal of the robotic arm's own region, selection of local regions of interest, calculation of the nearest distance, and judgment of risk zoning.

[0200] Step 33: Determine the real-time pose of the robotic arm.

[0201] In one possible implementation, the current state information of the robotic arm is read from the robotic arm controller. The current state information includes at least one of the following: current joint angle, joint velocity, end-effector pose, current trajectory execution speed, and load status.

[0202] Among them, the current joint angle is used to represent the rotational position of each joint of the robotic arm at the current moment; the joint speed is used to represent the movement speed of each joint of the robotic arm at the current moment; the end-effector pose is used to represent the position and orientation of the end of the robotic arm at the current moment; the current trajectory execution speed is used to represent the execution speed of the robotic arm when it moves along a predetermined trajectory; and the load state is used to represent the state of the tool or object carried by the end of the robotic arm.

[0203] In one possible implementation, the URDF model of the robotic arm includes the connection relationships and relative pose relationships between the links, joints, and end effector of the robotic arm. Based on the URDF model, the kinematic chain structure of the robotic arm is determined, and by substituting the current joint angles into the forward kinematic relationships, the real-time poses of each link, joint, and end effector relative to the robotic arm's base coordinate system can be calculated step-by-step.

[0204] Furthermore, based on the transformation relationship between the robot arm's base coordinate system and the unified coordinate system, the real-time poses of each link, joint, and end effector of the robot arm in the robot arm's base coordinate system can be transformed to the unified coordinate system, thus obtaining the real-time pose of the robot arm body in the unified coordinate system.

[0205] This allows us to determine the spatial position of the robotic arm relative to the unified sensing information at the current moment, providing a basis for the subsequent generation of the robotic arm body mask, its own occlusion area, dynamic safety shell, and risk partition.

[0206] Step 34: Generate the robot arm body mask and its own occlusion area.

[0207] In one possible implementation, based on the URDF model of the robotic arm, the current joint angle, and sensor extrinsic parameters, the current position and orientation of the robotic arm links, joint housings, end-effectors, and other components in each sensor coordinate system are determined, and the models of the robotic arm links, joint housings, end-effectors, and other components are projected onto the image or depth map of each sensor to obtain a robotic arm body mask.

[0208] Specifically, the URDF model of the robotic arm is used to provide the geometric structure, connection relationships, and relative pose relationships of the robotic arm links, joint housings, end effectors, etc.; the current joint angle is used to determine the actual pose of each component of the robotic arm at the current moment; and the sensor extrinsic parameters are used to determine the transformation relationship between the robotic arm coordinate system and the coordinate systems of each sensor. The robotic arm body mask is used to represent the regions in the image or depth map corresponding to the robotic arm's own structure, such as the robotic arm links, joint housings, and end effectors.

[0209] For point cloud data, the URDF model of the robotic arm can be converted into a 3D model or bounding volume in the current pose, and it can be determined whether the point cloud is within the robotic arm's main body model or its tolerance range. The tolerance range is used to compensate for dimensional deviations, sensor measurement errors, and coordinate transformation errors between the URDF model and the actual shape of the robotic arm.

[0210] Specifically, points located within the tolerance range of the robot arm's body mask or body model can be marked as robot arm points; image regions or depth regions occluded by the robot arm body can be marked as self-occluded regions. In subsequent obstacle detection, the robot arm's own point cloud and self-occluded regions can be removed to avoid the robot arm links, joints, or end effectors being misidentified as external obstacles.

[0211] In one possible implementation, if the deviation between the robot arm projection generated based on the current joint angle and URDF model and the robot arm's position in the actual image or point cloud exceeds a preset threshold, it can be determined that there is a calibration anomaly, encoder anomaly, or URDF model matching anomaly. In this case, the user can be prompted to check, and a conservative strategy can be adopted in subsequent risk assessments, such as reducing the speed multiplier, limiting acceleration, or pausing the current task.

[0212] Therefore, the robot arm's body mask, its own points, and its own occluded areas can be updated in real time according to the robot arm's movement state, thereby improving the accuracy and safety of subsequent obstacle judgment.

[0213] Step 35: Coarse judgment of high frame rate depth map.

[0214] In one possible implementation, in the current depth map, panoramic depth map, or fused depth representation, the depth data is first filtered for validity, and invalid depth points, low confidence points, robot body mask areas, and self-occluded areas are excluded. Then, the depth statistics of the external environment are quickly calculated.

[0215] Among them, the current depth map is used to represent the depth information acquired by the depth sensors participating in obstacle monitoring at the current moment; the panoramic depth map is used to represent the depth information formed after coordinate unification or view fusion of depth information from multiple sensors or multiple perspectives; the fused depth representation is used to represent the depth distribution result obtained by fusing depth maps, point clouds or other spatial perception information.

[0216] For example, depth statistics may include one or more of the following: minimum depth in effective depth, lower percentile distance of effective depth, minimum depth or average proximity of block regions, number of pixels smaller than the warning baseline boundary or area of ​​connected regions, depth confidence and effective pixel ratio.

[0217] If the depth statistics show that the external target has not entered the user-configured warning range, normal monitoring can continue, and high-overhead precision judgment will not be performed on the entire point cloud. Specifically, when the statistical results of the minimum depth, lower percentile distance, minimum depth of the block region, or average proximity distance in the effective depth do not meet the warning conditions, or when the number of pixels smaller than the warning baseline boundary or the area of ​​the connected region does not reach the preset scale, it can be determined that the current external target has not entered the warning range.

[0218] In one possible implementation, when a critical safety area is obstructed by the robotic arm itself or external objects, making it impossible to reliably determine distance, a manual inspection can be prompted, and a conservative deceleration strategy can be adopted for the robotic arm. If effective depth or effective point cloud cannot be obtained for several consecutive frames, and the robotic arm is still in motion, a deceleration state or a stop state can be output.

[0219] Therefore, by using a high frame rate coarse judgment at the depth map level, unnecessary point cloud fine judgment calculations can be reduced when the external target is far away or the risk is low. When the coarse judgment result shows that the external target may enter the warning range, the subsequent local ROI fine judgment step is then entered, thus balancing the real-time monitoring and the accuracy of obstacle judgment.

[0220] Step 36: Select the Region of Interest (ROI).

[0221] In one possible implementation, when the coarse depth map result indicates that a target has entered the warning range, candidate areas where external obstacles may exist are determined in the depth map, panoramic depth map, or fused depth representation based on the depth statistics obtained in step 35.

[0222] Specifically, the region near the minimum depth point in the depth map can be selected, or all connected regions smaller than the warning boundary can be selected, and then this region can be expanded to form a local ROI. The local ROI is used to represent a local image region in the depth map where there may be nearby external targets. Subsequently, only the point cloud or spatial data corresponding to the local ROI is refined to reduce the computational overhead caused by refining the point cloud of the entire image.

[0223] Optionally, the ROI can be determined as follows: take a fixed-size or distance-related pixel neighborhood centered on the minimum depth point; perform connected component analysis on pixels smaller than the warning boundary and take the bounding box of the connected component; expand the bounding box by a certain proportion; sort multiple candidate ROIs by nearest distance, region area or confidence level, and retain only the top few ROIs for fine-tuning.

[0224] In another possible implementation, in addition to the minimum depth neighborhood, a connected region less than the warning threshold, the projection area of ​​the robot arm's movement direction, the area near the key link, or the multi-strategy fusion area can also be used as the ROI. This application does not limit this.

[0225] Therefore, the subsequent precision judgment steps do not need to process all the perceived information, but instead perform distance calculation, obstacle confirmation, or risk assessment for the external target candidate area corresponding to the local ROI.

[0226] Step 37: Local ROI point cloud back projection and candidate obstacle segmentation.

[0227] In one possible implementation, the effective depth pixels within the ROI are back-projected to obtain the ROI point cloud. Here, the effective depth pixels within the ROI refer to the depth pixels located in the local ROI determined in step 36 that have not been judged as invalid depth or low confidence; back-projection refers to converting pixels in the two-dimensional depth map into three-dimensional points in the sensor coordinate system based on the pixel coordinates, depth values, and sensor intrinsic parameters of the depth pixels.

[0228] If the sensor directly outputs a point cloud, a local point cloud can be extracted based on the Region of Interest (ROI) or its corresponding 3D frustum. Specifically, when the sensor outputs a depth map, the ROI point cloud can be obtained through back projection; when the sensor outputs a point cloud, 3D points whose projection falls within the ROI can be selected based on the ROI's range in the image plane, or 3D points located within the corresponding 3D frustum can be extracted from the sensor-output point cloud based on the ROI's corresponding 3D frustum range.

[0229] Then, based on the robotic arm body mask and URDF model obtained in step 34, point clouds belonging to the robotic arm itself are removed. Specifically, for the ROI point cloud obtained by backprojection of the depth map, depth pixels located within the robotic arm body mask area can be excluded first, and then the 3D points obtained by backprojection can be further judged; for the obtained 3D point cloud, it can be determined whether the 3D points in the point cloud are located within the space occupied by the robotic arm's own structure, based on the URDF model or its tolerance range under the current joint angle.

[0230] The remaining point cloud after removal is used as the candidate external obstacle point cloud. The candidate external obstacle point cloud is used to represent the point cloud of external objects that still exist within the spatial range corresponding to the local ROI after excluding the structure of the robotic arm itself.

[0231] In multi-sensor scenarios, ROI point clouds obtained from multiple sensors can be uniformly transformed into the robot arm's base coordinate system and then fused. Fusion can employ methods such as timestamp matching, voxel downsampling, confidence weighting, and merging of duplicate points.

[0232] In one possible implementation, if multiple sensors give inconsistent assessments of the obstacle's position, distance, or presence in the same spatial area, the result with higher risk or closer distance can be selected as the basis for subsequent judgment. Alternatively, the speed can be reduced to wait for confirmation in the next frame. This can reduce the risk of misjudgment caused by conflicts in multi-sensor fusion.

[0233] Step 38: Generate the dynamic safety shell for the robotic arm.

[0234] In one possible implementation, the current 3D model of the robotic arm is calculated based on the link geometry, joint shape, and end-effector model in the URDF model, combined with the current joint angle. Then, the current 3D model is extended outward by a basic safety distance to form a dynamic safety shell.

[0235] The dynamic safety shell represents the safety buffer zone outside the current 3D model of the robotic arm. This safety buffer zone is updated in real time as the current joint angle of the robotic arm changes. In other words, when the posture of the robotic arm changes, the position and posture of the current 3D model of the robotic arm change accordingly, and the dynamic safety shell also changes synchronously.

[0236] In one possible implementation, the containment structure can be represented by one or more of the following: a link capsule, a bounding box, a bounding sphere, an extended mesh model, a signed distance field of the link model, a voxel occupancy model, and a tool safety envelope generated based on the end-tool shape. These representations can be used individually or in combination to balance computational speed and containment accuracy.

[0237] In one possible implementation, the containment enclosure's outward extension distance can be determined based on the linkage type, end-effector size, load, operating mode, and user configuration. Larger outward extension distances can be provided for end-effectors, grippers, high-speed moving links, tools with sharp edges, parts carrying heavy workpieces, or areas where clamping risks may occur.

[0238] Therefore, the dynamic containment structure can not only cover the robotic arm's own structure, but also form differentiated safety buffer ranges according to the risk level of different parts, and serve as a spatial benchmark for subsequent judgment of whether candidate external obstacle point clouds have entered the robotic arm's danger zone.

[0239] Step 39: Calculate the shortest distance from the candidate obstacle to the containment.

[0240] In one possible implementation, the three-dimensional Euclidean distance from each point in the candidate external obstacle point cloud to the surface of the robotic arm's dynamic safety shell is calculated. The candidate external obstacle point cloud is the remaining external obstacle point cloud obtained in step 37 after removing the robotic arm's own point cloud; the robotic arm's dynamic safety shell is the safety buffer area generated in step 38 that changes with the robotic arm's current posture.

[0241] Specifically, when the dynamic containment is represented by a link capsule, bounding box, or bounding sphere, the shortest distance from the 3D points in the candidate external obstacle point cloud to the corresponding geometric surface can be calculated respectively; when the dynamic containment is represented by an extended mesh model, the shortest distance from the 3D points in the candidate external obstacle point cloud to the mesh surface can be calculated; when the dynamic containment is represented by a signed distance field of a link model, the distance values ​​of the 3D points in the candidate external obstacle point cloud in the signed distance field can be directly queried.

[0242] If a candidate external obstacle point is located inside the dynamic containment, the distance corresponding to that point can be recorded as zero or a negative value. When the distance is recorded as zero, it can be used to indicate that the obstacle has entered the dynamic containment; when the distance is recorded as a negative value, the absolute value of the negative value can be used to indicate the depth of the obstacle inside the dynamic containment.

[0243] To reduce the risk of false triggering caused by a single noise point, instead of using a single nearest point as the sole criterion, the three-dimensional nearest distance for risk assessment can be determined by using the average or median distance of the nearest N points, the low percentile value of the distance distribution, the nearest effective cluster distance, or the confirmation distance of consecutive K frames.

[0244] Furthermore, the x, y, and z components of the nearest point or cluster of nearest points relative to the dynamic containment of the robotic arm can be recorded to determine whether an obstacle is located above, below, to the left, to the right, in front of, or behind the robotic arm, and to provide directional cues to the operator. However, the primary criterion for determining the risk level remains the three-dimensional nearest distance from the candidate external obstacle point cloud to the dynamic containment, or the three-dimensional nearest distance after statistical filtering.

[0245] Step 310: Dynamically calculate the risk zone boundary based on the robot arm speed.

[0246] In one possible implementation, the user can configure the basic boundaries of the warning zone, deceleration zone, and stop zone according to the specific operational scenario. The warning zone, deceleration zone, and stop zone are different risk areas defined based on the distance from candidate external obstacles to the robotic arm's dynamic safety housing.

[0247] Specifically, the three-dimensional closest distance from the candidate external obstacle to the dynamic safety shell of the robotic arm, obtained in step 39, is used as the judgment distance. When the judgment distance is greater than the boundary of the warning zone, the obstacle is considered to be within a relatively safe range; when the judgment distance is less than or equal to the boundary of the warning zone and greater than the boundary of the deceleration zone, the obstacle is considered to have entered the warning zone; when the judgment distance is less than or equal to the boundary of the deceleration zone and greater than the boundary of the stop zone, the obstacle is considered to have entered the deceleration zone; when the judgment distance is less than or equal to the boundary of the stop zone, the obstacle is considered to have entered the stop zone.

[0248] In one possible implementation, the base boundary is not fixed to a specific set of meters, but can be configured based on the robotic arm model, workspace, personnel access risk, load, end effector, operating speed, braking capacity, and safety level. For scenarios with high personnel access risk, large load, high end effector risk, or long braking distance, a larger base boundary can be configured.

[0249] During the operation of the robotic arm, the boundaries of each risk zone can be dynamically expanded based on the current speed. For example, the dynamic boundaries can be calculated using the following formula:

[0250]

[0251] in, This represents the dynamic boundary of the i-th risk zone, where i can be a warning zone, a deceleration zone, or a stop zone; This indicates the basic boundary of the risk partition configured by the user; This indicates the current speed of the robotic arm, which can be the end effector speed, the maximum linear velocity of the link, the key point speed obtained by mapping the joint speed, or the maximum value among the above speeds. This indicates response time, including sensing latency, computation latency, communication latency, and / or controller response latency; This indicates the allowable deceleration or braking capacity of the robotic arm under the current load and operating mode; This represents the safety margin, which can be configured based on sensor error, depth noise, calibration error, load uncertainty, and scenario requirements.

[0252] Alternatively, the dynamic boundary calculation method is not limited to the above braking distance formula, and can also use speed segment lookup table, user configuration table, model prediction results or safety level configuration, etc.

[0253] To ensure a clear zoning relationship, the dynamic boundary of the alert zone can be larger than that of the deceleration zone, and the dynamic boundary of the deceleration zone can be larger than that of the stop zone. When the dynamic boundaries of each zone calculated in the above manner do not satisfy this order relationship, constraints can be imposed by setting minimum intervals, boundary limits, or reordering to maintain a hierarchical relationship from near to far between the stop zone, deceleration zone, and alert zone.

[0254] Step 311: Risk level determination.

[0255] In one possible implementation, the three-dimensional nearest distance from the candidate external obstacle to the dynamic containment obtained in step 39 is compared with the dynamic partition boundary obtained in step 310, and the risk level is output.

[0256] Specifically, when the closest three-dimensional distance between a candidate external obstacle and the dynamic containment is greater than the dynamic boundary of the deceleration zone but less than or equal to the dynamic boundary of the warning zone, the candidate external obstacle is determined to have entered the warning zone, and a warning status is output. The warning status may include audible and visual alarms, interface prompts, directional prompts, display of risk areas, or sending a warning signal to the host computer.

[0257] When the closest three-dimensional distance between a candidate external obstacle and the dynamic containment is greater than the dynamic boundary of the stopping zone and less than or equal to the dynamic boundary of the deceleration zone, the candidate external obstacle is determined to have entered the deceleration zone, and a deceleration state is output. The deceleration state may include reducing the speed multiplier of the robotic arm, limiting the maximum linear velocity, limiting the joint velocity, or limiting the acceleration, and may trigger subsequent local obstacle avoidance processes.

[0258] When the closest three-dimensional distance between a candidate external obstacle and the dynamic containment is less than or equal to the dynamic boundary of the stop zone, or when the candidate external obstacle is located inside the dynamic containment, the candidate external obstacle is determined to have entered the stop zone, and a stop status is output. The stop status can include stopping the current trajectory execution, pausing the task, controlling the robotic arm to brake, entering a safety holding state, or triggering an emergency stop signal.

[0259] In one possible implementation, if abnormal situations such as low depth confidence, sparse point cloud, sensor occlusion, URDF projection matching failure, or conflicting multi-sensor fusion results occur, a conservative strategy can be adopted while prompting for manual inspection. The conservative strategy may include reducing the speed multiplier, limiting acceleration, pausing the task, or entering a safe hold state.

[0260] Specifically, when only a short-term decrease in depth confidence occurs, the speed can be reduced and monitoring can continue; when the point cloud remains sparse or the sensor remains occluded, the task can be paused and a manual check prompted; when URDF projection matching fails or multi-sensor fusion results continue to conflict, a safe hold state can be entered to avoid continuing high-speed operation when the spatial perception results are unreliable.

[0261] Therefore, the risk level determination is based not only on the distance between the candidate external obstacle and the dynamic containment, but also on the perceived reliability and model matching status for safety control.

[0262] In one possible implementation, if the initial risk zone where the target object is located has been determined by the aforementioned depth map coarse judgment, the initial risk zone can be updated based on the three-dimensional nearest distance from the candidate external obstacle point cloud to the dynamic safety shell, as well as the partition boundaries of the warning zone, deceleration zone, and stop zone, to obtain the updated target risk zone.

[0263] Step 312: Distance field repulsion local obstacle bypass within the deceleration zone.

[0264] In one possible implementation, when the risk level is deceleration and the candidate external obstacle has not entered the stop zone, a partial obstacle avoidance process can be performed.

[0265] Specifically, the repulsion direction of an obstacle relative to the robotic arm can be determined based on the nearest point pair from the candidate external obstacle point to the robotic arm's dynamic safety housing. The nearest point pair refers to the pair formed by the target obstacle point in the candidate external obstacle point cloud and the point on the surface of the robotic arm's dynamic safety housing that is closest to the target obstacle point.

[0266] In one possible implementation, the repulsion direction can be determined by the opposite direction from the nearest point of the dynamic containment to the obstacle point, the normal direction from the obstacle point to the surface of the nearest link, or the weighted average direction of multiple nearest points. When multiple proximity points exist, the multiple directions can be weighted according to distance or risk coefficient to obtain the comprehensive repulsion direction.

[0267] Furthermore, the repulsion strength can be generated based on the nearest distance. The closer the distance is to the boundary of the stopping zone, the greater the repulsion strength; the closer the distance is to the outer boundary of the deceleration zone, the smaller the repulsion strength. The repulsion strength can be a linear function, a quadratic function, a reciprocal function, or a piecewise function.

[0268] For example, the distance within the deceleration zone can be normalized to a risk coefficient r, where a larger r indicates a higher risk, and obstacle avoidance speed or pose offset can be generated based on r.

[0269] In one possible implementation, the adjustment targets for local obstacle avoidance include the end effector and the critical link closest to the obstacle. For the end effector, the repulsion direction can be mapped to an end effector Cartesian velocity correction or an end effector pose offset; for the critical link, the repulsion velocity of the critical link can be mapped to a joint velocity correction using the robotic arm Jacobian matrix.

[0270] When multiple critical linkage points pose risks simultaneously, they can be weighted and synthesized according to risk coefficient, distance, or linkage priority. The local obstacle avoidance correction is combined with the original mission speed to obtain the corrected control quantity.

[0271] During the synthesis process, the modified control variables should meet the following constraints: not exceeding the speed multiplier corresponding to the current risk level; not exceeding the allowable acceleration; not exceeding joint limits, speed limits, and workspace constraints; not causing the robotic arm to enter a new stop zone; and not violating the attitude constraints or end-effector constraints allowed by the current task.

[0272] Alternatively, the local obstacle avoidance method is not limited to the distance field repulsion method, but can also employ local replanning, velocity barriers, control barrier functions, artificial potential fields, or short-time trajectory optimization methods.

[0273] In one possible implementation, if local obstacle avoidance is feasible, the robotic arm performs a corrected trajectory or speed adjustment under speed-limited conditions to move the end effector and key links away from the obstacle. If local obstacle avoidance is not feasible, the risk increases for multiple consecutive frames, the obstacle avoidance correction exceeds the control constraints, the local obstacle avoidance correction remains saturated, the risk distance does not increase after obstacle avoidance, or the obstacle enters the stop zone, then the output of the local obstacle avoidance correction stops, and triggers a deceleration wait, a safe stop, or a wait for manual confirmation.

[0274] Step 313: Output the risk assessment results, control instructions, and / or reminder information.

[0275] In one possible implementation, the robotic arm control system outputs at least one of the following: risk assessment result, control command, and warning information, based on the risk level obtained in step 311 and the local obstacle avoidance assessment result obtained in step 312.

[0276] The risk assessment result may include at least one of a warning state, a deceleration state, and a stop state. The control commands may include at least one of a suggested speed multiplier, acceleration limit, local obstacle avoidance correction, stop sign, joint speed limit command, end-of-range speed limit command, trajectory pause command, and safety braking command. The warning information may include at least one of the following: obstacle direction, nearest distance, corresponding sensor, risk area, confidence level, and abnormal state.

[0277] Specifically, it is suggested that the speed multiplier be used to limit the current task speed of the robotic arm; the acceleration limit be used to limit the range of motion changes of the robotic arm; the local obstacle avoidance correction amount can be the speed correction, pose offset or joint speed correction generated in the deceleration zone in step 312; and the stop flag be used to indicate that the robotic arm pauses the current trajectory, stops the current task, maintains the current position or performs a safety braking procedure.

[0278] In one possible implementation, the obstacle orientation can be determined based on the position of the nearest point or cluster of nearest points relative to the dynamic safety housing of the robotic arm obtained in step 39; the nearest distance can be the three-dimensional nearest distance from the candidate external obstacle to the dynamic safety housing or the distance after statistical filtering; the corresponding sensor is used to identify which sensor mainly detects the obstacle; the risk area is used to indicate that the candidate external obstacle is currently in the warning area, deceleration area or stop area; the confidence level is used to indicate the reliability of the current distance judgment or obstacle detection result; the abnormal state is used to indicate whether there are situations such as low depth confidence, sparse point cloud, sensor occlusion, URDF projection matching failure or multi-sensor fusion conflict.

[0279] Specifically, in a risk level of alert, the robotic arm can maintain its original movement or slightly limit its speed, and alert the operator through an interface, audio-visual aids, logs, remote monitoring, or human-machine interface devices. In a risk level of deceleration, the robotic arm can reduce its speed ratio, limit joint speed, end effector speed, or trajectory execution speed, and perform local obstacle avoidance corrections when local obstacle avoidance is feasible. In a risk level of stop, the robotic arm can pause trajectory execution, safely stop, maintain its current position, or execute a safety braking procedure permitted by the controller.

[0280] Optionally, the form of the control command is not limited to speed multiplier, acceleration limit, local obstacle avoidance correction amount and stop flag, but can also be converted into joint speed limit command, end speed limit command, trajectory pause command, local replanning constraint or safety controller input.

[0281] This application does not limit the specific interface of the robotic arm controller. The risk assessment results, control commands, or alerts mentioned above can be sent to the robotic arm controller, host computer, alarm device, or human-machine interface via the controller application programming interface, fieldbus, industrial Ethernet, robot operating system interface, digital signals, or host computer control interface. Different control architectures can choose to execute all or part of the output content according to their own capabilities.

[0282] Therefore, the embodiments of this application can be adapted to different control architectures such as position control, speed control, trajectory control or host computer supervision control, and realize closed-loop processing from environmental perception, risk partition generation to hierarchical motion control of the robotic arm.

[0283] In one possible implementation, based on Figure 3 As illustrated in the flowchart, this application also provides an example of zoned motion control for a robotic arm. This example uses sensor depth maps and / or point clouds, the current joint angle q of the robotic arm, the current velocity v of the robotic arm, the Unified Robot Description Format (URDF) model of the robotic arm, the basic boundaries of the user-configured warning zone, deceleration zone, and stop zone, response time, allowable deceleration, and safety margin as inputs to assess the risk of obstacles around the robotic arm and output corresponding motion control results.

[0284] Specifically, the robotic arm control system calculates the current link pose of the robotic arm based on the current joint angle q and the URDF model, and projects the robotic arm model onto the depth map by combining the intrinsic and extrinsic parameters of the sensors to generate a robotic arm body mask. Subsequently, the robotic arm body area, invalid depth areas, and low confidence areas are removed from the depth map, and the minimum depth corresponding to the external environment or the near-distance depth after statistical filtering is calculated.

[0285] If the minimum depth does not enter the warning range, it is determined that the current external target does not pose an approach risk, a normal state is output, and high frame rate monitoring continues. If the minimum depth enters the warning range, a local ROI is generated based on the neighborhood or nearby connected region of the minimum depth point, and the effective depth within the ROI is back-projected into a point cloud; simultaneously, the robot arm's own points are removed by combining the robot arm's body mask and URDF model to obtain a candidate external obstacle point cloud.

[0286] Furthermore, the robotic arm control system generates a dynamic safety shell for the robotic arm based on the URDF model and the current link pose, and calculates the nearest statistical distance d from the candidate external obstacle point cloud to the dynamic safety shell. The nearest statistical distance d can be the average distance, median distance, lower percentile distance, nearest effective cluster distance, or distance after confirmation over multiple consecutive frames among the nearest N points.

[0287] After obtaining the most recent statistical distance d, the boundaries of the warning zone, deceleration zone, and stop zone are dynamically calculated based on the current speed v of the robotic arm. For example, the dynamic boundaries can be determined based on the base boundary, response time, allowable deceleration, and safety margin.

[0288] Subsequently, the most recent statistical distance d is compared with the dynamic partition boundary: if d is in the warning zone, a warning status is output; if d is in the deceleration zone, a deceleration status is output, and a local obstacle avoidance correction is generated based on the direction and distance of the candidate external obstacle relative to the dynamic safety shell; if d is in the stopping zone, or local obstacle avoidance is not feasible, a stop flag is output.

[0289] During deceleration, the robotic arm control system can synthesize the local obstacle avoidance correction amount with the original task speed to obtain the corrected control amount. If the corrected control amount meets the constraints of speed ratio, allowable acceleration, joint limits, workspace, and task posture, the robotic arm will execute the corrected trajectory or corrected speed under the speed limit state; if the corrected control amount does not meet the constraints, or the risk distance does not increase after obstacle avoidance, the output of the obstacle avoidance correction amount will stop, and a safety stop will be triggered or manual confirmation will be awaited.

[0290] The output information in this embodiment may include risk level, suggested speed multiplier, acceleration limit, local obstacle avoidance correction, stop sign, obstacle warning direction, and nearest distance. Through this method, rapid early warning of obstacles around the robotic arm, accurate distance judgment, dynamic risk zoning, deceleration obstacle avoidance, and safe stopping control can be achieved without continuously performing high-overhead calculations on the entire scene point cloud.

[0291] Figure 3 This paper illustrates the complete process of generating and controlling risk zones for a robotic arm. This process utilizes multi-source sensing information, the real-time pose of the robotic arm, the robotic arm's body mask, the dynamic safety shell, and the distance relationship between the target object's point cloud to determine the risk level and implement corresponding control. To reduce the computational overhead of continuous point cloud fine-tuning while ensuring safety, this embodiment of the application can also adopt a graded triggering risk judgment method. Specifically, a high-frame-rate coarse judgment can be performed first based on the depth map, and the boundaries of the warning zone, deceleration zone, and stop zone can be dynamically determined in conjunction with the current speed of the robotic arm. When the external target only enters the warning zone or the first half of the deceleration zone, a rapid warning or speed limit is prioritized based on the depth map. When the external target further enters the second half of the deceleration zone or may enter the stop zone, local ROI point cloud fine-tuning, dynamic safety shell distance calculation, and precise stop judgment are then initiated. Thus, point cloud back-projection and full point cloud computing can be reduced in the lower-risk stage, while the reliability of distance judgment and stop control can be improved in the higher-risk stage.

[0292] See Figure 4 The specific steps are as follows:

[0293] Step 41: Acquire multi-source sensing information.

[0294] Step 42: Preprocess the multi-source sensing information and unify the coordinate system.

[0295] Step 43: Read the encoder or controller status and user partition configuration to obtain joint angles, speeds, end effectors or link poses.

[0296] The encoder information or controller status may include at least one of the following: joint angle, joint speed, end effector pose, end effector speed, link pose, current trajectory execution speed, and load status. The user partition configuration may include at least one of the following: basic boundaries of the warning zone, deceleration zone, and stop zone; response time; allowable deceleration; and safety margin. This provides a foundation for subsequent real-time body model calculation of the robotic arm, dynamic containment generation, and risk partition boundary calculation.

[0297] Step 44: Calculate the real-time body model of the robotic arm based on URDF, and project it to generate the robotic arm body mask and its own occlusion area.

[0298] In one possible implementation, based on the URDF model of the robotic arm and the current joint state, the real-time body model of the robotic arm at the current moment is calculated, and the spatial positions of each link, joint and end effector of the robotic arm are determined.

[0299] Step 45: Eliminate the robotic arm's own region, invalid depth regions, and low confidence regions.

[0300] In one possible implementation, regions of the robotic arm itself, invalid depth regions, and low-confidence regions are removed from the depth map or fused depth data to obtain effective depth data for coarse judgment of external targets.

[0301] Step 46: Coarse judgment of high frame rate depth map.

[0302] In one possible implementation, a coarse depth map is made based on the effective depth data obtained in step 45, and depth statistics such as nearest depth, local minimum depth, low percentile depth, or close connected regions are calculated.

[0303] Step 47: Dynamically calculate the boundaries of the stop zone, deceleration zone, and warning zone based on the current speed.

[0304] In one possible implementation, the boundaries of the stop zone, deceleration zone, and warning zone are dynamically calculated based on the current speed of the robotic arm, response time, allowable deceleration, and safety margin.

[0305] Step 48: Determine if the depth map has entered the warning area.

[0306] In one possible implementation, the coarse depth map result is compared with the boundary of the dynamic alert zone to determine whether an external target has entered the alert zone.

[0307] If the alert area is not entered, proceed to the next frame loop; if the alert area is entered, proceed to step 49.

[0308] In one possible implementation, when the coarse depth map result does not enter the warning area, the robotic arm maintains normal monitoring or normal operation and enters the next frame loop.

[0309] Step 49: Quickly determine or provide a warning based on the depth map, without performing point cloud back projection, and then proceed to step 410.

[0310] In one possible implementation, when the coarse depth map result indicates that an external target has entered the warning zone, but it has not yet been determined that it has entered the deceleration zone, the warning status can be output based on the nearest depth, low percentile depth, or nearby connected regions in the depth map, without immediately performing ROI point cloud back projection. This reduces the computational overhead for fine point cloud judgment during lower-risk phases while maintaining high frame rate monitoring.

[0311] Step 410: Determine whether to enter the deceleration zone based on the depth map.

[0312] In one possible implementation, if the depth map has already entered the warning zone, the system can further quickly determine whether an external target has entered the deceleration zone based on the nearby target in the depth map, the boundary of the dynamic deceleration zone, and the current motion state of the robotic arm.

[0313] If the deceleration zone is not entered, proceed to the next frame loop; if the deceleration zone is entered, proceed to step 411.

[0314] Step 411: Quickly determine or provide a warning based on the depth map, without performing point cloud back projection, and then proceed to step 412.

[0315] In one possible implementation, when the coarse depth map result indicates that an external target has entered the deceleration zone, a deceleration status or speed limit control can be output based on the depth map result without immediately performing point cloud back projection. If the external target is only located in the first half of the deceleration zone, the speed limit can be maintained by continuously monitoring with the depth map; if the external target further approaches and enters the second half of the deceleration zone, the ROI point cloud fine judgment process is initiated to improve the accuracy of close-range risk assessment.

[0316] Step 412: Determine whether you have entered the second half of the deceleration zone.

[0317] If yes, proceed to step 413; otherwise, proceed to step 425.

[0318] In one possible implementation, the deceleration zone can be divided into a front half and a rear half along the direction approaching the robotic arm. When an external target enters the front half of the deceleration zone, the robotic arm can implement speed limiting or issue a warning based on the depth map results and continue high-frame-rate monitoring. When the external target enters the rear half of the deceleration zone, it indicates that the external target is closer to the stopping zone, and the risk level further increases. At this point, a local Region of Interest (ROI) is generated, and ROI point cloud precision judgment is initiated. This reduces the continuous computational overhead on the global or local point cloud while ensuring safety.

[0319] Step 413: Generate a local ROI based on the depth region of the deceleration zone, and begin fine-grained ROI point cloud analysis;

[0320] Step 414: ROI depth back projection or cropping of local point cloud multi-sensor point cloud fusion and removal of body points.

[0321] Step 415: Generate a dynamic containment based on URDF and calculate the three-dimensional nearest statistical distance from the obstacle point to the containment.

[0322] Step 416: Determine whether the stop zone has been entered.

[0323] If the stop zone is entered, proceed to step 417; if the stop zone is not entered, proceed to step 418.

[0324] In one possible implementation, step 416 can update the target risk partition obtained based on the coarse assessment of the depth map by comparing the three-dimensional nearest statistical distance obtained in step 415 with the partition boundary in the risk partition information. For example, when the three-dimensional nearest statistical distance is less than or equal to the stop zone boundary, the target risk partition can be updated to a stop zone; when the three-dimensional nearest statistical distance is greater than the stop zone boundary and less than or equal to the deceleration zone boundary, the target risk partition can be updated to a deceleration zone.

[0325] Step 417: Gradually expand the ROI range, accurately compare the robotic arm safety shell with the point cloud, and then execute step 419.

[0326] In one possible implementation, when step 416 determines that an external target may enter the stopping zone, the ROI range can be gradually expanded based on the current ROI, and a precise comparison can be performed between the expanded ROI point cloud and the dynamic safety shell of the robotic arm. This precise comparison may include calculating the three-dimensional nearest statistical distance, the nearest effective cluster distance, or the distance confirmed after multiple consecutive frames from the candidate external obstacle point cloud to the dynamic safety shell. This improves the reliability of stopping zone determination and reduces the risk of false triggering due to single noise points, incomplete ROI truncation, or local depth anomalies.

[0327] Step 419: If the distance between the point cloud and the safe house is confirmed to be lower than the stop threshold through precise comparison, then trigger pause, safe stop, or maintain the current position, and then execute step 425.

[0328] In one possible implementation, when the 3D nearest statistical distance between the point cloud after expanding the ROI and the dynamic safety shell of the robotic arm is less than or equal to the stop zone threshold, or when the obstacle point is located inside the dynamic safety shell, it can be determined that an external target has entered the stop zone, triggering a pause, a safe stop, or maintaining the current position. Subsequently, information such as the stop status, stop flag, nearest distance, risk area, and logs are output uniformly.

[0329] Step 418: Limit speed and acceleration, enter the distance field to repel local obstacle avoidance, and then execute step 420.

[0330] Step 420: Local obstacle avoidance algorithm, generate repulsion direction and repulsion strength based on the nearest point pair to protect the end effector and the nearest critical link.

[0331] Step 421: The original task speed and obstacle avoidance speed are combined to satisfy the speed ratio, acceleration, joint limit and workspace constraints.

[0332] Step 422: Determine if bypassing the obstacle is feasible.

[0333] If feasible, proceed to step 423; if not feasible, proceed to step 424.

[0334] Step 423: Execute the trajectory correction, bypass the obstacles and continue the mission, then proceed to step 425.

[0335] Step 424: Trigger a safe stop, then proceed to step 425.

[0336] Step 425: Unified output: Risk level, alert status, speed multiplier, obstacle avoidance correction amount, stop flag, log.

[0337] See Figure 5 This application provides a control device for a robot device, the device 500 including:

[0338] The acquisition module 501 is used to acquire distance information between the target object and the robot device, as well as risk zoning information of the robot device, wherein the risk zoning information includes the range of at least two risk zoning areas and the risk level corresponding to each risk zoning area;

[0339] The determination module 502 is used to determine the target risk zone where the target object is located based on the distance information and the risk zone information;

[0340] The control module 503 is used to control the operation of the robot device according to the target risk zone.

[0341] In one possible implementation, the acquisition module 501 is further configured to acquire the obstacle point cloud corresponding to the target object, acquire the safety shell corresponding to the robot device, determine the closest distance between the obstacle point cloud and the safety shell, and use the closest distance as the distance information between the target object and the robot device.

[0342] In one possible implementation, the acquisition module 501 is further configured to acquire an environmental point cloud of the area surrounding the robot device, determine a local region of interest corresponding to the target object based on the environmental point cloud, remove the robot device's own point cloud from the point cloud corresponding to the local region of interest to obtain the target object point cloud, and use the target object point cloud as the obstacle point cloud.

[0343] In one possible implementation, the acquisition module 501 is further configured to acquire a depth map of the area surrounding the robot device, determine the depth distance information corresponding to the target object based on the depth map, and use the depth distance information as the distance information between the target object and the robot device.

[0344] In one possible implementation, the acquisition module 501 is further configured to determine a candidate image region based on the depth value in the depth map, determine the object corresponding to the candidate image region as the target object, and determine the depth distance information based on the depth information corresponding to the candidate image region.

[0345] In one possible implementation, the acquisition module 501 is further configured to determine an image region that satisfies at least one of the following conditions as the candidate image region: the minimum depth in the image region is less than or equal to a first distance threshold; the low percentile depth in the image region is less than or equal to a second distance threshold; the number of pixels in the image region whose depth is less than or equal to a third distance threshold is greater than or equal to a number threshold; the area of ​​the connected region in the image region whose depth is less than or equal to a fourth distance threshold is greater than or equal to an area threshold; and the depth change rate in the image region is greater than or equal to a speed threshold.

[0346] In one possible implementation, after determining the initial risk zone where the target object is located based on the depth distance information and the risk zone information, the acquisition module 501 is further configured to, if the risk level corresponding to the initial risk zone meets the preset risk conditions, determine a local region of interest containing the target object based on the depth map, generate a target object point cloud corresponding to the target object based on the depth information corresponding to the local region of interest, acquire the safety shell corresponding to the robot device, and determine the closest distance between the target object point cloud and the safety shell; the determination module 502 is further configured to determine the target risk zone where the target object is located based on the closest distance and the risk zone information.

[0347] In one possible implementation, the acquisition module 501 is further configured to determine a candidate region corresponding to the target object based on the depth pixel distribution of the target object in the depth map; determine the candidate region as the local region of interest; or, expand the candidate region based on at least one of the position of the target object in the candidate region, the depth pixel distribution corresponding to the target object, and the distance between the target object and the boundary of the candidate region to obtain the local region of interest.

[0348] In one possible implementation, the acquisition module 501 is further configured to perform 3D point cloud conversion on the depth pixels in the local region of interest to obtain a local point cloud; and based on at least one of the geometric model of the robot device, the pose information of the robot device, and the safety shell corresponding to the robot device, remove the point cloud of the robot device itself from the local point cloud to obtain the point cloud of the target object.

[0349] In one possible implementation, the safety shell is a dynamic safety shell determined based on the current state of the robot device. The acquisition module 501 is further configured to acquire device model information and current state information of the robot device, determine the real-time spatial range of the robot device based on the device model information and the current state information, and determine the dynamic safety shell based on the real-time spatial range and a preset safety margin.

[0350] In one possible implementation, the acquisition module 501 is further configured to determine the distances from multiple points in the target point cloud to the containment, wherein the target point cloud is the obstacle point cloud or the target object point cloud; and determine the closest distance based on the distances from the multiple points to the containment.

[0351] In one possible implementation, the acquisition module 501 is further configured to determine a distance set based on the distances from the plurality of points to the containment; determine the minimum distance in the distance set as the nearest distance; or, perform abnormal distance elimination and / or effective point filtering on the distance set, and determine a distance characterization value based on the minimum distance in the filtered distance set, the average distance of the nearest plurality of points, the low percentile distance, the nearest effective cluster distance, or the distance after confirmation of multiple consecutive frames, and use the distance characterization value as the nearest distance.

[0352] In one possible implementation, the at least two risk zones include at least two of a warning zone, a deceleration zone, and a stop zone; wherein the risk level of the warning zone is lower than the risk level of the deceleration zone, and the risk level of the deceleration zone is lower than the risk level of the stop zone.

[0353] In one possible implementation, the control module 503 is further configured to, when the target risk zone is a warning zone, control the robot device to maintain its current operating state and output a warning message; when the target risk zone is a deceleration zone, adjust the original operating state of the robot device to obtain an operating state that reduces the risk of collision; and when the target risk zone is a stop zone, control the robot device to stop operating, pause operating, or maintain its current position.

[0354] In one possible implementation, the control module 503 is further configured to acquire the orientation information of the target object relative to the robot device; determine the running adjustment direction and / or running adjustment amount based on the distance information and the orientation information; and adjust the original running state of the robot device based on the running adjustment direction and / or the running adjustment amount to obtain the adjusted running state.

[0355] In one possible implementation, the control module 503 is further configured to determine the operating adjustment direction for moving the robot away from the target object based on the orientation information, and to determine the operating adjustment amount related to the distance to the target object based on the distance information, wherein the operating adjustment amount is negatively correlated with the distance value corresponding to the distance information.

[0356] In one possible implementation, the control module 503 is further configured to determine at least one of trajectory correction, velocity correction, acceleration correction, end-effector pose correction, joint position correction, and joint velocity correction based on the operation adjustment direction and / or the operation adjustment amount, and to adjust the original operating state based on at least one of the trajectory correction, velocity correction, acceleration correction, end-effector pose correction, joint position correction, and joint velocity correction.

[0357] In one possible implementation, the control module 503 is further configured to, after obtaining the adjusted operating state and before controlling the robot device to operate according to the operating state to be executed, determine whether the adjusted operating state meets the operating restrictions of the robot device; if the operating restrictions are met, then the adjusted operating state is determined as the operating state to be executed of the robot device, and the robot device is controlled to operate according to the operating state to be executed; if the operating restrictions are not met, then the operating speed of the robot device is reduced, or the robot device is controlled to stop operating.

[0358] In one possible implementation, the device 500 further includes:

[0359] The output module is used to output a reminder message when the target object is in the reminder area, the reminder message being used to indicate that the target object is in the reminder area.

[0360] The apparatus provided in this application embodiment can achieve... Figure 1 The various processes implemented in the method embodiments shown achieve the same technical effects, and will not be described again here to avoid repetition.

[0361] like Figure 6 As shown, this application embodiment also provides a robot device 600, which includes a processor 601, a memory 602, and a program or instructions stored in the memory 602 and executable on the processor 601. When the program or instructions are executed by the processor 601, they implement the above-mentioned... Figure 1 The various processes in the method embodiments can achieve the same technical effect. To avoid repetition, they will not be described again here.

[0362] This application embodiment also provides a readable storage medium storing a program or instructions that, when executed by a processor, implement the above-described functionality. Figure 1 The various processes of the method embodiments shown can achieve the same technical effect, and will not be described again here to avoid repetition.

[0363] The processor mentioned above is the processor in the robot device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0364] The steps of the methods or algorithms described in conjunction with the disclosure of this application can be implemented in hardware or by executing software instructions on a processor. The software instructions can consist of corresponding software modules, which can be stored in RAM, flash memory, ROM, EPROM, EEPROM, registers, hard disk, portable hard disk, read-only optical disk, or any other form of storage medium well known in the art. An exemplary storage medium is coupled to the processor, enabling the processor to read information from and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and storage medium can be housed within an ASIC.

[0365] Those skilled in the art will recognize that, in one or more of the examples above, the functions described in this application can be implemented using hardware, software, firmware, or any combination thereof. When implemented in software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or code on a computer-readable medium. Computer-readable media include computer storage media and communication media, wherein communication media include any medium that facilitates the transfer of a computer program from one place to another. Storage media can be any available medium accessible to a general-purpose or special-purpose computer.

[0366] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of this application. It should be understood that the above description is only a specific embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made on the basis of the technical solution of this application should be included within the scope of protection of this application.

[0367] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, embodiments of this application can take the form of entirely hardware embodiments, entirely software embodiments, or embodiments combining software and hardware aspects. Furthermore, embodiments of this application can take the form of computer program products implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0368] This application describes embodiments of methods, apparatus (systems), and computer program products according to embodiments of this application with reference to flowchart illustrations and / or block diagrams. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0369] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0370] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0371] Obviously, those skilled in the art can make various modifications and variations to the embodiments of this application without departing from the spirit and scope of this application. Therefore, if these modifications and variations to the embodiments of this application fall within the scope of the claims of this application and their equivalents, this application also intends to include these modifications and variations.

Claims

1. A control method for a robot device, characterized in that, include: Obtain distance information between the target object and the robot device, as well as risk zoning information of the robot device, wherein the risk zoning information includes the range of at least two risk zoning areas and the risk level corresponding to each risk zoning area; Based on the distance information and the risk zoning information, the target risk zoning where the target object is located is determined; The robot equipment is operated and controlled according to the target risk zone.

2. The control method according to claim 1, characterized in that, The acquisition of distance information between the target object and the robot device includes: Obtain the obstacle point cloud corresponding to the target object; Obtain the safety case corresponding to the robot device; Determine the closest distance between the obstacle point cloud and the containment structure; The nearest distance is used as the distance information between the target object and the robot device.

3. The control method according to claim 2, characterized in that, The step of obtaining the obstacle point cloud corresponding to the target object includes: Obtain the environmental point cloud of the area surrounding the robot device; The local region of interest corresponding to the target object is determined based on the environmental point cloud. The point cloud of the target object is obtained by removing the point cloud of the robot device itself from the point cloud corresponding to the local region of interest. The point cloud of the target object is used as the point cloud of the obstacle.

4. The control method according to claim 1, characterized in that, The acquisition of distance information between the target object and the robot device includes: Obtain a depth map of the area surrounding the robot device; Determine the depth distance information corresponding to the target object based on the depth map; The depth distance information is used as the distance information between the target object and the robot device.

5. The control method according to claim 4, characterized in that, Determining the depth distance information corresponding to the target object based on the depth map includes: Candidate image regions are determined based on the depth values ​​in the depth map; The object corresponding to the candidate image region is determined as the target object; Based on the depth information corresponding to the candidate image region, the depth distance information corresponding to the target object is determined.

6. The control method according to claim 5, characterized in that, The step of determining candidate image regions based on depth values ​​in the depth map includes: Image regions that meet at least one of the following conditions are identified as candidate image regions: The minimum depth in the image region is less than or equal to a first distance threshold; The depth of the low percentile in the image region is less than or equal to the second distance threshold; The number of pixels in the image region whose depth is less than or equal to the third distance threshold is greater than or equal to the number threshold; The area of ​​a connected region in the image region whose depth is less than or equal to the fourth distance threshold is greater than or equal to the area threshold. The rate of depth change in the image region is greater than or equal to a velocity threshold.

7. The control method according to claim 4, characterized in that, After determining the initial risk zone where the target object is located based on the depth distance information and the risk zone information, the method further includes: If the risk level corresponding to the initial risk partition meets the preset risk conditions, a local region of interest containing the target object is determined based on the depth map; Based on the depth information corresponding to the local region of interest, a point cloud of the target object is generated. Obtain the safety case corresponding to the robot device; Determine the closest distance between the target object point cloud and the containment structure; Based on the nearest distance and the risk zoning information, the target risk zoning where the target object is located is determined.

8. The control method according to claim 7, characterized in that, Determining the local region of interest containing the target object based on the depth map includes: Based on the depth pixel distribution of the target object in the depth map, the candidate region corresponding to the target object is determined; The candidate region is determined as the local region of interest; or, the candidate region is expanded according to at least one of the following: the position of the target object in the candidate region, the depth pixel distribution corresponding to the target object, and the distance between the target object and the boundary of the candidate region, to obtain the local region of interest.

9. The control method according to claim 7, characterized in that, The step of generating a target object point cloud corresponding to the target object based on the depth information corresponding to the local region of interest includes: A 3D point cloud transformation is performed on the depth pixels within the local region of interest to obtain a local point cloud; Based on at least one of the geometric model of the robot device, the pose information of the robot device, and the safety shell corresponding to the robot device, the point cloud of the robot device itself is removed from the local point cloud to obtain the point cloud of the target object.

10. The control method according to claim 2, 7 or 9, characterized in that, The safety containment is a dynamic safety containment determined based on the current state of the robot equipment.

11. The control method according to claim 10, characterized in that, The dynamic containment structure is determined in the following ways: Obtain the device model information and current status information of the robot device; Based on the device model information and the current state information, the real-time spatial range of the robot device is determined; The dynamic safety shell is determined based on the real-time spatial range and the preset safety margin.

12. The control method according to claim 2 or 7, characterized in that, Determining the closest distance between the obstacle point cloud and the containment, or determining the closest distance between the target object point cloud and the containment, includes: Determine the distances from multiple points in the target point cloud to the containment structure, wherein the target point cloud is the obstacle point cloud or the target object point cloud; The closest distance is determined based on the distances from the plurality of points to the containment vessel.

13. The control method according to claim 12, characterized in that, Determining the closest distance based on the distances from the plurality of points to the containment vessel includes: A distance set is determined based on the distances from the plurality of points to the containment structure; The minimum distance in the set of distances is determined as the nearest distance; or, The distance set is subjected to abnormal distance removal and / or valid point filtering. Based on the minimum distance, the average distance of the nearest multiple points, the low percentile distance, the nearest valid cluster distance, or the distance confirmed after multiple consecutive frames in the filtered distance set, a distance characterization value is determined and the distance characterization value is used as the nearest distance.

14. The control method according to any one of claims 1 to 13, characterized in that, The at least two risk zones include at least two of the following: warning zone, deceleration zone, and stop zone; The risk level of the warning zone is lower than that of the deceleration zone, and the risk level of the deceleration zone is lower than that of the stop zone.

15. The control method according to claim 14, characterized in that, The step of controlling the operation of the robot device according to the target risk zone includes: If the target risk zone is a warning zone, the robot device is controlled to maintain its current operating state and output a warning message; When the target risk zone is a deceleration zone, the original operating state of the robot equipment is adjusted to obtain an operating state that reduces the risk of collision. If the target risk zone is a stop zone, control the robot to stop running, pause running, or maintain its current position.

16. The control method according to claim 15, characterized in that, The adjustment of the original operating state of the robot device to obtain an operating state with reduced collision risk includes: Obtain the orientation information of the target object relative to the robot device; Based on the distance information and the orientation information, determine the direction and / or amount of operation adjustment; The original operating state of the robot device is adjusted according to the operating adjustment direction and / or the operating adjustment amount to obtain the adjusted operating state.

17. The control method according to claim 16, characterized in that, The step of determining the operation adjustment direction and / or operation adjustment amount based on the distance information and the orientation information includes: Based on the orientation information, determine the direction of the robot device to move away from the target object; The operational adjustment amount related to the distance to the target object is determined based on the distance information, wherein the operational adjustment amount is negatively correlated with the distance value corresponding to the distance information.

18. The control method according to claim 16, characterized in that, The step of adjusting the original operating state of the robot device according to the operating adjustment direction and / or the operating adjustment amount includes: Based on the said running adjustment direction and / or the said running adjustment amount, determine at least one of the following: trajectory correction amount, velocity correction amount, acceleration correction amount, end pose correction amount, joint position correction amount, and joint velocity correction amount; The original operating state is adjusted based on at least one of the trajectory correction amount, the velocity correction amount, the acceleration correction amount, the end pose correction amount, the joint position correction amount, and the joint velocity correction amount.

19. The control method according to claim 16, characterized in that, After obtaining the adjusted operating state, and before controlling the robot device to operate according to the operating state to be executed, the method further includes: Determine whether the adjusted operating state meets the operating restrictions of the robot device; If the operating restrictions are met, the adjusted operating state is determined as the pending operating state of the robot device, and the robot device is controlled to operate according to the pending operating state. If the operating restrictions are not met, the operating speed of the robot device will be reduced, or the robot device will be controlled to stop operating.

20. The control method according to any one of claims 1 to 18, characterized in that, The method further includes: When the target object is within the warning area, a warning message is output, which is used to indicate that the target object is in the warning area.

21. A control device for a robot, characterized in that, include: The acquisition module is used to acquire distance information between the target object and the robot device, as well as risk zoning information of the robot device. The risk zoning information includes the range of at least two risk zoning areas and the risk level corresponding to each risk zoning area. The determination module is used to determine the target risk zone where the target object is located based on the distance information and the risk zone information; The control module is used to control the operation of the robot equipment according to the target risk zone.

22. A robotic device, characterized in that, It includes a processor, a memory, and a program or instructions stored in the memory and executable on the processor, wherein the program or instructions, when executed by the processor, implement the steps of the method as claimed in any one of claims 1 to 20.

23. A readable storage medium, characterized in that, The readable storage medium stores a computer program, the computer program comprising a program or instructions that, when executed by a processor, implement the steps of the method as described in any one of claims 1 to 20.

Images