An edge control method and system for a robotic arm

Abstract

This application discloses an edge-end control method and system for a robotic arm. The method includes: marking the image acquisition time, joint sampling time, and expected command activation time at the edge end using a global control time reference, and writing joint state data and obstacle position data that has passed verification into a short-term state cache; reading the joint state data according to the image acquisition time, converting the visual perception result into obstacle position data in the robotic arm's base coordinate system, and associating it with the expected positions of the link and end effector at the expected command activation time to form a result to be verified; determining the latest available time based on joint control margin, processing time, end effector speed, obstacle distance, obstacle change rate, and depth measurement confidence; performing timing, timeliness, and joint state offset verification on the result to be verified, and generating a set of safety constraints based on the adopted results to correct the nominal control command and output it to the joint controller.

Description

Technical Field

[0001] This application relates to the field of robotic arm control technology, specifically to an edge control method and system for a robotic arm. Background Technology

[0002] Robotic arms have been widely used in industrial scenarios such as sorting, assembly, handling, and grasping. To improve the autonomous operation capability of robotic arms in complex environments, existing systems are usually equipped with depth cameras, joint encoders, torque or current sensors, and perform tasks such as visual perception, kinematic calculation, obstacle avoidance correction, and joint control at the edge. Such solutions can shorten the data transmission path, reduce dependence on remote servers, and are suitable for occasions with high response time requirements, such as high-speed sorting, narrow space grasping, and human-robot collaboration.

[0003] However, in actual control processes, there is an inevitable time lag between image acquisition, model inference, message transmission, state sampling, and joint command activation. The robotic arm configuration corresponding to the visual perception result may be different from the joint state in the current control cycle. The position of obstacles may also shift due to environmental changes. If the perception result is still directly used for obstacle avoidance or constraint calculation, it is easy to cause inaccurate obstacle position judgment, delayed control command correction, or even command conflict or collision risk.

[0004] In existing technologies, some solutions use a fixed timeout threshold to determine whether the perceived data has expired, or select the latest result only based on the order of reception. This approach does not take into account factors such as the current speed of the robotic arm, joint limit margin, obstacle distance, depth data confidence, and obstacle change rate, making it difficult to adapt to the data availability judgment requirements under different risk conditions. Summary of the Invention

[0005] This application provides an edge control method and system for a robotic arm to at least solve some of the technical problems existing in the related technologies described above.

[0006] According to a first aspect of the embodiments of this application, an edge-end control method for a robotic arm is provided, comprising: At the edge, the image acquisition time, joint sampling time, and expected command effective time are marked with a global control time reference, and the joint status data and the obstacle position data that has been submitted for verification the most recently are written into the short-term status cache; The edge end generates visual perception results for the image, reads joint state data according to the image acquisition time, converts the visual perception results into obstacle position data in the robot arm base coordinate system, and associates them with the expected positions of each link and end effector at the expected time of command activation to form the result to be verified. In each control cycle, the joint control margin is calculated from the current joint status data and the obstacle position data that was last submitted for verification. Based on the joint control margin, actual processing time, end effector linear velocity, minimum obstacle distance, obstacle position change estimation speed, and depth measurement confidence, the latest available time for the verification result is determined. The verification results are subjected to source data timing checks, latest available time checks, and joint state offset checks. If all three checks pass, the result is recorded as accepted; otherwise, it is discarded. A set of safety constraints is generated based on the adopted results. The nominal control command is then modified using the set of safety constraints to obtain the modified control command, which is then output to the joint controller.

[0007] As an optional approach, the short-term state cache also writes the end effector pose and spatial position of each link obtained from the forward kinematics, the joint control command output in the previous control cycle and the expected effective time of the command, and retains the written data of the most recent control cycles in a circular manner.

[0008] As an optional approach, the visual perception results include the position and corresponding depth value of the obstacle region in the image coordinate system; the edge end obtains the spatial position of each link and end effector at the time of image acquisition by forward kinematics based on the joint state data at the time of image acquisition, and converts the position and corresponding depth value of the obstacle region in the image coordinate system into obstacle position data in the robot arm base coordinate system based on the spatial position.

[0009] As an optional approach, when determining the latest available time at the edge, the time difference between the image acquisition time and the inference completion time is used as the actual processing time, and the magnitude of the linear velocity component obtained by multiplying the current joint angular velocity by the Jacobian matrix is ​​used as the linear velocity of the end effector; the obstacle position change estimation velocity takes a preset upper limit value in the first frame, and subsequent frames are determined according to the position difference and time difference of the same obstacle in two consecutive frames; the depth measurement confidence is determined based on the depth camera.

[0010] As an optional approach, determining the latest available time also includes dividing the minimum obstacle distance by the sum of the end effector linear velocity, the estimated obstacle position change velocity, and a preset small positive number to obtain the estimated time; multiplying the estimated time by the depth measurement confidence and the monotonically increasing scaling function value with respect to joint control margin, and then subtracting the actual processing time to obtain the effective time period; when the effective time period is greater than zero, adding the effective time period to the inference completion time to obtain the latest available time; when the effective time period is less than or equal to zero, discarding the results to be verified.

[0011] As an optional solution, source data timing checks and latest available time checks include: the source data acquisition time is the image acquisition time of the image corresponding to the visual perception result; when there is already an adopted result in the current control cycle and the source data acquisition time of the result to be verified is earlier than the source data acquisition time of the adopted result, the edge end discards the result to be verified; when the current time exceeds the latest available time, the edge end discards the result to be verified.

[0012] As an optional approach, in the joint state offset check, the edge end reads the joint angle vector at the source data acquisition time and the joint angle vector in the current control cycle of the result to be verified, and calculates the norm of the difference between the two as the joint state offset; the edge end takes the maximum value of the estimated displacement of each joint as the upper limit of allowable offset, and discards the result to be verified when the joint state offset exceeds the product of the upper limit of allowable offset and the preset safety factor.

[0013] As an optional approach, a quadratic programming model of the control barrier function is established at the edge. The set of safety constraints is used as the constraint conditions, and the deviation between the corrected control command and the nominal control command is minimized as the objective function to obtain the corrected control command. The set of safety constraints includes the minimum allowable distance between the end effector and each obstacle, the minimum allowable distance between the geometric envelope of each link and each obstacle, the position limits of each joint, the velocity limits of each joint, and the acceleration limits of each joint. The nominal control command is generated by the path planning module or the end-to-end control model in the current control cycle, and the nominal control command is one of the following: joint angular velocity command, joint angle increment command, or joint torque command.

[0014] As an alternative approach, when the last valid visual perception result has exceeded the latest available time and the new visual perception result has not passed the submission verification, the edge device generates temporary safety constraints based on the obstacle position data, obstacle position change estimated speed, current end effector linear velocity, and braking capability in the last valid result.

[0015] According to a second aspect of the embodiments of this application, an edge-end control system for a robotic arm is also provided, comprising: The time stamping and caching module is used at the edge to mark the image acquisition time, joint sampling time, and expected command effective time with a global control time reference, and writes the joint status data and the obstacle position data that has been submitted for verification the most recently into the short-term status cache. The verification result generation module is used to generate visual perception results for images at the edge end. It reads joint state data according to the image acquisition time, converts the visual perception results into obstacle position data in the robot arm base coordinate system, and associates them with the expected positions of each link and end effector at the expected time of command activation to form the verification result. The latest available time determination module is used for each control cycle. It calculates the joint control margin from the current joint state data and the obstacle position data that was last submitted for verification. Based on the joint control margin, actual processing time, end effector linear velocity, minimum obstacle distance, obstacle position change estimation speed and depth measurement confidence, it determines the latest available time of the verification result. The submission verification module is used to perform source data timing checks, latest available time checks, and joint state offset checks on the verification results. If all three checks pass, the result is recorded as accepted; otherwise, it is discarded. The control command correction module is used to generate a set of safety constraints based on the adopted results, correct the nominal control commands with the set of safety constraints, obtain the corrected control commands, and output them to the joint controller.

[0016] This application achieves this by uniformly marking image acquisition time, joint sampling time, and expected command activation time at the edge, and by storing joint states, obstacle positions, and historical control information in a short-term state cache. This ensures that visual perception results have clear temporal and state basis before entering control calculations. Based on this, the latest usable time of the results is determined according to joint control margin, actual processing time, end effector speed, minimum obstacle distance, obstacle change rate, and depth measurement confidence level, avoiding the problem of fixed timeout thresholds being difficult to adapt to different working conditions. Simultaneously, by checking the source data timing, the latest usable time, and joint state offset, outdated or incompatible perception results are eliminated, reducing the risk of misjudgment caused by old data participating in obstacle avoidance calculations. The verified and adopted results are used to generate a safety constraint set and modify nominal control commands, improving the obstacle avoidance reliability and operational safety of the robotic arm in high-speed motion, narrow-space operations, and obstacle-prone scenarios while ensuring control continuity.

[0017] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Furthermore, no embodiment in this disclosure is required to achieve all the effects described above. Attached Figure Description

[0018] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0019] Figure 1 This is a schematic diagram of an edge control method for a robotic arm provided in an embodiment of the present disclosure.

[0020] Figure 2 A flowchart for determining the latest available time provided in embodiments of this disclosure.

[0021] Figure 3This is a flowchart illustrating the verification process for the verification result provided in this embodiment of the disclosure.

[0022] Figure 4 This is a schematic diagram of the structure of an edge control system for a robotic arm provided in an embodiment of this disclosure.

[0023] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure. Detailed Implementation

[0024] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0025] This embodiment provides an edge-end control method for a robotic arm, applicable to robotic arm systems where visual perception, kinematic calculation, obstacle avoidance correction, and joint control operate on the same edge computing board. The robotic arm can be a six-axis or seven-axis industrial robotic arm, or a collaborative robotic arm. The hardware includes the robotic arm body and its joint encoders, joint torque sensors or current sensors, at least one depth camera, an edge computing board, and a joint controller. The software can employ a robot operating system or a functionally equivalent robot software framework. In situations such as high-speed sorting, narrow-space grasping, joint approach to limits, or end-effector approach to obstacles, the time difference between image acquisition, model inference, message transmission, and joint command execution can cause deviations between the perception results and the current joint state, leading to obstacle avoidance inaccuracies or command conflicts. This method determines whether each perception result is still suitable for the current control cycle based on the real-time joint state of the robotic arm and adjusts the control behavior accordingly.

[0026] The implementation process of the method described in this application will be described in detail below with reference to specific embodiments. It should be noted that this embodiment is only used to explain this application and is not intended to limit the scope of protection of this application. Conventional adjustments or substitutions of each step by those skilled in the art without departing from the concept of this application should be included in the scope of protection of this application.

[0027] Please see Figure 1 , Figure 1 A flowchart of the edge-end control method for a robotic arm according to an embodiment of the present invention is shown, as follows: Figure 1 As shown, the method includes steps S1-S5: In step S1, the edge end marks the image acquisition time, joint sampling time, and expected command effective time with a global control time reference, and writes the joint state data and the obstacle position data that was last submitted for verification into the short-term state cache.

[0028] Specifically, after the edge computing board is started, a global control time reference is established using the system clock of the edge computing board. The hardware timestamp generated when the depth camera acquires images is converted to this global control time reference after clock synchronization. Each joint encoder, torque sensor or current sensor reports joint status data at a fixed sampling frequency, and each set of data carries the sampling time under the global control time reference. The expected effective time of the joint control command is also marked according to this time reference. The above clock synchronization ensures that the image acquisition time, joint sampling time and command effective time can be compared under the same time coordinate.

[0029] A short-term state cache is maintained at the edge to store joint state data and related calculation results from the most recent control cycles. Joint state data includes joint angles, angular velocities, and torques or currents for each joint. In addition to joint state data, the short-term state cache also contains the end effector pose and spatial positions of each link calculated from forward kinematics, the joint control commands output in the previous control cycle and their expected activation times, the obstacle position data that passed the most recent submission verification, and the corresponding depth measurement confidence level. Each written data entry is marked with a write time marker. The cache is managed in a circular manner, overwriting the oldest data after a new entry is written, always retaining the state information from the most recent time period.

[0030] In step S2, the edge end generates visual perception results for the image, reads joint state data according to the image acquisition time, converts the visual perception results into obstacle position data in the robot arm base coordinate system, and associates them with the expected positions of each link and end effector at the expected time of command activation to form a result to be verified.

[0031] Specifically, the visual perception stage employs a lightweight instance segmentation model to identify obstacles. In some embodiments, the YOLOv8n-seg network can be used. This network is a single-stage instance segmentation structure, with the backbone network responsible for extracting multi-scale features and the neck network fusing these features. The detection head outputs the bounding box, category, confidence score, and pixel-wise segmentation mask of the target region. The model input is a color image or depth image captured by a depth camera, and the output visual perception results include the position, segmentation mask, confidence score, and corresponding depth value of the obstacle region in the image coordinate system. This model is pre-trained on a labeled dataset using common training methods in the fields of object detection and instance segmentation, and is deployed in inference mode on an edge computing board. This embodiment will not be elaborated further here.

[0032] In one embodiment, after the visual model completes the inference of a frame of image, the edge device writes the visual perception result, along with the acquisition time and inference completion time of the frame of image, into the queue of results to be verified. Specifically, before writing, the coordinate transformation includes the edge device retrieving the joint state data corresponding to the image acquisition time from the short-term state cache, obtaining the spatial position of each link and end effector at the image acquisition time through forward kinematics, and converting the pixel coordinates of the obstacle area into three-dimensional obstacle position data in the robot arm base coordinate system by combining the depth value.

[0033] Based on the above, the edge device extrapolates the joint angle at the expected effective time of the joint control command from the current joint state according to the expected effective time of the joint control command in the current control cycle, the current joint angle, the angular velocity of each joint, and the acceleration limit of each joint. It then obtains the expected position of each link and end effector at the expected effective time through forward kinematics. After associating the obstacle position data with the above expected positions, it writes the data into the queue of results to be verified. Thus, when subsequent steps evaluate obstacle constraints, the spatial relationship used corresponds to the expected effective time of the command, rather than the image acquisition time.

[0034] In step S3, for each control cycle, the joint control margin is calculated from the current joint state data and the obstacle position data that was last submitted for verification. The latest available time for the verification result is determined based on the joint control margin, actual processing time, end effector linear velocity, minimum obstacle distance, obstacle position change estimated velocity, and depth measurement confidence.

[0035] Please see Figure 2 , Figure 2 A flowchart illustrating the latest available time determination provided in an embodiment of this disclosure is shown, such as... Figure 2 As shown in box 201, in each control cycle, the joint control margin is calculated from the current joint state data and the obstacle position data that was last submitted for verification.

[0036] In some embodiments, at the beginning of each control cycle, the edge device reads the current joint state data and the obstacle position data that has most recently passed the submission verification from the short-term state cache, and uses these as inputs to calculate the joint control margin. If the edge device starts up and there is no obstacle position data that has passed the submission verification yet, the obstacle position data that has most recently passed the submission verification is recorded as empty, and the obstacle distance margin is set to 1 as if there are no known obstacles. After the first frame of visual perception results passes the submission verification, the edge device updates the short-term state cache with the obstacle position data corresponding to the visual perception results, and participates in the joint control margin calculation in subsequent control cycles.

[0037] Joint control margin characterizes the minimum safe space remaining when the robotic arm continues to execute nominal control commands in its current state. The smaller the value, the closer the robotic arm is to the boundary of a certain constraint, and the less suitable it is to continue moving according to the original command. Thus, unifying the safety judgment criteria scattered across different physical dimensions into a single comparable quantity can provide a unified adjustment basis for subsequent determination of effective time periods, generation of control task deadlines, and submission verification.

[0038] Specifically, the edge layer first calculates the safety margin, with each margin normalized to between 0 and 1. For the normalized result of each dimension's safety margin, if the calculated value is less than 0, it is set to 0; if the calculated value is greater than 1, it is set to 1. The safety margin specifically includes: Joint position margin: for the first Each joint, take the current joint angle. Reaching the upper limit With the lower limit The smaller of the distances is divided by the total range of the joint's stroke, i.e.:

[0039] in and These are the lower and upper mechanical limits of the joint, respectively. The minimum positional margin for each joint is taken as denoted as . This margin reflects how much available travel each joint has from its position limit; the closer to the limit, the lower the margin.

[0040] Braking distance margin: for the first Each joint, based on the current joint angular velocity and the maximum braking deceleration of the joint The angular displacement required for braking is estimated using the uniform deceleration model, i.e.:

[0041] Subtract the distance from the current joint angle to the nearest limit. Divide by the total travel range to get the first... The braking distance margin of each joint; if the braking angular displacement of a joint has exceeded its distance to the nearest limit, the braking distance margin of that joint is recorded as zero; the minimum braking distance margin of each joint is taken as denoted as . This margin measures whether the joint can still stop within the limit range after emergency braking at the current speed.

[0042] The maximum braking deceleration of the joint can be obtained in advance based on the rated braking capacity of the joint motor and reducer, the maximum allowable braking torque, the equivalent moment of inertia of the joint and the current limiting parameters of the controller. Alternatively, it can be measured during the equipment commissioning stage through braking tests under no-load and typical load conditions and written into the edge end parameter table.

[0043] Obstacle distance margin: At the edge end, the Euclidean distance from the geometric envelope of each link and the end effector to the nearest obstacle is calculated using forward kinematics. The minimum value among all links and the end effector is taken as the minimum obstacle distance. ,Will Divide by the pre-defined upper limit of the safe distance and truncate to between 0 and 1 to obtain the obstacle distance margin. The upper limit of the safe distance indicates that it is considered completely safe when the distance is exceeded. Its value can be determined by offline simulation or actual measurement, for example, by calibrating based on the maximum motion envelope of the robotic arm and the distribution of obstacles in a typical working environment.

[0044] Joint velocity margin: for the first For each joint, the difference between the absolute value of the maximum permissible angular velocity and the current angular velocity is divided by the maximum permissible angular velocity, and the minimum value for each joint is recorded as . This margin reflects the clearance for the speed limits of each joint. The joint angular velocity can be directly obtained from the joint state data reported by each joint encoder or joint controller at a fixed sampling frequency, with the sampling time marked using a global control time reference. Alternatively, when only the joint angle is obtained, the current joint angular velocity can be calculated based on the difference in joint angles and the time difference between adjacent sampling times. Joint torque margin: for the first For each joint, take the difference between the absolute value of the maximum permissible torque and the current torque, divide it by the maximum permissible torque, and record the minimum value for each joint as . When the torque borne by a joint is close to its upper limit of output capacity, the margin is low, which means that there is insufficient torque reserve available for acceleration or deceleration.

[0045] End-effector amplification margin: The mapping characteristics from joint motion to end-effector motion in the current configuration are evaluated using the Jacobian matrix of the robotic arm. When the robotic arm approaches a singular configuration, the condition number of the Jacobian matrix increases, and even small changes in joint angles can lead to significant displacement of the end-effector position, increasing the risk of collision. The ratio of the minimum singular value to the maximum singular value of the Jacobian matrix is ​​calculated at the edge and used as the end-effector amplification margin. The value approaches 1 when far from a singular configuration and tends to zero when close to a singular configuration.

[0046] At the edge, the minimum value among the normalized safety margins of the above six dimensions is taken as the joint control margin for the current control cycle:

[0047] Taking the minimum value allows the margin to be determined by the most pressing constraint dimension at present; for example, even if the joint position and velocity margin are high, if the end effector is very close to the obstacle, the obstacle distance margin dominates the value of the margin, and the subsequent effective time period and task deadline are tightened accordingly; after the joint control margin is calculated, the edge end writes it into the record of the current control cycle in the short-term state cache.

[0048] Optionally, in some embodiments, weighting coefficients can be configured for each dimension of safety margin, making the margin more sensitive to specific dimensions; the value range of each weighting coefficient is between 0 and 1, and can be determined through offline calibration: run the robotic arm under various typical working conditions, record the distribution characteristics of each dimension's margin and the actual occurrence of constraint violations, and adjust the weights accordingly.

[0049] In box 202, the latest available time for the verification result is determined based on the joint control margin, actual processing time, end effector linear velocity, minimum obstacle distance, obstacle position change estimated velocity, and depth measurement confidence.

[0050] Existing solutions typically use a fixed timeout threshold to determine whether the perception results have expired. This approach fails to adjust to the level of risk associated with the real-time movement of the robotic arm: in high-risk situations, data that deviates too much from the current state may be used, while in low-risk situations, usable data may be discarded prematurely. Therefore, this embodiment introduces an effective time period, which is the range of time from the moment the visual perception result is acquired that it can still participate in the generation of control commands. This time period is dynamically determined by the current movement state of the robotic arm and changes in the environment.

[0051] Specifically, for a single frame of visual perception result, the input quantities used to determine its effective time period at the edge include: the actual processing time obtained from the image acquisition time to the inference completion time. End effector linear velocity The minimum Euclidean distance from each link and end effector to the nearest obstacle. Estimating the velocity of the obstacle's position change Current joint control margin and depth measurement confidence .

[0052] The linear velocity of the end effector The velocity estimate for obstacle position change is obtained by multiplying the current joint angular velocity by the Jacobian matrix and taking the magnitude of the linear velocity component. In the first frame after the visual model is started, a preset upper limit value is taken. This preset upper limit value is a non-negative velocity quantity, obtained through offline simulation or actual measurement calibration based on the maximum expected moving velocity of the obstacle relative to the robot arm's base coordinate system in the robot arm's working scene. Because adjacent frame data is lacking at this time, a conservative estimate is used, making the effective time period of the first frame shorter. Subsequent frames are calculated based on the position difference and time difference of the same obstacle in two consecutive frames. Depth measurement confidence level. Provided by the depth camera, the value ranges from 0 to 1, reflecting the reliability of the depth data.

[0053] In one embodiment, the formula for calculating the effective time period is:

[0054] in the formula This is a preset small positive number to prevent the denominator from being zero. Regarding joint control margin The scaling function is monotonically increasing; a larger margin results in a larger scaling factor, thus extending the effective time period; a smaller margin results in a smaller scaling factor, thus shortening the effective time period. In one implementation... Using linear form:

[0055] and The lower and upper bound coefficients are pre-calibrated and are both positive numbers. They can be determined through simulation experiments. For example, the control stability and obstacle detection timeliness corresponding to different values ​​can be tested under various typical motion scenarios, and the range with good overall performance can be selected.

[0056] This represents the estimated time required for the current end-effector velocity and the obstacle's speed to reduce from the current distance to contact. and Scaling is applied to this time; a more conservative value is used when the margin is small or the reliability of depth measurement is low; subtract the actual processing time. Then we obtain the remaining available time from the moment the reasoning is completed.

[0057] Among them, the effective period When the value is greater than zero, the edge terminal adds the inference completion time. The latest available time for the visual perception result of a frame is recorded. When the effective time period is less than or equal to zero, it indicates that the result of the frame is no longer suitable for use in the current control state when the inference is completed. The edge end marks it as expired and discards it, and it will no longer enter the subsequent submission and verification stage. In the subsequent control cycle, if the current time exceeds the latest available time of a certain frame result, the result is also regarded as expired. In this way, whether to use a frame of visual result is determined by the current motion risk level of the robotic arm and the quality of the perception data, rather than a fixed time threshold.

[0058] In step S4, source data timing check, latest available time check, and joint state offset check are performed on the verification result. If all three checks pass, it is recorded as an accepted result; otherwise, it is discarded.

[0059] Submission verification involves the edge device making an admission judgment on the visual perception results to be entered into each control cycle. Even if a result is correct in content, insufficient timeliness or a significant deviation between the corresponding robotic arm configuration and the current state can still cause command conflicts or obstacle avoidance inaccuracies. Therefore, multiple checks are required before the result enters the control calculation. For visual perception results, the source data acquisition time is the image acquisition time of the frame image corresponding to the visual perception result; the inference completion time and the time of writing to the verification result queue are not considered as source data acquisition times.

[0060] Specifically, such as Figure 3 As shown, Figure 3 A flowchart illustrating the verification process for the result to be verified, provided in an embodiment of this disclosure, is shown. In block 301, source data timing is checked. When an adopted result already exists in the current control cycle, the edge device compares the source data acquisition time on which the result to be entered into the current control cycle depends with the source data acquisition time of the adopted result. If the source data acquisition time of the result to be entered is earlier than the source data acquisition time of the adopted result, it indicates that the result was generated based on older input data. The edge device discards this result, preventing it from overwriting a result already adopted based on updated data. This check prevents older data from replacing newer data due to message transmission delays.

[0061] In box 302, the latest available time is checked. The edge reads the latest available time determined in the previous steps and compares it with the current time. If the current time has exceeded the latest available time, the result is considered expired and the edge discards the result. This check ensures that the result entering the control cycle is still timely under the current motion risk level.

[0062] In box 303, joint state offset check. The edge device retrieves the joint angle vector at the source data acquisition time corresponding to the result to be entered into the current control cycle from the short-time state buffer, and reads the joint angle vector of the current control cycle, calculating the norm of the difference between the two as the joint state offset; at the same time, the edge device performs displacement estimation for each joint, that is, multiplying the absolute value of the current joint angular velocity by the time difference from the source data acquisition time to the current time, and adding the braking deceleration additional displacement obtained from the maximum braking deceleration, to obtain the displacement estimation result of the joint; wherein, the braking deceleration additional displacement can be calculated in the same way as the angular displacement required for braking in the previous steps, which will not be elaborated here.

[0063] The maximum value of the displacement estimation result of each joint is taken as the upper limit of allowable offset. If the offset of the joint state exceeds the product of the upper limit of allowable offset and the preset safety factor, it means that the deviation between the robot arm configuration when the result was generated and the current configuration has exceeded the allowable level within the current motion capability range, and the result is discarded at the edge. The preset safety factor is between 0 and 1 and can be calibrated by experiment. The smaller the value, the stricter the verification judgment.

[0064] In box 304, if all three checks pass, the result is recorded as adopted; otherwise, it is discarded. When all three checks pass, the edge device records the result as adopted and participates in the instruction generation of the current control cycle. If any check fails, the result is discarded directly and not written into the adopted record. The edge device continues to perform control calculations based on the adopted results already existing in the current cycle or the results of the previous cycle that have not yet exceeded the latest available time. If the results of the previous cycle have exceeded the latest available time, temporary safety constraints are generated according to the transitional processing for missing visual perception results. The control loop is not suspended due to the discarding of a certain result.

[0065] In some embodiments, during actual operation, multiple visual perception results from the visual inference node may arrive sequentially within the same control cycle, and the arrival order may not be consistent with the order of the source data acquisition time. The edge terminal performs the above three checks on each arriving result. If multiple results pass the three checks, the edge terminal adopts the result with the latest source data acquisition time. The adopted result is not replaced by the later result with the earlier source data acquisition time. This process avoids the old result accidentally overwriting the control result generated based on the updated data due to transmission queuing delay, thereby maintaining the consistency between the control command and the latest perception state.

[0066] In some embodiments, optionally, the edge terminal uses the start time of the current control cycle as a reference to generate the deadline of each control task within the current cycle based on the joint control margin; the control tasks are divided into safety-related tasks and non-safety-related tasks; wherein safety-related tasks may include joint control margin calculation, submission verification, nominal control command correction, and joint control command output; non-safety-related tasks include visual model inference, target re-identification, and log recording; the deadline of safety-related tasks is the product of the control cycle length and a coefficient that decreases as the joint control margin decreases, the lower the margin, the faster the safety task must be completed; the deadline of non-safety-related tasks is the control cycle length.

[0067] In one implementation, the coefficient according to It is confirmed that, among them, Allowance for joint control. This is the lower limit of the coefficient. It is the upper limit of the coefficient, and satisfies ; and Through simulation or actual testing, it is determined that safety-related tasks can be completed within the corresponding control cycle.

[0068] When the joint control margin is lower than the preset emergency threshold, the edge terminal skips non-safety-related tasks in the current control cycle and only executes safety-related tasks. The emergency threshold can be determined through simulation or actual testing, and its value ranges from 0 to 1. Control tasks that are not skipped are ordered and executed according to the earliest due date first (EDF) algorithm. The EDF algorithm selects the ready task with the earliest due date to run first at each scheduling time.

[0069] In step S5, a set of safety constraints is generated based on the adopted results. The nominal control command is then modified using the set of safety constraints to obtain the modified control command, which is then output to the joint controller. In some embodiments, for the accepted results that have passed the submission verification, the edge device generates a set of safety constraints for the current control cycle. The set of safety constraints includes: the minimum allowable distance between the end effector and each obstacle, the minimum allowable distance between the geometric envelope of each link and each obstacle, the position limit of each joint, the velocity limit of each joint, and the acceleration limit of each joint. The specific values ​​of the above constraints can be set according to actual needs, and this embodiment does not impose specific restrictions.

[0070] In each control cycle, the path planning module or end-to-end control model first generates a nominal control command, which can be a joint angular velocity command, a joint angle increment command, or a joint torque command. In one embodiment, a control barrier function quadratic programming (CBF-QP) model is established at the edge, using the set of safety constraints as constraints and minimizing the deviation between the corrected control command and the nominal control command as the objective function to obtain the corrected control command. Specifically, CBF-QP uses the nominal control command as the desired solution and incorporates inequality constraints derived from the control barrier function when solving the quadratic programming. When the robotic arm is far from the constraint boundary, the correction amount is minimal; when it approaches the constraint boundary, the solver automatically increases the correction magnitude of the relevant joint command components to meet the safety conditions. This allows the system to maintain compliance with safety constraints without frequent emergency stops, reducing path jitter caused by excessive avoidance. After the corrected control command is generated, it is output to the joint controller for execution.

[0071] In some embodiments, optionally, when the last valid visual perception result has exceeded the latest available time and the new visual perception result fails the submission verification, the edge device generates temporary safety constraints based on the obstacle position data, the estimated speed of obstacle position change, the current end effector linear velocity, and the braking capability in the last valid result. The temporary safety constraints include an increased minimum allowable distance and a reduced nominal control command amplitude, causing the robotic arm to gradually reduce its speed during the period of missing perception data.

[0072] Specifically, the edge device estimates the speed based on the obstacle position data and the change in obstacle position in the last valid frame, and extrapolates the obstacle position to the current moment. Then, based on the current linear velocity and braking capability of the end effector, it determines the distance at which the end effector may continue to approach the obstacle during braking, and uses this distance as the increase in the minimum allowable distance. The edge device reduces the nominal control command amplitude according to the ratio of this increase to a pre-calibrated upper limit of the safe distance, and the reduced nominal control command amplitude is not lower than the preset minimum output amplitude. The lower the joint control margin or the longer the time elapsed since the last valid frame expired, the larger the increase and the greater the reduction in the nominal control command amplitude.

[0073] After the new visual perception result is submitted for verification, the edge device replaces the temporary security constraints with the security constraint set corresponding to the new result and restores normal control. This transition process is only used for periods when the perception result is temporarily missing.

[0074] Therefore, using the above method, the decision to adopt each frame's visual perception result at the edge is jointly determined by three dimensions: joint control margin, effective time period, and joint state offset. This replaces the traditional approach of using fixed timeout thresholds or fixed queue lengths. The joint control margin unifies multi-dimensional constraint information into a single comparable quantity; the effective time period dynamically adjusts the expiration judgment according to the level of motion risk and environmental changes; and the three conditions for submission for verification filter results unsuitable for the current state from the perspectives of time, temporal sequence, and spatial offset. The combination of these three factors ensures that the scheduling of control tasks and the selection of perception results can follow the real-time state changes of the robotic arm, maintaining control stability and obstacle avoidance accuracy even under conditions of computing power fluctuations and message out-of-order delivery.

[0075] Please see Figure 4 , Figure 4 This is a schematic diagram of the edge control system of a robotic arm provided in an embodiment of this application. As shown in the figure, the system includes: The time stamping and caching module 401 is used at the edge to mark the image acquisition time, joint sampling time and the expected effective time of the command with a global control time reference, and to write the joint status data and the obstacle position data that has been submitted for verification the most recently into the short-term status cache. The verification result generation module 402 is used to generate visual perception results for the image at the edge end, read joint state data according to the image acquisition time, convert the visual perception results into obstacle position data in the robot arm base coordinate system, and associate them with the expected positions of each link and end effector at the expected time of command activation to form the verification result; The latest available time determination module 403 is used for each control cycle to calculate the joint control margin from the current joint state data and the obstacle position data that was last submitted for verification. Based on the joint control margin, actual processing time, end effector linear velocity, minimum obstacle distance, obstacle position change estimation speed and depth measurement confidence, the latest available time of the verification result is determined. Submit the verification module 404, which is used to perform source data timing checks, latest available time checks, and joint state offset checks on the verification results. If all three checks pass, the result is recorded as accepted; otherwise, it is discarded. The control command correction module 405 is used to generate a set of safety constraints based on the adopted results, correct the nominal control commands with the set of safety constraints, obtain the corrected control commands, and output them to the joint controller.

[0076] Each processing unit and / or module in the embodiments of this application can be implemented by an analog circuit that implements the functions described in the embodiments of this application, or by software that executes the functions described in the embodiments of this application.

[0077] Please see Figure 5It shows a schematic diagram of the structure of an electronic device according to an embodiment of this application, which can be used to implement... Figure 1 The method in the illustrated embodiment. (As shown) Figure 5 As shown, the electronic device may include: The system includes at least one processor 501, at least one network interface 504, a user interface 503, a memory 505, and at least one communication bus 502. The communication bus 502 is used to enable connection and communication between the components. The user interface 503 may include buttons, and optionally include a standard wired or wireless interface. The network interface 504 may include, but is not limited to, a Bluetooth module, an NFC module, a Wi-Fi module, etc.

[0078] The processor 501 may include one or more processing cores and connect to various parts within the electronic device through various interfaces and lines. It implements various functions and data processing of the electronic device by running or executing instructions, programs, code sets, or instruction sets stored in the memory 505, and by accessing data in the memory 505. Optionally, the processor 501 may be implemented using at least one hardware form of DSP, FPGA, or PLA. The processor 501 may also integrate one or more combinations of CPU, GPU, and modem.

[0079] Memory 505 may include random access memory (RAM) or read-only memory (ROM). Optionally, memory 505 includes a non-transitory computer-readable medium for storing instructions, programs, code, code sets, or instruction sets. Memory 505 may be divided into a program storage area and a data storage area, wherein the program storage area can be used to store instructions for implementing an operating system and instructions for implementing the foregoing method embodiments; the data storage area can be used to store data related to the relevant method embodiments. Memory 505 may also be at least one storage device located remotely from processor 501. Figure 5 As shown, the memory 505, which serves as a computer storage medium, may contain an operating system, a network communication module, a user interface module, and program instructions.

[0080] In particular, the methods and / or embodiments in this application can be implemented as computer software programs. For example, the embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowchart. When the computer program is executed by processor 501, it performs the functions defined in the methods of this application.

[0081] Another embodiment of this application provides a storage medium storing computer program instructions thereon, which can be executed by a processor to implement the methods and / or technical solutions of any one or more embodiments of this application.

[0082] In the above embodiments, the descriptions of each embodiment have different focuses. Parts not described in detail in a certain embodiment can be referred to in the relevant descriptions of other embodiments. The above descriptions are merely preferred embodiments of this application and explanations of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in this application is not limited to the technical solutions formed by specific combinations of the above technical features, but should also cover other technical solutions formed by arbitrary combinations of the above technical features or their equivalent features without departing from the inventive concept.

Claims

1. An edge-end control method for a robotic arm, characterized in that, include: At the edge, the image acquisition time, joint sampling time, and expected command effective time are marked with a global control time reference, and the joint status data and the obstacle position data that has been submitted for verification the most recently are written into the short-term status cache; The edge end generates visual perception results for the image, reads joint state data according to the image acquisition time, converts the visual perception results into obstacle position data in the robot arm base coordinate system, and associates them with the expected positions of each link and end effector at the expected time of command activation to form the result to be verified. In each control cycle, the joint control margin is calculated from the current joint status data and the obstacle position data that was last submitted for verification. Based on the joint control margin, actual processing time, end effector linear velocity, minimum obstacle distance, obstacle position change estimation speed, and depth measurement confidence, the latest available time for the verification result is determined. The verification results are subjected to source data timing checks, latest available time checks, and joint state offset checks. If all three checks pass, the result is recorded as accepted; otherwise, it is discarded. A set of safety constraints is generated based on the adopted results. The nominal control command is then modified using the set of safety constraints to obtain the modified control command, which is then output to the joint controller.

2. The method according to claim 1, characterized in that, The short-term state buffer also writes the end effector pose and spatial position of each link obtained from positive kinematics, the joint control command output in the previous control cycle and the expected effective time of the command, and retains the written data of the most recent control cycles in a circular manner.

3. The method according to claim 2, characterized in that, The visual perception results include the position of the obstacle region in the image coordinate system and its corresponding depth value; Based on the joint state data at the moment of image acquisition, the edge end obtains the spatial position of each link and end effector at the moment of image acquisition through forward kinematics, and converts the position and corresponding depth value of the obstacle area in the image coordinate system into obstacle position data in the robot arm base coordinate system based on the spatial position.

4. The method according to claim 3, characterized in that, When determining the latest available time at the edge, the time difference between the image acquisition time and the inference completion time is used as the actual processing time, and the magnitude of the linear velocity component obtained by multiplying the current joint angular velocity by the Jacobian matrix is ​​used as the linear velocity of the end effector. The estimated velocity of obstacle position change is set to a preset upper limit in the first frame, and in subsequent frames it is determined by the position difference and time difference of the same obstacle in two consecutive frames; the confidence level of depth measurement is determined based on the depth camera.

5. The method according to claim 4, characterized in that, Determining the latest available time also includes dividing the minimum obstacle distance by the sum of the end effector linear velocity, the estimated obstacle position change velocity, and a preset small positive number to obtain the estimated time; multiplying the estimated time by the depth measurement confidence and the monotonically increasing scaling function value with respect to the joint control margin, and then subtracting the actual processing time to obtain the effective time period; If the valid time period is greater than zero, the inference completion time plus the valid time period is used as the latest available time. If the valid time period is less than or equal to zero, the result to be verified is discarded.

6. The method according to claim 5, characterized in that, The source data timing check and latest available time check include: the source data acquisition time is the image acquisition time of the image corresponding to the visual perception result; when there is already an adopted result in the current control cycle and the source data acquisition time of the result to be verified is earlier than the source data acquisition time of the adopted result, the edge end discards the result to be verified; when the current time exceeds the latest available time, the edge end discards the result to be verified.

7. The method according to claim 6, characterized in that, During the joint state offset check, the edge end reads the joint angle vector at the source data acquisition time and the joint angle vector in the current control cycle of the result to be verified, and calculates the norm of the difference between the two as the joint state offset; the edge end takes the maximum value of each joint displacement estimation result as the upper limit of allowable offset, and discards the result to be verified when the joint state offset exceeds the product of the upper limit of allowable offset and the preset safety factor.

8. The method according to claim 7, characterized in that, A quadratic programming model of the control barrier function is established at the edge, with the set of safety constraints as the constraint conditions and the deviation between the corrected control command and the nominal control command as the objective function to minimize, and the corrected control command is obtained by solving the problem. The set of safety constraints includes the minimum allowable distance between the end effector and each obstacle, the minimum allowable distance between the geometric envelope of each link and each obstacle, the position limit of each joint, the velocity limit of each joint, and the acceleration limit of each joint; the nominal control command is generated by the path planning module or the end-to-end control model in the current control cycle, and the nominal control command is one of the following: joint angular velocity command, joint angle increment command, or joint torque command.

9. The method according to claim 8, characterized in that, When the last valid visual perception result has exceeded the latest available time and the new visual perception result has not passed the submission verification, the edge device generates temporary safety constraints based on the obstacle position data, the speed estimated by the change of obstacle position, the current end effector linear velocity, and the braking capability in the last valid result.

10. An edge-end control system for a robotic arm, characterized in that, include: The time stamping and caching module is used at the edge to mark the image acquisition time, joint sampling time, and expected command effective time with a global control time reference, and writes the joint status data and the obstacle position data that has been submitted for verification the most recently into the short-term status cache. The verification result generation module is used to generate visual perception results for images at the edge end. It reads joint state data according to the image acquisition time, converts the visual perception results into obstacle position data in the robot arm base coordinate system, and associates them with the expected positions of each link and end effector at the expected time of command activation to form the verification result. The latest available time determination module is used for each control cycle. It calculates the joint control margin from the current joint state data and the obstacle position data that was last submitted for verification. Based on the joint control margin, actual processing time, end effector linear velocity, minimum obstacle distance, obstacle position change estimation speed and depth measurement confidence, it determines the latest available time of the verification result. The submission verification module is used to perform source data timing checks, latest available time checks, and joint state offset checks on the verification results. If all three checks pass, the result is recorded as accepted; otherwise, it is discarded. The control command correction module is used to generate a set of safety constraints based on the adopted results, correct the nominal control commands with the set of safety constraints, obtain the corrected control commands, and output them to the joint controller.

Images