Migration correction system, method and device for dynamic robot visual action strategy

Abstract

The invention belongs to the technical field of robots, and particularly relates to a migration correction system, method and device for a dynamic robot visual action strategy. The method comprises the steps that actions which are not executed yet in a historical first action sequence serve as a third action sequence, a first action sequence is generated based on current first image data, the first action sequence and the third action sequence are merged based on a time sequence and then interpolated, and a second action sequence is generated, wherein the time step of the second action sequence is smaller than that of the first action sequence; the pose of the robot body is obtained, the future pose is estimated, the action offset of a target action is determined based on a pre-corrected delay parameter and the future pose, and the delay parameter is used for representing time delay from pose estimation to action execution; and performing real-time correction on the action in the third action sequence based on the action offset, and generating the action to be executed.

Description

Technical Field

[0001] The present disclosure belongs to the field of robotics technology, and in particular relates to a system, method, and device for migrating and correcting visual motion strategies of a dynamic robot. Background Art

[0002] Vision-action strategies, a novel control paradigm that directly maps visual perception to robot action, have demonstrated significant advantages in learning complex tasks. This strategy utilizes human-instructional data through an end-to-end imitation learning framework to directly translate sensor inputs into action outputs, effectively avoiding the complex modular process design required by traditional approaches.

[0003] However, when targeting dynamic robotic platforms (such as quadruped robots, drones, and other autonomous mobile systems), the actual deployment (also known as migration) of visual-motion strategies faces multiple obstacles. First, the inherent motion instability of dynamic platforms makes it extremely difficult to collect high-quality teaching data. Therefore, visual-motion strategies for dynamic robotic platforms are often trained using static data. However, the motion disturbances of dynamic platforms can easily cause traditional static training strategies to fail, making it difficult to ensure the stability of high-precision action execution. Second, existing visual-motion strategies suffer from real-time performance issues in dynamic scenarios. Specifically, inference delays caused by low-computing platforms can cause generated actions to become outdated and ineffective due to platform displacement. Furthermore, there is a timing mismatch between batch action generation mechanisms and real-time control. Summary of the Invention

[0004] The disclosed embodiments propose a migration correction solution for dynamic robot visual motion strategies to address various problems in existing solutions caused by platform motion interference, response lag, and timing mismatch.

[0005] A first aspect of the disclosed embodiments provides a migration and correction system for a dynamic robot visual motion strategy, comprising a dynamic robot platform, a controller, and at least two image acquisition devices, wherein the controller is data-connected to the dynamic robot platform and the image acquisition devices, respectively, wherein:

[0006] The dynamic robot platform comprises at least a robot body and an execution part, wherein the robot body is used to carry the execution part and is in a moving state, and the execution part is used to execute the to-be-executed action generated by the controller;

[0007] The first image acquisition device is used to acquire first image data of the target area as input of the visual action strategy;

[0008] The second image acquisition device is used to acquire second image data of a preset area of the robot body as input to the visual SLAM module;

[0009] The controller is deployed with a visual motion strategy and a migration correction module,

[0010] The visual action strategy generates a first action sequence based on the first image data;

[0011] The migration correction module includes a visual SLAM module that estimates the robot body posture based on the second image data. The migration correction module generates a second action sequence by interpolation for the first action sequence, and performs real-time correction on the actions in the second action sequence based on the pre-corrected delay parameter and the robot body posture to generate the action to be executed, wherein the delay parameter is used to characterize the time delay from posture estimation to action execution, and the time step of the second action sequence is smaller than that of the first action sequence.

[0012] A second aspect of the embodiments of the present disclosure provides a migration correction method for a dynamic robot visual motion strategy, which is applied to the system described in the first aspect of the embodiments of the present disclosure, including:

[0013] Taking the actions that have not been executed in the historical first action sequence as a third action sequence, generating a first action sequence based on the current first image data, merging the first action sequence and the third action sequence based on time sequence and then interpolating to generate a second action sequence, wherein the time step of the second action sequence is smaller than that of the first action sequence;

[0014] Acquiring the posture of the robot body and estimating a future posture, and determining an action offset of a target action based on a pre-corrected delay parameter and the future posture, wherein the delay parameter is used to characterize a time delay between posture estimation and action execution;

[0015] The actions in the second action sequence are corrected in real time based on the action offset to generate the action to be executed.

[0016] In some embodiments of the present disclosure, the generating the second action sequence by merging the first action sequence and the third action sequence based on a time sequence and then interpolating the combined results includes:

[0017] determining an overlapping interval between the first action sequence and the third action sequence;

[0018] In the overlapping interval, based on minimizing the Euclidean distance between the action sequences, determining a timestamp correspondence between the first action sequence and the third action sequence;

[0019] Performing exponential weighted fusion on the first action sequence and the third action sequence based on the following formula to generate a fourth action sequence;

[0020]

[0021] The new and old weights satisfy:

[0022]

[0023] in, is the third action sequence, W is the weight ( represents the weight of the third action sequence, represents the weight of the fourth action sequence, represents the weight of the first action sequence), is an action in the first action sequence, is an action in the third action sequence, is an action in the fourth action sequence, i and k are indexes of the overlapping parts of the first action sequence and the third action sequence, α is a preset parameter, and m a is the length of the first action sequence, and t is the timestamp;

[0024] The fourth action sequence is interpolated to generate the second action sequence.

[0025] In some embodiments of the present disclosure, interpolating the fourth action sequence to generate the second action sequence includes:

[0026] The fourth action sequence is continuously interpolated using a linear interpolation method to generate the second action sequence, wherein the time step of the second action sequence is smaller than that of the first action sequence.

[0027] In some embodiments of the present disclosure, obtaining the posture of the robot body and estimating the future posture includes:

[0028] Based on the second image acquisition device, perform visual SLAM on the robot body to obtain a posture sequence of the robot body;

[0029] Taking the current pose as the reference coordinate system, calculate the relative pose sequence of the past l0 frames in p0 is the current pose, Δp -i is the relative pose of the past i-th frame relative to the current frame, p -i is the pose of the past i-th frame;

[0030] The future pose of the robot body is predicted based on the relative pose sequence.

[0031] In some embodiments of the present disclosure, predicting the future posture of the robot body based on the relative posture sequence includes:

[0032] Take the relative pose sequence as the input of the prediction network and output the relative pose prediction sequence of the next I1 frame

[0033] pass Calculate the future pose, where is the predicted relative pose of the future i-th frame relative to the current frame, p0 is the pose of the current frame, is the predicted pose of the future i-th frame.

[0034] In some embodiments of the present disclosure, the prediction network can be expressed as:

[0035]

[0036] in, Represents the prediction network, LSTM(·) represents a long short-term memory network, GRU(·) represents a gated recurrent unit, concat(·) represents concatenation along the feature dimension, and FC(·) represents a fully connected layer.

[0037] In some embodiments of the present disclosure, determining the motion offset of the target motion based on the delay parameter and the future posture includes:

[0038]

[0039] Where δ represents the movement of the robot body to the target action A τ The motion offset caused by ε represents the external parameter matrix between the pre-calibrated visual SLAM coordinate system and the execution unit coordinate system; Δt τ represents the time interval from when the visual action strategy generates an action to when it is executed, f represents the output frequency of the SLAM system, and Δt is the delay parameter.

[0040] In some embodiments of the present disclosure, the delay parameter pre-correction method includes:

[0041] The trajectory of the marker set at the end of the execution part is tracked online, and a time delay when the ratio of the marker pixel plane movement speed to the movement speed of the robot body is minimized is linearly searched from 0 delay, and the time delay is used as the delay parameter.

[0042] A third aspect of the embodiments of the present disclosure provides a migration and correction device for a dynamic robot visual motion strategy, which is applied to the system described in the first aspect of the embodiments of the present disclosure, including:

[0043] an interpolation module, configured to use actions that have not yet been executed in the historical first action sequence as a third action sequence, generate a first action sequence based on the current first image data, merge the first action sequence and the third action sequence based on a time sequence, and interpolate them to generate a second action sequence, wherein the time step of the second action sequence is smaller than that of the first action sequence;

[0044] a prediction module, configured to obtain the posture of the robot body and estimate a future posture, and determine an action offset of a target action based on a pre-corrected delay parameter and the future posture, wherein the delay parameter is used to characterize a time delay between posture estimation and action execution;

[0045] A correction module is used to perform real-time correction on the actions in the third action sequence based on the action offset to generate the to-be-executed action.

[0046] In summary, the system, method, and apparatus for migrating and correcting dynamic robot visual motion strategies provided by various embodiments of the present disclosure address the core contradiction of static training strategies when deployed on dynamic platforms by introducing a real-time motion correction framework. By using time series integration and high-frequency interpolation techniques, the strategy output frequency is increased to the hardware limit, effectively alleviating the timing misalignment problem between asynchronous reasoning and real-time control. Based on a lightweight prediction network combined with visual SLAM pose tracking, a spatiotemporal mapping model of platform motion and end-effector state is constructed to achieve feedforward compensation for platform disturbances. Its prediction time domain covers the entire link delay cycle from perception to execution. Online delay estimation dynamically corrects system parameter deviations through end-stage stable warm-up calibration, forming a closed-loop optimization mechanism. The synergistic effect of these three technical components enables the static strategy to resolve the end-stage pose offset caused by platform motion in real time while retaining its original operational capabilities, thereby maintaining centimeter-level operational accuracy even when the platform is in continuous motion. Experiments have verified the comprehensive advantages of this method in maintaining strategy generalization, improving system responsiveness, and enhancing platform adaptability. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] The features and advantages of the present disclosure will be more clearly understood by referring to the accompanying drawings, which are schematic and should not be construed as limiting the present disclosure in any way. In the accompanying drawings:

[0048] Figure 1 is a schematic diagram of a migration correction system for a dynamic robot visual motion strategy applicable to the present disclosure;

[0049] Figure 2 is a schematic diagram of a dynamic robotic platform according to one embodiment of the present disclosure;

[0050] Figure 3 is a schematic diagram of a dynamic robotic platform according to one embodiment of the present disclosure;

[0051] Figure 4 It is an example of a migration correction system for a dynamic robot vision-action strategy shown in one embodiment of the present disclosure;

[0052] Figure 5is a flow chart of a migration correction method for a dynamic robot visual motion strategy according to some embodiments of the present disclosure;

[0053] Figure 6 is a schematic diagram of a verification task for verifying the method in one embodiment of the present disclosure;

[0054] Figure 7 This method is based on Figure 6 The task execution process of the verification task shown;

[0055] Figure 8 is Figure 7 The distribution diagram of the end of the robot arm during the task execution shown;

[0056] Figure 9 2 is a schematic diagram of a migration correction device for a dynamic robot visual motion strategy according to some embodiments of the present disclosure. DETAILED DESCRIPTION

[0057] In the detailed description that follows, many specific details of the present disclosure are set forth by way of example in order to provide a thorough understanding of the relevant disclosure. However, it will be apparent to one of ordinary skill in the art that the present disclosure can be implemented without these details. It should be understood that the use of the terms "system," "device," "unit," and / or "module" in the present disclosure is a method for distinguishing between different parts, elements, parts, or assemblies at different levels in a sequential arrangement. However, these terms may be replaced by other expressions if they can achieve the same purpose.

[0058] It should be understood that when a device, unit, or module is referred to as being "on," "connected to," or "coupled to" another device, unit, or module, it may be directly on, connected to, coupled to, or in communication with the other device, unit, or module, or there may be intervening devices, units, or modules, unless the context clearly indicates an exception. For example, the term "and / or" as used in this disclosure includes any and all combinations of one or more of the associated listed items.

[0059] The terms used in this disclosure are only for describing specific embodiments and are not intended to limit the scope of this disclosure. As shown in the specification and claims of this disclosure, unless the context clearly indicates an exception, the words "a", "an", "a kind" and / or "the" do not specifically refer to the singular and may also include the plural. Generally speaking, the terms "include" and "comprise" only indicate the inclusion of clearly identified features, wholes, steps, operations, elements and / or components, and such expressions do not constitute an exclusive list, and other features, wholes, steps, operations, elements and / or components may also be included.

[0060] These and other features and characteristics of the present disclosure, as well as the methods of operation, the functions of the related elements of the structure, the combination of parts, and the economy of manufacture may be better understood with reference to the following description and accompanying drawings, which form a part of this specification. However, it is to be expressly understood that the drawings are for illustration and description purposes only and are not intended to limit the scope of protection of the present disclosure. It is to be understood that the drawings are not drawn to scale.

[0061] Various structural diagrams are used in this disclosure to illustrate various variations of the embodiments of the present disclosure. It should be understood that the preceding or following structures are not intended to limit the present disclosure. The scope of protection of the present disclosure is subject to the claims.

[0062] Figure 1 It is a schematic diagram of a migration correction system for a dynamic robot visual motion strategy applicable to the present disclosure. Figure 1 The system includes a controller connected to a dynamic robotic platform and at least two image acquisition devices. The image acquisition devices respectively acquire image data of a target area and the dynamic robotic platform, and the controller generates a to-be-performed action of the dynamic robotic platform in the target area based on the image data.

[0063] The dynamic robot platform refers to a robot system with autonomous movement capability, which includes at least a robot body and an execution unit. The robot body is used to carry the execution unit and is in motion, and the execution unit is used to execute the action to be executed generated by the controller. In one embodiment of the present disclosure, Figure 2 As shown, the dynamic robot platform is a crawler-type mobile robot arm 20, and the robot body 22 in the form of a crawler is easy to move and can carry a three-degree-of-freedom robot arm 21 as an execution part. The robot arm 21 is used to perform specific actions to be performed; in another embodiment of the present disclosure, as shown Figure 3 As shown, the dynamic robotic platform is a quadcopter drone 30 with an operating rod. The quadcopter body 32, which serves as the robot's main body, can fly autonomously. The lower portion of the quadcopter body carries an operating rod 31, which serves as an actuator. The operating rod 31 can perform operations during the flight of the quadcopter drone 30. The present disclosure does not limit the specific form of the dynamic robotic platform.

[0064] The first image acquisition device is used to acquire image data of the target area as an input of the visual action strategy to generate the action to be executed by the execution unit in the target area. The second image acquisition device is used to acquire image data of a preset area of the robot body to perform visual SLAM on the robot body to obtain the posture of the robot body. Generally speaking, the second image acquisition device acquires image data at a relatively high frequency, and the first image acquisition device acquires image data at a relatively low frequency. For example, in one embodiment of the present disclosure, the second image acquisition device is a T265 camera. However, the present disclosure does not limit the type of image acquisition device. Any image acquisition device that meets the requirements of the visual action strategy for input image data can be used as the first image acquisition device, and any image acquisition device that meets the requirements of visual SLAM can be used as the second image acquisition device.

[0065] The controller disclosed herein is equipped with a visual action strategy and a visual SLAM module. The visual action strategy generates an action sequence based on image data of a target area acquired by a first image acquisition device. The visual SLAM module estimates the posture sequence of the robot body based on image data of a preset area of the robot body acquired by a second image acquisition device. The controller estimates the future posture based on the posture sequence, and compensates the action sequence based on the future posture to generate the pending action to be executed by the execution unit. The controller can be any of a stand-alone, clustered, or distributed server. In particular, the controller can be built into a dynamic robot platform.

[0066] In one embodiment of the present disclosure, the migration correction system of the dynamic robot visual action strategy is as follows Figure 4 shown. Figure 4 The system shown consists of a platform that can be moved on an operating table, equipped with a three-degree-of-freedom robotic arm. Two cameras are mounted on the shoulder and wrist of the robotic arm as inputs for visual action strategies, and a T265 camera is also installed as input for visual SLAM. A programmable lead screw is used to provide motion for the robotic arm platform. The controller is not shown.

[0067] SLAM (Simultaneous Localization and Mapping) refers to simultaneous localization and environmental mapping. Visual SLAM uses image data to estimate the body's position and construct a map of the environment in real time. It is a core technology in robotics, autonomous driving, AR / VR, and other fields.

[0068] Vision-Action Strategy, a novel control paradigm that directly maps visual perception to robot action, has demonstrated significant advantages in learning complex tasks. This strategy utilizes human-instructional data through an end-to-end imitation learning framework to directly convert sensor inputs to action outputs, effectively avoiding the complex modular process design required by traditional approaches. Vision-Action Strategy has been successfully applied to delicate operations such as battery assembly, food processing, and textile handling, demonstrating its superior performance in static environments. The deployment of vision-action strategies across different platforms is called migration.

[0069] Although visual-action strategies have demonstrated excellent performance in static environments, they encounter the following difficulties when migrating visual-action strategies from static environments to dynamic platforms or between different dynamic platforms:

[0070] First, the inherent motion instability of dynamic platforms makes it extremely difficult to collect high-quality teaching data. Therefore, visual motion strategies for dynamic robotic platforms are often trained using static data. However, the motion disturbances of dynamic platforms can easily cause traditional static training strategies to fail, making it difficult to ensure the stability of high-precision motion execution. Second, existing visual motion strategies suffer from real-time defects in dynamic scenarios. Specifically, inference delays caused by low-computing platforms cause generated actions to become outdated and ineffective due to platform displacement. In addition, there is a timing mismatch between batch action generation mechanisms and real-time control. Although some solutions have attempted to improve timing consistency through techniques such as action segmentation, this further weakens the system's adaptability to dynamic environmental changes and cannot meet the centimeter-level precision requirements of dynamic platforms.

[0071] Therefore, the present disclosure proposes a migration correction method for dynamic robot visual motion strategy to solve the above-mentioned problems encountered when migrating visual motion strategy from a static environment to a dynamic robot platform, or migrating between different dynamic robot platforms.

[0072] Figure 5 is a flow chart of a method for migrating and correcting a dynamic robot visual motion strategy according to some embodiments of the present disclosure. In some embodiments, the method for migrating and correcting a dynamic robot visual motion strategy is composed of Figure 1 The controller shown executes the method, which includes the following steps:

[0073] S510, taking the actions that have not been executed in the historical first action sequence as the third action sequence, generating the first action sequence based on the current first image data, merging the first action sequence and the third action sequence based on time sequence and interpolating them to generate a second action sequence, wherein the time step of the second action sequence is smaller than that of the first action sequence.

[0074] based on Figure 4The migration correction system of the dynamic robot visual action strategy shown in the figure uses the basic visual action strategy network as the starting point of the overall system. The visual action strategy is denoted as π, and the m o Observation data (Including images and the robot arm joint status ) as input. Based on these observations, we predict the future m a The action sequence of time steps. The number of observation steps m o and the number of prediction steps m a All are preset based on experience. At time t, the action sequence output by strategy π is The process can be formally expressed as:

[0075]

[0076] To prevent the policy inference process from blocking the system for extended periods, the policy network π is assigned to a separate subprocess. After each inference cycle, the policy network immediately retrieves the latest observation data to ensure continuous generation of action sequences. This design increases the decision-making frequency of the policy network and helps it cope with dynamic scenarios.

[0077] In order to effectively select appropriate actions from the continuously generated action sequences, this embodiment designs an action manager module to process the output of the policy network. First, the action manager maintains an action buffer The buffer stores the most recently executed actions and the predicted actions to be executed in sequence. Then, a time series smoothing algorithm is used to combine the newly generated action sequence u with the action buffer. Finally, the robot arm control frequency is improved through action interpolation. The specific implementation is as follows:

[0078] Timing smoothing:

[0079] The policy network π reads the most recent m0 observations at time t1 After inference time t Inf Post-generation action sequence The action manager at time t2≈t1+t Inf The new action sequence needs to be compared with the current action buffer Merge, where "≈" indicates that t2 is the first execution time after the action sequence is generated by obtaining the observation at time t1. The corresponding action index is determined by establishing a timestamp alignment relationship:

[0080]

[0081] where d(·) represents the Euclidean distance, for and In the overlapping interval after time alignment, m s is the preset search range. Determine the corresponding relationship t u Then, exponentially weighted fusion is used to update the action buffer:

[0082]

[0083] The new and old weights satisfy:

[0084]

[0085] in is the action buffer, w is the weight ( represents the weight of the action in the action buffer before updating, represents the weight of the action in the updated action buffer, represents the weight of the newly generated action sequence), is the action in the newly generated action sequence, is the action of the action buffer before updating, is the action of the updated action buffer, and α is the preset parameter.

[0086] Action interpolation:

[0087] To address the problem that the frequency of action sequences output by the policy network is lower than the actual execution requirements, a linear interpolation method is used to generate actions with fine-grained time steps τ:

[0088] A τ =(τ-t)A t +(t+1-τ)A t+1

[0089] The interpolation process achieves internal interpolation through a continuously updated action buffer, avoiding the risks brought by extrapolation.

[0090] S520, obtaining the posture of the robot body and estimating the future posture, and determining the motion offset of the target action based on the pre-corrected delay parameter and the future posture, wherein the delay parameter is used to characterize the time delay between posture estimation and action execution.

[0091] Stabilizer based on prediction compensation:

[0092] After obtaining real-time, high-frequency motion through the motion manager, the impact of robot motion on manipulator operation still needs to be addressed, such as the robot's active movement or shaking caused by its own instability. The most direct approach is to monitor the robot's motion data and adjust the manipulator's motion accordingly. However, considering the computational, communication, and control delays, using outdated robot position data to adjust the manipulator's motion can significantly reduce the success rate of high-precision manipulation tasks. Therefore, this paper designs a stabilizer based on predictive compensation, which effectively corrects the motion by predicting the platform's short-term motion.

[0093] The prediction module consists of two parts: a SLAM module for pose estimation and a lightweight prediction network. In terms of pose estimation, a stereo camera equipped with an IMU is used to implement real-time high-frequency visual SLAM to obtain high-frequency pose sequences. Each prediction takes the current pose p0 as the reference coordinate system and calculates the high-frequency relative pose sequence of the past l0 frames in p0 is the current pose, Δp -i is the relative pose of the past i-th frame relative to the current frame, p -i is the pose of the past i-th frame. As the input of the prediction network. The prediction network directly outputs the high-frequency relative pose prediction sequence of the next l1 frames pass Calculate the future pose, where is the relative pose of the predicted future i-th frame relative to the current frame, is the predicted pose of the future i-th frame.

[0094] Considering the computational resource limitations and system real-time requirements, this paper implements a lightweight prediction network Specifically, by parallelizing the LSTM and GRU networks so that they process the same input, their time series outputs are concatenated along the feature dimension, and finally a prediction result is generated through a fully connected layer, namely:

[0095]

[0096] in, denotes a prediction network, LSTM(·) denotes a long short-term memory network, GRU(·) denotes a gated recurrent unit, concat(·) denotes concatenation along the feature dimension, and FC(·) denotes a fully connected layer.

[0097] For dynamic robot scenarios dominated by periodic or regular motion, some embodiments of the present disclosure predict future poses based on adjacent frame differences and TCN networks. Specifically:

[0098] Taking the current pose as the reference coordinate system, calculate the relative pose sequence of the past l0 frames in p0 is the current pose, Δp -i is the relative pose of the past i-th frame relative to the i+1-th frame, p -i is the pose of the past i-th frame;

[0099] Then use the relative pose sequence as the input of the prediction network and output the relative pose prediction sequence of the next l1 frames pass Calculate the future pose, where is the relative pose of the predicted future i-th frame relative to the future i-1-th frame, p0 is the pose of the current frame, is the predicted pose of the future i-th frame.

[0100] The prediction network can be expressed as:

[0101]

[0102] in, Represents the prediction network, TCN(.) represents the temporal convolutional network, L is the number of convolutional layers of TCN, k is the convolution kernel size, d is the dilation factor growth strategy, and PC(·) represents the fully connected layer.

[0103] In trajectory modeling methods that use relative starting frame positions as input, the input for each time step is represented as the absolute displacement between that frame and the starting frame. While this representation preserves a certain degree of global trajectory shape, the constant growth of its value over time leads to non-stationary input features. This is particularly true in scenarios with periodic or regular motion. These displacements no longer reflect the repetitive patterns of the trajectory, making it difficult for recurrent neural networks (such as LSTMs) based on temporal dependencies to effectively capture periodic structure. Furthermore, LSTMs suffer from state forgetting and gradient decay when modeling long sequences. When the input data lacks discernible local dynamics, its memory mechanism is easily corrupted by accumulated errors, further weakening the model's ability to learn short-term patterns. Compared to the initial frame + LSTM approach, the adjacent frame difference + TCN approach is more suitable for the periodic or regular nature of real-world robot motion. Adjacent frame differences can better represent local dynamics over short periods of time. Furthermore, the smaller range of variation in the difference values makes it easier for the prediction network to fit. TCNs can effectively enhance the ability to model long-term dependencies, resulting in faster and more stable training.

[0104] Finally, the robot moves to the target action A τ The resulting motion offset can be expressed as:

[0105]

[0106] Where ε represents the external parameter matrix between the visual SLAM coordinate system and the robotic arm coordinate system (manually calibrated in advance), Δt τ Indicates that the policy network π generates action A τ The time interval from pose estimation to action execution, f represents the output frequency of the SLAM system. Δt represents the total delay from pose estimation to action execution (including pose estimation processing time, pose prediction time, data transmission time, and action execution time). This parameter will be obtained through online estimation.

[0107] Online estimation of delay parameters:

[0108] To align the optimization goals of the entire system with the task execution goals, the present disclosure designs a motion stabilization warm-up program for online calibration of system delay parameters, thereby enabling the stabilizer module to accurately adjust motion execution. To achieve this goal, the present disclosure utilizes a stabilizer module to assist the robot in performing end-effector stabilization tasks, by quantitatively evaluating the impact of system delay on operational stability. A specific implementation method is as follows: During the movement of the robotic arm base, the gripper continuously holds a red marker while tracking its position offset through a fixed monocular third-view camera. A linear search strategy starting from 0 delay is executed, and the system delay is determined by identifying the motion trajectory of the marker. To ensure that changes in robot motion do not affect the accuracy of delay estimation, the present invention uses the ratio of the marker pixel plane motion speed to the platform motion speed estimated by visual SLAM as an evaluation indicator, and the time delay when the ratio of the marker pixel plane motion speed to the robot base motion speed is minimum is used as the delay parameter. By controlling the motion sequence, online calibration of system parameters is achieved, and ultimately a time synchronization relationship between motion execution and platform motion is established.

[0109] S530: Correct the actions in the second action sequence in real time based on the action offset to generate the action to be executed.

[0110] Get the robot's movement to the target action A τ After calculating the motion offset and pre-corrected delay parameters, the final motion performed by the robot can be corrected to:

[0111] A′ τ =δ(τ, Δt, ε)·A τ

[0112] Among them, A′ τ It is the action to be executed after correction. τ is the target action cached in the updated action buffer, δ(τ, Δt, ε) is the robot motion for the target action A τ The amount of motion offset caused.

[0113] One embodiment of the present disclosure will Figure 5 The method described in S510-S530 (hereinafter referred to as this method) was verified in the task of grasping and placing objects, the task of stacking cylinders, and the task of carrying bags. Figure 6 The task execution process is as follows. Figure 7 During the task execution, the distribution diagram of the end of the robotic arm is as shown in Figure 8 shown. Figure 8 It can be seen that during the movement of the robotic arm, this method can effectively ensure the stability of the robotic arm's end-to-end execution of tasks.

[0114] Figure 9 Schematic diagram of a migration correction device for a dynamic robot visual action strategy according to some embodiments of the present disclosure. Figure 9 As shown, the migration correction device 900 of the dynamic robot visual motion strategy includes an interpolation module 910, a prediction module 920, and a correction module 930. In some embodiments of the present disclosure, the migration correction function of the dynamic robot visual motion strategy is Figure 1 The controller shown executes as follows.

[0115] An interpolation module 910 is configured to use unexecuted actions in the historical first action sequence as a third action sequence, generate a first action sequence based on the current first image data, and interpolate the first action sequence and the third action sequence based on a time sequence to generate a second action sequence, wherein the time step of the second action sequence is smaller than that of the first action sequence;

[0116] a prediction module 920 for acquiring the posture of the robot body and estimating a future posture, and determining an action offset of a target action based on a pre-calibrated delay parameter and the future posture, wherein the delay parameter is used to characterize a time delay between posture estimation and action execution;

[0117] The correction module 930 is configured to perform real-time correction on the actions in the third action sequence based on the action offset to generate the to-be-executed actions.

[0118] In summary, the system, method, and apparatus for migrating and correcting dynamic robot visual motion strategies provided by various embodiments of the present disclosure address the core contradiction of static training strategies when deployed on dynamic platforms by introducing a real-time motion correction framework. By using time series integration and high-frequency interpolation techniques, the strategy output frequency is increased to the hardware limit, effectively alleviating the timing misalignment problem between asynchronous reasoning and real-time control. Based on a lightweight prediction network combined with visual SLAM pose tracking, a spatiotemporal mapping model of platform motion and end-effector state is constructed to achieve feedforward compensation for platform disturbances. Its prediction time domain covers the entire link delay cycle from perception to execution. Online delay estimation dynamically corrects system parameter deviations through end-stage stable warm-up calibration, forming a closed-loop optimization mechanism. The synergistic effect of these three technical components enables the static strategy to resolve the end-stage pose offset caused by platform motion in real time while retaining its original operational capabilities, thereby maintaining centimeter-level operational accuracy even when the platform is in continuous motion. Experiments have verified the comprehensive advantages of this method in maintaining strategy generalization, improving system responsiveness, and enhancing platform adaptability.

[0119] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices and modules can refer to the corresponding descriptions in the aforementioned device embodiments and will not be repeated here.

[0120] Although the subject matter described herein is provided in the general context of being executed in conjunction with the execution of an operating system and application programs on a computer system, those skilled in the art will recognize that other implementations may also be performed in conjunction with other types of program modules. Generally speaking, program modules include routines, programs, components, data structures, and other types of structures that perform specific tasks or implement specific abstract data types. Those skilled in the art will appreciate that the subject matter described herein may be practiced using other computer system configurations, including handheld devices, multiprocessor systems, microprocessor-based or programmable consumer electronics, minicomputers, mainframe computers, and the like, and may also be used in distributed computing environments in which tasks are performed by remote processing devices connected via a communication network. In a distributed computing environment, program modules may be located in both local and remote memory storage devices.

[0121] Those skilled in the art will appreciate that the units and method steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this disclosure.

[0122] It should be understood that the above-described specific embodiments of the present disclosure are merely illustrative of or explanation of the principles of the present disclosure and do not constitute limitations on the present disclosure. Therefore, any modifications, equivalent substitutions, improvements, etc. made without departing from the spirit and scope of the present disclosure shall be included within the scope of protection of the present disclosure. In addition, the claims appended to the present disclosure are intended to cover all variations and modifications that fall within the scope and metes and bounds of the appended claims, or equivalents of such scope and metes and bounds.

Claims

1. A migration and correction system for a dynamic robot visual motion strategy, comprising a dynamic robot platform, a controller, and at least two image acquisition devices, wherein the controller is data-connected to the dynamic robot platform and the image acquisition devices, respectively, wherein: The dynamic robot platform comprises at least a robot body and an execution part, wherein the robot body is used to carry the execution part and is in a moving state, and the execution part is used to execute the to-be-executed action generated by the controller; The first image acquisition device is used to acquire first image data of the target area as input of the visual action strategy; The second image acquisition device is used to acquire second image data of a preset area of the robot body as input to the visual SLAM module; The controller is deployed with a visual motion strategy and a migration correction module, The visual action strategy generates a first action sequence based on the first image data; The migration correction module includes a visual SLAM module that estimates the robot body posture based on the second image data. The migration correction module generates a second action sequence by interpolation for the first action sequence, and performs real-time correction on the actions in the second action sequence based on the pre-corrected delay parameter and the robot body posture to generate the action to be executed, wherein the delay parameter is used to characterize the time delay from posture estimation to action execution, and the time step of the second action sequence is smaller than that of the first action sequence.

2. A migration correction method for a dynamic robot visual motion strategy, applied to the system described in claim 1, characterized in that: include: Taking the actions that have not been executed in the historical first action sequence as a third action sequence, generating a first action sequence based on the current first image data, merging the first action sequence and the third action sequence based on time sequence and then interpolating to generate a second action sequence, wherein the time step of the second action sequence is smaller than that of the first action sequence; Acquiring the posture of the robot body and estimating a future posture, and determining an action offset of a target action based on a pre-corrected delay parameter and the future posture, wherein the delay parameter is used to characterize a time delay between posture estimation and action execution; The actions in the second action sequence are corrected in real time based on the action offset to generate the action to be executed.

3. The method according to claim 2, characterized in that The generating a second action sequence by merging the first action sequence and the third action sequence based on time sequence and then interpolating the combined results includes: determining an overlapping interval between the first action sequence and the third action sequence; In the overlapping interval, based on minimizing the Euclidean distance between the action sequences, determining a timestamp correspondence between the first action sequence and the third action sequence; Performing exponential weighted fusion on the first action sequence and the third action sequence based on the following formula to generate a fourth action sequence; The new and old weights satisfy: in, is the third action sequence, w is the weight ( represents the weight of the third action sequence, represents the weight of the fourth action sequence, represents the weight of the first action sequence), is an action in the first action sequence, is an action in the third action sequence, is an action in the fourth action sequence, i and k are indexes of the overlapping parts of the first action sequence and the third action sequence, α is a preset parameter, and m a is the length of the first action sequence, and t is the timestamp; The fourth motion sequence is interpolated to generate the second motion sequence, wherein the time step of the second motion sequence is smaller than that of the first motion sequence.

4. The method according to claim 3, characterized in that Interpolating the fourth action sequence to generate the second action sequence includes: The fourth action sequence is continuously interpolated using a linear interpolation method to generate the second action sequence, wherein the time step of the second action sequence is smaller than that of the first action sequence.

5. The method according to claim 1, characterized in that: The obtaining the posture of the robot body and estimating the future posture includes: Based on the second image acquisition device, perform visual SLAM on the robot body to obtain a posture sequence of the robot body; Taking the current pose as the reference coordinate system, calculate the relative pose sequence of the past l0 frames in p0 is the current pose, Δp -i is the relative pose of the past i-th frame relative to the current frame, p -i is the pose of the past i-th frame; The future pose of the robot body is predicted based on the relative pose sequence.

6. The method according to claim 5, characterized in that Predicting the future posture of the robot body based on the relative posture sequence includes: Take the relative pose sequence as the input of the prediction network and output the relative pose prediction sequence of the next l1 frames pass Calculate the future pose, where is the predicted relative pose of the future i-th frame relative to the current frame, p0 is the pose of the current frame, is the predicted pose of the future i-th frame.

7. The method according to claim 6, characterized in that: The prediction network can be expressed as: in, Represents the prediction network, LSTM(·) represents a long short-term memory network, CRU(·) represents a gated recurrent unit, concat(·) represents concatenation along the feature dimension, and FC(·) represents a fully connected layer.

8. The method according to claim 2, characterized in that: Determining the motion offset of the target motion based on the delay parameter and the future posture includes: Where δ represents the movement of the robot body to the target action A τ The motion offset caused by ε represents the external parameter matrix between the pre-calibrated visual SLAM coordinate system and the execution unit coordinate system; Δt τ represents the time interval from when the visual action strategy generates an action to when it is executed, f represents the output frequency of the SLAM system, and Δt is the delay parameter.

9. The method according to claim 2, characterized in that: The delay parameter pre-correction method includes: The trajectory of the marker set at the end of the execution part is tracked online, and a time delay when the ratio of the marker pixel plane movement speed to the movement speed of the robot body is minimized is linearly searched starting from 0 delay, and the time delay is used as the delay parameter.

10. A migration and correction device for dynamic robot visual motion strategy, applied to the system according to claim 1, characterized in that: include: an interpolation module, configured to use actions that have not yet been executed in the historical first action sequence as a third action sequence, generate a first action sequence based on the current first image data, merge the first action sequence and the third action sequence based on a time sequence, and interpolate them to generate a second action sequence, wherein the time step of the second action sequence is smaller than that of the first action sequence; a prediction module, configured to obtain the posture of the robot body and estimate a future posture, and determine an action offset of a target action based on a pre-corrected delay parameter and the future posture, wherein the delay parameter is used to characterize a time delay between posture estimation and action execution; A correction module is used to perform real-time correction on the actions in the third action sequence based on the action offset to generate the to-be-executed action.

Images