Action timing alignment method and electronic device

Abstract

The application provides an action timing alignment method and an electronic device, wherein the method comprises: acquiring an inference delay, an action field of view, multi-modal context information and an action block of a current inference period, and determining a timing weight sequence; in a current generation step of the current inference period, generating a current update vector, and determining an estimated action block corresponding to the current generation step; determining action intervention information corresponding to the current generation step; applying the action intervention information corresponding to the current generation step to the current update vector to obtain an updated vector, iteratively executing, and after the iteration ends, taking an intermediate action block at the end of the iteration as an action block of a next inference period. The application can achieve hard alignment of the action to be executed and soft alignment of future actions on each generation step under the premise of zero training cost and without gradient backpropagation, and can achieve sub-millisecond alignment overhead while preserving generation prior.

Description

Technical Field

[0001] This application relates to the field of action generation technology, and more specifically, to an action timing alignment method and an electronic device. Background Technology

[0002] Current mainstream large-scale vision-language-action (VLA) models generally adopt a generative architecture, relying on diffusion or flow matching mechanisms. To improve the model's real-time capabilities, they employ an asynchronous inference paradigm: while the robot executes the previous action block, the next action block is generated in parallel. Although this paradigm significantly reduces end-to-end latency, it introduces a structural misalignment of action blocks on the time axis. Specifically, when the inference latency does not match the action block length or the execution field of view, the generated action block cannot naturally align with the actual local action range to be executed, leading to execution interruption, trajectory jitter, or even task failure.

[0003] Existing technologies typically propose intervention schemes during the training phase and system design phase to address this temporal misalignment problem. Specifically, training phase intervention schemes usually modify the model structure or adjust the policy network during VLA training. Training-free online correction schemes typically calculate the gradient of the action head output with respect to the current generated state using the Vector-Jacobian Product (VJP) at each step of denoising / stream sampling. This gradient is then superimposed onto the original velocity field as a guiding signal, thereby pulling the generated trajectory towards a subspace aligned with the committed action.

[0004] However, these existing processing methods suffer from poor scalability, high online computational overhead, weak hyperparameter robustness, and rigid timing constraints. Summary of the Invention

[0005] The purpose of this application is to provide an action timing alignment method, apparatus, device, and storage medium to address the shortcomings of the prior art, thereby solving the problems of poor scalability, high online computing overhead, weak hyperparameter robustness, and rigid timing constraints in the prior art.

[0006] To achieve the above objectives, the technical solutions adopted in the embodiments of this application are as follows: In a first aspect, one embodiment of this application provides an action timing alignment method, the method comprising: The system acquires inference latency, action field of view, multimodal context information, and action blocks of the current inference cycle. Based on the inference latency and the action field of view, it determines a temporal weight sequence, which includes multiple temporal weights arranged in sequence. In the current generation step of the current inference cycle, the current generation time is determined, and the current update vector is generated based on the intermediate action block of the current generation step, the current generation time, the pre-trained VLA action head, and the multimodal context information. The estimated action block corresponding to the current generation step is determined based on the intermediate action block of the current generation step, the current generation time, and the current update vector. Based on the temporal weight sequence, the action block of the current inference cycle, and the estimated action block corresponding to the current generation step, determine the action intervention information corresponding to the current generation step; Apply the action intervention information corresponding to the current generation step to the current update vector to obtain the updated vector, and generate the intermediate action block of the next generation step of the current generation step according to the updated vector. Iterate and execute, and after the iteration ends, use the intermediate action block at the end of the iteration as the action block of the next inference cycle.

[0007] Optionally, determining the action intervention information corresponding to the current generation step based on the temporal weight sequence, the action block of the current inference cycle, and the estimated action block corresponding to the current generation step includes: Based on the temporal weight sequence, the action block of the current inference cycle, and the estimated action block corresponding to the current generation step, a correction direction vector sequence corresponding to the current generation step is determined, wherein the correction direction vector sequence includes multiple correction direction vectors; Based on the current generation time, the temporal weight sequence, the action block of the current inference cycle, the correction direction vector sequence corresponding to the current generation step, and the estimated action block corresponding to the current generation step, determine the direction correction coefficient corresponding to the current generation step; Calculate the product of each corrected direction vector in the corrected direction vector sequence and the direction correction coefficient to obtain the action intervention information corresponding to the current generation step.

[0008] Optionally, determining the correction direction vector sequence corresponding to the current generation step based on the temporal weight sequence, the action block of the current inference cycle, and the estimated action block corresponding to the current generation step includes: Traverse each sub-action in the estimated action block, for the current sub-action, determine the temporal weight corresponding to the current sub-action from the temporal weight sequence, determine the reference sub-action of the current sub-action from the action block of the current inference cycle, and calculate the correction direction vector corresponding to the current sub-action based on the temporal weight of the current sub-action, the current sub-action, and the reference sub-action. After the traversal is completed, the correction direction vectors of all sub-actions are combined into a sequence of correction direction vectors corresponding to the current generation step.

[0009] Optionally, calculating the correction direction vector corresponding to the current sub-action based on the temporal weight of the current sub-action, the current sub-action, and the reference sub-action includes: Calculate the difference between the current sub-action and the reference sub-action to obtain the action difference result; Based on the action difference result and the temporal weight of the current sub-action, the correction direction vector corresponding to the current sub-action is calculated.

[0010] Optionally, determining the direction correction coefficient corresponding to the current generation step based on the current generation time, the temporal weight sequence, the action block of the current inference cycle, the correction direction vector sequence corresponding to the current generation step, and the estimated action block corresponding to the current generation step includes: The temporal consistency score is determined based on the temporal weight sequence, the action block of the current inference cycle, and the estimated action block corresponding to the current generation step. Based on the temporal consistency score and the current generation time, the obstacle boundary information is determined; Based on the obstacle boundary information and the correction direction vector sequence corresponding to the current generation step, the direction correction coefficient corresponding to the current generation step is determined.

[0011] Optionally, determining the temporal consistency score based on the temporal weight sequence, the action block of the current inference cycle, and the estimated action block corresponding to the current generation step includes: Traverse each sub-action in the estimated action block, for the current sub-action, determine the temporal weight of the current sub-action from the temporal weight sequence, determine the reference sub-action of the current sub-action from the action block of the current inference cycle, and calculate the weighted temporal residual corresponding to the current sub-action based on the temporal weight of the current sub-action, the current sub-action, and the reference sub-action. After the traversal is completed, the timing consistency score is calculated based on the weighted timing residuals corresponding to each sub-action.

[0012] Optionally, determining the obstacle boundary information based on the temporal consistency score and the current generation time includes: Determine the time coefficient based on the current generation time; The obstacle boundary information is calculated based on the time coefficient, the temporal consistency score, and the preset correction intensity parameter.

[0013] Optionally, determining the direction correction coefficient corresponding to the current generation step based on the obstacle boundary information and the correction direction vector sequence corresponding to the current generation step includes: The basic correction amount is determined based on the correction direction vector sequence corresponding to the current generation step and the current update vector; The normalization factor is determined based on the sequence of correction direction vectors corresponding to the current generation step; Based on the obstacle boundary information, the basic correction amount, and the normalization factor, the direction correction coefficient corresponding to the current generation step is determined.

[0014] Optionally, determining the direction correction coefficient corresponding to the current generation step based on the obstacle boundary information, the basic correction amount, the normalization factor, and the preset boundary conditions includes: Calculate the difference between the obstacle boundary information and the basic correction amount to obtain the difference result; The ratio of the difference result to the normalization factor is calculated to obtain the correction result; If the correction result is greater than or equal to the preset lower bound value, the correction result is used as the direction correction coefficient; If the correction result is less than a preset lower bound, the lower bound is used as the direction correction coefficient.

[0015] Secondly, another embodiment of this application provides an action timing alignment device, the device comprising: The acquisition module is used to acquire inference latency, action field of view, multimodal context information and action block of the current inference cycle, and determine the temporal weight sequence based on the inference latency and the action field of view, wherein the temporal weight sequence includes multiple temporal weights arranged in sequence. The first determining module is used to determine the current generation time in the current generation step of the current inference cycle, generate the current update vector based on the intermediate action block of the current generation step, the current generation time, the pre-trained VLA action head and the multimodal context information, and determine the estimated action block corresponding to the current generation step based on the intermediate action block of the current generation step, the current generation time and the current update vector. The second determining module is used to determine the action intervention information corresponding to the current generation step based on the time-series weight sequence, the action block of the current inference cycle, and the estimated action block corresponding to the current generation step. The iteration module is used to apply the action intervention information corresponding to the current generation step to the current update vector to obtain the updated vector, and generate the intermediate action block of the next generation step of the current generation step according to the updated vector, execute iteratively, and after the iteration ends, use the intermediate action block at the end of the iteration as the action block of the next inference cycle.

[0016] Optionally, the second determining module is specifically used for: Based on the temporal weight sequence, the action block of the current inference cycle, and the estimated action block corresponding to the current generation step, a correction direction vector sequence corresponding to the current generation step is determined, wherein the correction direction vector sequence includes multiple correction direction vectors; Based on the current generation time, the temporal weight sequence, the action block of the current inference cycle, the correction direction vector sequence corresponding to the current generation step, and the estimated action block corresponding to the current generation step, determine the direction correction coefficient corresponding to the current generation step; Calculate the product of each corrected direction vector in the corrected direction vector sequence and the direction correction coefficient to obtain the action intervention information corresponding to the current generation step.

[0017] Optionally, the second determining module is specifically used for: Traverse each sub-action in the estimated action block, for the current sub-action, determine the temporal weight corresponding to the current sub-action from the temporal weight sequence, determine the reference sub-action of the current sub-action from the action block of the current inference cycle, and calculate the correction direction vector corresponding to the current sub-action based on the temporal weight of the current sub-action, the current sub-action, and the reference sub-action. After the traversal is completed, the correction direction vectors of all sub-actions are combined into a sequence of correction direction vectors corresponding to the current generation step.

[0018] Optionally, the second determining module is specifically used for: Calculate the difference between the current sub-action and the reference sub-action to obtain the action difference result; Based on the action difference result and the temporal weight of the current sub-action, the correction direction vector corresponding to the current sub-action is calculated.

[0019] Optionally, the second determining module is specifically used for: The temporal consistency score is determined based on the temporal weight sequence, the action block of the current inference cycle, and the estimated action block corresponding to the current generation step. Based on the temporal consistency score and the current generation time, the obstacle boundary information is determined; Based on the obstacle boundary information and the correction direction vector sequence corresponding to the current generation step, the direction correction coefficient corresponding to the current generation step is determined.

[0020] Optionally, the second determining module is specifically used for: Traverse each sub-action in the estimated action block, for the current sub-action, determine the temporal weight of the current sub-action from the temporal weight sequence, determine the reference sub-action of the current sub-action from the action block of the current inference cycle, and calculate the weighted temporal residual corresponding to the current sub-action based on the temporal weight of the current sub-action, the current sub-action, and the reference sub-action. After the traversal is completed, the timing consistency score is calculated based on the weighted timing residuals corresponding to each sub-action.

[0021] Optionally, the second determining module is specifically used for: Determine the time coefficient based on the current generation time; The obstacle boundary information is calculated based on the time coefficient, the temporal consistency score, and the preset correction intensity parameter.

[0022] Optionally, the second determining module is specifically used for: The basic correction amount is determined based on the correction direction vector sequence corresponding to the current generation step and the current update vector; The normalization factor is determined based on the sequence of correction direction vectors corresponding to the current generation step; Based on the obstacle boundary information, the basic correction amount, and the normalization factor, the direction correction coefficient corresponding to the current generation step is determined.

[0023] Optionally, the second determining module is specifically used for: Calculate the difference between the obstacle boundary information and the basic correction amount to obtain the difference result; The ratio of the difference result to the normalization factor is calculated to obtain the correction result; If the correction result is greater than or equal to the preset lower bound value, the correction result is used as the direction correction coefficient; If the correction result is less than a preset lower bound, the lower bound is used as the direction correction coefficient.

[0024] Thirdly, another embodiment of this application provides an electronic device, including: a processor, a storage medium, and a bus, wherein the storage medium stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor communicates with the storage medium via the bus, and the processor executes the machine-readable instructions to perform the steps of any of the methods described in the first aspect above.

[0025] Fourthly, another embodiment of this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, performs the steps of any of the methods described in the first aspect above.

[0026] The beneficial effects of this application are as follows: By acquiring inference latency, action vision, multimodal context information, and action blocks in the current inference cycle, and determining the temporal weight sequence, and in each generation step of the current inference cycle, generating the current update vector based on the intermediate action block of the current generation step, the current generation time, the pre-trained VLA action head, and the multimodal context information, and determining the estimated action block corresponding to the current generation step based on the intermediate action block of the current generation step, the current generation time, and the current update vector, the optimal estimation of the real clean action block is achieved. Thus, through the temporal weight sequence, the action block of the current inference cycle, and the estimated action block corresponding to the current generation step, the action intervention information corresponding to the current generation step is determined, the deviation between the action that the model currently believes should be output and the action that is actually promised to be executed is estimated, and the action intervention information is used to implement minimal intrusive correction of the current update vector. Thus, with zero training cost and no gradient backpropagation required, hard alignment of the action to be executed and soft alignment of the future action are achieved in each generation step, which can achieve sub-millisecond alignment overhead while preserving the generation prior.

[0027] Furthermore, the temporal weight sequence is determined through inference delay and action vision, which improves the flexibility of temporal constraints. Meanwhile, the determination process of action intervention information is only related to element-wise subtraction, vector inner product, and a single scalar division of the action block, ensuring hyperparameter robustness. In addition, it can be embedded as a pluggable module in the iterative sampling loop of the VLA action head, improving scalability and compatibility with the VLA backbone. Attached Figure Description

[0028] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0029] Figure 1 A schematic diagram of asynchronous inference provided for an embodiment of this application; Figure 2 A flowchart illustrating an action timing alignment method provided in an embodiment of this application; Figure 3 This is a flowchart illustrating the process of determining the action intervention information corresponding to the current generation step in the action timing alignment method provided in this application embodiment; Figure 4 This is a flowchart illustrating the process of determining the direction correction coefficient corresponding to the current generation step in the action timing alignment method provided in this application embodiment; Figure 5 This is another flowchart illustrating the determination of the direction correction coefficient corresponding to the current generation step in the action timing alignment method provided in this application embodiment; Figure 6 This is another flowchart illustrating the process of determining the direction correction coefficient corresponding to the current generation step in the action timing alignment method provided in the embodiments of this application. Figure 7 A schematic diagram of an action timing alignment device provided in an embodiment of this application; Figure 8 This is a schematic diagram of the electronic device structure provided in an embodiment of this application. Detailed Implementation

[0030] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. It should be understood that the accompanying drawings in this application are for illustrative and descriptive purposes only and are not intended to limit the scope of protection of this application. Furthermore, it should be understood that the schematic drawings are not drawn to scale. The flowcharts used in this application illustrate operations implemented according to some embodiments of this application. It should be understood that the operations in the flowcharts may not be implemented in sequence, and steps without logical contextual relationships may be reversed or implemented simultaneously. In addition, those skilled in the art, guided by the content of this application, may add one or more other operations to the flowcharts, or remove one or more operations from the flowcharts.

[0031] Furthermore, the described embodiments are merely some, not all, of the embodiments of this application. The components of the embodiments of this application described and illustrated herein can typically be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0032] It should be noted that the term "comprising" will be used in the embodiments of this application to indicate the presence of the features declared thereafter, but does not exclude the addition of other features.

[0033] Current mainstream large-scale vision-language-action (VLA) models generally adopt a generative architecture, relying on diffusion or flow matching mechanisms, and employing an asynchronous inference paradigm. Specifically, Figure 1 This is a schematic diagram of asynchronous inference provided for an embodiment of this application, with reference to... Figure 1 As shown, the inference start time of the current inference cycle t is sub-action a. x-1 The corresponding time, the end time of reasoning is sub-action a. x-1+d At the corresponding moment, d represents the inference delay. In the current inference cycle, the next action block B is being generated based on the VLA obtained from training. At the same time, the lower-level motion controller is executing the unconsumed portion of action block A generated in the previous inference cycle. Action block A has a length of H and includes sub-actions a0~a0. H 1. The sub-actions that have been executed are a0~ a x-1+d There is still a left. x+d ~ a H 1. Pending execution, if the first action b0 of the newly generated action block B is directly connected to a... x-1+d If the action is executed later, kinematic discontinuities and a lack of connection in the state space will occur.

[0034] In kinematics, a motion fault refers to the last executed sub-motion a. x-1+d There is no kinematic / dynamic continuity guarantee between the first action b0 of the newly generated action block B and the first action b0.

[0035] In this context, the inability to connect in the state space means that the first action b0 of the newly generated action block B cannot connect with the last executed sub-action a of the action block A generated in the previous inference cycle. x-1+d Connect the end states.

[0036] The VLA used in inference was not trained to model the aforementioned issues of action gaps and the inability to connect cross-cycle states. Therefore, in asynchronous inference, timing alignment is needed to address these two problems.

[0037] Specifically, timing alignment refers to, in an asynchronous pipeline, forcing the newly generated action block B to align with the last executed sub-action a of action block A generated in the previous inference cycle on the timeline. d 1. Achieve seamless transitions that are continuous in state, connectable in dynamics, and consistent in semantics.

[0038] Existing technologies mainly propose two types of solutions to address this timing misalignment problem.

[0039] The first category is intervention schemes during the training phase, which specifically include introducing a delay-aware attention module in VLA training, constructing data augmentation samples with artificial delays, or designing a dedicated temporal alignment loss function (such as chunk-wise temporal consistency loss).

[0040] The second category is training-free online correction schemes, represented by Real-Time Chunking (RTC). Specifically, in each step of denoising / stream sampling, the gradient of the action head output with respect to the current generated state is calculated through Vector-Jacobian Product (VJP). This gradient is then superimposed on the original velocity field as a guiding signal, thereby pulling the generated trajectory toward a subspace aligned with committed actions.

[0041] However, the aforementioned existing technology has the following problems: (1) Due to the conflict between training methods and the evolution paradigm of large-scale VLA pre-training and lightweight fine-tuning, each backbone model upgrade requires repeated alignment training, making it difficult to deploy as a general plugin on existing diffusion and flow matching VLAs, resulting in poor scalability. (2) Since RTC-type methods trigger backpropagation once in each sampling step, the single inference delay is nearly doubled, resulting in a near doubling of the single inference latency. In high-frequency closed-loop control scenarios, this offsets the latency benefits brought by asynchronous inference itself and occupies a large amount of computing resources, resulting in excessive online overhead. (3) Since the VJP guiding weights need to be finely tuned, if they are too small, the alignment will be insufficient, and if they are too large, the generation prior will be destroyed. Furthermore, they cannot be reused when transferring across models, resulting in weak hyperparameter robustness. (4) Because existing solutions either impose uniform hard constraints on the entire action block or only constrain a very short commitment interval, strong jumps occur within the action block, making it impossible to balance execution continuity and task adaptability. Furthermore, under high latency and long-term tasks, the policy performance drops sharply, resulting in the problem of rigid timing constraints.

[0042] Therefore, existing processing methods suffer from poor scalability, excessive online computation overhead, weak hyperparameter robustness, and rigid timing constraints.

[0043] Based on the aforementioned problems, this application proposes an action timing alignment method. Inspired by the Control Barrier Function (CBF), it corrects the prediction results of the VLA action head, achieving timing alignment with zero training cost and without gradient backpropagation. This preserves the generation prior while achieving sub-millisecond alignment overhead. Furthermore, since the computational cost of CBF is only related to element-wise subtraction of action blocks, vector inner products, and a single scalar division, it also ensures hyperparameter robustness. In addition, it can be embedded as a pluggable module in the iterative sampling loop of the VLA action head.

[0044] It is understood that the action timing alignment method provided in this application embodiment can be deployed in any electronic device that needs to generate actions under asynchronous inference, such as a robot or a robotic arm. Specifically, the robot can be a humanoid robot or a humanoid robot.

[0045] The action timing alignment method provided in this application will be described in detail below with reference to several embodiments.

[0046] Figure 2 A flowchart illustrating an action timing alignment method provided in an embodiment of this application is shown below. Figure 2 As shown, the method includes: S201. Obtain inference delay, action view, multimodal context information and action block of the current inference cycle, and determine the temporal weight sequence based on inference delay and action view.

[0047] Here, the inference delay 'd' refers to the number of control steps taken from the start of the current inference cycle to the completion of this asynchronous inference and the output of the complete action block. The inference delay can be obtained through real-time observation; continue to refer to... Figure 1 Inference delay can be obtained from the time difference between the start and end of inference.

[0048] Here, the motion field of view *h* represents the upper limit of the motion index that the low-level motion controller can stably and reliably execute within the current control cycle. The motion field of view characterizes the boundary of the deterministic execution capability of the low-level motion controller. The motion field of view can be calculated jointly by the state feedback frequency of the low-level controller, the trajectory interpolation accuracy, the joint dynamic response bandwidth, and the safety monitoring module.

[0049] Here, the multimodal context information c refers to the joint representation vector after alignment and normalization in each inference cycle. The multimodal context information includes the current camera observation, natural language instructions, and the ontology perception state of the electronic device.

[0050] The action block of the current inference cycle includes multiple sub-actions. For example, continue to refer to... Figure 1 As shown, the action block for the current inference cycle can be action block A. The action block for the next inference cycle can be B. The length of the action block for the current inference cycle is the same as the number of sub-actions in the action block for the next inference cycle, which is H.

[0051] Where, the inference delay d ≤ action field of view h ≤ length of action block H.

[0052] Optionally, the action block of the current inference cycle can be generated by inference from the previous inference cycle. After obtaining the original action block of the current inference cycle in the previous inference cycle, the original action block of the current inference cycle can be time-aligned to align the original action block of the current inference cycle to the current local view, thereby obtaining the action block of the current inference cycle.

[0053] For example, sub-actions that have been consumed by the lower-level motion controller can be removed from the original action block of the current inference cycle, and the remaining sub-actions can be translated forward d steps along the time axis to align with the current local view, thus obtaining the action block of the current inference cycle.

[0054] For example, continue to refer to Figure 1 As shown, at the start of inference, the currently executed sub-actions a0~a0 in action block A can be... x-1 Perform elimination, and then remove the remaining sub-actions a. x~ aH 1. Translate forward d steps along the time axis to align with the current local view, and use zero padding after each remaining sub-action to maintain a fixed length H, thus obtaining the action block for the current inference cycle.

[0055] Optionally, after obtaining the inference delay and action field of view, the type of the temporal interval to which each sub-action in the action block belongs can be determined based on the inference delay and action field of view, and the corresponding temporal weight can be determined according to the type of the temporal interval to which each sub-action belongs, and after obtaining all the temporal weights, a temporal weight sequence can be obtained.

[0056] The temporal weight sequence includes multiple temporal weights arranged sequentially. This sequence is used to apply temporal constraints to each sub-action within the action block along the action axis during the action block generation process in the next inference cycle.

[0057] Specifically, the temporal weight sequence includes multiple temporal weights arranged in sequence. The number of temporal weights in the temporal weight sequence is the same as the number of sub-actions of the action block, and they correspond one-to-one. One temporal weight is used to apply a temporal constraint to a corresponding sub-action.

[0058] The types of time intervals include: intervals for actions to be executed, near-future intervals, and far-future intervals.

[0059] Specifically, the action range to be executed refers to the range that has been or is being consumed in real time by the lower-level motion controller. Pre-set near-hard constraints are applied to the action range to ensure the immediacy and stability of task execution. The near future range refers to the range that will be consumed in real time by the lower-level motion controller. Soft constraints with exponential decay are applied to the near future range to achieve a smooth transition. The long-term future range refers to the range for future long-term strategy planning. Constraints are lifted on the long-term future range to preserve the long-term task adaptability of VLM.

[0060] For example, the i-th time-series weight in the time-series weight sequence For example, the i-th time series weight can be determined. Corresponding original time series weights And obtain the i-th time series weight. Corresponding control parameters And calculate control parameters and original time series weights The product of these terms yields the i-th time-series weight. .

[0061] In one example, the i-th time-series weight Corresponding original time series weights The following formula (1) can be used to obtain the result: (1) in, For reasoning delay, The action view is H, where H is the length of the action block. , These are preset hyperparameters. This is the soft-constraint decay parameter, used to control the exponential decay rate within the soft-constraint interval. >0, A preset small constant, Used to prevent the denominator from being zero.

[0062] For example, =1, or, =0.

[0063] Specifically, the original temporal weights in the action interval [0, d) to be executed are assigned a value of 1, forming a near-hard constraint to ensure execution continuity; the original temporal weights in the near future interval [d, h) decay exponentially along the index, so that the first action to be executed is assigned a weight less than 1, thereby avoiding a jump between hard constraints and no constraints; the original temporal weights in the far future interval [h, H) are assigned a value of 0, and no temporal constraints are applied to this interval to preserve the long-range adaptability of VLA, thereby achieving triple robustness to changes in action vision length, task complexity, and deployment delay.

[0064] In other words, based on inference latency and action horizon, the system divides the action to be executed into three intervals: near-future interval and far-future interval. Then, according to the index value of the temporal weight in the temporal weight sequence, the original temporal weight is calculated within its corresponding action interval. Finally, based on the original temporal weight and the corresponding control parameters, the final temporal weight is calculated. This ensures execution continuity while preserving the long-term task adaptability of VLA. Furthermore, controlling the temporal weights through the control parameters corresponding to each temporal weight improves the flexibility in applying the temporal weights.

[0065] In another example, in addition to the exponential decay form, the original time series weights in the near future interval [d, h) can also be any monotonically non-increasing form, such as linear decay, cosine decay, and polynomial decay, to take the maximum value in the interval of actions to be executed [0, d), take zero in the far future interval [h, H), and smoothly transition in the near future interval [d, h).

[0066] Specifically, the original time series weights in the near future interval [d, h) under the linear decay form You can refer to the following formula (2): (2) Specifically, the original time series weights in the near-future interval [d, h) under the cosine decay form You can refer to the following formula (3): (3) Specifically, the original time series weights in the near-future interval [d, h) under the polynomial decay form You can refer to the following formula (4): (4) in, These are the preset parameters.

[0067] S202. In the current generation step of the current inference cycle, determine the current generation time, generate the current update vector based on the intermediate action block of the current generation step, the current generation time, the pre-trained VLA action head, and the multimodal context information, and determine the estimated action block corresponding to the current generation step based on the intermediate action block of the current generation step, the current generation time, and the current update vector.

[0068] It is understandable that after determining the temporal weight sequence of the current inference cycle, the current inference cycle includes multiple generation steps, and the action block of the next inference cycle can be generated by iterating through the multiple generation steps in the current inference cycle.

[0069] If the current generation step is the first generation step, the initial action block is sampled and used as the intermediate action block of the current generation step to generate the current update vector and the estimated action block corresponding to the current generation step.

[0070] For example, initial action blocks can be obtained by randomly sampling from a standard normal distribution. The shape of the initial action block is the same as the shape of the action block in the current inference cycle.

[0071] If the current generation step is not the first generation step, the intermediate action block output by the previous generation step is used as the intermediate action block of the current generation step to generate the current update vector and the estimated action block corresponding to the current generation step.

[0072] It is understandable that VLA action heads can be of various types, including stream matching and diffusion.

[0073] Optionally, in the current generation step, the current generation time is determined, and the intermediate action block of the current generation step, the current generation time, and the multimodal context information are input into the pre-trained VLA action head. The VLA action head infers and generates the current update vector, and the estimated action block corresponding to the current generation step is calculated based on the current update vector, the current generation time, and the intermediate action block of the current generation step.

[0074] Here, the estimated action block corresponding to the current generation step refers to the optimal estimate of the real clean action block predicted by the VLA action head under the current generation step. The estimated action block corresponding to the current generation step is the unique observation anchor point in temporal alignment and is used to characterize the temporal alignment target.

[0075] It is worth noting that in asynchronous inference scenarios of flow matching and diffusion-type VLA, the real target to be aligned is not the intermediate state under the current sampling step, but the current best estimate of the clean action block to which the trajectory will eventually converge, that is, the estimated action block corresponding to the current generation step.

[0076] In one example, taking a VLA action head of type stream matching as an example, under the current generation step, the current generation step... intermediate action block Current generation time and multimodal context information The current velocity field is generated by inputting the pre-trained VLA action head. and the current velocity field As the current update vector.

[0077] Based on this, according to the current generation time The time coefficient was calculated. And calculate the current velocity field With time coefficient The product is obtained by multiplying the product and then calculating the product result with the current generation step. intermediate action block The sum of, as the current generation step Corresponding estimated action block , For details, please refer to the following formula (5): (5) In another example, taking the VLA action head of type diffusion as an example, at the current generation step, the intermediate action block of the current generation step, the current generation time, and the multimodal context information are input into the pre-trained VLA action head, and the VLA action head outputs noise prediction, and the noise prediction is used as the current update vector.

[0078] Based on this, according to the current update vector, the current generation time, and the intermediate action block of the current generation step, the clean sample estimate is derived from the intermediate sample output under a given noise level using a preset denoising formula, thus obtaining the estimated action block corresponding to the current generation step.

[0079] S203. Based on the temporal weight sequence, the action block of the current inference cycle, and the estimated action block corresponding to the current generation step, determine the action intervention information corresponding to the current generation step.

[0080] It is understandable that after obtaining the estimated action block corresponding to the current generation step, the action block of the current inference cycle represents the action that the electronic device actually promises to perform, and the estimated action block corresponding to the current generation step represents the action that the model believes should be output. Therefore, based on the action block of the current inference cycle and the estimated action block corresponding to the current generation step, combined with the time-series weights in the time-series weight sequence, and inspired by CBF, the deviation between the action that the model currently believes should be output and the action that the electronic device actually promises to perform can be estimated. Under this deviation, the action intervention information corresponding to the current generation step can be determined, so that the current update vector output by VLA can be modified with minimal intrusion under the current generation step by using the action intervention information corresponding to the current generation step.

[0081] Alternatively, inspired by CBF, the constraint control problem that requires online optimization can be reduced to a scalar inequality that can be solved instantly. The solution is then obtained by solving the problem based on the temporal weight sequence, the action block of the current inference cycle, and the estimated action block corresponding to the current generation step.

[0082] For example, inspired by CBF, the action intervention information corresponding to the current generation step is obtained by solving the problem based on the temporal weight sequence, the action block of the current inference cycle, and the estimated action block corresponding to the current generation step, according to the preset temporal consistency condition.

[0083] Among them, the temporal consistency condition refers to the constraint relationship that must be satisfied between the action block generation flow and the execution flow in order to ensure the physical continuity of the electronic device's actions and the security of the task during asynchronous reasoning.

[0084] Specifically, the timing consistency condition can be: the starting segment of the newly generated action block can be continuously connected with the tail action that has been committed to be executed at the joint velocity level.

[0085] Among them, the action intervention information is the minimum amount of correction required in the action space to make the current update vector satisfy the temporal consistency condition.

[0086] In one example, taking the VLA action head type as flow matching, the action intervention information refers to the minimum corrective velocity that must be injected into the current velocity field in the current sampling step along the direction of the resultant force of the misalignment in order to eliminate the current temporal misalignment.

[0087] For example, taking the VLA action head type as flow matching, the temporal consistency condition includes: the weighted temporal deviation between the action block of the current inference cycle after being weighted by the temporal weight sequence and the estimated action block corresponding to the current generation step, in the analytical correction direction formed in the action space, the projected component of its velocity field after correction must not be less than the corresponding lower bound of the obstacle.

[0088] S204. Apply the action intervention information corresponding to the current generation step to the current update vector to obtain the updated vector. Based on the updated vector, generate the intermediate action block of the next generation step of the current generation step, execute iteratively, and after the iteration ends, use the intermediate action block at the end of the iteration as the action block of the next inference cycle.

[0089] Optionally, after obtaining the action intervention information corresponding to the current generation step, the action intervention information corresponding to the current generation step is applied to the current update vector to obtain the updated vector. The product of the updated vector and the step size is then calculated, and the product is added to the current generation step. intermediate action block Above, obtain the intermediate action block for the next generation step of the current generation step.

[0090] In one example, taking a VLA action head of type flow matching as an example, the current velocity field is... Apply current generation step Corresponding action intervention information The updated vector is obtained, and the updated vector and step size are calculated. The product of the two is then added to the current generation step. intermediate action block Above, obtain the current generation step. The next generation step intermediate action block For details, please refer to the following formula (6): (6) Among them, step size It can be pre-configured, or obtained through the next generation step. The generation time and the current generation step The generation time was calculated.

[0091] Optionally, the process is iterated until all generation steps in the current inference cycle have been completed. After the iteration ends, the intermediate action block at the end of the iteration is used as the action block for the next inference cycle.

[0092] For example, the action block of the next inference cycle is consumed step by step during the asynchronous inference cycle as the electronic device executes, and S201 is re-executed as the action block of the current inference cycle when the next current inference cycle arrives, thereby forming a closed-loop real-time aligned inference pipeline.

[0093] In this embodiment, by acquiring inference latency, action vision, multimodal context information, and action blocks of the current inference cycle, and determining the temporal weight sequence, and in each generation step of the current inference cycle, generating the current update vector based on the intermediate action block of the current generation step, the current generation time, the pre-trained VLA action head, and the multimodal context information, and determining the estimated action block corresponding to the current generation step based on the intermediate action block of the current generation step, the current generation time, and the current update vector, the optimal estimation of the real clean action block is achieved. Thus, by using the temporal weight sequence, the action block of the current inference cycle, and the estimated action block corresponding to the current generation step, the action intervention information corresponding to the current generation step is determined, and the deviation between the action that the model currently believes should be output and the action that is actually promised to be executed is estimated. The action intervention information is used to perform minimal intrusive correction on the current update vector, thereby achieving hard alignment of the action to be executed and soft alignment of the future action in each generation step with zero training cost and no gradient backpropagation required. This can achieve sub-millisecond alignment overhead while preserving the generation prior.

[0094] Furthermore, the temporal weight sequence is determined through inference delay and action vision, which improves the flexibility of temporal constraints. Meanwhile, the determination process of action intervention information is only related to element-wise subtraction, vector inner product, and a single scalar division of the action block, ensuring hyperparameter robustness. In addition, it can be embedded as a pluggable module in the iterative sampling loop of the VLA action head, improving scalability and compatibility with the VLA backbone.

[0095] As one possible implementation method, Figure 3 This is a flowchart illustrating the process of determining the action intervention information corresponding to the current generation step in the action timing alignment method provided in this application embodiment, with reference to... Figure 3 As shown, in S203 above, based on the temporal weight sequence, the action block of the current inference cycle, and the estimated action block corresponding to the current generation step, the action intervention information corresponding to the current generation step is determined, including: S301. Based on the temporal weight sequence, the action block of the current inference cycle, and the estimated action block corresponding to the current generation step, determine the correction direction vector sequence corresponding to the current generation step.

[0096] It can be understood that the action intervention information represents the control direction and control intensity when correcting the current update vector. Therefore, the action intervention information can be decoupled into the correction direction and correction magnitude, and determined separately.

[0097] Optionally, based on the action block of the current inference cycle and the estimated action block corresponding to the current generation step, the deviation between the action that the model currently believes should be output and the action that the electronic device actually promises to execute is estimated, and the deviation is weighted by the time-series weight sequence to obtain the correction direction vector sequence corresponding to the current generation step.

[0098] The correction direction vector sequence includes multiple correction direction vectors. The number of correction direction vectors in the sequence is the same as the number of sub-actions in the action block, and they correspond one-to-one. Each correction direction vector represents the correction direction of a corresponding sub-action.

[0099] By using the temporal weight sequence, the action block of the current inference cycle, and the estimated action block corresponding to the current generation step, the corrected direction vector sequence corresponding to the current generation step is determined. This makes the generation of the corrected direction vector independent of the VLA model structure, relying only on the action block of the current inference cycle and the estimated action block corresponding to the current generation step. This avoids backpropagation to the VLA backbone and VLA action head, reducing computational overhead. Furthermore, by providing the corrected direction vector with spatiotemporal sensitivity and action importance through the temporal weight sequence, the flexibility of temporal constraints is improved.

[0100] In one example, each correction direction vector in the sequence of correction direction vectors corresponding to the current generation step can also be obtained by calculating the subgradient of the temporal consistency score.

[0101] S302. Based on the current generation time, time-series weight sequence, action block of the current inference cycle, correction direction vector sequence corresponding to the current generation step, and estimated action block corresponding to the current generation step, determine the direction correction coefficient corresponding to the current generation step.

[0102] It is understandable that, inspired by CBF, in the process of timing alignment, if the action block space is regarded as the state space, then the action trajectory should always be within the safe set, that is, it cannot cross the safe boundary, and the closer the action is to the safe boundary, the greater the correction force should be given.

[0103] Optionally, a barrier boundary is constructed through the temporal consistency condition, and the direction correction coefficient corresponding to the current generation step is obtained by solving under the barrier boundary based on the current generation time, temporal weight sequence, action block of the current inference cycle, correction direction vector sequence corresponding to the current generation step, and estimated action block corresponding to the current generation step. Thus, the correction magnitude under the correction direction is controlled through the direction correction coefficient.

[0104] Specifically, the temporal consistency condition can be characterized as follows: the inner product of the corrected direction vector sequence corresponding to the current generation step and the updated vector is not less than the corresponding obstacle boundary.

[0105] The direction correction factor is used to characterize the correction magnitude and is the minimum scalar scaling factor required to satisfy the timing consistency condition.

[0106] S303. Calculate the product of each correction direction vector and the direction correction coefficient in the correction direction vector sequence to obtain the action intervention information corresponding to the current generation step.

[0107] Optionally, the product of each corrected direction vector and the direction correction coefficient in the corrected direction vector sequence is calculated to obtain the action intervention information corresponding to the current generation step.

[0108] As one possible implementation, in step S301 above, the correction direction vector sequence corresponding to the current generation step is determined based on the temporal weight sequence, the action block of the current inference cycle, and the estimated action block corresponding to the current generation step, including: The algorithm iterates through each sub-action in the estimated action block. For the current sub-action, it determines the temporal weight corresponding to the current sub-action from the temporal weight sequence and determines the reference sub-action from the action block of the current inference cycle. Based on the temporal weight of the current sub-action, the current sub-action, and the reference sub-action, it calculates the correction direction vector corresponding to the current sub-action. After the iteration is completed, the correction direction vectors of all sub-actions are combined into the correction direction vector sequence corresponding to the current generation step.

[0109] Optionally, iterate through the estimated action blocks. Each sub-action in the process, for the current sub-action being traversed. Determine the current sub-action from the temporal weight sequence. Corresponding time series weights And determine the current sub-action from action block A of the current inference cycle. Reference sub-action And based on the current sub-action Time-series weights Current sub-action and reference sub-actions Calculate the current sub-action Corresponding correction direction vector .

[0110] In one example, the current sub-action can be calculated. and reference sub-actions The difference is calculated, the result is obtained, and then processed through the current sub-action. Time-series weights The difference results are weighted to obtain the current sub-action. Corresponding correction direction vector .

[0111] Optionally, after the traversal is complete, the correction direction vectors of all sub-actions are arranged in order to obtain the sequence of correction direction vectors corresponding to the current generation step. .

[0112] As one possible implementation, the above steps calculate the correction direction vector corresponding to the current sub-action based on the temporal weight of the current sub-action, the current sub-action, and the reference sub-action, including: Calculate the difference between the current sub-action and the reference sub-action to obtain the action difference result; based on the action difference result and the temporal weight of the current sub-action, calculate the correction direction vector corresponding to the current sub-action.

[0113] Optionally, the difference between the current sub-action and the reference sub-action can be calculated to obtain the action difference result. Then, the action difference result can be weighted according to the analytical negative direction of the temporal weight to obtain the correction direction vector. Thus, the correction direction vector can be obtained without performing gradient calculation. At the same time, the calculation of the correction direction vector involves element-wise subtraction and bit-wise weighting of the action. Its computational cost is linearly related to the size of the action block and is completely independent of the VLA backbone parameters. It also improves compatibility, reduces computational overhead, and reduces noise sensitivity.

[0114] Furthermore, by weighting the action difference results according to the temporal weights to obtain the correction direction vector, the long-term task adaptability of VLA can be preserved while ensuring the continuity of execution.

[0115] Among them, the action difference result is used to characterize the temporal difference between the action that the model currently believes should be output and the action that the electronic device actually promises to perform.

[0116] For example, obtain the current sub-action Time-series weights and reference sub-actions Then, the current sub-action can be calculated by referring to the following formula (7). Corresponding correction direction vector : (7) As one possible implementation method, Figure 4 This is a flowchart illustrating the process of determining the direction correction coefficient corresponding to the current generation step in the action timing alignment method provided in this application embodiment, referring to... Figure 4 As shown, in S302 above, the direction correction coefficient corresponding to the current generation step is determined based on the current generation time, the temporal weight sequence, the action block of the current inference cycle, the correction direction vector sequence corresponding to the current generation step, and the estimated action block corresponding to the current generation step, including: S401. Determine the temporal consistency score based on the temporal weight sequence, the action block of the current inference cycle, and the estimated action block corresponding to the current generation step.

[0117] Optionally, based on the action block of the current inference cycle and the estimated action block corresponding to the current generation step, the deviation between the action that the model currently believes should be output and the action that the electronic device actually promises to perform can be estimated, and the deviation can be weighted and summed through the temporal weight sequence to obtain the temporal consistency score.

[0118] The timing consistency score is used to characterize the total timing deviation between the action that the model currently believes should be output and the action that the electronic device actually promises to perform. The timing consistency score is not greater than zero, and the closer the value is to zero, the smaller the total timing deviation.

[0119] In the first example, the time consistency score is obtained by weighting the deviations in L2 form using the time-weighted sequence.

[0120] In the second example, the time series consistency score is obtained by weighting and summing the deviations in the form of weighted L1 residuals using the time series weight sequence.

[0121] In the third example, the time consistency score is obtained by weighting and summing the deviations in the form of Huber residuals using a time-weighted sequence.

[0122] In the fourth example, an anisotropic weight matrix can be introduced into the action dimension to reflect the differences in importance of different joint / end-effector coordinates, and a temporal consistency score can be obtained by weighted summation through a temporal weight sequence.

[0123] S402. Determine obstacle boundary information based on the temporal consistency score and the current generation time.

[0124] Optionally, after obtaining the temporal consistency score, obstacle boundary information can be calculated based on the temporal consistency score and the current generation time.

[0125] Among them, obstacle boundary information is used to indicate the safety boundary of the action.

[0126] In one example, obstacle boundary information can be characterized by an exponential family of lower bounds.

[0127] In another example, obstacle boundary information can be characterized by a lower bound on a family of numbers.

[0128] S403. Based on the obstacle boundary information and the correction direction vector sequence corresponding to the current generation step, determine the direction correction coefficient corresponding to the current generation step.

[0129] Optionally, after obtaining the obstacle boundary information and the correction direction vector sequence corresponding to the current generation step, the direction correction coefficient corresponding to the current generation step can be obtained by combining the obstacle boundary information and the correction direction vector sequence corresponding to the current generation step with the temporal consistency condition, and applying non-negativity constraints. This allows for the generation step to be modified without any correction when the current update vector itself already satisfies the temporal consistency condition, thus fully preserving the generation prior of VLA. When the current update vector violates the temporal consistency condition, a projection correction is applied with the minimum magnitude that exactly cancels out the violation, achieving on-demand correction. This avoids continuous interference with the generation prior learned by VLA, significantly improving the continuity and smoothness of the generation action, resulting in a smoother execution trajectory and fewer sudden stops or starts during the operation of real electronic devices.

[0130] As one possible implementation, in step S401 above, the temporal consistency score is determined based on the temporal weight sequence, the action block of the current inference cycle, and the estimated action block corresponding to the current generation step, including: The algorithm iterates through each sub-action in the estimated action block. For the current sub-action, it determines the temporal weight of the current sub-action from the temporal weight sequence and determines the reference sub-action of the current sub-action from the action block of the current inference cycle. Based on the temporal weight of the current sub-action, the current sub-action, and the reference sub-action, it calculates the weighted temporal residual corresponding to the current sub-action. After the iteration is completed, it calculates the temporal consistency score based on the weighted temporal residuals corresponding to each sub-action.

[0131] Optionally, iterate through the estimated action blocks. Each sub-action in the process, for the current sub-action being traversed. Determine the current sub-action from the temporal weight sequence. Corresponding time series weights And determine the current sub-action from action block A of the current inference cycle. Reference sub-action And based on the current sub-action Time-series weights Current sub-action and reference sub-actions Calculate the current sub-action Corresponding weighted time series residuals .

[0132] Optionally, the weighted temporal residuals corresponding to each sub-action... Take the norm, square it, and finally take the negative half and sum them up to obtain the timing consistency score. .

[0133] For example, the timing consistency score is calculated. You can refer to the following formulas (8)-(9): (8) (9) In another example, the current sub-action Corresponding weighted time series residuals It can also be done by analyzing the current sub-action. Corresponding correction direction vector Taking the negative yields the result.

[0134] By calculating the temporal weight of the current sub-action, the current sub-action, and the reference sub-action, the weighted temporal residual corresponding to the current sub-action is obtained. Then, the temporal consistency score is calculated using the weighted temporal residuals corresponding to each sub-action. Thus, the temporal consistency score is obtained without gradient solving. At the same time, the calculation of the temporal consistency score involves element-wise subtraction and bit-wise weighting of actions. Its computational cost is linearly related to the size of the action block and is completely independent of the VLA backbone parameters. It also improves compatibility, reduces computational overhead, and reduces noise sensitivity.

[0135] As one possible implementation, in step S402 above, obstacle boundary information is determined based on the timing consistency score and the current generation time, including: Based on the current generation time, the time coefficient is determined; based on the time coefficient, the temporal consistency score, and the preset correction intensity parameters, the obstacle boundary information is calculated.

[0136] Optionally, based on the current generation time The time coefficient was calculated. and time coefficient Timing consistency score and preset correction strength parameters Obtain obstacle boundary information For details, please refer to the following formula (10): (10) in, For asymptotically shrinking normalization factor, It can be calculated using the following formula (11): (11) in, This is the preset minimum value.

[0137] in, At the current generation time As it approaches 1, it tends to a minimum value, thus equivalently causing the projection intensity to gradually increase as the generation process progresses. Because... ,therefore, ,and The value increases significantly as the generation process approaches 1, which in turn requires the projection of the corrected velocity along the direction g to become increasingly larger.

[0138] In another example, taking the lower bound of the exponential family as an example, barrier boundary information It can also be calculated using the following formula (12): (12) In another example, taking the lower bound of the logarithmic family as an example, barrier boundary information It can also be calculated using the following formula (13): (13) In calculating obstacle boundary information Different calculation methods can be used to obtain different degrees of radical correction, thereby achieving finer-grained adjustment of alignment performance on different tasks.

[0139] As one possible implementation method, Figure 5 This is another flowchart illustrating the determination of the direction correction coefficient corresponding to the current generation step in the action timing alignment method provided in this application embodiment, referred to [reference needed]. Figure 5 As shown, in step S403 above, the direction correction coefficient corresponding to the current generation step is determined based on the obstacle boundary information and the correction direction vector sequence corresponding to the current generation step, including: S501. Determine the basic correction amount based on the correction direction vector sequence corresponding to the current generation step and the current update vector.

[0140] Optionally, the sequence of corrected direction vectors corresponding to the current generation step can be calculated. transpose Compared with the current update vector The product of these two factors yields the basic correction amount. .

[0141] S502. Determine the normalization factor based on the correction direction vector sequence corresponding to the current generation step.

[0142] Optionally, based on the correction direction vector sequence corresponding to the current generation step. The normalization factor was calculated. .in, This is the preset minimum value.

[0143] S503. Based on the obstacle boundary information, the basic correction amount, and the normalization factor, determine the direction correction coefficient corresponding to the current generation step.

[0144] Optionally, after obtaining the basic correction amount and the normalization factor, the initial direction correction coefficient can be calculated based on the obstacle boundary information, the basic correction amount and the normalization factor, and a non-negativity constraint can be applied to the initial direction correction coefficient to obtain the direction correction coefficient corresponding to the current generation step.

[0145] As one possible implementation method, Figure 6 This is another flowchart illustrating the determination of the direction correction coefficient corresponding to the current generation step in the action timing alignment method provided in this application embodiment, referred to [reference needed]. Figure 6 As shown, based on obstacle boundary information, basic correction amount, normalization factor, and preset boundary conditions, the direction correction coefficient corresponding to the current generation step is determined, including: S601. Calculate the difference between the obstacle boundary information and the basic correction amount to obtain the difference result.

[0146] For example, calculating obstacle boundary information With basic correction amount The difference is obtained by finding the difference result. .

[0147] S602. Calculate the ratio of the difference result to the normalization factor to obtain the corrected result.

[0148] For example, the difference results are calculated. With normalization factor The ratio is used to obtain the corrected result.

[0149] S603. If the correction result is greater than or equal to the preset lower limit value, the correction result shall be used as the direction correction coefficient.

[0150] For example, the correction result is judged, and if the correction result is greater than or equal to a preset lower bound value, the correction result is used as the direction correction coefficient. The preset lower bound value can be 0.

[0151] S604. If the correction result is less than the preset lower bound value, the lower bound value shall be used as the direction correction coefficient.

[0152] For example, the correction result is judged, and if the correction result is less than the preset lower bound value, the lower bound value is used as the direction correction coefficient.

[0153] For example, the direction correction factor It can be calculated using the following formula (14): (14) In practical applications, taking the VLA action head type as a stream matching type as an example, we can use K = 10 and Δt = 1 / K as the stream matching sampling settings, and take α = 1.0, β = 2.0, and ε = 1e 6. Execute the action timing alignment method provided in the embodiments of this application to achieve timing alignment.

[0154] Based on the same inventive concept, this application also provides an action timing alignment device corresponding to the action timing alignment method. Since the principle of the device in this application is similar to the action timing alignment method described above in this application, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be described again.

[0155] Reference Figure 7 As shown, Figure 7 This is a schematic diagram of an action timing alignment device provided in an embodiment of this application. The device includes: The acquisition module 701 is used to acquire inference delay, action vision, multimodal context information and action block of the current inference cycle, and determine the temporal weight sequence based on the inference delay and action vision. The temporal weight sequence includes multiple temporal weights arranged in sequence. The first determining module 702 is used to determine the current generation time in the current generation step of the current inference cycle, generate the current update vector based on the intermediate action block of the current generation step, the current generation time, the pre-trained VLA action head and multimodal context information, and determine the estimated action block corresponding to the current generation step based on the intermediate action block of the current generation step, the current generation time and the current update vector. The second determining module 703 is used to determine the action intervention information corresponding to the current generation step based on the time-series weight sequence, the action block of the current inference cycle, and the estimated action block corresponding to the current generation step. The iteration module 704 is used to apply the action intervention information corresponding to the current generation step to the current update vector to obtain the updated vector, and generate the intermediate action block of the next generation step of the current generation step according to the updated vector, execute iteratively, and after the iteration ends, use the intermediate action block at the end of the iteration as the action block of the next inference cycle.

[0156] Optionally, the second determining module 703 is specifically used for: Based on the temporal weight sequence, the action block of the current inference cycle, and the estimated action block corresponding to the current generation step, the correction direction vector sequence corresponding to the current generation step is determined, and the correction direction vector sequence includes multiple correction direction vectors. Based on the current generation time, the temporal weight sequence, the action block of the current inference cycle, the correction direction vector sequence corresponding to the current generation step, and the estimated action block corresponding to the current generation step, determine the direction correction coefficient corresponding to the current generation step. Calculate the product of each corrected direction vector and the direction correction coefficient in the corrected direction vector sequence to obtain the action intervention information corresponding to the current generation step.

[0157] Optionally, the second determining module 703 is specifically used for: Traverse each sub-action in the estimated action block. For the current sub-action, determine the temporal weight corresponding to the current sub-action from the temporal weight sequence, and determine the reference sub-action of the current sub-action from the action block of the current inference cycle. Based on the temporal weight of the current sub-action, the current sub-action, and the reference sub-action, calculate the correction direction vector corresponding to the current sub-action. After the traversal is completed, the correction direction vectors of all sub-actions are combined into a sequence of correction direction vectors corresponding to the current generation step.

[0158] Optionally, the second determining module 703 is specifically used for: Calculate the difference between the current sub-action and the reference sub-action to obtain the action difference result; Based on the action difference result and the temporal weight of the current sub-action, the correction direction vector corresponding to the current sub-action is calculated.

[0159] Optionally, the second determining module 703 is specifically used for: The temporal consistency score is determined based on the temporal weight sequence, the action block of the current inference cycle, and the estimated action block corresponding to the current generation step. Based on the temporal consistency score and the current generation time, the obstacle boundary information is determined; Based on the obstacle boundary information and the correction direction vector sequence corresponding to the current generation step, determine the direction correction coefficient corresponding to the current generation step.

[0160] Optionally, the second determining module 703 is specifically used for: Iterate through each sub-action in the estimated action block. For the current sub-action, determine the temporal weight of the current sub-action from the temporal weight sequence, and determine the reference sub-action of the current sub-action from the action block of the current inference cycle. Based on the temporal weight of the current sub-action, the current sub-action, and the reference sub-action, calculate the weighted temporal residual corresponding to the current sub-action. After the traversal is completed, the timing consistency score is calculated based on the weighted timing residuals corresponding to each sub-action.

[0161] Optionally, the second determining module 703 is specifically used for: Determine the time coefficient based on the current generation time; The obstacle boundary information is calculated based on the time coefficient, the temporal consistency score, and the preset correction intensity parameters.

[0162] Optionally, the second determining module 703 is specifically used for: The basic correction amount is determined based on the correction direction vector sequence corresponding to the current generation step and the current update vector; The normalization factor is determined based on the sequence of correction direction vectors corresponding to the current generation step. Based on obstacle boundary information, basic correction amount, and normalization factor, determine the direction correction coefficient corresponding to the current generation step.

[0163] Optionally, the second determining module 703 is specifically used for: Calculate the difference between the obstacle boundary information and the basic correction amount to obtain the difference result; The ratio of the difference result to the normalization factor is calculated to obtain the corrected result; If the correction result is greater than or equal to the preset lower bound value, the correction result will be used as the direction correction coefficient. If the correction result is less than the preset lower bound value, the lower bound value will be used as the direction correction coefficient.

[0164] The processing flow of each module in the device and the interaction flow between each module can be referred to the relevant descriptions in the above method embodiments, and will not be detailed here.

[0165] This application also provides an electronic device, such as... Figure 8 As shown, Figure 8 The schematic diagram of the electronic device structure provided in the embodiments of this application includes: a processor 801 and a memory 802, and optionally, a bus 803. The memory 802 stores machine-readable instructions that can be executed by the processor 801. When the electronic device is running, the processor 801 and the memory 802 communicate through the bus 803, and the processor 801 executes the machine-readable instructions to perform the steps of the above-mentioned action timing alignment method.

[0166] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the steps of the above-described timing alignment method.

[0167] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems and devices described above can be referred to the corresponding processes in the method embodiments, and will not be repeated here. In the several embodiments provided in this application, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed mutual coupling or direct coupling or communication connection can be through some communication interfaces; the indirect coupling or communication connection of devices or modules can be electrical, mechanical, or other forms.

[0168] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. If the functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes: USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, optical disks, and other media capable of storing program code.

[0169] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

Claims

1. A method for aligning action sequences, characterized in that, include: The system acquires inference latency, action field of view, multimodal context information, and action blocks in the current inference cycle. Based on the inference latency and the action field of view, it determines a temporal weight sequence, which includes multiple temporal weights arranged in sequence. In the current generation step of the current inference cycle, the current generation time is determined, and the current update vector is generated based on the intermediate action block of the current generation step, the current generation time, the pre-trained VLA action head, and the multimodal context information. The estimated action block corresponding to the current generation step is determined based on the intermediate action block of the current generation step, the current generation time, and the current update vector. Based on the temporal weight sequence, the action block of the current inference cycle, and the estimated action block corresponding to the current generation step, determine the action intervention information corresponding to the current generation step; Apply the action intervention information corresponding to the current generation step to the current update vector to obtain the updated vector, and generate the intermediate action block of the next generation step of the current generation step according to the updated vector. Iterate and execute, and after the iteration ends, use the intermediate action block at the end of the iteration as the action block of the next inference cycle.

2. The action timing alignment method according to claim 1, characterized in that, The step of determining the action intervention information corresponding to the current generation step based on the temporal weight sequence, the action block of the current inference cycle, and the estimated action block corresponding to the current generation step includes: Based on the temporal weight sequence, the action block of the current inference cycle, and the estimated action block corresponding to the current generation step, a correction direction vector sequence corresponding to the current generation step is determined, wherein the correction direction vector sequence includes multiple correction direction vectors; Based on the current generation time, the temporal weight sequence, the action block of the current inference cycle, the correction direction vector sequence corresponding to the current generation step, and the estimated action block corresponding to the current generation step, determine the direction correction coefficient corresponding to the current generation step; Calculate the product of each corrected direction vector in the corrected direction vector sequence and the direction correction coefficient to obtain the action intervention information corresponding to the current generation step.

3. The action timing alignment method according to claim 2, characterized in that, The step of determining the correction direction vector sequence corresponding to the current generation step based on the temporal weight sequence, the action block of the current inference cycle, and the estimated action block corresponding to the current generation step includes: Traverse each sub-action in the estimated action block, for the current sub-action, determine the temporal weight corresponding to the current sub-action from the temporal weight sequence, determine the reference sub-action of the current sub-action from the action block of the current inference cycle, and calculate the correction direction vector corresponding to the current sub-action based on the temporal weight of the current sub-action, the current sub-action, and the reference sub-action. After the traversal is completed, the correction direction vectors of all sub-actions are combined into a sequence of correction direction vectors corresponding to the current generation step.

4. The action timing alignment method according to claim 3, characterized in that, The step of calculating the correction direction vector corresponding to the current sub-action based on the temporal weight of the current sub-action, the current sub-action, and the reference sub-action includes: Calculate the difference between the current sub-action and the reference sub-action to obtain the action difference result; Based on the action difference result and the temporal weight of the current sub-action, the correction direction vector corresponding to the current sub-action is calculated.

5. The action timing alignment method according to claim 2, characterized in that, The step of determining the direction correction coefficient corresponding to the current generation step based on the current generation time, the temporal weight sequence, the action block of the current inference cycle, the correction direction vector sequence corresponding to the current generation step, and the estimated action block corresponding to the current generation step includes: The temporal consistency score is determined based on the temporal weight sequence, the action block of the current inference cycle, and the estimated action block corresponding to the current generation step. Based on the temporal consistency score and the current generation time, the obstacle boundary information is determined; Based on the obstacle boundary information and the correction direction vector sequence corresponding to the current generation step, the direction correction coefficient corresponding to the current generation step is determined.

6. The action timing alignment method according to claim 5, characterized in that, The step of determining the temporal consistency score based on the temporal weight sequence, the action block of the current inference cycle, and the estimated action block corresponding to the current generation step includes: Traverse each sub-action in the estimated action block, for the current sub-action, determine the temporal weight of the current sub-action from the temporal weight sequence, determine the reference sub-action of the current sub-action from the action block of the current inference cycle, and calculate the weighted temporal residual corresponding to the current sub-action based on the temporal weight of the current sub-action, the current sub-action, and the reference sub-action. After the traversal is completed, the timing consistency score is calculated based on the weighted timing residuals corresponding to each sub-action.

7. The action timing alignment method according to claim 5, characterized in that, The step of determining obstacle boundary information based on the temporal consistency score and the current generation time includes: Determine the time coefficient based on the current generation time; The obstacle boundary information is calculated based on the time coefficient, the temporal consistency score, and the preset correction intensity parameter.

8. The action timing alignment method according to claim 5, characterized in that, The step of determining the direction correction coefficient corresponding to the current generation step based on the obstacle boundary information and the correction direction vector sequence corresponding to the current generation step includes: The basic correction amount is determined based on the correction direction vector sequence corresponding to the current generation step and the current update vector; The normalization factor is determined based on the sequence of correction direction vectors corresponding to the current generation step; Based on the obstacle boundary information, the basic correction amount, and the normalization factor, the direction correction coefficient corresponding to the current generation step is determined.

9. The action timing alignment method according to claim 8, characterized in that, The step of determining the direction correction coefficient corresponding to the current generation step based on the obstacle boundary information, the basic correction amount, the normalization factor, and the preset boundary conditions includes: Calculate the difference between the obstacle boundary information and the basic correction amount to obtain the difference result; The ratio of the difference result to the normalization factor is calculated to obtain the correction result; If the correction result is greater than or equal to the preset lower bound value, the correction result is used as the direction correction coefficient; If the correction result is less than a preset lower bound, the lower bound is used as the direction correction coefficient.

10. An electronic device, characterized in that, include: A processor and a memory, the memory storing machine-readable instructions executable by the processor, which, when the electronic device is running, are executed by the processor to perform the steps of the action timing alignment method as described in any one of claims 1 to 9.

Images