Robot asynchronous operation method and device based on optical flow prediction

Abstract

The invention discloses a robot asynchronous operation method and device based on optical flow prediction, and relates to the technical field of robot asynchronous operation strategies, and the method comprises the steps: carrying out the feature extraction and modeling of a historical visual frame, outputting an object flow through a lightweight flow prediction module, and carrying out the rendering and synthesis of future observation; comparing and learning with a time mask, aligning and synthesizing real future observation characteristics, and forming consistent future visual representation; inputting the future visual representation and the ontology perception state into a diffusion strategy action generation network, generating an action sequence aligned with a time sequence, and constructing a queue; depending on a diffusion strategy asynchronous architecture, a modeling system delays and determines a prediction view, a queue is dynamically maintained, an alignment action is output, and robot asynchronous operation in a delay scene is completed. According to the method, by supplementing future observation and realizing time sequence alignment, dynamic task optimization is realized, and the problems of asynchronous reasoning time sequence deviation and response lag are solved.

Description

Technical Field

[0001] This invention relates to the field of asynchronous robot operation strategy technology, and in particular to an asynchronous robot operation method and apparatus based on optical flow prediction. Background Technology

[0002] With the continuous development of intelligent perception and decision-making algorithms, robot autonomous control technology has made breakthroughs in scenarios such as dynamic interaction, precision operation, and autonomous movement, showing broad application prospects in fields such as industrial automation, intelligent warehousing, and service robots. However, existing end-to-end control strategies generally have inherent defects: the policy inference process is accompanied by significant computational overhead, and the resulting latency problem severely restricts the deployment of robot systems in real-world scenarios. In the traditional synchronous execution mode, observation and acquisition, policy inference, and low-level control are executed sequentially, which easily leads to problems such as delayed robot action response and stiff and discontinuous motion trajectories. Especially when facing high-speed interactive tasks such as dynamic target tracking and moving object grasping, the environmental state is constantly changing, and the slow response speed cannot match the real-time requirements of dynamic scenarios, directly resulting in a significant decrease in task success rate.

[0003] To alleviate the latency bottleneck caused by synchronous execution, asynchronous inference frameworks, which process model inference and action execution in parallel, have gradually become the mainstream technical approach. However, asynchronous architectures do not fundamentally eliminate system latency. The total latency generated by the superposition of observation transmission, model inference, and low-level control still results in a significant temporal misalignment between the action commands output by the policy and the real-time environment state. To ensure execution safety and rationality, existing systems are usually forced to discard early action fragments generated in each round of inference. This not only further exacerbates execution lag but also causes breaks in the connection between consecutive action chunks, significantly reducing motion continuity and control precision. In dynamic target interaction scenarios, the temporal misalignment problem is further amplified, making it difficult for control policies generated based on lagging information to consistently align with the target state, thus failing to achieve stable and accurate dynamic control.

[0004] Existing research has attempted to optimize policy timing by predicting future proprioceptive states. Representative methods such as VLASH alleviate action lag by introducing future proprioceptive state predictions. However, these methods still heavily rely on current and historical visual observations, failing to acquire future visual context information. Due to the lack of future visual cues, policy networks can only make approximate inferences about future actions based on incomplete historical information, rather than conducting precise reasoning based on a complete spatiotemporal context. This results in robustness and control precision in dynamic environments failing to meet practical requirements. Faced with a series of technical challenges such as temporal misalignment, missing visual information, and delayed action response in complex dynamic scenarios, current asynchronous policy frameworks still cannot achieve efficient, accurate, and coherent real-time robot control.

[0005] Therefore, in the field of asynchronous robot control for dynamic interactive tasks, there is an urgent need for an efficient strategy reasoning method that can fully utilize future visual context, eliminate timing deviations, and improve the accuracy of dynamic manipulation. Summary of the Invention

[0006] The main objective of this invention is to provide a method for asynchronous robot operation based on optical flow prediction.

[0007] Another objective of this invention is to provide an asynchronous robot operation device based on optical flow prediction.

[0008] The third objective of this invention is to provide an electronic device.

[0009] A fourth objective of this invention is to provide a non-transitory computer-readable storage medium.

[0010] To achieve the above objectives, a first aspect of the present invention proposes a method for asynchronous robot operation based on optical flow prediction, comprising:

[0011] Feature extraction and spatiotemporal modeling are performed on historical visual frames. A lightweight flow prediction module outputs an object flow that represents the motion trend of objects. Based on the object flow, rendering and synthesis of future observations are completed. A contrastive learning approach with temporal masking is used to align the synthesized future observation features with the real future observation features, forming a future visual representation consistent with real perception. The aligned future visual representation and the ontology perception state are jointly input into the action generation network based on the diffusion strategy to generate action sequences that are explicitly aligned with future timestamps, thus constructing a temporally unified action queue. An asynchronous decision-making architecture is constructed based on a diffusion strategy. The aligned future visual representation is accessed, and the predicted field of view length is determined by the total delay of the explicit modeling system. The action queue is dynamically maintained and an action sequence aligned with the real timestamp is generated to complete the time-series asynchronous operation in robot delay scenarios.

[0012] Optionally, feature extraction and spatiotemporal modeling are performed on historical visual frames, and a lightweight flow prediction module is used to output an object flow representing the motion trend of objects, including: Extract corresponding image feature information from consecutive historical visual frames and generate a heat map for locating object regions; A weighted clustering algorithm is used to perform soft sampling. Based on the response values ​​of each point in the heatmap, the corresponding spatial coordinates are aggregated in a weighted manner to obtain a rough location result of the starting point of the object flow. The local correlation characteristics between features of different time frames are calculated, and the fine-grained correction values ​​of the starting point of the stream and the corresponding motion displacement vector are output by the stream decoder to complete the prediction output of the object stream.

[0013] Optionally, the process of rendering and compositing future observations based on object flow includes: The predicted object flow vectors are assigned corresponding representation information according to preset rules and then rendered and mapped onto the currently observed image. The rendered flow vectors visually represent the magnitude and direction of displacement from the origin, and the flow information is used to synthesize and generate future observation images.

[0014] Optionally, a contrastive learning approach with a temporal mask is employed to impose alignment constraints on the synthesized future observation features and the real future observation features, including: The synthetic future observations and the real future observations are respectively input into a shared feature encoder and mapped to a unified latent feature space; A temporal mask screening mechanism is constructed to filter frame features with close time intervals, and only samples whose temporal distance meets the set conditions are retained to participate in comparative learning; With positive sample feature alignment and true negative sample feature separation as the optimization directions, bidirectional constraints are imposed on predicted features and true features, and feature space alignment is completed through symmetric optimization.

[0015] Optionally, the process of forming a future visual representation consistent with real perception includes: A contrastive learning objective is used to narrow the distribution distance between the predicted features after flow enhancement and the actual future observed features; In the feature matching process, feature interference from frames with adjacent times is eliminated to improve the effectiveness of feature alignment; By measuring the degree of feature matching through normalized similarity, the reliability of the representation of predicted observations is continuously optimized, the representational differences between synthetic and real observations are eliminated, and a future visual representation consistent with real perception is formed.

[0016] Optionally, the aligned future visual representation and the ontology-aware state are jointly input into an action generation network constructed based on a diffusion strategy to generate action sequences explicitly aligned with future timestamps, thus constructing a temporally unified action queue, including: An action generation network structure is built based on a diffusion strategy, and an encoding and decoding structure adapted to visual features is adopted. The aligned future visual representation is fused with the robot's own ontological perception state to form a complete decision input. Using future timestamps as alignment benchmarks, the action generation network outputs continuous and time-ordered action trajectories. Based on the explicit timestamps corresponding to the generated actions, a time-consistent and directly callable action queue is constructed.

[0017] Optionally, an asynchronous decision-making architecture is constructed based on a diffusion strategy, which integrates aligned future visual representations. The predicted field of view length is determined by explicitly modeling the total system latency. The action queue is dynamically maintained, and action sequences aligned with real timestamps are generated, including: In the asynchronous decision-making architecture built on the diffusion strategy, the future visual representation with alignment constraints is accessed; By combining observation delay, inference delay, and controller delay, an explicit model of the total system delay is completed. By combining the total system delay and the control time step, the prediction field length required for asynchronous decision-making is calculated and determined. Based on the calculated predicted field of view length, an action queue aligned with the real timestamp is dynamically maintained, and an action sequence matching the real timeline is output.

[0018] To achieve the above objectives, a second aspect of the present invention provides an asynchronous robot operation device based on optical flow prediction, comprising: The observation synthesis module is used to extract features and perform spatiotemporal modeling on historical visual frames. It outputs an object flow that represents the motion trend of objects through a lightweight flow prediction module, and completes the rendering and synthesis of future observations based on the object flow. The feature alignment module is used to employ a contrastive learning approach with a temporal mask to align the synthesized future observation features with the real future observation features, thereby forming a future visual representation consistent with real perception. The queue construction module is used to input the aligned and constrained future visual representation and the ontology perception state into the action generation network based on the diffusion strategy, generate action sequences that are explicitly aligned with future timestamps, and build a temporally unified action queue. The asynchronous decision module is used to construct an asynchronous decision architecture based on a diffusion strategy, access the aligned future visual representation, determine the predicted field of view length by explicitly modeling the total system delay, dynamically maintain the action queue and generate action sequences aligned with the real timestamps, and complete the time-series asynchronous operation in robot delay scenarios.

[0019] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0020] To achieve the above objectives, a third aspect of this application provides an electronic device, including a processor and a memory; wherein the processor runs a program corresponding to the executable program code stored in the memory, for implementing an asynchronous robot operation method based on optical flow prediction as described in the first aspect embodiment.

[0021] To achieve the above objectives, a fourth aspect of this application provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements an asynchronous robot operation method based on optical flow prediction as described in the first aspect embodiment.

[0022] The embodiments of the present invention have the following beneficial effects: 1. By effectively supplementing future observations through F2F-AP, a complete future spatiotemporal context is provided for robot strategies, significantly improving the rationality of decision-making and the accuracy of execution in dynamic interactive tasks, and achieving a higher task completion success rate compared to existing asynchronous reasoning methods.

[0023] 2. Introducing future observation can effectively enhance the robot's ability to predict environmental changes, significantly improve the system's response speed, simplify redundant motion trajectories, optimize motion execution paths, greatly shorten task completion time, and improve overall work efficiency.

[0024] 3. By relying on future observations to achieve precise alignment between actions and real timestamps, the problem of timing deviation is effectively alleviated, action loss and trajectory correction are avoided, and the robot's movement is made more coherent and smooth, with stronger robustness and practicality in dynamic target manipulation scenarios. Attached Figure Description

[0025] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein: Figure 1 A flowchart illustrating an asynchronous robot operation method based on optical flow prediction, provided as an embodiment of the present invention; Figure 2This is a schematic diagram of the experimental hardware platform and implementation scenario provided in the embodiments of the present invention; Figure 3 This is a schematic diagram of the overall algorithm flow and asynchronous inference provided for embodiments of the present invention; Figure 4 A schematic diagram of the dynamic task thermodynamic flow field and experimental results provided in an embodiment of the present invention; Figure 5 This is a schematic diagram of comprehensive verification in a multi-task extension scenario provided by an embodiment of the present invention; Figure 6 This is a structural diagram of an asynchronous robot operation device based on optical flow prediction, provided in an embodiment of the present invention. Detailed Implementation

[0026] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0027] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0028] The following description, with reference to the accompanying drawings, describes an asynchronous robot operation method and apparatus based on optical flow prediction according to an embodiment of the present invention.

[0029] Example 1 This invention provides a method for asynchronous robot operation based on optical flow prediction. Figure 1 This is a flowchart illustrating an asynchronous robot operation method based on optical flow prediction, provided in an embodiment of the present invention. Figure 1 As shown, the method includes the following steps: Step S1: Feature extraction and spatiotemporal modeling are performed on historical visual frames. The object flow representing the motion trend of objects is output through the lightweight flow prediction module. The rendering and synthesis of future observations are completed based on the object flow.

[0030] In the embodiments of this application, S1 is the foundation of the entire asynchronous operation method of the robot. Its core purpose is to obtain an object flow that can characterize the motion trend of the object, and to synthesize future observations based on the object flow, so as to provide a complete future spatiotemporal context for subsequent feature alignment and action generation.

[0031] Specifically, the lightweight flow prediction module used in this application embodiment is Flow2Future (hereinafter referred to as the F2F module). As one of the core innovations of this application embodiment, this module aims to build a lightweight spatiotemporal vision model to predict the motion flow of interactive objects, thereby realizing the synthesis of future observations. Compared with the computationally expensive video generation model in the prior art, the F2F module in this application embodiment is more lightweight, can better adapt to the real-time deployment requirements of physical robot systems, effectively reduce hardware resource consumption, and ensure the real-time response of the system.

[0032] In this embodiment, the specific process of feature extraction and spatiotemporal modeling of historical visual frames, and outputting object flow through an F2F module, is as follows: First, a dedicated neural network is built. This neural network is used to extract image feature maps from multiple consecutive historical visual frames, and simultaneously outputs a heatmap for locating object regions. The heatmap has a resolution of 56. 56. This method can accurately locate the position of objects in an image, providing a reliable spatial basis for locating the starting point of the subsequent object flow. Secondly, this embodiment uses a weighted clustering algorithm to perform soft sampling. Based on the response values ​​of each point in the heatmap, the spatial coordinates of the corresponding positions are weighted and aggregated to obtain a rough location result of the starting point of the object flow. This process can effectively filter out the core area of ​​the object and avoid irrelevant areas from interfering with the object flow prediction. Next, this embodiment calculates the local correlation characteristics between the image features of different time frames, uses the stream decoder in the F2F module to output the fine-grained correction value of the flow starting point and the corresponding motion displacement vector. The rough starting point is combined with the fine-grained correction value to obtain the accurate flow starting point. Then, combined with the motion displacement vector, the prediction output of the object flow is finally completed.

[0033] The prediction process of the above-mentioned object flow can be accurately described by the following formula, where the specific meanings of each parameter are as follows: The formula is:

[0034] in, This represents the flow of objects from the current time t to the future time t+H, and this flow of objects can completely represent the motion trend of objects during this time. This represents the soft sampling operation corresponding to the weighted clustering algorithm, used to aggregate the spatial coordinates of the heatmap; Represents image feature maps extracted from historical visual frames. The 56 obtained after processing 56-resolution heatmap Image feature maps of historical visual frames; The "coarse" part represents the coarse location result of the starting point of the object flow obtained through soft sampling operation; That is, the "fine" part. This represents the fine-grained correction module of the stream decoder. For feature maps of different time frames, this part is used to output fine-grained correction values ​​for the start point of the stream, achieving accurate correction of the coarse start point; That is, the "motion" part. The displacement vector output module of the stream decoder is used to output the motion displacement vector of the object stream, which represents the direction and magnitude of the object's motion.

[0035] In this embodiment, the rendering and synthesis of future observations is completed based on the predicted object flow. The specific process is as follows: This embodiment employs a "Paint" operation, assigning corresponding representation information to the predicted object flow vectors according to preset rules. Specifically, each flow vector is assigned a specific color, and the color difference visually reflects the displacement magnitude and direction of movement of the flow vector from the origin. Subsequently, the flow vectors with color representations are rendered and mapped onto the current observation image. Relying on the object motion information carried by the flow vectors, the synthesis and generation of future observation images are completed. This synthesized future observation can completely preserve the motion trend information of the objects, providing a reliable predictive observation basis for feature alignment in the subsequent S2 step.

[0036] Furthermore, in this embodiment, to ensure the training effect of the F2F module and improve the accuracy of object flow prediction, targeted data acquisition and processing were also performed. This embodiment utilizes a UMI device as the main hardware to achieve rapid data acquisition for a single-task strategy. The acquired fisheye images serve as the basic training data for the flow predictor in the F2F module. Considering the unique characteristics of fisheye images, this embodiment employs an image segmentation method to process them. Compared to existing methods such as point tracking or optical flow estimation, this image segmentation method is more reliable and can effectively extract the core features of interactive objects.

[0037] Specifically, the system extracts the most representative key points of interactive objects through segmentation and clustering processes. These key points are used as the starting points of the real flow, and the pixel distance between these key points in different frames is calculated as the displacement of the real flow, thereby constructing the training labels for the flow predictor. In particular, to address the common problem in grasping and placing tasks where objects are occluded by the gripper, causing flow prediction failure, this application proposes a targeted training data synthesis scheme: first, real flow labels are obtained from video sequences without gripper interference, and then gripper masks are superimposed on these video sequences to synthesize training samples with gripper occlusion scenes. The flow predictor trained in this way can perform effective object flow prediction even in the presence of occlusion, further improving the robustness and practicality of the method in this application.

[0038] Step S2: A contrastive learning method with temporal masking is used to align the synthesized future observation features with the real future observation features, forming a future visual representation consistent with real perception.

[0039] In this embodiment, S2 is a key step connecting S1 and S3. Its core purpose is to solve the representational difference between the synthesized future observation and the real future observation in S1. By contrastive learning, the features of the two are aligned to ensure that the future visual representation on which subsequent action generation depends is consistent with the real scene, thus avoiding the confusion of the policy network due to representational differences and affecting the accuracy of action prediction.

[0040] Specifically, although the predicted future observations generated through streaming in S1 retain comprehensive streaming information, simply concatenating them with the current observations still fails to produce a future-aware representation comparable to the real future observations. This representational difference prevents the policy network from accurately capturing the true state of the future scene, thereby weakening the core value of synthesized future observations for asynchronous reasoning. To alleviate this problem, this application's embodiments introduce a contrastive learning framework during the policy training phase, aiming to minimize the representational difference between the streaming future observations (i.e., the synthesized future observations in S1) and the real future observations, achieving feature alignment between the two.

[0041] In this embodiment, a contrastive learning approach with temporal masks is used to perform alignment constraints on the synthesized future observation features and the real future observation features. The specific process is as follows: First, in this embodiment, the synthesized future observations and the real future observations in S1 are input into a shared feature encoder. Through this shared feature encoder, the two observations are mapped to a unified latent feature space, ensuring that the features of the two can be compared in the same dimension, providing a unified metric basis for subsequent feature similarity calculation and alignment constraints. Second, this embodiment constructs a temporal mask filtering mechanism. Considering the high visual correlation between adjacent frames, if all temporally dissimilar frames are indiscriminately filtered... Treating identical features as negative samples can lead to deviations in the optimization direction of contrastive learning and affect the feature alignment effect. Therefore, this temporal mask mechanism is used to filter out frame features from temporally adjacent steps. Only when the temporal distance between two frames exceeds a set threshold is it considered a valid negative sample, thereby eliminating invalid interference samples and improving the accuracy and stability of contrastive learning. Finally, this embodiment takes positive sample feature alignment and true negative sample feature separation as the optimization direction, and applies bidirectional constraints to the predicted features of the synthesized future observations and the ground truth features of the real future observations. At the same time, feature space alignment is completed through symmetric optimization to ensure that the distribution of the two types of features in the latent space tends to be consistent.

[0042] In this embodiment, the contrastive learning process with time masking described above achieves constrained optimization through the corresponding loss function. The specific meanings of the relevant formulas and parameters are as follows: First, the formula for calculating the time mask matrix:

[0043] in, The elements in the temporal mask matrix are used to determine whether the i-th frame and the k-th frame can be used as valid samples for comparative learning. This indicates an indicator function that executes when the condition within the parentheses is true. The value is 1 if it is 1, otherwise it is 0. τ represents the time distance between the i-th frame and the k-th frame; τ represents a preset time distance threshold. In this embodiment, the value of τ can be flexibly adjusted according to actual task requirements, and is usually set to 3-5 time steps to ensure effective filtering of interference from adjacent frames; when When = 0, it means that the i-th frame and the k-th frame are the same frame and belong to a positive sample pair. =1; when When the time interval between the i-th frame and the k-th frame exceeds the threshold, it indicates that the time distance between them is a valid negative sample pair. =1; when 0 < When <τ, it means that the i-th frame and the k-th frame are neighboring frames and belong to invalid negative samples. =0, does not participate in comparative learning.

[0044] Secondly, the formula for calculating the directed contrast loss function is as follows:

[0045] in, The value represents the directional contrast loss, which measures the degree of alignment between the predicted features and the ground truth features. The smaller the loss value, the better the feature alignment between the two. N represents the total number of samples participating in the contrastive learning. This represents the predicted feature vector of the i-th synthetic future observation. Let represent the true eigenvector of the i-th true future observation; Represents the predicted feature vector With true eigenvectors The inner product between them is used to measure the similarity between the two; τ represents the temperature parameter, which is used to adjust the weight of feature similarity to avoid gradient vanishing or gradient explosion. In the embodiments of this application, the value of τ can be the same as or different from the time distance threshold τ, and can be flexibly adjusted according to the actual training effect. M_(i,k) represents the weighted sum of similarities between all valid samples (positive samples and valid negative samples) and the i-th predicted feature vector. M_(i,k) is used to filter valid samples, and only valid samples participate in this summation calculation.

[0046] Finally, here is the formula for calculating the symmetric objective function:

[0047] Where L represents the final contrastive learning loss value, which is used to optimize the entire contrastive learning process; This represents the predicted features based on synthetic future observations. True-value features for input, real future observations Targeted contrast loss; Represents the true characteristics of real future observations. Input and synthesize predictive features for future observations The target is a directional contrast loss; by averaging the two, a symmetric objective function is formed, which can achieve bidirectional alignment between predicted features and ground truth features, further improving the feature alignment effect and ensuring that the features of the synthesized future observations can be consistent with the features of the real future observations.

[0048] In this embodiment, the specific process of forming a future visual representation consistent with real perception is as follows: First, this embodiment uses the aforementioned contrastive learning objective to narrow the distribution distance between the predicted features after flow enhancement (i.e., the features of the synthesized future observation) and the features of the real future observation, gradually reducing the representational difference between the two; Second, during feature matching, the temporal mask mechanism effectively eliminates feature interference caused by temporally adjacent frames, avoids invalid negative samples affecting the effectiveness of feature alignment, and improves the accuracy and reliability of feature alignment; Finally, this embodiment calculates the normalized cosine similarity in the latent feature space to measure the matching degree between the predicted features and the ground truth features, continuously optimizes the representational reliability of the synthesized future observation based on the similarity results, gradually eliminates the representational difference between the synthesized observation and the real observation, and finally forms a future visual representation consistent with real perception. This future visual representation can accurately reflect the state of the future scene and provide a reliable input basis for the action generation in the subsequent S3 step.

[0049] Step S3: Input the aligned and constrained future visual representation and the ontology perception state into the action generation network constructed based on the diffusion strategy to generate action sequences that are explicitly aligned with future timestamps and construct a temporally unified action queue.

[0050] In this embodiment, S3 is the core step of action generation. The core purpose is to generate a time-accurate action sequence through the action generation network based on the aligned future visual representation obtained in S2 and combined with the robot's body perception state, and to build an action queue that can be directly invoked, so as to provide an executable action basis for subsequent asynchronous decision-making, ensure that the action is accurately aligned with the future timestamp, and avoid action lag or timing deviation.

[0051] Specifically, the action generation network in this embodiment is built on a diffusion policy framework. Diffusion policy has strong temporal modeling capabilities and action generation stability, effectively adapting to the action prediction needs in dynamic scenes. Compared to other existing action generation frameworks, it can better integrate future visual context to generate coherent and accurate action sequences. In this embodiment, the action generation network uses a pre-trained Visual Transformer (ViT) as a feature encoder to further extract and encode features from the future visual representation obtained in S2, ensuring the effectiveness and representativeness of the features. Simultaneously, a U-Net is used as a decoder to map the encoded features into specific robot action commands, thereby generating the action sequence.

[0052] In this embodiment, the process of inputting the aligned future visual representation and the robot's own proprioceptive state into the action generation network to generate an action sequence explicitly aligned with the future timestamp and construct a temporally unified action queue is as follows: First, based on a diffusion strategy, this embodiment builds an action generation network structure that adapts to future visual features and robot proprioceptive features. The visual Transformer (ViT) encoder is responsible for processing the future visual representation and extracting spatiotemporal information from the visual features. The U-Net decoder is responsible for decoding the features after fusing the visual features and proprioceptive features into specific robot action parameters, ensuring that the action generation network can accurately adapt to the input features and output action commands that meet the actual control requirements. Second, this embodiment fuses the aligned future visual representation and the robot's own proprioceptive state in S2. The proprioceptive state includes the robot's own state information such as joint angles, speed, and torque. After fusing the two, a complete decision input for the action generation network is formed, ensuring that action generation can not only consider the future environmental state but also combine the robot's own state, improving the feasibility and safety of the action.

[0053] Furthermore, this embodiment uses future timestamps as alignment benchmarks and outputs continuous and time-ordered action trajectories through an action generation network. Since the synthesized future observations in S1 correspond to explicit future timestamps (time t+H), the action generation network generates action sequences explicitly aligned with these timestamps, ensuring that each action corresponds to a specific future time and avoiding timing deviations. Finally, this embodiment constructs a time-unified and directly callable action queue based on the explicit timestamps corresponding to the generated actions. This action queue is arranged in the order of timestamps, eliminating the need for post-processing corrections or discarding head action segments. It can directly provide an executable action basis for subsequent asynchronous decisions, laying the foundation for decoupling reasoning and execution.

[0054] In this embodiment, the action sequence generated by the action generation network is explicitly aligned with the future timestamp. Its core advantage is that, compared with the existing asynchronous inference methods that require discarding head action segments, the action sequence in this embodiment can directly adapt to the future timestamp without discarding any action segments. This effectively avoids the problems of action lag and discontinuous action block connections. At the same time, by combining future visual representation and ontological perception state, the generated action sequence can better adapt to changes in the dynamic environment, improving the accuracy and coherence of robot control.

[0055] Step S4: Construct an asynchronous decision-making architecture based on the diffusion strategy, access the aligned future visual representation, determine the predicted field of view length through the total delay of the explicit modeling system, dynamically maintain the action queue and generate an action sequence aligned with the real timestamp, and complete the time-series asynchronous operation in the robot delay scenario.

[0056] In the embodiments of this application, S4 is the final execution step of the entire asynchronous operation method. The core purpose is to build an asynchronous decision architecture, realize the complete decoupling of policy reasoning and action execution, determine the predicted field of view length through explicit modeling system delay, dynamically maintain the action queue, ensure that the generated action sequence is accurately aligned with the real timestamp, and finally complete the robot's time-series asynchronous operation in the delayed scenario, solving the problems of action lag and inaccurate control in existing asynchronous reasoning.

[0057] Specifically, the embodiments of this application construct an asynchronous decision-making architecture based on a diffusion strategy. This architecture is consistent with the action generation network in S3 and can achieve efficient access to future visual representations and dynamic maintenance of action queues. Its core advantage lies in its ability to parallelize policy reasoning and action execution, completely solving the latency bottleneck in synchronous execution mode. At the same time, by explicitly modeling system latency, it ensures accurate alignment between action sequences and real timestamps, improving the robustness of control in dynamic scenes.

[0058] In this embodiment, the asynchronous decision-making architecture based on a diffusion strategy is constructed, and the aligned future visual representation is accessed. The predicted field of view length is determined by explicitly modeling the total system delay, and the action queue is dynamically maintained to generate an action sequence aligned with the real timestamp. The specific process is as follows: First, in the asynchronous decision-making architecture based on the diffusion strategy, the aligned future visual representation in S2 is accessed. This future visual representation provides a complete future spatiotemporal context for asynchronous decision-making, ensuring that the decision-making process can reason based on the future environmental state and avoid decision bias caused by missing information. Second, this embodiment performs explicit modeling of the total system delay. The total system delay is the core reason for action timing deviation, which includes three parts: observation delay, inference delay, and controller delay. Among them, observation delay refers to the time consumption from acquiring historical visual frames to obtaining image features; inference delay refers to the time consumption of the F2F module predicting object flow, contrastive learning feature alignment, and the action generation network generating action sequences; and controller delay refers to the time consumption of the robot's underlying controller receiving action instructions and executing actions. This embodiment completes the total system delay by accurately calculating the delays of the above three parts. Explicit modeling, followed by combining the robot's single-step control time step. The required prediction horizon length H for asynchronous decision-making is determined by calculating the corresponding formula.

[0059] The calculation formula for the predicted visual field length H and the specific meanings of each parameter are as follows:

[0060] Where H represents the prediction field of view length, i.e., the number of future time steps to be predicted, in units of time steps. This indicates a rounding up operation to ensure that the predicted field of view length can fully cover the total system latency, avoiding action lag caused by insufficient predicted field of view. This represents the total system delay, which is the sum of observation delay, inference delay, and controller delay, expressed in seconds. The time step of the robot's single-step control is the time required for the robot to execute one action step, expressed in seconds. The predicted field of view length H calculated by this formula ensures that the action sequence generated by the action generation network is precisely aligned with the real timestamp after the system delay ends, effectively eliminating timing deviations.

[0061] Finally, based on the calculated predicted field of view length H, this embodiment dynamically maintains an action queue aligned with the real timestamp. Specifically, the asynchronous decision architecture obtains changes in the total system latency in real time, dynamically adjusts the predicted field of view length H, and then adjusts the action sequence in the action queue to ensure that each action in the action queue strictly matches the real physical timestamp. At the same time, since the actions in the action queue are explicitly aligned with future timestamps, and the predicted field of view length can cover the total system latency, the asynchronous decision architecture can directly call the actions in the action queue and output an action sequence that matches the real timeline, achieving complete decoupling between policy reasoning and action execution without the need for post-event correction or trajectory discarding. The robot's underlying controller executes actions according to this action sequence, accurately responding to changes in the dynamic environment, effectively solving the problems of action lag and discontinuous control in existing asynchronous reasoning, and ultimately completing the time-series asynchronous operation in robot latency scenarios, improving the robot's control accuracy, efficiency, and robustness in dynamic interactive tasks.

[0062] In this embodiment, through the coordinated operation of S1 to S4, a complete process from future observation synthesis and feature alignment to action generation and asynchronous decision-making is realized. The entire process is logically rigorous and tightly connected. Future observations are supplemented by the F2F module, feature alignment is achieved through contrastive learning with time masking, an action generation network and asynchronous decision-making architecture are constructed through a diffusion strategy, and the predicted field of view length is determined through explicit delay modeling. Ultimately, the robot achieves accurate, coherent, and efficient asynchronous control in delayed scenarios, effectively solving the technical problems of missing observation information, temporal deviation, and action lag in the prior art. It has high practicality and application value.

[0063] In the application of one embodiment of the present invention, the implementation process is as follows: This embodiment uses the F2F-AP (Flow2Future-Asynchronous Policy) model to achieve asynchronous control of multiple platforms such as robotic arms and robot dogs in dynamic scenarios, as detailed below: like Figure 2 As shown, this embodiment employs two heterogeneous robot platforms for experimental verification: a fixed-platform single-arm robotic arm platform and a composite mobile operation platform with a lightweight robotic arm mounted on the back of a robot dog. The fixed-platform single-arm robotic arm is mounted on a fixed base on a standard experimental workbench and is primarily used for precise interactive operations such as grasping and transferring dynamic targets within the desktop area. The robot dog platform, based on a quadrupedal mobile robot with a multi-degree-of-freedom lightweight robotic arm integrated on its back, combines mobility and manipulation capabilities, enabling collaborative operations in more complex dynamic environments. Both platforms are equipped with synchronous visual acquisition devices, real-time motion control units, and data processing units. Model deployment and task execution are completed under a unified experimental environment and identical task parameter settings, ensuring consistent experimental conditions and providing a reliable foundation for subsequent performance comparisons and effect verification.

[0064] The overall reasoning and execution process of the model in this application embodiment can be uniformly represented by the following formula:

[0065] in, This represents a continuous historical visual observation frame sequence from time tn to time t. This refers to the Flow2Future lightweight optical flow prediction module. This represents the feature encoder trained using contrastive learning with a temporal mask. This represents an action generation decoder built based on a diffusion strategy. represents the future H-step observation information obtained by reasoning in the feature space, and s represents the robot's own body perception state information such as joint angles, posture, and velocity.

[0066] This model takes historical visual observations and the robot's own state as input, and outputs the action instructions that the robotic arm or robot dog should execute in the future time sequence through optical flow prediction, feature encoding, and motion decoding. The execution method of the action instructions differs for different hardware platforms: for a fixed-stage single-arm robotic arm, the system calculates the inverse kinematics based on the generated target motion, solves for the rotation angles of each joint, and directly drives the joint motors to complete the motion; for a robot dog platform with an added robotic arm, the system uses a unified whole-body controller to simultaneously control the drive motors of the four legs and the actuator motors of the back robotic arm, achieving coordinated movement of the hands and feet, enabling the end effector of the back robotic arm to accurately reach the designated target position.

[0067] like Figure 3 As shown in the figure, the left side is a schematic diagram of asynchronous inference timing. The blue dots represent the actual trajectory of the object (objtraj.), and the green and yellow dots represent the predicted trajectories (predicted traj.) at different stages. It intuitively shows the action alignment process from inference start to future time step, reflecting the core idea of ​​this application to achieve temporal decoupling by predicting future states.

[0068] The middle section is the core algorithm pipeline: First, the historical visual frame sequence ( … The robot's states are input into the ViT encoder, and the features are fed into the LocalCorr. local correlation calculation module and the Heatmap generation module, respectively. The FlowDec. stream decoder outputs the predicted object flow, completing the Paint rendering and synthesis of future observations. Subsequently, the real future observations ( ) and Synthetic Future Observation ( The predicted features are first processed by a shared Enc. encoder, then mapped to the same latent feature space by Projector I and Projector II, and then enter the contrastive learning stage. Through a similarity matrix constraint with a temporal mask, the distribution of predicted features is aligned with that of true features. Finally, the observed features carrying future context and the ontology state are input into the policy network, which outputs a temporally aligned action sequence via the Enc. encoder and Dec. decoder. … This enables asynchronous decoupling of reasoning and execution.

[0069] The entire process, through the coordinated efforts of ViT feature extraction, lightweight flow prediction, contrastive learning feature alignment, and diffusion strategy action generation, provides the robot with accurate future temporal action instructions, effectively solving the problems of observation gaps and temporal deviations in asynchronous reasoning.

[0070] like Figure 4As shown, the left side of the figure is a 56×56 resolution heatmap output by the model. In the figure, the blue area represents the low response area of ​​the feature, which corresponds to the background or non-critical area of ​​the image. The red area represents the high response area of ​​the feature, which is used to accurately locate the position and core area of ​​the interactive object. The predicted object flow is visualized using pink lines that gradually change from dark to light. The direction of the line represents the direction of the object's movement, and the length of the line corresponds to the magnitude of the displacement, which intuitively reflects the movement trend from the current frame to the future frame. The central image shows the execution of three tasks: hammer handover, car retrieval, and banana retrieval. The upper right corner displays timestamps corresponding to 1.2s / 1.9s / 2.7s, 0.7s / 1.5s / 2.1s, and 0.5s / 1.9s / 2.4s respectively, visually demonstrating the dynamic characteristics of the tasks and the robot's responsiveness. Quantitative comparative data on the right shows that, under the same experimental conditions, F2F-AP achieved a success rate of 15 / 20 and an execution time of 3.4s in the hammer handover task, 8 / 20 and a execution time of 2.3s in the car retrieval task, and 13 / 20 and a execution time of 2.5s in the banana retrieval task. All of these significantly outperform the three baseline methods: Synchronous UMI (Syn.UMI), Asynchronous UMI (Asyn.UMI), and VLASH.

[0071] like Figure 5 As shown, the left side of the figure displays heatmaps and predicted optical flow for different tasks. A 56×56 resolution heatmap is used to locate the target object, and the pink gradient flow field clearly presents the movement trends of different objects, verifying the generalization ability of the Flow2Future module in multi-target and multi-scenario scenarios. The middle section shows the execution screens of the carrot transfer and car retrieval tasks, with timestamps of 0.1s / 1.0s / 1.6s and 0.4s / 1.1s / 1.9s respectively, demonstrating the collaborative efficiency of the robot dog and robotic arm platform. The quantitative comparative data on the right shows that F2F-AP achieved a success rate of 6 / 10 and an execution time of 1.6s in the carrot transfer task, and a success rate of 7 / 10 and an execution time of 1.9s in the car retrieval task, maintaining the highest success rate and shortest execution time.

[0072] comprehensive Figure 4 and Figure 5 The data shows that F2F-AP effectively solves the problems of action lag and timing deviation in asynchronous inference by using future observations to fill in the gaps and align the timing. It significantly outperforms existing baseline methods in multi-task and multi-platform dynamic interaction scenarios, greatly improving task success rate and execution efficiency.

[0073] Example 2 This invention provides an asynchronous robot operation device based on optical flow prediction. Figure 6 This is a schematic flowchart of an asynchronous robot operation device based on optical flow prediction, provided as an embodiment of the present invention. Figure 6As shown, the device includes: The observation synthesis module 100 is used to extract features and perform spatiotemporal modeling on historical visual frames. It outputs an object flow that represents the motion trend of objects through a lightweight flow prediction module, and completes the rendering and synthesis of future observations based on the object flow. The feature alignment module 200 is used to perform alignment constraints on the synthesized future observation features and the real future observation features using a contrastive learning method with a time mask, so as to form a future visual representation consistent with real perception. The queue construction module 300 is used to input the aligned and constrained future visual representation and the ontology perception state into the action generation network based on the diffusion strategy, generate action sequences that are explicitly aligned with future timestamps, and construct a temporally unified action queue. The asynchronous decision module 400 is used to construct an asynchronous decision architecture based on a diffusion strategy, access the aligned future visual representation, determine the predicted field of view length through explicit modeling of the total system delay, dynamically maintain the action queue and generate an action sequence aligned with the real timestamp, and complete the time-series asynchronous operation in robot delay scenarios.

[0074] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0075] Example 3 To implement the methods of the above embodiments, the present invention also provides an electronic device, which includes a memory and a processor; wherein the processor reads executable program code stored in the memory to run a program corresponding to the executable program code, so as to implement the various steps of the methods described above.

[0076] Example 4 To implement the above embodiments, this application also proposes a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the method described in the foregoing embodiments.

[0077] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

[0078] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0079] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this invention, "a plurality of" means at least two, such as two, three, etc., unless otherwise explicitly specified.

Claims

1. A method for asynchronous robot operation based on optical flow prediction, characterized in that, include: Feature extraction and spatiotemporal modeling are performed on historical visual frames. A lightweight flow prediction module outputs an object flow that represents the motion trend of objects. Based on the object flow, rendering and synthesis of future observations are completed. A contrastive learning approach with temporal masking is used to align the synthesized future observation features with the real future observation features, forming a future visual representation consistent with real perception. The aligned future visual representation and the ontology perception state are jointly input into the action generation network based on the diffusion strategy to generate action sequences that are explicitly aligned with future timestamps, thus constructing a temporally unified action queue. An asynchronous decision-making architecture is constructed based on a diffusion strategy. The aligned future visual representation is accessed, and the predicted field of view length is determined by the total delay of the explicit modeling system. The action queue is dynamically maintained and an action sequence aligned with the real timestamp is generated to complete the time-series asynchronous operation in robot delay scenarios.

2. The method according to claim 1, characterized in that, Feature extraction and spatiotemporal modeling are performed on historical visual frames. A lightweight flow prediction module outputs an object flow representing the motion trend of objects, including: Extract corresponding image feature information from consecutive historical visual frames and generate a heat map for locating object regions; A weighted clustering algorithm is used to perform soft sampling. Based on the response values ​​of each point in the heatmap, the corresponding spatial coordinates are aggregated in a weighted manner to obtain a rough location result of the starting point of the object flow. The local correlation characteristics between features of different time frames are calculated, and the fine-grained correction values ​​of the starting point of the stream and the corresponding motion displacement vector are output by the stream decoder to complete the prediction output of the object stream.

3. The method according to claim 2, characterized in that, The process of rendering and compositing future observations based on object flow includes: The predicted object flow vectors are assigned corresponding representation information according to preset rules and then rendered and mapped onto the currently observed image. The rendered flow vectors visually represent the magnitude and direction of displacement from the origin, and the flow information is used to synthesize and generate future observation images.

4. The method according to claim 3, characterized in that, A contrastive learning approach with temporal masking is employed to impose alignment constraints between synthesized future observation features and real future observation features, including: The synthetic future observations and the real future observations are respectively input into a shared feature encoder and mapped to a unified latent feature space; A temporal mask screening mechanism is constructed to filter frame features with close time intervals, and only samples whose temporal distance meets the set conditions are retained to participate in comparative learning; With positive sample feature alignment and true negative sample feature separation as the optimization directions, bidirectional constraints are imposed on predicted features and true features, and feature space alignment is completed through symmetric optimization.

5. The method according to claim 4, characterized in that, The process of forming a future visual representation consistent with real perception includes: A contrastive learning objective is used to narrow the distribution distance between the predicted features after flow enhancement and the actual future observed features; In the feature matching process, feature interference from frames with adjacent times is eliminated to improve the effectiveness of feature alignment; By measuring the degree of feature matching through normalized similarity, the reliability of the representation of predicted observations is continuously optimized, the representational differences between synthetic and real observations are eliminated, and a future visual representation consistent with real perception is formed.

6. The method according to claim 5, characterized in that, The aligned future visual representation and ontology-aware state are jointly input into an action generation network constructed based on a diffusion strategy to generate action sequences explicitly aligned with future timestamps, thus constructing a temporally unified action queue, including: An action generation network structure is built based on a diffusion strategy, and an encoding and decoding structure adapted to visual features is adopted. The aligned future visual representation is fused with the robot's own ontological perception state to form a complete decision input. Using future timestamps as alignment benchmarks, the action generation network outputs continuous and time-ordered action trajectories. Based on the explicit timestamps corresponding to the generated actions, a time-consistent and directly callable action queue is constructed.

7. The method according to claim 6, characterized in that, An asynchronous decision-making architecture is constructed based on a diffusion strategy. Aligned future visual representations are accessed, and the predicted field of view length is determined by explicitly modeling the total system latency. The action queue is dynamically maintained, and action sequences aligned with real timestamps are generated, including: In the asynchronous decision-making architecture built on the diffusion strategy, the future visual representation with alignment constraints is accessed; By combining observation delay, inference delay, and controller delay, an explicit model of the total system delay is completed. By combining the total system delay and the control time step, the prediction field length required for asynchronous decision-making is calculated and determined. Based on the calculated predicted field of view length, an action queue aligned with the real timestamp is dynamically maintained, and an action sequence matching the real timeline is output.

8. An asynchronous robot manipulation device based on optical flow prediction, characterized in that, include: The observation synthesis module is used to extract features and perform spatiotemporal modeling on historical visual frames. It outputs an object flow that represents the motion trend of objects through a lightweight flow prediction module, and completes the rendering and synthesis of future observations based on the object flow. The feature alignment module is used to employ a contrastive learning approach with a temporal mask to align the synthesized future observation features with the real future observation features, thereby forming a future visual representation consistent with real perception. The queue construction module is used to input the aligned and constrained future visual representation and the ontology perception state into the action generation network based on the diffusion strategy, generate action sequences that are explicitly aligned with future timestamps, and build a temporally unified action queue. The asynchronous decision module is used to construct an asynchronous decision architecture based on a diffusion strategy, access the aligned future visual representation, determine the predicted field of view length by explicitly modeling the total system delay, dynamically maintain the action queue and generate action sequences aligned with the real timestamps, and complete the time-series asynchronous operation in robot delay scenarios.

9. An electronic device, characterized in that, Including processor and memory; The processor reads executable program code stored in the memory to run a program corresponding to the executable program code, so as to implement the method as described in any one of claims 1-7.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1-7.

Images