Mechanical arm coffee latte art trajectory self-adaptive inference method based on diffusion transformer model

Abstract

The application relates to the technical field of robot intelligent control, and discloses a mechanical arm coffee latte trajectory self-adaptive reasoning method based on a diffusion transformer model, which acquires multi-modal teaching data through a handheld acquisition device, converts the multi-modal teaching data into Lie algebra relative trajectories to train a conditional diffusion transformer. In an online execution stage, a hybrid dynamic model composed of a nominal model and a real-time updated residual neural network is constructed and used for capturing fluid nonlinear characteristics. A dynamic energy function is constructed by using the hybrid model, an energy gradient guide is introduced in a diffusion reverse sampling process, and a smooth trajectory conforming to current physical constraints is generated. Finally, combined with dynamic liquid surface compensation, an optimal moment command is solved through model predictive control to drive the mechanical arm. The application effectively solves the fluid control problem in an unstructured environment and realizes high-precision mechanical arm latte with physical self-adaptive capability.

Description

Technical Field

[0001] This invention relates to the field of robot intelligent control technology, specifically to an adaptive reasoning method for coffee latte art trajectory of a robotic arm based on a diffusion converter model. Background Technology

[0002] With the development of service robot technology, automated coffee making has expanded from simple grinding and extraction to more complex latte art techniques. Latte art not only requires robotic arms with high-precision motion control capabilities but also necessitates handling complex fluid dynamics interactions. Existing robotic latte art systems mostly employ the traditional teach-and-playback control mode, where the operator records a standard trajectory through drag-and-teach or teleoperation, and the robotic arm repeats this trajectory in subsequent operations.

[0003] However, this control strategy based on the rigid body motion assumption has inherent flaws when dealing with fluid media. The latte art process involves nonlinear coupling of multiphase fluids, and the flow characteristics of the liquid are extremely sensitive to pouring speed, acceleration changes, and liquid level. In practical applications, even small fluctuations in environmental parameters, such as viscosity differences due to milk frothing, initial height deviations in the coffee liquid, or time-varying characteristics of robotic arm joint friction, can cause the preset taught trajectory to fail, leading to pattern deformation, flow interruption, or even liquid overflow. Furthermore, traditional motion planning algorithms struggle to maintain trajectory smoothness while simultaneously considering fluid dynamic constraints, and end-to-end deep learning methods often lack clear physical interpretability. The generated trajectories are prone to unpredictable oscillations in unseen state spaces and are difficult to adapt to containers of different sizes or shapes, resulting in weak system generalization capabilities. Current solutions cannot achieve real-time physical adaptation to dynamic fluid environments while maintaining the geometric aesthetics of the pattern. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention provides an adaptive reasoning method for the trajectory of a robotic arm in latte art based on a diffusion converter model. This method solves the technical problems of existing robotic latte art systems that rely on rigid teaching trajectories and cannot adapt to unstructured environmental changes such as fluid viscosity and liquid level, as well as the lack of physical and dynamic constraints in pure data-driven generation strategies, which leads to motion trajectory oscillations, pattern deformation, or fluid overflow.

[0005] The method of this invention first acquires a multimodal teaching sequence containing image data and inertial data using a handheld demonstration acquisition device. In the data preprocessing stage, a monocular dual-view system consisting of a wide-angle fisheye camera and an auxiliary reflecting mirror is used to construct virtual stereoscopic visual constraints based on the principle of planar reflection. By establishing a visual reprojection residual model between the direct observation area and the specular reflection area, and combining it with inertial pre-integration residual terms, a nonlinear least-squares optimization problem is constructed and solved to calculate the six-degree-of-freedom absolute pose sequence of the teaching actions.

[0006] To enhance the spatial generalization ability of the trajectory, this invention converts the calculated absolute pose sequence into a relative trajectory vector in Lie algebra space. Specifically, it calculates the relative transformation matrix of each time step in the trajectory relative to the initial pose and converts it into a six-dimensional motion vector through a logarithmic mapping. Simultaneously, it extracts the visual feature vectors of the synchronized images and the robotic arm's body state vector, fuses them through a projection layer to generate an observation condition vector, and constructs a multimodal demonstration dataset.

[0007] This invention utilizes the aforementioned dataset to train a conditional diffusion model offline. The model employs a Transformer-based denoising network architecture, introducing an adaptive normalization layer in each encoder block to modulate features using the observation conditional vector and the diffusion time embedding vector. The training process minimizes the error between predicted noise and true Gaussian noise, enabling the network to learn to recover a sequence of relative trajectory vectors conforming to the teaching distribution from the Gaussian noise distribution.

[0008] During the online execution phase, this invention constructs a hybrid dynamics model comprising a nominal dynamics model and a residual neural network model to adapt to changes in fluid load. The nominal dynamics model describes the rigid body dynamics, while the residual neural network model characterizes the state deviations caused by fluid sloshing and nonlinear friction. The system maintains an experience replay buffer, stores dynamic transformation tuples in real time, and updates the parameters of the residual neural network online using a stochastic gradient descent algorithm by calculating the residual between the nominal model's predicted values ​​and the actual measured values, thereby capturing the nonlinear dynamic characteristics of the system in real time.

[0009] To ensure that the generated trajectory conforms to the current physical constraints, this invention constructs a dynamic energy function, which is a weighted combination of a dynamic consistency energy term and a trajectory smoothness energy term. During the backsampling process of the diffusion model, the system calculates the gradient of this dynamic energy function with respect to the trajectory and uses this gradient as a guiding term to be added to the denoising step to correct the predicted noise. This process allows the generated trajectory to shift towards the manifold region with lower dynamic energy while maintaining the taught geometric features, thereby generating a physically feasible motion trajectory.

[0010] Finally, the model predictive control execution module constructs a dynamic reference coordinate system based on the generated relative trajectory and the real-time updated liquid level, mapping the relative trajectory to the target pose in absolute space. By solving a quadratic programming problem with constraints from a hybrid dynamics model, the optimal control torque command is calculated to drive the robotic arm, achieving precise adaptive control of the fluid latte art process.

[0011] This invention provides an adaptive inference method for the trajectory of latte art using a robotic arm based on a diffusion transformer model. It offers the following advantages:

[0012] 1. This invention solves the depth occlusion problem in teaching data acquisition and improves the spatial generalization ability of actions by employing a combination of a monocular dual-view acquisition device and Lie algebra relative trajectory mapping. Virtual stereo vision is constructed using an auxiliary reflective mirror, enabling a single camera to simultaneously capture the vertical height and horizontal position of the latte art nozzle relative to the liquid surface, eliminating the uncertainty in depth estimation inherent in monocular vision. Simultaneously, the absolute pose is converted into a relative motion vector in Lie algebra space, decoupling the learned latte art strategy from the absolute world coordinate system. This allows the robotic arm to adaptively execute correct relative motion even when the coffee cup's position changes, reducing dependence on environmental position calibration.

[0013] 2. This invention utilizes an energy field-guided diffusion converter model to achieve trajectory generation that combines diversity and physical consistency. Unlike traditional deterministic interpolation methods, the conditional diffusion model can reconstruct diverse trajectory distributions from noise based on the current visual state, conforming to human operating habits. More importantly, an energy field gradient based on dynamic residuals is introduced during the reverse denoising sampling process, forcing the generated trajectory to converge towards lower energy and higher smoothness. This mechanism ensures that the generated latte art not only geometrically conforms to the characteristics of a heart-shaped pattern but also dynamically satisfies the physical constraints of fluid manipulation, avoiding fluid overflow or pattern jitter caused by sudden acceleration changes.

[0014] 3. This invention, by constructing a neuro-physical hybrid dynamics model and combining it with an online residual learning mechanism, endows the system with real-time adaptability to fluid load changes in unstructured environments. The system utilizes a nominal rigid body model to ensure the correctness of fundamental physical laws and employs a real-time updated residual neural network to capture nonlinear characteristics such as fluid sloshing, viscosity changes, and mechanical friction. This hybrid architecture enables the model predictive control module to adjust the motor torque command in real time based on the updated dynamic characteristics. For example, it automatically compensates for the driving force when the milk viscosity increases, thus ensuring that the robotic arm can control the end effector to perform complex latte art operations under different working conditions without the need for retraining offline. Attached Figure Description

[0015] Figure 1 This is a system framework diagram of the present invention; Figure 2 This is a flowchart of the method of the present invention.

[0016] Among them, 110 is the robotic arm body; 111 is the end effector; 120 is the handheld demonstration and acquisition device; 121 is the wide-angle fisheye camera; 122 is the auxiliary reflector; 123 is the inertial measurement unit; 130 is the computational control unit; 201 is the multimodal data preprocessing module; 202 is the conditional diffusion model training module; 203 is the online residual dynamics learning module; 204 is the energy field guided inference module; and 205 is the model predictive control execution module. Detailed Implementation

[0017] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0018] Please see the appendix Figure 1 - Appendix Figure 2 This invention provides an adaptive reasoning system for the trajectory of latte art by a robotic arm based on a diffusion converter model. The system mainly includes a robotic arm body 110, a handheld demonstration and acquisition device 120, and a calculation and control unit 130.

[0019] The robotic arm body 110 is a multi-degree-of-freedom serial robotic arm, with an end effector 111 installed at its end. The end effector 111 is configured to grip a pull cylinder and can perform tilting, translation, and shaking movements. The robotic arm body 110 integrates position sensors and torque sensors to provide feedback on joint angle, angular velocity, and joint torque information.

[0020] The handheld demonstration and acquisition device 120 is a portable teaching tool used to collect data on the latte art movements of a human operator. The handheld demonstration and acquisition device 120 includes a wide-angle fisheye camera 121, an auxiliary reflector 122, and an inertial measurement unit 123. The auxiliary reflector 122 is fixed within one side of the field of view of the wide-angle fisheye camera 121, allowing the imaging plane of the wide-angle fisheye camera 121 to simultaneously capture images from both the direct frontal view and the side view reflected by the auxiliary reflector 122. The inertial measurement unit 123 is rigidly fixed to the wide-angle fisheye camera 121 and is used to acquire high-frequency triaxial acceleration and triaxial angular velocity data.

[0021] The computational control unit 130 is connected to the robotic arm body 110 and the handheld demonstration and acquisition device 120 via communication interfaces. The computational control unit 130 includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to perform logical operations on multiple functional modules. These functional modules include: a multimodal data preprocessing module 201, a conditional diffusion model training module 202, an online residual dynamics learning module 203, an energy field-guided inference module 204, and a model prediction control execution module 205.

[0022] The multimodal data preprocessing module 201 is configured to receive raw sensor data from the handheld demonstration acquisition device 120. This module first constructs a virtual stereoscopic vision based on images from the wide-angle fisheye camera 121 and the reflection characteristics of the auxiliary reflective mirror 122. It then combines this with data from the inertial measurement unit 123 to perform visual inertial state estimation, calculating the six-degree-of-freedom pose sequence of the hand movement. Subsequently, the module converts the absolute pose sequence into relative trajectory vectors in Lie algebra space relative to the initial time, extracts the corresponding visual feature vectors, and constructs a multimodal demonstration dataset.

[0023] The conditional diffusion model training module 202 is configured to build and train the diffusion transformer network offline. This module employs a Transformer-based denoising network architecture, using visual features output from the multimodal data preprocessing module 201, the current robotic arm state, and the gripper opening as conditional inputs. By minimizing the noise prediction error, this module trains the network to learn how to recover a sequence of relative trajectory vectors conforming to a human demonstration distribution from a Gaussian noise distribution.

[0024] The online residual dynamics learning module 203 is configured to build a system dynamics model in real time during system operation. This module defines the hybrid dynamic equations consisting of a nominal dynamics model and a residual neural network model. During the robot arm's task execution, this module collects state transition data in real time and stores it in an experience replay buffer. It then uses the collected data to perform online gradient updates on the parameters of the residual neural network model to capture nonlinear dynamic characteristics arising from fluid sloshing, frictional changes, and system delays.

[0025] The energy field-guided inference module 204 is configured to perform adaptive sampling during the trajectory generation phase. This module constructs a dynamic energy function using a residual neural network model updated in real-time by the online residual dynamics learning module 203. The dynamic energy function characterizes the degree of dynamic violation and unsmoothness of the generated trajectory under the current physical environment. During the inverse denoising process of the diffusion model, this module calculates the gradient of the dynamic energy function relative to the current intermediate trajectory and adds this gradient as a guiding term to the denoising sampling step, causing the generated trajectory to shift in the direction of decreasing dynamic energy function value.

[0026] The model predictive control execution module 205 is configured to perform trajectory tracking control. This module receives the relative trajectory sequence output by the energy field-guided inference module 204 and converts it into a reference state sequence in an absolute coordinate system. The module constructs a quadratic programming optimization problem with constraints from hybrid dynamics equations, solves for the optimal control sequence in the prediction time domain, and sends the first control command to the controller of the robotic arm body 110.

[0027] This invention also provides an adaptive inference method for the latte art trajectory of a robotic arm based on a diffusion transformer model, comprising the following steps: Step S1: Collect a multimodal teaching sequence containing image data and inertial data using a handheld demonstration acquisition device 120.

[0028] Step S2: The six-degree-of-freedom pose of the teaching action is solved by the multimodal data preprocessing module 201, and the absolute pose sequence is converted into relative trajectory vectors in Lie algebra space to construct the training dataset.

[0029] Step S3: Use the training dataset to train the diffusion transformer network in the conditional diffusion model training module 202 offline to learn the conditional probability distribution of the relative trajectory.

[0030] Step S4: Enter the online execution stage. The online residual dynamics learning module 203 updates the parameters of the residual neural network model online based on the real-time feedback data of the robotic arm body 110.

[0031] In step S5, the energy field-guided inference module 204 combines the trained diffusion converter network with the currently updated residual neural network model, and generates a latte art trajectory that conforms to the current physical constraints by calculating the gradient of the dynamic energy function to guide the back sampling process.

[0032] In step S6, the model prediction control execution module 205 solves for the optimal control command based on the generated latte art trajectory, drives the robotic arm body 110 to perform actions, and repeats steps S4 to S6 until the task ends.

[0033] The wide-angle fisheye camera 121 on the handheld demonstration acquisition device 120 has a field of view covering the front and sides. An auxiliary reflector 122 is positioned at one edge of the field of view of the wide-angle fisheye camera 121. The auxiliary reflector 122 acts as a planar reflector, reflecting the side view image of the scene to a predetermined area of ​​the imaging sensor plane of the wide-angle fisheye camera 121. Through a calibration process, the plane equation of the auxiliary reflector 122 in the coordinate system of the wide-angle fisheye camera 121 is determined. Let the plane equation be... ; in, The unit normal vector of the plane ( ), The distance from the origin to the plane. Let be the coordinates of any point on the plane.

[0034] Based on the principle of planar reflection, the optical center relative to the real camera is calculated. Virtual camera light center pose transformation matrix Define the reflection transformation matrix. as follows: ; in, It is the identity matrix. Let be the unit normal vector of the plane. This is the distance from the origin to the plane.

[0035] For any point in space Its coordinates in the virtual camera coordinate system Coordinates in the real camera coordinate system Satisfying Relationships ; in, Let P be the coordinates of any point P in space within the virtual camera coordinate system. Here is the reflection transformation matrix. Let P be the coordinates of any point P in space in the real camera coordinate system.

[0036] During image acquisition, the processor divides the imaging sensor plane into a direct observation area and a specular reflection area. For feature points extracted from the direct observation area, a real camera intrinsic parameter model is used. Projection is performed; for feature points extracted from the specular reflection region, the projection is equivalent to the spatial points being projected via... The transformed projection onto a real camera. A single wide-angle fisheye camera 121 simultaneously acquires images with baseline distance in each frame. Binocular observation data, among which for The translation components provide geometric constraints for restoring scene depth and absolute scale.

[0037] The inertial measurement unit 123 and the wide-angle fisheye camera 121 undergo time synchronization and external parameter calibration. (System definition...) State vector at time step Including location ,speed Posture Quaternions and the zero bias of the acceleration of the inertial measurement unit and gyroscope zero bias : ; In adjacent image frames and Between these steps, pre-integration is performed using measurement data from the inertial measurement unit to construct inertial constraint terms. The pre-integrated measurement values ​​are defined. , , It relies solely on inertial measurement data and zero bias. The state estimation problem is constructed as a nonlinear least squares optimization problem, with the objective function... Includes marginalized prior terms and inertial pre-integration residual terms. and visual reprojection residuals

[0038] ; in, This is the set of all state variables and inverse depths of landmarks within the sliding window. A set of keyframes For the set of characteristic observations, This refers to the prior residuals generated when marginalizing the old state. This is the corresponding prior Jacobian matrix. Indicates the first Frame and the Raw observation data (acceleration and angular velocity) from the inertial measurement unit between frames. Indicates the camera frame index. Indicates the landmark index. For the first The first road sign point Pixel observation coordinates on the frame image and These are the covariance matrices for inertial measurement and visual measurement, respectively. This represents the square of the distance to Maharanobis. This is a robust kernel function.

[0039] Visual reprojection residual Specifically defined as the pixel coordinates of the observed feature points. with road signs Errors between projected coordinates. For feature points in the directly observed area, the projection model adopts the standard pinhole model. For feature points in the reflection region, the projection model includes the reflection matrix. Transformation: (Direct observation); (Reflection observation); in, For visual reprojection residuals, The pixel coordinates of the observed feature points. Projection of a standard pinhole model This is the extrinsic parameter matrix from the inertial measurement unit to the camera. Acting on a spatial point in the camera coordinate system For the first The pose of the frame machine in the world coordinate system. These are the three-dimensional coordinates of the landmark in the world coordinate system.

[0040] The matrix multiplication order in the formula represents transforming points from the world coordinate system to the camera coordinate system, applying reflection transformation, and finally projecting onto the imaging plane. By solving the above optimization problem, the six-DOF pose sequence of the hand can be calculated. Regarding the back-end optimization solver and feature point extraction algorithm involved in state estimation, those skilled in the art can choose conventional implementation methods based on actual computing resources. These are well-known technologies in the field and will not be elaborated upon here.

[0041] In order to enable the learned strategy to have spatial generalization ability, the system calculates the absolute pose sequence in step S202. Convert to the starting pose relative to the action The relative motion sequence. Here, a Lie group is used. The logarithmic mapping method using the Lie algebra se(3) compresses the rigid body transformation matrix into a six-dimensional motion vector. For the th... Calculate the relative pose transformation matrix at each time step. Subsequently, logarithmic mapping and vectorization operations were used to... Mapped to a six-dimensional vector : ; in, For the first A six-dimensional vector mapped by the relative pose transformation at each time step. This is the pose matrix of the first frame of the taught trajectory. For special Euclidean groups To its tangent space logarithmic mapping, It is a vectorization operator.

[0042] The generated vector The first three dimensions represent the translation components, and the last three dimensions represent the rotation vector components. The resulting relative trajectory sequence... It includes geometric features that are only related to the action itself, decoupling its absolute position in the world coordinate system.

[0043] For synchronously acquired wide-angle image frames The system utilizes a pre-trained visual encoder network to extract semantic and spatial features. The original image is adjusted to a preset resolution and normalized. It is then input into the visual encoder network, and the output of the deeper layers is extracted as the visual feature vector. ,in This represents the dimension of the feature embedding. The visual feature vector encodes the liquid surface state, milk foam texture, and relative position to the coffee cup within the latte art pitcher. For the visual encoder network, a Transformer-based model or a convolutional neural network model can be used.

[0044] The system synchronously reads the current joint angles of the robotic arm. Joint angular velocity and the opening and closing degree of the end grippers ,in This represents the number of joint degrees of freedom of the robotic arm. To eliminate the influence of different physical dimensions, the above data is subjected to min-max normalization and mapped to the interval [-1, 1] to form the body state vector. ,in This represents the normalized value.

[0045] To integrate visual perception with ontological perception, the system constructs a projection layer. The projection layer It consists of a multi-layer fully connected network and a non-linear activation function. It processes visual feature vectors... With the ontology state vector Cascade and input to the projection layer In this context, it is mapped to a fixed-dimensional observation condition vector. : ; in, For the observation condition vector, For projection layer, This is a vector concatenation operation. The hidden layer feature dimension of the transformer model. For visual feature vectors, This is the ontological state vector, and the observation condition vector. It includes a description of the current environment and its own state, which serves as a constraint for the diffusion model to generate the trajectory at the next moment.

[0046] The processed relative trajectory sequence As the prediction target, the corresponding multimodal conditional observation vector sequence As input, the data is divided into fixed-length sequence windows based on time steps. For the data within each window, the system also calculates the action sequence, i.e., the relative trajectory difference between adjacent time steps, to assist in training the dynamic model. All processed data tuples are stored in a storage unit for subsequent use by the offline training module.

[0047] The input to the denoising network consists of three parts: the current diffusion time step. Noisy trajectory sequences Diffusion Step Size Index and the generated multimodal observation condition vector .in, The length of the trajectory sequence. The dimension of the relative trajectory features includes a six-dimensional Lie algebra motion vector and a one-dimensional gripper opening.

[0048] The system first uses a linear projection layer to transform the noisy trajectory sequence. Mapping to hidden layer dimension To preserve the temporal information of the sequence, learnable positions are encoded. Adding the projected trajectory features to obtain the initial hidden state sequence .

[0049] Meanwhile, for diffusion step size index The vector is mapped using sinusoidal positional encoding, and then transformed using a multilayer perceptron (MLP) to obtain the temporal embedding vector. The formula for calculating the sinusoidal position code is as follows: ; ; in, For dimensional indexing, This represents the numerical value of the location encoding vector in the corresponding dimension. For diffusion step size index, This represents the hidden layer feature dimension. This is the temporal embedding vector. Subsequently, it was used with the multimodal observation condition vector. By concatenating or adding them together, a unified global condition vector can be formed. , used for modulation of subsequent network layers.

[0050] The backbone of the denoising network consists of It consists of stacked Transformer encoder blocks. Each Transformer block contains a multi-head self-attention mechanism (MHSA) and a feedforward neural network (FFN). This is to integrate the global conditional vector... In this embodiment, an adaptive layer normalization (AdaLN) layer is introduced into each Transformer block. In the... In each Transformer block, the input features First, it is processed by global conditions. Control normalization processing. The system utilizes a linear projection layer. from Calculate the scaling factor Translation factor The modulation process is defined as follows: ; in, For adaptive layer normalization operation, For the first Input features of each Transformer block This is the global condition vector. For scaling factor (vector). It is the translation factor (vector). For element-wise multiplication, This is a standard layer normalization operation. The modulated features are input to the multi-head self-attention mechanism layer to calculate the dependencies within the sequence: ; Subsequently, the features undergo the same adaptive modulation process again and are then input into the feedforward neural network: ; in, For the first The final output features of each Transformer block, The output features of the multi-head self-attention mechanism layer are... For the first Input features of each Transformer block This is a multi-head self-attention mechanism. It is a feedforward neural network. For adaptive layer normalization, This is the global condition vector.

[0051] Through this hierarchical modulation mechanism, multimodal observation information and diffusion time information directly affect the network's layer-by-layer abstraction and reconstruction of trajectory features.

[0052] go through Hidden state sequence after processing by the Transformer block Through final layer normalization and a linear decoding layer, the original trajectory dimension is mapped back. The predicted noise vector is obtained. Perform the offline training process. During the training phase, real relative trajectories are randomly sampled from the preprocessed dataset. and corresponding conditions Random sampling diffusion time step Uniform and Gaussian noise ,in The maximum number of diffusion steps is preset. A noisy trajectory is generated based on the diffusion process formula. : ; in, For adding noise to the trajectory, Preset noise scheduling parameters The cumulative product, For real relative trajectory samples, It is Gaussian noise. Input the noise to the denoising network and calculate the predicted noise. Network parameters Optimization is achieved by minimizing the mean squared error loss function: ; in, Let the mean squared error loss function be . It is Gaussian noise. For the predicted noise vector, For adding noise to the trajectory, For diffusion step size index, This represents the multimodal conditional observation vector.

[0053] After training, the network possesses the ability to progressively remove random noise to generate latte art trajectories that conform to the human teaching distribution, given visual and ontological states. The specific parameter settings and optimizer selection within the Transformer architecture can be routinely adjusted by those skilled in the art based on actual memory resources and task complexity, and will not be elaborated upon here.

[0054] The forward diffusion process is constructed as a Markov chain, and the system configuration is based on a preset noise scheduling scheme. To real relative trajectory samples Gaussian noise is gradually added to the process. For each time step... From the state of the previous moment Current state transition probability distribution The definition is as follows: ; in, The mean is Covariance is The normal distribution It is the identity matrix. The current state. For the first The variance parameter of the step, This represents the state at the previous moment.

[0055] Based on the superposition property of Gaussian distributions, the reparameterization technique is used to derive the initial state... Sample any Moment State Marginal probability distribution

[0056] ; in, To start from the initial state Sample the state at any time t Marginal probability distribution, for Current state The cumulative noise figure, This is the initial state. It is an identity matrix.

[0057] The corresponding sampling formula is: ; in, for Current state The cumulative noise figure, This is the initial state. The noise is Gaussian. Based on this formula, the training module directly generates noisy samples at any time step for network input.

[0058] The goal of the reverse process is to learn the transition distribution. Approximate distribution Thus, it is possible to extract pure noise Initially, under the observation conditions The trajectory was gradually reconstructed under constraints. The system defines this reverse transition distribution as a Gaussian distribution: ; in, This is an approximate distribution of the reverse transition distribution. This refers to the state at the previous moment. The current state. For observation conditions, To predict the mean, To predict variance.

[0059] In one implementation, variance Set as a constant . The value can be selected as or posterior variance ,in And set For the mean The system outputs predicted noise through a denoising network. Perform the calculation. Based on the derivation of Bayes' theorem, the mean... The relationship with the noise prediction network is defined as follows: ; in, To predict the mean, This is a quantity related to the noise figure. The current state. For variance parameter, This refers to the noise component predicted by the neural network.

[0060] Based on this parameterization, at each time step of the generation process The system calls a neural network to predict the noise components contained in the current trajectory and calculates the denoising estimate of the previous moment according to the above formula. For The specific value range is set at 10. -4 The value is between 0.02 and 0.02, depending on the total number of steps. Decide.

[0061] To train the denoising network, the system uses a simplified variational lower bound as the optimization objective, directly calculating the mean square error between the predicted noise and the actual noise. Define the... loss function of the next iteration as follows: ; in, For the desired operation, For diffusion step size index, These are real relative trajectory samples. It is Gaussian noise. for The predicted noise output by the denoising network. for The trajectory of adding noise at all times, To represent the square of the Euclidean norm, This is the multimodal observation condition vector.

[0062] To support classifier-free guidance during the inference phase, the system introduces a random condition masking mechanism during training. The system is configured to use preset probabilities. Multimodal observation condition vector Replace with empty embedding vector At this point, the input to the denoising network becomes... This allows a single network to possess both conditional and unconditional generation capabilities after training, enabling it to balance the consistency between generated trajectories and conditions, as well as the diversity of generated trajectories, by adjusting the guiding scale during inference.

[0063] The system randomly samples a small batch of data from the training database and calculates the loss function described above for the network parameters. gradient The AdamW optimizer is used to update the parameters based on the calculated gradients. The optimizer's update rule combines first-order moment estimation and second-order moment estimation, and introduces a weight decay term. For the specific implementation logic of the AdamW optimizer, those skilled in the art can refer to the documentation of relevant open-source deep learning libraries; it is well-known technology in this field and will not be elaborated upon here.

[0064] To improve the model's generalization ability and the stability of the generation process, the system stores and updates exponential moving average copies of the network parameter weights. In each parameter After the update, update according to the following formula.

[0065] ; in, For the first Step-updated network parameters, For the first Step-updated EMA parameters, The attenuation rate, For the first The updated EMA parameters are used by the system during the inference phase after training. Replace original parameters Trajectory generation is performed. This strategy smooths out oscillations during parameter updates, generating smoother and more robust latte art trajectories.

[0066] In the initial training phase, a linear warm-up strategy is employed, linearly increasing the learning rate from 0 to a preset base learning rate to stabilize the gradient at the beginning of training. After the warm-up phase, a cosine annealing strategy is used to gradually decrease the learning rate, allowing the model to converge to a better local minimum in the later stages of training. The specific learning rate... The calculation formula is determined by the current iteration step, ensuring continuous optimization until the loss function converges.

[0067] System Definition The combined state vector at time step and control input vector ,in, and These represent the dimensions of the state vector and the input vector, respectively. Constructed as a relative pose (six-dimensional Lie algebra vector) including the end effector of the robotic arm. ), end velocity, and the estimated position of the fluid centroid and the angle of inclination of the liquid surface inside the drawing cylinder. Includes torque commands for the robotic arm joints or acceleration commands for the end effector. Combined state vector. By concatenating the relative trajectory features and their first derivatives into vectors, a complete state description of the dynamic evolution is provided.

[0068] Nominal dynamic model Based on the principles of rigid body dynamics, this model describes the deterministic motion characteristics of a system that conform to the laws of physics. This part assumes the fluid inside the vat is a rigid load and neglects liquid sloshing and deformation. The discrete-time state transition equations of the nominal model are defined as follows: ; in, To consider only the state predicted by rigid body dynamics at the next moment, For the nominal dynamic model function, This is the current state. To control the input.

[0069] The calculation process of this function includes the kinematic Jacobian matrix and dynamic parameters (mass, inertia tensor) of the robotic arm. This nominal model provides a low-frequency fundamental estimate of the system's motion, ensuring the basic physical reasonableness of the prediction results. For the specific rigid body dynamic parameters in the nominal model, those skilled in the art can use the least squares method or genetic algorithm for offline identification.

[0070] To compensate for the neglected fluid dynamics characteristics and nonlinear errors such as system friction in the nominal model, a residual neural network was constructed. The network employs a multilayer perceptron (MLP) or gated recurrent unit (GRU) architecture, with the current state as its input. and control input The output is the predicted value of the state deviation. The mathematical expression of the residual model is as follows: ; in, The dynamic residual term predicted by the neural network, This is the set of learnable weights and bias parameters for a residual neural network. This is the current state. To control the input, For residual neural network models, i.e. In this way, neural networks are used to characterize high-frequency nonlinear features such as fluid sloshing that are difficult to describe using analytical formulas.

[0071] By superimposing the outputs of the nominal model and the residual model, we obtain the complete hybrid power prediction equation: ; in, To predict the system state at the next moment, For the complete hybrid dynamics mapping relationship, For the nominal dynamic model, For residual models, These are the weighting parameters. This equation clarifies the specific structure of the "hybrid dynamics model" in the claims: that is, it consists of analytical terms... and learning items It is formed by linear superposition. This structure ensures that the model can still maintain the constraints based on physical laws in sparse training data regions, while in data-rich regions, the prediction accuracy can be improved through neural networks.

[0072] The system utilizes the collected demonstration dataset The hybrid model is trained end-to-end. The training objective is to minimize the multi-step prediction error, and the loss function is... Defined as: ; in, For batch size, For the first The true next-moment state of each sample and The first The current state and control input of each sample. To represent the square of the Euclidean norm, This is a regularization coefficient used to prevent overfitting in neural networks. For weight parameters, This is a residual neural network. The weights are updated using the backpropagation algorithm. This allows the hybrid model to approximate the real fluid manipulation dynamics process.

[0073] The system allocates a fixed-size storage space in memory to construct an experience replay buffer. The buffer is configured as a circular queue structure with a maximum capacity. The buffer stores the dynamics transformation tuples from the most recent time step. Each tuple... Includes the current state, control input, and the observed state at the next time step: ; in, This is the current state. To control the input, This is the actual measured value. When the amount of data in the buffer exceeds... When this happens, the system performs an overwrite operation, removing the tuple with the oldest timestamp and writing the latest tuple to ensure that the data used for model updates always reflects the current physical characteristics of the system.

[0074] Step S407: Perform real-time data acquisition and sliding window construction. During the latte art action, the sensor samples at a frequency of... Collect the robot's joint angles, end-effector pose, and torque information, and assemble them into the aforementioned tuple. Store in buffer To update parameters, the system truncates a value from buffer B containing the most recent... A sliding window dataset of samples ,in The batch size (BatchSize) is adjusted online and satisfies the following conditions: This sliding window dataset serves as the micro-batch input for online training.

[0075] Step S408: Define the online adaptive loss function. To quickly correct the prediction bias of the residual model, the system defines an online loss function based on a sliding window. The loss function calculates the residual between the nominal model predictions and actual measurements for all samples within the sliding window, and uses this residual as a label to supervise the output of the residual neural network. The loss function formula is as follows: ; in, To adapt the loss function online, This is the batch size for the sliding window. These are actual measured values. These are the current weight parameters of the residual neural network. The output is a fixed nominal dynamic model. The output of the residual neural network to be updated. This represents the square of the Euclidean norm.

[0076] The system parameter update frequency is set to every The process executes once per control cycle. When an update condition is triggered, the processor calculates the online loss function. Regarding weight gradient The weights are updated using stochastic gradient descent (SGD) or its variants: ; in, For the updated weight parameters, The weight parameters before the update. For online learning rate, For the online loss function with respect to the weights The gradient. Updated parameters. Immediately loaded into the neuro-physical hybrid model to participate in subsequent time steps. The model predicts and controls the computation. The backpropagation process in gradient calculation is implemented using existing automatic differentiation frameworks, which are well-known techniques in the field and will not be elaborated upon here.

[0077] To ensure the generated relative trajectory The system exhibits dynamic characteristics consistent with those described by a neuro-physical hybrid model, and the system constructs a dynamic prediction error term. This measure assesses the deviation between the transitions between adjacent states in the trajectory and the predictions of the hybrid dynamics model. The calculation formula is as follows: ; in, For the dynamic prediction error term, The length of the trajectory sequence. This is the current state. A neurophysical hybrid model. For the nominal inverse dynamics operator based on rigid body dynamics, This represents the square of the Euclidean norm. The physical meaning of this energy term is that, for a physically feasible trajectory, its state evolution should be characterized by the hybrid dynamics equations, i.e., the prediction error should approach zero.

[0078] To prevent high-frequency material movement during the generated motion, which could lead to deformation of the latte art pattern or fluid overflow, the system incorporates a smoothness constraint. This penalty is based on the second-order difference of the trajectory (i.e., changes in acceleration / jerk). The calculation formula is as follows: ; in, For trajectory smoothness energy term, The length of the trajectory sequence. For the state in The acceleration or second derivative vector at time step. Minimizing this term constrains the rate of change of the system's control energy, ensuring that the generated trajectory maintains continuous and smooth motion characteristics to match the motion distribution characteristics of the taught data.

[0079] The system weights and combines the dynamic consistency energy term and the trajectory smoothness energy term to obtain the final total energy function used for guiding sampling.

[0080] ; in, Let be the total energy function. These are the weighting coefficients for the dynamic constraints. For the kinetic uniformity energy term, These are the weighting coefficients for the smoothness constraint. This represents the trajectory smoothness energy term. Both coefficients are non-negative real numbers, used to balance the importance of physical consistency and motion smoothness in the optimization process. In a preferred embodiment, The value is set to 5.0. The value is set to 0.05. This total energy function... For input trajectory It is totally differential and supports calculating information about the trajectory through an automatic differentiation mechanism. gradient This provides guidance for subsequent Langevin dynamic sampling.

[0081] To achieve gradient backpropagation, the system reconstructs the kinematic state defined in the preceding steps and performs nominal inverse dynamics calculation. Neuro-physical hybrid forward prediction The energy function summation operation constructs a connected directed acyclic graph. Within this computational graph, the generated trajectory... The variable nodes are marked as to be optimized, while the weight parameters in the hybrid power model are... These nodes are set as fixed-parameter nodes and do not participate in gradient calculation. This ensures that the optimization process only applies to the trajectory itself and does not change the parameters of the already trained dynamic model. Step S505: Perform the chain rule gradient calculation. The system calculates the total energy function through inverse mode differentiation. For each time step state in the trajectory The partial derivatives. According to the chain rule, the total gradient. The calculation is decomposed into the sum of the products of the partial derivatives with respect to each intermediate variable: ; in, The energy function with respect to the sequence of integrated state vectors gradient, Construct the Jacobian matrix of the function with respect to the trajectory coordinates for the state. The total energy function with respect to the trajectory The gradient. Due to the state Through trajectory It is constructed by linear difference, and this term is a sparse constant matrix. This represents the transpose operation of a matrix or vector.

[0082] For the core dynamic energy term Its state The gradient propagation path flows through the hybrid dynamics model, specifically as follows: ; in, For the dynamic energy term in relation to the state gradient, The state predicted by the model for the next time step. The next state of the target. For hybrid dynamics model, To control the input, This is the inverse dynamic equation. The above formula shows that gradient information is transmitted not only directly through the state, but also through the control input. Indirect transmission, in which The analytical Jacobian matrix corresponding to the inverse dynamic equation.

[0083] In calculation items At that time, the system calculates and sums the gradient contributions of the nominal model and the residual network respectively: ; in, The gradient of the hybrid power model with respect to the state. For the nominal dynamic model The analytic Jacobian matrix, The gradient of the residual neural network with respect to the input. These are the weight parameters.

[0084] After a reverse traversal of the computational graph, the system summarizes the gradient components at all time steps, and outputs the trajectory along with the input trajectory. gradient vectors of the same dimension : ; in, The gradient vector, Let be the gradient of the total energy function with respect to the input trajectory x.

[0085] Each element of the gradient vector represents the direction and intensity that the trajectory coordinates in the corresponding dimension need to be adjusted to reduce physical energy. For the specific implementation of automatic differentiation, those skilled in the art can utilize the automatic differentiation modules in existing deep learning frameworks such as PyTorch or TensorFlow. The underlying implementation is well-known in the field and will not be elaborated upon here.

[0086] At each time step of the back diffusion process (from Decrease to 1), the system receives the current noisy trajectory. The denoising network is used in unconditional input... Conditional input The next step is to perform forward inference and output unconditional prediction noise. and conditional prediction noise The system uses a classifier-free guidance formula to synthesize initial prediction noise. : ; in, For initial prediction noise, For unconditional prediction of noise, To guide the scaling of the classifier Predict noise based on conditions.

[0087] Based on the current noisy trajectory and synthetic prediction noise The system estimates the current clean trajectory based on the inverse formula of the diffusion model. This estimate represents the final noise-free trajectory predicted by the model at the current level of denoising. The calculation formula is as follows: ; in, This is the current clean trajectory estimate. For the current noisy trajectory, For a moment The cumulative signal retention coefficient, This is used to synthesize prediction noise.

[0088] The system will estimate the clean trajectory Input to the aforementioned constructed kinetic energy function In the process, the gradient of the energy function with respect to the trajectory is calculated using automatic differentiation. This gradient reflects the degree and direction to which the currently estimated trajectory violates the physical dynamics constraints and motion smoothness constraints. To control the strength of the physical guidance, the system introduces a physical guidance scale. Calculate the corrected pilot noise

[0089] ; in, This is the corrected pilot noise. To synthesize prediction noise, For physical guidance scale. Let be the gradient of the energy function with respect to the trajectory. The physical meaning of this formula is: by superimposing a correction term proportional to the energy gradient into the prediction noise, a cleaner trajectory... With noise They are negatively correlated. Increasing the noise component is equivalent to updating the clean trajectory along the negative gradient direction of the energy function, thereby reducing the energy of the generated trajectory. The term is used to adaptively adjust the gradient size based on the current noise level, ensuring that physical guidance does not disrupt the overall structure of the trajectory in the initial stages when noise is high.

[0090] Using the corrected noise The system calculates the state of the previous time step (after denoising) based on the deterministic recursive formula of DDIM. The updated formula is as follows: ; in, For a moment The cumulative signal retention coefficient. The term within parentheses in the formula represents the value based on the corrected noise. The recalculated physical consistency clean trajectory estimate.

[0091] The system receives the physically consistent trajectory generated by the energy-guided DDIM algorithm. The trajectory consists of a series of discrete-time step relative state vectors, containing the pose of the robotic arm's end effector relative to the current liquid surface, represented as a six-dimensional Lie algebra vector. To perform coordinate transformation, the processor first uses an exponential map to convert the Lie algebra vector into a homogeneous transformation matrix in Lie group form. : ; in, It is a matrix exponential function. An operator that maps a six-dimensional vector to a 4×4 antisymmetric matrix. For relative rotation matrices, This is the relative displacement vector. This transformation matrix describes the spatial relationship between the drawing needle or milk cup nozzle and the center point of the current fluid surface.

[0092] Because the milk injected during the latte art process will cause the liquid level to rise... As time progresses, the system needs to establish a dynamic liquid level reference coordinate system. The system updates the current time frame based on the integral value of the flow velocity from the flow sensor or feedback from the visual observation module. liquid level height Based on the liquid level and the fixed pose of the cup in the world coordinate system. Construct the transformation matrix of the liquid surface relative to the world coordinate system. : ; in, The transformation matrix is ​​the center of the flower cylinder base relative to the coordinate system of the robotic arm base. This matrix is ​​obtained in advance through hand-eye calibration. The transformation matrix for liquid level rise is defined as the translation along the axis of the drawing cylinder (usually the Z-axis). Distance transformation matrix: ; in, It is a 3×3 identity matrix. The height of the liquid level. This is the transformation matrix for liquid level rise. This step ensures that the generated trajectory can automatically adapt to changes in liquid level under different capacities of frothing cylinders and different injection flow rates, thereby compensating for the relative height deviation caused by the rise in liquid level in absolute space.

[0093] Using the chain rule of homogeneous transformation matrices, the system calculates the absolute target pose of the robotic arm's end effector in the world coordinate system. : ; in, This represents the absolute target pose of the robotic arm's end effector in the world coordinate system. Let be the transformation matrix of the liquid surface relative to the world coordinate system. For relative motion command transformation matrix, this formula maps relative motion commands to operation commands in absolute space. Includes end effector in The coordinates of the position that should be reached at that time And attitude direction (quaternion or rotation matrix form).

[0094] In order to drive the robotic arm joints, the system calculates the absolute pose sequence. The input is fed into the Inverse Kinematics (IK) solver. Based on the link parameters (DH parameters) of the robotic arm, the solver calculates a set of joint angle vectors that satisfy the target pose. : ; in, IK represents the number of degrees of freedom of the robotic arm. For the inverse kinematics solver function, For the absolute target pose, This represents the joint angles at the previous moment in the system. For cases where there are multiple solutions in the inverse kinematics, the system is configured to calculate all feasible solutions and the joint angles at the previous moment. The Euclidean distance is calculated, and the solution with the smallest distance is selected as the current control command. The calculated joint angle sequence... After three spline interpolation smoothing processes, position, velocity, and acceleration commands are generated and sent to the servo driver for execution.

[0095] To construct the optimization problem, the system first defines a state vector. and control input vector State vector Including the robotic arm in the Predicting joint angles of the step and joint angular velocity ,Right now Control input vector The driving torque vector of each joint motor Reference trajectory sequence It consists of the joint angles output by the aforementioned inverse kinematics solution module and the velocities obtained through differential calculation.

[0096] The system establishes a quadratic objective function. The configuration is to calculate the weighted sum of tracking errors between the actual state and the reference state, while constraining the energy consumption of the control input. The objective function of the optimization problem is as follows: ; in, To predict the time domain length, it is set to cover the number of steps (e.g., 10 to 50 steps) from 0.5 to 1.0 seconds into the future, U = The control input sequence to be optimized. For the predicted first Step state, For the corresponding reference trajectory state, Here is the state error weight matrix. To control the input weight matrix, both are positive semi-definite diagonal matrices. This is the terminal cost weight matrix, used to constrain the convergence of the predicted state at the end of the time domain. This represents the square of the Euclidean norm weighted by the weight matrix W, i.e. .

[0097] To ensure that the calculated control commands are physically executable, the optimization process must satisfy a series of constraints. The equality constraints are constituted by the previously trained neuro-physical hybrid model, while the inequality constraints are determined by the hardware limits of the robotic arm. ; in, This is a hybrid dynamics model, which serves as the nonlinear state transition constraint equation. Mechanical limit for joint angle. This is the joint velocity limit. This represents the saturation limit of the motor's torque output. For the joint angle. By embedding the hybrid dynamics model into the constraints, the controller is able to handle the nonlinear dynamic effects of fluid load variations.

[0098] The controller employs either Sequential Quadratic Programming (SQP) or the nonlinear interior-point method to solve the aforementioned constrained nonlinear programming problem. During the solution process, the system utilizes an automatic differentiation mechanism to calculate the hybrid dynamics model. Regarding the status The Jacobian matrix of the control u is used to construct a local quadratic programming subproblem. The optimal control sequence is then obtained. Then, according to the rolling time-domain control strategy, the system only applies the first control variable in the sequence. The command is sent to the servo driver for execution. In the next control cycle... The system reads the actual state of the robotic arm through an encoder. This is then used as a new initial state to repeat the optimization process described above. In this embodiment, the optimal solution sequence from the previous time step is used as the initial value for the current optimization step to reduce the number of iterations and ensure that the algorithm's running frequency meets real-time control requirements (e.g., 50Hz to 200Hz).

Claims

1. An adaptive reasoning method for the latte art trajectory of a robotic arm based on a diffusion transformer model, characterized in that, Includes the following steps: A multimodal teaching sequence containing image data and inertial data is acquired using a handheld demonstration acquisition device (120); The absolute pose sequence of the teaching action is obtained by the multimodal data preprocessing module (201) based on the multimodal teaching sequence, and the absolute pose sequence is converted into a relative trajectory vector in the Lie algebra space to construct a training dataset. The conditional probability distribution of the relative trajectory vector is learned by offline training of the diffusion transformer network in the conditional diffusion model training module (202) using the training dataset. A hybrid dynamics model comprising a nominal dynamics model and a residual neural network model is constructed, and the parameters of the residual neural network model are updated online based on real-time feedback data from the robotic arm body (110) through an online residual dynamics learning module (203), wherein the residual neural network model is used to compensate for the prediction bias of the nominal dynamics model; The energy field-guided inference module (204) constructs a dynamic energy function using the updated hybrid power model and calculates the gradient of the dynamic energy function to guide the backsampling process of the diffusion converter network, generating a latte art trajectory that conforms to physical constraints. The model predictive control execution module (205) solves for the optimal control command based on the latte art trajectory to drive the robotic arm body (110), and repeatedly executes the steps of updating the parameters of the residual neural network model online and generating the latte art trajectory that conforms to the physical constraints until the task ends.

2. The adaptive inference method for the latte art trajectory of a robotic arm based on a diffusion transformer model according to claim 1, characterized in that, The specific process of calculating the absolute pose sequence of the taught actions includes: The image acquired by the wide-angle fisheye camera (121) on the handheld demonstration acquisition device (120) is divided into a direct observation area and a specular reflection area reflected by an auxiliary reflective mirror (122). Based on the plane equation of the auxiliary reflecting mirror (122), a reflection transformation matrix is ​​constructed. The reflection transformation matrix is ​​used to establish a visual reprojection residual model of feature points in the reflection area of ​​the mirror, and a visual reprojection residual model of feature points in the direct observation area is also established. Inertial pre-integration residuals are constructed by combining the inertial data, and a nonlinear least squares optimization objective function is constructed that includes marginalization priors, the inertial pre-integration residuals, and the visual reprojection residual model. Solve the optimization objective function to calculate the six-degree-of-freedom pose sequence of the handheld demonstration acquisition device (120), which is used as the absolute pose sequence.

3. The adaptive inference method for the latte art trajectory of a robotic arm based on a diffusion transformer model according to claim 1, characterized in that, The specific process of converting the absolute pose sequence into relative trajectory vectors in Lie algebra space to construct the training dataset includes: Calculate the relative pose transformation matrix of the k-th time step relative to the initial pose in the teaching motion trajectory; Using the logarithmic mapping from Lie group to Lie algebra, the relative pose transformation matrix is ​​mapped to a six-dimensional motion vector, which serves as the relative trajectory vector; Extract the visual feature vector of the synchronously acquired image frame and obtain the current body state vector of the robotic arm body (110). The body state vector includes the normalized joint angle, joint angular velocity and end gripper opening degree. The visual feature vector and the ontology state vector are concatenated by a projection layer to generate an observation condition vector, and a multimodal demonstration dataset containing the relative trajectory vector and the observation condition vector is constructed as the training dataset.

4. The adaptive inference method for the latte art trajectory of a robotic arm based on a diffusion transformer model according to claim 1, characterized in that, The process of offline training of the conditional diffusion model training module (202) using the training dataset includes: Construct a denoising network based on the Transformer architecture, wherein the denoising network comprises multiple stacked Transformer encoder blocks; An adaptive normalization layer is introduced into each Transformer encoder block, and the input features are scaled and translated using a global condition vector generated by the observation condition vector and the diffusion time embedding vector. Real relative trajectory samples are sampled from the training dataset, and Gaussian noise is added according to preset noise scheduling parameters to generate noisy trajectories; The noise trajectory, diffusion step size index, and observation condition vector are input into the denoising network. The parameters of the denoising network are updated by minimizing the mean square error between the predicted noise and the added Gaussian noise.

5. The adaptive inference method for the latte art trajectory of a robotic arm based on a diffusion transformer model according to claim 1, characterized in that, The process of constructing a hybrid dynamics model comprising a nominal dynamics model and a residual neural network model, and the process of updating the parameters of the residual neural network model online, includes: The hybrid dynamics model is defined as a linear superposition of the nominal dynamics model and the residual neural network model, wherein the nominal dynamics model is used to describe the rigid body dynamics characteristics, and the residual neural network model is used to characterize the state deviations caused by fluid sloshing and nonlinear friction. Construct an experience replay buffer and store dynamic transformation tuples containing the current state, control input, and actual measurement value at the next moment in a circular queue structure; Extract a sliding window dataset from the experience playback buffer and calculate the residual between the nominal dynamic model prediction and the actual measurement; Based on the residual, the gradient of the online adaptive loss function is calculated, and the weight parameters of the residual neural network model are updated online using the stochastic gradient descent algorithm.

6. The adaptive inference method for the latte art trajectory of a robotic arm based on a diffusion transformer model according to claim 5, characterized in that, The dynamic energy function is a weighted combination of the dynamic consistency energy term and the trajectory smoothness energy term; The dynamic consistency energy term is obtained by calculating the deviation between the transition between adjacent states in the trajectory and the predicted value of the hybrid dynamics model; The trajectory smoothness energy term is calculated based on the second-order difference of the trajectory state sequence; The energy field guided inference module (204) calculates the gradient of the total energy function with respect to the input trajectory through an automatic differentiation mechanism, and superimposes it as a guiding term into the backsampling process of the diffusion converter network.

7. The adaptive inference method for the latte art trajectory of a robotic arm based on a diffusion transformer model according to claim 6, characterized in that, The specific process of generating the latte art trajectory that conforms to physical constraints includes: At each time step of the back diffusion process, the unconditional prediction noise and conditional prediction noise are calculated using the diffusion converter network, and the initial prediction noise is synthesized using the classifier-free guided formula. Based on the current noisy trajectory and the initial predicted noise, estimate the current clean trajectory; Calculate the gradient of the kinetic energy function with respect to the clean trajectory; Based on a preset physical guidance scale, the gradient is used to correct the initial prediction noise, causing the generated trajectory to shift in the direction of decreasing dynamic energy function value; Using the corrected noise, the state at the previous moment is calculated according to the recursive formula of the denoised diffusion implicit model until the final physically consistent trajectory is generated.

8. The adaptive inference method for the latte art trajectory of a robotic arm based on a diffusion transformer model according to claim 1, characterized in that, The process by which the model predictive control execution module (205) solves for the optimal control command based on the latte art trajectory includes: Update the liquid level based on the current fluid flow rate integral or visual feedback; Based on the liquid level height, a dynamic liquid level reference coordinate system is constructed, and the Lie algebra relative trajectory vector in the latte art trajectory is converted into an absolute target pose sequence in the world coordinate system. Construct a quadratic programming optimization problem that includes the hybrid dynamics model as equality constraints and the limits of the robot arm joint position, velocity, and torque as inequality constraints; The optimization problem is solved in the prediction time domain to obtain the optimal control sequence, and the first control command of the optimal control sequence is sent to the robotic arm body (110).

9. The adaptive inference method for the latte art trajectory of a robotic arm based on a diffusion transformer model according to claim 1, characterized in that, During the offline training process, a random conditional masking mechanism is also introduced to replace the observation conditional vector with an empty embedding vector with a preset probability, so that the diffusion transformer network has both conditional generation capability and unconditional generation capability. Furthermore, during the training process, an exponential moving average copy of the network parameter weights is maintained, and this exponential moving average copy is used for trajectory generation in the step of generating a latte art trajectory that conforms to physical constraints.

10. The adaptive inference method for the latte art trajectory of a robotic arm based on a diffusion transformer model according to claim 1, characterized in that, Before acquiring the multimodal teaching sequences, a calibration step is also included: Determine the planar parameters of the auxiliary reflector in the wide-angle fisheye camera coordinate system. The planar parameters include the unit normal vector of the plane and the distance from the origin of the coordinate system to the plane. Based on the plane parameters, a reflection transformation matrix is ​​constructed, and the virtual camera optical center pose relative to the real camera optical center is calculated using the reflection transformation matrix. This allows for the construction of binocular observation constraints with a fixed baseline distance during monocular vision acquisition, which are used to recover scene depth and absolute scale.

Images