Deformable object interactive operation control method based on visual touch-language-action multi-mode model

Abstract

The invention relates to a deformable object interactive operation control method based on a visual touch-language-action multi-mode model, and the method comprises the steps: carrying out the image, touch and language data coding of a deformable object, and obtaining visual, touch and language features; performing cross-modal feature alignment processing on the visual features, the tactile features and the language features to obtain multi-modal fusion features; inputting the multi-modal fusion features into a large model for environment understanding; iteratively performing action planning and execution by adopting a'thinking-decision 'planning mode; and repeatedly executing the steps until the current deformable object interaction operation task is completed. Compared with the prior art, the multi-modal feature alignment capability, the action planning precision and the task adaptability are improved, efficient recognition and interaction of the robot on the deformable object can be achieved, deformation and state changes of the object can be effectively dealt with especially in a complex environment, the operation strategy can be dynamically adjusted, and the operation efficiency is improved. And more intelligent and accurate deformable object operation is realized.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent robot interactive control, and in particular to a method for interactive operation control of a deformable object based on a visual-touch-language-action multimodal model. Background Art

[0002] Intelligent robot systems mainly rely on visual perception and tactile feedback to complete object grasping and manipulation tasks. Traditional visual perception methods are more than computer vision technologies, such as convolutional neural networks for object detection and recognition, which can provide efficient object positioning information. However, when faced with deformable objects in complex environments, traditional visual methods cannot fully consider the deformation, elasticity and tactile feedback of objects, resulting in low grasping accuracy and success rate. On the other hand, tactile perception can help robots better understand the hardness, texture and deformation process of objects, and provide important operational feedback, but without an effective fusion strategy, tactile information alone is difficult to achieve efficient object manipulation.

[0003] In recent years, the rapid development of multimodal fusion technology has provided a solution to this problem. By integrating visual, tactile and language information, multimodal models can more comprehensively perceive the environment and reason about operation plans. However, the existing multimodal fusion technology is still used in the field of robot interactive operations, and the following problems still exist:

[0004] Insufficient cross-modal feature alignment. Different modal features have differences in expression form and semantic space, resulting in poor information fusion effect.

[0005] The dynamic planning and real-time adjustment capabilities are limited. When faced with complex operation tasks, the existing models have weak robustness in motion planning in dynamic environments.

[0006] Insufficient use of historical information and lack of modeling and storage of task history information affect the optimization and generalization capabilities of operation strategies.

[0007] In addition, existing object manipulation methods mainly rely on static perception data, such as images or tactile signals, which ignore the dynamic changes of object states and environmental feedback in actual operations. Especially for deformable objects, how to achieve dynamic perception and decision-making through multimodal fusion models to ensure the safety and stability of robots during operation is still a difficult point. Summary of the invention

[0008] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and to provide a method for interactive operation control of deformable objects based on a vision-touch-language-action multimodal model, which can improve the multimodal feature alignment capability, action planning accuracy and task adaptability, and can dynamically adjust the operation strategy to achieve more intelligent and precise deformable object operation.

[0009] The object of the present invention can be achieved by the following technical solution: A method for interactive operation control of a deformable object based on a visual-touch-language-action multimodal model, comprising the following steps:

[0010] S1. Encode the image, tactile and language data of the deformable object to obtain visual features, tactile features and language features;

[0011] S2, perform cross-modal feature alignment processing on visual features, tactile features and language features to obtain multi-modal fusion features;

[0012] S3, input multimodal fusion features into the large model for environmental understanding;

[0013] S4, adopt the “thinking-decision-making” planning method to iteratively plan and execute actions;

[0014] S5. Repeat steps S1 to S4 until the current deformable object interactive operation task is completed.

[0015] Furthermore, the specific process of step S1 is: mapping the visual image and tactile data of the deformable object into image embedding and tactile embedding respectively through visual and tactile encoders, and mapping the language instructions into language embedding, so as to extract visual, tactile and language features.

[0016] Furthermore, the visual encoder and tactile encoder both adopt a multi-layer Transformer structure, which are used to extract the visual features and tactile features of the object respectively. The visual encoder adopts a standard Vision Transformer (ViT) structure, divides the input image into patches of fixed size, converts them into embedded vectors through linear mapping, and inputs them into a Transformer encoder with a multi-layer multi-head self-attention mechanism to extract global and local features;

[0017] The tactile encoder adopts a dual-tower Transformer architecture, one tower is used to model the spatial features of tactile data, and the other tower is used to capture the temporal relationship of tactile signals. It achieves deep representation of tactile signals through a multi-head self-attention mechanism and combines cross-modal feature fusion to enhance the collaborative understanding ability of touch and vision.

[0018] Furthermore, the specific process of step S2 is: using a projector to map the embedding of the visual and tactile encoders and the language embedding to the input space of the language model to achieve cross-modal alignment of multimodal features, and the projector uses a linear transformation method to convert the visual and tactile feature embeddings into a format compatible with the language model input space to ensure a unified representation of multimodal information.

[0019] Furthermore, the projector uses a cross-attention mechanism to map the embedding and language embedding of the visual and tactile encoders to the input space of the language model, achieving cross-modal feature alignment and ensuring efficient fusion of object and environmental information.

[0020] Furthermore, the step S3 specifically inputs the multimodal fusion features into the large model to complete current object detection and recognition, scene understanding, instance segmentation and object attribute recognition.

[0021] Furthermore, step S4 includes the following process:

[0022] S41, combining the large language model backbone Llama2, generating an action plan for the current operation state through step-by-step prediction, the large language model is a pre-trained language model based on the Transformer structure, combining historical tasks, environmental feedback and tactile signals to generate an action plan, the action plan adopts an iterative method, through the "thinking-decision-making" planning method, each step generates the next action and evaluates its effect;

[0023] S42. During the operation, a multimodal fusion model is used to dynamically predict the grasping and placement points, control the robotic arm to perform operations and update the operation status. The multimodal fusion model adopts a cross-modal alignment method to fuse visual, tactile and language information, generate grasping and placement points related to the operation status, control the motion trajectory and strength of the robotic arm, continuously optimize the interaction behavior with the object according to environmental feedback and tactile signals, adjust the grasping strength and posture, and avoid damage or excessive deformation of the object.

[0024] Furthermore, in step S41, the large language model Llama2 is combined with a multimodal shared memory module to perform natural language understanding and action planning, and a time series knowledge base is formed by recording the historical visual, tactile and language features of the task. In the action planning process, the historical features of the relevant tasks are retrieved based on the memory module to assist the current state reasoning and improve the interaction accuracy and safety;

[0025] The large language model Llama2 is fine-tuned in combination with low-rank adaptation technology (LoRA). The LoRA fine-tuning of Llama2 adds low-rank matrix adjustment to the weight W of the large model:

[0026]

[0027] Where W is the original weight matrix, W′ is the fine-tuned weight matrix, A and B are low-rank matrices used for parameter optimization, r is the dimension of the low-rank matrix, satisfying r\ll d, and d is the dimension of the weight matrix;

[0028] The action plan is:

[0029] a t+1 = argmax a∈A π(a|H t ,E t ,F t )

[0030] Among them, a {t+1} is the next action, π is the action strategy function, H t is the current task history, E t is the environmental feedback, F t For tactile signals;

[0031] The update calculation formula of the dynamic programming is:

[0032] π(a t+1 )=π(a t )+Δπ(a t )

[0033] Among them, π(a t ) is the current action strategy, Δπ(a t ) is the action strategy update amount.

[0034] Furthermore, the prediction formula for the grab and place points in step S42 is:

[0035] G,P=Predict(V,T,L)

[0036] Among them, G is the predicted grasping point, P is the predicted placement point, V, T, L are visual, tactile and language embeddings respectively;

[0037] The formula for adjusting the grasping force and posture is:

[0038] F g =Adaptive(F t ),θ=Pose(F t )

[0039] Among them, F g is the adjusted grasping force, θ is the adjusted robot arm posture, F t It is a tactile feedback signal.

[0040] Furthermore, when repeating steps S1 to S4, step S5 is specifically based on operation history analysis and strategy update of the temporal graph neural network (TGN), based on time feature modeling, using TGN to perform time series modeling of the operation history, dynamically capturing the law of change of operation status over time, and based on decision optimization driven by environmental feedback, combining TGN and tactile feedback signals, predicting the probability of task completion and adjusting the operation strategy in real time, giving priority to executing action plans with high success rates.

[0041] Compared with the prior art, the present invention has the following advantages:

[0042] Multimodal fusion: Traditional methods usually use vision, touch and language as independent inputs, while the present invention maps visual, tactile and language features into a unified embedding space and uses a cross-attention mechanism for cross-modal alignment, ensuring efficient fusion of information between modalities. This multimodal fusion capability not only improves the perception of complex attributes such as object shape, texture, hardness, etc., but also enhances the system's adaptability in multi-tasks and complex environments.

[0043] Adaptive action planning: Traditional action planning methods are often based on fixed rules or preset models, lacking a dynamic adjustment mechanism for real-time environmental feedback. However, the present invention combines the fine-tuned Llama2 large language model with the feedback of tactile signals, and adopts a "thinking-decision" iterative planning method, which can evaluate and adjust the plan in real time after each action is generated, thereby improving the flexibility and safety of the interaction process.

[0044] Enhanced tactile understanding: Traditional tactile perception is more than simple tactile sensor data, but lacks the ability to deeply understand tactile signals. This invention adopts a dual-tower Transformer architecture to model the spatial features and temporal relationships of tactile data respectively, and extracts deep tactile features through a multi-head self-attention mechanism, so as to more accurately capture the shape, hardness and other physical properties of objects, effectively preventing the risk of object damage or excessive deformation during operation.

[0045] Real-time dynamic adjustment: Traditional object grasping methods often ignore the adjustment of real-time feedback. However, the present invention uses a multimodal fusion model to dynamically predict the grasping and placement points during the operation, control the robot arm to perform operations in real time, and continuously optimize the interaction behavior with the object based on environmental feedback and tactile signals. This feature enables the present invention to make instant adjustments based on real-time environmental changes, improving the mission success rate and operation accuracy.

[0046] Enhanced task history understanding and optimization: Traditional methods lack the ability to deeply learn and memorize historical tasks and environmental changes when dealing with complex tasks. This invention introduces a multimodal shared memory module to record the visual, tactile and language features of historical tasks, and combines Temporal TGN for time series modeling to analyze the operation history and optimize the strategy. This enables real-time adjustment of the operation strategy based on historical experience and environmental feedback, improving the accuracy and success rate of task execution. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 It is a schematic diagram of the method flow of the present invention;

[0048] Figure 2 is a schematic diagram of an application framework of an embodiment;

[0049] Figure 3 It is a schematic diagram of the action planning process in the embodiment;

[0050] Figure 4 Schematic diagram of repeated execution and update operation strategy in the embodiment. DETAILED DESCRIPTION

[0051] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0052] Example

[0053] like Figure 1 As shown, a method for controlling interactive operation of a deformable object based on a visual-touch-language-action multimodal model comprises the following steps:

[0054] S1. Encode the image, tactile and language data of the deformable object to obtain visual features, tactile features and language features;

[0055] S2, perform cross-modal feature alignment processing on visual features, tactile features and language features to obtain multi-modal fusion features;

[0056] S3, input multimodal fusion features into the large model for environmental understanding,

[0057] Specifically include: object detection and recognition, through object detection and recognition of objects and obstacles in the environment; scene understanding, through natural language description recognition and understanding of environmental semantics, complete instance segmentation; object attribute recognition, based on tactile data, recognize the shape, texture and hardness characteristics of objects;

[0058] S4, adopt the “thinking-decision-making” planning method to iteratively plan and execute actions;

[0059] S5. Repeat steps S1 to S4 until the current deformable object interactive operation task is completed.

[0060] This embodiment applies the above solution to build Figure 2 The application framework shown mainly includes:

[0061] 1. The visual image and tactile data of the deformable object are respectively mapped into image embedding and tactile embedding through the visual and tactile encoders, and the language instructions are mapped into language embedding, so as to extract visual, tactile and language features. In this embodiment, the visual encoder and the tactile encoder both adopt a multi-layer Transformer structure, and extract image features and tactile features through the self-attention mechanism. The language encoder uses the Transformer model to map the language instructions into language embedding; this process can extract the visual features and tactile features of the object and capture its key information such as shape, texture, hardness, etc.

[0062] Specifically, visual feature extraction uses the standard patch embedding process of Vision Transformer (ViT) to divide the input image I into N patches of fixed size.

[0063] P i =PatchEmbed(I), i=1,…,N (1)

[0064] I: input visual image, representing the two-dimensional pixel data of the current deformable object, N: the number of patches divided into images, P i : The embedding vector of the i-th Patch, which represents the feature vector after linear mapping.

[0065] Use a multi-head self-attention mechanism to encode patch embeddings.

[0066]

[0067] Q: query vector (Query), generated by Patch embedding through linear transformation, K: key vector (Key), generated by Patch embedding through linear transformation, V: value vector (Value), generated by Patch embedding through linear transformation, d k : The dimension of the key vector, indicating the size of the feature space.

[0068] Output global feature embedding V,

[0069] V = TransformerEncoder(P) (3)

[0070] V: visual global feature embedding, representing the comprehensive visual features of the image, P: the set of all patch embeddings.

[0071] Tactile feature extraction is to extract spatial features T from tactile data T. s and time feature T t ,

[0072] T s =Conv(T), T t = TransformerEncoder(T) (4)

[0073] T: input tactile signal, which represents the raw data collected by the tactile sensor, T s : Tactile spatial features extracted by convolution operation, T t : Tactile temporal features extracted by Transformer.

[0074] The two features are concatenated and passed through the fusion layer.

[0075] T=Concat(T s ,T t ) (5)

[0076] T: tactile comprehensive feature embedding, Concat: feature concatenation operation.

[0077] Language feature extraction, using the Transformer model to encode the language instruction C into an embedding L,

[0078] L = Transformer (C) (6)

[0079] C: input natural language instruction, L: language feature embedding, which represents the language semantic vector after Transformer encoding.

[0080] Second, a projector is used to map the embeddings of the visual and tactile encoders and the language embeddings to the input space of the language model to achieve cross-modal alignment of multimodal features. In this embodiment, the projector uses a linear transformation method to convert the visual and tactile feature embeddings into a format compatible with the input space of the language model to ensure a unified representation of multimodal information.

[0081] Specifically, the cross-attention mechanism is used to align the visual embedding V, tactile embedding T and language embedding L to ensure the efficient fusion of object and environment information.

[0082] M=CrossAttention(L,V,T) (7)

[0083] M: unified feature embedding after cross-modal alignment, L: language embedding, V: visual embedding, T: tactile embedding.

[0084]

[0085] The unified feature representation of the output is,

[0086] M=Fusion(L,V,T) (9)

[0087] 3. Combined with the large language model backbone Llama2, an action plan for the current operation state is generated through step-by-step prediction. In this embodiment, the large language model is a pre-trained language model based on the Transformer structure, which combines historical tasks, environmental feedback, and tactile signals to generate action plans. The action planning adopts an iterative method through the "thinking-decision-making" planning method (such as Figure 3 As shown), each step generates the next action and evaluates its effect;

[0088] Llama2's LoRA fine-tuning adds low-rank matrix adjustments to the weights W of the large model.

[0089]

[0090] W: original weight matrix, W′: fine-tuned weight matrix, A and B: low-rank matrices for parameter optimization, r: the dimension of the low-rank matrix, satisfying r\ll d, d: the dimension of the weight matrix.

[0091] Action plan generation,

[0092] a t+1 = argmax a∈A π(a|H t ,E t ,F t ) (11)

[0093] a {t+1} : next action, π: action strategy function, H t : Current task history, E t : Environmental feedback, F t : Tactile signal.

[0094] Dynamic planning updates,

[0095] π(a t+1 )=π(a t )+Δπ(a t ) (12)

[0096] π(a t ): current action strategy, Δπ(a t ): Action strategy update amount.

[0097] 4. During the operation, the multimodal fusion model is used to dynamically predict the grasping and placing points, control the robot arm to perform the operation and update the operation status. The multimodal fusion model adopts a cross-modal alignment method to fuse visual, tactile and language information, generate grasping and placing points related to the operation status, control the movement trajectory and strength of the robot arm, and continuously optimize the interaction behavior with the object according to environmental feedback and tactile signals to avoid damage or excessive deformation of the object;

[0098] Pick and place point prediction,

[0099] G,P=Predict(V,T,L) (13)

[0100] G: grasp point prediction, P: placement point prediction, V, T, L: visual, tactile, and language embeddings.

[0101] Real-time adjustment,

[0102] F g =Adaptive(F t ), θ=Pose(F t ) (14)

[0103] F g : Adjusted grasping force, θ: Adjusted robot arm posture, F t : Tactile feedback signal.

[0104] 5. Repeat steps 1 to 4 above until the task is completed, wherein the operation status is updated after each feedback, and the operation strategy is adjusted according to the new feedback.

[0105] Temporal Graph Neural Network (TGN) models operation history,

[0106] H t =TGN(H t-1 ,E t ,F t ) (15)

[0107] H t : Current operation history characteristics, H {t-1} : The operation history of the previous step, E t : Current environment feedback, F t :

[0108] The current haptic signal.

[0109] Decision optimization,

[0110] π′=argmax π P success (H t ,F t ,E t) (16)

[0111] π′: optimized strategy, P success : The probability function of task success.

[0112] In the above process one, both the visual encoder and the tactile encoder adopt a multi-layer Transformer structure, which is used to extract the visual features and tactile features of the object respectively. The visual encoder adopts the standard Vision Transformer (ViT) structure, divides the input image into patches of fixed size, converts them into embedding vectors through linear mapping, and inputs the Transformer encoder with a multi-layer multi-head self-attention mechanism to extract global and local features. The tactile encoder adopts a dual-tower Transformer architecture, one tower is used to model the spatial features of tactile data, and the other tower is used to capture the temporal relationship of tactile signals. The deep representation of tactile signals is achieved through the multi-head self-attention mechanism, and combined with cross-modal feature fusion to enhance the collaborative understanding ability of touch and vision.

[0113] In the above process 2, the projector uses a cross-attention mechanism to map the embedding and language embedding of the visual and tactile encoders to the input space of the language model, achieving cross-modal feature alignment and ensuring efficient fusion of object and environmental information.

[0114] In the above process three, the large language model Llama2 is fine-tuned in combination with low-rank adaptation technology (LoRA) to optimize its action plan generation capability in the interactive scene of deformable objects. By integrating historical tasks, environmental feedback, and tactile signals, Llama2 infers the current state of the object and generates a specific interactive action plan.

[0115] It should be noted that the fine-tuning of the large language model Llama2 adopts an iterative planning method of "thinking-decision-making", which gradually generates an action plan for interacting with objects and combines tactile feedback for judgment and adjustment in each step.

[0116] The fine-tuned Llama2 is used as the core of natural language understanding and action planning. By combining the multimodal shared memory module, the visual, tactile and language features involved in the task are recorded to build a time series knowledge base. In the action planning stage, the model can retrieve historical task-related features through the memory module to provide support for the reasoning of the current state, improve the interaction accuracy and system security.

[0117] In the above process four, the multimodal fusion model dynamically predicts the grasping and placement points, controls the robotic arm in real time to perform interactive operations with the object, and adjusts the grasping force and posture according to tactile feedback to ensure safe operation of the object.

[0118] In the above process 5, if Figure 4 As shown in the figure, the model uses a temporal graph neural network (TGN) to analyze the operation history and update the strategy. TGN models the time characteristics of the operation history and dynamically captures the law of the change of the operation state over time. On this basis, the system combines environmental feedback to optimize the decision, predicts the probability of task completion through TGN and tactile signals, adjusts the operation strategy in real time, and prioritizes the action plan with a high success rate.

[0119] In summary, this solution combines visual, tactile and language information, based on the Transformer architecture, low-rank adaptation (LoRA) fine-tuning and temporal graph neural network (TGN), to improve the multimodal feature alignment capability, action planning accuracy and task adaptability, and improve the generalization ability of deformable object interaction. It is suitable for deformable object interaction tasks in various complex scenarios, and can dynamically adjust the operation strategy to achieve more intelligent and precise deformable object operation.

Claims

1. A method for interactive operation control of a deformable object based on a visual-touch-language-action multimodal model, characterized in that: The following steps are involved: S1. Encode the image, tactile and language data of the deformable object to obtain visual features, tactile features and language features; S2, perform cross-modal feature alignment processing on visual features, tactile features and language features to obtain multi-modal fusion features; S3, input multimodal fusion features into the large model for environmental understanding; S4, adopt the "thinking-decision" planning method to iteratively plan and execute actions; S5. Repeat steps S1 to S4 until the current deformable object interactive operation task is completed.

2. The method for controlling interactive operation of a deformable object based on a visual-touch-language-action multimodal model according to claim 1, characterized in that: The specific process of step S1 is: mapping the visual image and tactile data of the deformable object into image embedding and tactile embedding respectively through the visual and tactile encoders, and mapping the language instructions into language embedding, so as to extract visual, tactile and language features.

3. The method for controlling interactive operation of a deformable object based on a visual-touch-language-action multimodal model according to claim 2, characterized in that: The visual encoder and tactile encoder both adopt a multi-layer Transformer structure, which are used to extract the visual features and tactile features of the object respectively. The visual encoder adopts a standard Vision Transformer structure, divides the input image into patches of fixed size, converts them into embedded vectors through linear mapping, and inputs them into the Transformer encoder with a multi-layer multi-head self-attention mechanism to extract global and local features; The tactile encoder adopts a dual-tower Transformer architecture, one tower is used to model the spatial features of tactile data, and the other tower is used to capture the temporal relationship of tactile signals. It achieves deep representation of tactile signals through a multi-head self-attention mechanism and combines cross-modal feature fusion to enhance the collaborative understanding ability of touch and vision.

4. The method for controlling interactive operation of a deformable object based on a visual-touch-language-action multimodal model according to claim 2, characterized in that: The specific process of step S2 is: using a projector to map the embedding of the visual and tactile encoders and the language embedding to the input space of the language model to achieve cross-modal alignment of multimodal features. The projector uses a linear transformation method to convert the visual and tactile feature embeddings into a format compatible with the language model input space to ensure a unified representation of multimodal information.

5. The method for controlling interactive operation of a deformable object based on a visual-touch-language-action multimodal model according to claim 1, characterized in that: The projector uses a cross-attention mechanism to map the embeddings of the visual and tactile encoders and the language embedding to the input space of the language model, achieving cross-modal feature alignment and ensuring efficient fusion of object and environment information.

6. The method for controlling interactive operation of a deformable object based on a visual-touch-language-action multimodal model according to claim 1, characterized in that: The step S3 specifically inputs the multimodal fusion features into the large model to complete current object detection and recognition, scene understanding, instance segmentation and object attribute recognition.

7. The method for controlling interactive operation of a deformable object based on a visual-touch-language-action multimodal model according to claim 1, characterized in that: Step S4 include The following process: S41, combining the large language model backbone Llama2, through step-by-step prediction, generating an action plan for the current operation state, the large language model is a pre-trained language model based on the Transformer structure, combining historical tasks, environmental feedback and tactile signals to generate an action plan, the action plan adopts an iterative method, through the "thinking-decision" planning method, each step generates the next action and evaluates its effect; S42. During the operation, a multimodal fusion model is used to dynamically predict the grasping and placement points, control the robotic arm to perform operations and update the operation status. The multimodal fusion model adopts a cross-modal alignment method to fuse visual, tactile and language information, generate grasping and placement points related to the operation status, control the motion trajectory and strength of the robotic arm, continuously optimize the interaction behavior with the object according to environmental feedback and tactile signals, adjust the grasping strength and posture, and avoid damage or excessive deformation of the object.

8. The method for controlling interactive operation of a deformable object based on a visual-touch-language-action multimodal model according to claim 7, characterized in that: In step S41, the large language model Llama2 is combined with a multimodal shared memory module to perform natural language understanding and action planning, and a time series knowledge base is formed by recording the historical visual, tactile and language features of the task. In the action planning process, the historical features of the relevant tasks are retrieved based on the memory module to assist the current state reasoning and improve the interaction accuracy and safety; The large language model Llama2 is fine-tuned in combination with the low-rank adaptation technology LoRA. The LoRA fine-tuning of Llama2 adds low-rank matrix adjustment to the weight W of the large model: Where W is the original weight matrix, W′ is the fine-tuned weight matrix, A and B are low-rank matrices used for parameter optimization, r is the dimension of the low-rank matrix, satisfying r\ll d, and d is the dimension of the weight matrix; The action plan is: a t+1 =argmax a∈A π(a|H t ,E t ,F t ) Among them, a {t+1} is the next action, π is the action strategy function, H t is the current task history, E t is the environmental feedback, F t For tactile signals; The update calculation formula of the dynamic programming is: π(a t+1 )=π(a t )+Δπ(a t ) Among them, π(a t ) is the current action strategy, Δπ(a t ) is the action strategy update amount.

9. The method for controlling interactive operation of a deformable object based on a visual-touch-language-action multimodal model according to claim 7, characterized in that: The prediction formula for the grab and place points in step S42 is: G, P = Predict(V, T, L) Among them, G is the predicted grasping point, P is the predicted placement point, V, T, and L are visual, tactile, and language embeddings respectively; The formula for adjusting the grasping force and posture is: F g =Adaptive(F t ),θ=Pose(F t ) Among them, F g is the adjusted grasping force, θ is the adjusted robot arm posture, F t It is a tactile feedback signal.

10. The method for controlling interactive operation of a deformable object based on a visual-touch-language-action multimodal model according to claim 1, characterized in that: When repeating steps S1 to S4, step S5 specifically analyzes the operation history and updates the strategy based on the time graph neural network TGN, models the operation history based on time features, uses TGN to perform time series modeling on the operation history, dynamically captures the law of operation state changes over time, and optimizes the decision-making driven by environmental feedback. It combines TGN and tactile feedback signals to predict the probability of task completion and adjust the operation strategy in real time, giving priority to executing action plans with high success rates.

Images