A flexible cloth robust operation method and system based on optical flow enhancement

Abstract

The application belongs to the technical field of robot flexible object operation, and specifically discloses a flexible cloth robust operation method and system based on optical flow enhancement, which comprises the following steps: constructing a robust benchmark for flexible cloth operation, taking the robust benchmark as a unified anchor point, and calculating a benchmark-dependent two-dimensional optical flow field; performing three-channel dynamic geometric coding on the optical flow field and zero-channel splicing data to obtain pose-invariant geometric features related to operation intention; taking the pose-invariant geometric features as a reference benchmark to obtain fusion features strongly related to operation intention; splicing the fusion features, joint state features of a robot body and task instructions with position coding, and using an intention-action mapping model based on Transformer to output continuous robot action sequences aligned with the operation intention of the flexible cloth. The application eliminates the information aliasing problem caused by internal self-observation uncertainty, and significantly improves the decision robustness in a complex interference scene.

Description

Technical Field

[0001] This invention relates to the field of robotic flexible object manipulation technology, and in particular to a robust manipulation method and system for flexible fabric based on optical flow enhancement. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] Manipulating flexible fabrics is a core task for robots in scenarios such as home services and industrial textiles, with typical scenarios including folding, hanging, and organizing clothing. However, the characteristics of flexible fabrics, such as an infinite continuous state space and high degrees of freedom of deformation, place extremely high demands on the robot's perception accuracy, decision-making robustness, and environmental adaptability.

[0004] Imitation learning is a technology that enables robots to learn to perform various tasks by observing and imitating the teaching behaviors of experts, just like humans. It has become the mainstream paradigm for robots to learn complex operational skills. End-to-end frameworks, represented by ACT (Action Chunking with Transformers) and Diffusion Policy (a policy based on a diffusion model), have achieved excellent operational performance in structured laboratory environments.

[0005] However, when robots are deployed in unstructured open-world scenarios, they face coupled dual-perceptual uncertainties: first, external target state uncertainties, including random changes in cloth pose, background interference, and lighting fluctuations, leading to a mismatch in target perception distribution between the training and deployment phases; second, internal self-observation uncertainties, including partial camera occlusion, sensor noise, and viewpoint shifts, resulting in degraded self-observation data quality. These coupled dual-perceptual uncertainties collectively cause severe perception distribution shifts, disrupting the robot's learned state-operation intention mapping relationship, ultimately leading to a catastrophic decline in the performance of flexible cloth manipulation tasks.

[0006] Mainstream imitation learning frameworks lack a unified, invariant reference benchmark and rely excessively on visual features specific to the training scene, making them prone to representation drift in out-of-distribution (OOD) scenarios. For flexible cloth with complex deformation and an infinite state space, existing methods cannot extract invariant features bound to the operation intent, resulting in extremely poor generalization ability. Furthermore, existing imitation methods are mostly designed for operations on rigid objects and lack targeted modeling for the geometric deformation of flexible cloth. They cannot cope with the perceptual uncertainty brought about by the large-scale deformation of cloth, resulting in low success rate of cloth operation tasks in open scenes and difficulty in achieving large-scale application. Summary of the Invention

[0007] To address the aforementioned issues, this invention proposes a robust manipulation method and system for flexible cloth based on optical flow enhancement. Using the standardized pose of the flexible cloth as an anchoring reference, dynamic geometric representation learning enhanced by optical flow suppresses perceptual representation drift at its source. Combined with a dual-stream adaptive gating fusion module, dynamic and reliable filtering and fusion of multi-source perceptual information are achieved. A stable intention-action mapping is constructed through a Transformer architecture, significantly improving the robustness and task success rate of the robot's flexible cloth manipulation in open-world (OOD) scenarios.

[0008] In some implementations, the following technical solutions are adopted: A robust manipulation method for flexible fabrics based on optical flow enhancement includes: To construct a robust benchmark for flexible fabric manipulation, we acquire multi-view camera images of the robot on the flexible fabric in the current scene, as well as the joint state features of the robot body. Using a robust benchmark as a unified anchor point, the benchmark-dependent two-dimensional optical flow field between the current input image and the benchmark image is calculated; a Z-axis zero channel is introduced, and the stitched data of the optical flow field and the zero channel is subjected to three-channel dynamic geometric encoding to obtain pose-invariant geometric features related to the operation intention. Using the pose-invariant geometric features as a reference, the original image features of the multi-view cameras are adaptively weighted for effectiveness through camera information flow gating branches to obtain gated effective features for all cameras; the pose-invariant geometric features are weighted and optimized through optical flow mode gating branches to obtain optical flow gating effective features. The effective gating features and optical flow gating features of all cameras are fused to obtain fused features that are strongly correlated with the operation intention; The fusion features, joint state features of the robot body, task instruction embedding and position encoding are spliced ​​together, and a continuous robot action sequence aligned with the flexible cloth operation intention is output using a Transformer-based intention-action mapping model. The robot is controlled to perform corresponding flexible fabric operations based on the action sequence.

[0009] As a further option, the robustness benchmark is a key structural feature image of the flexible fabric in a standard pose.

[0010] As a further approach, a robust benchmark is used as a unified anchor point to calculate the benchmark-dependent two-dimensional optical flow field between the current input image and the benchmark image. Specifically, the two-dimensional optical flow field between the current input image and the benchmark image is calculated using the RAFT algorithm to encode the pose offset and motion trend of the flexible fabric relative to the benchmark.

[0011] As a further solution, a Z-axis zero channel is introduced to perform three-channel dynamic geometric encoding on the stitched data of the optical flow field and the zero channel, specifically: Feature extraction is performed on the spliced ​​data of optical flow field and zero channel using a pre-trained ResNet network, and the output is a three-channel pose-invariant geometric feature related to the operation intention.

[0012] As a further approach, the original image features of the multi-view cameras are adaptively weighted for effectiveness through camera information flow gating branches to obtain the gated effective features of all cameras, specifically: For each camera, a corresponding gating branch is constructed. First, the visual features of the input image of each camera are extracted through the ResNet18 network. Then, the validity weight of each visual feature is calculated through the corresponding gating branch. The effective camera features after gating are obtained by multiplying the visual features and weights element by element.

[0013] As a further approach, pose-invariant geometric features are weighted and optimized using optical flow mode-gated branches to obtain optical flow-gated effective features, specifically: The pose-invariant geometric features are input into a gated branch consisting of convolution and a sigmoid activation function to obtain the effectiveness weights of the optical flow geometric features; the pose-invariant geometric features are multiplied by the effectiveness weights to obtain the gated effective optical flow features.

[0014] As a further approach, a Transformer-based intention-action mapping model is used to output a continuous sequence of robot actions aligned with the manipulation intentions of the flexible cloth, specifically: Each modality feature undergoes intramodal multi-head self-attention to achieve feature purification, denoising, and consistency enhancement; after being normalized by the LayerNorm layer, it enters intermodal multi-head cross-attention to achieve cross-modal global semantic alignment; after being normalized again by LayerNorm, it enters the feedforward network to complete feature transformation. The above process involves multiple layers of Transformers that iterate through intramodal purification and cross-modal alignment, ultimately outputting the mapping result from intent to action.

[0015] In other embodiments, the following technical solutions are adopted: A robust operating system for flexible fabric based on optical flow enhancement includes: The data acquisition module is used to build a robust benchmark for flexible fabric operation, acquire multi-view camera images of the flexible fabric in the current scene, and the joint state features of the robot body. The benchmark construction module is used to calculate the benchmark-dependent two-dimensional optical flow field between the current input image and the benchmark image, using a robust benchmark as a unified anchor point; a Z-axis zero channel is introduced to perform three-channel dynamic geometric encoding on the stitched data of the optical flow field and the zero channel to obtain pose-invariant geometric features related to the operation intention. The dual-stream adaptive gating fusion module is used to adaptively weight the original image features of the multi-view cameras through camera information flow gating branches to obtain gated effective features for all cameras, using the pose-invariant geometric features as a reference. It also performs weighted optimization on the pose-invariant geometric features through optical flow mode gating branches to obtain optical flow gating effective features. Finally, it fuses the gated effective features of all cameras and the optical flow gating effective features to obtain fused features that are strongly correlated with the operation intent. The action sequence prediction module is used to splice the fused features, the joint state features of the robot body, the embedded task instructions and the position encoding, and output a continuous robot action sequence that is aligned with the flexible cloth operation intention using the Transformer-based intention-action mapping model. The execution control module is used to control the robot to perform corresponding flexible fabric operations based on the action sequence.

[0016] In other embodiments, the following technical solutions are adopted: A robot includes: a robot body and a multi-view camera, a flexible fabric manipulation actuator, a controller, and a computer program stored on and executed by the controller, all mounted on the robot body; when the controller executes the computer program, it is able to implement the optical flow enhancement-based robust manipulation method for flexible fabric as described in any one of claims 1-7.

[0017] In other embodiments, the following technical solutions are adopted: A computer-readable storage medium storing a plurality of instructions adapted for loading and execution by a processor of a terminal device of the above-described optical flow-enhanced robust fabric handling method.

[0018] Compared with the prior art, the beneficial effects of the present invention are: This invention addresses the characteristics of flexible fabrics, such as high deformation and infinite state space, by constructing a standardized pose robustness benchmark bound to the operational intent. Through benchmark-dependent optical flow encoding, pose-invariant geometric features unaffected by pose, background, and illumination are extracted. This suppresses representation drift caused by uncertainty in the external target state in open scenes from the source, solving the core problem of poor generalization ability of existing methods for flexible fabric deformation.

[0019] This invention designs a dual-stream adaptive gating fusion module, which uses the invariant geometric features anchored to the benchmark as a reliability reference to achieve dynamic adaptive filtering and fusion of multi-view visual information and optical flow geometric information. It can accurately identify and shield invalid information caused by camera occlusion and sensor noise, eliminate the information aliasing problem caused by internal self-observation uncertainty, and significantly improve the decision robustness in complex interference scenarios.

[0020] Other features and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0021] Figure 1 This is a flowchart of a robust operation method for flexible fabric based on optical flow enhancement in an embodiment of the present invention; Figure 2 This is a schematic diagram of the process for extracting pose-invariant geometric features of flexible fabric in an embodiment of the present invention; Figure 3 This is a schematic diagram of the dual-stream adaptive gating fusion architecture in an embodiment of the present invention; Figure 4 This is a schematic diagram of the intent-action mapping model architecture based on Transformer in an embodiment of the present invention. Detailed Implementation

[0022] It should be noted that the following detailed description is illustrative and intended to provide further explanation of the invention. Unless otherwise specified, all technical and scientific terms used in this invention have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0023] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of exemplary embodiments according to the invention. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0024] Example 1 In one or more embodiments, a robust manipulation method for flexible fabrics based on optical flow enhancement is disclosed, combined with... Figure 1 Specifically, it includes the following process: S101: Construct a robust benchmark for flexible fabric manipulation, acquire multi-view camera images of the robot on the flexible fabric in the current scene, and the joint state features of the robot body.

[0025] In this embodiment, the robustness benchmark is the RGB image of the key structural features of the flexible fabric in a flat, unfolded, and standardized pose, corresponding to the initial standard state of the fabric folding and suspension task. This benchmark strictly satisfies the strong correlation between feature integrity and operational intent, is unaffected by external background, lighting, changes in fabric appearance, or internal camera occlusion and sensor noise, and is always closely bound to the core operational intent of fabric folding and suspension; its core function is to achieve anchoring calibration, converting dynamic multi-source input information into an invariant description relative to the benchmark, severing the correlation between representation and scene distribution, and directly suppressing representation drift.

[0026] Among them, those that strongly correlate with operational intent include: Anchoring the starting point of the intention: The baseline is the standard initial flat state of the folding and hanging task, which is itself a concrete anchor point of the operational intention; Strong feature binding: Only retain the key structural features of the cloth necessary for operation, and eliminate all interference information that is irrelevant to the operation; Locking in the direction of intent: Suppressing representation drift through benchmark anchoring calibration to ensure that inputs always revolve around the core operational intent.

[0027] In this embodiment, the robot is equipped with three synchronous RGB cameras, including two wrist-mounted hand-eye cameras and one fixed global camera; it acquires RGB images of the flexible fabric in the current scene captured by the robot's multi-view cameras, and simultaneously acquires the 6-DOF joint state information of the robot body (angle information of the 6 joints of each of the two robotic arms).

[0028] S102: Using a robust benchmark as a unified anchor point, calculate the benchmark-dependent two-dimensional optical flow field between the current input image and the benchmark image; introduce a Z-axis zero channel, and perform three-channel dynamic geometric encoding on the stitched data of the optical flow field and the zero channel to obtain pose-invariant geometric features related to the operation intention.

[0029] In this embodiment, the reference-dependent optical flow field is first calculated: using a robust reference as a unified anchor point, the two-dimensional optical flow field between the current input image I and the robust reference B is calculated using the RAFT algorithm, thereby encoding the pose offset and motion trend of the flexible fabric relative to the reference. The specific formula is as follows: (1) in, , These are the height and width of the image, respectively. In this embodiment, we take... =480, =640.

[0030] Then, three-channel dynamic geometric encoding is performed: a zero channel along the Z-axis is introduced, and features are extracted from the concatenated data of the optical flow field and the zero channel using a pre-trained ResNet network (residual network). This yields three-channel pose-invariant geometric features related to the operational intent, as shown in the formula: (2) in, For the Z-axis zero channel, Enhance the geometric features of the final output optical flow.

[0031] This process uses standardized benchmarks as a fixed reference, effectively filtering out invalid flow field information caused by internal and external interference, and ensuring a strong correlation between the encoding results and the intention of the fabric operation.

[0032] In this embodiment, the two-dimensional optical flow field (x, y directions) is expanded into three channels by introducing a zero channel along the Z-axis, which is adapted to the network structure of ResNet. At the same time, the spatial dimension of the geometric encoding is aligned structurally, which is the basis for the implicit spatial correlation of subsequent feature extraction.

[0033] ResNet network is selected to extract features from the spliced ​​data of optical flow field and zero channel. The pre-trained weights of ResNet network in visual tasks can be transferred to learn, which can accelerate convergence and generalize well to dynamic optical flow features. The convolutional structure can effectively capture the correlation between local motion patterns and global deformation of optical flow, which fits the dynamic geometric requirements of cloth operation. At the same time, it can solve the gradient vanishing problem of deep network and extract deep abstract motion features of optical flow field.

[0034] This embodiment uses three-channel pose-invariant geometric features as the core input, combined with reference anchoring, to convert dynamic multi-source inputs into invariant descriptions relative to the reference, severing the association with scene distribution, allowing the model to focus on the operation intention itself, and providing subsequent intention-action mapping with geometric representations that are only related to the operation intention and are not affected by pose changes, thereby improving the stability of recognition and decision-making for the same operation under different lighting and backgrounds.

[0035] S103: Using pose-invariant geometric features as a reference, the original image features of the multi-view cameras are adaptively weighted for effectiveness through the camera information flow gating branch to obtain gated effective features for all cameras; the pose-invariant geometric features are weighted and optimized through the optical flow mode gating branch to obtain optical flow gated effective features.

[0036] In this embodiment, using the invariant geometric features obtained in step S102 as a reference, adaptive filtering and fusion are performed on the multi-view visual features and optical flow geometric features respectively, combined with... Figure 3 The specific process is as follows: (1) Camera information flow gating branch: We constructed dedicated gating branches for each of the three cameras, with each branch consisting of a 1×1 convolutional layer and a Sigmoid activation function.

[0037] First, visual features of each camera input image are extracted using a ResNet18 network. Let i ∈ {1, 2, 3} be the camera index; then, the effectiveness weight of each visual feature is calculated using a 1×1 convolution and a sigmoid activation function, as shown in the formula: (3) in, A value close to 1 indicates that the corresponding feature is reliable, while a value close to 0 indicates that the feature is invalid due to occlusion or noise interference. Invalid visual information is filtered out by element-wise multiplication of the features and weights, resulting in gated valid camera features. .

[0038] (2) Optical flow mode-gated branch: A 1×1 convolutional layer and sigmoid activation function structure, originating from the camera branch, are used to adapt to the invariant properties of optical flow features. The effectiveness weights of the optical flow geometric features are calculated using the following formula: (4) This weight is used to quantify the reliability of the optical flow field, identify and filter invalid optical flow values ​​caused by background interference and motion blur, and purify the optical flow features by element-wise multiplication to obtain the gated effective optical flow features. .

[0039] S104: The effective gating features and optical flow gating features of all cameras are fused to obtain fused features that are strongly correlated with the operation intention.

[0040] In this embodiment, the effective gating features and optical flow gating features of all cameras are concatenated along the channel dimension to obtain the final fused features, as shown in the formula: (5) This fusion feature is strongly correlated with the fabric manipulation intent, filtering out redundant information caused by sensor failure and scene interference.

[0041] In this embodiment, the effective gating features and optical flow gating features of each camera have been preprocessed by the gating module. The gating will automatically assign high weights to "effective information" (such as camera images that clearly capture the fabric and optical flow that conforms to motion logic) and low weights or even directly suppress invalid information (such as occluded camera areas, sensor noise, and falsely detected optical flow). What is stitched together is the purified effective features, not the original data, thus avoiding the mixing of invalid information from the source.

[0042] Furthermore, different cameras have different perspectives. When one camera is blocked, other cameras may capture valid information about the same area. By stitching together the gated valid features of each camera, the influence of a single camera blocking is naturally shielded. At the same time, camera features are spatial or appearance information, while optical flow features are motion or geometric information. By cross-validating the two, the information aliasing of sensor noise and self-observation uncertainty can be eliminated.

[0043] S105: The fusion features, joint state features of the robot body, task instruction embedding and position encoding are spliced ​​together, and a continuous robot action sequence aligned with the flexible cloth operation intention is output using a Transformer-based intention-action mapping model.

[0044] In this embodiment, since the Transformer's self-attention is not sensitive to the order or position of the sequence, the positional encoding is used to assign a positional label to each input feature, so that the model knows which step the feature corresponds to and which one corresponds to the base joint, thereby capturing the temporal dependence of the action and the spatial hierarchy of the joints.

[0045] In this embodiment, combined with Figure 4 First, the multi-source information is concatenated to provide a comprehensive and reliable input foundation for the model, reducing the risk of single-source information failure due to distribution shift. The concatenation formula is as follows: (6) in, It includes learnable style tokens, joint embeddings, and task instruction text embeddings. Among them, the learnable style tokens are a set of fixed-dimensional vector parameters that are continuously optimized and updated during model training with backpropagation. The core of the tokens is to vectorize and encode different action execution strategies, operation paradigms, and motion preferences under the same task objective in flexible cloth operations.

[0046] These are the joint state characteristics of the robot body. It provides sine or cosine position encoding for capturing spatiotemporal correlations.

[0047] Then, a continuous robot motion sequence aligned with the manipulation intent of the flexible cloth is obtained by mapping using a Transformer encoder-decoder network. The Transformer encoder-decoder network adopts a Transformer architecture with 4 layers of encoder, 6 layers of decoder, 8 attention heads, and a hidden layer dimension of 512. The correlation between intramodal and intermodal modes is mined through self-attention layers: intramodal capture enhances the consistency representation of the cloth and the reliability of pose offset, and filters out noise caused by distribution offset; intermodal capture achieves cross-modal semantic alignment through global similarity calculation, without the need to manually define weights, which improves the robustness of fused features. Finally, the autoregressive output is a 15-step continuous robot joint motion sequence aligned with the manipulation intent.

[0048] In a specific implementation, each modal feature undergoes intramodal multi-head self-attention to achieve feature purification, denoising, and consistency enhancement; after normalization by the LayerNorm layer, it enters intermodal multi-head cross-attention to achieve cross-modal global semantic alignment; after normalization by LayerNorm again, it enters the feedforward network to complete feature transformation; the above process iterates through multiple Transformer layers to perform intramodal purification and cross-modal alignment, and finally outputs the mapping result from intent to action.

[0049] In this embodiment, the process of constructing the training dataset for the Transformer-based intent-action mapping model is as follows: (1) Data collection: Core data from real teaching demonstrations; Robots such as Franka Panda collect tens of thousands of human demonstrations through kinesthetic or teleoperation to ensure the ability to be deployed.

[0050] (2) Data standardization processing, including spatiotemporal alignment, action segmentation and normalization; The spatiotemporal alignment process involves hardware-level synchronization of timestamps for multi-camera images, optical flow, and joint states (error < 1ms). The process of action segmentation and normalization is as follows: the continuous teaching action is broken down into fixed lengths, and the action (joint angle) and visual features are normalized by Z-score for stable training.

[0051] (3) Sample tuple construction: This forms a strongly bound sample consisting of "task instructions + style tags + gated multimodal observation + cloth reference image to action ground truth".

[0052] S106: Control the robot to perform the corresponding flexible fabric operation based on the action sequence.

[0053] The robot controller receives and outputs the sequence of actions, and controls the robotic arm to perform corresponding flexible fabric operations, such as folding, hanging, or arranging the fabric.

[0054] During execution, images and joint status are acquired in real time, and steps S101-S104 are executed cyclically to achieve closed-loop control of the operation process.

[0055] This embodiment constructs a closed-loop pipeline from invariant benchmark construction, dynamic feature purification, stable intent to action mapping through the collaborative optimization of benchmark anchored geometric representation and adaptive gating fusion. It can achieve excellent flexible cloth operation performance in various out-of-distribution scenarios without relying on large-scale OOD data training, greatly reducing the model's dependence on training data and possessing strong open-world deployment capabilities and engineering implementation value.

[0056] The method described in this embodiment is not only applicable to typical flexible operation tasks such as fabric hanging and folding, but can also be extended to robot operation scenarios involving other deformable objects. It has good versatility and scalability and can be widely used in many practical fields such as home service, industrial textiles, and warehousing sorting.

[0057] To verify the effectiveness of the method in this embodiment, five progressive OOD scenarios were designed to cover typical interferences in open-world deployments: S1: Single translational offset (uncertainty in the state of the basic external target); S2: Translation-rotation composite offset (severe external pose uncertainty); S3: Pose displacement + background interference (multidimensional external distribution displacement); S4: Translation offset + random camera occlusion (internal and external coupling interference); S5: Translation offset + Gaussian image noise (extreme full-scene distribution offset).

[0058] Using task success rate as the core evaluation metric, each method underwent 100 independent tests per scenario, and was compared with three mainstream baseline methods: ACT, InterACT, and Diffusion Policy. The key results are as follows: Fabric suspension task: The overall average success rate of the method in this embodiment is 31.0%, which exceeds the baseline best ACT method by 12.0 percentage points; Fabric folding task: The overall average success rate of the method in this embodiment is 51.2%, which exceeds the baseline best ACT method by 12.8 percentage points; Average across all scenarios: The overall average success rate of the method in this embodiment is 45.1%, which is 11.4 percentage points higher than the baseline best ACT method. The performance advantage is even more significant in extreme coupled interference scenarios such as occlusion and noise.

[0059] Experimental results demonstrate that the method in this embodiment can effectively address the coupled dual-sensor uncertainty in an open world, significantly improve the robustness and OOD generalization ability of flexible cloth operations, and possess excellent practical deployment performance.

[0060] Example 2 In one or more embodiments, a robust operating system for flexible fabric based on optical flow enhancement is disclosed, specifically including: The data acquisition module is used to build a robust benchmark for flexible fabric operation, acquire multi-view camera images of the flexible fabric in the current scene, and the joint state features of the robot body. The benchmark construction module is used to calculate the benchmark-dependent two-dimensional optical flow field between the current input image and the benchmark image, using a robust benchmark as a unified anchor point; a Z-axis zero channel is introduced to perform three-channel dynamic geometric encoding on the stitched data of the optical flow field and the zero channel to obtain pose-invariant geometric features related to the operation intention. The dual-stream adaptive gating fusion module is used to adaptively weight the original image features of the multi-view cameras through camera information flow gating branches to obtain gated effective features for all cameras, using the pose-invariant geometric features as a reference. It also performs weighted optimization on the pose-invariant geometric features through optical flow mode gating branches to obtain optical flow gating effective features. Finally, it fuses the gated effective features of all cameras and the optical flow gating effective features to obtain fused features that are strongly correlated with the operation intent. The action sequence prediction module is used to splice the fused features, the joint state features of the robot body, the embedded task instructions and the position encoding, and output a continuous robot action sequence that is aligned with the flexible cloth operation intention using the Transformer-based intention-action mapping model. The execution control module is used to control the robot to perform corresponding flexible fabric operations based on the action sequence.

[0061] It should be noted that the specific implementation methods of the above modules are exactly the same as those in Example 1, and will not be repeated here.

[0062] Example 3 In one or more embodiments, a robot is disclosed, comprising: a robot body and a multi-view image acquisition mechanism, a flexible fabric manipulation execution mechanism, a controller mounted on the robot body, and a computer program stored on and executed by the controller; when the controller executes the computer program, it can realize the optical flow enhancement-based flexible fabric robust manipulation method described in Embodiment 1.

[0063] Example 4 In one or more embodiments, a computer-readable storage medium is disclosed, wherein a plurality of instructions are stored, the instructions being adapted to be loaded by a processor of a terminal device and executed by the optical flow-enhanced robust fabric manipulation method described in Embodiment 1.

[0064] While the specific embodiments of the present invention have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of the present invention are still within the scope of protection of the present invention.

Claims

1. A robust manipulation method for flexible fabrics based on optical flow enhancement, characterized in that, include: To construct a robust benchmark for flexible fabric manipulation, we acquire multi-view camera images of the robot on the flexible fabric in the current scene, as well as the joint state features of the robot body. Using a robust benchmark as a unified anchor point, the benchmark-dependent two-dimensional optical flow field between the current input image and the benchmark image is calculated; a Z-axis zero channel is introduced, and the stitched data of the optical flow field and the zero channel is subjected to three-channel dynamic geometric encoding to obtain pose-invariant geometric features related to the operation intention. Using the pose-invariant geometric features as a reference, the original image features of the multi-view cameras are adaptively weighted for effectiveness through camera information flow gating branches to obtain gated effective features for all cameras; the pose-invariant geometric features are weighted and optimized through optical flow mode gating branches to obtain optical flow gating effective features. The effective gating features and optical flow gating features of all cameras are fused to obtain fused features that are strongly correlated with the operation intention; The fusion features, joint state features of the robot body, task instruction embedding and position encoding are spliced ​​together, and a continuous robot action sequence aligned with the flexible cloth operation intention is output using a Transformer-based intention-action mapping model. The robot is controlled to perform corresponding flexible fabric operations based on the action sequence.

2. The robust operation method for flexible fabric based on optical flow enhancement as described in claim 1, characterized in that, The robustness benchmark is the key structural feature image of the flexible fabric in a standard pose.

3. The robust operation method for flexible fabric based on optical flow enhancement as described in claim 1, characterized in that, Using a robust benchmark as a unified anchor point, the benchmark-dependent two-dimensional optical flow field between the current input image and the benchmark image is calculated. Specifically, the two-dimensional optical flow field between the current input image and the benchmark image is calculated using the RAFT algorithm to encode the pose offset and motion trend of the flexible fabric relative to the benchmark.

4. The robust operation method for flexible fabric based on optical flow enhancement as described in claim 1, characterized in that, it introduces... The Z-axis zero channel performs three-channel dynamic geometric encoding on the stitched data of the optical flow field and the zero channel, specifically as follows: Feature extraction is performed on the spliced ​​data of optical flow field and zero channel using a pre-trained ResNet network, and the output is a three-channel pose-invariant geometric feature related to the operation intention.

5. A robust handling method for flexible fabric based on optical flow enhancement as described in claim 1, characterized in that, Adaptive validity weighting is applied to the original image features of the multi-view cameras through camera information stream gating branches to obtain the gated valid features of all cameras, specifically: For each camera, a corresponding gating branch is constructed. First, the visual features of the input image of each camera are extracted through the ResNet18 network. Then, the validity weight of each visual feature is calculated through the corresponding gating branch. The effective camera features after gating are obtained by multiplying the visual features and weights element by element.

6. The robust handling method for flexible fabric based on optical flow enhancement as described in claim 1, characterized in that, By performing weighted optimization of pose-invariant geometric features through optical flow modal gating, effective optical flow-gated features are obtained, specifically: The pose-invariant geometric features are input into a gated branch consisting of convolution and a sigmoid activation function to obtain the effectiveness weights of the optical flow geometric features; the pose-invariant geometric features are multiplied by the effectiveness weights to obtain the gated effective optical flow features.

7. A robust handling method for flexible fabric based on optical flow enhancement as described in claim 1, characterized in that, Using a Transformer-based intent-action mapping model, a continuous sequence of robot actions aligned with the manipulation intent of the flexible cloth is output, specifically: Each modality feature undergoes intramodal multi-head self-attention to achieve feature purification, denoising, and consistency enhancement; after being normalized by the LayerNorm layer, it enters intermodal multi-head cross-attention to achieve cross-modal global semantic alignment; after being normalized again by LayerNorm, it enters the feedforward network to complete feature transformation. The above process involves multiple layers of Transformers that iterate through intramodal purification and cross-modal alignment, ultimately outputting the mapping result from intent to action.

8. A robust operating system for flexible fabric based on optical flow enhancement, characterized in that, include: The data acquisition module is used to build a robust benchmark for flexible fabric operation, acquire multi-view camera images of the flexible fabric in the current scene, and the joint state features of the robot body. The benchmark construction module is used to calculate the benchmark-dependent two-dimensional optical flow field between the current input image and the benchmark image, using a robust benchmark as a unified anchor point; a Z-axis zero channel is introduced to perform three-channel dynamic geometric encoding on the stitched data of the optical flow field and the zero channel to obtain pose-invariant geometric features related to the operation intention. The dual-stream adaptive gating fusion module is used to adaptively weight the original image features of the multi-view cameras through camera information flow gating branches to obtain gated effective features for all cameras, using the pose-invariant geometric features as a reference. It also performs weighted optimization on the pose-invariant geometric features through optical flow mode gating branches to obtain optical flow gating effective features. Finally, it fuses the gated effective features of all cameras and the optical flow gating effective features to obtain fused features that are strongly correlated with the operation intent. The action sequence prediction module is used to splice the fused features, the joint state features of the robot body, the embedded task instructions and the position encoding, and output a continuous robot action sequence that is aligned with the flexible cloth operation intention using the Transformer-based intention-action mapping model. The execution control module is used to control the robot to perform corresponding flexible fabric operations based on the action sequence.

9. A robot, comprising: The robot body, a multi-view camera mounted on the robot body, a flexible fabric manipulation actuator, a controller, and a computer program stored on and executed by the controller; characterized in that, when the controller executes the computer program, it is able to implement the optical flow-enhanced robust manipulation method for flexible fabric as described in any one of claims 1-7.

10. A computer-readable storage medium storing a plurality of instructions, characterized in that, The instructions are adapted to be loaded by the processor of the terminal device and executed by the robust operation method for flexible fabric based on optical flow enhancement as described in any one of claims 1-7.

Images