A vision-tactile fusion control method and system for embodied intelligent robots for biochemical experimental tasks
By constructing a multimodal fusion model, visual and tactile information are aligned and fused for inference to generate reliable robot control commands. This solves the problems of unstable grasping and insufficient visual observability of biochemical experimental robots under complex working conditions, and improves operational robustness and generalization ability.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- HUNAN UNIV
- Filing Date
- 2026-06-08
- Publication Date
- 2026-07-10
AI Technical Summary
Existing biochemical experimental robots suffer from problems such as unstable grasping, slipping when opening lids, and spillage when tipped over in complex working conditions. Furthermore, insufficient visual observability leads to inadequate robustness and generalization ability, and tactile information lacks a systematic connection with the semantics of task steps.
By synchronously acquiring visual frames, language commands, and ontological states, a multimodal fusion model is constructed. The three-axis force of the end-effector tactile dot matrix is converted into a three-channel two-dimensional tactile image and aligned with visual observation. The Transformer model is used for multimodal fusion inference to generate joint space increments and gripper control commands.
It improves the operational robustness and cross-scenario generalization ability of biochemical experiments, and achieves real-time response and fine-grained operation quality for complex contact processes.
Smart Images
Figure CN122353625A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of embodied intelligent robot control technology, and in particular to a visual-tactile fusion control method and system for embodied intelligent robots for biochemical experimental tasks. Background Technology
[0002] Currently, most automated robots for biochemical experiments employ fixed workstations and structured fixtures, relying on pre-set scripts, teaching playback, or traditional motion planning to perform operations such as grasping, opening, inserting / removing, pipetting, mixing, and transferring. However, in real-world biochemical laboratory settings, issues such as glassware transparency and reflectivity, liquid level variations, batch differences in consumables, deformation of flexible packaging, fluctuations in friction coefficients, and slight positional shifts are common. This makes systems relying solely on fixed procedures prone to problems like unstable grasping, alignment failures, slippage when opening lids, and spillage. Furthermore, biochemical tasks have limited margins for error and high requirements for safety and cleanliness. Collisions or spills not only affect results but can also introduce contamination and safety risks. Therefore, the robustness and generalization capabilities of existing methods are still insufficient to support stable operation under complex conditions.
[0003] To enhance adaptability, some solutions introduce visual perception and learning-based policy control, generating actions by recognizing the positions of containers and tools. However, biochemical experiments involve numerous interactions that are "not sufficiently observable visually": transparent containers and liquids produce refraction and highlights, making it difficult to reliably estimate contact boundaries and insertion depths; critical states such as whether clamping is secure, whether micro-slippage occurs, whether there is skew during insertion or removal, and whether tightening is properly engaged often cannot be reliably judged visually alone. Even when using torque or tactile sensors, existing systems often use them for threshold triggering or over-limit protection, lacking a systematic correlation between tactile information and task step semantics, operational stages, and failure modes. Furthermore, tactile data is complex in form and has a high temporal frequency, making integration with vision difficult in terms of temporal synchronization, spatial alignment, and representational unification. This results in visual-tactile fusion often remaining at the engineering splicing level, with unstable performance when transferred to different consumables and tasks. Therefore, there is an urgent need to design a visual-tactile fusion control method for embodied intelligent robots for biochemical experimental tasks, which can express tactile sensation in a structured way and effectively integrate it with vision and task semantics, thereby improving the reliability, robustness and cross-scenario generalization ability of complex biochemical experimental tasks. Summary of the Invention
[0004] This invention provides a visual-tactile fusion control method and system for embodied intelligent robots for biochemical experimental tasks, in order to solve the technical problems mentioned in the background art.
[0005] To achieve the above objectives, the technical solution of the present invention is implemented as follows: This invention provides a visual-tactile fusion control method for embodied intelligent robots for biochemical experimental tasks, comprising the following steps: S1. For biochemical experimental tasks, visual frames, language commands, body states and action sequences are collected simultaneously and aligned with a unified timestamp to form a standard trajectory sample sequence sorted by time. S2. The raw triaxial force readings of the end tactile dot matrix are converted into a three-channel two-dimensional tactile image through a mapping operator; after normalization and temporal smoothing, they are aligned with the nearest neighbor time of visual observation to form a tactile representation sequence. S3. Construct a multimodal fusion model, which includes a shared image encoder, a pre-trained text encoder, a projection module, a stitching module, a fusion inference network, and an action head. The shared image encoder, the pre-trained text encoder, and the projection module are connected in parallel and then connected to the stitching module, the fusion inference network, and the action head in sequence. S4. Combine the standard trajectory sample sequence with the structured tactile representation to obtain the fusion inference observation sequence; use the fusion inference observation sequence to train the multimodal fusion model to obtain the trained multimodal fusion model; S5. Deploy the trained multimodal fusion model to the device, then input the real-time collected fusion inference observation sequence to obtain the action vector, generate the joint space increment and gripper control command for the current cycle based on the action vector, and finally control the embodied intelligent robot based on the joint space increment and gripper control command for the current cycle.
[0006] Furthermore, step S1 specifically includes the following steps: S11. In biochemical experiment tasks, for each complete teaching trajectory, record RGB visual images, robot body state, and task instructions at a fixed sampling frequency, and assign a unified timestamp to each moment. This forms multiple complete teaching trajectory sequences; S12. Summarize multiple complete teaching trajectory sequences to obtain a standard trajectory sample sequence sorted by time.
[0007] Furthermore, the expression for the complete teaching trajectory sequence in S11 is as follows: ; in, Indicates the first m A complete sequence of teaching trajectories; express t RGB visual image at any given time; This indicates the state of the embodied intelligent robot. Indicates task instructions; Indicates a real action tag; T This indicates the total number of time steps in the current teaching trajectory; The physical state of the embodied intelligent robot The expression is as follows: ; in, Joint angle, For joint velocity, n The degrees of freedom of the joints of an embodied intelligent robot; The end position, This represents the end-effector attitude. The grippers are in the open / closed state; Represents the set of real numbers; Realistic Action Tags The expression is as follows: ; in, This represents the increase in joint space. The expression for the standard trajectory sample sequence in S12 is as follows: ; in, Represents a standard trajectory sample sequence; This represents the total number of teaching trajectories. Indicates the first m The teaching trajectory is shown in the text.
[0008] Furthermore, step S2 specifically includes the following steps: S21. Obtain the three-axis force vector of each tactile point through tactile data sampling; S22. Calculate the original tactile observation matrix of the entire frame based on the triaxial force vector of each tactile point, that is, the original triaxial force reading of the end tactile point array; S23. Map all tactile points in the entire frame's original tactile observation matrix to... In the grid positions, a three-channel two-dimensional tactile image is thus constructed; Indicates the height of the three-channel two-dimensional tactile image; Indicates the width of the three-channel two-dimensional tactile image; S24. Perform channel normalization processing on the three-channel two-dimensional tactile image to obtain the normalized three-channel two-dimensional tactile image; S25. An exponential moving average mechanism is introduced to perform temporal smoothing on the normalized three-channel two-dimensional tactile image to obtain a temporally smoothed three-channel two-dimensional tactile image. S26. Use the nearest neighbor alignment strategy to timestamp the visual frames. Adjacent tactile sampling timestamps Pairing is performed to synchronize visual and tactile timestamps; S27. Combine the time-smoothed three-channel two-dimensional tactile image with the corresponding tactile sampling timestamp. Pairing yields tactile representation sequences. tactile representation sequence It includes multiple tactile heat maps.
[0009] Furthermore, the expression for the three-axis force vector of the tactile point in S21 is: ; in, express t At this moment i Three-axis force vectors at each tactile point; They represent t At this moment i The three-axis force vector of each tactile point is x, y, z Three components along the axial direction; The expression for the original tactile observation matrix of the entire frame in S22 is: ; in, This represents the original tactile observation matrix for the entire frame; N Indicates the total number of tactile points; The expression for the three-channel two-dimensional tactile image in S23 is as follows: ; in, Represents a three-channel two-dimensional tactile image; This represents the layout mapping function; The expression for channel normalization processing in S24 is as follows: ; in, Indicates the first c Two-dimensional tactile image after channel normalization Indicates the first c Two-dimensional tactile images of each channel; and These are the mean and standard deviation of the corresponding channel on the calibration set, respectively. To avoid the tiny constants caused by division by zero anomalies; The expression for the three-channel two-dimensional tactile image after time-series smoothing in S25 is as follows: ; in, Indicates the smoothing coefficient; express t Three-channel two-dimensional tactile image after time-series smoothing at time -1; This represents a normalized three-channel two-dimensional tactile image; express t Three-channel two-dimensional tactile image after time-series smoothing; The expression for the nearest neighbor alignment strategy in S26 is as follows: ; in, Indicates tactile index; Indicates taking The smallest haptic frame index is used as the output.
[0010] Furthermore, step S4 specifically includes the following steps: S41. The standard trajectory sample sequence is combined with the tactile representation sequence to obtain the fusion inference observation sequence, which includes a fusion inference observation matrix at multiple time points, wherein... t The fusion inference observation matrix at time step is: ; in, express t The fusion inference observation matrix at each moment; Represents a tactile heatmap; S42, will t The fusion inference observation matrix at each time step is input into the multimodal fusion model, and a shared image encoder is used to process the RGB visual images respectively. tactile heat map Isomorphic encoding is performed to obtain a sequence of tokens in the same feature space, including visual sequence features. and tactile sequence features ; S43. Use a pre-trained text encoder to process the task instructions. Encoding as semantic features Then, the state of the embodied intelligent robot body. The projection module maps the state features to the same dimension as the token sequence. ; S44. Visual sequence features tactile sequence features semantic features and state characteristics The sequences are assembled into a unified sequence using the splicing module. S45. Input the unified sequence into the fusion inference network, perform cross-modal fusion inference, and obtain the fusion latent variables. ; S46. Integrate latent variables Input is fed into the action head, and the action vector is output. That is, the predicted action output by the multimodal fusion model; S47. Calculate the loss value of the pre-constructed loss function based on the action vector and the real action label, and then adjust the parameters of the multimodal fusion model based on the loss value. S48. Determine whether the preset training stop condition has been met. If so, stop training and obtain the trained multimodal fusion model. Otherwise, return to S42.
[0011] Furthermore, the visual sequence features in S42 and tactile sequence features The expressions are as follows: ; ; in, For shared image encoders; The semantic features in S43 State characteristics The expressions are as follows: ; ; in, For a pre-trained text encoder, This is the projection matrix in the projection module.
[0012] Furthermore, the expression for the unified sequence in S44 is as follows: ; in, Represents a unified sequence; Fusion of latent variables in S45 The expression is as follows: ; in, Represents a fusion reasoning network; The expression for the action vector in S46 is as follows: ; in, For action heads.
[0013] Furthermore, the fusion inference network uses the backbone network of the Transformer model.
[0014] In another aspect, the present invention provides a visual-tactile fusion control system for an embodied intelligent robot, configured to execute the above-described visual-tactile fusion control method for an embodied intelligent robot; the visual-tactile fusion control system for the embodied intelligent robot includes: A visual acquisition system is used to collect visual information to obtain a time-ordered sequence of standard trajectory samples. An end-effector tactile acquisition system is used to collect tactile information to obtain a tactile representation sequence; A robot execution system is used to execute joint space increments and gripper control commands.
[0015] The beneficial effects of this invention are: 1. To address the problem of discrete tactile readings in robot end-effector sensing and the difficulty in characterizing physical interaction features, this invention proposes a visual-tactile fusion control method for embodied intelligent robots designed for biochemical experimental tasks. The original triaxial force readings are converted into spatially correlated three-channel two-dimensional tactile images using a mapping operator, and further enhanced by temporal smoothing and channel normalization to obtain a tactile representation sequence. This sequence stably characterizes contact intensity, spatial distribution, and temporal evolution of contact dynamics.
[0016] 2. This invention designs a multimodal fusion model. The multimodal fusion model employs a shared image encoder to achieve isomorphic encoding of visual and tactile images, ensuring semantic alignment of heterogeneous perceptual data within a unified feature space. Simultaneously, the multimodal fusion model in this invention includes a fusion inference network. This network incorporates the self-attention mechanism of the Transformer model, which models long-range dependencies between multimodal tokens, enabling the multimodal fusion model to automatically associate visual target boundaries with tactile force features.
[0017] 3. This invention unifies the timestamp synchronization mechanism through the nearest neighbor alignment strategy and completes the tightly coupled loop of perception-inference-execution within a millisecond period through a multimodal fusion model, ensuring real-time response to complex contact processes and improving the quality of precise operation in biochemical experiments. Attached Figure Description
[0018] Figure 1 This is a flowchart of the visual-tactile fusion control method for embodied intelligent robots in this invention; Figure 2 This is a schematic diagram of tactile data acquisition in an embodiment of the present invention. Figure 1 ; Figure 3 This is a schematic diagram of tactile data acquisition in an embodiment of the present invention. Figure 2 . Detailed Implementation
[0019] To facilitate understanding of the present invention, a more complete description will be given below with reference to the accompanying drawings. Preferred embodiments of the invention are shown in the drawings. However, the invention can be implemented in many other different forms and is not limited to the embodiments described herein. Rather, these embodiments are provided to provide a thorough and complete understanding of the disclosure of the invention.
[0020] The purpose of this invention is to provide a visual-tactile fusion control method and system for embodied intelligent robots designed for biochemical experiments. The visual-tactile fusion control method addresses issues encountered in biochemical experiments, such as insufficient vision due to transparent containers and liquid environments, difficulty in reliably judging contact stages, and slippage caused by batch differences in consumables and containers and slight positional deviations. It structurally represents the tactile information at the end point and aligns and fuses it with visual and task command semantics. Through fusion reasoning, executable actions are generated, and during execution, visual-tactile feedback is used to achieve online correction and anomaly recovery, thereby significantly improving the robustness, safety, and cross-consumable generalization ability of multi-step operations in biochemical experiments. Specifically: Reference Figure 1 and Figure 2 This application provides a vision-tactile fusion control method for embodied intelligent robots for biochemical experimental tasks, including the following steps: S1. For biochemical experimental tasks, visual frames, language commands, body states and action sequences are collected simultaneously and aligned with a unified timestamp to form a standard trajectory sample sequence sorted by time. S2. The raw triaxial force readings of the end tactile dot matrix are converted into a three-channel two-dimensional tactile image through a mapping operator; after normalization and temporal smoothing, they are aligned with the nearest neighbor time of visual observation to form a tactile representation sequence. S3. Construct a multimodal fusion model, which includes a shared image encoder, a pre-trained text encoder, a projection module, a stitching module, a fusion inference network, and an action head. The shared image encoder, the pre-trained text encoder, and the projection module are connected in parallel and then connected to the stitching module, the fusion inference network, and the action head in sequence. S4. Combine the standard trajectory sample sequence with the structured tactile representation to obtain the fusion inference observation sequence; use the fusion inference observation sequence to train the multimodal fusion model to obtain the trained multimodal fusion model; S5. Deploy the trained multimodal fusion model to the device, then input the real-time collected fusion inference observation sequence to obtain motion vectors. Generate joint space increments and gripper control commands for the current cycle based on the motion vectors. Finally, control the embodied intelligent robot based on the joint space increments and gripper control commands for the current cycle. Complete the motion loop of this control cycle and enter the next cycle.
[0021] In some embodiments, S1 specifically includes the following steps: S11. In biochemical experiment tasks, for each complete teaching trajectory, record RGB visual images, robot body state, and task instructions at a fixed sampling frequency, and assign a unified timestamp to each moment. This forms multiple complete teaching trajectory sequences; S12. Summarize multiple complete teaching trajectory sequences to obtain a standard trajectory sample sequence sorted by time. Through the above multimodal trajectory acquisition method, the data maintains a high degree of consistency with the real robot interface at the input and output levels, which facilitates model training and evaluation, and provides a unified data foundation for transfer learning in biochemical experiments under different vessel, consumable, and workstation conditions.
[0022] In some embodiments, the expression for the complete teaching trajectory sequence in S11 is as follows: ; in, Indicates the first m A complete sequence of teaching trajectories; express t RGB visual image at any given time; This indicates the state of the embodied intelligent robot. Indicates task instructions, task instructions It can be a global task objective or a step-by-step description. The recording layer retains the original text or its token (lexical) sequence to ensure consistent interface with the subsequent text encoder. Indicates a real action tag; T This indicates the total number of time steps in the current teaching trajectory; The physical state of the embodied intelligent robot The expression is as follows: ; in, Joint angle, For joint velocity, n The degrees of freedom of the joints of an embodied intelligent robot; The end position, This represents the end-effector attitude. The grippers are in the open / closed state; Represents the set of real numbers; Realistic Action Tags The expression is as follows: ; in, This represents the increase in joint space. The expression for the standard trajectory sample sequence in S12 is as follows: ; in, Represents a standard trajectory sample sequence; This represents the total number of teaching trajectories. Indicates the first m The teaching trajectory is shown in the text.
[0023] In some embodiments, S2 specifically includes the following steps: S21. Obtain the three-axis force vector for each tactile point through tactile data sampling; see the schematic diagram of tactile data sampling. Figure 2 and Figure 3 As shown; where, Figure 2 and Figure 3 These are two different sampling diagrams from the same batch of tactile data. S22. Calculate the original tactile observation matrix of the entire frame based on the triaxial force vector of each tactile point, that is, the original triaxial force reading of the end tactile point array; S23. Considering that the fusion inference network has a stronger spatial correlation learning ability for two-dimensional structured data, all tactile points in the original tactile observation matrix of the entire frame are mapped to... In the grid positions, a three-channel two-dimensional tactile image is thus constructed; Indicates the height of the three-channel two-dimensional tactile image; This represents the width of the three-channel two-dimensional tactile image. The core value of this step lies in converting the disordered point set readings into tensors with spatial adjacency relationships, thereby transforming the original triaxial force readings of the end tactile point array into a two-dimensional representation, enabling the multimodal fusion model to more accurately represent the stability of the contact state, sliding changes, and changes in the contact center position. S24. To eliminate the interference of different experimental rounds, range differences and sensor noise on the learning process, the three-channel two-dimensional tactile images are normalized to obtain normalized three-channel two-dimensional tactile images. Through channel normalization, the numerical scale of each axial force is unified to a similar range, which significantly improves the generalization ability of the model in cross-sensor and cross-task scenarios. S25. To address the jitter and instantaneous spike noise that may occur during high-frequency sampling, an exponential moving average mechanism is introduced to perform temporal smoothing on the normalized three-channel two-dimensional tactile image, resulting in a temporally smoothed three-channel two-dimensional tactile image. S26. To ensure strong consistency between tactile and visual observations in physical time, a nearest neighbor alignment strategy is adopted to timestamp the visual frames. Adjacent tactile sampling timestamps Pairing is performed to synchronize the timestamps of vision and touch. For each frame of visual input, the system automatically retrieves the closest tactile state on the timeline and pairs it. This alignment process effectively eliminates state mismatch caused by asynchronous sampling rates between modalities, enabling the subsequent fusion inference network to jointly analyze visual environment, ontological actions, and tactile feedback information from a unified physical state perspective. S27. Combine the time-smoothed three-channel two-dimensional tactile image with the corresponding tactile sampling timestamp. Pairing yields tactile representation sequences. tactile representation sequence It includes multiple tactile heat maps.
[0024] In some embodiments, the expression for the triaxial force vector of the tactile point in S21 is: ; in, express t At this moment i Three-axis force vectors at each tactile point; They represent t At this moment i The three-axis force vector of each tactile point is x, y, z Three components along the axial direction; The expression for the original tactile observation matrix of the entire frame in S22 is: ; in, This represents the original tactile observation matrix for the entire frame; N Indicates the total number of tactile points; The expression for the three-channel two-dimensional tactile image in S23 is as follows: ; in, Represents a three-channel two-dimensional tactile image; This represents the layout mapping function; the layout mapping function can determine the layout based on the physical topological arrangement of the haptic points at the end of the robotic arm. Each tactile point is mapped to In the grid positions, a three-channel two-dimensional tactile image is constructed; The expression for channel normalization processing in S24 is as follows: ; in, Indicates the first c Two-dimensional tactile image after channel normalization Indicates the first c Two-dimensional tactile images of each channel; and These are the mean and standard deviation of the corresponding channel on the calibration set, respectively. To avoid the tiny constants caused by division by zero anomalies; The expression for the three-channel two-dimensional tactile image after time-series smoothing in S25 is as follows: ; in, Represents the smoothing coefficient. Used to control smoothing intensity, a larger smoothing coefficient It can more effectively suppress environmental noise, reduce the probability of false triggering, and at the same time preserve the main evolution trend of contact distribution; express t Three-channel two-dimensional tactile image after time-series smoothing at time -1; This represents a normalized three-channel two-dimensional tactile image; express t Three-channel two-dimensional tactile image after time-series smoothing; The expression for the nearest neighbor alignment strategy in S26 is as follows: ; in, Indicates tactile index; Indicates taking The smallest haptic frame index is used as the output.
[0025] In some embodiments, S4 specifically includes the following steps: S41. The standard trajectory sample sequence is combined with the tactile representation sequence to obtain the fusion inference observation sequence, which includes a fusion inference observation matrix at multiple time points, wherein... t The fusion inference observation matrix at time step is: ; in, express t The fusion inference observation matrix at each moment; Represents a tactile heatmap; S42, will t The fusion inference observation matrix at each time step is input into the multimodal fusion model. Since both RGB images and tactile heatmaps are image tensors in terms of data format, this invention uses a shared image encoder to process the RGB visual images separately. tactile heat map Isomorphic encoding is performed to obtain a sequence of tokens in the same feature space, including visual sequence features. and tactile sequence features ; S43. Use a pre-trained text encoder to process the task instructions. Encoding as semantic features Then, the state of the embodied intelligent robot body. The projection module maps the state features to the same dimension as the token sequence. ; S44. Visual sequence features tactile sequence features semantic features and state characteristics The sequences are assembled into a unified sequence using the splicing module. S45. Input the unified sequence into the fusion inference network, perform cross-modal fusion inference, and obtain the fusion latent variables. ; Fusing latent variables It represents a unified expression of the current multimodal state under the semantic constraints of the task. It includes target location clues, contact distribution clues, and execution step clues, and is the direct basis for the generation of subsequent actions. S46. Integrate latent variables Input is fed into the action head, and the action vector is output. That is, the predicted action output by the multimodal fusion model; action vector. The expression and control interface are consistent, and are usually expressed in the form of "end-effector increment + gripper command" or "joint space increment + gripper command": ; in, and These represent the end-effector position and attitude increment, respectively. Indicates the joint space increment, used for direct joint control; This indicates the gripper opening and closing command; S47. Calculate the loss value of the pre-constructed loss function based on the action vector and the real action label, and then adjust the parameters of the multimodal fusion model based on the loss value. S48. Determine whether the preset training stop condition has been met. If so, stop obtaining the trained multimodal fusion model; otherwise, return to S42.
[0026] In some embodiments, visual sequence features in S42 and tactile sequence features The expressions are as follows: ; ; in, For shared image encoders; The semantic features in S43 State characteristics The expressions are as follows: ; ; in, For a pre-trained text encoder, This is the projection matrix in the projection module.
[0027] In some embodiments, the expression for the unified sequence in S44 is as follows: ; in, Represents a unified sequence; Fusion of latent variables in S45 The expression is as follows: ; in, Represents a fusion reasoning network; The expression for the action vector in S46 is as follows: ; in, For action heads.
[0028] In some embodiments, the fusion inference network is selected from the backbone network of the Transformer model.
[0029] To verify the visual-tactile fusion control capabilities of the embodied intelligent robot, it was manipulated to perform liquid retrieval, liquid transfer, and liquid pouring operations. During the liquid retrieval phase, the embodied intelligent robot continuously adjusted its end-effector posture and approach speed in real time using tactile heatmap feedback on contact distribution changes, ensuring the stability of the contact process between the end-effector and the target. When the contact state reached the preset grasping conditions, the embodied intelligent robot output a gripper closing command to reliably grasp the liquid container, and then lifted and disengaged by outputting joint increments. Entering the liquid transfer phase, the embodied intelligent robot, while maintaining stable gripping, moved above the target container and achieved secondary precise alignment through continuous inference. After alignment, the multimodal fusion model output continuous joint increments to change the end-effector posture, performing a pouring action and returning to a stable posture after the task was completed. To further verify the visual-tactile fusion control capabilities of the embodied intelligent robot, it was manipulated to perform solid retrieval, solid transfer, and solid pouring operations. The operation process for solid retrieval and solid transfer was basically the same as that for the liquid phase. After the embodied intelligent robot moves above the beaker or solid container and completes alignment, it continuously analyzes the tactile thermogram during the approach, contact, and gripping phases, using information such as the stability of the contact distribution and the presence of any offset to aid in judgment. Once gripping is complete, the embodied intelligent robot lifts and moves to the target placement position, achieving solid transfer through fine-tuning of posture or position.
[0030] To generate the experimental time-series results diagram, this embodiment records the corresponding RGB image frames, stage labels, and timestamps at key nodes in each stage (such as core nodes like "moving to test tube," "clamping test tube," and "pouring liquid"), arranging and labeling them in chronological order. The system uses stage names to visually represent the temporal sequence of task execution and the effects of key actions. Throughout the experiment, the system synchronously saves RGB images, haptic heatmaps, ontological state observations, action outputs, and execution readback states for each control cycle, facilitating subsequent experimental reproduction, comparison of effects under different settings, and expansion of the training dataset.
[0031] In another aspect, the present invention provides a visual-tactile fusion control system for an embodied intelligent robot, configured to execute the above-described visual-tactile fusion control method for an embodied intelligent robot; the visual-tactile fusion control system for the embodied intelligent robot includes: A visual acquisition system is used to collect visual information to obtain a time-ordered sequence of standard trajectory samples. An end-effector tactile acquisition system is used to collect tactile information to obtain a tactile representation sequence; A robot execution system is used to execute joint space increments and gripper control commands.
[0032] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Furthermore, the technical solutions of the various embodiments of the present invention can be combined with each other, but this must be based on the ability of those skilled in the art to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed by the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.
Claims
1. A method for controlling embodied intelligent robots through visual-tactile fusion for biochemical experimental tasks, characterized in that, Includes the following steps: S1. For biochemical experimental tasks, visual frames, language commands, body states and action sequences are collected simultaneously and aligned with a unified timestamp to form a standard trajectory sample sequence sorted by time. S2. The raw triaxial force readings of the end tactile dot matrix are converted into a three-channel two-dimensional tactile image through a mapping operator; after normalization and temporal smoothing, they are aligned with the nearest neighbor time of visual observation to form a tactile representation sequence. S3. Construct a multimodal fusion model, which includes a shared image encoder, a pre-trained text encoder, a projection module, a stitching module, a fusion inference network, and an action head. The shared image encoder, the pre-trained text encoder, and the projection module are connected in parallel and then connected to the stitching module, the fusion inference network, and the action head in sequence. S4. Combine the standard trajectory sample sequence with the structured tactile representation to obtain the fusion inference observation sequence; use the fusion inference observation sequence to train the multimodal fusion model to obtain the trained multimodal fusion model; S5. Deploy the trained multimodal fusion model to the device, then input the real-time collected fusion inference observation sequence to obtain the action vector, generate the joint space increment and gripper control command for the current cycle based on the action vector, and finally control the embodied intelligent robot based on the joint space increment and gripper control command for the current cycle.
2. The method for visual-tactile fusion control of an embodied intelligent robot for biochemical experimental tasks according to claim 1, characterized in that, S1 specifically includes the following steps: S11. In biochemical experiment tasks, for each complete teaching trajectory, record RGB visual images, robot body state, and task instructions at a fixed sampling frequency, and assign a unified timestamp to each moment. This forms multiple complete teaching trajectory sequences; S12. Summarize multiple complete teaching trajectory sequences to obtain a standard trajectory sample sequence sorted by time.
3. The method for visual-tactile fusion control of an embodied intelligent robot for biochemical experimental tasks according to claim 2, characterized in that, The expression for the complete teaching trajectory sequence in S11 is as follows: ； in, Indicates the first m A complete sequence of teaching trajectories; express t RGB visual image at any given time; This indicates the state of the embodied intelligent robot. Indicates task instructions; Indicates a real action tag; T This indicates the total number of time steps in the current teaching trajectory; The physical state of the embodied intelligent robot The expression is as follows: ； in, Joint angle, For joint velocity, n The degrees of freedom of the joints of an embodied intelligent robot; The end position, This represents the end-effector attitude. The grippers are in the open / closed state; Represents the set of real numbers; Realistic Action Tags The expression is as follows: ； in, This represents the increase in joint space. The expression for the standard trajectory sample sequence in S12 is as follows: ； in, Represents a standard trajectory sample sequence; This represents the total number of teaching trajectories. Indicates the first m The teaching trajectory is shown in the text.
4. The method for visual-tactile fusion control of an embodied intelligent robot for biochemical experimental tasks according to claim 3, characterized in that, S2 specifically includes the following steps: S21. Obtain the three-axis force vector of each tactile point through tactile data sampling; S22. Calculate the original tactile observation matrix of the entire frame based on the triaxial force vector of each tactile point, that is, the original triaxial force reading of the end tactile point array; S23. Map all tactile points in the entire frame's original tactile observation matrix to... In the grid positions, a three-channel two-dimensional tactile image is thus constructed; Indicates the height of the three-channel two-dimensional tactile image; Indicates the width of the three-channel two-dimensional tactile image; S24. Perform channel normalization processing on the three-channel two-dimensional tactile image to obtain the normalized three-channel two-dimensional tactile image; S25. An exponential moving average mechanism is introduced to perform temporal smoothing on the normalized three-channel two-dimensional tactile image to obtain a temporally smoothed three-channel two-dimensional tactile image. S26. Use the nearest neighbor alignment strategy to timestamp the visual frames. Adjacent tactile sampling timestamps Pairing is performed to synchronize visual and tactile timestamps; S27. Combine the time-smoothed three-channel two-dimensional tactile image with the corresponding tactile sampling timestamp. Pairing yields tactile representation sequences. tactile representation sequence It includes multiple tactile heat maps.
5. The visual-tactile fusion control method for an embodied intelligent robot for biochemical experimental tasks according to claim 4, characterized in that, The expression for the three-axis force vector of the tactile point in S21 is: ； in, express t At this moment i Three-axis force vectors at each tactile point; They represent t At this moment i The three-axis force vector of each tactile point is x, y, z Three components along the axial direction; The expression for the original tactile observation matrix of the entire frame in S22 is: ； in, This represents the original tactile observation matrix for the entire frame; N Indicates the total number of tactile points; The expression for the three-channel two-dimensional tactile image in S23 is as follows: ； in, Represents a three-channel two-dimensional tactile image; This represents the layout mapping function; The expression for channel normalization processing in S24 is as follows: ； in, Indicates the first c Two-dimensional tactile image after channel normalization Indicates the first c Two-dimensional tactile images of each channel; and These are the mean and standard deviation of the corresponding channel on the calibration set, respectively. To avoid the tiny constants caused by division by zero anomalies; The expression for the three-channel two-dimensional tactile image after time-series smoothing in S25 is as follows: ； in, Indicates the smoothing coefficient; express t Three-channel two-dimensional tactile image after time-series smoothing at time -1; This represents a normalized three-channel two-dimensional tactile image; express t Three-channel two-dimensional tactile image after time-series smoothing; The expression for the nearest neighbor alignment strategy in S26 is as follows: ； in, Indicates tactile index; Indicates taking The smallest haptic frame index is used as the output.
6. The method for visual-tactile fusion control of an embodied intelligent robot for biochemical experimental tasks according to claim 5, characterized in that, S4 specifically includes the following steps: S41. The standard trajectory sample sequence is combined with the tactile representation sequence to obtain the fusion inference observation sequence, which includes a fusion inference observation matrix at multiple time points, wherein... t The fusion inference observation matrix at time step is: ； in, express t The fusion inference observation matrix at each moment; Represents a tactile heatmap; S42, will t The fusion inference observation matrix at each time step is input into the multimodal fusion model, and a shared image encoder is used to process the RGB visual images respectively. tactile heat map Isomorphic encoding is performed to obtain a sequence of tokens in the same feature space, including visual sequence features. and tactile sequence features ; S43. Use a pre-trained text encoder to process the task instructions. Encoding as semantic features Then, the state of the embodied intelligent robot body. The projection module maps the state features to the same dimension as the token sequence. ; S44. Visual sequence features tactile sequence features semantic features and state characteristics The sequences are assembled into a unified sequence using the splicing module. S45. Input the unified sequence into the fusion inference network, perform cross-modal fusion inference, and obtain the fusion latent variables. ; S46. Integrate latent variables Input is fed into the action head, and the action vector is output. That is, the predicted action output by the multimodal fusion model; S47. Calculate the loss value of the pre-constructed loss function based on the action vector and the real action label, and then adjust the parameters of the multimodal fusion model based on the loss value. S48. Determine whether the preset training stop condition has been met. If so, stop training and obtain the trained multimodal fusion model. Otherwise, return to S42.
7. The method for visual-tactile fusion control of an embodied intelligent robot for biochemical experimental tasks according to claim 6, characterized in that, The visual sequence features in S42 and tactile sequence features The expressions are as follows: ； ； in, For shared image encoders; The semantic features in S43 State characteristics The expressions are as follows: ； ； in, For a pre-trained text encoder, This is the projection matrix in the projection module.
8. The method for visual-tactile fusion control of an embodied intelligent robot for biochemical experimental tasks according to claim 7, characterized in that, The expression for the unified sequence in S44 is as follows: ； in, Represents a unified sequence; Fusion of latent variables in S45 The expression is as follows: ； in, Represents a fusion reasoning network; The expression for the action vector in S46 is as follows: ； in, For action heads.
9. A visual-tactile fusion control method for an embodied intelligent robot for biochemical experimental tasks according to claim 8, characterized in that, The fusion inference network uses the backbone network of the Transformer model.
10. A vision-tactile fusion control system for an embodied intelligent robot, characterized in that, The embodied intelligent robot visual-tactile fusion control method according to any one of claims 1 to 9 is configured to perform the control method. The embodied intelligent robot visual-tactile fusion control system includes: A visual acquisition system is used to collect visual information to obtain a time-ordered sequence of standard trajectory samples. An end-effector tactile acquisition system is used to collect tactile information to obtain a tactile representation sequence; A robot execution system is used to execute joint space increments and gripper control commands.