Robot grasping recognition method and system based on visual and tactile information fusion

Abstract

The invention discloses a robot grasping recognition method and system based on visual and tactile information fusion. The method comprises the following steps that S1, tactile signals are preprocessed; the method comprises the following steps: S1, integrating data acquired by each tactile sensor into a four-dimensional tensor, and constructing a basic tactile snapshot sequence; a three-level coding network composed of a CNN (Convolutional Neural Network), an LSTM (Long Short Term Memory) and a Transform Encoder is adopted to code the tactile information; S3, visual signal preprocessing and coding; s4, carrying out cross-modal alignment; and S5, large model reasoning. According to the robot grabbed object recognition method and system combining the visual features and the tactile features, the grabbed object of the robot is understood and analyzed based on the visual information and the tactile information, and the accuracy of robot actions and the completeness of tasks can be improved.

Description

Technical Field

[0001] The present invention belongs to the field of robot environment perception, and in particular relates to a robot grasping recognition method and system based on the fusion of visual and tactile information. Background Art

[0002] With the widespread adoption of service robots in areas such as homes, healthcare, and logistics, precise grasping and manipulation of objects by robots has become a key challenge. Traditional grasping systems primarily rely on single-source visual information (such as RGB or depth cameras) for object recognition and positioning, but face numerous limitations in complex scenarios: visual occlusion, reflective surfaces, and difficulty recognizing transparent objects significantly reduce grasping success rates. Furthermore, visual information alone cannot perceive the physical properties of an object (such as hardness, material, surface texture, temperature and humidity, or weight distribution), leading to improper control of grasping force, which can easily cause slippage or damage the object.

[0003] MEMS electronic skin, a new type of tactile sensor characterized by flexibility, miniaturization, and multifunctional integration, has demonstrated high-density, high-resolution tactile acquisition capabilities in multiple studies. It can sense a variety of tactile signals, including pressure, torque, vibration, surface texture, and temperature, and possesses rich physical semantic parsing capabilities and dynamic environmental adaptability. Therefore, multimodal perception methods that integrate visual and tactile information have become an important research direction for improving the operational capabilities of service robots.

[0004] In recent years, large models have demonstrated significant advantages over traditional specialized algorithmic models due to their powerful cross-modal representation capabilities, end-to-end learning characteristics, and emerging generalized intelligence. Large-model reasoning methods based on multimodal information fusion are widely used for cognitive and decision-making tasks in complex scenarios. Therefore, this invention leverages the powerful reasoning capabilities of large models based on visual and tactile information to enable service robots to recognize and understand grasped objects, thus overcoming the perceptual limitations of traditional grasping systems and achieving human-like adaptive precision manipulation. Summary of the Invention

[0005] The technical problem to be solved by the present invention is: for the multi-target path planning of a mobile robot, a multi-layer algorithm with a feedback mechanism is adopted, the upper-layer algorithm quickly provides a safe path with obstacle avoidance performance and the middle-layer and lower-layer algorithms effectively cooperate, thereby achieving the timeliness and reliability of multi-target path planning.

[0006] To solve the above technical problems, the present invention adopts a technical solution: a robot grasping recognition method and system based on the fusion of visual and tactile information, comprising the following steps:

[0007] Step S1: tactile signal preprocessing;

[0008] The data acquired by each tactile sensor is integrated into a four-dimensional tensor, constructing a basic tactile snapshot sequence. The raw tactile signals are then preprocessed, including offset removal and normalization. Adaptive filtering techniques (such as wavelet reconstruction) are then used to remove high-frequency noise components, resulting in a pure tactile signal. Finally, action cycles are marked based on event triggering mechanisms (such as contact / release boundary detection), ultimately outputting structured data in the standard [Batch, Time, X, Y, Channel] format.

[0009] Step S2: tactile signal encoding;

[0010] A three-layer encoding network consisting of a CNN, LSTM, and Transformer Encoder is used to encode tactile information. The CNN is used to extract the local contact structure and force distribution, the LSTM is used to analyze the temporal patterns, and the Transformer Encoder is responsible for extracting a 512-dimensional semantic vector containing tactile features.

[0011] Step S3: visual signal preprocessing and encoding;

[0012] The image signals collected simultaneously with the tactile signals undergo a series of image preprocessing operations to obtain a standardized visual signal. The preprocessed visual signal then passes through the Vision Encoder and the Transformer Encoder, ultimately obtaining a 512-dimensional semantic vector containing visual features.

[0013] Step S4: cross-modal alignment;

[0014] By calculating attention weights between visual and tactile features, dynamic mapping and alignment of feature spaces are achieved, representing a unified "vision + tactile" embedding. During the training phase, a hard sample mining technique is employed. Each training sample consists of an anchor (tactile semantic vector), a hard positive sample (matching score <0.6 in the matching image semantic vector), and a hard negative sample (matching score >0.4 in the mismatching image semantic vector). A hard sample mining triplet contrast loss function is employed, where a triplet consists of an anchor plus a hard positive sample or an anchor plus a hard negative sample. Using pre-annotated object labels (e.g., "a colorless plastic bottle filled with red liquid") as supervision, the algorithm constrains matching visual-tactile feature pairs to be close in distance in the shared space (e.g., L2 distance <0.3), while mismatching feature pairs are far apart (e.g., L2 distance >1.2). Furthermore, a temporal synchronization constraint loss is introduced to ensure temporal alignment of the two modalities during dynamic interaction.

[0015] The contrast loss expression of hard example mining triples is as follows:

[0016]

[0017] in, represents the tactile features in the i-th pair of samples, and Denote the visual features of the hard positive sample and the hard negative sample in the i-th pair of samples, respectively. The hard positive and negative samples are obtained by calculating the similarity between the positive and negative sample features and the tactile features. α>0 is a boundary hyperparameter. [·] + =max(0,·) is used to ensure non-negative loss.

[0018] The temporal synchronization constraint loss expression is as follows:

[0019]

[0020] in, and Represent the visual and tactile feature vectors of the i-th sample at time step t, Δt is the time offset between the two modalities, τ is the synchronization tolerance threshold, is an indicator function, which takes 1 when the time series deviation is within the threshold, otherwise it takes 0. N is the batch size, and T is the time series length.

[0021] Step S5: large model reasoning;

[0022] The Flamingo-VT (Vision-Tactile) large-scale visual-tactile model, built on Flamingo, is used to infer visual and tactile features. Specifically, the aligned 512-dimensional visual features and 512-dimensional tactile features are input into the fine-tuned Flamingo-VT large-scale model. Flamingo-VT establishes feature associations through a cross-attention layer and activates its built-in knowledge base to parse physical semantics (such as identifying material, name, and physical state).

[0023] In summary, due to the adoption of the above technical solution, the beneficial effects of the present invention are:

[0024] 1. The present invention has developed a method and system for robot grasping object recognition that combines visual features and tactile features.

[0025] 2. The present invention understands and analyzes the robot's grasping objects based on visual information and tactile information, which can improve the accuracy of the robot's movements and the integrity of the task.

[0026] 3. This invention effectively integrates visual information and tactile information, laying a key technical foundation for the autonomous operation and decision-making system of future intelligent service robots. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] To facilitate understanding by those skilled in the art, the present invention is further described below with reference to the accompanying drawings.

[0028] Figure 1 This is the overall block diagram of the robot grasping object recognition and understanding system.

[0029] Figure 2 Flowchart for large model reasoning based on the fusion of visual and tactile information. DETAILED DESCRIPTION

[0030] The following will clearly and completely describe the technical solutions of the present invention with reference to the embodiments. Obviously, the embodiments described are only some of the embodiments of the present invention, rather than all of them. The embodiments of the present invention and all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0031] Example 1

[0032] The present invention provides a robot grasping recognition method and system based on the fusion of visual and tactile information, which includes the following steps:

[0033] Step S1: tactile signal preprocessing;

[0034] The data acquired by each tactile sensor is integrated into a four-dimensional tensor, constructing a basic tactile snapshot sequence. The raw tactile signals are then preprocessed, including offset removal and normalization. Adaptive filtering techniques (such as wavelet reconstruction) are then used to remove high-frequency noise components, resulting in a pure tactile signal. Finally, action cycles are marked based on event triggering mechanisms (such as contact / release boundary detection), ultimately outputting structured data in the standard [Batch, Time, X, Y, Channel] format.

[0035] Step S2: tactile signal encoding;

[0036] A three-layer encoding network consisting of a CNN, LSTM, and Transformer Encoder is used to encode tactile information. The CNN is used to extract the local contact structure and force distribution, the LSTM is used to analyze the temporal patterns, and the Transformer Encoder is responsible for extracting a 512-dimensional semantic vector containing tactile features.

[0037] Step S3: visual signal preprocessing and encoding;

[0038] The image signals collected simultaneously with the tactile signals undergo a series of image preprocessing operations to obtain a standardized visual signal. The preprocessed visual signal then passes through the Vision Encoder and the Transformer Encoder, ultimately obtaining a 512-dimensional semantic vector containing visual features.

[0039] Step S4: cross-modal alignment;

[0040] By calculating attention weights between visual and tactile features, dynamic mapping and alignment of feature spaces are achieved, representing a unified "vision + tactile" embedding. During the training phase, a hard sample mining technique is employed. Each training sample consists of an anchor (tactile semantic vector), a hard positive sample (matching score <0.6 in the matching image semantic vector), and a hard negative sample (matching score >0.4 in the mismatching image semantic vector). A hard sample mining triplet contrast loss function is employed, where a triplet consists of an anchor plus a hard positive sample or an anchor plus a hard negative sample. Using pre-annotated object labels (e.g., "a colorless plastic bottle filled with red liquid") as supervision, the algorithm constrains matching visual-tactile feature pairs to be close in distance in the shared space (e.g., L2 distance <0.3), while mismatching feature pairs are far apart (e.g., L2 distance >1.2). Furthermore, a temporal synchronization constraint loss is introduced to ensure temporal alignment of the two modalities during dynamic interaction.

[0041] The contrast loss expression of hard example mining triples is as follows:

[0042]

[0043] in, represents the tactile features in the i-th pair of samples, and Denote the visual features of the hard positive sample and the hard negative sample in the i-th pair of samples, respectively. The hard positive and negative samples are obtained by calculating the similarity between the positive and negative sample features and the tactile features. α>0 is a boundary hyperparameter. [·] + =max(0,·) is used to ensure non-negative loss.

[0044] The temporal synchronization constraint loss expression is as follows:

[0045]

[0046] in, and Represent the visual and tactile feature vectors of the i-th sample at time step t, Δt is the time offset between the two modalities, τ is the synchronization tolerance threshold, is an indicator function, which takes 1 when the time series deviation is within the threshold, otherwise it takes 0. N is the batch size, and T is the time series length.

[0047] Step S5: large model reasoning;

[0048] The Flamingo-VT (Vision-Tactile) large-scale visual-tactile model, built on Flamingo, is used to infer visual and tactile features. Specifically, the aligned 512-dimensional visual features and 512-dimensional tactile features are input into the fine-tuned Flamingo-VT large-scale model. Flamingo-VT establishes feature associations through a cross-attention layer and activates a built-in knowledge base to resolve physical semantics (such as identifying material, name, and physical state).

[0049] The preferred embodiments of the present invention disclosed above are intended only to help illustrate the present invention. These preferred embodiments do not exhaustively describe all details, nor do they limit the present invention to specific embodiments. Obviously, many modifications and variations are possible based on the contents of this specification. These embodiments are selected and described in detail in this specification to better explain the principles and practical applications of the present invention, thereby enabling those skilled in the art to better understand and utilize the present invention. The present invention is limited only by the claims and their full scope and equivalents.

Claims

1. A robot grasping recognition method and system based on the fusion of visual and tactile information, characterized in that: The following steps are involved: Step S1: tactile signal preprocessing; The data acquired by each tactile sensor is integrated into a four-dimensional tensor, constructing a basic tactile snapshot sequence. The raw tactile signals are then preprocessed, including offset removal and normalization. Adaptive filtering techniques (such as wavelet reconstruction) are then used to remove high-frequency noise components, resulting in a pure tactile signal. Finally, action cycles are marked based on event triggering mechanisms (such as contact / release boundary detection), ultimately outputting structured data in the standard [Batch, Time, X, Y, Channel] format. Step S2: tactile signal encoding; A three-layer encoding network consisting of a CNN, LSTM, and Transformer Encoder is used to encode tactile information. The CNN is used to extract the local contact structure and force distribution, the LSTM is used to analyze the temporal patterns, and the Transformer Encoder is responsible for extracting a 512-dimensional semantic vector containing tactile features. Step S3: visual signal preprocessing and encoding; The image signals collected simultaneously with the tactile signals undergo a series of image preprocessing operations to obtain a standardized visual signal. The preprocessed visual signal then passes through the Vision Encoder and the Transformer Encoder, ultimately obtaining a 512-dimensional semantic vector containing visual features. Step S4: cross-modal alignment; By calculating attention weights between visual and tactile features, dynamic mapping and alignment of feature spaces are achieved, representing a unified "visual + tactile" embedding. During the training phase, a hard sample mining technique is employed. Each training sample consists of an anchor (tactile semantic vector), a hard positive example (matching score <0.6 in the matching image semantic vector), and a hard negative example (matching score >0.4 in the mismatching image semantic vector). A hard sample mining triplet contrast loss function is employed, where a triplet consists of an anchor plus a hard positive example or an anchor plus a hard negative example. Using pre-annotated object labels (e.g., "a colorless plastic bottle filled with red liquid") as supervision, the algorithm constrains matching visual-tactile feature pairs to be close in distance in the shared space (e.g., L2 distance <0.3), while mismatching feature pairs are far apart (e.g., L2 distance >1.2). Furthermore, a temporal synchronization constraint loss is introduced to ensure temporal alignment of the two modalities during dynamic interaction. The contrast loss expression of hard example mining triples is as follows: in, represents the tactile features in the i-th pair of samples, and Denote the visual features of the hard positive sample and the hard negative sample in the i-th pair of samples, respectively. The hard positive and negative samples are obtained by calculating the similarity between the positive and negative sample features and the tactile features. α>0 is a boundary hyperparameter. [·] + =max(0,·) is used to ensure non-negative loss. The temporal synchronization constraint loss expression is as follows: in, and Represent the visual and tactile feature vectors of the i-th sample at time step t, Δt is the time offset between the two modalities, τ is the synchronization tolerance threshold, is an indicator function, which takes 1 when the time series deviation is within the threshold, otherwise it takes 0. N is the batch size, and T is the time series length. Step S5: large model reasoning; The Flamingo-VT (Vision-Tactile) large-scale visual-tactile model, built on Flamingo, is used to infer visual and tactile features. Specifically, the aligned 512-dimensional visual features and 512-dimensional tactile features are input into the fine-tuned Flamingo-VT large-scale model. Flamingo-VT establishes feature associations through a cross-attention layer and activates a built-in knowledge base to resolve physical semantics (such as identifying material, name, and physical state).

Images