A deep learning-based visual haptic multi-modal object recognition method

Abstract

The application relates to a kind of visual haptic multimodal object recognition methods based on deep learning, comprising the following steps: collecting touch and slip data and visual data;Touch and slip-visual data set is made, as follows: touch and slip data processing: the touch and slip signal collected in the first step is obtained by demodulating pressure data and vibration data respectively;Visual data processing: the required visual image is obtained after the original image data is preprocessed;Visual image label acquisition: set several silica gel blocks with different surface hardness and roughness, and after collecting visual images of different silica gel blocks, label the visual images with the surface hardness and roughness, realize the connection between visual image and object surface specific condition;Touch and slip-visual data set is made;Establish and train object recognition neural network model.

Description

Technical Field

[0001] This invention relates to the field of fiber optic sensing, and in particular to a multimodal recognition system that obtains tactile and slip data of an object surface through fiber optic grating sensing and integrates visual data of the object. Background Technology

[0002] With the advent of the artificial intelligence era, intelligent robots are increasingly coming into people's view, and their applications are no longer limited to factory production, but are increasingly entering homes, offices, medical facilities, and other places. Tactile characteristics are crucial for humans to complete various tasks. The human tactile receptor network provides sensory feedback, making grasping dexterous, a technology difficult to replicate in robots. For example, when touching an object, through tactile feedback such as sliding and pressing, people continuously adjust their movements to complete lifting, throwing, or other interactive actions without damaging the object's structure. Similarly, robotic arms are the bridge between robots and the physical world. Before contacting an object, a robotic arm may need to know the object's tactile characteristics to complete the work requirements set by humans while ensuring the object's structural integrity or stability. Robotic arms acquire tactile information from the surface of objects they contact and collect and utilize this information to help robots analyze objects more smoothly, enhancing their perception capabilities and thus assisting people in completing various complex tasks.

[0003] Humans rely on multiple senses, such as sight, touch, hearing, smell, and taste, to judge and interact with the world, but this presents a significant challenge for robots. Simulating human contact with objects not only endows robots with tactile perception but also with visual capabilities. Traditional tactile sensors have high recognition capabilities only for object surfaces, but struggle to acquire information such as volume, shape, and color, leading to recognition difficulties. Although computer vision-based robot grasping strategies have made progress with abundant visual data and emerging machine learning tools, some problems remain. For example, traditional tactile sensors experience performance degradation in special scenarios such as strong magnetic fields and strong electric fields, and also suffer from complex sensor packaging, numerous wires, and heavy weight. Summary of the Invention

[0004] The purpose of this invention is to provide a visual-tactile multimodal object recognition method, which utilizes a trained multimodal neural network model to achieve relatively accurate object recognition. This invention is based on a fiber optic grating tactile sensor to detect tactile signals, which include both dynamic vibration signals and static pressure signals from the object surface, for tactile data acquisition. Simultaneously, an industrial camera's visual sensor acquires images of the object, obtaining a visual dataset. Both datasets are then fed into a multimodal neural network algorithm for training, resulting in a deep learning model. This model can effectively recognize objects. The technical solution is as follows:

[0005] A deep learning-based visual-tactile multimodal object recognition method includes the following steps:

[0006] The first step is to collect tactile and visual data, as follows:

[0007] A fiber Bragg grating (FBG) tactile sensor is encapsulated in a silicone sleeve, which is placed at the end of a robotic arm whose trajectory is controlled by a program. During tactile signal acquisition, the silicone sleeve containing the FBG serves as the sensing area, directly contacting the object being measured. When the silicone sleeve with the tactile sensor contacts and presses down on the object, the pressure on the silicone sleeve causes tangential deformation of the FBG, resulting in a change in its center wavelength. When the silicone sleeve with the tactile sensor slides and rubs against the object, vibrations are induced between the silicone sleeve and the contact surface and transmitted to the FBG. The center wavelength information of the FBG's reflection spectrum is modulated into the interference signal of each scan frame in the form of a signal envelope, used to demodulate the static pressure data detected by the tactile sensor. The dynamic vibration signal is coupled into the interference signal of the scan frame in the form of instantaneous frequency changes, used to demodulate the dynamic vibration data of the tactile sensor. The tactile signal is thus acquired using the FBG tactile sensor.

[0008] Visual data acquisition process: The camera is used to take pictures of the object under test from different angles and distances to obtain a visual dataset; when the silicone sleeve equipped with the tactile sensor slides and rubs against the object, the contact surface between the silicone sleeve and the object is photographed to collect raw image data.

[0009] The second step is to create a tactile-visual dataset, as follows:

[0010] (1) Tactile and slip data processing: The tactile and slip signals collected in the first step are demodulated to obtain pressure data and vibration data respectively;

[0011] (2) Visual data processing: The original image data is preprocessed to obtain the required visual image;

[0012] (3) Acquisition of visual image labels: Set up several silicone blocks with different surface hardness and roughness. After acquiring visual images of different silicone blocks, label the visual images with the relevant surface hardness and roughness to realize the connection between the visual images and the specific conditions of the object surface.

[0013] (4) Create a touch-slip-visual dataset. Each sample in the touch-slip-visual dataset is described as a quadruple Di = {Vi, Ti, Fi, Ri}; where V represents the visual image of the object, T represents the vibration data obtained by contacting the object surface to reflect the smoothness or roughness of the object surface, F represents the pressure data obtained by contacting the object surface, the granularity of which is controlled by the scanning rate frame of the scanning light source to reflect the softness or hardness of the object surface; R represents the label of the specific condition of the object surface, i ∈ (1, N), and N represents the number of samples.

[0014] The third step is to build and train an object recognition neural network model.

[0015] Furthermore, step (1) of the second step includes pressure data demodulation processing, taking the interference signal corresponding to the scan frame when the robot arm and the object are not in contact as the reference signal; performing Gaussian smoothing on the interference signal of all scan frames; extracting the envelope of the interference signal through Hilbert transform; and calculating the offset between the signal envelope extracted from each frame and the reference signal envelope using cross-correlation, which is used as a measure of the pressure data.

[0016] Furthermore, the second step (1) includes vibration data demodulation processing: performing Hilbert transform on the interference signal to extract its envelope, and using the extracted signal envelope to normalize the original interference signal; performing wavelet packet time-frequency analysis transform on the normalized interference signal to demodulate its instantaneous frequency, and the waveform of this instantaneous frequency is the waveform of the vibration signal in a scanning frame; by calculating the variance of the vibration signal in each scanning frame, it is used as a measure of the intensity of the vibration signal detected in that scanning frame.

[0017] Furthermore, the object recognition neural network model employs a feature-level fusion method, extracting features from pressure data and vibration data separately and then fusing them to discover the relationship between the two types of data, thereby obtaining tactile and slippery features; extracting features from the visual image and then concatenating them with the tactile and slippery features to obtain the total features of multimodal fusion; inputting this total feature into a softmax layer for classification tasks; and using the cross-entropy loss function to evaluate the performance of the neural network model for object recognition.

[0018] The advantages of this invention compared to existing technologies are as follows:

[0019] 1) Compared with the tactile data obtained by traditional electrical signal tactile sensors, the fiber optic grating tactile sensor used in this system has a stronger anti-electromagnetic interference capability in data acquisition and can detect vibrations by light interference, thereby achieving high-frequency and finer vibration detection and obtaining a more accurate tactile data set.

[0020] 2) Compared to traditional tactile data acquisition, which focuses on the analysis of dynamic vibration data, this algorithm uses vibration signals and pressure signals acquired by a single sensor to analyze simultaneously, explore the intrinsic relationship between the two signals, and work together with visual data to achieve better results in the training of recognition models.

[0021] 3) Compared with traditional single-tactile recognition algorithms or single-visual recognition algorithms, this algorithm uses tactile and visual data, and integrates tactile and visual signals for analysis, which has higher recognition accuracy and can perform object surface recognition and classification more accurately. Attached Figure Description

[0022] Figure 1 This is the optical path diagram for data acquisition in this invention.

[0023] Figure 2 This is a fiber grating packaging diagram of the experimental platform of this invention.

[0024] Figure 3 This is the system flowchart of the present invention.

[0025] Figure 4 This is a signal demodulation diagram of the data collected by this invention.

[0026] Figure 5 This is a diagram of the algorithm structure used in this invention. Detailed Implementation

[0027] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be described in further detail below.

[0028] Example:

[0029] The object recognition method and data acquisition system provided in this embodiment of the invention refer to 2022114450111. The invention is implemented in detail according to the following steps:

[0030] The first step is to collect tactile and visual data.

[0031] This embodiment is based on Figure 1 The data acquisition optical path shown includes a fiber grating 1 with a reflection bandwidth of 1nm, a Faraday rotator mirror 2, a 2×2 coupler 3, a swept wave light source module 4, a photodetector 5, a data acquisition card 6, and a host computer for data processing 7. The fiber grating tactile sensor in the optical path is encapsulated within... Figure 2 The silicone sleeve shown is placed inside the robotic arm and then installed on the end of the robotic arm.

[0032] During the acquisition of tactile and slip signals, a silicone sleeve encapsulating a fiber Bragg grating serves as the sensing area, directly contacting the object under test. When the silicone sleeve containing the tactile and slip sensor comes into contact with the object, the pressure acting on the silicone sleeve causes tangential deformation of the fiber Bragg grating, resulting in a change in its center wavelength. When the silicone sleeve containing the tactile and slip sensor slides against the object, vibrations are induced between the silicone sleeve and the contact surface and transmitted to the fiber Bragg grating. The center wavelength information of the fiber Bragg grating's reflection spectrum is modulated into the interference signal of each scan frame in the form of a signal envelope, and used to demodulate the static pressure information detected by the tactile and slip sensor. The dynamic vibration signal is coupled into the interference signal of the scan frame in the form of instantaneous frequency changes, and used to demodulate the dynamic vibration information of the tactile and slip sensor.

[0033] Tactile data acquisition process: A computer program is written to collect data at a pressure distance of 10mm and a sliding speed of 160mm / s. When the encapsulated silicone sleeve contacts the surface of the object being tested, the object sends feedback to the sensor. The sensor collects light information, which propagates through the optical path in the acquisition system and is finally received by the signal acquisition card to obtain the tactile information. The experiment is repeated 30 times on one object to obtain its tactile data. A robotic arm is used to collect data from different object surfaces to obtain the tactile data for the entire system.

[0034] The parameters for each module in the above-mentioned process of acquiring tactile and slip information are as follows: the scanning rate of the swept wave light source module is 50 Hz, the acquisition rate of the data acquisition card is 100 KSa / s, and the optical path difference between the two arms is set to 5 cm. Correspondingly, the scanning period for each scan frame is 20 ms, the interference signal wavenumber for each scan frame is approximately 80, and the interference signal length for each scan frame is 2000 sampling points. According to the Nyquist theorem, the highest detectable frequency for static pressure information is 25 Hz, and the highest detectable frequency for dynamic vibration information is 1 kHz.

[0035] Visual data acquisition process: Industrial-grade cameras are used to capture images of the object under test from different angles and distances, obtaining a visual dataset. Each time the robotic arm releases a sliding command, it captures an image of the contact surface between the silicone sleeve and the object. The image is projected onto an image sensor through a lens, and the image sensor with a CFA (Computer-Aided Image Sensor) acquires the raw image data.

[0036] The second step is to create a tactile-visual dataset.

[0037] Tactile data processing: The signals acquired in the first step cannot be used directly and need to be demodulated to obtain pressure and vibration information. The wavelength of the probe light output by the sweeping light source module is linearly related to time and is scanned at a fixed repetition period. Within one scanning period, when the wavelength of the sweeping light source reaches the reflection wavelength of the fiber optic grating, the probe light and the reference light interfere at the 2×2 coupler. After one scanning period, an interference signal of one scanning frame is formed and transmitted as an electrical signal through the photodetector 5. The interference signal g(λ) finally acquired by the data acquisition card 6 can be expressed by formula (1):

[0038]

[0039] Here, R(λ) represents the reflection spectrum of the fiber grating, reflecting the outer envelope characteristics of the interference signal. The envelope R(λ) contains strain and temperature information. When temperature or pressure strain causes a change in the center wavelength of the fiber grating, a shift in the envelope of the interference signal can be observed in each scan frame, such as... Figure 4 As shown in the left figure. L is the optical path difference of the grating, and n is the effective refractive index of the optical fiber. Vibrations can induce phase modulation of coherent signals during light wave propagation, such as... Figure 4 As shown in the right figure, R(λ) contains the required pressure information, while It contains vibration information.

[0040] Pressure information demodulation processing: First, the interference signal corresponding to the scan frame when the robotic arm is not in contact with the object is used as the reference signal. To prevent noise from affecting the extraction of the interference signal envelope, Gaussian smoothing is applied to the interference signals of all scan frames. The envelope of the interference signal is extracted using Hilbert transform. Finally, cross-correlation is used to calculate the offset between the signal envelope extracted from each frame and the reference signal envelope; this offset is the measure representing the static pressure information.

[0041] Vibration information demodulation processing: The Hilbert transform is performed on the interference signal to extract its envelope, and the extracted signal envelope is used to normalize the original interference signal. Wavelet packet time-frequency analysis is then performed on the normalized interference signal to demodulate its instantaneous frequency. The waveform of this instantaneous frequency is the waveform of the vibration signal in one scan frame. The variance of the vibration signal in each scan frame is calculated as a measure of the vibration intensity detected in that frame.

[0042] It should be noted that the time granularity of the pressure information and vibration information collected in the experiment is different. The time granularity of the pressure information depends on the scanning rate of the wavelength scanning light source, while the dynamic information is continuous.

[0043] Visual data processing: The original image data undergoes the following preprocessing: After color interpolation, image data represented by RGB is obtained. Using the OpenCV tool, the original image is resized to 224*224 pixels. An interpolation algorithm is used to smoothly scale the image while preserving its content. The original image pixel values ​​are divided by 255 to normalize them to a range between 0 and 1, ultimately yielding the image data required for this experiment.

[0044] Explanation of the relationship between visual images and object surfaces: Object surfaces exhibit characteristics such as texture, reflection, boundaries, and shape variations in images. These characteristics can be used to some extent to judge the roughness or hardness of an object's surface. However, the characteristics observed in images can only provide some hints or clues to infer the roughness or hardness of an object. A comprehensive judgment requires combining tactile and slip data. In the experiment, silicone blocks with different surface conditions were used to represent specific objects. The initial hardness and roughness (smoothness) of these silicone blocks were set manually. Therefore, after collecting visual information from different silicone blocks, the images were labeled with the corresponding hardness and roughness (smoothness), thus establishing a connection between the visual image and the specific surface conditions.

[0045] Each sample in this dataset can be described as a quadruple Di = {Vi, Ti, Fi, Ri}. Here, V represents the visual image of the object, acquired by a camera and assisted by tactile feedback for classification. T represents the vibration data obtained from contacting the object's surface, derived from the frequency variance demodulated by the interference signal, reflecting the smoothness or roughness of the surface. F represents the pressure data obtained from contacting the object's surface; this data is a time series, its granularity controlled by the scanning rate frame of the scanning light source, reflecting the hardness or softness of the surface. R represents the label of the specific surface condition, derived from different silicone blocks simulating the object's surface in the experiment, where i ∈ (1, N), and N represents the number of samples.

[0046] The third step is to train the object recognition neural network model using the prepared dataset.

[0047] 1) Feature extraction and mapping

[0048] RseNet, CNN, and LSTM networks were used to extract features from V, T, and F, respectively.

[0049] C V =E V (V) (2)

[0050] C T =E T (T) (3)

[0051] C F =E F(F) (4)

[0052] in These represent the features of the image, vibration data, and pressure data, respectively; E V E T and E F These represent feature extractors for image, vibration data, and pressure data, respectively; where d v d t d f These represent the feature dimensions of the image, vibration data, and pressure data, respectively.

[0053] Using (5)-(7), the features are mapped to obtain image representation, vibration data representation, and pressure data representation, respectively.

[0054] P V =M V (C V (5)

[0055] P T =M T (C T (6)

[0056] P F =M F (C F (7)

[0057] Among them, P V P T P F M represents the representation of image, vibration data, and pressure data, respectively. V M T M F These represent learnable feature mapping modules for image, vibration, and pressure data, respectively. V P T , d k This represents the dimension of the representation.

[0058] P S =S(P T P F (8)

[0059] Next, the vibration and pressure data from the tactile sensing are fused, whereby... S represents the feature representation of the fusion, and S represents the feature fusion module.

[0060] Using a ResNet50 network pre-trained on ImageNet as the image feature extractor, the output feature maps of the last residual block are averaged to obtain the image features C. VVibration features C were extracted using an LSTM network. T and pressure characteristics C F .

[0061] After extracting image, vibration, and pressure data, a feature mapping module is designed to further map these features. This feature mapping module for image, vibration, and pressure data is used to extract meaningful feature representations from the input data. After passing the features through the feature mapping module, P is obtained. V P T P F As shown in (7)-(9).

[0062] 2) Feature fusion

[0063] First, the vibration data layer and pressure data layer features are fused into a total tactile feature, such as... Figure 5 As shown. First, P T and P F They are connected. They pass through a fully connected layer with equal input and output dimensions. Then, they are activated by a sigmoid function to obtain a weight vector. Finally, the first half of the weight vector and P are... T Perform the Hadamard product operation, and perform the second half and P. F Perform the Hadamard product operation. The output is the sum of the two Hadamard products.

[0064] A1 = Up_half(Sigmoid1(CON(P) T ,P F )))) (9)

[0065] A2=Down_half(Sigmoid(L1(CON(P T ,P F )))) (10)

[0066] S(P T ,P F ) = P T ⊙A1+P F ⊙A2 (11)

[0067] L1 and L2 represent two fully connected layers, where Up_half represents obtaining the upper half of the vector and Down_half represents obtaining the lower half of the vector.

[0068] 3) Classifier

[0069] The output is then concatenated with the image feature output to obtain the total feature of the item. The obtained fused features are then input into a softmax layer to produce the corresponding classification and recognition result. This process can be represented as follows:

[0070] h = tanh(W) v ·P v +W s ·P S +b) (12)

[0071] Among them W v Represented as visual weight, W s denoted as the total tactile data weight, b represents the bias. This indicates the overall integration characteristics.

[0072] p = softmax(W h ·h+b h (13)

[0073] Among them W h b represents the weight of the total fusion feature. h Representing the bias, p is a probability vector where each dimension of p represents the probability of belonging to the corresponding class.

[0074] 4) Calculate the network's loss function.

[0075] During network training, the cross-entropy loss function is used as a regularization method to prevent overfitting. The loss function is calculated as follows:

[0076]

[0077] Where p i =(p1,p2,…,p k ), identify the probability distribution of a sample belonging to each category, and the actual label is represented as y. i =(y1,y2,…,y k The distribution is denoted by a matrix (where only one element is 1 and the rest are 0). By minimizing the cross-entropy loss function, the model can better approximate the true distribution, thereby improving classification performance.

[0078] The fourth step is to use the trained model to identify objects.

[0079] (1) For the object to be identified, the system collects its tactile and visual data, processes them separately, and then uses them as input to the recognition network model.

[0080] (2) Calculate the accuracy of object recognition by this model and evaluate the recognition performance of the system on object surfaces.

[0081] Table 1 ResNet50 network structure parameters

[0082]

[0083]

[0084] The network structure parameters are detailed in Table 1.

Claims

1. A deep learning-based visual-tactile multimodal object recognition method, characterized in that, Includes the following steps: The first step is to collect tactile and visual data, as follows: The fiber Bragg grating tactile sensor is encapsulated in a silicone sleeve, which is placed at the end of a robotic arm. The robotic arm's trajectory is controlled by a program. During tactile signal acquisition, the silicone sleeve containing the fiber Bragg grating serves as the sensing area and directly contacts the object being measured. When the silicone sleeve containing the tactile sensor contacts the object and is pressed down, the pressure acting on the silicone sleeve causes tangential deformation of the fiber Bragg grating, resulting in a change in its center wavelength. When the silicone sleeve containing the tactile sensor slides and rubs against the object, vibrations are induced between the silicone sleeve and the contact surface and transmitted to the fiber Bragg grating. The center wavelength information of the fiber Bragg grating's reflection spectrum is modulated in the interference signal of each scan frame in the form of a signal envelope, which is used to demodulate the static pressure data detected by the tactile sensor. The dynamic signal of the vibration will be coupled into the interference signal of the scanning frame in the form of instantaneous frequency changes, in order to demodulate the dynamic vibration data of the tactile sensor; Tactile signals are acquired using a fiber optic grating tactile sensor. Visual data acquisition process: The camera is used to take pictures of the object under test from different angles and distances to obtain a visual dataset; when the silicone sleeve equipped with a tactile sensor slides and rubs against the object, the contact surface between the silicone sleeve and the object is photographed to acquire raw image data. The second step is to create a tactile-visual dataset, as follows: (1) Tactile and slip data processing: The tactile and slip signals collected in the first step are demodulated to obtain pressure data and vibration data respectively; (2) Visual data processing: The original image data is preprocessed to obtain the required visual image; (3) Acquisition of visual image labels: Set up several silicone blocks with different surface hardness and roughness. After acquiring visual images of different silicone blocks, label the visual images with the relevant surface hardness and roughness to realize the connection between the visual images and the specific conditions of the object surface. (4) Create a touch-slip-visual dataset. Each sample in the touch-slip-visual dataset is described as a quadruple Di = {Vi, Ti, Fi, Ri}; where V represents the visual image of the object, T represents the vibration data obtained by contacting the object surface to reflect the smoothness or roughness of the object surface, F represents the pressure data obtained by contacting the object surface, the granularity of which is controlled by the scanning rate frame of the scanning light source to reflect the softness or hardness of the object surface; R represents the label of the specific condition of the object surface, i ∈ (1, N), and N represents the number of samples. The third step is to build and train an object recognition neural network model.

2. The visual-tactile multimodal object recognition method according to claim 1, characterized in that, The second step (1) includes pressure data demodulation processing, taking the interference signal corresponding to the scan frame when the robot arm and the object are not in contact as the reference signal; performing Gaussian smoothing on the interference signal of all scan frames; extracting the envelope of the interference signal through Hilbert transform; calculating the offset between the signal envelope extracted from each frame and the reference signal envelope using cross-correlation, and using this offset as a measure of the pressure data.

3. The visual-tactile multimodal object recognition method according to claim 1, characterized in that, The second step (1) includes vibration data demodulation processing: the Hilbert transform is performed on the interference signal to extract its envelope, and the extracted signal envelope is used to normalize the original interference signal; the wavelet packet time-frequency analysis transform is performed on the normalized interference signal to demodulate its instantaneous frequency, and the waveform of the instantaneous frequency is the waveform of the vibration signal in a scanning frame; the variance of the vibration signal in each scanning frame is calculated as a measure of the intensity of the vibration signal detected in that scanning frame.

4. The visual-tactile multimodal object recognition method according to claim 1, characterized in that, The object recognition neural network model employs a feature-level fusion method. Features are extracted from pressure and vibration data separately and then fused to explore the relationship between the two types of data, yielding tactile and slippery features. Features extracted from the visual image are then concatenated with the tactile and slippery features to obtain the total multimodal fusion features. These total features are then input into a softmax layer for classification tasks. The cross-entropy loss function is used to evaluate the performance of the object recognition neural network model.

Images