Tactile sensor contact force classification and numerical prediction method and storage medium

Abstract

The application provides a haptic sensor contact force classification and numerical prediction method, comprising: acquiring a mechanical information label and a picture, and combining and preprocessing the mechanical information label and the picture to construct a mechanical deep learning data set; and performing feature extraction on the mechanical deep learning data set based on a multi-scale deep learning residual network to perform contact force category and mechanical information prediction. The method can perform feature extraction on images of different scales, and finally input the extracted features into the multi-scale deep learning residual network for contact force classification and mechanical numerical prediction. The haptic sensing system and computer scale storage medium applying the haptic sensor contact force classification and numerical prediction method can realize expansion of the range of the mechanical sensor, and at the same time improve the accuracy of the mechanical sensor.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of human-computer interaction, and particularly relates to a touch sensor contact force classification and numerical prediction method and a storage medium. BACKGROUND

[0002] The fingertip touch sensing technology of a robot plays a crucial role in the design of an industrial robot and a bionic robot. Among them, the flexible touch sensor based on optics and the flexible touch sensor based on optical fiber transmission are the mainstream direction of current robot fingertip touch research.

[0003] The advantage of this sensor compared with the flexible touch sensor based on optics is that the camera is moved to the far end of the contact point, which can make the volume of the sensor smaller. The numerical prediction method of the conventional such sensor is an ordinary Alex convolution network, which has the advantages of small volume and simplicity. However, the mechanical numerical prediction accuracy of the network is low, so the mechanical sensing range and accuracy of the sensor calibrated based on the algorithm are low, which is very unfriendly to the interaction between the robot and the object. SUMMARY

[0004] Therefore, in order to overcome the defects of the prior art, the present application provides a touch sensor contact force classification and numerical prediction method, system and storage medium.

[0005] A touch sensor contact force classification and numerical prediction method, characterized in that it comprises:

[0006] obtaining a mechanical information label and a picture, combining and preprocessing the mechanical information label and the picture to construct a mechanical deep learning data set;

[0007] performing feature extraction on the mechanical deep learning data set based on a multi-scale deep learning residual network to perform contact force category and mechanical information prediction.

[0008] The "obtaining a mechanical information label and a picture, combining and preprocessing the mechanical information label and the picture to construct a mechanical deep learning data set" comprises:

[0009] performing automatic data acquisition of normal pressure, shear force and torsion;

[0010] According to the touch sensing device, the mechanical sensor is used to collect the mechanical information in real time, and the image acquisition device is used to collect the corresponding picture to establish the original mechanical deep learning data set.

[0011] Further, the temperature drift check point is collected at the same time as the "mechanical information collected by the touch sensing device and the corresponding picture collected by the image acquisition device".

[0012] According to the temperature drift check point, the mechanical information in the original mechanical deep learning data set is corrected to construct a corrected mechanical deep learning data set.

[0013] Specifically, the "feature extraction of the mechanical deep learning data set based on the multi-scale deep learning residual network for contact force type and mechanical information prediction" includes:

[0014] A multi-scale residual network is constructed, and random network parameter initialization is performed.

[0015] The multi-scale residual network is iteratively trained to make the multi-scale residual network converge, and the obtained converged multi-scale residual network is saved as a multi-scale deep learning residual network.

[0016] The multi-scale deep learning residual network is used to predict the mechanical information of the mechanical deep learning data set.

[0017] The multi-scale residual network structure in the "construction of a multi-scale residual network" is composed of 34 convolutional blocks.

[0018] It starts with a feature extraction block composed of 7x7 convolutional layers, 9x9 convolutional layers, 11x11 convolutional layers, and 13x13 convolutional layers.

[0019] Then a stack of convolutional blocks composed of 1x1 convolutional layers and 3x3 convolutional layers is formed, and residual calculations are performed between multiple convolutional blocks.

[0020] The "iterative training of the multi-scale residual network to make the multi-scale residual network converge" includes:

[0021] The mechanical deep learning data set is input into the multi-scale residual network, and forward propagation is performed to obtain the classification prediction of the contact force and the mechanical information prediction value.

[0022] According to the classification prediction of the contact force and the mechanical information prediction value, a loss function is calculated.

[0023] According to the cross function, the gradient of the multi-scale residual network is updated, and the weight of the multi-scale residual network is updated.

[0024] The "calculation of the loss function according to the classification prediction of the contact force and the mechanical information prediction value" includes:

[0025] The cross-entropy loss function is used to calculate the classification prediction of the contact force to obtain the distance between the classification prediction of the contact force and the actual classification of the contact force.

[0026] The mechanical information prediction value is calculated by using a mean square error loss function to obtain the distance between the mechanical information prediction value and the actual value of the mechanical information.

[0027] The gradient update of the multi-scale residual network according to the cross function comprises:

[0028] According to the formula The hypothesis function of the linear regression function is calculated, wherein h θ The hypothesis function of the linear regression function is calculated, wherein h j The function variable is x j The input parameter is n, and the number of input parameters is n;

[0029] The loss function corresponding to the hypothesis function of the linear regression function is: Wherein, m is the total amount of data, and i is the number of data groups;

[0030] The derivative of the loss function is:

[0031] The gradient iteration direction is:

[0032] The application also provides a tactile sensing system, comprising: a tactile sensing device and a chip, the tactile sensing device comprising a mechanical sensor and an image acquisition device, the mechanical sensor and the image acquisition device are electrically connected with the chip.

[0033] The chip comprises a programmable logic circuit and / or program instructions, and the chip is used for executing the tactile sensor contact force classification and value prediction method when running.

[0034] The application also provides a computer readable storage medium, which stores a computer program, and the program is executed by a processor to realize the tactile sensor contact force classification and value prediction method.

[0035] In summary, the tactile sensor contact force classification and value prediction method, system and storage medium have the following beneficial effects: the tactile sensor contact force classification and value prediction method adds different scale convolution blocks in front of the traditional residual network, thereby establishing a multi-scale residual network, which can extract different scale features of the image, and finally input the extracted features into the multi-scale residual network for contact force classification and mechanical value prediction. According to experiments, the multi-scale residual network can improve the accuracy of the mechanical sensor while expanding the range of the mechanical sensor.

[0036] The tactile sensor contact force classification and numerical prediction method of the application can realize the expansion of the range of the mechanical sensor and improve the accuracy of the mechanical sensor. BRIEF DESCRIPTION OF DRAWINGS

[0037] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0038] Figure 1 The flowchart of the tactile sensor contact force classification and numerical prediction method of the present application is shown.

[0039] Figure 2 The flowchart of the tactile sensor contact force classification and numerical prediction experiment of embodiment 2 is shown.

[0040] Figure 3 The schematic diagram of the experimental platform of embodiment 2 is shown.

[0041] Figure 4 The schematic diagram of the distribution and basic information of the collected data of embodiment 2 is shown.

[0042] Figure 5 The structure diagram of the multi-scale deep learning residual network of the present application is shown.

[0043] Figure 6 The result comparison diagram of the multi-scale deep learning residual network of the present application and other networks in the same data set for contact force classification and numerical prediction is shown.

[0044] Figure 7 The time comparison diagram of the multi-scale deep learning residual network of the present application and other networks for processing one picture is shown. DETAILED DESCRIPTION

[0045] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some embodiments of the present application, not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0046] Embodiment 1

[0047] The present application provides a tactile sensor contact force classification and numerical prediction method, which adopts a tactile sensing device based on optical fiber transmission to collect data and perform classification.

[0048] Referring to the accompanying drawings Figure 1 The tactile sensor contact force classification and value prediction method of the embodiment includes:

[0049] A: Obtain the mechanical information label and the corresponding picture, combine and preprocess the mechanical information label and the corresponding picture to construct a mechanical deep learning data set.

[0050] Specifically, it includes:

[0051] A1: Perform automatic data acquisition of normal pressure, shear force and torque.

[0052] Data acquisition programming is performed based on LabVIEW to realize automatic acquisition of normal pressure, shear force and torque.

[0053] A2: According to the mechanical information collected by the mechanical sensor carried by the tactile sensing device and the corresponding picture collected by the image acquisition device, an original mechanical deep learning data set is established.

[0054] In this embodiment, the tactile sensing device is a mechanical arm, and the mechanical sensor for collecting mechanical information carried on the surface of the mechanical arm is a six-axis torque sensor, and the image acquisition device is an industrial CCD camera. When the mechanical arm performs mechanical information collection work, the mechanical arm is driven to the target data point, and then the mechanical sensor collects the mechanical information, at the same time, the image acquisition device collects the corresponding picture of the mechanical information to complete the collection of one data point; and collect temperature drift check points. After completing the collection of one target data point, drive the mechanical arm to move to the initial position to correct the temperature drift. In this way, the collection of multiple target data points is realized.

[0055] Further, while the mechanical arm is correcting the temperature drift, the collected mechanical information of the target data point and the corresponding image are arranged into a standard data set format for deep learning. The mechanical information obtained for multiple target data points and the corresponding pictures are constructed to form a mechanical deep learning data set.

[0056] Then, according to the temperature drift detection point, the corresponding mechanical information in the mechanical deep learning data set is corrected, and the corrected mechanical information and the corresponding picture are constructed to form a corrected mechanical deep learning data set.

[0057] B: Based on the multi-scale deep learning residual network, the corrected mechanical deep learning data set is feature extracted to predict the contact force type and the mechanical information.

[0058] B1: Construct a multi-scale residual network and initialize random network parameters.

[0059] The structure of the multi-scale residual network is composed of 34 convolutional blocks, which are initially composed of a 7x7 convolutional layer, a 9x9 convolutional layer, an 11x11 convolutional layer, and a 13x13 convolutional layer to form a feature extraction block, and then 32 stacks of a convolutional block composed of one 1x1 convolutional layer, one 3x3 convolutional layer, and one 1x1 convolutional layer are formed. At the same time, residual calculation is performed between multiple convolutional blocks, that is, the output of the previous convolutional block is combined with the output of the current convolutional block and output to the next convolutional block.

[0060] The residual calculation between the multiple convolutional blocks of the multi-scale residual network can be performed using the formula F(x) = H(x) - x, where H(x) is the output when the input of the convolutional block is x, and F(x) is the residual.

[0061] B2: iteratively training the multi-scale residual network to make the multi-scale residual network converge, and saving the obtained converged multi-scale residual network as the multi-scale deep learning residual network.

[0062] B21: inputting the mechanical deep learning data set into the multi-scale residual network, performing forward propagation, and obtaining the classification prediction of the contact force and the prediction value of the mechanical information.

[0063] B22: calculating the loss function according to the classification prediction of the contact force and the prediction value of the mechanical information.

[0064] Specifically, for the contact force classification prediction, a cross-entropy loss function is used to calculate the distance between the classification prediction of the contact force and the actual classification of the contact force. The cross-entropy loss function can be represented as: where L is the cross-entropy loss function, N is the total number of samples, y i represents the label of sample i, p i represents the probability that sample i is predicted as a positive class.

[0065] For the mechanical information prediction value, a mean square error loss function is used to calculate the distance between the mechanical information prediction value and the actual value of the mechanical information. The mean square error loss function can be represented as: where J is the mean square error loss function, is the output result of the multi-scale residual network, x i is the true value.

[0066] B23: performing gradient update on the multi-scale residual network according to the cross function, updating the weight value of the multi-scale residual network, and saving the obtained multi-scale residual network as the multi-scale deep learning residual network. Wherein, SGD is used as the optimizer.

[0067] According to the formula the hypothesis function of the linear regression function is calculated, where h θThe hypothesis function is a linear regression function, θ j The function variable is x j The input parameter is n, and the number of input parameters is n.

[0068] The loss function corresponding to the hypothesis function of the linear regression function is: Where m is the total amount of data, and i is the number of data groups.

[0069] The derivative of the loss function is:

[0070] The gradient iteration direction is:

[0071] B3: Using a multi-scale deep learning residual network to predict mechanical information of a mechanical deep learning data set.

[0072] Embodiment 2

[0073] The embodiment provides a method for establishing a mechanical deep learning data set by using the tactile sensor contact force classification and numerical prediction method of embodiment 1, and performing a tactile sensor contact force classification and numerical prediction experiment. As shown in the accompanying drawings of the specification, Figure 2 The experimental process includes:

[0074] The functional requirements of the tactile sensor are determined, and then a sensor manufacturing mold is prepared by using a 3D printing technology through a fiber sensor preparation process. The sensor is prepared by a practical mold method based on a fiber conduction mechanical arm. The preparation material can be selected from flexible materials such as silicone resin.

[0075] A data acquisition platform is formed by using a fiber conduction mechanical arm and a horizontal operation table, and a sensor mounting platform is made by a lathe process to collect information by the sensor. The final experimental platform formed by mounting is shown in Figure 3 The experimental platform includes a camera, a light source, a fiber array, a mechanical sensor, and a mechanical arm. The mechanical arm is used to carry the mechanical sensor and mechanical programming.

[0076] In this embodiment, the fiber array includes 121 single-mode optical fibers, including 61 white single-mode optical fibers for guiding the light source into the sensor and 60 black single-mode optical fibers for guiding the scattered light inside the sensor to the camera end to be captured by the camera. Specifically, the white fiber guided light source is an LED brightness adjustable bulb equipped with a cooling device, and the camera captured picture is shown in the accompanying drawings of the specification Figure 3The mechanical sensor is a six-axis torque sensor that can output real-time mechanical information such as fx, fy, fz, etc. The mechanical sensor is also used as a platform for bearing the indenter, and the function of the indenter is to contact the flexible sensor to deform it. Two types of indenters are provided on the mechanical sensor, and the indenters are used to interact with the mechanical sensor. The 3mm diameter indenter in the two types of indenters is used to collect normal pressure, and the 20mm diameter indenter is used to collect shear force and torque.

[0077] Using the experimental equipment and experimental platform with the above structure, data collection is performed according to a predetermined data collection scheme. The data collection content includes collecting the type, position range and size of the contact force.

[0078] Specifically, LabVIEW programming is used to realize automatic acquisition of pictures, and the corresponding mechanical information tags of the pictures are output. In this embodiment, the collected data distribution is as shown in Figure 4 In the normal pressure experiment, 1250 positions are collected, and 8 depth data points are collected at each position, a total of 10000 data points. In the shear force experiment, 1000 positions are collected, and 2 depth data points are collected at each position, a total of 2000 data points. The specific collection method is that the mechanical sensor first reaches the preset depth, and then moves to 1000 different positions parallel to the sensing surface. In the torque experiment, 2 positions are collected, and 4 depth, 150 angle data points are collected at each position, a total of 1200 contact points. When the mechanical sensor reaches the preset data collection position and depth, LabVIEW obtains the mechanical information output by the mechanical sensor and the real-time acquired image. After the data point collection is completed, the original data set is formed. In addition, after the data collection at each collection point is completed, the mechanical arm is driven to move to the initial position to correct the temperature drift of the mechanical sensor. While performing temperature drift correction, the original data set is arranged into a standard data set format for deep learning.

[0079] Then the original data set is divided into three parts for deep learning training, testing and evaluation to form a mechanical deep learning data set. Among them, 70% of the original data set is used for deep learning training, 15% is used for deep learning testing, and 15% is used for deep learning verification.

[0080] The mechanical deep learning dataset is processed, unnecessary pictures and mechanical information labels are deleted, that is, information unnecessary for the mechanical deep learning dataset collected in the mechanical information collection process is deleted, such as the torque information of the motor output when the six-axis torque sensor is used for mechanical information collection in the embodiment. Then, the corresponding mechanical information in the mechanical deep learning dataset is corrected according to the temperature drift detection point, and the corrected mechanical information and the corresponding picture are used to form a corrected mechanical deep learning dataset.

[0081] The mechanical sensor is calibrated by using different deep learning networks, the last deep learning network with the best effect is selected, and the network is further optimized. The multi-scale deep learning residual network designed in the embodiment is as shown in Figure 5 The image is convolved using four convolution kernels of different sizes to extract feature information of different scales, and a structure similar to Resnet is used to obtain higher features so that the multi-scale deep learning residual network can extract more features to obtain better results. Finally, a 7-output connection layer is used to output the position and size of the contact force. A full connection layer is used to connect a 1-dimensional output sigmoid function to output the type of contact force. 1 is normal force, and 0 is shear force and torsional force.

[0082] The multi-scale residual network of the present application and other deep learning networks are used to predict the mechanical information of the mechanical deep learning dataset and evaluate the results.

[0083] The mean and variance are used as evaluation indexes for mechanical information prediction: Where m is the total amount of data, y i is the network prediction output value, and y true is the true mechanical information value.

[0084] The results are shown in the accompanying drawings of the specification Figure 6 Under the same dataset, the multi-scale deep learning residual network of the present application has the best overall effect and can improve the accuracy of the sensor while expanding the range of the mechanical sensor.

[0085] Further, to verify whether the multi-scale deep learning residual network of the present application can process video streams, a test on pictures is also conducted. The multi-scale residual network of the present application and other deep learning networks are used to process the same picture, and the results are shown in the accompanying drawings of the specification Figure 7 The multi-scale residual network of the present application can achieve 30 frames per second and can realize video stream processing, meeting the actual application requirements.

[0086] Other contents same as those in embodiment 1 are not repeated here.

[0087] Embodiment 3

[0088] The embodiment provides a tactile sensing system, comprising a tactile sensing device and a chip, the tactile sensing device comprising a mechanical sensor and an image acquisition device, and the mechanical sensor and the image acquisition device are electrically connected with the chip.

[0089] The chip comprises a programmable logic circuit and / or program instructions, and the chip is used for executing the tactile sensor contact force classification and value prediction method as described in Embodiment 1 when running.

[0090] Other contents same as Embodiment 1 are not repeated here.

[0091] Embodiment 4

[0092] The embodiment provides a computer readable storage medium, and a computer program is stored on the computer readable storage medium, and the program is executed by a processor to realize the tactile sensor contact force classification and value prediction method as described in Embodiment 1.

[0093] Other contents same as Embodiment 1 are not repeated here.

[0094] In summary, the tactile sensor contact force classification and value prediction method provided by the embodiment adds different scale convolution blocks in front of a traditional residual network, thereby establishing a multi-scale residual network, which can extract features of different scales for an image, and finally input the extracted features into the multi-scale residual network for contact force classification and mechanical value prediction. According to experiments, the multi-scale residual network can improve the accuracy of the mechanical sensor while expanding the range of the mechanical sensor.

[0095] The tactile sensing system and the computer readable storage medium using the tactile sensor contact force classification and value prediction method of the embodiment can expand the range of the mechanical sensor and improve the accuracy of the mechanical sensor at the same time.

[0096] The above only describes the preferred embodiments of the present application, and does not limit the present application. In addition to the above embodiments, different variants can be provided, and the technical features of the above embodiments can be combined with each other. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A method for classifying and numerically predicting contact force of a tactile sensor, characterized in that, include: Obtain mechanical information labels and images, and combine and preprocess the mechanical information labels and images to construct a mechanical deep learning dataset; Feature extraction is performed on the aforementioned deep learning dataset of mechanics based on a multi-scale deep learning residual network to predict the types of contact forces and mechanical information.

2. The method for classifying and numerically predicting contact force of a tactile sensor according to claim 1, characterized in that, The step of "obtaining mechanical information labels and images, combining and preprocessing the mechanical information labels and images to construct a mechanical deep learning dataset" includes: Automated acquisition of data on normal force, shear force, and torque; The tactile sensing device collects mechanical information in real time using its mechanical sensors, and the image acquisition device collects corresponding images to build an original mechanical deep learning dataset.

3. The method for classifying and numerically predicting contact force of a tactile sensor according to claim 2, characterized in that, While "collecting mechanical information in real time based on the mechanical sensor on the tactile sensing device and collecting corresponding images from the image acquisition device", temperature drift checkpoints are also collected. The mechanical information in the original mechanical deep learning dataset is corrected based on the temperature drift checkpoint to construct a corrected mechanical deep learning dataset.

4. The method for classifying and numerically predicting contact force of a tactile sensor according to claim 1, characterized in that, The phrase "using a multi-scale deep learning residual network to extract features from the mechanical deep learning dataset to predict contact force types and mechanical information" includes: Construct a multi-scale residual network and initialize the random network parameters; The multi-scale residual network is iteratively trained to make it converge, and the converged multi-scale residual network is saved as a multi-scale deep learning residual network. The multi-scale deep learning residual network is used to predict mechanical information in the mechanical deep learning dataset.

5. The method for classifying and numerically predicting contact force of a tactile sensor according to claim 4, characterized in that, The multi-scale residual network structure in the "construction of multi-scale residual network" consists of 34 convolutional blocks; Initially, the feature extraction blocks consist of 7×7 convolutional layers, 9×9 convolutional layers, 11×11 convolutional layers, and 13×13 convolutional layers; Then, convolutional blocks consisting of 1×1 convolutional layers and 3×3 convolutional layers are stacked together, and residual calculations are performed between multiple convolutional blocks.

6. The method for classifying and numerically predicting contact force of a tactile sensor according to claim 4, characterized in that, The phrase "iteratively training the multi-scale residual network to make the multi-scale residual network converge" includes: The mechanical deep learning dataset is input into the multi-scale residual network for forward propagation to obtain the classification prediction of contact force and the mechanical information prediction value. Calculate the loss function based on the classification prediction of the contact force and the predicted value of the mechanical information; The gradient of the multi-scale residual network is updated based on the cross function calculation, and the weights of the multi-scale residual network are also updated.

7. The method for classifying and numerically predicting contact force of a tactile sensor according to claim 6, characterized in that, The phrase "calculating the loss function based on the classification prediction of the contact force and the predicted value of the mechanical information" includes: The classification prediction of the contact force is calculated using the cross-entropy loss function to obtain the distance between the classification prediction of the contact force and the actual classification of the contact force; The mean squared error loss function is used to calculate the distance between the predicted mechanical information value and the actual mechanical information value.

8. The method for classifying and numerically predicting contact force of a tactile sensor according to claim 6, characterized in that, The phrase "calculating gradient updates for the multi-scale residual network based on the cross function" includes: According to the formula Calculate the hypothesis function of the linear regression function, where h θ Let θ be the hypothesis function of the linear regression function. j x is a function variable. j Here, n represents the number of input parameters. The loss function corresponding to the hypothesis function of the linear regression function is: Where m is the total amount of data and i is the number of data sets; Differentiating the loss function yields: The gradient iteration direction is:

9. A tactile sensing system, characterized in that, include: A tactile sensing device and a chip, wherein the tactile sensing device includes a force sensor and an image acquisition device, both of which are electrically connected to the chip; The chip includes programmable logic circuits and / or program instructions, and when the chip is run, it is used to execute the tactile sensor contact force classification and numerical prediction method as described in any one of claims 1-8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by the processor, the program implements the tactile sensor contact force classification and numerical prediction method as described in any one of claims 1-8.

Images