Gradient calibration method, device and medium for visual-tactile sensor based on deep learning

Abstract

The present invention discloses a deep learning-based visual-tactile sensor gradient calibration method, device, and medium. The method comprises: obtaining sensor image information of a preset sphere pressing on a target touch film of the visual-tactile sensor to be calibrated; inputting the sensor image information into a pre-trained deep learning segmentation model to obtain the sphere contact area of the target touch film; determining the coordinate values and RGB values of each point within the sphere contact area, and calculating the gradient information of each point within the sphere contact area based on the radius of the preset sphere; constructing a training data set based on the coordinate values, RGB values, and gradient information, and training a gradient recognition model based on the training data set; and calibrating the mapping relationship between the RGB light intensity and gradient of each point on the target touch film based on the gradient recognition model. The present invention improves the efficiency and accuracy of visual-tactile sensor gradient calibration and can be widely applied in the field of artificial intelligence technology.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a method, device, and medium for gradient calibration of a visual-tactile sensor based on deep learning. Background Art

[0002] The basic principle of visual-tactile sensors, such as Gelsight, is to calculate the height gradient generated by a point in two directions on a plane based on the amount of light reflected from the three different colors of RGB in the three directions. By integrating the gradient (solving the Poisson equation), the surface depth data can be obtained. The basic principle of this type of sensor design is that light is diffusely reflected from the surface of the touch film. When the film has undulations, the incident angle of the light during diffuse reflection on the film surface changes. Based on the change in the intensity of the light captured at that point, the surface normal vector of the film at that point, that is, the gradient direction, can be inferred. Therefore, calibration of this nonlinear relationship between light intensity and gradient is crucial for visual-tactile sensors.

[0003] Because sensors like Gelsight have assembly errors in the positions of the three color light sources and the film during manufacturing, and the reflectivity parameters of the film used in each sensor cannot be completely consistent, a light intensity to gradient calibration process is essential for each sensor before it leaves the factory or before the user uses it.

[0004] At present, the method for calibrating the light intensity to gradient is generally to press the film onto experimental molds with different gradients under single experimental conditions, observe the light intensity, and thus establish a mapping relationship between light intensity and gradient. This method requires professionals to complete it under good experimental conditions, and the completion process is relatively complicated. A method officially provided by Gelsight simplifies this experimental mold, that is, using a sphere with a known radius as the experimental mold. According to the radius of the sphere, the gradient of the point on the surface of the sphere can be solved. However, this method still requires professionals to mark the position of the sphere and record the data, which is inefficient and prone to errors. Summary of the Invention

[0005] The purpose of the present invention is to solve one of the technical problems existing in the prior art to at least a certain extent.

[0006] To this end, an object of an embodiment of the present invention is to provide a visual-tactile sensor gradient calibration method based on deep learning, which improves the efficiency and accuracy of visual-tactile sensor gradient calibration.

[0007] Another object of an embodiment of the present invention is to provide a visual-tactile sensor gradient calibration device based on deep learning.

[0008] In order to achieve the above technical objectives, the technical solutions adopted by the embodiments of the present invention include:

[0009] In one aspect, an embodiment of the present invention provides a method for gradient calibration of a visual-tactile sensor based on deep learning, comprising the following steps:

[0010] Obtaining sensor image information of a preset ball pressing on a target touch film of the visual-tactile sensor to be calibrated, inputting the sensor image information into a pre-trained deep learning segmentation model to obtain a contact area of the ball on the target touch film;

[0011] Determine the coordinate value and RGB value of each point in the contact area of the sphere, and calculate the gradient information of each point in the contact area of the sphere according to the radius of the preset sphere;

[0012] Constructing a training data set according to the coordinate values, the RGB values, and the gradient information, and training a gradient recognition model according to the training data set;

[0013] The mapping relationship between the RGB light intensity and the gradient of each point on the target touch film is calibrated according to the gradient recognition model.

[0014] Furthermore, in one embodiment of the present invention, the deep learning segmentation model is trained by the following steps:

[0015] Acquire a plurality of sensor sample images of the preset ball pressing at different positions on the sample touch film with different forces;

[0016] Segmenting and labeling the sensor sample images to determine contact area labels corresponding to each sensor sample image;

[0017] Inputting the sensor sample image into a pre-built deep learning neural network to obtain a predicted contact area;

[0018] determining a first loss value according to the predicted contact area and the contact area label;

[0019] The parameters of the deep learning neural network are updated according to the first loss value to obtain the trained deep learning segmentation model.

[0020] Furthermore, in one embodiment of the present invention, the step of calculating the gradient information of each point within the contact area of the sphere according to the radius of the preset sphere specifically includes:

[0021] determining an actual bending radius according to the radius of the preset sphere and the thickness of the target touch film;

[0022] Fitting the sphere contact area to obtain a fitted circular area, and determining the center position of the fitted circular area;

[0023] The gradient information of each point in the contact area of the sphere is calculated according to the center position of the circle and the actual bending radius.

[0024] Furthermore, in one embodiment of the present invention, the gradient information is calculated by the following formula:

[0025]

[0026]

[0027] in, and They represent the gradient values of the point (x, y) in the x direction and the y direction respectively, (x0, y0) represents the center position of the circle, and R represents the actual bending radius.

[0028] Furthermore, in one embodiment of the present invention, constructing a training data set according to the coordinate values, the RGB values, and the gradient information, and training a gradient recognition model according to the training data set specifically includes:

[0029] Determining an x coordinate and a y coordinate according to the coordinate values, determining an R value, a G value, and a B value according to the RGB values, and determining a training sample according to the R value, the G value, the B value, the x coordinate, and the y coordinate;

[0030] Determine an x-direction gradient value and a y-direction gradient value according to the gradient information, and determine a gradient label corresponding to the training sample according to the x-direction gradient value and the y-direction gradient value;

[0031] Inputting the training sample into a pre-built ANN neural network to obtain a gradient recognition result;

[0032] Determine a second loss value according to the gradient recognition result and the gradient label;

[0033] The parameters of the ANN neural network are updated according to the second loss value to obtain the trained gradient recognition model.

[0034] Furthermore, in one embodiment of the present invention, the ANN neural network includes an input layer, a first hidden layer, a second hidden layer, a third hidden layer, a fourth hidden layer and an output layer, the input layer is used to input the training sample, the output layer is used to output the gradient recognition result, the first hidden layer, the second hidden layer, the third hidden layer and the fourth hidden layer include 64, 128, 128 and 64 neurons respectively.

[0035] Furthermore, in one embodiment of the present invention, calibrating the mapping relationship between the RGB light intensity and the gradient of each point on the target touch film according to the gradient recognition model specifically includes:

[0036] Dividing the target touch film into a plurality of grid points according to a preset spacing;

[0037] Determining the gradient of each grid point under different RGB light intensities according to the gradient recognition model, and establishing a mapping relationship according to the coordinate position of the grid point, the RGB light intensity, and the corresponding gradient;

[0038] A three-dimensional lookup table including coordinate values, RGB values, and gradient values is generated according to the mapping relationship.

[0039] On the other hand, an embodiment of the present invention provides a device for calibrating a visual-tactile sensor gradient based on deep learning, comprising:

[0040] A contact area recognition module is configured to obtain sensor image information of a preset ball pressing on a target touch film of the visual-tactile sensor to be calibrated, input the sensor image information into a pre-trained deep learning segmentation model, and obtain the contact area of the ball on the target touch film;

[0041] a gradient information calculation module, configured to determine the coordinate values and RGB values of each point within the contact area of the sphere, and calculate the gradient information of each point within the contact area of the sphere according to the radius of the preset sphere;

[0042] A model training module is used to construct a training data set according to the coordinate values, the RGB values and the gradient information, and train a gradient recognition model according to the training data set;

[0043] A calibration module is used to calibrate the mapping relationship between the RGB light intensity and the gradient of each point on the target touch film according to the gradient recognition model.

[0044] On the other hand, an embodiment of the present invention provides an electronic device, comprising a memory, a processor, a program stored on the memory and executable on the processor, and a data bus for implementing connection and communication between the processor and the memory. When the program is executed by the processor, the visual-tactile sensor gradient calibration method based on deep learning as described above is implemented.

[0045] On the other hand, an embodiment of the present invention further provides a storage medium, which is a computer-readable storage medium for computer-readable storage, and stores one or more programs, which can be executed by one or more processors to implement the deep learning-based visual-tactile sensor gradient calibration method as described above.

[0046] The advantages and benefits of the present invention will be described in part in the following description and will become apparent from the following description or learned through practice of the present invention:

[0047] An embodiment of the present invention obtains sensor image information of a preset sphere pressing on a target touch film of a visual tactile sensor to be calibrated, inputs the sensor image information into a pre-trained deep learning segmentation model, obtains the sphere contact area of the target touch film, determines the coordinate values and RGB values of each point in the sphere contact area, and calculates the gradient information of each point in the sphere contact area based on the radius of the preset sphere. A training data set is constructed based on the coordinate values, RGB values, and gradient information, and a gradient recognition model is obtained by training based on the training data set. The mapping relationship between the RGB light intensity and the gradient of each point on the target touch film is calibrated based on the gradient recognition model. An embodiment of the present invention uses a pre-trained deep learning segmentation model to identify the sphere contact area, constructs a training data set based on the coordinate values, RGB values, and gradient information of each point in the sphere contact area, and trains to obtain a gradient recognition model. The gradient recognition model is used to complete the gradient calibration of each point on the target touch film, thereby improving the efficiency and accuracy of the gradient calibration of the visual tactile sensor. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following introduction is made to the drawings required for use in the embodiments of the present invention. It should be understood that the drawings introduced below are only for the convenience of clearly describing some embodiments of the technical solutions of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without any creative work.

[0049] Figure 1 A flowchart of the steps of a deep learning-based visual-tactile sensor gradient calibration method provided by an embodiment of the present invention;

[0050] Figure 2 A flowchart of the steps for training a deep learning segmentation model provided by an embodiment of the present invention;

[0051] Figure 3 A flowchart of step S102 provided in an embodiment of the present invention;

[0052] Figure 4 A flowchart of step S103 provided in an embodiment of the present invention;

[0053] Figure 5 A flowchart of step S104 provided in an embodiment of the present invention;

[0054] Figure 6A schematic diagram of the structure of a deep learning-based visual-tactile sensor gradient calibration device provided by an embodiment of the present invention;

[0055] Figure 7 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present invention;

[0056] Figure 8 A schematic diagram of the structure of a storage medium provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0057] The embodiments of the present invention are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements with the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application, and are not to be construed as limitations on the present application. It should be noted that, although the functional modules are divided in the system schematic and the logical order is shown in the flow chart, in some cases, the steps shown or described may be performed in a different order than the module division in the system schematic or the order in the flow chart. For the step numbers in the following embodiments, they are only provided for the convenience of explanation, and no limitation is placed on the order between the steps. The execution order of each step in the embodiment can be adaptively adjusted according to the understanding of those skilled in the art.

[0058] In the description of the present invention, the meaning of "a plurality" is two or more. If there is a description of "first" or "second", it is only used to distinguish technical features and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features or implicitly indicating the order of the indicated technical features. In addition, unless otherwise defined, all technical and scientific terms used in this document have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used in this document are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0059] The visual-tactile sensor gradient calibration method based on deep learning provided in the embodiments of the present application can be applied to a terminal, can also be applied to a server side, and can also be software running in a terminal or a server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a set-top box, etc.; the server side can be configured as an independent physical server, or as a server cluster or distributed system composed of multiple physical servers, or as a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements the visual-tactile sensor gradient calibration method based on deep learning, etc., but is not limited to the above forms.

[0060] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multi-processor systems, microprocessor-based systems, set-top boxes, programmable consumer electronic devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, and the like. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments in which tasks are performed by remote processing devices connected via a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media, including storage devices.

[0061] It should be noted that in each specific embodiment of the present application, when it comes to the need to perform relevant processing based on data related to the user's identity or characteristics, such as user information, user behavior data, user historical data, and user location information, the user's permission or consent will be obtained first, and the collection, use, and processing of such data will comply with the relevant laws, regulations, and standards of the relevant countries and regions. In addition, when the embodiment of the present application needs to obtain the user's sensitive personal information, the user's separate permission or consent will be obtained through a pop-up window or by jumping to a confirmation page. After clearly obtaining the user's separate permission or consent, the necessary user-related data for the normal operation of the embodiment of the present application will be obtained.

[0062] The process of establishing a mapping relationship between RGB light intensity and gradient for visual-tactile sensors, known as RGB intensity-to-gradient calibration, currently involves collecting experimental data on RGB light intensity at different gradients in the laboratory to establish the mapping relationship. Gelsight offers a method that uses a simple sphere to simplify experimental tools and improve calibration efficiency to a certain extent. However, this still requires professionals to mark the sphere's position and record the data, making automated calibration impossible.

[0063] The present invention uses deep learning segmentation to identify the specific position where the film is pressed on the sphere, eliminating the need for experimenters to manually mark the range of the sphere. At the same time, by training a gradient recognition model to complete the gradient calibration of each point on the touch film, automated visual tactile sensor gradient calibration can be achieved.

[0064] like Figure 1The figure shows a flowchart of a method for gradient calibration of visual tactile sensors based on deep learning according to an embodiment of the present invention. Figure 1 The embodiment of the present invention provides a method for gradient calibration of a visual-tactile sensor based on deep learning, which specifically includes the following steps:

[0065] S101, obtaining sensor image information of a preset ball pressing on a target touch film of a visual-tactile sensor to be calibrated, inputting the sensor image information into a pre-trained deep learning segmentation model, and obtaining a contact area of the ball on the target touch film;

[0066] S102, determining the coordinates and RGB values of each point in the contact area of the sphere, and calculating the gradient information of each point in the contact area of the sphere according to a preset radius of the sphere;

[0067] S103, constructing a training data set according to the coordinate values, RGB values and gradient information, and training a gradient recognition model based on the training data set;

[0068] S104 , calibrating the mapping relationship between the RGB light intensity and the gradient of each point on the target touch film according to the gradient recognition model.

[0069] The present invention uses deep learning segmentation to detect the position of the ball pressing on the film, deriving the contact area of the ball, automatically extracting the coordinates and RGB values of each point in the area, and calculating the gradient information of each point in the area. The gradient recognition model is trained based on the coordinates, RGB values, and gradient information. This gradient recognition model can then calibrate the mapping relationship between the RGB light intensity and gradient of each point on the touch film. The specific process is as follows:

[0070] 1) Press the ball onto a certain position on the membrane;

[0071] 2) Collect the image of the current sensor;

[0072] 3) For the current image, use the pre-trained deep learning segmentation model to identify the location where the ball touches and record this area as the valid area;

[0073] 4) Within the valid area, calculate the gradient of the height of each point in the area in the x and y directions according to the radius of the sphere;

[0074] 5) Record the gradient information (Gx, Gy) of each point in the valid area, and extract the coordinate value (x, y) and RGB value (r, g, b) of each point based on the image;

[0075] 6) Repeat steps 1) to 5) by changing the position where the ball is pressed on the touch screen, so that the ball is pressed at different positions on the touch screen with different forces;

[0076] 7) Arrange the data collected in the above steps, and record each set of data as (r, g, b, x, y) → (Gx, Gy);

[0077] 8) Construct a neural network with five input nodes and two output nodes;

[0078] 9) Using the data in 7), a neural network model is trained. The trained gradient recognition model can be directly deployed on the visual-tactile sensor or a corresponding host computer, thereby performing a mapping conversion from RGB light intensity to gradient information when the visual-tactile sensor is used. Alternatively, a mapping relationship between the RGB light intensity and gradient at each point on the touch film can be directly established based on the gradient recognition model, forming a mapping table that is stored in the visual-tactile sensor or a host computer, thereby completing the gradient calibration of the visual-tactile sensor.

[0079] The embodiment of the present invention uses a pre-trained deep learning segmentation model to identify the contact area of the sphere, constructs a training data set based on the coordinate values, RGB values and gradient information of each point in the contact area of the sphere, trains a gradient recognition model, and uses the gradient recognition model to complete the gradient calibration of each point on the target touch film, thereby improving the efficiency and accuracy of the gradient calibration of the visual tactile sensor.

[0080] The specific implementation process of the embodiment of the present invention is further described below with reference to the accompanying drawings.

[0081] like Figure 2 The figure shows a flowchart of the steps of training a deep learning segmentation model provided by an embodiment of the present invention, referring to Figure 2 , further as an optional implementation, the deep learning segmentation model is trained by the following steps:

[0082] S201, acquiring a plurality of sensor sample images of a preset ball pressing at different positions on a sample touch film with different forces;

[0083] S202, segmenting and labeling the sensor sample images to determine contact area labels corresponding to each sensor sample image;

[0084] S203, inputting the sensor sample image into a pre-built deep learning neural network to obtain a predicted contact area;

[0085] S204, determining a first loss value according to the predicted contact area and the contact area label;

[0086] S205. Update the parameters of the deep learning neural network according to the first loss value to obtain a trained deep learning segmentation model.

[0087] Specifically, a ball is pressed on different positions on the sample touch film with different forces, and image data of the sensor is collected; for the collected sample images, the position where the ball is squeezed is marked using a segmentation and annotation method to obtain a contact area label; a deep learning segmentation model is constructed, and the sample images and contact area labels are used to train the segmentation model. The model can use a commonly used deep learning segmentation model, such as yolov8-seg.

[0088] After inputting a sample image into an initialized deep learning neural network, the model outputs a predicted contact area. The predicted contact area and the aforementioned contact area labels can be used to evaluate the accuracy of the model's predictions, thereby updating the model's parameters. For deep learning segmentation models, the accuracy of the model's predictions can be measured using a loss function. This loss function is defined on a single training data point and measures the prediction error for that training data point. Specifically, the loss value for that training data point is determined by combining the label of the training data point with the model's prediction result for that training data point. In actual training, a training dataset often contains many training data points, so a cost function is typically used to measure the overall error of the training dataset. This cost function is defined over the entire training dataset and averages the prediction errors across all training data points, providing a better measure of the model's prediction performance. For typical machine learning models, the aforementioned cost function, combined with a regularization term to measure model complexity, serves as the training objective function. This objective function can then be used to calculate the loss value for the entire training dataset. There are many types of commonly used loss functions, such as 0-1 loss function, square loss function, absolute loss function, logarithmic loss function, cross entropy loss function, etc., which can all be used as loss functions of machine learning models, which will not be elaborated here one by one. In an embodiment of the present invention, any one of the loss functions can be selected to determine the loss value of training. Based on the loss value of training, the back propagation algorithm is used to update the parameters of the model, and a trained deep learning segmentation model can be obtained by iterating several rounds. The specific number of iterations can be pre-set, or the training is considered to be completed when the test set meets the accuracy requirements.

[0089] like Figure 3 FIG. 1 is a flowchart of step S102 provided in an embodiment of the present invention, referring to FIG. Figure 3 As an optional implementation, the gradient information of each point in the contact area of the sphere is calculated according to the preset radius of the sphere, which specifically includes:

[0090] S1021, determining an actual bending radius according to a preset radius of the sphere and a thickness of the target touch film;

[0091] S1022, fitting the contact area of the sphere to obtain a fitted circular area, and determining the center position of the fitted circular area;

[0092] S1023. Calculate the gradient information of each point in the contact area of the sphere according to the center position of the circle and the actual bending radius.

[0093] As an optional implementation, the gradient information is calculated using the following formula:

[0094]

[0095]

[0096] in, and Represent the gradient values of the point (x, y) in the x direction and y direction respectively, (x0, y0) represents the center position of the circle, and R represents the actual bending radius.

[0097] Specifically, assuming that the radius of the sphere is r and the thickness of the touch film is d, the actual bending radius can be calculated to be R=r+d.

[0098] The depth of the ball's extrusion is expressed as z=f(x,y). When the center of the contact area of the ball is taken as the coordinate origin, we have:

[0099]

[0100] The gradient of z in the x and y directions is solved as follows:

[0101]

[0102]

[0103] It should be noted that the x and y coordinates in the above formula are calculated with the center of the sphere contact area as the coordinate origin. When converted to the image coordinates of the sensor, x = x-x0 and y = y-y0 are substituted, where (x0, y0) is the center position of the sphere contact area. After the sphere contact area is segmented, the center position can be calculated by circle fitting, which will not be elaborated here.

[0104] like Figure 4 FIG. 1 is a flowchart of step S103 provided in an embodiment of the present invention, referring to FIG. Figure 4 As an optional implementation, a training data set is constructed based on the coordinate values, RGB values, and gradient information, and a gradient recognition model is obtained by training the training data set, which specifically includes:

[0105] S1031, determining an x coordinate and a y coordinate according to the coordinate values, determining an R value, a G value, and a B value according to the RGB values, and determining a training sample according to the R value, the G value, the B value, the x coordinate, and the y coordinate;

[0106] S1032, determining the gradient value in the x-direction and the gradient value in the y-direction according to the gradient information, and determining the gradient label corresponding to the training sample according to the gradient value in the x-direction and the gradient value in the y-direction;

[0107] S1033, inputting the training sample into a pre-built ANN neural network to obtain a gradient recognition result;

[0108] S1034. Determine a second loss value according to the gradient recognition result and the gradient label;

[0109] S1035. Update the parameters of the ANN neural network according to the second loss value to obtain a trained gradient recognition model.

[0110] Further as an optional implementation, the ANN neural network includes an input layer, a first hidden layer, a second hidden layer, a third hidden layer, a fourth hidden layer and an output layer. The input layer is used to input training samples, and the output layer is used to output gradient recognition results. The first hidden layer, the second hidden layer, the third hidden layer and the fourth hidden layer include 64, 128, 128 and 64 neurons respectively.

[0111] Specifically, when the positions of the three light sources (red (R), green (G), and blue (B)) are fixed, and the touch film is also fixed, the relationship between the gradient and the RGB light intensity at each point on the film is fixed. This relationship is a fixed nonlinear relationship f, namely G(x, y) = f(r, g, b, x, y). The gradient calculation is related to five input parameters: the R, G, and B values of the point under the three light sources, as well as the x and y coordinates of the point. The gradient of a point is the gradient in both the x and y directions.

[0112] In this embodiment of the present invention, the input layer of the ANN neural network consists of five nodes, which input the R value, G value, B value, x coordinate, and y coordinate, namely (r, g, b, x, y). The output layer consists of two nodes, which respectively input the x-direction gradient Gx and the y-direction gradient Gy. The middle layer contains four hidden layers, and the number of nodes in the four hidden layers is 64, 128, 128, and 64, respectively. During the training process, the MSE loss function is used as the loss function, and the parameters of the ANN neural network are updated through the backpropagation algorithm until the model converges.

[0113] like Figure 5 FIG. 1 is another flow chart of step S104 provided in an embodiment of the present invention, referring to FIG. Figure 5As an optional embodiment, the mapping relationship between the RGB light intensity and the gradient of each point on the target touch film is calibrated according to the gradient recognition model, which specifically includes:

[0114] S1041, dividing the target touch film into a plurality of grid points according to a preset spacing;

[0115] S1042, determining the gradient of each grid point under different RGB light intensities according to the gradient recognition model, and establishing a mapping relationship according to the coordinate position of the grid point, the RGB light intensity, and the corresponding gradient;

[0116] S1043: Generate a three-dimensional lookup table including coordinate values, RGB values, and gradient values according to the mapping relationship.

[0117] Specifically, the touch film is divided into grid points with a spacing of 0.1 mm. For each grid point, its gradient value under different RGB light intensities is determined through the gradient recognition model, thereby establishing a mapping relationship between coordinate position, RGB light intensity and gradient. Finally, a three-dimensional lookup table of coordinate values, RGB values and gradient values is generated to complete the gradient calibration of the visual tactile sensor.

[0118] After the visual-tactile sensor captures the touch image, it can identify gradient information using a gradient recognition model. Alternatively, it can use a calibrated 3D lookup table to find gradient information, and then use an integration method to obtain depth data. Depth data can be used for other applications, such as calculating pressure based on the elastic properties of a membrane. This embodiment of the present invention is not described in detail here.

[0119] The above describes the method steps of an embodiment of the present invention. It can be appreciated that the embodiment of the present invention utilizes a pre-trained deep learning segmentation model to identify the contact area of the sphere, constructs a training dataset based on the coordinate values, RGB values, and gradient information of each point within the contact area of the sphere, and trains a gradient recognition model to complete the gradient calibration of each point on the target touch film using this gradient recognition model, thereby improving the efficiency and accuracy of the visual tactile sensor gradient calibration.

[0120] like Figure 6 The figure shows a schematic diagram of the structure of the visual tactile sensor gradient calibration device based on deep learning provided by an embodiment of the present invention, referring to Figure 6 , an embodiment of the present invention provides a visual-tactile sensor gradient calibration device based on deep learning, comprising:

[0121] A contact area recognition module is used to obtain sensor image information of a preset ball pressing on the target touch film of the visual-tactile sensor to be calibrated, input the sensor image information into a pre-trained deep learning segmentation model, and obtain the contact area of the ball on the target touch film;

[0122] A gradient information calculation module is used to determine the coordinate values and RGB values of each point in the contact area of the sphere, and calculate the gradient information of each point in the contact area of the sphere according to the preset radius of the sphere;

[0123] The model training module is used to construct a training data set based on coordinate values, RGB values and gradient information, and train a gradient recognition model based on the training data set;

[0124] The calibration module is used to calibrate the mapping relationship between the RGB light intensity and the gradient of each point on the target touch film according to the gradient recognition model.

[0125] The contents of the above method embodiments are all applicable to the present device embodiments. The functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0126] An embodiment of the present invention further provides an electronic device comprising: a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for communicating between the processor and the memory. When the program is executed by the processor, the aforementioned deep learning-based visual-tactile sensor gradient calibration method is implemented. The electronic device can be any intelligent terminal, including a tablet computer and an in-vehicle computer.

[0127] like Figure 7 FIG2 is a schematic diagram of the hardware structure of the electronic device provided by the embodiment of the present invention, referring to FIG2 Figure 7 , an embodiment of the present invention provides an electronic device, including:

[0128] The processor 701 may be implemented as a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is configured to execute relevant programs to implement the technical solutions provided by the embodiments of the present invention.

[0129] Memory 702 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). Memory 702 can store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in memory 702 and is called by processor 701 to execute the deep learning-based visual tactile sensor gradient calibration method of the embodiment of the present invention.

[0130] Input / output interface 703, used to implement information input and output;

[0131] Communication interface 704, used to implement communication interaction between this device and other devices, which can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WiFi, Bluetooth, etc.);

[0132] Bus 705 , which transmits information between various components of the device (e.g., processor 701 , memory 702 , input / output interface 703 , and communication interface 704 );

[0133] The processor 701 , the memory 702 , the input / output interface 703 and the communication interface 704 are connected to each other in communication within the device via a bus 705 .

[0134] like Figure 8 FIG2 is a schematic diagram of the structure of the storage medium provided by the embodiment of the present invention, referring to FIG2 Figure 8 An embodiment of the present invention further provides a storage medium, which is a computer-readable storage medium for computer-readable storage. The storage medium stores one or more programs 801, and the one or more programs 801 can be executed by one or more processors to implement the above-mentioned deep learning-based visual-tactile sensor gradient calibration method.

[0135] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely arranged relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0136] The embodiment of the present invention also discloses a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs Figure 1 The method shown.

[0137] In some optional embodiments, the function / operation mentioned in the block diagram may not occur in the order mentioned in the operation diagram. For example, depending on the function / operation involved, the two boxes shown in succession can actually be executed substantially simultaneously or the above-mentioned boxes can sometimes be executed in reverse order. In addition, the embodiment presented and described in the flow chart of the present invention is provided in an exemplary manner for the purpose of providing a more comprehensive understanding of the technology. The disclosed method is not limited to the operation and logic flow presented herein. Optional embodiments are contemplated in which the order of the various operations is changed and the sub-operations described as a part of a larger operation are performed independently.

[0138] In addition, although the present invention is described in the context of functional modules, it should be understood that, unless otherwise stated, one or more of the above-mentioned functions and / or features can be integrated into a single physical device and / or software module, or one or more functions and / or features can be implemented in separate physical devices or software modules. It is also understood that a detailed discussion of the actual implementation of each module is not necessary for understanding the present invention. More specifically, given the properties, functions, and internal relationships of the various functional modules in the devices disclosed herein, the actual implementation of the module will be understood within the routine skills of an engineer. Therefore, a person skilled in the art can implement the present invention set forth in the claims using ordinary skills without undue experimentation. It is also understood that the specific concepts disclosed are merely illustrative and are not intended to limit the scope of the present invention, which is determined by the full scope of the appended claims and their equivalents.

[0139] If the above functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the above methods in various embodiments of the present invention. The aforementioned storage medium includes various media that can store program code, such as USB flash drives, mobile hard drives, read-only memories (ROMs), random access memories (RAMs), magnetic disks, or optical disks.

[0140] The logic and / or steps represented in the flowcharts or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing the logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (e.g., a computer-based system, a system including a processor, or other system that can fetch and execute instructions from an instruction execution system, apparatus, or device). For purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transport a program for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0141] More specific examples (a non-exhaustive list) of computer-readable media include the following: an electrical connection with one or more wires (electronic devices), a portable computer disk cartridge (magnetic devices), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disc read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable media on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other media, and then editing, interpreting, or processing it in another suitable manner as necessary, and then storing it in a computer memory.

[0142] It should be understood that various components of the present invention may be implemented using hardware, software, firmware, or a combination thereof. In the above-described embodiments, multiple steps or methods may be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented using hardware, as in another embodiment, any one of the following technologies known in the art or a combination thereof may be used: a discrete logic circuit having logic gate circuits for implementing logic functions on data signals, an application-specific integrated circuit having suitable combinational logic gate circuits, a programmable gate array (PGA), a field-programmable gate array (FPGA), etc.

[0143] In the above description of this specification, reference to the terms "one embodiment / example," "another embodiment / example," or "certain embodiments / examples" means that the specific features, structures, materials, or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

[0144] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to the embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the claims and their equivalents.

[0145] The above is a specific description of the preferred implementation of the present invention, but the present invention is not limited to the above embodiments. Those skilled in the art can make various equivalent modifications or substitutions without violating the spirit of the present invention. These equivalent modifications or substitutions are all included in the scope defined by the claims of this application.

Claims

1. A method for gradient calibration of visual tactile sensors based on deep learning, characterized in that: The following steps are involved: Obtaining sensor image information of a preset ball pressing on a target touch film of the visual-tactile sensor to be calibrated, inputting the sensor image information into a pre-trained deep learning segmentation model to obtain a contact area of the ball on the target touch film; Determine the coordinate value and RGB value of each point in the contact area of the sphere, and calculate the gradient information of each point in the contact area of the sphere according to the radius of the preset sphere; Constructing a training data set according to the coordinate values, the RGB values, and the gradient information, and training a gradient recognition model according to the training data set; calibrating a mapping relationship between RGB light intensity and gradient at each point on the target touch film according to the gradient recognition model; The deep learning segmentation model is trained by the following steps: Acquire a plurality of sensor sample images of the preset ball pressing at different positions on the sample touch film with different forces; Segmenting and labeling the sensor sample images to determine contact area labels corresponding to each sensor sample image; Inputting the sensor sample image into a pre-built deep learning neural network to obtain a predicted contact area; determining a first loss value according to the predicted contact area and the contact area label; The parameters of the deep learning neural network are updated according to the first loss value to obtain the trained deep learning segmentation model.

2. The method for gradient calibration of visual tactile sensors based on deep learning according to claim 1, characterized in that: The step of calculating the gradient information of each point in the contact area of the sphere according to the radius of the preset sphere specifically includes: determining an actual bending radius according to the radius of the preset sphere and the thickness of the target touch film; Fitting the sphere contact area to obtain a fitted circular area, and determining the center position of the fitted circular area; Gradient information of each point in the contact area of the sphere is calculated according to the center position of the circle and the actual bending radius.

3. The method for gradient calibration of visual tactile sensors based on deep learning according to claim 2, characterized in that: The gradient information is calculated by the following formula: in, and They represent the gradient values of the point (x, y) in the x direction and the y direction respectively, (x0, y0) represents the center position of the circle, and R represents the actual bending radius.

4. The method for gradient calibration of visual tactile sensors based on deep learning according to claim 1, characterized in that: The step of constructing a training data set according to the coordinate values, the RGB values, and the gradient information, and training a gradient recognition model according to the training data set specifically includes: Determining an x coordinate and a y coordinate according to the coordinate values, determining an R value, a G value, and a B value according to the RGB values, and determining a training sample according to the R value, the G value, the B value, the x coordinate, and the y coordinate; Determine an x-direction gradient value and a y-direction gradient value according to the gradient information, and determine a gradient label corresponding to the training sample according to the x-direction gradient value and the y-direction gradient value; Inputting the training sample into a pre-built ANN neural network to obtain a gradient recognition result; Determine a second loss value according to the gradient recognition result and the gradient label; The parameters of the ANN neural network are updated according to the second loss value to obtain the trained gradient recognition model.

5. The method for gradient calibration of a visual-tactile sensor based on deep learning according to claim 4, characterized in that: The ANN neural network includes an input layer, a first hidden layer, a second hidden layer, a third hidden layer, a fourth hidden layer and an output layer. The input layer is used to input the training sample, and the output layer is used to output the gradient recognition result. The first hidden layer, the second hidden layer, the third hidden layer and the fourth hidden layer include 64, 128, 128 and 64 neurons respectively.

6. A method for gradient calibration of visual tactile sensors based on deep learning according to any one of claims 1 to 5, characterized in that: The step of calibrating the mapping relationship between the RGB light intensity and the gradient of each point on the target touch film according to the gradient recognition model specifically includes: Dividing the target touch film into a plurality of grid points according to a preset spacing; Determining the gradient of each grid point under different RGB light intensities according to the gradient recognition model, and establishing a mapping relationship according to the coordinate position of the grid point, the RGB light intensity, and the corresponding gradient; A three-dimensional lookup table including coordinate values, RGB values, and gradient values is generated according to the mapping relationship.

7. A visual-tactile sensor gradient calibration device based on deep learning, characterized in that: include: A contact area recognition module is configured to obtain sensor image information of a preset ball pressing on a target touch film of the visual-tactile sensor to be calibrated, input the sensor image information into a pre-trained deep learning segmentation model, and obtain the contact area of the ball on the target touch film; a gradient information calculation module, configured to determine the coordinate values and RGB values of each point within the contact area of the sphere, and calculate the gradient information of each point within the contact area of the sphere according to the radius of the preset sphere; A model training module is used to construct a training data set according to the coordinate values, the RGB values and the gradient information, and train a gradient recognition model according to the training data set; a calibration module, configured to calibrate a mapping relationship between RGB light intensity and gradient at each point on the target touch film according to the gradient recognition model; The deep learning segmentation model is trained by the following steps: Acquire a plurality of sensor sample images of the preset ball pressing at different positions on the sample touch film with different forces; Segmenting and labeling the sensor sample images to determine contact area labels corresponding to each sensor sample image; Inputting the sensor sample image into a pre-built deep learning neural network to obtain a predicted contact area; determining a first loss value according to the predicted contact area and the contact area label; The parameters of the deep learning neural network are updated according to the first loss value to obtain the trained deep learning segmentation model.

8. An electronic device, characterized in that: The electronic device includes a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for realizing connection and communication between the processor and the memory. When the program is executed by the processor, the steps of the deep learning-based visual-tactile sensor gradient calibration method according to any one of claims 1 to 6 are implemented.

9. A storage medium, which is a computer-readable storage medium and is used for computer-readable storage, characterized in that: The storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps of the deep learning-based visual-tactile sensor gradient calibration method according to any one of claims 1 to 6.

Images