Force Estimation Using Deep Learning

Abstract

A force estimation using deep learning is disclosed. A computer system generates a tactile force model for a tactile force sensor by performing a number of calibration tasks. In various embodiments, the calibration tasks include attaching the tactile force sensor to a pressure gauge, pressing the tactile force sensor, interacting with a sphere, and pushing an object along a plane. The data collected from these calibration tasks is used to train a neural network. The resulting tactile force model allows the computer system to convert signals received from the tactile force sensor into the magnitude and direction of the force with greater accuracy than traditional methods. In one embodiment, the force on the tactile force sensor is inferred by interacting with the object, determining the movement of the object, and estimating the force on the object based on a physical model of the object.

Description

Background Art

[0001] Robotic systems are being used to perform increasingly complex and delicate tasks. An important part of being able to perform many delicate tasks is the ability to adjust the robot's control capabilities based on sensory feedback. For example, many kitchen tasks, such as opening a sugar bowl, pouring into a container, or flipping a pancake, require precise observation of forces to be successfully performed. With a method to estimate forces from tactile signals, these tasks can be learned more quickly. In another example, when handling soft or compliant objects, tactile force sensing allows the robot to adjust its actions based on changes to the object. Advanced tactile sensors, such as the Syn Touch BioTac, provide soft, flesh-like, compliant gripping surfaces that are able to handle delicate objects without damage. However, the nature of the sensor makes it difficult to estimate the force applied by the tactile sensor. Without accurate force estimates, successful performance of many tasks would be very difficult. Therefore, it is important to provide a system that estimates the force on the tactile sensor in a manner that is accurate in both magnitude and direction over a wide range of forces. BRIEF DESCRIPTION OF THE DRAWINGS

[0002] Various techniques will be described with reference to the accompanying drawings, in which:

[0003] Figure 1 An example of a manipulator equipped with a tactile force sensor in one embodiment is shown;

[0004] Figure 2 An example of a tactile force sensor in one embodiment is shown;

[0005] Figure 3 An example of a process for training a tactile force model for estimating forces on a tactile force sensor in one embodiment is shown;

[0006] Figure 4 An example of a force prediction neural network in one embodiment is shown;

[0007] Figure 5 An example of a test fixture for measuring force applied to a tactile force sensor in one embodiment is shown.

[0008] Figure 6 An example of a test fixture for measuring forces resulting from manipulation of a ball in one embodiment is shown.

[0009] Figure 7 An example of a test fixture for measuring the force resulting from pushing an object onto a flat surface is shown in one embodiment.

[0010] Figure 8 An example of a process in one embodiment as a result of execution by a computer system that generates a tactile force model that estimates forces on a tactile force sensor is shown;

[0011] Fig. 9 An example of a process as a result of being executed by a computer system in one embodiment to determine a force on a tactile force sensor based on the motion of an object being manipulated by the sensor is shown.

[0012] Fig.10 illustrates an example of a parallel processing unit ("PPU") according to one embodiment;

[0013] Fig.11 shows an example of a general processing cluster ("GPC") according to one embodiment;

[0014] Fig.12 shows an example of a streaming multiprocessor according to one embodiment; and

[0015] Fig.13 A computer system is shown in which various examples may be implemented according to one embodiment. DETAILED DESCRIPTION

[0016] This document describes a system for calibrating tactile force sensors for use in a robotic system. In one embodiment, a probe, claw, gripper, or manipulator is mounted with a set of tactile force sensors that allow a control system to receive signals representing forces applied to the sensors. In one embodiment, the tactile force sensor provides a plurality of force measurements distributed over a surface of the force sensor that provide an indication of the direction of the force as well as the magnitude of the force. In one embodiment, the tactile force sensor is covered with a soft surface (e.g., silicone) that conforms the surface of the sensor to the object being manipulated.

[0017] In one embodiment, a number of test devices are used to collect calibration information for a tactile force sensor. In one embodiment, the back of the tactile force sensor is fixed to a pressure sensor, and an object is pressed against the force sensor. In one embodiment, the tactile force sensor generates a signal as a result of pressing the object against the force sensor, and the pressure sensor records the amount of force generated. In one embodiment, a manipulator equipped with a tactile force sensor is used to manipulate a ball connected to a calibrated force sensor. In one embodiment, when the ball is manipulated, the signal received by the tactile force sensor is associated with the corresponding force measured by the calibrated force sensor. In one embodiment, a probe equipped with a tactile sensor is used to push an object on a plane and measure the movement of the object. In one embodiment, a physical model of the object is determined, allowing the force applied to the object to be estimated based on the movement of the object. In one embodiment, the estimated force and the signal received from the tactile force sensor are used to calibrate the tactile force sensor.

[0018] In one embodiment, information received from the test fixture is used as training data for the neural network. In one embodiment, the neural network learns a tactile force model that can be used to convert the signal provided by the tactile force sensor into the magnitude and direction of the force. In one embodiment, the tactile force model can be used to estimate the force applied by the tactile force sensor for various tasks other than the training tasks described above by training with ground truth from various different test devices as described above. In one embodiment, the system uses a voxelized input feature layer for the spatial signal and normalizes the loss function with information about the sensor surface. In one embodiment, it has been shown that the generated tactile force model produces a median angle accuracy of 0.06 radians and a median amplitude accuracy of 0.06N radians on a test data set. In one embodiment, the tactile force model is verified by performing object lifting and placement tasks, because if the force estimate is incorrect, the lifting task tends to fail, and if the force direction estimate is incorrect, the placement task tends to fail.

[0019] In one embodiment, the spatial information of the electrodes in the tactile force sensor is encoded by voxelizing the electrode values ​​in a 3D grid to provide a better estimate of the force direction. In one embodiment, the system provides surface normal regularization, which improves the accuracy of force prediction closer to the surface normal at the contact point. In one embodiment, the tactile force model is validated in a real robotic manipulation task using rigid and flexible objects with different surface properties.

[0020] As would be appreciated by those skilled in the art based on this disclosure, certain examples may be able to achieve certain advantages, including some or all of the following: (1) increasing the accuracy of force estimates in both direction and magnitude, (2) quantifying force precision in standardized metrics, (3) providing accurate force inferences through interaction with a calibrated object model, (4) autonomously collecting ground truth force measurement data, and (5) providing validation tasks to confirm the accuracy of force measurement models.

[0021] Figure 1An example of a manipulator equipped with a tactile force sensor is shown. In one embodiment, a computer control system controls the operation of the manipulator 100. In one embodiment, the manipulator 100 includes a palm portion 102 connected to a plurality of joint fingers. In one embodiment, the joint fingers include a first finger 104, a second finger 106, a third finger 108, and a fourth finger 110. In one embodiment, the jointed fingers include relative fingers 112. In one embodiment, each finger includes a plurality of joint points and a servo mechanism that allow the finger to move relative to the palm portion 102. In some embodiments, each finger is controlled by a servo motor and a cable of the length of the finger, so that the finger can be flexed and curled like a human hand. In other embodiments, the finger may include one or more servo motors at each joint point, thereby allowing each joint point to be operated separately. In another embodiment, each joint point may include an articulation motor that controls rotation and articulation on one or more axes.

[0022] In one embodiment, a control system is connected to the robot hand 100. In one embodiment, the control system includes a computing machine system having one or more processors and a memory storing instructions that, when executed by the one or more processors, cause the computer system to issue electronic commands to the servo motors that control the various fingers of the robot hand 100. In one embodiment, the electronic commands are issued through an I / O port connected to the computer system and amplified using relays, power transistors, or other circuits. In various embodiments, the electronic commands can be voltage-based or current-based signals. In one embodiment, the electronic commands can include a series of pulses that indicate a step or position change to be performed by the servo motor.

[0023] In one embodiment, one or more fingers of the manipulator 100 may include a tactile sensor. In one embodiment, the tactile sensor is a force sensor, such as a SynTouch BioTac. In one embodiment, the tactile sensor can provide temperature, humidity, pressure, or conductivity measurements of the surface being contacted. In one embodiment, the tactile sensor can provide the above measurements for multiple locations on the sensor surface. In one embodiment, a separate measurement is provided by the tactile sensor for each location in a grid of locations distributed on the sensor surface. In one embodiment, force measurements are provided by a grid of electrodes on the surface of the tactile sensor. In one embodiment, the tactile sensor provides a digital electrical interface, such as a parallel or serial interface, which allows signals from the tactile sensor to be provided to a control system. In one embodiment, the tactile sensor is constructed using a compliant contact surface such as silicone that covers the sensor and protects the electronic components of the tactile sensor from moisture and wear.

[0024] exist Figure 1 In the example shown, Figure 1 As shown, the first finger 104 includes a first tactile force sensor 114, the second finger 106 includes a second tactile force sensor 116, the third finger 108 includes a third tactile force sensor 118, the fourth finger 110 includes a fourth tactile force sensor 120, and the opposing finger 112 includes a fifth tactile force sensor 122. In one embodiment, each force sensor is positioned so that one or more fingers can be manipulated by a control system so that the surface of the tactile force sensor is used to manipulate an object. In one embodiment, the signal provided by the tactile force sensor is provided to the control system, and the control system is able to manipulate the servos in the various fingers of the robotic hand 100 to adjust the force sensed by the sensor. In one embodiment, this technology allows for more reliable manipulation of soft, compliant or soft objects. For example, in one embodiment, by adaptively adjusting the force sensed by the tactile force sensor, the control system is able to reliably pick up and place soft or flexible objects onto a hard surface.

[0025] In one embodiment, the manipulator 100 can include various numbers of fingers and opposing fingers in various configurations. In one embodiment, the manipulator 100 includes two fingers implemented with tactile sensors in a claw configuration. In one embodiment, the manipulator 100 includes one or more fingers and one or more opposing fingers, and only the opposing fingers are equipped with tactile sensors, and the force on the one or more fingers is estimated based on the force detected on the opposing fingers.

[0026] Figure 2 An example of a tactile force sensor in one embodiment is shown. Figure 2 An example of a tactile force sensor 200 such as the Syn TouchBioTac is shown. In one embodiment, the tactile force sensor 200 includes a sensor assembly 202, a flexible cover 204, a sensor body 206, and a flange 208. In one embodiment, the tactile force sensor 200 is mounted to the end of a probe, a claw, or a finger on a manipulator. In one embodiment, an object is manipulated by the probe, the claw, or the manipulator, and the surface of the flexible cover 204 is pressed against the surface of the object to be manipulated. In one embodiment, the flexible cover 204 transmits force to a set of force sensors on the sensor assembly 202. In one embodiment, the sensor assembly 202 includes an electrode grid that provides a set of signals that are approximately proportional to the magnitude of the force at the electrode locations.

[0027] In one embodiment, the interface cable 210 connects the sensor assembly 202 to the control system. In one embodiment, the interface cable 210 is part of a serial interface that transmits the individual values ​​of the electrodes to the control system. In one embodiment, the control system processes the set of values ​​to identify the magnitude and direction of the force (e.g., force vector). In one embodiment, the control system includes a neural network that is trained to estimate the force and magnitude on the tactile force sensor based on the signals provided by the sensor assembly 202.

[0028] In one embodiment, the tactile force sensor 200 is a BioTac sensor from Syn Touch. The BioTac sensor has a rigid core wrapped by a high friction elastic skin, and the space between the core and the skin is filled with a weakly conductive liquid. In the BioTac sensor, 19 impedance sensing electrodes are dispersed on the surface of the core, and a thermistor coupled to a heater measures the fluid temperature. In one embodiment, the transducer measures static pressure, and the high frequency changes in pressure are measured by the transducer at 2.2kHz and sent to the system in a buffer. In one embodiment, the system determines the contact points and surface normals on the Bio Tac sensor by approximating the BioTac surface geometry as a half cylinder connected to a quarter cylinder cap, both of which have the same radius.

[0029] Figure 3 An example of a process 300 for training a tactile force model to estimate forces on a tactile force sensor in one embodiment is shown. In one embodiment, given a set of sensor readings at the contact point of the sensor, the system estimates the forces on the tactile force sensor with reference to the sensor frame. In one embodiment, the sensor readings are voxelized 304 and provided to a feed neural network 310. In one embodiment, learning of the tactile sensor is improved by introducing a loss function 308 in training that scales the loss as a function of the angular distance between the applied force and the surface normal 302. In one embodiment, a feedforward neural network 310 is used to learn a function that maps the sensor readings to an estimated force on the sensor. In one embodiment, the accuracy of the model is improved by using a training data set that covers a wide range of forces in magnitude and direction. In one embodiment, ground truth forces 306 are collected from multiple ground truth data collection sources, thereby improving the robustness of the model and the ability to use the force model in various additional tasks.

[0030] In one embodiment, the source includes force data collected by securely attaching a tactile force sensor to a wrist force / torque ("FT") sensor and using an object to press against the tactile force sensor. In one embodiment, pressing against the sensor is accomplished by having a person interact with the object and the sensor. In one embodiment, the FT sensor is attached to a ball and then, for example, Figure 1 The robotic hand-arm system shown interacts with a ball to create another source. In one embodiment, the ball interaction source adds randomness to the orientation of the BioTac sensor frame relative to the force torque sensor frame. In one embodiment, the signal-to-noise ratio of the experimentally generated wrist force / torque sensor is in the range of 0.01N to 0.1N, making small force readings very noisy. In one embodiment, to reduce noise, another source of force data involves collecting sensor readings from a planar push device. In one embodiment, the planar push setup is configured so that the robot uses a tactile force sensor to push a box on a plane. In one embodiment, the ground truth force used for the planar push is calculated by least squares optimization, as described elsewhere in this document.

[0031] In one embodiment, the data set used for system training is collected as follows. In one embodiment, the system uses an OptoForce HEX-E 6-DOF force torque sensor to collect stiffness-ft and ball-ft data. For stiffness-ft data, the BioTac is mounted on the force-torque sensor, and then the object is pressed on the finger to collect data. In one embodiment, a total of 20k force samples were collected. In one embodiment, for the ball-ft method, the manipulator pushes the tactile sensor to a hard plastic ball mounted on a vertical rod attached to the force-torque sensor. For example, in one embodiment, the ball-ft method generates 200 random trajectories for the middle fingertip to make contact with the ball from 10 different wrist postures, thereby generating a total of 20k force samples. In one embodiment, for the plane push method, the tactile sensor is mounted on a robot that can push a box weighing up to 0.65 kg. In one embodiment, the system generates a single straight line, task space position trajectory for the fingertip using trajectory optimization. In one embodiment, the task space trajectory is executed using a Riemann motion strategy. In one embodiment, for each execution of the task space position trajectory, a random initial orientation is selected for the tactile force sensor and the box. In one embodiment, the orientation of the box is sampled from a small range so that each push keeps the contact point on the same face of the box. In one embodiment, a total of 600 trials were collected on the robot, resulting in a total of 100k force samples. In one example, the final data set collected from three data sources contains a total of 140k force samples. In one embodiment, for the ball-ft and plane push operations, the robot is tracked with a digital camera such as an ASUS Xtion RGB-D camera. In one embodiment, depth-based tracking techniques with physical constraints are used to determine the position and motion of the ball and box objects. In one embodiment, contact is enabled before the FT sensor measures a force greater than 2N on the ball-ft and before the BioTac pressure signal value rises by more than 10 units of the plane push setting. In one embodiment, the parameters for optimization are selected in a least squares minimization problem, namely n=80 and k=10. In one embodiment, the number of particles n does not affect the force beyond this magnitude, and any value of k greater than 2 will produce similar performance. In one embodiment, the coefficient of friction between the box and the plane is determined to be 0.1.

[0032] Figure 4An example of a force prediction neural network 400 in one embodiment is shown. In one embodiment, a set of tactile force sensor signals, such as BioTac signals 402, are fed into the neural network. In one embodiment, the BioTac signals include a set of electrode signals 404 and surface contact points 406. In one embodiment, the neural network 400 uses the BioTac signals 402 to generate an estimated force vector. In one embodiment, a voxel grid 408 is created based on the value of each electrode and the location of the electrode on the sensor. In one embodiment, the output of the first voxel grid 408 is fed to a first 3-D convolution 410. In one embodiment, a second voxel grid 412 is created, which represents the contact points on the sensor. In one embodiment, the output of the second voxel grid 412 is fed to a second 3-D convolution 414.

[0033] In one embodiment, the outputs of the 3D convolutions are concatenated 416 and the results are passed through a first layer of 3D convolutions 418 and a second layer of 3D convolutions 420. The features are then flattened and passed through a 2-D convolution layer 422, which is then further flattened into a vector 424. In one embodiment, the vector passes through fully connected layers 426, 428, and 430 to produce a prediction force vector f of length 3. p .

[0034] In one embodiment, the predicted force vector f p The scaled surface normal 432 is compared to the ground truth force 30 .

[0035] Q 3d (f 3d f p )=(1 / ||f 3d ||2)||f 3d -f p ||2

[0036] In one embodiment, for the planar push dataset, we use the projection norm 434 to generate Q 436 because there may be forces acting perpendicular to the planar surface that are not accounted for by the physical model.

[0037]

[0038] where Ψ is the direction of the support surface plane.

[0039] In one embodiment, the high friction surface of the tactile force sensor allows the force to be applied at the contact point of the sensor from directions other than the surface normal, so the force can be applied at the contact point on the surface other than the contact point where the surface normal matches the force direction. In one embodiment, as the angle between the force and the surface normal increases, the signal of the sensor may become less meaningful. In one embodiment, the loss function is scaled with an adaptive weighting function.

[0040] α(S n , f 3d )=2^(β(1-D(S n ,f 3d )))

[0041]

[0042] where β is a scalar weight and is the ground truth force vector f 3d is the unit vector of , and D(·) is the normalized cosine distance function. The loss function used in our network is defined as:

[0043] Q(·)=∝(·)Q proj (f 3d ,f p , w R B )-If the plane pushes

[0044] Q(·)=∝(·)Q 3d (f 3d ,f p )-otherwise

[0045] In one embodiment, the neural network can be implemented using TensorFlow, and the dataset can be divided into 80% training, 10% validation, and 10% testing by partitioning the experiment (throughout the experiment). In one embodiment, the training can be run for 200 epochs with a batch size of 512, and the model is stored as the loss on the validation set improves. In one embodiment, the Adam Optimizer can be used to assist in the optimization of the neural network. In one embodiment, an adaptive learning rate is used, which is increased from 10 -4 Start with 2 for the first two periods [(i / 50)] , and then decreases by 0.95 for each iteration of the remaining epochs.

[0046] In one embodiment, a signal from the tactile force sensor is sent to the network when the robot detects that the sensor's fingertip is in contact with an object. In one embodiment, the absolute pressure signal from the tactile force sensor is used to determine contact and classify the sensor as in contact with the object. In one example, if the pressure signal maintains a value above 10 for at least 10 steps, then the BioTac sensor is determined to be in contact. In one embodiment, by checking for contact before sending data to the neural network, false positives are avoided when the sensor is moving in free space.

[0047] Figure 5 An example of a test fixture 500 in one embodiment is shown that measures the force applied to a tactile force sensor 502. In one embodiment, the tactile force sensor 502 includes a force sensor covered by a flexible membrane 504, and a signal representing the force on the membrane is transmitted to a computer system via an interface cable 506. In one embodiment, the force signal includes multiple signals representing the force at various points distributed throughout the surface of the membrane 504. In one embodiment, the force signal is transmitted in a format that allows the computer system to associate each force signal with a specific location on the membrane. In one embodiment, the force signal is transmitted as a position-value pair. In one embodiment, the force signal is transmitted in a two-dimensional array that represents a corresponding array of force sensors on the surface of the membrane 504. In one embodiment, the tactile force sensor 502 is mounted to a force sensor 508 via a mounting block 510. In one embodiment, the force sensor 508 is an OptoForce HEX-E 6-DOF force torque sensor.

[0048] In one embodiment, an object is manually 512 pressed against the membrane 504, and the force is transmitted to the force sensor 508 through the mounting block 510. In one embodiment, a pattern of signals representing the force is sent to a computer system through an interface cable, and additional measurements of the force are received from the force sensor 508. In one embodiment, these two measurements provide a source of ground truth force information for the neural network. In one embodiment, an object such as a flat object, a curved object, a compliant object, etc. is manually 512 pressed against the membrane 504. In one embodiment, the object pressed against the membrane 504 is selected based on the object to be manipulated in performing the task so that the estimated force generated by the neural network most accurately predicts the force on the actual object.

[0049] Figure 6An example of a test fixture 600 in one embodiment is shown that measures the forces generated by manipulating a ball. In one embodiment, the test fixture includes a force sensor 602, such as an OptoForce HEX-E 6-DOF force torque sensor. In one embodiment, the force sensor 602 is connected to the ball 604 via a rod. In one embodiment, a control computer system controls the operation of a robotic hand 606. In one embodiment, the robot includes a first finger 608, a second finger 610, and an opposing finger 612 in 606. In one embodiment, the first finger 608 is equipped with a tactile force sensor 614, such as a BioTac force sensor.

[0050] In one embodiment, the control computer system directs the manipulator 606 to manipulate the ball 604. In one embodiment, signals are sent from the tactile force sensor 614 to the control computer system via the interface cable 616. In one embodiment, each manipulation of the ball 604 by the robotic tank 606 generates a set of force signals from both the tactile force sensor 614 and the force sensor 602. In one embodiment, each set of force signals and corresponding measurements from the force sensor 602 are used as training data for the neural network. In one embodiment, the control computer system directs the manipulator 606 to autonomously interact with the ball 604 to establish ground truth from different approach directions.

[0051] Figure 7 An example of a test fixture 700 in one embodiment is shown that, in one embodiment, measures the force generated by pushing an object 702 onto a flat surface 704. In one embodiment, the force is inferred from a planar push of the object 702 using a tactile force sensor 706 using rigid body physics. In one embodiment, this force inference allows for ground truth force estimates at small force levels. In one embodiment, the force inference also infers the forces that cause linear and angular motion of the object. In one embodiment, the inferred forces are two-dimensional, and a loss function is introduced that penalizes errors only along these two dimensions. Additionally, in one embodiment, the loss is regularized based on the cosine distance between the force direction at the contact point on the sensor membrane 708 and the surface normal. In one embodiment, this regularization allows for higher accuracy in force predictions when the force is closer to the surface normal.

[0052] In one embodiment, a control computer system directs the operation of a robotic probe or hand equipped with a tactile force sensor 706. In one embodiment, the control computer system directs the hand to move the object 702 by pressing the tactile membrane 708 of the force sensor 706 against one side of the object 702. In one embodiment, the object is moved along the plane 704 by the hand. In one embodiment, the position of the object 702 is monitored by a camera, and based on the movement of the object 702, the force applied by the membrane 708 to the object is determined. In one embodiment, a set of sensor signals are transmitted from the force sensor 706 to the control computer system via an interface cable 710.

[0053] In one embodiment, assuming an object with known mass m in SE(2) plane space moves with linear velocity v and angular velocity ω, the net force causing the motion can be determined by:

[0054] f c =m(dv / dt)

[0055] xf c =I(dw / dt)

[0056] in, It's on point The net force acting at the center of mass (CM) of a reference object.

[0057] In one embodiment, given the linear acceleration of an object, the net force can be determined. However, in one embodiment, it is difficult to determine the force if the measurement system cannot observe small linear accelerations. In one embodiment, linear accelerations can be small in the following cases: 1) the force applied to the object is small, resulting in small linear and angular accelerations; 2) the force applied is perpendicular to a radial line, in which case the object will have a large angular acceleration. In one embodiment, in the latter case, the above equation can be used to determine the net force f c In one embodiment, we solve the above equation as the loss function in a least squares minimization problem:

[0058]

[0059] where the weight k scales the linear acceleration loss, and [c] is the antisymmetric matrix of the vector c.

[0060] In one embodiment, when the object is now at a position with a friction coefficient μ between the object and the surface s When the object is placed on a plane, the friction between the object and the plane will be the friction force f f and moment n fIn one embodiment, if the contact area between the object and the surface is R, and r is any point on the object in this area, the force and torque can be defined using Coulomb's law as:

[0061]

[0062]

[0063] where v(·) is a function giving the velocity of a point.

[0064] In one embodiment, the pressure at r is given by p(·), and dA is the differential element of the area at r. In one embodiment, the torque is derived with reference to the center of mass of the object. In one embodiment, to make the friction force of the flat push easy to calculate, the system assumes:

[0065] 1) The pressure distribution in the contact area R is uniform.

[0066] 2) The rigid body is made of n uniformly distributed particles.

[0067] 3) Decompose the contact area into N small areas, where the centroid of area i is r i , the normal force applied in region i is mg / N.

[0068] In one embodiment, using the assumptions listed above, the friction force may be determined as:

[0069]

[0070] And, the torque due to friction becomes:

[0071]

[0072] The friction force f included in the minimization problem f and moment n f gives:

[0073]

[0074] Optimizing the above equation yields the force f c In one embodiment, the force f c is 2D and parallel to the support plane. In one embodiment, assume that f 3d The ground truth force is provided as follows, which is converted from the object's reference frame o to the BioTac sensor reference frame B:

[0075] f 3d = B R0·f c

[0076] In one embodiment, the directional error is calculated as the cosine similarity between the vectors of the predicted force on the ground truth force. In one embodiment, the cosine similarity is scaled to give a percentage of directional accuracy. In one embodiment, the magnitude error is reported as the symmetric mean absolute percentage accuracy between the ground truth and predicted force magnitudes.

[0077] Figure 8 An example of a process as a result of execution by a computer system is shown, which in one embodiment generates a tactile force model that estimates the forces on a tactile force sensor. In one embodiment, process 800 begins at block 802, where a computer system mounts a tactile sensor in a test fixture. In one embodiment, the test fixture can be a planar motion, spherical manipulation, or direct compression fixture, as described elsewhere in this application.

[0078] In one embodiment, the computer system begins with a process of collecting force data for a tactile force sensor. In one embodiment, at box 804, a force is applied to the tactile force sensor. In one embodiment, the force may be applied manually or by autonomous control using a controlled manipulator. In one embodiment, at box 806, the computer system receives a force signal generated by the tactile force sensor applying the force. In one embodiment, the force signal may be a plurality of electrode signals. In one embodiment, at box 808, the computer system measures the force applied to the tactile sensor using a force sensor on a test fixture. In one embodiment, at box 810, the computer system uses the signal from the tactile sensor and the measured force to train the tactile model. In one embodiment, at decision box 812, the computer system determines whether additional training is required. In one embodiment, additional training may be required based on the number of configurations of the requested training samples. In one embodiment, additional training is necessary based on the measurement errors generated by the tactile force model. In one embodiment, if additional training is required, the process returns to box 804, where another force is applied to the tactile sensor. In one embodiment, if the training is completed, execution proceeds to box 814.

[0079] In one embodiment, at block 814, the system estimates the force on the sensor using the tactile force model. In one embodiment, at block 814, the computer system receives a signal from the tactile force sensor. In one embodiment, at block 816, the computer system estimates the actual force on the sensor based on the received signal using the tactile force model. In one embodiment, the estimated force is in the form of a vector, which includes the magnitude and direction of the force. In one embodiment, execution returns to block 814 and an updated force signal is received from the tactile sensor.

[0080] Fig. 9An example of a process 900 as a result of being executed by a computer system in one embodiment is shown, which determines the force on a tactile force sensor based on the movement of an object manipulated by the sensor. In one embodiment, the process begins at box 902, and the computer system builds a physical model for an environment including an object. In one embodiment, the model may include one or more of the mass of the object, the center of mass of the object, the geometry of the object, the pressure center of the object, and one or more environmental parameters. In one embodiment, the environmental parameters may include one or more of friction, air resistance, fluid resistance, temperature, pressure, or wind speed. In one embodiment, at box 904, the computer system commands a robotic mechanism equipped with a tactile force sensor to manipulate the object. In one embodiment, at box 906, the computer system receives a signal from the tactile force sensor as a result of manipulating the object. In one embodiment, the signal is a collection of individual signals, where each individual signal is associated with one of a plurality of electrodes on the tactile force sensor. In one embodiment, at box 908, the computer system measures the movement of the object resulting from the manipulation. In one embodiment, the movement of the object is measured using a camera that obtains a video clip of the object moving. In one embodiment, the video clip is analyzed by the computer system to determine the movement of the object.

[0081] In one embodiment, at box 910, a computer system estimates a force on an object based at least in part on the determined motion of the object and a physical model. For example, in one embodiment, the physical model includes the center of mass and mass of the object, and the force on the object can be determined as the mass of the object multiplied by the acceleration of the object. In one embodiment, the force acting on the object can include a linear component and an angular component, and can be constrained to a two-dimensional plane as described above. In one embodiment, at box 912, the signal obtained from the tactile sensor and the estimated force are used as force data to train a tactile force model. In one embodiment, the tactile force model can be used to convert the signal generated by the tactile force sensor into an estimated force and force vector.

[0082] One embodiment includes a processor comprising: one or more arithmetic logic units (ALUs) configured to control movement of a robotic device based at least in part on one or more neural networks trained using a two-dimensional representation of an object surface and pressure information from one or more tactile sensors.

[0083] An embodiment includes a method comprising training one or more neural networks to control one or more robotic devices based at least in part on a two-dimensional representation of a surface of an object and pressure information from one or more tactile sensors.

[0084] An embodiment includes a machine-readable medium having stored thereon a set of instructions that, if executed by one or more processors, cause the one or more processors to perform at least the following operations: train one or more neural networks to control one or more robotic devices based at least in part on a two-dimensional representation of a surface of an object and pressure information from one or more tactile sensors.

[0085] An embodiment includes a machine-readable medium comprising information that, if used by one or more computer instructions to configure one or more processors, causes the one or more processors to control a robotic device based at least in part on a machine learning model, the machine learning model comprising: an embedding layer for encoding a two-dimensional representation of a three-dimensional object and pressure information representing pressure sensed by a tactile sensor applied to the three-dimensional object; and a neural network for predicting one or more actions of the robotic device based at least in part on the two-dimensional representation and the pressure information.

[0086] Fig.10 A parallel processing unit ("PPU") 1000 is shown according to one embodiment. In one embodiment, the PPU 1000 is configured with machine-readable code that, if executed by the PPU, causes the PPU to perform some or all of the processes and techniques described throughout the present disclosure. In one embodiment, the PPU 1000 is a multithreaded processor implemented on one or more integrated circuit devices that utilizes multithreading as a latency hiding technique to process computer-readable instructions (also referred to as machine-readable instructions or simple instructions) in parallel on multiple threads. In one embodiment, a thread refers to an execution thread and is an instance of a group of instructions configured to be executed by the PPU 1000. In one embodiment, the PPU 1000 is a graphics processing unit ("GPU") configured as an image rendering pipeline for processing three-dimensional ("3D") graphics data to generate two-dimensional ("2D") image data for display on a display device such as a liquid crystal display ("LCD") device. In one embodiment, the PPU 1000 is used to perform calculations such as linear algebra operations and machine learning operations. Fig.10 An example parallel processor is shown for illustration purposes only, which should be construed as a non-limiting example of a processor architecture within the scope of the present disclosure, and any suitable processor may be used in addition to and / or in place of the same.

[0087] In one embodiment, one or more PPUs are configured to accelerate high performance computing ("HPC"), data centers, and machine learning applications. In one embodiment, PPU 1000 is configured to accelerate deep learning systems and applications, including the following non-limiting examples: autonomous vehicle platforms, deep learning, high-precision speech, image, text recognition systems, intelligent video analysis, molecular simulation, drug discovery, disease diagnosis, weather forecasting, big data analysis, astronomy, molecular dynamics simulation, financial modeling, robotics, factory automation, real-time language translation, online search optimization, personalized user recommendations, etc.

[0088] In one embodiment, the PPU 1000 includes an input / output ("I / O") unit 1006, a front end unit 1010, a scheduler unit 1012, a work distribution unit 1014, a hub 1016, a crossbar switch ("Xbar") 1020, one or more general processing clusters ("GPCs") 1018, and one or more partition units 1022. In one unit, the PPU 1000 is connected to a host processor or other PPUs 1000 via one or more high-speed GPU interconnects 1008. In one embodiment, the PPU 1000 is connected to a host processor or other peripheral devices via an interconnect 1002. In one embodiment, the PPU 1000 is connected to a local memory including one or more memory devices 1004. In one embodiment, the local memory includes one or more dynamic random access memories ("DRAMs"). In one embodiment, one or more DRAM devices are configured and / or configurable as a high bandwidth memory ("HBM") subsystem, and multiple DRAM dies are stacked within each device.

[0089] The high-speed GPU interconnect 1008 may refer to a wire-based multi-channel communication link that a system may use to scale and include one or more PPUs 1000 in combination with one or more CPUs, supporting cache coherency between the PPU 1000 and the CPU, and CPU mastering. In one embodiment, data and / or commands are transmitted / received by the high-speed GPU interconnect 1008 to / from other units of the PPU 1000 through the hub 1016, such as one or more copy engines, video encoders, video decoders, power management units, and other units that may not be in the PPU 1000. Fig.10 Other components explicitly shown in FIG.

[0090] In one embodiment, I / O unit 1006 is configured to receive data from a host processor ( Fig.10In one embodiment, the I / O unit 1006 communicates with the host processor directly via the system bus 1002 or through one or more intermediate devices (e.g., a memory bridge). In one embodiment, the I / O unit 1006 can communicate with one or more other processors (e.g., one or more PPUs 1000) via the system bus 1002. In one embodiment, the I / O unit 1006 implements a peripheral component express ("PCIe") interface for communicating over a PCIe bus. In one embodiment, the I / O unit 1006 implements an interface for communicating with external devices.

[0091] In one embodiment, I / O unit 1006 decodes packets received via system bus 1002. In one embodiment, at least some of the packets represent commands configured to cause PPU 1000 to perform various operations. In one embodiment, I / O unit 1006 sends the decoded commands to various other units of PPU 1000 as specified by the commands. In one embodiment, the commands are sent to front end unit 1010 and / or to hub 1016 or other units of PPU 1000, such as one or more replication engines, video encoders, video decoders, power management units, etc. ( Fig.10 In one embodiment, I / O unit 1006 is configured to route communications between and among the various logical units of PPU 1000.

[0092] In one embodiment, a program executed by a host processor encodes a command stream in a buffer that provides a workload to the PPU 1000 for processing. In one embodiment, the workload includes instructions and data to be processed by these instructions. In one embodiment, the buffer is an area in a memory that can be accessed (e.g., read / write) by both the host processor and the PPU 1000 - the host interface unit can be configured to access the buffer in the system memory, which is connected to the system bus 1002 via a memory request transmitted by the I / O unit 1006 via the system bus 1002. In one embodiment, the host processor writes the command stream to the buffer and then sends a pointer to the starting point of the command stream to the PPU 1000 so that the front end unit 1010 receives a pointer to one or more command streams and manages the one or more streams, reads commands from these streams and forwards the commands to the various units of the PPU 1000.

[0093] In one embodiment, the front end unit 1010 is coupled to a scheduler unit 1012 that configures various GPCs 1018 to process tasks defined by one or more streams. In one embodiment, the scheduler unit 1012 is configured to track state information related to various tasks managed by the scheduler unit 1012, where the state information may indicate which GPC 1018 the task is assigned to, whether the task is active or inactive, a priority associated with the task, etc. In one embodiment, the scheduler unit 1012 manages the execution of multiple tasks on one or more GPCs 1018.

[0094] In one embodiment, the scheduler unit 1012 is coupled to a work distribution unit 1014, which is configured to distribute tasks for execution on the GPCs 1018. In one embodiment, the work distribution unit 1014 tracks a number of scheduled tasks received from the scheduler unit 1012, and the work distribution unit 1014 manages a pending task pool and an active task pool for each GPC 1018. In one embodiment, the pending task pool includes a plurality of time slots (e.g., 32 time slots) containing certain tasks assigned to be processed by a particular GPC 1018; the active task pool may include a plurality of time slots (e.g., 4 time slots) for tasks that are being actively processed by the GPC 1018, such that when a GPC 1018 completes executing a task, the task is evicted from the active task pool of the GPC 1018, and one of the other tasks in the pending task pool is selected and scheduled for execution on the GPC 1018. In one embodiment, if the active task is idle on GPC 1018 , for example while waiting for data dependencies to be resolved, the active task is evicted from GPC 1018 and returned to the pending task pool while another task is selected from the pending task pool and scheduled for execution on GPC 1018 .

[0095] In one embodiment, work distribution unit 1014 communicates with one or more GPCs 1018 via a crossbar switch (XBar) 1020. In one embodiment, crossbar switch 1020 is an interconnect network that couples many units of PPU 1000 to other units of PPU 1000, and can be configured to couple work distribution unit 1014 to a specific GPC 1018. Although not explicitly shown, one or more other units of PPU 1000 can also be connected to crossbar switch 1020 through hub 1016.

[0096] Tasks are managed by a scheduler unit 1012 and assigned to GPCs 1018 by a work distribution unit 1014. GPCs 1018 are configured to process tasks and produce results. The results may be consumed by other tasks in GPCs 1018, may be routed to different GPCs 1018 via a crossbar 1020, or may be stored in memory 1004. The results may be written to memory 1004 via a partition unit 1202, implementing a memory interface for reading data from and writing data to memory 1004. The results may be transmitted to another PPU 1004 or a CPU via a high-speed GPU interconnect 1008. In one embodiment, PPU 1000 includes a number U of partition units 1022, where U is equal to the number of separate and distinct storage devices 1004 coupled to PPU 1000. Partition units 1022 will be described in more detail elsewhere.

[0097] In one embodiment, a host processor executes a driver kernel that implements an application programming interface (“API”) that enables one or more applications executing on the host processor to schedule operations to be performed on the PPU 1000. In one embodiment, the PPU 1000 executes multiple computing applications simultaneously, and the PPU 1000 provides isolation, quality of service (“QoS”), and independent address spaces for multiple computing applications. In one embodiment, the application generates instructions (e.g., in the form of API calls) that cause the driver kernel to generate one or more tasks to be executed by the PPU 1000, and the driver kernel outputs the tasks to one or more streams processed by the PPU 1000. In one embodiment, each task includes one or more related thread groups, which may be referred to as warps. In one embodiment, a warp includes multiple related threads (e.g., 32 threads) that may be executed in parallel. In one embodiment, a cooperative thread may refer to multiple threads that include instructions to execute tasks and exchange data through a shared memory. Threads and cooperative threads are described in more detail elsewhere in this specification, according to one embodiment.

[0098] Fig.11 According to one embodiment, Fig.10In one embodiment, each GPC 1100 includes a plurality of hardware units for processing tasks, and each GPC 1100 includes a pipeline manager 1102, a front raster operation unit ("PROP") 1104, a raster engine 1108, a work distribution crossbar ("WDX") 1116, a memory management unit ("MMU") 1118, one or more data processing clusters ("DPCs") 1106, and any suitable combination of components. It will be understood that Fig.11 The GPC 1100 may include instead of or in addition to Fig.11 Other hardware units of the unit shown.

[0099] In one embodiment, the operation of the GPC 1100 is controlled by a pipeline manager 1102. The pipeline manager 1102 handles the configuration of one or more DPCs 1106 to process the tasks assigned to the GPC 1100. In one embodiment, the pipeline manager 1102 configures at least one of the one or more DPCs 1106 to implement at least a portion of a graphics rendering pipeline. In one embodiment, the DPC 1106 is configured to execute a vertex shader program on a programmable streaming multiprocessor ("SM") 1114. In one embodiment, the pipeline manager 1102 is configured to route packets received from work assignments to appropriate logic units within the GPC 1100, and some packets may be routed to fixed-function hardware units in the PROP 1104 and / or the raster engine 1108, while other packets may be routed to the DPC 1106 to be processed by the primitive engine 1112 or the SM 111. In one embodiment, the pipeline manager 1102 configures at least one of the one or more DPCs 1106 to implement a neural network model and / or a compute pipeline.

[0100] In one embodiment, the PROP unit 1104 is configured to route data generated by the raster engine 1108 and the DPC 1106 to a raster operation ("ROP") unit in the memory partition unit, as described in more detail above. In one embodiment, the PROP unit 1104 is configured to perform optimized color blending, organize pixel data, perform address translation, and the like. In one embodiment, the raster engine 1108 includes a plurality of fixed-function hardware units configured to perform various raster operations, and the raster engine 1108 includes a setup engine, a coarse raster engine, a culling engine, a clipping engine, a fine raster engine, a tile merging engine, and any suitable combination thereof. In one embodiment, the setup engine receives transformed vertices and generates plane equations associated with geometric primitives defined by the vertices; the plane equations are passed to the coarse raster engine to generate coverage information for the primitives (e.g., an x, y coverage mask for a tile); the output of the coarse raster engine is transmitted to the culling engine, where fragments associated with primitives that fail the z test are culled, and to the clipping engine, where fragments outside the frustum are clipped. In one embodiment, the clipped and culled fragments are passed to a fine raster engine to generate attributes for the pixel fragments based on the plane equations generated by the setup engine. In one embodiment, the output of the raster engine 1108 includes fragments to be processed by any suitable entity (e.g., by a fragment shader implemented within the DPC 1106).

[0101] In one embodiment, each DPC 1106 included in the GPC 1100 includes an M-pipeline controller ("MPC") 1110; a primitive engine 1112; one or more SMs 1114, and any suitable combination thereof. In one embodiment, the MPC 1110 controls the operation of the DPC 1106, routing packets received from the pipeline manager 1102 to appropriate units in the DPC 1106. In one embodiment, packets associated with vertices are routed to the primitive engine 1112, which is configured to fetch vertex attributes associated with the vertices from memory; conversely, packets associated with shader programs may be sent to the SM 1114.

[0102] In one embodiment, SM1114 includes a programmable stream processor configured to process processing tasks represented by multiple threads. In one embodiment, SM1114 is multithreaded and configured to execute multiple threads (e.g., 32 threads) from a specific thread group simultaneously, and implements a SIMD (single instruction, multiple data) architecture, wherein each thread in a group of threads (e.g., a warp) is configured to process different data sets based on the same set of identical instructions. In one embodiment, all threads in a thread group execute the same instructions. In one embodiment, SM1114 implements a SIMT (single instruction, multiple thread) architecture, wherein each thread in a group of threads is configured to process different data sets based on the same instruction set, but wherein a single thread in a thread group is allowed to diverge during execution. In one embodiment, a program counter, a call stack, and an execution state are maintained for each warp, so that when threads within a warp diverge, concurrency between warps and serial execution within the warp is achieved. In another embodiment, a program counter, a call stack, and an execution state are maintained for each individual thread, so that equal concurrency is achieved within and between warps, and between all threads. In one embodiment, execution state is maintained for each individual thread, and threads executing the same instruction may be converged and executed in parallel for better efficiency. In one embodiment, SM1114 will be described in more detail below.

[0103] In one embodiment, MMU 1118 provides an interface between GPC 1100 and the memory partition unit, and MMU 1118 provides virtual address to physical address translation, memory protection, and arbitration for memory requests. In one embodiment, MMU 1118 provides one or more translation lookaside buffers ("TLBs") for translating virtual addresses to physical addresses in memory.

[0104] Fig.12 According to one embodiment, Fig.111200. In one embodiment, SM 1200 includes: an instruction cache 1202; one or more scheduler units 1204; a register file 1208; one or more processing cores 1210; one or more special function units ("SFUs") 1212; one or more load / store units ("LSUs") 1214; an interconnect network 1216; a shared memory / L1 cache 1218; and any suitable combination thereof. In one embodiment, a work distribution unit schedules tasks for execution on a GPC of a PPU, and each task is assigned to a specific DPC within a GPC, and, if the task is associated with a shader program, the task is assigned to SM 1200. In one embodiment, scheduler unit 1204 receives tasks from the work distribution unit and manages instruction scheduling for one or more thread blocks assigned to SM 1200. In one embodiment, scheduler unit 1204 schedules thread blocks for execution as warps of parallel threads, wherein each thread block is assigned at least one warp. In one embodiment, each warp executes a thread. In one embodiment, the scheduler unit 1204 manages multiple different thread blocks, assigns warps to different thread blocks, and then distributes instructions from multiple different cooperation groups to various functional units (e.g., core 1210, SFU 1212, and LSU 1214) in each clock cycle.

[0105] A cooperative group may refer to a programming model for organizing groups of communicating threads that allows developers to express the granularity of communicating threads, thereby being able to express richer and more efficient parallel decompositions. In one embodiment, a cooperative launch API supports synchronization between thread blocks to execute parallel algorithms. In one embodiment, the application of a conventional programming model provides a single, simple construct for synchronizing cooperative threads: a barrier across all threads of a thread block (e.g., a syncthreads() function). However, programmers often want to define thread groups in a manner smaller than the granularity of a thread block and synchronize within the defined group to achieve higher performance, design flexibility, and reuse software in the form of a collectively scoped functional interface. Cooperative groups enable programmers to explicitly define thread groups at sub-block (i.e., as small as a single thread) and multi-block granularity, and perform collective operations (e.g., synchronization) on threads in cooperative groups. The programming model supports clean composition across software boundaries so that libraries and utility functions can be safely synchronized in their local contexts without having to assume convergence. Cooperative group primitives enable new cooperative parallel modes, including producer-consumer parallelism, opportunistic parallelism, and global synchronization across the entire thread block grid.

[0106] In one embodiment, the dispatch unit 1206 is configured to send instructions to one or more functional units, and the scheduler unit 1204 includes two dispatch units 1206, which enables two different instructions from the same thread warp to be dispatched in each clock cycle. In one embodiment, each scheduler unit 1204 includes a single dispatch unit 1206 or additional dispatch units 1206.

[0107] In one embodiment, each SM 1200 includes a register file 1208 that provides a set of registers for the functional units of the SM 1200. In one embodiment, the register file 1208 is divided between each functional unit so that each functional unit is allocated a dedicated portion of the register file 1208. In one embodiment, the register file 1208 is divided between different thread warps executed by the SM 1200, and the register file 1208 provides temporary storage for operands connected to the data paths of the functional units. In one embodiment, each SM 1200 includes a plurality of L processing cores 1210. In one embodiment, the SM 1200 includes a large number (e.g., 128 or more) of different processing cores 1210. In one embodiment, each core 1210 includes a full pipeline, single precision, double precision and / or mixed precision processing unit (including a floating point arithmetic logic unit and an integer arithmetic logic unit). In one embodiment, the floating point arithmetic logic unit implements the IEEE 754-2008 standard for floating point arithmetic. In one embodiment, core 1210 includes 64 single precision (32-bit) floating point cores, 64 integer cores, 32 double precision (64-bit) floating point cores, and 8 tensor cores.

[0108] According to one embodiment, the tensor core is configured to perform matrix operations. In one embodiment, one or more tensor cores are included in core 1210. In one embodiment, the tensor core is configured to perform deep learning matrix arithmetic, such as convolution operations for neural network training and reasoning. In one embodiment, each tensor core operates on a 4×4 matrix and performs matrix multiplication and accumulation operations D=A×B+C, where A, B, C, and D are 4×4 matrices.

[0109] In one embodiment, the matrix multiplication inputs A and B are 16-bit floating point matrices, and the accumulation matrices C and D are 16-bit floating point or 32-bit floating point matrices. In one embodiment, the tensor core performs accumulation operations on the 16-bit floating point input data and the 32-bit floating point. In one embodiment, this 16-bit floating point multiplication requires 64 operations and produces a full-precision product, which is then added to other intermediate products using 32-bit floating point addition to perform a 4×4×4 matrix multiplication. In one embodiment, the tensor core is used to perform larger two-dimensional or higher dimensional matrix operations that are composed of these smaller elements. In one embodiment, an API such as the CUDA 9C++ API exposes specialized matrix loads, matrix multiplications and accumulations, and matrix storage operations to efficiently use tensor cores from CUDA-C++ programs. In one embodiment, at the CUDA level, the warp-level interface assumes that a matrix of size 16×16 spans all 32 threads of the warp.

[0110] In one embodiment, each SM 1200 that performs a specific function (e.g., attribute evaluation, reciprocal square root, etc.) includes M SFUs 1212. In one embodiment, the SFs 1212 include a tree traversal unit configured to traverse a hierarchical tree data structure. In one embodiment, the SFUs 1212 include a texture unit configured to perform texture map filtering operations. In one embodiment, the texture unit is configured to load a texture map (e.g., a 2D array of textures) from a memory and sample the texture map to generate sampled texture values ​​for use by a shader program executed by the SM 1200. In one embodiment, the texture map is stored in a shared memory / LI cache. According to one embodiment, the texture unit performs texture operations, such as filtering operations using mip-maps (e.g., a texture map with varying levels of detail). In one embodiment, each SM 1200 includes two texture units.

[0111] In one embodiment, each SM 1200 includes N LSUs that implement load and store operations between shared memory / LI cache and register file 1208. In one embodiment, each SM 1200 includes an interconnect network 1216 that connects each functional unit to register file 1208 and connects LSUs 1214 to register file 1208, shared memory / LI cache 1218. In one embodiment, interconnect network 1216 is a crossbar switch that can be configured to connect any functional unit to any register in register file 1208 and connect LSUs 1214 to memory locations in register file and shared memory / LI cache 1218.

[0112] In one embodiment, shared memory / L1 cache 1218 is an array of on-chip memory that allows data storage and communication between SM 1200 and primitive engines and between threads in SM 1200. In one embodiment, shared memory / L1 cache 1218 includes 128KB of storage capacity and is in the path from SM 1200 to the partition unit. In one embodiment, shared memory / L1 cache 1218 is used to cache reads and writes. One or more of shared memory / L1 cache 1218, L2 cache, and memory are backing stores.

[0113] In one embodiment, data cache and shared memory functionality are combined into a single memory block, providing improved performance for both types of memory accesses. In one embodiment, the capacity is used or used as a cache by programs that do not use shared memory, for example, if the shared memory is configured to use half of the capacity, textures and load / store operations can use the remaining capacity. According to one embodiment, integration within the shared memory / LI cache 1218 enables the shared memory / LI cache 1218 to be used as a high throughput pipeline for streaming data while providing high bandwidth and low latency access to frequently reused data. When configured for general parallel computing, a simpler configuration can be used compared to graphics processing. In one embodiment, the fixed function graphics processing unit is bypassed, creating a simpler programming model. In one embodiment, in a general parallel computing configuration, the work distribution unit directly dispatches and assigns blocks of threads to DPCs. According to one embodiment, threads in a block execute the same program, use unique thread IDs in computations to ensure that each thread generates unique results, use SM 1200 to execute the program and perform computations, shared memory / L1 cache 1218 to communicate between threads, and LSU 1214 to read and write global memory through shared memory / L1 cache 1218 and memory partition unit 1214. In one embodiment, when configured for general parallel computation, SM 1200 writes commands that the scheduler unit can use to start new work on a DPC.

[0114] In one embodiment, the PPU is included in or coupled to a desktop computer, a laptop computer, a tablet computer, a server, a supercomputer, a smartphone (e.g., wireless, handheld device), a personal digital assistant ("PDA"), a digital camera, a vehicle, a head mounted display, a handheld electronic device, etc. In one embodiment, the PPU is packaged on a single semiconductor substrate. In one embodiment, the PPU is included in a system on a chip ("SoC") along with one or more other devices (e.g., additional PPUs, memory, a reduced instruction set computer ("RISC") CPU), a memory management unit ("MMU"), a digital-to-analog converter ("DAC"), etc.

[0115] In one embodiment, the PPU may be included on a graphics card that includes one or more storage devices. The graphics card may be configured to interact with a PCIe slot on a motherboard of a desktop computer. In yet another embodiment, the PPU may be an integrated graphics processing unit ("iGPU") included in a chipset of the motherboard.

[0116] Fig.13 A computer system 1300 is shown in which various architectures and / or functions may be implemented according to one embodiment. In one embodiment, the computer system 1300 is configured to implement the various processes and methods described throughout this disclosure.

[0117] In one embodiment, computer system 1300 includes at least one central processing unit 1302 connected to a communication bus 1310 implemented using any suitable protocol, such as PCI (Peripheral Component Interconnect), PCI-Express, AGP (Accelerated Graphics Port), HyperTransport, or any other bus or point-to-point communication protocol. In one embodiment, computer system 1300 includes main memory 1304 and control logic (e.g., implemented as hardware, software, or a combination thereof), and data is stored in main memory 1304, which may take the form of random access memory ("RAM"). In one embodiment, network interface subsystem 1322 provides an interface to other computing devices and networks for receiving data from computer system 1300 and transmitting data from computer system 1300 to other systems.

[0118] In one embodiment, the computer system 1300 includes an input device 1308, a parallel processing system 1312, and a display device 1306, which may use a conventional CRT (cathode ray tube), LCD (liquid crystal display), LED (light emitting diode), plasma display, or other suitable display technology. In one embodiment, user input is received from the input device 1308 such as a keyboard, mouse, touch pad, microphone, etc. In one embodiment, each of the aforementioned modules may be located on a single semiconductor platform to form a processing system.

[0119] In this specification, a single semiconductor platform may refer to a unique unified semiconductor-based integrated circuit or chip. It should be noted that the term "single semiconductor platform" may also refer to a multi-chip module with increased connectivity that simulates operations on a chip and is substantially improved by utilizing a traditional central processing unit ("CPU") and bus implementation. Of course, the individual modules may also be placed separately or in various combinations of semiconductor platforms, depending on the needs of the user.

[0120] In one embodiment, a computer program in the form of a machine-readable executable code or computer control logic algorithm is stored in the main memory 1304 and / or the auxiliary memory. If executed by one or more processors, the computer program enables the system 1300 to perform various functions according to one embodiment. The main memory 1304, memory and / or any other memory are possible examples of computer-readable media. Auxiliary storage can refer to any suitable storage device or system, such as a hard drive and / or a removable storage drive, in the form of a floppy disk drive, a tape drive, an optical drive, a digital versatile disk ("DVD") drive, a recording device, a universal serial bus ("USB") flash memory.

[0121] In one embodiment, the architecture and / or functionality of the various previous figures are implemented in the context of a central processor 1302; a parallel processing system 1312; an integrated circuit having at least a portion of the capabilities of both the central processor 1302 and the parallel processing system 1312; a chipset (e.g., a group of integrated circuits designed to work and sold as units that perform related functions, etc.), and any suitable combination of integrated circuits.

[0122] In one embodiment, the architecture and / or functionality of the various previous figures are implemented in the context of a general purpose computer system, a circuit board system, a game console system dedicated for entertainment purposes, an application specific system, etc. In one embodiment, the computer system 1300 may take the form of a desktop computer, a laptop computer, a tablet computer, a server, a supercomputer, a smartphone (e.g., wireless, handheld device), a personal digital assistant ("PDA"), a digital camera, a vehicle, a head mounted display, a handheld electronic device, a mobile telephone device, a television, a workstation, a game console, an embedded system, and / or any other type of logic.

[0123] In one embodiment, the parallel processing system 1312 includes multiple PPUs 1314 and associated memory 1316. In one embodiment, the PPU is connected to a host processor or other peripheral device via an interconnect 1318 and a switch 1320 or multiplexer. In one embodiment, the parallel processing system 1312 distributes computing tasks across (one or more) PPUs 1314, and the computing tasks can be parallel, for example, as part of the distribution of computing tasks across multiple GPU thread blocks. In one embodiment, the memory is shared between the entire PPU 1314 or all PPUs 1314 and the entire PPU 1314 or all PPUs 1314 are accessed (e.g., for read and / or write access), although such shared memory may incur performance losses relative to the use of local memory and registers resident in the PPU. In one embodiment, the operation of the PPU 1314 is synchronized by using a command such as _syncthreads(), which requires that all threads in a block (e.g., executed across multiple PPUs 1314) must reach a specific execution point of the code before continuing execution.

[0124] Accordingly, the specification and drawings are to be regarded in an illustrative rather than a restrictive sense. It will, however, be evident that various modifications and changes may be made thereto without departing from the broader spirit and scope of the invention as set forth in the claims.

[0125] Other variations are within the spirit of the present disclosure. Thus, while the disclosed technology is susceptible to various permutations and alternative constructions, certain illustrated embodiments are thus shown in the drawings and have been described in detail above. However, it should be understood that there is no intention to limit the invention to the particular form or forms disclosed, but on the contrary, it is intended to cover all modifications, alternative constructions, and equivalents that fall within the spirit and scope of the present invention, as defined by the appended claims.

[0126] The use of the terms "a", "an", and "the" and similar references in the context of describing the disclosed embodiments (particularly in the context of the appended claims) should be interpreted as covering the singular and the plural, unless otherwise specified herein or clearly contradicted by the context. Unless otherwise indicated, the terms "include", "have", "include" and "contain" should be interpreted as open terms (i.e., meaning "including but not limited to"). The term "connected" (unmodified and referring to a physical connection) should be understood as being fully or partially contained in, attached to or connected together, even if something intervenes. The reference to the numerical range herein is intended to be used only as a shorthand method, unless otherwise specified herein, to refer to each individual value falling within the range, and each individual value is incorporated into the specification, just as if it were quoted separately here. The use of the term "set" (e.g., "a set of items)" or "subset" should be interpreted as a non-empty set containing one or more members, unless otherwise specified or contradicted by the context. In addition, unless otherwise specified or contradicted by the context, the term "subset" of the corresponding set does not necessarily represent a proper subset of the corresponding set, but the subset and the corresponding set may be equal.

[0127] Connective language, such as phrases of the form "at least one of A, B, and C" or "at least one of A, B, and C," unless expressly stated otherwise or clearly contradicted by context, may be used as commonly understood with context to present an item, clause, or the like, which may be A or B or C, or any non-empty subset of the set of A and B and C. For example, in the illustrative example of a set having three members, the connective phrases "at least one of A, B, and C" and "at least one of A, B, and C" refer to any of the following sets: {A}, {B}, {C}, {A, B}, {A, C}, {B, C}, {A, B, C}. Thus, such connective language is not generally intended to imply that certain embodiments require at least one A, at least one B, and at least one C, each of which is used for presentation. In addition, unless expressly stated otherwise or contradicted by context, the term "plurality" indicates a plural state (e.g., "plurality of items" means a plurality of items). The number of items in "plurality" is at least two, but may be more when expressly or by context indicates. Further, the phrase "based on" means "based at least in part on" rather than "based solely on" unless stated otherwise or clear from context.

[0128] The operations of the processes described herein may be performed in any suitable order unless otherwise indicated herein or clearly contradicted by context. In one embodiment, processes such as those described herein (or variations and / or combinations thereof) are performed by hardware or a combination thereof under the control of one or more computer systems, one or more computer systems being configured with executable instructions and implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) that is executed together on one or more processors. In one embodiment, the code is stored on a computer-readable storage medium in the form of a computer program that includes a computer program of multiple instructions that can be executed by one or more processors. In one embodiment, the computer-readable storage medium is a non-transitory computer-readable storage medium that does not include transient signals (e.g., propagated transient electrical or electromagnetic transmissions) but includes non-transitory data storage circuits (e.g., buffers, caches, and queues) within a transceiver of transient signals. In one embodiment, code (e.g., executable code or source code) is stored on a set of one or more non-transitory computer-readable storage media having executable instructions stored thereon (or other storage to store executable instructions) that, when executed (e.g., due to being executed by one or more processors of a computer system), cause the computer system to perform the operations described herein. In one embodiment, the set of non-transitory computer-readable storage media includes a plurality of non-transitory computer-readable storage media, and one or more individual non-transitory storage media of the plurality of non-transitory computer-readable storage media lack all of the code, while the plurality of non-transitory computer-readable storage media collectively store all of the code. In one embodiment, the executable instructions are executed such that different instructions are executed by different processors—e.g., a non-transitory computer-readable storage medium stores instructions, and a main CPU executes some instructions, while a graphics processor unit executes other instructions. In one embodiment, different components of the computer system have independent processors, and different processors execute different subsets of instructions.

[0129] Thus, in one embodiment, a computer system is configured to implement one or more services that individually or collectively perform the operations of the processes described herein, and such a computer system is configured with applicable hardware and / or software that enables the execution of the operations. In addition, a computer system implementing an embodiment of the present disclosure is a single device, and in another embodiment, is a distributed computer system that includes multiple devices that operate in different ways, such that the distributed computer system performs the operations described herein, and such that a single device does not perform all operations.

[0130] Unless otherwise required, the use of any and all examples or exemplary language (e.g., "such as") provided herein is intended only to better illustrate embodiments of the invention and is not intended to limit the scope of the invention. Language in this specification should not be construed as indicating that any non-claimed element is essential to the practice of the invention.

[0131] Embodiments of the present disclosure are described herein, including the best mode known to the inventor for implementing the present invention. Variations of these embodiments will become apparent to those of ordinary skill in the art after reading the foregoing description. The inventor expects that a skilled person will appropriately adopt such variations, and the inventor hopes that the embodiments of the present disclosure may be implemented in a manner different from that specifically described herein, and therefore, the scope of the present disclosure includes all modifications and equivalent forms of the enumerated themes. In addition, unless otherwise noted herein or clearly contradictory to the context, any combination of the above-mentioned elements in all possible variations thereof is included within the scope of the present disclosure.

[0132] All references, including publications, patent applications, and patents, cited herein are hereby incorporated by reference to the same extent as if each reference were individually and specifically indicated to be incorporated by reference and were set forth in its entirety herein.

[0133] In the specification and claims, the terms "coupled" and "connected" and their derivatives may be used. It should be understood that these terms may not be intended as synonyms for each other. On the contrary, in specific examples, "connected" or "coupled" may be used to indicate that two or more elements are in direct or indirect physical or electrical contact with each other. "Coupled" may also mean that two or more elements are not in direct contact with each other, but still cooperate or interact with each other.

[0134] Unless otherwise indicated, it should be understood that throughout the specification, terms such as "processing", "computing", "calculating", "determining", etc. refer to the actions and / or processes of a computer or computing system or similar electronic computing device that operates on and / or transforms data (e.g., electronic) represented as physical quantities (e.g., electronic) in the registers and / or memories of the computing system into other data similarly represented as physical quantities in the memory, registers or other such information storage, transmission or display devices of the computing system.

[0135] In a similar manner, the term "processor" may refer to any device or part of a device that processes electronic data from registers and / or memory and converts the electronic data into other electronic data that can be stored in registers and / or memory. As a non-limiting example, a "processor" may be a central processing unit (CPU) or a graphics processing unit (GPU). A "computing platform" may include one or more processors. As used herein, a "software" process may include, for example, software and / or hardware entities that perform work over time, such as tasks, threads, and intelligent agents. Similarly, each process may refer to multiple processes to execute instructions sequentially or in parallel, continuously or intermittently. The terms "system" and "method" are used interchangeably herein to the extent that the system may embody one or more methods and the method may be considered a system.

[0136] In this document, reference may be made to obtaining, acquiring, receiving or inputting analog or digital data into a subsystem, a computer system or a computer-implemented machine. The process of obtaining, acquiring, receiving or inputting analog and digital data may be accomplished in a variety of ways, such as by receiving data as a parameter of a function call or a call to an application programming interface. In some embodiments, the process of obtaining, acquiring, receiving or inputting analog or digital data may be accomplished by transmitting data via a serial or parallel interface. In another embodiment, the process of obtaining, acquiring, receiving or inputting analog or digital data may be accomplished by transferring data from a providing entity to an acquiring entity via a computer network. Reference may also be made to providing, outputting, transmitting, sending or presenting analog or digital data. In various examples, the process of providing, outputting, transmitting, sending or presenting analog or digital data may be accomplished by transmitting data as an input or output parameter of a function call, a parameter of an application programming interface or an inter-process communication mechanism.

[0137] Although the above discussion sets forth example implementations of the described technology, other architectures may be used to implement the described functionality and are intended to be within the scope of the present disclosure. In addition, although specific responsibilities are defined above for discussion purposes, various functions and responsibilities may be allocated and divided in different ways depending on the circumstances.

[0138] Furthermore, although the subject matter has been described in language specific to structural features and / or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described. Rather, the specific features and acts are disclosed as exemplary forms of implementing the claims.

Claims

1. A method for determining a force on a probe, comprising: Manipulate objects with probes equipped with force sensors; receiving a signal from the force sensor; estimating a force on the probe based on the motion of the object and a model of the object; training a tactile force model of the force sensor using the signal and the estimated force; and The force on the probe is determined using the tactile force model.

2. The method according to claim 1, wherein: manipulating the object by pushing the object on a plane; The model of the object includes the mass of the object and a measure of friction between the object and the plane.

3. The method according to claim 1, further comprising: receiving a video of the subject; and Motion of the object is determined based at least in part on the video.

4. The method according to claim 1, wherein: The motion of the object includes a linear component and an angular component; and The force on the probe is determined as a vector parallel to the plane on which the object is pushed.

5. The method according to claim 1, wherein: The signal comprises a plurality of voxelized electrode signals; and A contact point on the force sensor is determined based at least in part on a plurality of the voxelized electrode signals.

6. The method according to claim 1, wherein: The estimated force is based at least in part on a loss function; and The loss function is a function of the force exerted on the object and the angular distance between the surface normal of the object at the point of contact.

7. The method according to claim 1, further comprising: Force data were collected as a result of performing planar pushing, ball manipulation, and rigid object squeezing tasks; generating the tactile force model using the force data; and The tactile force model is used to estimate the forces generated on the force sensor when performing different tasks.

8. A system for determining a force applied to an object, comprising: one or more processors; and A memory for storing computer executable instructions which, if executed, cause the one or more processors to perform the following operations: receiving a signal from a tactile force sensor generated in response to a force exerted on the object; generating an estimate of a force exerted on the object based on the motion of the object and a model of the object; and The tactile force sensor is calibrated based at least in part on the estimate of the force and the signal to determine a force exerted on the object using the calibrated tactile force sensor.

9. The system according to claim 8, wherein: The computer executable instructions cause the system to further: training a neural network using the signal and the force estimate; and Signals from the tactile force sensors are converted into force estimates using the trained neural network.

10. The system according to claim 8, wherein: The force estimate includes a force magnitude and a force direction relative to an orientation of the tactile force sensor.

11. The system of claim 8, wherein: The object is a rectangular object; and The motion of the object includes planar motion along a surface and angular motion of the object.

12. The system according to claim 11, wherein: The force estimate is determined by solving a least squares minimization of the linear and angular accelerations of the object.

13. The system according to claim 8, wherein: The computer executable instructions, if executed, cause the one or more processors to further: A tactile force model is determined for the tactile force sensor based at least in part on the estimate of force and the signal.

14. The system according to claim 13, wherein: The computer executable instructions, if executed, cause the one or more processors to further: receiving a second signal from the tactile force sensor; and The haptic force model is used to estimate an amount of force represented by the second signal.

15. One or more non-transitory computer-readable storage media having executable instructions stored thereon, whereby the executable instructions are executed by one or more processors of a computer system, causing the computer system to at least perform the following operations: receiving a set of signals from a tactile force sensor, the set of signals representing a first force resulting from an interaction with an object; training a tactile force model using the set of signals and the estimate of the first force; and A second force on the tactile force sensor is estimated using the tactile force model.

16. The one or more non-transitory computer-readable storage media of claim 15, wherein: The set of signals is collected from a set of electrodes on the tactile force sensor; and The set of signals is stored as a first voxel grid.

17. One or more non-transitory computer-readable storage media according to claim 16, wherein: Contact points on the tactile force sensor are stored in a second voxel grid; and The first voxel grid and the second voxel grid are concatenated together and passed through two layers of 3D convolutions.

18. One or more non-transitory computer-readable storage media according to claim 17, wherein: The 3D convolutional features are flattened and passed through a layer of 2D convolution; and The 2D convolved features are flattened into a vector for estimating the second force.

19. The one or more non-transitory computer-readable storage media of claim 15, wherein: The executable instructions further cause the computer system to: Verifying the force generated by the tactile force model by performing a task of grasping a deformable object; and The direction of the force vector generated by the tactile force model is verified by placing a deformable object on a plane.

20. The one or more non-transitory computer-readable storage media of claim 15, wherein: The tactile force sensor provides a plurality of force measurements from a plurality of different points on a surface of the tactile force sensor; and The tactile force sensor provides information identifying a location on the surface of the tactile force sensor for each of the plurality of force measurements.

21. The one or more non-transitory computer-readable storage media of claim 15, wherein: The executable instructions further cause the computer system to: enabling a robot equipped with one or more tactile force sensors to manipulate an object; receiving signals from the one or more tactile force sensors; and A force on the object is determined using the tactile force model and the signal.

22. The one or more non-transitory computer-readable storage media of claim 21, wherein: The executable instructions further cause the computer system to cause the robot to adjust manipulation of the object based at least in part on the determined forces.

Images