Tactile Pin Field Sensors and Manipulators

Abstract

The present application discloses a tactile pin field sensor and manipulator. A pin field includes a plurality of pins mounted parallel to each other in a frame. The plurality of pins can be extended and retracted from the frame for sensing an object in a working environment or for manipulating an object.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of U.S. Provisional Patent Application No. 62 / 655,584, filed on April 10, 2018. Technical Field

[0003] The present disclosure generally relates to systems and methods for sensing and manipulating objects. More specifically, the present disclosure relates to methods and devices for sensing and manipulating objects using haptic pin fields. Background Art

[0004] The need for effective telerobotic sensing and object manipulation arises from a cross-industry demand to be able to sense and manipulate objects or environments in areas where it is impossible, difficult, or unsafe for humans to approach or perform work.

[0005] Haptic technology recreates the sense of touch by applying force, vibration, or motion to the user. This mechanical stimulation can be used to help create virtual objects in computer simulations, to control such virtual objects, and to enhance remote control of machines and devices. Haptic devices can include tactile sensors that measure the force applied by the user on the interface.

[0006] Current haptic technology attempts to recreate human sensations through technology. For example, existing haptic and telerobotic solutions attempt to relay touch sensation information to a person through a telerobotic. For example, a glove or other device that engages with a person's hand can be used to relay information. The glove or other device can be equipped with an electronic system that attempts to recreate sensations from the environment in which the sensing device is telerobotically positioned. For example, a rough surface can feed a relay of vibrations to convey the "bumpy" feel of that type of surface. In another example, a small articulated arm stylus can restrict motion when in contact with a static object to convey the boundaries of the object.

[0007] Both of these methods convey some tactile information, but they are very limited in the fidelity of the sensed information they can convey. They also require a high level of interpretation of the information they provide to make it meaningful in terms of sensing. With an articulated arm, a person must try to visualize and remember where resistance was and where no resistance was received to form a mental configuration of the object or environment being sensed. Similarly, when a glove provides information through means such as vibration, the person wearing the glove must interpret which structure the different vibration levels are associated with.

[0008] Conventional claw-type end effectors can be used to grasp and manipulate objects in a variety of environments. However, conventional claw-type end effectors may not be universally well used for objects with a variety of different and irregular shapes or other features.

[0009] What is needed is a method and apparatus that takes into account at least some of the above-mentioned issues as well as possible other issues. Summary of the invention

[0010] The illustrative embodiments provide a pin field that includes a plurality of pins mounted parallel to each other in a frame. The plurality of pins can be extended and retracted from the frame for sensing an object in a working environment or for manipulating an object.

[0011] The illustrative embodiments also provide a pin field sensing system comprising a first pin field and a sensing pin field. The first pin field comprises a first plurality of pins mounted parallel to each other in a frame, wherein each pin of the first plurality of pins can be independently extended and retracted from the frame to sense an object. The sensing pin field comprises a plurality of parallel actuator driven pins, wherein the pins in the sensing pin field are controlled to replicate the surface of the object against which the first plurality of pins are placed.

[0012] The illustrative embodiments also provide a method for sensing or manipulating an object in a work environment. A plurality of pins in a pin field are placed against the object. The pin field includes a plurality of pins mounted parallel to each other in a frame, wherein each of the plurality of pins is independently extendable and retractable from the frame. Sensor information is received from the plurality of pins placed against the object to sense the object, or the plurality of pins placed against the object are controlled to manipulate the object in the work environment.

[0013] The features and functions can be achieved independently in various embodiments of the present disclosure or may be combined in yet other embodiments in which further details can be seen with reference to the following description and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The novel features which are believed to be characteristic of the illustrative embodiments are set forth in the appended claims. However, the illustrative embodiments together with the preferred modes of use, further objects and features thereof will be best understood by reference to the following detailed description of illustrative embodiments of the disclosure when read in conjunction with the accompanying drawings, in which:

[0015] Figure 1 is an illustration of a block diagram of a tactile pin field sensing and manipulation system according to one illustrative embodiment;

[0016] Figure 2 is a diagram of a pin field end effector according to an illustrative embodiment;

[0017] Figure 3is a diagram of a pin field according to an illustrative embodiment;

[0018] Figure 4 is an illustration of a pin in a retracted state according to an illustrative embodiment;

[0019] Figure 5 is an illustration of a pin in an extended state according to an illustrative embodiment;

[0020] Figure 6 is an illustration of objects in a work environment according to an illustrative embodiment;

[0021] Figure 7 is a diagram of an actuated tactile pin field according to an illustrative embodiment;

[0022] Figure 8 is an illustration of a three-dimensional CAD model generated according to one illustrative embodiment;

[0023] Fig. 9 is a diagram of a pin field sensing two wires according to an illustrative embodiment;

[0024] Fig.10 is generated according to an illustrative embodiment Fig. 9 Illustration of a 3D CAD model of the electrical wires in;

[0025] Fig.11 is an illustration of a pin field configured to turn a nut according to an illustrative embodiment;

[0026] Fig.12 is an illustration of a pin field configured to grasp an object according to an illustrative embodiment;

[0027] Fig.13 is an illustration of a pin field with tool access holes according to one illustrative embodiment; and

[0028] Fig.14 is an illustration of a block diagram of a data processing system in accordance with an illustrative embodiment. DETAILED DESCRIPTION

[0029] The illustrative embodiments recognize and take into account different considerations. For example, the illustrative embodiments recognize and take into account that current haptic solutions for telerobotics may lack effectiveness. For example, current haptic solutions may not provide useful haptic representations of sensed objects.

[0030] Illustrative embodiments provide a tactile, sensing, actuation pin field that can be used as an end effector in telerobotic applications. An end effector according to one illustrative embodiment handles tactile feedback in a completely new way by recreating the environment that an operator senses through a telerobotic rather than attempting to recreate and relay human sensations through technology. Recreating the environment to be sensed and allowing humans to interpret using natural sensations can achieve maximum effectiveness for a variety of applications in hazardous environments while maintaining operator safety.

[0031] According to one illustrative embodiment, each pin in a pin field is equipped with a combined pressure and temperature sensor on its head. These sensors allow the pins in the pin field to relay data about temperature and weight changes. The pressure or force data obtained from the pins as the pin field moves against an object can be analyzed for material type determination. Additionally, sensing the temperature at each pin head data point can also provide data to be analyzed for material type determination.

[0032] The illustrative embodiments also provide a method and apparatus for grasping a variety of objects that provide an improvement over claw-type end effectors and other conventional end effectors and manipulators that may not work universally well. Rather than seeking to find a perfect size or shape end effector design for grasping, the illustrative embodiments allow a pin field to contact an object of any geometry, engaging with all of its protrusions and intrusions to securely and universally hold when engaged when the actuator is locked in place.

[0033] The data processing system can use sensed information such as pin position, force information, temperature changes, and / or object shape to identify the material composition of the object (e.g., plastic, metal, wood, etc.). The system can also use this information to identify the shape of the object. The system controller can generate a three-dimensional model of the object, such as a computer-aided design (CAD) model, from the sensed object shape. For example, a pin field end effector according to one illustrative embodiment can grasp and release objects from varying orientations to obtain a three-dimensional outline of the object shape.

[0034] The pins in the pin field can then be manipulated from a CAD model of the object or from a control station physical pin field to move the pins in the end effector pin field to manipulate the object. For example, a model of the object can be used to set the pins into the appropriate shape to engage a portion of the object to be manipulated. This allows an operator to remotely mechanically engage and manipulate the object. This feature can be used to complete tasks such as moving and separating wires, installing or removing hardware (such as nuts), or picking up objects while only touching certain features.

[0035] Go to Figure 1, depicts an illustration of a block diagram of a tactile pin field sensing and manipulation system, according to an illustrative embodiment. The tactile pin field sensing and manipulation system 100 includes a remote pin field 102. The remote pin field 102 includes a plurality of pins 104. The plurality of pins 104 can be mounted substantially parallel to each other in a frame 105, and can be independently extendable 138 and retractable 140 from the frame 105.

[0036] The plurality of pins 104 may be configured to perform operations 113 on objects 108 in the working environment 110. For example, the plurality of pins 104 may be configured to sense 106 objects 108 in the working environment 110. Alternatively or additionally, the plurality of pins 104 in the remote pin field 102 may be configured to control manipulation 112 of objects 108 in the working environment 110. Sensing 106 and manipulating 112 are examples of operations 113 that may be performed in the working environment 110, but are not limited thereto.

[0037] Remote pin field 102 may be configured to sense 106 various different characteristics of object 108. For example, without limitation, remote pin field 110 may be configured to sense shape 142 of object 108, material 144 from which object 108 is made, or any other suitable characteristic or combination of characteristics of object 108.

[0038] For example, without limitation, a pin in plurality of pins 104 may include pressure sensor 132, temperature sensor 134, combined pressure and temperature sensor 136, or any other suitable sensor or combination of sensors for sensing 106 object 108. Pressure sensor 132 may provide pressure information 133. Temperature sensor 134 may provide temperature information 135. Combined pressure and temperature sensor 136 may provide both pressure information 133 and temperature information 135. According to one illustrative embodiment, pressure sensor 132 may include a force sensor, and pressure information 133 may include force information.

[0039] Working environment 110 may be any suitable environment in which it may be desirable to sense object 108 or manipulate object 108 in any suitable manner. Working environment 110 may be an environment in which it is impossible, difficult, or otherwise undesirable for a human to perform sensing or manipulation of object 108. For example, without limitation, working environment 110 may be an area in which it may be hazardous for humans to work or to which humans may not be able to access. For example, without limitation, working environment 110 may be a confined space on an aircraft 114 or other vehicle or other structure, or a factory 116, or other facility in which manufacturing, inspection, or other processes may be performed using remote pin field 102. Object 108 may be any suitable object in working environment 110.

[0040] For example, without limitation, work environment 110 may be factory 116 or other manufacturing or assembly facility in which aircraft 114 is being manufactured. For example, without limitation, object 108 may be electrical wires in a location that is difficult for humans to reach, such as inside a wing box of aircraft 114. In this example, remote pin field 102 may be used to sense 106 the electrical wires, for example, to determine whether the electrical wires are hot.

[0041] A robotic manipulator or other appropriate system or device may be used to move the remote pin field 102 into contact with the object 108 to perform sensing 106 or manipulation 112 of the object 108. The sensing 106 of the object 108 may be performed using appropriate sensors, such as pressure and temperature sensors, in the plurality of pins 104. The manipulation 112 of the object 108 may include moving the plurality of pins 104 to a desired configuration to interact with the object 108.

[0042] The controller 118 may be configured to control a robotic manipulator or other system or device to move the remote pin field 102 into contact with the object 108, and when the remote pin field 102 is in contact with the object 108, to move the robotic manipulator or other system or device to move the remote pin field 102 to perform a desired manipulation 112 of the object 108. The controller 118 may also be configured to control the movement of the plurality of pins 104 to configure the plurality of pins 104 into a desired shape for manipulating the object 108. The controller 118 may be configured to generate control signals 148 to control the operation of the remote pin field 102. For example, but not limitation, the controller 118 may provide control signals 148 to control the extension and retraction of pins in the plurality of pins 104.

[0043] Controller 118 may also be configured to process sensor information 146 from sensing 106 of object 108 by remote pin field 102 to determine and display various characteristics of object 108. Pressure information 133 and temperature information 135 are examples of sensor information 146, but are not limited thereto.

[0044] The controller 118 may be implemented in hardware or in hardware combined with software. For example, but not limited to, the controller 118 may be implemented using a computer or other data processing system.

[0045] The controller 118 may use information from the sensing 106 of the object 108 to create or modify a model 120 of the object 108. For example, the model 120 of the object 108 may be a computer-aided design, CAD, model 122. The model 120 of the object 108 may be used by the controller 118 to control positioning of the remote pin field relative to the object 108 or to control configuration of the plurality of pins 104 in the remote pin field 102 for manipulation 112 of the object 108.

[0046] The shape of the object 108 determined using the remote pin field 102 can be presented to the user 124 through the tactile pin field 126. The tactile pin field 126 includes a plurality of pins 128. The controller 118 can control the plurality of pins 128 in the tactile pin field 126 to represent the shape of the object 108 determined by the remote pin field 102. Therefore, the user 124 can feel the shape of the object 108 in the work environment 110 by feeling the plurality of pins 128 in the tactile pin field 126 with the hand 130.

[0047] Alternatively or additionally, the tactile pin field 126 may be used to control the plurality of pins 104 in the remote pin field 102. The user 124 may move the plurality of pins 128 in the tactile pin field 126 with the hand 130. The configuration of the plurality of pins 128 in the tactile pin field 126 (as controlled by the user 124) may be detected and used by the controller 118 to control the plurality of pins 104 in the remote pin field 102 to match the configuration of the plurality of pins 128 in the tactile pin field 126. Thus, for example, the user 124 may use the remote pin field 102 to perform manipulation 112 of the object 108 in the work environment 110 by manipulating the plurality of pins 128 in the tactile pin field 126.

[0048] Figure 1 The illustration of the tactile pin field sensing and manipulation system 100 in the illustrative embodiments is not meant to imply physical or architectural limitations to the manner in which the illustrative embodiments may be implemented. Other components may be used in addition to or in place of the components shown. Some components may not be necessary. In addition, the blocks are presented to illustrate some functional components. When implemented in the illustrative embodiments, one or more of these blocks may be combined, divided, or combined and divided into different blocks.

[0049] Go to Figure 2 , depicts a diagram of a pin field end effector according to an illustrative embodiment. Pin field end effector 200 includes pin field 201 and pin field 210. Pin field 201 and pin field 210 are Figure 1 An example of an implementation of the remote pin field 102 in FIG.

[0050] Pin field 201 includes a plurality of pins 202 mounted substantially parallel to one another in a frame 204. Frame 204 forms a housing for supporting pins 202. Each of pins 202 can be independently extended and retracted from frame 204 to sense an object or structure in a working environment 206, or to manipulate an object in working environment 206. Working environment 206 can be Figure 1 An example of a working environment 110 in FIG.

[0051] The size, number, and strength of the pins 202 in the pin field 201 may be selected based on the target application. For example, the strength of the pins 202 may be determined by the size of the pins 202 and the material from which the pins 202 are made. The size, shape, and thickness of the frame 204 may be selected based on the target application. The pin field 201 may be made more robust for use in harsh environments. For example, but not limited to, a rubber skirt may be provided around each pin 202 where each pin 202 exits the frame 204 housing.

[0052] The pin field 201 may be provided as an end effector attached to a manipulator arm 208. For example, but not limitation, the manipulator arm 208 may be part of a fixed or mobile robotic system configured for operation in the work environment 206. For example, but not limitation, the manipulator arm 208 may be part of a robotic platform, a crawler robot, or any other suitable robotic system. The manipulator arm 208 may be operated in a known manner to position the pin field 201 at a desired location in the work environment 206.

[0053] Pin field 210 includes a plurality of pins 212 mounted parallel to one another in a frame 214. Each of pins 212 can be independently extended and retracted from frame 214 to sense an object or structure in working environment 206 or to manipulate an object in working environment 206. Pin field 210 can be the same as or different from pin field 201. For example, in some embodiments, pin field 201 can include pressure sensing pins (not shown) while pin field 210 includes temperature sensing pins (also not shown). As another example, the position of each pin relative to the frame can be determined to sense the shape of an object.

[0054] The pin field 210 may be provided as an end effector attached to a manipulator arm 218. The manipulator arm 218 may be part of the same fixed or mobile robotic system as the manipulator arm 208, or may be part of a different fixed or mobile robotic system. In any case, the manipulator arm 218 may be operated in a known manner to position the pin field 210 at a desired location in the work environment 206.

[0055] Manipulator arm 208 and manipulator arm 218 may be simultaneously operated in a coordinated manner to position pin field 201 and pin field 210 to perform desired operations in work environment 206. For example, but not limited to, manipulator arm 208 may be operated to move pin field 201 in the direction indicated by arrow 222 to position pin field 201 on one side of object 220 in work environment 200. At the same time, manipulator arm 218 may be operated to move pin field 210 in the direction of arrow 224 to position pin field 210 on the other side of object 220. Thus, object 220 may be grasped between pin field 201 and pin field 210 so that object 220 may be moved or otherwise manipulated.

[0056] According to one illustrative embodiment, more or less than two pin fields mounted on more or less than two manipulator arms may be provided to perform desired operations in any appropriate working environment.

[0057] Go to Figure 3 , depicts a diagram of a pin field according to an illustrative embodiment. Pin field 300 is Figure 1 An example of a possible implementation of the remote pin field 102 in FIG. The pin field 300 is Figure 2 201 in which the pins 202 are shown separated from the frame 204.

[0058] Pin field 300 includes any suitable number of pins 302. Pins 302 are oriented in pin field 300 generally perpendicular to one another.

[0059] All pins 302 in pin field 300 may be identical. Alternatively, pin field 300 may include different types of pins that are specifically configured for different purposes. For example, but not limited to, pin field 300 may include pins that are specifically configured for grasping and manipulating objects, which are interspersed with pins that are specifically configured for sensing. For example, pins 302 configured for sensing may include appropriate sensors mounted therein, while pins 302 configured for manipulation may not include sensors. As another example, pins 302 configured for manipulating an object may include a surface material that prevents sliding between pins 302 and the object when pins 302 come into contact with the object. Conversely, the surface of pins 302 configured for sensing may be configured to enhance, or at least not detract from, the sensing of the sensors therein.

[0060] Wires 304 may be connected to each pin 302 in the pin field 300 to provide power and control signals to the pins 302 and to receive sensor signals from the pins 302 .

[0061] Go to Figure 4, depicts an illustration of a pin in a retracted state according to one illustrative embodiment. Pin 400 is Figure 2 An example of one possible implementation of a pin in a plurality of pins 104 in a remote pin field 102 in FIG. In this example, according to one illustrative embodiment, pin 400 is a miniature linear telescopic actuator that can be used to implement a pin in a pin field.

[0062] The distal end 402 of the pin 400 may have one or more sensors mounted therein or thereon in any suitable manner. For example, the distal end 402 of the pin 400 may include a pressure sensor, a temperature sensor, both a pressure sensor and a temperature sensor, or a combined temperature and pressure sensor. The combined pressure and temperature sensor at the distal end 402 of the pin 400 may include both a pressure sensor and a temperature sensor in one device. The distal end 402 of the pin 400 may also be referred to as the head of the pin 400.

[0063] The distal end 402 of the pin 400 is attached to the body 404 of the pin 400 by a telescoping portion 403. The telescoping portion 403 is configured to move inwardly and outwardly from the body 404, thereby moving the distal end 402 of the pin 400 toward and away from the body 404. The controlled linear movement of the telescoping portion 403 can be achieved in any suitable manner. For example, the pin 400 can be an electromechanical linear actuator, in which the telescoping portion 403 of the pin 400 moves by converting the rotational motion of a motor into a linear displacement in any suitable manner.

[0064] The body 404 of the pin 400 can be configured to be mounted in a frame or other suitable support structure. For example, but not limited to, the wire 406 can be connected to the body 404 at the proximal end 408 of the pin 400 to provide power and control signals to the pin 400 and receive sensor signals from the pin 400.

[0065] Go to Figure 5 , an illustration of a pin in an extended state is depicted in accordance with an illustrative embodiment. Figure 5 Shown in extended state Figure 4 In the extended state, the telescopic portion 403 of the pin 400 extends from the main body 404 of the pin 400 by a greater amount than when the pin 400 is in the retracted state (eg, Figure 4 According to an illustrative embodiment, the pin 400 can be controlled so that the telescopic portion 403 of the pin 400 can be extended to include Figure 4 The retracted state shown and Figure 5 The extended state shown and Figure 4 The retracted state shown and Figure 5 Any suitable extended position between the extended states shown.

[0066] Providing pins with combined pressure and temperature sensors in the plurality of pins 104 in the remote pin field 102 allows the remote pin field 102 to relay data regarding changes in temperature and weight of the object 108. For example, the weight of the object 108 may be determined using the remote pin field 102 by measuring the pressure or force exerted by the object 108 on the plurality of pins 104 in the remote pin field 102 while the object 108 is supported by the plurality of pins 104.

[0067] This not only allows for changes in temperature and weight to be detected, but can also be used to provide valuable data about the object 108 as it displays the value and change for each data point corresponding to a pin in the plurality of pins 104 in the remote pin field 102. This allows the data to be used to determine component differences on the object 108. For example, but not limited to, one application may be to determine the temperature of an electrical wire to monitor for an overload condition. The ambient temperature may be determined, and then by comparing temperatures above or below ambient to the object, taking time interval temperature readings at data points corresponding to pins in the remote pin field 102, the material type of the various components of the object 108 may be determined based on their conductivity and the time required to return to ambient temperature. Additionally or alternatively, when more than one wire is present in the object 108, the temperature difference between the wires may be calculated to sense when a wire is heating relative to the other wires.

[0068] The pressure or force data obtained from the plurality of pins 104 when the remote pin field 102 moves into contact with the object 108 may be analyzed for material type determination. The information obtained by measuring the changes in pressure or force detected by the plurality of pins 104 in the remote pin field 102 as the remote pin field 102 moves into contact with the object 108, or as the object 108 moves into contact with the remote pin field 102, may be referred to as force feedback information. For example, according to one illustrative embodiment, such force feedback information from the remote pin field 102 may be used to determine whether the material of the object 108 in contact with the remote pin field 102 is rubber-coated wire, uncoated copper wire, a static metal surface, or other material.

[0069] The determination of the shape and material of the object 108 made using the remote pin field 102 can be used as an input to a system that uses machine learning to identify the object 108. The machine learning system can be trained using a repository of information for standard hardware components or other objects. For example, but not limited to, a repository of CAD data for standard hardware components can be used to train the machine learning system. The determination of the shape and material of known objects made using the remote pin field 102 can be used to further train the machine learning system. Then, according to one illustrative embodiment, the trained machine learning system can be used to identify unknown objects based on the shape and material of the objects determined using the remote pin field 102.

[0070] For example, but not limited to, based on a determined shape having six sides, a height of the component identified as 0.25 inches, and a material determination of stainless steel based on force and temperature feedback, a machine learning system according to one illustrative embodiment may identify that the object is a stainless steel nut. Based on this identification, a system according to one illustrative embodiment may be configured to know how to interact with the identified object based on criteria and learned interactions, such as rotating the component counterclockwise to remove it from a structure.

[0071] The remote pin field 102 can also be used to remotely mechanically manipulate components of an object 108 in a work environment 110 with fine precision. Multiple pins 104 in the remote pin field 102 can be actuated to extend selected pins of the multiple pins 104 by an appropriate amount so that the multiple pins 104 are shaped to engage the object 108 in a desired manner. For example, the desired shape presented by the multiple pins 104 can be achieved by controlling the actuation of the multiple pins 104 using a CAD model or other appropriate model of the object 108. Alternatively, the user can use a tactile pin field to control the actuation of the multiple pins 104. The remote pin field 102 can then be moved to engage and manipulate the object 108. This allows an operator to remotely mechanically engage with and manipulate an object. This feature can be used to complete tasks such as moving and separating wires, installing or removing hardware (such as nuts), or picking up objects while only touching certain features.

[0072] For example, the plurality of pins 104 in the remote pin field 102 may be actuated to extend selected ones of the plurality of pins 104 so that the plurality of pins 104 assumes a shape that is complementary to the shape of the nut. The remote pin field 102 may then be moved to engage over a nut that needs to be removed, wherein the pins of the plurality of pins 104 adjacent to the exterior of the nut are selected to be locked in place in an extended state, while the remaining pins are retracted. The remote pin field 102 may then be rotated about its axis to remove the nut. Thus, the remote pin field 102 may be controlled to function as a universal socket wrench.

[0073] Go to Figure 6 , depicts an illustration of an object in a work environment according to an illustrative embodiment. Object 600 may be Figure 1 1. An example of an object 108 in a working environment 110 in FIG. In this example, object 600 includes a planar surface 602 having a generally annular shape 604 extending outwardly therefrom.

[0074] According to one illustrative embodiment, pin field 606 may be applied to a surface of object 600 to replicate its geometry through movement of actuated pins 608 of pin field 606. Pin field 606 is Figure 1An example of one possible implementation of the remote pin field 102 in FIG. The extension amount of each actuated pin 608 can be relayed back to a controller such as Figure 1 Controller 118 in.

[0075] Go to Figure 7 , depicts an illustration of an actuated tactile pin field according to one illustrative embodiment. The tactile pin field 700 is Figure 1 An example of one possible implementation of the tactile pin field 126 in FIG. 700 may be provided at an operator control station.

[0076] The tactile pin field 700 includes a plurality of parallel actuator-driven pins 702. According to one illustrative embodiment, the pins 702 in the tactile pin field 700 are controlled to replicate the surface of the sensed object 600 based on extension information provided from the pins 608 in the pin field 606 placed against the object 600 in the work environment. Thus, the operator can effectively feel the surface of the object 600 by feeling an exact replica of the geometry of the object 600 disposed on the tactile pin field 700. Thus, the operator at the control station can use their hands and sense of touch to interact and perceive a replicated version of the object 600 in the work environment.

[0077] Go to Figure 8 , depicts an illustration of a three-dimensional CAD model according to an illustrative embodiment. CAD model 800 is Figure 1 An example of one possible implementation of the CAD model 122 in FIG. 800 can be generated from pins in a pin field placed against objects in a work environment. Different surface heights of the objects represented in the CAD model 800 can be color coded or otherwise indicated in any suitable manner.

[0078] Go to Fig. 9 , depicts a diagram of a pin field sensing two wires according to an illustrative embodiment. Pin field 900 is Figure 1 FIG. 1 is an example of one possible implementation of a remote pin field 102 in FIG. 1 . In this example, pin field 900 is positioned to sense wire 902 and wire 904 .

[0079] Go to Fig.10 , according to an illustrative embodiment, depicts Fig. 9 CAD model 800 is a diagram of a three-dimensional CAD model of an electrical wire in FIG. Figure 1 An example of a possible implementation of the CAD model 122 in FIG. Fig. 9 CAD model 1000 is generated based on the information provided by pin field 900 of wire 902 and wire 904 .

[0080] In this example, the shapes 1002 and 1004 of the real-world wires 902 and 904 are captured in the CAD model 1000. In addition, changes in temperature information or pressure information, or a combination thereof, provided by the pin field 900 may be used to distinguish different characteristics of the wires 902 and 904. This may be indicated by color coding of the shapes 1002 and 1004 of the wires 902 and 904 in the CAD model 1000, or in any other suitable manner.

[0081] Go to Fig.11 , depicts an illustration of a pin field configured to turn a nut according to an illustrative embodiment. The pin field 1100 may be Figure 1 An example of an implementation of the remote pin field 102 in FIG.

[0082] The pins in the pin field 1100 can be manipulated from a computer CAD model or a tactile pin field to move the pins in the pin field 1100. This allows an operator to mechanically engage and manipulate an object remotely.

[0083] In this example, the pin field 1100 has been engaged over the nut 1104. The pins 1102 adjacent to the nut 1104 are selected to be extended and locked in place in the extended state. The remaining pins 1106 are retracted. The pin field can then be rotated to turn the nut 1104.

[0084] Go to Fig.12 , an illustration of a pin field configured to grasp an object is depicted in accordance with an illustrative embodiment. Pin field 1200 is Figure 1 An example of a remote pin field 102 in FIG.

[0085] Rather than seeking to find the perfect size or shape of end effector design for grasping, the illustrative embodiments allow the pin field 1200 to contact the geometry of the object 1204 to be grasped, engaging all of its protrusions and intrusions for a secure, universal hold when the actuator is locked in place upon engagement. In this case, the pins 1202 provide a perfect interference-style grasp for any object or geometry that may be encountered.

[0086] Fig.13 is an illustration of a pin field with tool access holes according to one illustrative embodiment. Pin field 1300 is Figure 1 An example of a remote pin field 102 in FIG.

[0087] The design of the pin field according to one illustrative embodiment can be modified for specific needs. For example, but not limited to, a tool access hole 1302 can be placed in the pin field 1300 to allow appropriate tools to access the parts held by the pin field 1300. Tools such as screwdrivers, scissors, wire cutters, chemical applicators, and any other appropriate tools can access the parts through the tool access hole 1302.

[0088] Go to Fig.14 , depicts an illustration of a block diagram of a data processing system according to an illustrative embodiment. Data processing system 1400 is a system for implementing Figure 1 An example of one possible implementation of a data processing system for controller 118 in FIG.

[0089] In this illustrative example, data processing system 1400 includes communications fabric 1402. Communications fabric 1402 provides communications between processor unit 1404, memory 1406, persistent storage 1408, communications unit 1410, input / output (I / O) unit 1412, and display 1414. Memory 1406, persistent storage 1408, communications unit 1410, input / output (I / O) unit 1412, and display 1414 are examples of resources that processor unit 1404 may access via communications fabric 1402.

[0090] Processor unit 1404 is used to run instructions for software that may be loaded into memory 1406. Processor unit 1404 may be a number of processors, a multi-processor core, or some other type of processor, depending on the particular implementation. Further, processor unit 1404 may be implemented using a number of heterogeneous processor systems in which a main processor is present with secondary processors on a single chip. As another illustrative example, processor unit 1404 may be a symmetric multi-processor system containing multiple processors of the same type.

[0091] Memory 1406 and permanent storage 1408 are examples of storage devices 1416. A storage device is any hardware capable of storing information (such as, but not limited to, data, program code in the form of functions, and other suitable information) on a temporary or permanent basis. In these examples, storage devices 1416 may also be referred to as computer-readable storage devices. In these examples, memory 1406 may be, for example, a random access memory or any other suitable volatile or non-volatile storage device. Permanent storage 1408 may take various forms, depending on the particular implementation.

[0092] For example, permanent storage 1408 may include one or more components or devices. For example, permanent storage 1408 may be a hard drive, a flash memory, a rewritable optical disk, a rewritable magnetic tape, or some combination thereof. The media used by permanent storage 1408 may also be removable. For example, a removable hard drive may be used for permanent storage 1408.

[0093] Communications unit 1410, in these examples, provides communications with other data processing systems or devices. Communications unit 1410, in these examples, is a network interface card. Communications unit 1410 may provide communications using one or both of physical and wireless communications links.

[0094] Input / output (I / O) unit 1412 allows input and output of data with other devices that may be connected to data processing system 1400. For example, input / output (I / O) unit 1412 may provide a connection for user input through a keyboard, a mouse, and / or some other suitable input device. Additionally, input / output (I / O) unit 1412 may send output to a printer. Display 1414 provides a mechanism for displaying information to a user.

[0095] Instructions for the operating system, applications, and / or programs may be located in storage devices 1416, which are in communication with processor unit 1404 through communications fabric 1402. In these illustrative examples, the instructions are present in functional form on persistent storage 1408. These instructions may be loaded into memory 1406 for execution by processor unit 1404. The processes of the different embodiments may be performed by processor unit 1404 using computer implemented instructions, which may be located in a memory, such as memory 1406.

[0096] These instructions are referred to as program instructions, program code, computer usable program code, or computer readable program code, which can be read and executed by a processor in processor unit 1404. The program code in different embodiments may be embodied on different physical or computer readable storage media, such as memory 1306 or permanent storage device 1408.

[0097] Program code 1418 is located in functional form on a selectively removable computer readable medium 1420 and can be loaded into or transferred to data processing system 1400 for execution by processor unit 1404. In these examples, program code 1418 and computer readable medium 1420 form a computer program product 1422. In one example, computer readable medium 1420 may be computer readable storage medium 1424 or computer readable signal medium 1426.

[0098] Computer-readable storage media 1424 may include, for example, an optical or magnetic disk inserted or placed in a drive or other device (that is part of permanent storage 1408) for transfer to a storage device, such as a hard drive, that is part of permanent storage 1408. Computer-readable storage media 1424 may also take the form of a permanent storage device, such as a hard drive, a thumb drive, or flash memory, that is connected to data processing system 1400. In some cases, computer-readable storage media 1424 may not be removable from data processing system 1400.

[0099] In these examples, computer-readable storage media 1424 is a physical or tangible storage device for storing program code 1418, rather than a medium that propagates or transmits program code 1418. Computer-readable storage media 1424 is also referred to as a computer-readable tangible storage device or a computer-readable physical storage device. In other words, computer-readable storage media 1424 is a medium that can be touched by a person.

[0100] Alternatively, program code 1418 may be transmitted to data processing system 1400 using computer readable signal media 1426. Computer readable signal media 1426 may be, for example, a propagated data signal containing program code 1418. For example, computer readable signal media 1426 may be an electromagnetic signal, an optical signal, and / or any other suitable type of signal. These signals may be transmitted via a communication link, such as a wireless communication link, a fiber optic cable, a coaxial cable, an electrical wire, and / or any other suitable type of communication link. In other words, in the illustrative examples, the communication link and / or connection may be physical or wireless.

[0101] In some illustrative embodiments, program code 1418 may be downloaded over a network from another device or data processing system to persistent storage 1408 via computer readable signal media 1426 for use in data processing system 1400. For example, program code stored in a computer readable storage medium in a server data processing system may be downloaded over a network from the server to data processing system 1400. The data processing system providing program code 1418 may be a server computer, a client computer, or some other device capable of storing and transmitting program code 1418.

[0102] The different components shown for data processing system 1400 are not meant to provide architectural limitations to the manner in which the different embodiments may be implemented. The different illustrative embodiments may be implemented in a data processing system that includes components in addition to and / or in place of those shown for data processing system 1400. Fig.14The other components shown in the figure may be different from the illustrative examples shown. Any hardware device or system capable of running program code may be used to implement different embodiments. As an example, data processing system 1400 may include organic components integrated with inorganic components and / or may be entirely composed of organic components other than human. For example, a storage device may be composed of an organic semiconductor.

[0103] In another illustrative example, processor unit 1404 may take the form of a hardware unit having circuits that are manufactured or configured for a specific purpose. This type of hardware can perform operations without requiring program code to be loaded from a storage device into memory to be configured to perform the operations.

[0104] For example, when processor unit 1404 is in the form of a hardware unit, processor unit 1404 can be a circuit system, an application specific integrated circuit (ASIC), a programmable logic device, or some other suitable type of hardware configured to perform several operations. Using a programmable logic device, the device is configured to perform several operations. The device can be reconfigured later, or the device can be permanently configured to perform several operations. Examples of programmable logic devices include, for example, programmable logic arrays, programmable array logic, field programmable logic arrays, field programmable gate arrays, and other suitable hardware devices. Using this type of implementation, program code 1418 can be omitted because the processes for different embodiments are implemented in hardware units.

[0105] In yet another illustrative example, processor unit 1404 may be implemented using a combination of processors found in computers and hardware units. Processor unit 1404 may have several hardware units and several processors that are configured to run program code 1418. Using the described example, some processes may be implemented in several hardware units, while other processes may be implemented in several processors.

[0106] In another example, a bus system can be used to implement communication structure 1402 and can include one or more buses, such as a system bus or an input / output bus. Of course, any suitable type of architecture can be used to implement the bus system, which provides data transfer between different components or devices attached to the bus system.

[0107] In addition, the communication unit 1410 may include several devices that send data, receive data, or send and receive data. The communication unit 1410 may be, for example, a modem or a network adapter, two network adapters, or some combination thereof. In addition, the memory may be, for example, the memory 1406 or a cache, such as the cache found in the interface and memory controller hub that may be present in the communication structure 1402.

[0108] The flow charts and block diagrams described herein illustrate the architecture, functions and operations of possible implementations of the systems, methods and computer program products according to various illustrative embodiments. In this regard, each box in the flow chart or block diagram can represent a module, segment or part of a code, which includes one or more executable instructions for realizing (one or more) specified logical functions. It should also be noted that in some alternative implementations, the functions marked in the box may not occur in the order marked in the accompanying drawings. For example, the functions of the two boxes shown in succession can be performed substantially simultaneously, or the functions of the boxes can sometimes be performed in reverse order, depending on the functions involved.

[0109] Furthermore, the present disclosure includes embodiments according to the following clauses:

[0110] 1. A pin field, comprising:

[0111] a plurality of pins mounted in parallel with each other in the frame; and

[0112] Multiple pins can be independently extended and retracted from the frame to sense or manipulate objects in a working environment.

[0113] 2. The insertion site according to clause 1, wherein the working environment is an aircraft or a factory.

[0114] 3. The pin field according to clause 1 further comprises a manipulator arm, wherein the pin field is an end effector attached to the manipulator arm.

[0115] 4. The pin field of clause 3, wherein the pin field is a first pin field and the manipulator arm is a first manipulator arm, and further comprising:

[0116] a second pin field comprising a second plurality of pins mounted parallel to one another in a second frame, wherein the second plurality of pins are independently extendable and retractable from the second frame;

[0117] a second manipulator arm, wherein the second pin field is an end effector attached to the second manipulator arm; and

[0118] Wherein the first manipulator arm and the second manipulator arm are configured to operate simultaneously to position the first pin field and the second pin field to perform an operation in a work environment.

[0119] 5. The pin field of clause 4, wherein the first manipulator arm and the second manipulator arm are configured to position the first pin field and the second pin field on different sides of the object for manipulation by simultaneous operation of the first pin field and the second pin field.

[0120] 6. The pin field of clause 1, wherein the plurality of pins comprises pins configured for manipulation of an object interspersed with pins configured for sensing of an object.

[0121] 7. The pin field of clause 1, further comprising wires connected to the plurality of pins to provide control signals to the plurality of pins and receive sensor information from the plurality of pins.

[0122] 8. The pin field of clause 1, wherein each pin of the plurality of pins comprises a linear telescopic actuator.

[0123] 9. The pin field of clause 1, wherein the distal end of each pin of the plurality of pins comprises a combined pressure and temperature sensor.

[0124] 10. The pin field of clause 1, wherein the plurality of pins are configured to provide pressure information and temperature information of an object against which the plurality of pins are placed, and further comprising:

[0125] A data processing system is configured to use pressure information and temperature information provided by a plurality of pins to identify a material from which an object is made.

[0126] 11. A pin field sensing system, comprising:

[0127] a remote pin field comprising a first plurality of pins mounted parallel to one another in a frame, wherein the first plurality of pins are extendable and retractable from the frame to sense a shape of an object; and

[0128] A tactile pin field comprising a second plurality of pins mounted parallel to one another, wherein the second plurality of pins are actuator driven pins, and wherein the second plurality of pins in the tactile pin field are controlled to replicate the shape of an object against which the first plurality of pins are placed.

[0129] 12. A method for sensing or manipulating an object in a work environment, comprising:

[0130] placing a plurality of pins against the object in a pin field, wherein the pin field includes a plurality of pins mounted parallel to one another in a frame, wherein the plurality of pins are independently extendable and retractable from the frame; and

[0131] Sensor information is received from a plurality of pins placed against an object for sensing the object in a working environment, or a plurality of pins placed against an object is controlled for manipulating the object in a working environment.

[0132] 13. The method of clause 12, wherein the work environment is an aircraft or a factory.

[0133] 14. A method according to clause 12, wherein:

[0134] The pin field is an end effector attached to the manipulator arm; and

[0135] Placing the pin field against the object includes operating a manipulator arm to position the pin field against the object.

[0136] 15. The method of clause 12, wherein the plurality of pins includes pins configured for manipulating an object interspersed with pins configured for sensing an object.

[0137] 16. The method of clause 12, further comprising wires connected to the plurality of pins to provide control signals to the plurality of pins and to receive sensor information from the plurality of pins.

[0138] 17. The method of clause 12, wherein each pin of the plurality of pins comprises a linear telescopic actuator.

[0139] 18. A method according to clause 12, wherein:

[0140] A plurality of pins in a pin field are placed against an object including a first wire and a second wire;

[0141] Receiving sensor information includes receiving temperature information from pins in a pin field positioned against the first and second wires; and

[0142] Temperature information from pins placed against the first and second wires is compared to determine a condition of one of the first and second wires.

[0143] 19. The method of clause 12, wherein receiving sensor information from the plurality of pins comprises receiving pressure information and temperature information from the plurality of pins of an object against which the plurality of pins are placed, and further comprising:

[0144] The material an object is made of is identified using pressure and temperature information provided by multiple pins.

[0145] 20. The method of clause 12, further comprising controlling the plurality of pins in the tactile pin field to replicate a shape of an object against which the plurality of pins are placed.

[0146] The description of different illustrative embodiments has been presented for the purpose of illustration and description, and the description of different illustrative embodiments is not intended to be exhaustive or limited to the embodiments in the disclosed form. Many modifications and variations are apparent to those of ordinary skill in the art. In addition, different illustrative embodiments may provide different features compared to other desired embodiments. One or more selected embodiments are selected and described in order to best explain the principles of the embodiments, practical applications, and to enable others skilled in the art to understand the present disclosure of various embodiments with various modifications suitable for the intended specific use.

Claims

1. A pin field (102), comprising: A plurality of pins (104) mounted in parallel with each other in a frame (105); and wherein the plurality of pins (104) are independently extendable (138) and retractable (140) from the frame (105) for sensing (106) an object (108) or manipulating (112) the object (108) in a working environment (110); wherein the distal end (402) of each of the plurality of pins (104) comprises a combined pressure and temperature sensor (136); and wherein the plurality of pins (104) are configured to provide pressure information (133) and temperature information (135) of the object (108) against which the plurality of pins (104) are placed, and the pin field (102) further comprises: A data processing system (1400) is configured to use the pressure information (133) and the temperature information (135) provided by the plurality of pins (104) to identify a material (144) of which the object (108) is made.

2. The pin pad (102) of claim 1, wherein the work environment (110) is an aircraft (114) or a factory (116).

3. The pin field (102) of claim 1, further comprising a manipulator arm, wherein the pin field is an end effector attached to the manipulator arm.

4. The pin field (102) of claim 3, wherein the pin field is a first pin field (201) and the manipulator arm is a first manipulator arm (208), and further comprising: a second pin field (210) comprising a second plurality of pins (212) mounted parallel to one another in a second frame (214), wherein the second plurality of pins (212) are independently extendable (138) and retractable (140) from the second frame (214); a second manipulator arm (218), wherein the second pin field (210) is an end effector attached to the second manipulator arm (218); and The first manipulator arm (208) and the second manipulator arm (218) are configured to operate simultaneously to position the first pin field (201) and the second pin field (210) to perform an operation (113) in the working environment (110).

5. The pin field (102) of claim 4, wherein the first manipulator arm (208) and the second manipulator arm (218) are configured to position the first pin field (201) and the second pin field (210) on different sides of the object (108) to manipulate (112) by operating the first pin field (201) and the second pin field (210) simultaneously.

6. A pin field (102) according to any one of claims 1-5, wherein the plurality of pins (104) include pins configured for the manipulation (112) of the object (108), which are interspersed with pins configured for the sensing (106) of the object (108).

7. The pin field (102) according to any one of claims 1-5, further comprising wires (304) connected to the plurality of pins (104) to provide control signals (148) to the plurality of pins (104) and receive sensor information (146) from the plurality of pins (104).

8. The pin field (102) according to any one of claims 1-5, wherein each pin of the plurality of pins (104) comprises a linear telescopic actuator (400).

9. A pin field sensing system (100), comprising: A remote pin field (102) comprising a first plurality of pins (104) mounted parallel to one another in a frame (105), wherein the first plurality of pins (104) are extendable (138) and retractable (140) from the frame (105) to sense a shape (142) of an object (108); and wherein a distal end (402) of each pin in the first plurality of pins (104) comprises a combined pressure and temperature sensor (136); and a tactile pin field (126) comprising a second plurality of pins (128) mounted parallel to one another, wherein the second plurality of pins (128) are actuator driven pins, and wherein the second plurality of pins (128) in the tactile pin field (126) are controlled to replicate the shape (142) of the object (108) against which the first plurality of pins (104) are placed; in: The first plurality of pins are placed against an object; receiving pressure information and temperature information from the combined pressure and temperature sensor located at a distal end of each pin in the first plurality of pins; and The pressure information and the temperature information provided by the first plurality of pins are used to identify the material that the object is made of.

10. A method of sensing (106) or manipulating (112) an object (108) in a working environment (110), comprising: placing a plurality of pins (104) in a pin field (102) against the object (108), wherein the pin field (102) comprises a plurality of pins (104) mounted parallel to one another in a frame (105), wherein the plurality of pins (104) are independently extendable (138) and retractable (140) from the frame (105); and wherein a distal end (402) of each pin in the plurality of pins (104) comprises a combined pressure and temperature sensor (136); and receiving sensor information (146) from the plurality of pins (104) positioned against the object (108) for sensing (106) the object (108) in the working environment (110), or controlling the plurality of pins (104) positioned against the object (108) for manipulating (112) the object (108) in the working environment (110); wherein receiving sensor information (146) from the plurality of pins (104) comprises receiving pressure information (133) and temperature information (135) of the object (108) against which the plurality of pins (104) are placed from the combined pressure and temperature sensor located at a distal end of each of the plurality of pins (104), and the method further comprises: The pressure information (133) and the temperature information (135) provided by the plurality of pins (104) are used to identify a material (144) from which the object (108) is made.

11. The method according to claim 10, wherein: The pin field (102) is an end effector attached to a manipulator arm; and Placing the pin field (102) against the object (108) includes operating the manipulator arm to position the pin field (102) against the object (108); wherein the plurality of pins (104) includes pins configured for manipulating (112) the object (108) interspersed with pins configured for sensing (106) the object (108).

12. The method according to any one of claims 10-11, further comprising a wire (304) connected to the plurality of pins (104) to provide a control signal (148) to the plurality of pins (104) and receive sensor information (146) from the plurality of pins (104); and wherein each of the plurality of pins (104) comprises a linear telescopic actuator (400).

13. The method according to any one of claims 10-11, wherein: The plurality of pins (104) in the pin field (102) are placed against an object (108) comprising a first wire (902) and a second wire (904); Receiving the sensor information (146) includes receiving temperature information (135) from pins in the pin field (102) positioned against the first wire (902) and the second wire (904); and The temperature information (135) from the pins placed against the first wire (902) and the second wire (904) is compared to determine a condition of one of the first wire (902) and the second wire (904).

14. The method according to any one of claims 10-11, further comprising: A plurality of pins (128) in a tactile pin field (126) is controlled to replicate a shape (142) of the object (108) against which the plurality of pins (104) are placed.

Images