End effector and haptic control system

Abstract

An end effector and haptic control system for telerobotics is provided. The end effector is configured for attachment to a robot and comprises a vision hub having a plurality of sensors for viewing the environment for telerobotic control and a projector for projecting onto objects, surfaces or a user's body. A user interacts with the projections to direct the robot to perform various tasks. The user interaction is detected by the sensors in the vision hub to enable feedback control of the robot. The end effector can be used as part of a haptic control system in combination with auxiliary haptic devices to direct the robot to perform a variety of tasks.

Description

END EFFECTOR AND HAPTIC CONTROL SYSTEMTechnical Field

[0001] The embodiments disclosed herein relate to non-autonomous, semi- autonomous, autonomous, and teleoperated robots, and, in particular to the robot end effector and haptic control system for such robotic applications.Introduction

[0002] Robotic systems are typically purpose-built for specific applications and thus cannot be easily adapted to perform a wide range of specialized tasks. As such, there are limited “out of the box” telerobotic solutions that can be adapted for a wide range of tasks and applications. A further challenge in robotics, especially telerobotics, is that the user / operator must be specially trained to correctly interact with, and control, the robot using, for example, an input device, or virtual / augmented reality equipment. This is time consuming and requires specialized equipment.

[0003] The combination of telerobotics with a multi-purpose end effector and haptic feedback and control offers numerous potential use cases across various industries and applications. The versatility of this approach opens up possibilities for remote operations, training, healthcare, exploration, and various other fields where precise control, sensory feedback, and remote interaction are required. For example, by combining sensor fusion, a projected display, quick tool swapping, and haptic feedback and input, telerobotics can be further extended to be used by individuals with sensory (e.g., auditory) constraints and in loud and variable operating environments.

[0004] Accordingly, there is a need for new robotic systems - especially in the telerobotic field - which employ a projected display, multitude of sensors, automated tool swapping and haptic feedback and control that are easy and intuitive to use.Brief Description of the Drawings

[0005] The drawings included herewith are for illustrating various examples of articles, methods, and apparatuses of the present specification. In the drawings:

[0006] FIGS. 1 A and 1 B are perspective views of an end effector, according to an embodiment;

[0007] FIG. 1 C is a front view of the end effector shown in FIGS. 1 A and 1 B;

[0008] FIG. 1 D is a rear view of the end effector shown in FIGS. 1 A and 1 B;

[0009] FIG. 1 E side view of the end effector shown in FIGS. 1 A and 1 B;

[0010] FIG. 1 F is a front cross-sectional view through section A-A in FIG. 1 E;

[0011] FIGS. 1 G is a perspective view of the latching mechanism shown in FIG. 1 F.

[0012] FIG. 2A is perspective view of a vision hub, according to an embodiment;

[0013] FIG. 2B is a front view of the vision hub shown in FIG. 2A;

[0014] FIG. 20 is a perspective view of the vision hub shown in FIG. 2A, removed from the housing;

[0015] FIG. 2D is a side view of the vision hub shown in FIG. 2C;

[0016] FIG. 2E is a rear perspective view of the vision hub shown in FIG. 2C;

[0017] FIG. 3 is a diagram of a projected graphical user interface, according to an embodiment;

[0018] FIGS. 4A-4B are top and bottom perspective views, respectively, of a haptic device, according to an embodiment;

[0019] FIG. 40 is a perspective view of the haptic device of FIGS. 4A-4B shown mounted to a table station; and

[0020] FIG. 4D is a side view of the haptic device of FIGS. 4A-4B shown mounted to a robot.Detailed Description

[0021] Various apparatuses or processes will be described below to provide an example of each claimed embodiment. No embodiment described below limits any claimed embodiment and any claimed embodiment may cover processes or apparatusesthat differ from those described below. The claimed embodiments are not limited to apparatuses or processes having all of the features of any one apparatus or process described below or to features common to multiple or all of the apparatuses described below.

[0022] One or more systems described herein may be implemented in computer programs executing on programmable computers, each comprising at least one processor, a data storage system (including volatile and non-volatile memory and / or storage elements), at least one input device, and at least one output device. For example, and without limitation, the programmable computer may be a programmable logic unit, a mainframe computer, server, and personal computer, cloud based program or system, laptop, personal data assistance, cellular telephone, smartphone, or tablet device.

[0023] Each program is preferably implemented in a high level procedural or object oriented programming and / or scripting language to communicate with a computer system. However, the programs can be implemented in assembly or machine language, if desired. In any case, the language may be a compiled or interpreted language. Each such computer program is preferably stored on a storage media or a device readable by a general or special purpose programmable computer for configuring and operating the computer when the storage media or device is read by the computer to perform the procedures described herein.

[0024] The term “controller” as used herein refers to computer processors (e.g., central processing units, graphics processing units), integrated circuits, systems-on-a- chip, including associated hardware and software, being configured to execute instructions, perform calculations and / or process signals / data as the case may be.

[0025] A description of an embodiment with several components in communication with each other does not imply that all such components are required. On the contrary a variety of optional components are described to illustrate the wide variety of possible embodiments of the present invention.

[0026] Further, although process steps, method steps, algorithms or the like may be described (in the disclosure and I or in the claims) in a sequential order, such processes, methods and algorithms may be configured to work in alternate orders. In other words, any sequence or order of steps that may be described does not necessarily indicate arequirement that the steps be performed in that order. The steps of processes described herein may be performed in any order that is practical. Further, some steps may be performed simultaneously.

[0027] When a single device or article is described herein, it will be readily apparent that more than one device I article (whether or not they cooperate) may be used in place of a single device I article. Similarly, where more than one device or article is described herein (whether or not they cooperate), it will be readily apparent that a single device I article may be used in place of the more than one device or article.

[0028] Referring to FIGS. 1A-1 E, shown therein is an end effector 100, according to an embodiment. The end effector 100 is configured for attachment to robotic arms or robot apparatuses (e.g., the autonomous robot apparatus disclosed in U.S. Patent Publication No. 2022 / 0362944), for telerobotic applications. Generally, the end effector 100 electronically and physically connects to a robotic arm (not shown) at a robot end 130 and electronically and physically connects to a tool (not shown) at a tool end 140 to enable telerobotic control of the tool using a plurality of sensors and a projector contained within the end effector 100.

[0029] The end effector 100 includes servo connection points 108 on the robot end 130 for physically connecting the end effector 100 to the robot such that the end effector 100 (and the tool attached thereto) can be moved and oriented by the robot. The robot end 130 additionally includes a high-speed data connection port 111 (e.g., USB, CANbus) for electrically connecting the robot and the end effector 100 for transmitting data and power therebetween. Furthermore, the robot end 130 includes a pocket for routing fluid conduits which run within the robot, to allow for fluids to be passed through end effector 100 and out the tool end 140 (see FIGS. 1 F-1 G).

[0030] A plurality of LED channels 114 are located adjacent to the robot end 130. The LED channels 114 are translucent features through which light from indicator LEDs disposed on an interior of the end effector 110 can be seen from the exterior. The indicator LEDs can be individually controlled and are capable of achieving a wide color range, and can have their intensities varied. The indicator LEDs can be used for indicating an operational status, connection status, etc., of the end effector 100 and / or the robot. Forexample, a flashing red light may indicate a fault in connecting the tool to the end effector 100.

[0031] The end effector 100 includes a mounting interface 106 on the tool end 140 for physically and electronically connecting to a plurality of removably attachable tools for various applications. For example, the end effector 100 may be removably attached, at the mounting interface 106, to a claw or gripping tool for picking up or manipulating objects.

[0032] The mounting interface 106 includes an array of pogo pins 110 disposed thereon for forming an electrical connection with corresponding pogo pins on the tool attached to the end effector 100. The array of pogo pins 110 transmit data, control signals and power between the robot and the tool via the end effector 100. The data transfer can take the form of whatever communication protocol is necessary, including but not limited to, USB, I2C, CAN, and SPI. The versatility of this electronics interface allows for communication with not only tools designed specifically to be integrated with the robot / end effector 100, but also with off-the-shelf instrumentation. As an example, in the field of sonography, existing probes from OEMs such as GE®, Phillips®, Clarius®, Siemens®, Butterfly™, etc. can be integrated into a tool and communicate with the robot without requiring any modifications to the robot or end effector 100.

[0033] The end effector 100 is able to receive the fluids or slurries (not limited to compressed gasses, water, sanitizing liquids, ultrasound gel) being carried through the length of the robot and pass them from the robot end 130 to the mounting interface 106. The mounting interface 106 includes fluid connection points 113a, 113b which align with similar features on mating tools to form a face seal. Two working fluids (in any combination of gas, fluid, and slurry) can be exchanged between the robot and the tool via the two different fluid conduits 112a, 112b and fluid connection points 113a, 113b. One fluid conduit 112a and face seal 112a may be configured for a gas and a second fluid conduit 112b and face seal 112b may be configured for a liquid.

[0034] The mounting interface 106 includes a latching mechanism 120 for removably securing the tool to the end effector 100. While the end effector 100 is attached to the robot at the robot end 130 and latched to the tool at the tool end 140 the two movetogether, allowing for the robot to move the tool around. The latching mechanism 120 includes a pair of locking lugs 122, a latch 124 and a latch release 126.

[0035] Referring to FIGS. 1 F-1 G, the locking lugs 122 (curse threads) engage similar features on the tool to prevent relative axial motion between the end effector 100 and the tool on the condition that there is no relative rotation (similar to a screw). The latch 124 prevents relative rotation by swinging into a pocket on the tool. Together, the locking lugs 122 and the latch 124 prevent both relative axial and rotational motion, thus locking the end effector 100 to the tool. This system is “fail safe”, meaning in the event of power loss, the tool will remain fixed to the end effector 100 (i.e., fixed to the robot) and will not fall off risking injury to the user or damage to the tool.

[0036] The latch 124 is actuated using a constant force spring and a shape memory alloy (SMA) cable 123. The latch 124 is biased in a locked “fail safe” position, such that should power be lost, the latch 124 will remain locked. The constant force spring constantly maintains the latch 124 in the locked position (as shown in FIGS. 1 F-1 G). When a current passes through the SMA cable 123, its length shortens. The SMA cable 123 is arranged such that that when it shortens, it counters the constant force spring to move the latch 124 in the direction of arrow 125, thus opening / unlocking the latch mechanism 120. When current to the SMA cable 123 is cut, the constant force spring returns to being the stronger of the two forces, and pulls the latch 124 into the locked position while stretching the SMA cable 123 back into its “lengthened” state. Current is supplied to the SMA cable 123 via electrical connectors 128 allowing for electronic control of the latching mechanism 120 by the robot.

[0037] In addition, a latch release button 126 is connected to the latch 124 which allows for a user to manually unlatch the tool from the end effector 100 in case of emergency or power loss. When this button is depressed in the direction of arrow 127, it causes the latch 124 to move to the unlock position (direction of arrow 125) as when the SMA cable shortens.

[0038] Positioned within the end effector 100, is a multi-axis load cell. The load cell is configured to measure loads in 6 degrees of freedom: 3 linear forces and 3 moments. These forces can be used alone or through sensor fusion to determine if theend effector 100 is experiencing load. In telerobotic applications, the values measured by the load cell can be used by a haptic device (FIGS. 4A-4D) to provide haptic feedback to the user, indicating not only that the end effector 100 or a tool attached to it is contacting an object, but with how much force and moment and at what angle.

[0039] The end effector 100 includes a vision hub 102 housing a vision module having a plurality of sensors for sensing the environment around the robot. The vision hub 102 further includes a projector for projecting video and / or images (e.g., graphical user interfaces) onto surrounding objects, surfaces, or onto a user’s body. The vision hub 102 is described in detail below with reference to FIGS. 2A-2D. The projector within the vision hub 102 may be substituted for alternative sensors or devices as required for the specific application.

[0040] The vision hub 102 includes a visor 104 for protecting the sensors and projector from dust and environmental hazards. The visor 104 is substantially transparent to, at least, visible light and infrared radiation. The vision hub 102 may include a solid- state fan (e.g., AirJet®) configured to blow air over the visor 104 for cleaning the visor 104.

[0041] Referring to FIGS. 2A-2E, shown therein is a vision hub 200 according to an embodiment. The vision hub 200 may be the vision hub 102 shown in FIGS. 1A-1 F.

[0042] The vision hub 200 includes two RGB cameras 202a, 202b, a time-of-flight (TOF) sensor 204 disposed between the RGB cameras 202a, 202b and LED diodes 206a, 206b for the TOF sensor 204. The vision hub 200 further includes a thermopile (infrared heat) sensor or infrared camera 208. Collectively, the RGB cameras 202a, 202b, the TOF sensor 204, the LED diodes 206 and the thermopile / infrared camera 208 are referred to herein as “the sensors.”

[0043] The vision hub 200 is contained between a housing 201 and the visor (i.e. , visor 104). For ease of illustration the visor is not shown. The sensors 202, 204, 206, 208 are arranged to sense the environment through the visor.

[0044] The RGB cameras 202a, 202b are stereoscopic RGB cameras with a global shutter specifically positioned to create a parallax effect for enhanced depth perception.In addition, the RGB cameras 202a, 202b are spaced apart at a distance corresponding to the distance between human eyes, recreating the human parallax effect.

[0045] The vision hub 200 further includes a projector 210 for projecting images, video and / or graphical user interfaces onto nearby objects, surfaces or users. The vision hub 200 further includes a dynamic focusing system (FIGS. 20 and 2D) to maintain a clear and in-focus projection regardless of its distance from the surface / object onto which the projector 210 is projecting. This is important, as the distance from the projector 210 to the projection surface / object will vary between, and within, applications. It should not be assumed that the distance will always be the same.

[0046] Projector 210 focusing is accomplished through use of an SMA cable 212. The SMA cable 212 is fixed to a PCB 214 on both ends and is secured to a projector focus knob 216 about midway along its length. By controlling the current being run through the SMA cable 212, its length can be finely controlled, thus providing fine control over the focus knob 216 which it is secured to. Generally, the focus knob 216 turns in a direction (e.g., clockwise) when the SMA cable 212 shortens and turns in an opposite direction (e.g., counterclockwise) when the SMA cable 212 lengthens.

[0047] According to some embodiments, the position of the SMA cable 212 and focus knob 216 can be controlled by position-sensing sensors (optical, magnetic, capacitive, or inductive) in a feedback control loop by the dynamic control system.

[0048] According to other embodiments, the position of the SMA cable 212 and focus knob 216 can be controlled by monitoring the electrical resistance of the SMA cable 212 in a feedback control loop. The SMA cable’s resistance is directly linked to its length; therefore, by knowing the resistance, the state of the SMA cable 212 and focus knob 216 can be accurately determined and the focus of the projector 210 can be controlled. This method of focus control can be employed alone, or in combination with position-sensing sensors are noted above.

[0049] Generally, the focus of the projector 210 is automatically controlled by the dynamic control system using data measured by the sensors 202, 204, 206, 208 pertaining to the distance of the projection surface to the vision hub 200. The dynamic control system includes a controller configured to receive input from position-sensingsensors (including the sensors 202, 204, 206, 208 in the vision module) to determine a distance to the object or the surface onto which the projector 210 is projecting and vary the current through the SMA cable 212 according to the distance to keep the projected video / images in focus if the distance to the object / surface changes. If desired, direct control over projector 210 focus can be performed manually by the user.

[0050] The vision hub 200 includes a frame 230 to which the PCB 214, the sensors 202, 204, 206, 208 and the projector 210 are mounted. Preferably the frame is constructed of copper sheet metal for structural rigidity and for operation as a heat sink. Similarly, a pulley 232 over which the SMA cable 212 is wound is preferably constructed of copper sheet metal for heat dissipation purposes.

[0051] Using the sensors 202, 204, 206, 208 of the vision hub 200 in tandem with the projector 210 and the capabilities of the end effector 100, enables versatile technical applications when integrated with a robot (e.g., the robot disclosed in U.S. Patent Publication No. 2022 / 0362944) as described below. According to various embodiments, the PCB 214 may further include one or more controllers configured for various operations and applications as described below.

[0052] According to an embodiment, the end-effector 100 is configured to display image overlays onto workspace surfaces or specific objects to guide the robot's movements or provide visual instructions to the user. This can be used in guiding the user to comply with various prompts. Visual instructions can be particularly useful for users with auditory constraints, as they can rely on the projected images to understand and comply with the robot's prompts and instructions.

[0053] The vision hub 200 is configured to implement computer vision algorithms to analyze the environment, allowing the robot to recognize objects and interpret visual cues. Image processing techniques enhance the clarity and quality of the projected instructions. Augmented reality (AR) can be incorporated to blend virtual content with the real-world environment, providing an immersive experience. Integration of sensors such as cameras and depth sensors enables real-time data gathering for adaptive guidance. Machine learning and Al algorithms enable the robot to learn and interpret visual cues, generating appropriate instructions. Together, these technologies enable the robot toproject clear and accurate visual cues, aiding users in complying with instructions and enhancing task performance.

[0054] According to an embodiment, the computer vision algorithms include hand recognition artificial intelligence for gesture recognition and touch estimation, using both ToF sensor 204 and the RGB cameras 202a, 202b to overlay the projector 210 to draw a path or select a target onto a user or a patient’s body. The target is projected by the projector 210 onto the user’s body, and the robot will take an image of the projection and replicate it in real life or replicate the path.

[0055] According to an embodiment, the end effector projects anatomical information onto a patient’s body, enabling the operator to have a visual reference during procedures. This enhances the operator's understanding of the patient's anatomy and facilitates accurate manipulation of tools and instruments. This can be accomplished through the use of the ToF sensor 204 within the vision hub 200, and artificial intelligence such as semantic segmentation or pose estimation.

[0056] According to an embodiment, using advanced computer vision algorithms in conjunction with the end-effector 100, the robot can recognize and highlight specific objects or components. This facilitates picking and placing operations, reducing the chances of errors and miscommunication. Here too, the robot communicates with the user, e.g., using the end effector 100 to shine a light only at the object and / or project text or symbols onto the object. Fidelity of pick and place operations may be increased using the RGB cameras 202a, 202b and the projector 210 by depth mapping by projecting a dot pattern or a checkers pattern onto the object as the robot moves to pick it.

[0057] This capability is valuable for tasks such as picking and placing objects, as the robot can accurately identify the target objects and provide visual indications or highlights, reducing the chances of errors and improving overall efficiency. Object identification in the robot system leverages a suite of advanced technologies to enhance its capabilities. The system incorporates computer vision algorithms that analyze the visual input from sensors 202, 204, 206, 208 to identify specific objects or components in the environment. Machine learning and deep learning algorithms enable the robot to learn and recognize objects based on patterns, shapes, and characteristics. This allows foraccurate and reliable identification of objects, while reducing errors in picking and placing operations.

[0058] Additionally, the system can utilize the ToF sensor 204 to obtain precise distance information and create 3D models of objects for improved identification. The integration of advanced image processing techniques further enhances the clarity and accuracy of object recognition. By combining these approaches, the robot can efficiently identify and highlight objects, enabling more efficient and reliable operations in various tasks and industries.

[0059] According to an embodiment, the projector 210, in combination with the ToF sensor 204, is configured to project warning signals or indicators onto workspace objects / surfaces to alert the user or prevent collisions with nearby objects. The robot utilizes computer vision algorithms and image processing techniques to analyze the sensor data and identify potential collision risks. When the TOF sensor 204 detects an object within a predefined proximity range, it triggers the projector 210 to display warning signals or indicators onto workspace objects / surfaces.

[0060] These visual cues alert the user to the presence of obstacles or potential collision hazards, enabling them to take appropriate action and avoid accidents. By leveraging these technologies, the robot enhances situational awareness and promotes safe working practices by providing real-time visual feedback and proactive warnings about the proximity of objects. By visually highlighting these obstacles, the robot enhances safety and prevents accidental collisions, especially in dynamic or crowded environments.

[0061] According to an embodiment, the projector 210 is configured to project images and guides directly onto a patient's body or other surfaces. This projection can be applied to several areas to improve the accuracy and precision guidance of tools attached to the end effector 100. The TOF sensor 204 and / or the RGB cameras 202 may be employed to precisely detect the patient's body contours and spatial position. This information is then used to dynamically adjust the projected image to align perfectly with the patient's anatomy even though the surface is undulated / not uniform, ensuring optimal visualization and guidance. Furthermore, the vision hub 200 may incorporate calibrationmechanisms to account for variations in patient size and position, allowing for personalized and adaptable projection. If there is an image adjacent to the body the text is going to get distorted, the ToF sensor 204 cancels out the distortion by the soft body deformation.

[0062] By incorporating the projection feature into various non-healthcare fields, the end effector 100 becomes a versatile tool for providing visual guidance, enhancing training and education, improving productivity in assembly and maintenance tasks, creating interactive experiences in retail and hospitality, and facilitating design visualization in architecture and product development. The ability to project images and guides onto surfaces adds an extra layer of interactivity, precision, and efficiency to a wide range of tasks across different industries.

[0063] According to an embodiment, the vision hub 200 is configured with the pair of RGB cameras 202 being spaced apart at approximately the same distance of human eyes, recreating the parallax effect. Users are able to observe the robot’s view of the environment through the vision hub 200 with a virtual reality (VR) or augmented reality (AG) headset for telerobotic control. The user will have the option of indicating what they are looking at based on their head movement with the VR headset. As the users head is moving, the end effector 100 will move accordingly and project a visual cue as to what the user is observing.

[0064] According to an embodiment, VR / AR technologies are integrated with the end effector 100 to provide an immersive experience for the operator. VR allows the operator to have a first-person perspective of the robot's surroundings, enhancing situational awareness. AR overlays virtual information onto the operator's real-world view, providing additional guidance and visual cues during procedures.

[0065] According to an embodiment, a robot incorporates RGB cameras mounted on a gimbal at the robot's base. These cameras capture the robot's surroundings and transmit the visual data to the operator's VR headset. This enables the operator to perceive the environment from the robot's perspective, enhancing spatial awareness and facilitating more precise control of the robot's movements.

[0066] According to an embodiment, a further vision module embedded in the base of the robot and the vision hub 200 of the end effector 100 can be used to recreate depth perception and a dynamic view for patient and operator during a two-way video conferencing. Based on the head and body movement captured by a camera of the operator, the information can be relayed to the robot to contort the movement of the arm, the end effector, and gimbal system to capture the visual information and relay the information to the operator in the orientation of the perspective of the operator’s head position. Based on their head and bodily movement, the visual display of the operator will be changed and contorted to the perspective of the patient’s head position. Advantageously, this approach is less computationally intensive than conventional approaches of using multiple fixed cameras for tracking head and body movements to generate a dynamic visual display from the perspective of the viewer.

[0067] According to an embodiment, virtual reality is applied to a robot arm to make it visually resemble the operator's arm. By overlaying the operator's arm movements onto the robot arm, the system creates a more intuitive control interface, allowing the operator to perform tasks with a sense of embodiment and familiarity. Augmenting away the arm from the field of view.

[0068] According to an embodiment, visual cues, such as projected targets or markers, can be utilized during rehabilitation exercises or physical therapy sessions to help patients with movement guidance and tracking progress.

[0069] According to an embodiment, the projector 210 is configured to display realtime medical imaging data or visual instructions onto the patient's body during surgeries or other procedures to assist medical professionals in accurate localization and precise treatment.

[0070] Referring to FIG. 3, according to an embodiment, the end effector 100 is configured to project educational or entertainment content or a graphical user interface 150 onto surfaces like a wall 154, tables, or floors, enhancing the patient's experience during downtime or therapy sessions. For ease of illustration the robot connected to the end effector 100 is not shown in FIG. 3.

[0071] According to an embodiment, the ToF sensor 204 is employed to locate the user / patient relative to the room and the robot for better sound calibration EQ and the relative volume of the speakers.

[0072] According to various embodiments, the end effector 100 is configured for a haptic feedback and control system for a plurality of applications. Using the ToF sensor 204, the robot can provide touch feedback for GUI (Graphical User Interface) interaction. The ToF sensor 204 measures the distance between the end effector 200 and a surface, allowing for touch detection without physical contact enabling versatile technical applications as described below.

[0073] According to an embodiment, the ToF sensor 204 is configured to capture hand movements and gestures 152, enabling intuitive interaction with GUI elements projected by the projector 210. For example, swiping gestures or pinch-to-zoom actions by the user can be detected and interpreted.

[0074] With its ability to quickly (i.e. , in real time) and accurately measure the time it takes for emitted light to travel and return, the ToF sensor 204 provides depth information of the surrounding environment. This data can be utilized for gesture recognition, enabling the robot to interpret and respond to user gestures in real-time.

[0075] According to an embodiment, using advanced computer vision algorithms, the ToF sensor 204 is configured to capture the depth and movement of the user's hand or body, allowing for precise gesture recognition. This enables intuitive interaction between the user and the robot, eliminating the need for traditional input devices (e.g., keyboard, mouse) or physical contact (e.g., pressing buttons). Users can perform gestures such as waving, pointing, or making specific hand shapes, which are then recognized and translated into meaningful commands for the robot (sign language). The ToF sensor 204 can perceive subtle changes in hand or body positions, allowing for precise and responsive control of the robot. Additionally, since the ToF sensor operates in real-time, it enables immediate feedback and interaction between the user and the robot.

[0076] According to an embodiment, when the ToF sensor 204 detects a touch gesture, the robot can generate haptic feedback through an auxiliary haptic device tosimulate tactile sensations. This feedback enhances the user's perception and improves the overall user experience.

[0077] According to an embodiment, utilizing computer vision techniques such as OpenCV and OpenPose, the vision hub 200 is configured to estimate the pose and movements of the human operator. This allows the system to accurately map the operator's gestures and actions onto the robot, enabling intuitive and coordinated control.

[0078] According to an embodiment, both 2D and 3D semantic segmentation models are employed to identify and understand different anatomical regions, enabling precise robot movements and interactions. Segmentation models to analyze and understand the environment. By identifying different anatomical regions and objects, the system can guide the robot's movements and interactions more precisely, ensuring safe and accurate procedures. Human post estimation and semantic segmentation could both be used to assist in the higher frequency motion control.

[0079] In one embodiment, when a user brings their hand (or another body part) in the line of the sight of the projector 210, which would otherwise obstruct the projection and create a shadow, the ToF sensor 204 “sees” the hand and causes the projector 210 to mask the hand so the places where the hand is there is no shadow behind the hand. According to an embodiment, the images and video data captured by the vision hub 200 is transmitted and anonymized on a server to ensure patient / user privacy. Patient data is encrypted and securely transmitted to the servers for storage. To ensure patient privacy, identifiable images of patients are anonymized before being stored on the server. This involves removing any personal identifying information from the images, such as names or medical record numbers. The system employs robust encryption techniques to protect patient data during transmission to the servers, ensuring its confidentiality and integrity.

[0080] According to an embodiment, anonymization is also employed to address patient anonymity and privacy concerns during video transmission. This technology replaces all visual identifiers present in the video with an anonymized representation, typically an avatar or computer-generated image overlay. By overlaying the avatar onto the video, all identifiable features of the patient, including their face, tattoos, and otherdistinguishing characteristics, are effectively masked, while still capturing their eyetracking, gauze, and other features related to the patient’s response.

[0081] The video data undergoes a process where the patient's visual identifiers are detected and mapped. These identifiers are then replaced in real-time with the avatar overlay, ensuring that no identifiable information is visible in the transmitted video stream. This replacement process occurs before the video data is encrypted and securely transmitted to the recipient. Importantly, the system is designed to prioritize patient privacy. It ensures that no identifiable information from the original video is stored or retained within the system. Only the avatar image overlay data, which contains no patientspecific information, is stored for subsequent viewing or reference. On the recipient's end, such as the sonographer or operator, the secured video can be accessed without the avatar overlay. This allows the medical professional to have an unobstructed view of the patient's anatomical structures and perform the necessary assessments or procedures. However, it is crucial to note that the system does not store any data related to the patient's visual identifiers, thereby safeguarding patient privacy. By employing anonymization in this manner, patient anonymity and privacy are upheld throughout the video transmission process. The use of avatars to replace visual identifiers, coupled with encryption and strict data storage policies, contributes to maintaining the confidentiality and security of patient information, thereby fostering trust in the system and protecting patient rights.

[0082] When integrated with haptic feedback devices, the ToF sensor 204 provides real-time depth information that can be used to simulate the interaction between the user and virtual objects. As the user's hand or other body parts approach or make contact with virtual objects, the ToF sensor 204 captures the precise distance and position data, allowing an auxiliary haptic system to generate corresponding tactile feedback. The ToF sensor's fast response time and high accuracy contribute to realistic haptic feedback. It can detect even subtle changes in proximity and movement, ensuring precise synchronization between the user's actions and the haptic response. This level of detail enhances the user's perception and interaction with virtual objects, making the haptic feedback more intuitive and engaging.

[0083] Referring to FIG. 4A-4B, shown therein is a haptic device 300, according to an embodiment. The haptic device 300 functions as an input device allowing users to interact with the robot and its environment through touch and force feedback via the haptic control device 300. According to some embodiments the haptic device 300 may function as a tool attached to a robot end effector as described below.According to various embodiments, the haptic device 300 offers either 3 or 6 degrees of freedom (DOF), allowing for versatile and dexterous manipulation. The DOFs enable users to perceive and control movement in multiple directions, providing a natural and intuitive control interface and enhancing the realism and versatility of the haptic feedback.

[0084] According to the embodiment shown in FIGS. 4A-4D, the haptic device 300 utilizes a delta configuration, which consists of a parallel kinematic structure with at least three interconnected arms 301 . Each arm 301 includes an upper arm segment 302 and a lower arm segment 304 connected at a lower passive joint 308. The lower passive joint 308 provides up to 3 degrees of freedom. Each upper arm segment 302 is connected to a tool attachment point 310 by a passive upper joint 307. The passive upper joint 307 also provides up to 3 degrees of freedom. This configuration provides high rigidity, stability, and precision in controlling the haptic feedback, ensuring accurate and responsive force feedback during user interaction. The robot arm configuration follows the delta configuration, which offers advantages such as high precision, fast movements, and a compact design. This configuration enhances the robot's agility and maneuverability, enabling it to perform intricate tasks with precision.

[0085] Each arm 301 is connected to at least one servo 314 and preferably two servos 314, 316. Each servo 314, 316 adds one degree of freedom. The servos 314, 316 are mounted in a “shell” housing attached to the body 315. The servos 314, 316 are configured to “passively” measure the angle and movement of the arm 301 it is connected to as the user manipulates the haptic device 300. The servos 314, 316 are further configured to “actively” apply torque to the arm 301 to provide the user with haptic feedback. For example, if the haptic device 300 is remotely controlling a robotic arm, the haptic device provides haptic feedback to the user reflecting the resistance / load experienced by the robotic arm.

[0086] The arms 301 and the servo housing shells may be constructed of any suitable materials not limited to carbon fiber, metals, or polymer-based material.

[0087] The haptic device includes a body 315 housing a PCB and a controller for the haptic device 300. The body 315 further contains valves and fluid conduits for connecting to fluid lines passing through the arms 301 to a tool mounting point 310. The tool mounting point 310 is configured to removable attach a haptic tool to the haptic device 300.

[0088] According to an embodiment, the haptic device 300 features a detachable haptic tool (not shown), which can be easily removed and replaced. For example, the haptic tool may be a grip for grasping by a user. The haptic tool uses the same mounting interface as the end effector tools and the table station 350, ensuring compatibility and interchangeability. This allows users to switch between different tools for specific tasks while maintaining consistent haptic feedback capabilities.

[0089] According to an embodiment, the haptic tool can be easily detached from the robot and used independently, utilizing the same interface as the robot's end-effector at the table station. This allows for seamless integration and quick tool changes, facilitating efficient workflow and adaptability to different procedural needs.

[0090] Referring to FIG. 40, the haptic device 300 includes a mounting interface 306 configured to physically and electrically connect the haptic device 300 to a table station 350. When mounted to the table station 350, the haptic device 300 functions as an input device for teleoperated device (e.g., a computer or robot). The table station 350 includes I / O ports 352 (e.g., USB) to connect to the computer the robot to grant the user control over the movements of the robot. Included in the table station 350 is a 6 axis load cell similar to that described within the end effector 100. This load cell can be used to aid in gauging forces applied and received by the haptic system 300.

[0091] Referring to FIGS. 4B and 4D, the mounting interface 306 is also compatible with the end effector mounting interface (e.g., mounting interface 106) and is used to physically and electrically connect the haptic device 300 to the end effector 100 of a robot 400 (FIG. 4D). In this configuration, the haptic device 300 functions as a tool for the robot 400. This standardization ensures compatibility and interchangeability of tools, allowingfor flexibility and adaptability in various procedures. The mounting interface 306 includes locking lugs 322 (coarse threads) for engaging locking lugs on the end effector (e.g., locking lugs 122) and pogo pins 320 for transmitting for data and power between the haptic device 300 and the robot 400 via the end effector 100. The mounting interface 306 further includes fluid conduits 312 which mate with the fluid connection points (e.g., fluid connection points 113) on the end effector.

[0092] According to an embodiment, the haptic device 300 incorporates a pneumatic braking system to control and dampen the movement of the haptic device 300. This system uses compressed air to regulate the speed and force exerted by the device, allowing for smooth and controlled haptic interactions.

[0093] According to an embodiment, the haptic device 300 may further incorporate magnetic or piezoelectric braking systems. These systems use magnetic fields or piezoelectric materials, respectively, to provide additional braking forces and fine-tuned control over the haptic feedback, enabling precise and realistic touch sensations.

[0094] According to another embodiment (not shown), the haptic device utilizes a hybrid parallel configuration which consists of 3 sets of 2 interconnected arms, shells for servo motor mounting, and 2 servo motors connected serially closing the loop with an end effector. End effector consists of an electronic circuit board and an enclosing shell with mounting features for different attachments for operator use or for use by the robot.

[0095] According to an embodiment, there is a haptic control system that integrates vision systems, capacitive touch sensors, and load cells to detect and prevent collisions. Vision systems (e.g., the vision hub 200) provide real-time feedback about the robot's surroundings, while capacitive touch sensors and load cells measure contact forces and detect obstacles. If a collision is imminent, the system initiates collision avoidance measures. It limits the motions of the haptic control, taking into account inputs from the capacitive touch sensors, load cells, and vision systems.

[0096] According to an embodiment, the haptic control system also warns the operator of the limit reached, ensuring they are aware of potential collisions and can take corrective actions. The maximum speed and force applied by the robot are determined based on inputs from the vision system and the semantic segmentation model or other Almodels (Computer Vision, Sensor Fusion, 3D Mapping). These inputs help define the maximum stress the region of the body can withstand, as well as the contact area of the probe or tool. By considering these factors, the system ensures safe and appropriate force application during procedures. The region of the body to be operated on is determined by the segmentation model. This model analyzes the captured images and identifies the specific anatomical regions requiring attention. This information guides the robot's movements and enables precise targeting of the affected areas, improving procedural accuracy and patient safety. The combination of vision data, Al processing to segment and delineate regions, and maximum force and speed that can be applied using a haptic device can be applied to other interactions with different surfaces and / or components of a surface or object to ensure no excessive force is applied.

[0097] According to an embodiment, the haptic control system incorporates semi- autonomous features, enabling the robot to execute certain tasks independently under the supervision of the operator. The operator can define high-level commands and objectives, while the robot autonomously plans and executes lower-level actions to accomplish the task. This improves efficiency and reduces the operator's cognitive load. As an example, to direct a patient through the steps of a sonography, going to the location of interest, or to the site of sonography (kidney, liver), robot would autonomously go there.

[0098] Remote haptic is a guided motion, the robot is going to fill the gaps from each command. Intelligently fill the gaps between the commands coming from the haptic system and the actual commands coming from the robot, robot has to smoothly and predict between the robots coming from the command, maintain the path and pressure, while waiting for the commands from the haptic system, (10 rigid motions vs. 1 smooth motion to perform the action). This would feed into point 8 on human post estimation.

[0099] According to an embodiment, the ToF sensor 204 can be used to provide collision detection and avoidance in haptic control applications. By continuously monitoring the proximity of the user's hand or body parts to physical objects or boundaries, the ToF sensor 204 allows the haptic system to adjust the feedback accordingly. It can provide warnings or dynamically adapt the force or vibration intensity to prevent collisions or ensure user safety.

[0100] According to an embodiment, the ToF sensor 204, in combination with other pressure-sensing technologies, can differentiate between light touches and firm compressions of the end effector tool, allowing for more nuanced interactions with a projected GUI. Pressure sensing is a fundamental aspect of haptic feedback systems, and when combined with other pressure sensing technologies, it enables more nuanced and realistic tactile experiences within the robot system.

[0101] Spring force constants for biomechanical models can be felt at the haptic system and modelled through the system to simulate the feeling of different types of tissues types that the robot has detected with the pressure sensing technologies.

[0102] By integrating multiple pressure sensing technologies, such as resistive, capacitive, and piezoresistive sensors, the haptic system can capture and interpret a wide range of pressure signals to provide rich and detailed haptic feedback. Resistive pressure sensors, for example, measure pressure by detecting changes in electrical resistance. When a user applies pressure to a haptic device or object, the resistive sensor registers the compression and translates it into an electrical signal. This information is then processed by the haptic system to generate corresponding tactile sensations, allowing users to perceive different levels of pressure feedback.

[0103] Capacitive pressure sensors, on the other hand, rely on changes in capacitance to measure pressure. They consist of electrodes that detect the variations in electrical charge as pressure is applied. These sensors can provide precise pressure measurements, enabling the haptic system to deliver fine-grained feedback and capture subtle variations in touch or force. Piezoresistive pressure sensors utilize the piezoresistive effect, where the resistance of a material changes under applied pressure, to measure force and pressure. These sensors are highly sensitive and can accurately capture even small changes in pressure. By incorporating piezoresistive sensors into the haptic system, users can experience more realistic tactile sensations that mimic the texture, compliance, and responsiveness of virtual objects. The combination of these pressure sensing technologies allows the haptic system to capture a comprehensive range of pressure inputs, from light touches to firm presses, and translate them into appropriate haptic feedback. This integration enhances the realism and precision of thetactile experience, enabling users to perceive different levels of pressure and interact with virtual objects in a more natural and intuitive manner.

[0104] According to an embodiment, the end effector is configured for bounding volume speed control to ensure safe and controlled robotic motion within a defined workspace. The bounding volume speed control is used for both coarse and fine movements by regulating the robot's speed based on its position within a bounding volume. To implement bounding volume speed control, position control or speed control techniques are utilized to specify the desired position or by adjusting the speed at which the robot moves.

[0105] The bounding volume is defined as a region in space within which the robot operates. Within this volume, two or more nested cubes are set up. The inner cube serves as a fine motion control zone, where the robot's movements are controlled by a haptic device. This allows for precise and delicate manipulation of the robot's actions within this confined space. As the robot moves between the boundaries of the nested cubes, it transitions into a faster motion mode. The central location of the cubes shifts and resets accordingly to accommodate the robot's movement. This allows for smoother and more rapid traversal within the larger bounding volume, optimizing efficiency while maintaining safety.

[0106] To ensure the robot's awareness of its surroundings and to prevent collisions, the system leverages internal sensors, such as vision systems and capacitive sensors. These sensors enable the robot to perform look-ahead and prediction models, allowing it to anticipate obstacles or potential hazards in its path. By integrating these sensor inputs with the robot's guided movement capabilities, the system enables semi- autonomous behavior. The robot can dynamically adjust its motion to avoid obstacles or hazards, enhancing safety and reducing the need for human intervention. The vision system provides visual feedback to the robot, enabling it to perceive and analyze the environment in real-time. Capacitive sensors detect the proximity of objects or obstacles, allowing the robot to maintain a safe distance and avoid collisions.

[0107] By utilizing these sensors in conjunction with the bounding volume speed control, the system ensures that the robot operates within its designated workspace whileadapting its motion to the specific task requirements and environmental conditions. In summary, the Bounding Volume Speed Control feature enables safe and controlled robotic motion within a defined workspace. By setting speed limits based on the robot's position within the bounding volume, the system regulates the robot's movements for both coarse and fine tasks. The integration of vision and capacitive sensors enhances the robot's awareness of its surroundings, allowing for guided movement and obstacle avoidance. This combination of technologies promotes efficiency, safety, and adaptability in robotic operations.

[0108] While the above description provides examples of one or more apparatus, methods, or systems, it will be appreciated that other apparatus, methods, or systems may be within the scope of the claims as interpreted by one of skill in the art.

Claims

AMENDED CLAIMS received by the International Bureau on 21 October 2024 (21.10.2024)Claims1 . An end effector comprising: a robot end configured to physically and electrically connect the end effector to a robot; a mounting interface configured to removably attach a tool and electrically connect the tool to the end effector; a plurality of sensors for sensing an environment; and a projector for projecting video and / or images.

2. The end effector of claim 1 , further comprising a multi-axis load cell configured to measure load experienced by the end effector.

3. The end effector of claim 1 , wherein the robot end comprises: servo connection points for physically connecting the end effector to a robot; and a port for electrically connecting the end effector to the robot.

4. The end effector of claim 1 , wherein the mounting interface comprises: a latching mechanism for removably attaching a tool to the end effector.

5. The end effector of claim 4, wherein the latching mechanism comprises: a pair of lugs for engaging grooves on the tool to prevent relative axial motion of the tool when attached; andAMENDED SHEET (ARTICLE 19)a latch biased in a locked position by a constant force spring connected to a shape memory alloy (SMA) cable.

6. The end effector of claim 5, wherein the SMA cable is configured to shorten in length to overcome the constant force spring bias when a current passes through the SMA cable.

7. The end effector of claim 5, wherein the latching mechanism further comprises a manual latch release configured to overcome the constant force spring bias when the manual latch release is depressed.

8. The end effector of claim 1 , further comprising: fluid conduits passing from the robot end through to the mounting interface; and face seals on the mounting interface for covering ends of the fluid conduits.

9. The end effector of claim 1 , further comprising a visor covering the plurality of sensors and the projector.

10. The end effector of claim 9, further comprising a solid-state fan configured to blow air over the visor.11 . The end effector of claim 1 , wherein the plurality of sensors comprise: two stereoscopic RGB cameras having a global shutter positioned to create a parallax effect for enhanced depth perception; a time-of-flight (ToF) sensor disposed between the RGB cameras;LED diodes for the ToF sensor; and a thermopile or a thermal camera.AMENDED SHEET (ARTICLE 19)12. The end effector of claim 11 , wherein the RGB cameras are spaced apart at a distance corresponding to the space between human eyes.

13. The end effector of claim 11 , wherein the RGB cameras are configured for virtual reality teleportation.

14. The end effector of claim 1 , wherein the projector comprises a dynamic focusing system configured to maintain in-focus projection irrespective of a distance to an object or a surface onto which the projector is projecting.

15. The end effector of claim 14, wherein the dynamic focusing system comprises: a focus knob; and an SMA cable secured to the focus knob at about midway along a length of the SMA cable, wherein the focus knob rotates in a first direction when the SMA cable shortens and rotates in a second direction, when the SMA cable lengthens.

16. The end effector of claim 15, wherein the dynamic focusing system comprises a controller configured to: receive input from the plurality of sensors to determine the distance to the object or the surface onto which the projector is projecting; and vary the current through the SMA cable according to the distance.

17. A robot comprising: an end effector comprising:AMENDED SHEET (ARTICLE 19)a mounting interface configured to removably attach a tool and electrically connect the tool to the end effector; a plurality of sensors for sensing an environment; and a projector for projecting video and / or images, the projector comprising a dynamic focusing system configured to maintain in-focus projection irrespective of a distance to an object or a surface onto which the projector is projecting.

18. The robot of claim 17, wherein the dynamic focusing system comprises: a focus knob; and an SMA cable secured to the focus knob at about midway along a length of the SMA cable, wherein the focus knob rotates in a first direction when the SMA cable shortens and rotates in a second direction when the SMA cable lengthens.

19. The robot of claim 18, further comprising a controller configured to: receive input from the plurality of sensors to determine the distance to the object or the surface onto which the projector is projecting; and vary the current through the SMA cable according to the distance.

20. The robot of claim 17, further comprising a controller configured to: receive input from the plurality of sensors to determine the location of objects within a predefined proximity range; and cause the projector to project a warning indicator.AMENDED SHEET (ARTICLE 19)21 . The robot of claim 17, further comprising: a manipulator tool attached to the mounting interface; and a controller configured to: receive input from the plurality of sensors to determine the position of an object; orient the projector to project a visual indicator pattern onto the object.

22. The robot of claim 17, further comprising a controller configured to: cause the projector to project a graphical user interface (GUI) onto the object or the surface; receive input from the plurality of sensors to sense user interaction with the GUI.

23. A haptic device comprising: a parallel kinematic structure having three or more interconnected arms; at least 1 brushless servo motor connected to each arm; a braking system operably connected to each servo motor; a tool attachment point configured to removably attach a tool and electrically connect the tool to the haptic device; and a mounting interface configured to physically and electrically connect the haptic device to a second device.

24. The haptic device of claim 23, wherein the braking system is one of:AMENDED SHEET (ARTICLE 19)a pneumatic braking system; a magnetic braking system; and a piezoelectric braking system.

25. The haptic device of claim 23, wherein the mounting interface comprises: pogo pins for forming an electrical connection to the second device.

26. The haptic device of claim 23, wherein the second device is one of: a table station and an end effector of a robot.AMENDED SHEET (ARTICLE 19)Statement under Article 19(1)Claim 4 as filed contained a typographical error as noted in the international search report. The claim has been amended from "The end effector of claim 1 , wherein the mounting interface comprises: c." to "The end effector of claim 1 , wherein the mounting interface comprises: a latching mechanism for removably attaching a tool to the end effector". No new matter has been added as a result of this amendment.

Images