Segmenting multi-fingered soft grip portions based on optical flow

Abstract

Segmentation of multi-fingered soft grip portions based on optical flow is disclosed. A robotic system includes: a robot having an arm with a multi-finger soft grip formed from a compliant material; an image sensor fixedly mounted on the arm to move together with the multi-fingered soft grip and operable to capture image data of the multi-fingered soft grip composed of a compliant material; and processor circuitry operable to generate a model of a state of the multi-fingered soft grip by: controlling the arm to move in a predefined motion pattern while the image sensor captures image data; deriving an optical stream based on the image data and continuous arm kinematics and velocity frame states; and segmenting between the multi-fingered soft grip portion and a background portion of the image data based on the optical flow wherein a static region of the optical flow represents the multi-fingered soft grip portion and a dynamic region of the optical flow represents the background portion, and the incoherent region of the optical flow represents a contour between the multi-fingered soft grip portion and the background portion of the image data.

Description

Technical Field

[0001] The present disclosure relates to robotic systems, and in particular to robotic systems that segment between a multi-finger soft gripper portion and a background portion of image data based on optical flow. Background Art

[0002] Soft robots include deformable materials such as polymers, fluids, and gels that enable them to change shape and perform actions. One of the properties of soft robots is their compliance matching property. Compliance matching refers to the ability of materials in contact with each other to have similar mechanical stiffness. This property allows for uniform distribution of internal loads and minimizes stress concentrations at the interfaces between different robot parts.

[0003] In the field of industrial automation, the use of soft grippers presents practical challenges in creating cost-effective and precise automation products across sectors and in enterprises of different sizes. One of the main challenges is related to proprioceptive sensing. Soft grippers lack built-in joint state encoders due to difficulties arising from the elastic and compressive properties of the materials involved, which can change over time and during different tasks. In addition, soft robotic fingers exhibit hysteresis and non-stationary behavior due to previous interactions, chamber stresses, and object contact. Therefore, achieving reliable proprioception in soft robots remains an unsolved problem.

[0004] In industrial automation, modular soft grippers are used to handle objects with different shapes and compositions, such as in the food industry or for rigid objects. However, due to the lack of finger encoders, it is impractical to achieve precise control in these modular grippers. Therefore, grasping tasks are limited to force grasping (binary open / closed states) or centimeter-level pinching. Existing methods usually require the use of image sensors within the fingers or on a separate workbench, which increases the cost and fragility of soft grippers for precision tasks and leads to calibration and occlusion issues. BRIEF DESCRIPTION OF THE DRAWINGS

[0005] Figure 1 A schematic diagram of a soft robotic system according to aspects of the present disclosure is shown.

[0006] Figure 2 A flow chart illustrating activity stream self-segmentation and grasping according to aspects of the present disclosure is shown.

[0007] Figure 3 Figure illustrates smoothed velocity traces for activity sensing in dense optical flow in accordance with aspects of the present disclosure.

[0008] Figure 4 ( Figure 4A-4I) illustrates a method of activity sensing for foreground and background segmentation according to aspects of the present disclosure.

[0009] Figure 5 The diagram illustrates an overall motion picture according to aspects of the present disclosure.

[0010] Figure 6 The diagram illustrates finger segmentation and contour analysis for active finger modeling according to aspects of the present disclosure. DETAILED DESCRIPTION

[0011] The present disclosure relates to a soft robot that uses a single camera with a wide-angle lens mounted on the palm or wrist of the robot to obtain the continuous state of the fingers in its multi-fingered soft gripper in real time. Various aspects of the disclosure provide millimeter-precise control of grasping and manipulating objects via visual servoing without the limitations of previous solutions.

[0012] I. Overview

[0013] Figure 1 A schematic diagram of a soft robotic system 100 is shown in accordance with aspects of the present disclosure.

[0014] The soft robotic system 100 includes a processor circuitry 110 having a non-transitory storage medium, a robotic arm controller circuitry 120 , a soft gripper controller circuitry 130 , and an image sensor 140 , as well as a network / communication channel 150 .

[0015] The processor circuit system 110 is operable to model the state of the robot's multi-finger soft gripper. The robot arm controller circuit system 120 is operable to control the robot arm of the robot. The multi-finger soft gripper controller circuit system 130 is operable to control the multi-finger soft gripper 210 attached to the robot arm. In one or more of the implementations described herein, the processing circuit system or the controller circuit system may include a memory storing data and / or instructions. The "processor circuit system" used herein may also generally refer to the processor circuit system 110, the robot arm controller circuit system 120 and / or the multi-finger soft gripper controller circuit system 130. The processor circuit system is operable to control an arm with a multi-finger soft gripper to manipulate an object based on a state model of the multi-finger soft gripper.

[0016] As used herein, the terms "processor", "processor circuitry", "processor circuit", "controller" or "controller circuitry" may be understood as any technical entity that allows the handling of data. Data may be handled according to one or more specific functions performed by the processor, processor circuitry or controller. Further, as used herein, the processor circuitry, processor or controller may be understood as any circuit, such as any analog or digital circuit. The processor circuitry, processor or controller may therefore be or include an analog circuit, a digital circuit, a mixed signal circuit, a logic circuit, a processor, a microprocessor, a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), a field programmable gate array (FPGA), an integrated circuit, an application specific integrated circuit (ASIC), etc., or any combination thereof. Any other kind of implementation of the corresponding functions, which will be described in further detail below, may also be understood as a processor circuitry, a processor, a controller or a logic circuit. It should be understood that any two (or more) of the processors, controllers, logic circuits, or processing circuit systems detailed herein may be implemented as a single entity having equivalent or similar functionality, etc., and conversely, any single processor, controller, logic circuit, or processor circuit system detailed herein may be implemented as two (or more) separate entities having equivalent or similar functionality, etc.

[0017] The image sensor 140 is fixedly mounted on the robot arm to move with the multi-finger soft gripper 210 made of compliant material, and is operable to capture image data of the multi-finger soft gripper 210. The image sensor 140 may include any number of image acquisition devices and components. The image sensor 140 may include one or more image capture devices (e.g., a camera, a CCD (charge coupling device) or any other type of image sensor). The camera may be a red, green, blue (RGB) / grayscale camera. The lens 222 of the camera 220 has a wide field of view 224. For example, the field of view may be 160-220 degrees. The same image sensor 140 can be used to simultaneously determine finger position and identify an object.

[0018] "Storage medium" or "memory" is a computer-readable medium in which data or information can be stored for retrieval. Thus, references to "storage medium" or "memory" included herein can be understood to refer to volatile or non-volatile memory, including random access memory (RAM), read-only memory (ROM), flash memory, solid-state storage device, magnetic tape, hard disk drive, optical drive, etc., or any combination thereof. In this article, registers, shift registers, processor registers, and data buffers, etc. are also included by the term memory. The memory can be any known volatile and / or non-volatile memory, including read-only memory (ROM), random access memory (RAM), flash memory, magnetic storage medium, optical disk, erasable programmable read-only memory (EPROM), and programmable read-only memory (PROM). The memory can be non-removable, removable, or a combination of the two.

[0019] The network / communication channel 150 can be any wireless and / or wired communication channel. For example, the wireless communication channel can be configured according to a short-range mobile radio communication standard (such as Bluetooth, Zigbee, etc.). As another example, the wireless communication channel can be configured according to a medium-range or wide-range mobile radio communication standard (such as a 3G, 4G or 5G mobile radio communication standard). As a further example, the wireless communication channel can be configured according to a wireless local area network communication protocol or standard (such as IEEE 802.11). The wired communication channel can be configured according to standards such as Ethernet (IEEE 802.3), USB (Universal Serial Bus), HDMI (High-Definition Multimedia Interface), Thunderbolt, RS-232, RS-485, etc.

[0020] II. Contactless proprioceptive instrumentation: imaging and motion control

[0021] Figure 2 (include Figure 2A and Figure 2B ) illustrates a flowchart 200 of activity flow self-segmentation and grasping according to aspects of the present disclosure.

[0022] As an overview, flowchart 200 illustrates an instrumentation of active optical flow 230 for self-segmentation and grasping to output monitoring / debugging output visualization 250. A single camera 220 mounted near the multi-finger soft gripper 210 fully captures the soft finger from the mounting base to the tip. When high-precision picking is required, the placement of the camera principal point is directed toward the tip of the finger in a pinch configuration. This minimizes geometric compression distortion near fine-manipulation image regions. This is relevant to visual servoing tasks, where approach and closing actions are driven via the visual-Jacobian. The wide-angle lens 222 does not need to be calibrated (in terms of optical intrinsic parameters). The method requires as input: (1) the original image I c (t); and (2) wrist kinematic frame state W(t) and its velocity Using these two inputs, the method automatically performs the motion Θ(t0, t1) (velocity trajectory 232), estimates and publishes the state F(t) 240 of the fingers for any motion, planning and control to exploit the information for grasping and beyond grasping.

[0023] First, the RGB camera 220 captures a wide field of view 224 including the multi-finger soft gripper 210. This is a time-varying signal Its color information is not used in the active optical stream 230. By converting this color image data into brightness image data or brightness value image data 226, color dependencies are eliminated. Therefore, any color of the multi-finger soft gripper 210 or the background will work similarly following the continuous state of the position, orientation, and velocity (linear and angular) of the multi-finger soft gripper 210 relative to the optical flow 230. This helps limit the calculation of the search area along the direction of camera motion. This is called the wrist frame 212 and is represented as and the associated speed That is, twist. The time synchronization t is ensured via the known ROS-Crony mechanism and the ROS-TF7D interpolation function. Therefore, the formal system input α is expressed as:

[0024] The multi-finger soft gripper 210 can be designed without fingers or fingertips, such as a spherical shape. It is assumed that the camera 220 and the robot arm base are fixed, but the fingers themselves are not fixed because they are flexible. The camera 220 can be installed in different positions (e.g., on one finger or in the center of all fingers) depending on the area of ​​the segmented access and contact points.

[0025] III. Active image acquisition: for image wheel Profile tension and Coordinated motion for segmentation stability detection

[0026] Figure 3Smoothed velocity traces 300 of activity sensing in dense optical flow are illustrated in accordance with aspects of the present disclosure.

[0027] Activity sensing is a process in which a robot can better sense and model its state by purposefully moving the image sensor 120 / 220 in a predefined motion pattern while the image sensor 140 / 220 captures image data to obtain a better signal, thereby simplifying the perception problem or receiving more information. In this paper, the robot exploits the fact that the camera 120 / 220 and the multi-finger soft gripper 210 mounted on the robot arm or wrist will move in a similar manner. Therefore, the difference between the foreground and the background is exposed in the image through the controlled motion. Formally, this means that there is a visual mapping Process Γ, which generates certain image data as input to the wrist frame 212. Since the robot arm can be controlled in velocity mode, the velocity Can be set as a closed loop trajectory: (in The composition of the frame and the twist) includes the translation and rotation of the image plane, such as Figure 3 W(t0) is the instantaneous position at the start of the motion. Such motion is intended to provide enough unique changes in the image data to detect reliable outer boundaries of the soft finger with respect to the background. The shape of the velocity trajectory Θ(t0, t1)232 ranges from a simple circular outline to multiple loops, depending on the shape, fingers, and complexity of the multi-fingered soft gripper. In practice, assuming the diversity of image content in the background, the figure-of-eight motion pattern (see Figure 3 ) will work for most applications. In applications where the background lacks texture, patches with texture can be reprojected to the image plane and used in a scanning fashion to ensure visual gradients around boundaries. This mitigation strategy results in simple detectability. Specific trajectories can be planned or randomly chosen. Invariantly, the acquisition ensures sufficient variation to detect segmentation and contours as described in the following sections. Finally, acceleration is kept to the minimum possible to avoid blurring and shaky motion of the finger.

[0028] IV. Motion-biased quasi-dense optical flow

[0029] Figure 4 (include Figure 4 A-4I) illustrates a method 400 of activity sensing with foreground and background segmentation according to aspects of the present disclosure.

[0030] Figure 4 Figure A shows the optical flow resulting from a rotation about the optical axis of the camera. Figure 4 Figure B shows the use of Figure 4 A stream of image data in mode A.

[0031] Figure 4 Panel C shows the optical flow resulting from translational motion along the optical axis of the camera. Figure 4 D shows the use of Figure 4 A stream of image data in the mode C.

[0032] Figure 4 Figure E shows that the static region with respect to the optical flow is the segmentation of the multi-fingered soft gripper (fingers and robot arm). The zoomed region (indicated by circles) shows that the dense flow pattern is close to the contour area and the inside of the fingers is small. Figure 4 Figure F shows a contour or boundary region where the optical flow reveals abrupt edges in flow and amplitude. Such flow cues reliably separate moving areas from non-moving areas.

[0033] Figure 4 G shows how areas that expose minimum optical flow and accumulation (sum of vector magnitudes) are considered scene content and are pixel-wise labeled as background (shown in darker colors). Figure 4 H area in the motion accumulator χ T ([x, y]) have insufficient variation and are considered foreground or soft gripper pixels (shown in darker colors). Figure 4 I shows Figure 4 Combination of G and 4H. These two segmentation classes expose non-convex regions with excellent details. To remove over-segmented speckles, simple morphological erosion and dilation (aperture filter) cleans the process speckles, thereby providing reliable segmentation in a computationally efficient manner.

[0034] Optical flow is the apparent motion pattern of an object between two or more consecutive images. This is caused by the motion of the object, the camera 140 / 220, or both. Considering an intensity image, where I l (x, y, t) is the pixel coordinate at time t, then after the object in the image moves by a small interval (|[δx, δy]| < ε) without a sharp and immediate change in the vantage point, we can consider: I l (x, y, t)≈I l (x+δx, y+δy, t+δt). (Equation 4) ε is the pixel distance, proportional to the expected velocity and inversely proportional to the background depth. Smaller values ​​make the motion more linear, producing better results. However, small values ​​require more computational power, which is especially important in dense flows of high-resolution images.

[0035] The first-order Taylor approximation on the right side is expressed as:

[0036] Then, the optical flow formula can now be obtained by dividing this expression by δt, namely: I,s, where and The unit is pixels per second. The expression [u, v] is the linear motion velocity vector, is the (quasi-static) horizontal image gradient around the pixel. Similarly, is the vertical quasi-static gradient.

[0037] Known dense methods estimate the [u, v] model underlying image patch as a polynomial surface and search around the neighbors in the region to determine the best match. On the other hand, aspects of the present disclosure introduce information about the expected motion. Information, i.e. dynamic frame The impact on the camera greatly reduces calculations.

[0038] This is modeled as a range from a predefined depth range [D0, D1], typically between 10-50 cm, for grasping approach and closing. This is formally expressed as: Where [β0,β1] defines the minimum and maximum length (in pixels) of the displacement of the [D0,D1] input range. Consider how far away the objects in the image are from the image sensor 140 / 220. The change in pixels that are far away is much smaller than the change in pixels that are close. Finally, since K tells the search window the pixel range that has motion significance, it greatly reduces computational resources while increasing the robustness of the results. Similarly, the direction can also be modeled via the unit vector γ of the optical flow displacement, that is:

[0039] This biased sub-pixel linear search provides further optimization of narrow bounds in dense optical flow. and the deviation of the spatial range [D0, D1], which is technically and mathematically possible. The latest efficiency concept of half-density optical flow with prior knowledge is the fact that limited pixels with sufficiently prominent gradient magnitude will produce reliable results regardless of the color or texture of the material of the multi-finger soft gripper 210. Therefore, limited to those pixels whose local area exposes the motion gradient magnitude: Above a predefined stability threshold λ will be computed. This is to allow the optical flow equation to have a meaningful solution. It also reduces the computational load without sacrificing the accuracy of the vector field. The dense optical flow map is written as: and provide a displacement vector β0≤|[u,v]|≤β1 or If the pixel [x,y] does not have enough gradient for a stable flow search calculation.

[0040] V. Activity flow based segmentation and contouring

[0041] Figure 5 An overall motion picture 500 is illustrated in accordance with various aspects of the disclosure. Figure 6 Finger segmentation and contour analysis 600 for active finger modeling in accordance with various aspects of the disclosure is illustrated.

[0042] Exploiting motion cues for segmentation makes the algorithm invariant to image content. In flow-based segmentation, changes in motion (e.g. Figure 4 A and 4C) enable the collection of vectors for all regions, including those that do not intersect. Furthermore, by computing multiple images in motion, it can provide results that are invariant to the background if the intensities differ slightly. The discrete total motion of a pixel is defined as the sum of the magnitudes of the motion vectors along the trajectory (θ(t0, t1) see Figure 5 ). Therefore, the integral expression for segmenting the image into two categories is:

[0043] Binary version of Equation 11 Deformable blobs used to segment the fingers that describe the multi-finger soft gripper 210. These are free-form two-dimensional shapes. Note that the base of the fingers remains within these blobs. The idea of ​​removing the inactive elements of the multi-finger soft gripper (base and hose) 210 is straightforward in the same way, except that the motion is obtained by opening and closing the soft fingers while applying maximum aperture and maximum force in one state and minimum aperture and no force in another. This ensures a full range of deformation of the soft fingers. Of course, the rigid base of the multi-finger soft gripper 210 remains static. Figure 6 The figure shows the result of a finger blob that is completely segmented from any other element. The outer contour of each of these blobs (denoted as i (t)) is the outline of the chain encoding.

[0044] VI. Deformable multi-segment activity model

[0045] For each contour Π representing a finger at time t i (t), its convex hull can be calculated A subset of these points explicitly defines the j-ray connecting the centroid of the blob and the shell points like Figure 6 Finally, for each finger spot Π i(t), the light with the maximum change in the aperture value of the exposed holder Δ j,i These salient light features are stable in their behavior across actions and 6D pose configurations. To increase robustness, contour smoothing is applied. Figure 6 This smoothing effect is indicated on the top of the image while selecting the best ray from two nearby ones. Due to the continuous nature of the contour and highlight rays, the position is sub-pixel accurate and the resolution close to these areas is very high. Therefore, the tip contact point can be determined with sub-millimeter accuracy.

[0046] The compliant matching properties of soft robots provide them with various advantages. These include fast and robust grasping capabilities as they can adjust their shape according to the objects they interact with, enabling stable and dexterous manipulation skills. In addition, soft robots exhibit effortless passive impedance, allowing them to absorb collisions and slide smoothly during misalignment, making them resilient to program and execution errors. In addition, soft robots are versatile as they are suitable for physical interaction in wet, dirty and taut environments with adjustable softness for both objects and humans. They save a lot of cost compared to rigid robots; this is due to their efficient energy actuation, cheap parts replacement, heavy-duty capabilities and easy operation.

[0047] Various aspects of the disclosure are directed to obtaining real-time, continuous finger state information in a soft gripper using a single camera with a wide-angle lens mounted on the palm or wrist of a robot. This innovative solution enables heavy-duty and cost-effective automation by providing millimeter-precision, industrial-grade, vision-based closed-loop control to grasp and manipulate a variety of objects.

[0048] The technique employed is self-segmentation using active optical flow. By leveraging active sensing capabilities, a soft gripper mounted on a robotic arm can extract segmented regions and contours of objects independently of background appearance. The system uses these visual cues to create non-stationary, deformable gripper models and continuously measures them at frame rates.

[0049] Furthermore, millimeter-scale grasping tasks are reliably accomplished without camera calibration. There are no restrictions on the gripper’s shape, material, workspace, or robot’s motion capabilities.

[0050] The implementation on the CPU provides real-time performance by utilizing an inexpensive high definition (HD) camera operating at 20 frames per second (fps). This enables the system to demonstrate precision (e.g., grasping and inserting a coin), strength (e.g., lifting a heavy object), and delicacy (e.g., picking raspberries without causing damage).

[0051] The disclosed aspects provide scalability and generalization without (re)training the model. This is achieved through an online adaptive process that effectively avoids re-tuning the artificial neural network (ANN). Analytical derivations enable the method to have a verifiability and interpretability.

[0052] The ability to reliably measure finger states in soft grippers is fundamental to enabling vision-based flexible grasping and manipulation of objects across a variety of processes. The disclosed aspects effectively address barriers associated with cost-effective robotic automation in many applications, including those involving high-precision tasks. The resulting soft robots' advantages in productivity, minimal maintenance, and resilience (e.g., to collisions, twisting, and bending) translate directly into economic gains for companies. This is particularly important in contexts where the high cost and limited availability of human employees, particularly in industrialized countries, pose an existential threat. Such challenges can be mitigated through AI-based automation.

[0053] The technology of the present disclosure may also be described in the following examples.

[0054] Example 1. A robotic system comprising: a robot having an arm with a multi-finger soft gripper formed of a compliant material; an image sensor fixedly mounted on the arm to move with the multi-finger soft gripper and operable to capture image data of the multi-finger soft gripper; and a processor circuit system operable to generate a model of a state of the multi-finger soft gripper by: controlling the arm to move in a predefined motion pattern while the image sensor captures image data; deriving an optical flow based on the image data and successive arm kinematic and velocity frame states; and segmenting between a multi-finger soft gripper portion and a background portion of the image data based on the optical flow, wherein a static region of the optical flow represents the multi-finger soft gripper portion, a dynamic region of the optical flow represents the background portion, and an incoherent region of the optical flow represents a contour between the multi-finger soft gripper portion and the background portion of the image data.

[0055] Example 2. The robotic system of Example 1, wherein the processor circuit system is further operable to generate a model of the state of the multi-fingered soft gripper by deriving the optical flow as an active optical flow by restricting the derivation to pixels of the image data along a direction of expected motion and within a predefined depth range from the image sensor.

[0056] Example 3. The robotic system of any one or more of Examples 1-2, wherein the processor circuitry is further operable to generate a state model of the multi-fingered soft gripper by deriving active optical flow by restricting the derivation to pixels of the image data having motion gradient magnitudes above a predetermined stability threshold.

[0057] Example 4. The robotic system of any one or more of Examples 1-3, wherein the processor circuit system is further operable to generate a state model of the multi-finger soft gripper by: determining a discrete total motion for each pixel of the image data by summing the magnitudes of motion vectors along a trajectory of a predefined motion pattern; and segmenting the pixel of the image data between a portion of the multi-finger soft gripper and a background portion of the image data based on the discrete total motion.

[0058] Example 5. The robotic system of any one or more of Examples 1-4, wherein the processor circuitry is further operable to control an arm having a multi-fingered soft gripper to manipulate an object based on a state model of the multi-fingered soft gripper.

[0059] Example 6. The robotic system of any one or more of Examples 1-5, wherein the predefined motion pattern comprises a rotation around an optical axis of the image sensor or a translational motion along the optical axis of the image sensor.

[0060] Example 7. The robotic system of any one or more of Examples 1-6, wherein the predefined motion pattern comprises a figure eight motion pattern.

[0061] Example 8. The robotic system of any one or more of Examples 1-7, wherein the image sensor comprises a single camera having a lens with a field of view wide enough to simultaneously capture image data of all fingers of the multi-fingered soft gripper.

[0062] Example 9. The robotic system of any one or more of Examples 1-8, wherein the processor circuit system is further operable to generate a state model of the multi-finger soft gripper by computing a convex hull of each contour representing a corresponding finger of the multi-finger soft gripper, wherein the convex hull is a subset of points defining rays connecting the centroids of the corresponding fingers in the image data.

[0063] Example 10. The robotic system of any one or more of Examples 1-9, wherein the processor circuitry is further operable to generate a state model of the multi-fingered soft gripper by converting the image data from color image data to luminance value image data prior to deriving the optical flow.

[0064] Example 11. A component of a robotic system, wherein the robotic system includes: a robot having an arm with a multi-finger soft gripper; and an image sensor fixedly mounted on the arm to move with the multi-finger soft gripper formed of a compliant material and operable to capture image data of the multi-finger soft gripper; the component includes: a processor circuit system; a non-transitory computer-readable storage medium including instructions that, when executed by the processor circuit system, cause the processor circuit system to generate a state model of the multi-finger soft gripper by: controlling the arm to move in a predefined motion pattern while the image sensor captures image data; deriving an optical flow based on the image data and consecutive arm kinematic and velocity frame states; and segmenting between a multi-finger soft gripper portion and a background portion of the image data based on the optical flow, wherein a static region of the optical flow represents the multi-finger soft gripper portion, a dynamic region of the optical flow represents the background portion, and an incoherent region of the optical flow represents a contour between the multi-finger soft gripper portion and the background portion of the image data.

[0065] Example 12. The assembly of Example 11, wherein the processor circuitry is further operable to generate a state model of the multi-finger soft gripper by deriving the optical flow as an active optical flow by restricting the derivation to pixels of the image data along a direction of expected motion and within a predefined depth range from the image sensor.

[0066] Example 13. The assembly of any one or more of Examples 11-12, wherein the processor circuitry is further operable to generate a state model of the multi-finger soft gripper by deriving active optical flow by restricting the derivation to pixels of the image data having motion gradient magnitudes above a predetermined stability threshold.

[0067] Example 14. The component of any one or more of Examples 11-13, wherein the processor circuit system is further operable to generate a model of the state of the multi-finger soft gripper by: determining a discrete total motion for each pixel of the image data by summing the magnitudes of motion vectors along a trajectory of a predefined motion pattern; and segmenting pixels of the image data between a portion of the multi-finger soft gripper and a background portion of the image data based on the discrete total motion.

[0068] Example 15. The assembly of any one or more of Examples 11-14, wherein the processor circuitry is further operable to control an arm having a multi-fingered soft gripper to manipulate the object based on the model of the state of the multi-fingered soft gripper.

[0069] Example 16. The assembly of any one or more of Examples 11-15, wherein the predefined motion pattern comprises a rotational motion around an optical axis of the image sensor or a translational motion along the optical axis of the image sensor.

[0070] Example 17. The assembly of any one or more of Examples 11-16, wherein the predefined motion pattern comprises a figure eight motion pattern.

[0071] Example 18. The assembly of any one or more of Examples 11-17, wherein the image sensor comprises a single camera having a lens with a field of view wide enough to simultaneously capture image data of all fingers of the multi-finger soft gripper.

[0072] Example 19. The assembly of any one or more of Examples 11-18, wherein the processor circuitry is further operable to generate a model of the state of the multi-finger soft gripper by computing a convex hull of each contour representing a corresponding finger of the multi-finger soft gripper, wherein the convex hull is a subset of points defining rays connecting the centroids of the corresponding fingers in the image data.

[0073] Example 20. The assembly of any one or more of Examples 11-19, wherein the processor circuitry is further operable to generate a model of a state of the multi-finger soft gripper by converting the image data from color image data to luminance value image data prior to deriving the optical flow.

[0074] Although the foregoing has been described in conjunction with exemplary aspects, it should be understood that “exemplary” is merely meant as an example, not the best or optimal. Therefore, the present disclosure is intended to cover alternatives, modifications, and equivalents that may be included within the scope of the present disclosure.

[0075] Although various specific aspects have been illustrated and described herein, it will be appreciated by those skilled in the art that various alternative and equivalent implementations may be used to replace the specific aspects shown and described without departing from the scope of the present application. This application will cover any modifications or variations of the various aspects discussed herein.

Claims

1. A robot system, comprising: a robot having an arm with a multi-fingered soft gripper formed of a compliant material; an image sensor fixedly mounted on the arm to move with the multi-finger soft gripper and operable to capture image data of the multi-finger soft gripper; as well as A processor circuit system is operable to generate a model of the state of the multi-finger soft gripper by: controlling the arm to move in a predefined motion pattern while the image sensor captures the image data; deriving optical flow based on the image data and successive arm kinematics and velocity frame states; as well as Segmentation is performed between the multi-finger soft gripper portion and the background portion of the image data based on the optical flow, wherein a static region of the optical flow represents the multi-finger soft gripper portion, a dynamic region of the optical flow represents the background portion, and an incoherent region of the optical flow represents a contour between the multi-finger soft gripper portion and the background portion of the image data.

2. The robot system according to claim 1, wherein: The processor circuitry is further operable to generate the model of the state of the multi-finger soft gripper by: The optical flow is derived as a motion optical flow by restricting the derivation to pixels of the image data that are along a direction of expected motion and within a predefined depth range from the image sensor.

3. The robot system according to claim 2, wherein: The processor circuitry is further operable to generate the model of the state of the multi-finger soft gripper by: The motion optical flow is derived by limiting the derivation to pixels of the image data having motion gradient magnitudes above a predetermined stability threshold.

4. The robot system according to claim 1, wherein: The processor circuitry is further operable to generate the model of the state of the multi-finger soft gripper by: determining a discrete total motion for each pixel of the image data by summing the magnitudes of the motion vectors along the trajectory of the predefined motion pattern; and Pixels of the image data between the multi-finger soft gripper portion and the background portion of the image data are segmented based on the discrete total motion.

5. The robot system according to claim 1, wherein: The processor circuitry is further operable to control the arm having the multi-fingered soft gripper to manipulate an object based on the model of the state of the multi-fingered soft gripper.

6. The robot system according to claim 1, wherein: The predefined motion pattern comprises a rotational motion around an optical axis of the image sensor or a translational motion along the optical axis of the image sensor.

7. The robot system according to claim 1, wherein: The predefined motion pattern includes a figure eight motion pattern.

8. The robot system of claim 1, wherein: The image sensor includes a single camera having a lens with a field of view wide enough to simultaneously capture image data for all fingers of the multi-finger soft gripper.

9. The robot system of claim 1, wherein: The processor circuitry is further operable to generate the model of the state of the multi-finger soft gripper by: A convex hull is calculated for each contour representing a corresponding finger of the multi-finger soft gripper, wherein the convex hull is a subset of points defining rays connecting centroids of the corresponding fingers in the image data.

10. The robot system of claim 1, wherein: The processor circuitry is further operable to generate the model of the state of the multi-finger soft gripper by: Prior to deriving the optical flow, the image data is converted from color image data to luminance value image data.

11. A component of a robotic system, wherein: The robot system comprises: a robot having an arm with a multi-fingered soft gripper; and an image sensor fixedly mounted on the arm to move with the multi-fingered soft gripper formed of a compliant material and operable to capture image data of the multi-fingered soft gripper, the components comprising: processor circuitry; A non-transitory computer-readable storage medium comprising instructions that, when executed by the processor circuit system, cause the processor circuit system to generate a model of the state of the multi-finger soft gripper by: controlling the arm to move in a predefined motion pattern while the image sensor captures the image data; deriving optical flow based on the image data and successive arm kinematics and velocity frame states; and Segmentation is performed between the multi-finger soft gripper portion and the background portion of the image data based on the optical flow, wherein a static region of the optical flow represents the multi-finger soft gripper portion, a dynamic region of the optical flow represents the background portion, and an incoherent region of the optical flow represents a contour between the multi-finger soft gripper portion and the background portion of the image data.

12. The assembly of claim 11, wherein: The processor circuitry is further operable to generate the model of the state of the multi-finger soft gripper by: The optical flow is derived as a motion optical flow by restricting the derivation to pixels of the image data that are along a direction of expected motion and within a predefined depth range from the image sensor.

13. The assembly of claim 12, wherein: The processor circuitry is further operable to generate the model of the state of the multi-finger soft gripper by: The motion optical flow is derived by limiting the derivation to pixels of the image data having motion gradient magnitudes above a predetermined stability threshold.

14. The assembly of claim 11, wherein: The processor circuitry is further operable to generate the model of the state of the multi-finger soft gripper by: determining a discrete total motion for each pixel of the image data by summing the magnitudes of the motion vectors along the trajectory of the predefined motion pattern; and Pixels of the image data between the multi-finger soft gripper portion and the background portion of the image data are segmented based on the discrete total motion.

15. The assembly of claim 11, wherein: The processor circuitry is further operable to control the arm having the multi-fingered soft gripper to manipulate an object based on the model of the state of the multi-fingered soft gripper.

16. The assembly of claim 11, wherein: The predefined motion pattern comprises a rotational motion around an optical axis of the image sensor or a translational motion along the optical axis of the image sensor.

17. The assembly of claim 11, wherein: The predefined motion pattern includes a figure eight motion pattern.

18. The assembly of claim 11, wherein: The image sensor includes a single camera having a lens with a field of view wide enough to simultaneously capture image data for all fingers of the multi-finger soft gripper.

19. The assembly of claim 11, wherein: The processor circuitry is further operable to generate the model of the state of the multi-finger soft gripper by: A convex hull is calculated for each contour representing a corresponding finger of the multi-finger soft gripper, wherein the convex hull is a subset of points defining rays connecting centroids of the corresponding fingers in the image data.

20. The assembly of claim 11, wherein: The processor circuitry is further operable to generate the model of the state of the multi-finger soft gripper by: Prior to deriving the optical flow, the image data is converted from color image data to luminance value image data.

Images