Robot end effector and method for object interaction

Abstract

Embodiments relate generally a gripper unit configured to couple to an end effector to assist in robotic object interaction. An example gripper unit may include: a gripper body having proximal and distal ends, the gripper body defining a channel extending between the proximal and distal ends; a gripper base coupled to the proximal end of the gripper body and configured to couple to the end effector; and an imaging device positioned at the gripper base or the gripper body; wherein the channel includes an opening through which the imaging device sees.

Description

"Robot end effector and method for object interaction" Technical Field

[0001] Embodiments generally relate to an end effector for assisting in robotic object interaction. In particular, embodiments relate to a gripper unit of the end effector for assisting in robotic object interaction. Background

[0002] The adoption of a robotic apparatus for interaction with food items, such as fruits and eggs, or quality control and manufacturing situations may provide economic benefits, mitigate human inconsistencies in the inspection process, and reduce wear on a human worker under strenuous working conditions. For example, a robotic apparatus may increase an orchard’s production levels by integrating both inspection and harvesting processes, performing inspections immediately preceding harvesting. This approach can significantly reduce the costs associated with post-harvest inspection and sorting, while also reducing the likelihood of contaminating healthy fruit with defective or infectious ones, thereby mitigating potential losses. However, existing apparatuses and methods for interacting and inspecting fruit, or other objects, tend to be inefficient and suboptimal.

[0003] For example, existing fruit inspection methods are either conducted by humans, which is inconsistent and labour-intensive, or carried out post-harvest, requiring the fruit to be picked and transported to a controlled environment. This process wastes time and resources on picking and transporting defective fruits.

[0004] Existing solutions for monitoring robot-object interactions typically require complex and costly systems utilising gel materials and tactile sensing algorithms. These approaches focus on inferring object characteristics through the deformation of gel materials, which not only complicates the system but also obstructs direct visualinspection of the target object due to the gel material's opacity. Consequently, these solutions are limited in their ability to provide direct object inspection.

[0005] It is desired to address or ameliorate one or more shortcomings or disadvantages associated with prior apparatuses and methods for interacting with objects, or to at least provide a useful alternative thereto.

[0006] Throughout this specification the word "comprise", or variations such as "comprises" or "comprising", will be understood to imply the inclusion of a stated element, integer or step, or group of elements, integers or steps, but not the exclusion of any other element, integer or step, or group of elements, integers or steps.

[0007] Any discussion of documents, acts, materials, devices, articles or the like which has been included in the present specification is not to be taken as an admission that any or all of these matters form part of the prior art base or were common general knowledge in the field relevant to the present disclosure as it existed before the priority date of each of the appended claims. Summary

[0008] Some embodiments relate to a gripper unit configured to couple to an end effector to assist in robotic object interaction. The gripper unit may comprise: a gripper body having proximal and distal ends, the gripper body defining a channel extending between the proximal and distal ends; a gripper base coupled to the proximal end of the gripper body and configured to couple to the end effector; and an imaging device positioned at the gripper base or the gripper body; wherein the channel includes an opening through which the imaging device sees.

[0009] The gripper unit may further comprise a reflective surface extending within the channel between the proximal and distal ends of the gripper body, wherein the reflective surface is within a field of view of the imaging device. The reflective surface may be configured to deform with deformation of the gripper body. The reflectivesurface may be configured to provide the imaging device with additional imaging angles of an object, the object positioned near the distal end of the gripper unit.

[0010] The gripper unit may further comprise a plurality of tracking points disposed within the channel, wherein the plurality of tracking points are positioned within the field of view of the imaging device. Deformation of the gripper body may be determined based on the plurality of tracking points. The plurality of tracking points may comprise one or more of: a plurality of physical points within the channel or a plurality of software defined points within the channel.

[0011] The gripper unit may further comprise an actuator configured to pivot the gripper body relative to the end effector. The gripper unit may further comprise at least one light source positioned at the gripper base and configured to emit light toward the second end of the gripper body.

[0012] Some embodiments relate to an end effector for assisting in robotic object interaction. The end effector may comprise: an end effector body having proximal and distal ends, the proximal end configured to couple to a robotic system; and at least two gripper units previously described coupled to the distal end of the end effector body, the at least two gripper units spaced apart from each other.

[0013] The end effector may further comprise a second imaging device coupled to the distal end of the end effector and configured with a field of view (FOV), the at least two gripper units positioned within the FOV.

[0014] The end effector may further comprise a mechanism for adjusting a distance between the at least two gripper units.

[0015] The end effector may comprise three gripper units coupled to the distal end of the end effector body, wherein the three gripper units are spaced apart from each other. The end effector may comprise four gripper units coupled to the distal end of the end effector body, wherein the four gripper units are spaced apart from each other.

[0016] Some embodiments relate to a robotic system for robotic object interaction. The robotic system may comprise: a body housing a processor and a memory, the memory accessible to the processor; at least one robotic limb coupled to the body; and at least one end effector previously described coupled to the at least one robotic limb; wherein the processor is configured to execute instructions stored in the memory so as to cause the robotic system to interact with an object using at least one of: the robotic limb and the at least one end effector.

[0017] Some embodiments relate to a method of assisting robotic object interaction. The method may comprise: positioning a gripper unit proximal to an object, the gripper unit coupled to an end effector, wherein the gripper unit has a gripper body, and an imaging device positioned relative to the gripper body, wherein the imaging device has a view through an opening of the gripper body; and receiving a stream of imaging data of the object from the imaging device.

[0018] The gripper body may further include a reflective surface, and the receiving may comprise receiving imaging data relating to a direct view of the object; and receiving imaging data relating to an indirect view of the object via the reflective surface. The method may further comprise rotating the gripper unit about the object without gripping the object to inspect the object.

[0019] The method may further comprise determining, using a first machine learning model, based on the received stream of imaging data, a status of the object; and determining, using a second machine learning model, based on the status of the object, control instructions for at least one of the end effector and the gripper unit.

[0020] The method may further comprise manipulating the object using at least one of the end effector and the gripper unit based on the control instructions. The object may be a fruit, and the status may include one or more of: ripe, unripe, defective, not defective, slipping, obstructed, and unobstructed.

[0021] The method may further comprise determining, based on the received imaging data, deformation of the gripper unit. Determining the control instructions may be further based on the determined deformation of the gripper unit. Brief Description of Drawings

[0022] Embodiments are described in further detail below, by way of example and with reference to the accompanying drawings, in which:

[0023] Figure 1 is an illustration of an example end effector, according to some embodiments;

[0024] Figure 2 is an illustration of an example gripper unit, according to some embodiments;

[0025] Figures 3A and 3B are an illustration of an example interaction of gripper units with an object, according to some embodiments;

[0026] Figures 4A and 4B are an illustration the gripper unit of Figure 2, according to some embodiments;

[0027] Figure 5 is an illustration of the size adaptability of the end effector of Figure 1, according to some embodiments;

[0028] Figures 6A and 6B show an illustration of a size adaptability mechanism of the end effector of Figure 1, according to some embodiments;

[0029] Figures 7A and 7B show example illustrations of the field of view of an imaging device of the gripper unit of Figure 2, according to some embodiments;

[0030] Figure 8 is an illustration comparing prior end effector imaging methods and the end effector of Figure 1, according to some embodiments;

[0031] Figure 9 shows an example robotic system including the end effector of Figure 1, according to some embodiments;

[0032] Figure 10 is a flowchart illustrating a method for execution by the robotic system of Figure 9, according to some embodiments; and

[0033] Figures 11A and 11B illustrate an example of deformation determination of the gripper unit of Figure 2, according to some embodiments. Description of Embodiments

[0034] Figure 1 is an illustration of an example end effector 100 for assisting in robotic object interaction, according to some embodiments. End effector 100 comprises a main body 150 configured to couple to a robotic system (not shown). End effector 100 further comprises a mounting interface 170 for coupling the end effector 100 to the robotic system. In some embodiments, the end effector may be coupled to a robotic limb 199. The robotic system may comprise the electrical, electronic, and computing components to control the end effector 100 to assist in object interaction.

[0035] End effector 100 further comprises at least two gripper units 200. End effector 100 may comprise four gripper units 200 as shown in Figure 1, for example. The gripper units 200 may be configured such that the end effector 100 forms a figurative “hand”, with the gripper units 200 being the “fingers”. Each of the gripper units 200 is configured to couple to the main body 150 of the end effector 100.

[0036] In some embodiments, end effector 100 further comprises a size adaptability mechanism 152 for adjusting spacing between the at least two gripper units 200. The at least two gripper units 200 may be coupled to the size adaptability mechanism 152, thereby enabling the spacing between the gripper units 200 to be increased or reduced. The size adaptability mechanism 152 may be a four-follower face cam mechanism actuated by a motor 154, for example. In some embodiments, end effector 100 furthercomprises a plurality of pin rods 156 for coupling components of the end effector 100 to one another.

[0037] In some embodiments, end effector 100 further comprises an imaging device 190 positioned between the at least two gripper units 200 and toward their respective proximal ends. That is, the imaging device 190 may be positioned in the figurative “palm” of the end effector 100, for example. In some embodiments, the imaging device 190 is one of: a camera, a LiDAR, a radar, or a hyperspectral imaging device. The imaging device 190 may allow the robotic system, to which the end effector 100 is coupled, to gather data relating to the position of the end effector 100 and the at least two gripper units 200 within the environment in which the robotic system is being used. Data gathered from the imaging device 190 may also include information relating to an object with which the end effector 100 and gripper units 200 are interacting with. For example, the imaging device 190 may capture image data of the position of the end effector 100 and gripper units 200 with respect to a fruit and use this image data to reposition the end effector 100 and gripper units 200 accordingly.

[0038] Referring to Figure 2, there is shown an example gripper unit 200 for assisting in robotic object interaction, according to some embodiments. Gripper unit 200 comprises a gripper body 210 having a proximal end 212 and a distal end 214. The gripper body 210 may generally be considered to have a finger-like appearance, where the gripper body 210 narrows progressively from a wide base at the proximal end 212 to a pointed (but not sharp) tip at the distal end 214. The surface of the gripper body 210 may comprise a shape-adaptive flexible material such as thermoplastic polyurethane (TPU), thermoplastic elastomer (TPE), or thermoplastic copolyester (TPC) for example. The shape-adaptive flexible material may allow gripper body 210 to at least partially conform to the shape of the object with which it is interacting. In some embodiments, an inner wall 220 of the gripper body 210 may include a silicone- based or rubber-based surface finish to enhance gripping friction of the gripper unit 200.

[0039] The gripper unit 200 may further comprise a plurality of ribbed portions 218 for providing structural integrity to the gripper body 210 whilst allowing the gripper body 210 to also be shape-adaptive. The plurality of ribbed portions 218 may comprise the same flexible material as the gripper body 210, such as TPU (thermoplastic polyurethane), TPE (thermoplastic elastomer), or TPC (thermoplastic copolyester), or alternatively a lightweight and rigid metallic material such as aluminium, aluminium alloys, titanium, or titanium alloys, for example.

[0040] In embodiments where the plurality of ribbed portions 218 comprise a rigid material, the plurality of ribbed portions 218 act to reduce moment forces within the cross-section of the gripper body 210 to prevent undesired rotation along a longitudinal axis between the proximal end 212 and the distal end 214 when interacting with an object. This undesired rotation can result in loss of grip of the object during clasping or gripping thereof. In embodiments where the plurality of ribbed portions 218 comprise a flexible material, the plurality of ribbed portions 218 provide additional flexibility to protect both the gripper body 210 and the environment in which it operates when strong external forces are applied to the gripper body 210.

[0041] In some embodiments, to account for the finger-like appearance of the gripper body 210, the plurality of ribbed portions 218 may become progressively larger in terms of length and height from the tip (distal end 214) to the base (proximal end 212) of the gripper body 210. That is, a ribbed portion 218 toward the distal end 214 may be smaller than a ribbed portion 218 at the proximal end 212, for example. This particular structure may allow the gripper body 210 to have a natural curvature as illustrated in Figure 2. The described ribbed portion 218 structure may also allow greater flexion in the gripper body 210 when the gripper unit 200 is actuated.

[0042] The gripper body 210 further comprises a channel 216 extending between the proximal end 212 and the distal end 214 and defining an opening between the inner walls 220. The channel 216 may extend the entire length of the gripper body 210 or may extend a portion of the length of the gripper body 210.

[0043] Gripper unit 200 further comprises a gripper base 250 configured to couple to the main body 150 of the end effector 100. The gripper base 250 comprises a housing 252, configured to house, for example, electronic components and circuitry. The gripper body 210 is coupled at the proximal end 212 to the gripper housing 252.

[0044] The gripper base 250 further comprises an actuator 254 for actuating the gripper unit 200. The actuator 254 includes an actuating arm 256 coupled to the gripper housing 252. The gripper base 250 further comprises a pivot arm 258 coupled to the gripper housing 252 and forming a pivot point 260, about which the gripper housing 252, and therefore the gripper body 210 coupled thereto, pivot. In some embodiments, the actuator 254 is an electrically driven linear actuator. In some embodiments, the actuator 254 is a pneumatically driven linear actuator. Each of the actuator 254 and the pivot arm 258 are coupled to a base plate 262. In some embodiments, the base plate 262 of the gripper unit 200 is configured to couple to the size adaptability mechanism 152 of the end effector 100.

[0045] Referring to Figures 3A and 3B, there is shown an example illustration of the actuator 254 manipulating the gripper units 200 of the end effector 100 to interact with an object 300, according to some embodiments. The actuator 254 is configured to extend or retract the actuating arm 256 such that it exerts a pushing or pulling force onto the gripper housing 252. As shown in Figure 3A, the actuating arm 256 is retracted such that the gripper units 200 are in an open position. As the actuator 254 extends the actuating arm 256, the gripper housing 252 is pivoted about the pivot point 260, moving the gripper units 200 into a closed position to clasp the object 300. The actuator 254 may then retract the actuating arm 256, pivoting the gripper housing 252 about the pivot point 260, to move the gripper units 200 back into the open position.

[0046] In some embodiments, when the actuator arms 256 is fully retracted by the actuators 254 of the respective gripper units 200, the gripper units 200 may be considered to be in a fully open position. That is, to fully open the “hand” of the end effector 100, the actuators 254 retract the actuating arms 256 as much as they’re able, for example. In some embodiments, when the actuator arms 256 are fully extended bythe actuators 254 of the respective gripper units 200, the gripper units 200 may be considered to be in a fully closed position. That is, to fully close the “hand” of the end effector 100, the actuators 254 extend the actuating arms 256 as much as they’re able, for example. In some embodiments, the actuators 254 may be configured to stop extension and retraction of the actuating arms 256 at any position between the fully open and fully closed positions. That is, the actuators 254 may stop extending and retracting of the actuating arms 256 such that the “hand” of the end effector 100 is in a partially open position or a partially closed position, for example.

[0047] Figures 4A and 4B show an example configuration of the gripper unit 200 including an imaging device 232, according to some embodiments. Imaging device 232 may be a camera, a LiDAR, a radar, or a hyperspectral imaging device, for example. The imaging device 232 is configured such that its field of view (FOV) encompasses the opening defined by the channel 216 in a direction outward of the gripper body 210. That is, the imaging device 232 can see through the opening defined by the channel 216 in a direction pointing away from the gripper body 210, for example. The imaging device 232 is configured to capture image data of its FOV for use by the robotic system, for example.

[0048] In some embodiments, the imaging device 232 is positioned at the gripper base 250 or the gripper body 210. The imaging device 232 may be carried by, mounted to, coupled to, or positioned within the gripper base 250 or the gripper body 210, for example. In embodiments where the imaging device 232 is positioned within the gripper body 210, the imaging device 232 essentially creates an “eye-in-finger” configuration, in which the figurative eyes of a robotic system to which the end effector is coupled are within the figurative fingers, for example.

[0049] In some embodiments, and as shown in Figure 4A, the imaging device 232 is positioned at the proximal end 212 of the gripper body 210. The gripper body 210 may include an aperture 236 through which the imaging device 232 can extend through, as shown in Figure 4B, for example. The imaging device 232 may be positioned between the distal end 214 and the proximal end 212 of the gripper body 210, for example. Theimaging device 232 may be positioned toward the distal end 214 of the gripper body 210, for example.

[0050] Imaging device 232 may further include circuitry 234 coupled thereto. Circuitry 234 may enable the imaging device 232 to perform its functionality. In some embodiments, circuitry 234 and, in part, imaging device 232 may be housed within the gripper housing 252. In some embodiments, imaging device 232 is mounted at a distance from circuitry 234.

[0051] In some embodiments, gripper unit 200 further includes at least one light source 230. Gripper unit 200 may include four light sources 230, as shown in Figure 4A, for example. Light source 230 may be a light-emitting diode (LED), for example. In some embodiments, and as shown in Figure 4A, the at least one light source 230 is positioned at the proximal end 212 of the gripper body 210. The gripper body 210 may include at least one aperture 238 through which the at least one light source 230 can extend through, as shown in Figure 4B, for example. In some embodiments, the at least one light source 230 is positioned proximal to the imaging device 232 and configured to emit light in the direction in which the imaging device 232 is pointing to assist in enhancing the quality of captured image data.

[0052] In some embodiments, gripper unit 200 further includes a reflective surface 222. The reflective surface 222 may cover the base of the channel 216, or in other words, the side of the channel 216 opposite the opening between the inner walls 220, for example. In some embodiments, the reflective surface 222 covers the entire base of the channel 216. In some embodiments, the reflective surface 222 covers a portion of the base of the channel 216. In embodiments including the reflective surface 222, the imaging device 232 is configured such that it’s field of view (FOV) encompasses the opening of the channel 216 and at least a portion of the reflective surface 222. In some embodiments, the FOV of the imaging device 232 may encompass the opening of the channel and the entire reflective surface 222. The reflective surface 222 may act as a mirror, configured to reflect light to the imaging device 232 to subsequently becaptured as image data. This will be described in further detail below, in relation to Figures 7A and 7B.

[0053] Figure 5 shows an illustration exemplifying the size adaptability of the end effector 100, according to some embodiments. As shown, using the size adaptability mechanism 152 of end effector 100, the gripper units 200 move laterally to increase or decrease the spacing between them, for example. Referring to Figures 6A and 6B, there is shown an illustration of an example size adaptability mechanism 152 in the form of a four-follower face cam mechanism 600. The four-follower face cam mechanism 600 comprises a rotation plate 602 mounted to the motor 154 such that actuation of the motor 154 causes the rotation plate 602 to rotate. The rotation plate 602 includes a plurality of slots 604 corresponding to the number of gripper units 200 of the end effector 100. That is, for every gripper unit 200 of end effector 100, there is a corresponding slot 604 in the rotation plate 602, for example.

[0054] Each slot 604 is configured to receive a pin rod 606 coupled to the base plate 262 of the gripper unit 200. The pin rods 602 are configured to move freely within the slot 604. As shown in Figure 6B, the four-follower face cam mechanism 600 further comprises a guiding plate 608. The guiding plate 608 is coupled to the end effector 100 adjacent to the rotation plate 602. The guiding plate 608 includes a plurality of guiding grooves 610 corresponding to the number of gripper units 200 of the end effector 100. The guiding grooves 610 are configured to receive the base plates 262 of gripper units 200 and laterally limit the movement of the base plates 262. That is, the base plates 262 may only slide back and forth along a single axis denoted by arrows 612 within the guiding grooves 610, for example.

[0055] Spacing between the base plates 262, and therefore the gripper units 200, is determined by the rotation of the rotation plate 602. As the rotation plate 602 rotates in a first direction, the pin rods 606 are guided within the slots 604 toward the centre of the rotation plate 602 due to the limited lateral movement of each corresponding base plate 262 along the axis denoted by arrows 612. That is, the pin rods 606 are unable to move off the axis denoted by the arrows 612 as the base plates 262 coupled thereto arelimited in their movement by the guiding plate 610. Rotation of the rotation plate 602 in the first direction and movement of the pin rods 606 toward the centre of the rotation plates 602 causes the spacing between the base plates 262, and therefore the gripper units 200, to decrease.

[0056] Similarly, as the rotation plate 602 rotates in a second direction opposite the first direction, the pin rods 606 are guided within the slots 604 away from the centre of the rotation plate 602. Rotation of the rotation plate 602 in the second direction and movement of the pin rods 606 away from the centre of the rotation plates 602 causes the spacing between the base plates 262, and therefore the gripper units 200, to increase.

[0057] Referring to Figures 7A and 7B, there are shown example illustrations of the field of view (FOV) of the imaging device 232 of a gripper unit 200 with and without the reflective surface 222, respectively, according to some embodiments. Additionally, Table 1 below outlines the relevant variables and their respective ranges for determining coverage of the imaging device 232 with respect to an object 300. Table 1 – Variables used to determine coverage of the imaging device

[0058] Figure 7A exemplifies an embodiment of the gripper unit 200 without the reflective surface 220. The imaging device 232 tilt, ^^^, is determined by the actuator 254, wherein extending the actuator arm 256 increases the angle ^^^and retracting the actuator arm 256 decreases the angle ^^^. The FOV, ^^^, of the imaging device 232 may be dependent on the model of imaging device 232 used, or the settings applied to the imaging device 232 in software, for example. The distance ^^^is the distance from the centre of the imaging device 232 to the external surface of the gripper body 210. That is, ^^^is the distance from the centre of the imaging device 232 to the surface on which the reflective surface 222 would be placed, for example. The distance ^^^is the length of the reflective surface 222.

[0059] Object offset, ^^^, is the distance that the object 300 is from the centre point 703 between the imaging devices 232. The distance ^^^is the distance of the imaging device 232 from the centre point 703. The distance 2^^^between the opposing imaging devices 232 is determined by the size adaptability mechanism 152. The object 300 is positioned a distance ℎ^from a plane 701 formed between the centre of each imaging device 232 and has a radius ^^^.

[0060] Imaging device 232 has a coverage angle, ^^^, which is based on the amount of the surface of the object 300 that the FOV of imaging device 232 can capture. That is, coverage angle, ^^^, as shown in Figure 7A, is the portion of the object 300 surface between point 702 and point 704 which the imaging device 232 can capture, for example. To determine the distance ℎ^, equation (1) below is used. Once ℎ^is determined, equation (2) below is used to determine the coverage angle ^^^.ℎ^ ൌ ^^^ ^ ^^^^ െ ^^^^ cot^^^ ൌ arccos arctan ൬ ^^^ ^ ^^^ℎ^ ^ ^2^

[0061] In embodiments including the reflective surface 222, the coverage angle, ^^^, increases due to the additional viewing / imaging angles provided by the reflective surface 222. That is, the reflective surface 222 acts as a mirror, providing the imaging device 232 with alternate viewing angles of the object 300, for example. As shown in Figure 7B, the coverage angle, ^^^, increases to between point 710 and point 712 with the reflective surface 222 at the same conditions as Figure 7A. The optimal coverage angle, ^^^, may occur when the FOV of the imaging device 232 is positioned such that at its boundary is defined by point 706 which corresponds to a reflection of the centre of the object 300 at point 710, for example. To determine the optimal distance ℎ^when the boundary of the imaging device 232 is defined at point 706, equation (3) below is used. Once ℎ^is determined, function (4) below is used to determine the coverage angle, ^^^, based on the optimal distance ℎ^.ℎ^ ൌ ^^^ cos^^^^^cos^^^ െ ^^^ ^ sin^^^ െ ^^^ cot^^^ ^ 3^^^^sin^^^ ^ ^^^ ^ ^^^ ^ ^^^^ െ ^^^^ cot^^^ ^ 3^^^ ^3^^^ℎ^^^^^^:

[0062] Referring to Figure 8, there is an illustration comparing prior end effector imaging methods and end effector 100 including the gripper unit 200 with and withoutthe reflective surface 222. As shown in Figure 8, there may be several imaging devices 232 in different configurations, namely C1, C2, C3, Cf, and Cf’, wherein each imaging device 232 is positioned to view the object 300. C1-3represent example configurations of imaging devices 232 of prior systems. Cf represents an example configuration of two imaging devices 232 positioned proximal to the gripper bases 250 of two gripper units 200 of the end effector 100. Cf’represents an example configuration of a reflected view of the imaging device 232 of Cfpositioned proximal to the gripper base 250 of a gripper unit 200 including a reflective surface 222.

[0063] Using the equations outlined in relation to Figures 7A and 7B, C1, positioned the furthest away from the object 300, has a coverage angle corresponding to approximately 47%, or about 46.7-47.3%, of the surface area of the object 300, represented by the line 802. That is, C1 can see, or has within its field of view (FOV), approximately 47%, or about 46.7-47.3%, of the surface area of the object 300 as indicated by S1, for example. C2, positioned closer to the object 300 than C1, has a coverage angle corresponding to approximately 44%, or about 43.3-44.6%, of the surface area of the object 300, represented by the line 804. That is, C2can see, or has within its field of view (FOV), approximately 44%, or about 43.3-44.6%, of the surface area of the object 300 as indicated by S2, less than that of C1, for example. C3, positioned closer to the object 300 than C2, has a coverage angle corresponding to approximately 42%, or about 39.2-45.2%, of the surface area of the object 300, represented by the line 806. That is, C3 can see, or has within its field of view (FOV), approximately 42%, or about 39.2-45.2%, of the surface area of the object 300 as indicated by S3, less than that of C1and C2, for example.

[0064] The two imaging devices 232 of Cf are positioned in a configuration corresponding to an end effector 100 including at least two gripper units 200 interacting with an object 300. The imaging devices 232 of Cf, have a coverage angle corresponding to approximately 74%, or about 69.7-78.5%, of the surface area of the object 300, represented by the line 808. That is, the two Cf imaging devices 232 can see, or have within their respective fields of view (FOVs), approximately 74%, or about69.7-78.5%, of the surface area of the object 300 as indicated by Sf, greater than that of any one of C1-3, for example.

[0065] In embodiments where the gripper unit 200 further includes a reflective surface 222, the reflected view provided by the reflective surface 222 to the imaging device 232 positioned at Cf corresponds to Cf’, an equivalent position of the imaging device 232 positioned at Cfdue to the reflective view. That is, the reflective surface 222 increases the FOV of the imaging device 232 positioned at Cfby providing a mirrored image of the object 300 from an alternate angle, for example. The imaging devices 232 of Cf in combination with the additional view provided by Cf’ have a coverage angle corresponding to approximately 82%, or about 77.4-85.6%, of the surface area of the object 300, represented by the line 810. That is, the two Cf imaging devices 232 of gripper unit 200 including reflective surfaces 222 can see, or have within their fields of view (FOVs), approximately 82%, or about 77.4-85.6%, of the surface area of the object 300 as indicated by Sf’, greater than that of any one of C1-3 and Cf (with no reflective surface 122), for example.

[0066] Referring to Figure 9, there is shown an example robotic system 900 including at least one end effector 100, the at least one end effector 100 including at least two gripper units 200, according to some embodiments. Robotic system 900 may be a fruit picking apparatus or a quality control inspection unit, for example. Robotic system 900 comprises a processor 902 and a memory 904 accessible to processor 902. Processor 902 may be configured to access data stored in memory 904, to execute instructions stored in memory 904, and to read and write data to and from memory 904. Processor 902, and any processor defined hereafter unless otherwise stated, may comprise one or more microprocessors, microcontrollers, central processing units (CPUs), application specific instruction set processors (ASIPs), or other processor capable of reading and executing instruction code.

[0067] Memory 904, and any memory defined hereafter unless otherwise stated, may comprise one or more volatile or non-volatile memory types, such as RAM, ROM, EEPROM, or flash, for example. Memory 904 may be configured to store executableapplications for execution by processor 904. For example, memory 904 may store at least one robotic control module 906 configured to control at least one robotic limb 199, end effector 100, and at least two gripper units 200 of the robotic system 900, for example.

[0068] The robotic system 900 may comprise additional modules and / or components (not shown) such as a drive unit configured to enable movement of the robotic system 900, a vision subsystem (separate to the imaging device 232 of the gripper unit 200) configured to assist the robotic system 900 to navigate its surrounding environment and perform its function, or a power subsystem configured to provide the robotic system 900 with electrical energy to function, for example.

[0069] As shown in Figure 9, the robotic system 900 is configured to interact with an object 300. The dashed lines indicate an example flow of image data through the robotic system. The at least two imaging devices 232 receive capture direct image data 910 of the object 300. In embodiments where the gripper units 200 further include the reflective surface 222, the imaging devices 232 further capture indirect image data 912 of the object 300 via the reflective surfaces 222.

[0070] Referring to Figure 10, there is shown a method 1000 executed, for example, by the robotic system 900 for interacting with the object 300, according to some embodiments. Method 1000 may be performed by processor 902 executing at least one of robotic control module 906 and image processing module 908 stored in memory 904.

[0071] At step 1002 of method 1000, processor 902 of robotic system 900 executes robotic control module 906 to control the robotic limb 199, end effector 100, and the at least two gripper units 200 to position the gripper units 200 proximal to the object 300. That is, the robotic system 900 moves the gripper units 200, configured to be spaced evenly from one another, such that the object 300 is between them, for example.

[0072] At step 1004 of method 1000, processor 902 executes image processing module 908 to begin receiving a stream of direct image data 910 from the imaging devices 232. In embodiments including the reflective surfaces 222, the stream further includes the indirect image data 912 captured via the reflective surfaces 222.

[0073] In some embodiments, process 902 continuing to execute robotic control module 906, may perform step 1006 of method 1000 to control the end effector 100 to rotate about its central axis such that the at least two gripper units 200 are rotated about the object 300. Rotation of the gripper units 200 about the object 300, and consequently rotation of the imaging devices 232 about the object 300, may allow the robotic system 900 to capture additional image data 910, 912 of the object 300 from additional viewing angles, for example. In some embodiments, the rotation of the gripper units 200 about the object 300 occurs without the gripper units 200 contacting the object 300.

[0074] The end effector 100 may rotate the gripper units 200 about the object 300 depending on a number of gripper units 200 coupled thereto. For example, if the end effector 100 includes two gripper units 200, rotation of 180 degrees may be required to provide a full 360-degree view of the object 300, whereas if three gripper units 200 are provided, rotation of 120 degrees may be required. Step 1006 may be performed simultaneously to step 1004. That is, the stream of image data 910, 912 is received while the gripper units 200 are rotating about the object 300, for example.

[0075] At step 1008, processor 902 continuing to execute image processing module 908, processes the received stream of image data 910, 912 associated to the object 300. In some embodiments, step 1008 is performed by process 902 in tandem with step 1004. That is, processing of the received image data 910, 912 may begin when the robotic system 900 begins to receive the stream of image data 910, 912, for example.

[0076] In some embodiments, the robotic system 900 performs preprocessing and / or labelling of the image data 910, 912. For example, the received image data 910, 912 may be cropped such that only a particular region of interest, including a direct visual or indirect visual via the reflective surface 222 of the object 300 is included. This mayremove portions of the received image data 910, 912 not relevant to robotic system 900 performing its functions and decrease the amount of data to be processed, for example.

[0077] In some embodiments, image data 910, 912 received from imaging devices 232 of different gripper units 200 may be downscaled and subsequently stitched together to form a single stream of image data for processing. For example, in a robotic system 900 including four gripper units 200, each imaging device 232 may capture a stream of image data 910, 912 at a resolution of 1280x720. The four streams of image data 910, 912 captured at 1280x720 may then be downscaled to 960x540 and then further stitched together to form a single stream of image data at a resolution of 1920x1080, for example. In some embodiments, the plurality of streams of image data 910, 912 captured by each imaging device 232 are processed by a multiplexer to combine the plurality of streams of image data into a single stream of image data.

[0078] In some embodiments, depending on the position of the imaging device 232 relative to the object 300, a perspective transform of the captured image data 910, 912 is performed to resize the stream of image data 910, 912 for processing. The perspective transform may further homogenize the received streams of image data 910, 912 for processing, thereby increasing accuracy of image analysis. In some embodiments, a pre-calculated transformation matrix is utilised, enabling real-time image processing and analysis. The pre-calculated transformation matrix may be determined based on the configuration of the gripper units 200 coupled to the end effector 100 as well as the imaging devices 232 of each gripper unit 200 and the respective positioning, for example.

[0079] In some embodiments, the stream of image data 910, 912 is received in a video data format. In some embodiments, the received video data is processed and each frame saved as an individual image file for analysis. This process of saving each frame as an individual image file may be completed after transformation of the received image stream 910, 912. In some embodiments, the directly captured image data 910 and the indirectly captured image data 912 are saved as separate individual image files. For example, 182,400 individual image files saved from received video data maycomprise 91,200 directly captured images and 91,200 indirectly captured images. The indirectly captured image data 912 may be flipped vertically during preprocessing to account for the image being captured via reflective surface 222, for example.

[0080] At step 1010 of method 1000, processor 902 determines a status of the object 300 based on the processed image data. In some embodiments, robotic system 900 determines the status of the object 300 using a first trained machine learning model. In some embodiments, the first trained machine learning model may utilise deep learning techniques. The first trained machine learning model may utilise YOLO v8 model architecture, for example, which may include feature maps from P2 to P5 layers. In some embodiments, the processed image data is fed into a modified CSPDarknet53 backbone which utilises Cross Stage Partial (CSP) connections to improve gradient flow and network efficiency. Subsequently, the processed image data is passed through a Feature Pyramid Network (FPN) to generate a richer feature representation, enabling better detection of objects 300 of various sizes. Lastly, the first training machine learning model predicts bounding boxes, class probabilities, and object score.

[0081] In some embodiments, the first machine learning model is trained on a dataset of images captured by the robotic system 900 and subsequently processed at step 1008. To train the model, a random selection of the processed image data is obtained for labelling to generate training image data. Labelling of the randomly selected processed image data may be performed using, for example, the Roboflow platform to draw bounding boxes on areas of interest on the object 300 to be interacted with. For example, if the object 300 were apples, the areas of interest may include, but are not limited to, bruises, worm holes, or other defects of interest to assist in determining whether an apple is suitable for picking and / or consumption. Similarly, in instances where object 300 were manufactured machine parts, the areas of interest may include, but are not limited to, surface irregularities, burn marks, dimensional irregularities, and chatter marks, for example.

[0082] In some embodiments, augmentation techniques may be applied to the training image data to generate a plurality of variations of each training image, therebyincreasing the amount of data available to train the first machine learning model. Using augmentation techniques may also improve the robustness and generalisation of the first trained machine learning model. These augmentation techniques may involve the adjustment of parameters such as hue, saturation, brightness, and exposure. The first machine learning model is then trained on the augmented training image data to generate a first trained machine learning model.

[0083] In some embodiments, the first machine learning model is further trained to determine the presence of obstacles or foreign objects that may impede or inhibit interactions between the gripper units 200 and the object 300. In some embodiments, the first machine learning model is further trained to determine characteristics of the object. For example, in the case of fruit, the first machine learning model may be trained to determine a ripeness of the fruit. In embodiments in which the end effector 100 and gripper units 200 are configured to interact with fruit, the ability of the imaging devices 232 to view approximately 77.4% to approximately 85.6%, in embodiments including a reflective surface 222, of the surface area of a piece of fruit intended to be picked allows for the robotic system to more accurately determine ripeness by viewing the shaded side not typically visible.

[0084] To determine the status of the object 300, the processed image data is provided to the first trained machine learning model which outputs a status depending on preset tolerance levels for the application. For example, if the tolerance level is 10%, if 10% of the total processed image data provided to the first trained machine learning model are determined to contain defects, the model may output a status of indicating that the object 300 is ‘defective’.

[0085] At step 1012, processor 902 determines control instructions for controlling the robotic limb 199, the end effector 100, and the at least two gripper units 200 based on the status of the object 300. In some embodiments, determining of control instructions is performed by processor 902 in tandem to executing image processing module 908 and processing the received image data 910, 912 to determine a status. That is, the robotic system 900 may update in real time the control instructions in response to theobserved interaction between the robotic limb 199, the end effector 100, and the gripper units 200 with the object 300, for example.

[0086] For example, in the case of fruit picking, if the status indicates that the object 300 (fruit) is ‘defective’ the processor 902 may determine control instructions to ignore the fruit and move to inspect the next fruit. For example, in the case of fruit picking, if the status indicates that the object 300 (fruit) is ‘not defective’ the processor 902 may determine control instructions to pick the fruit and place it into a storage container. In some embodiments, where the object 300 is a fruit, the status may include one or more of: ripe, unripe, defective, not defective, slipping, obstructed, and unobstructed.

[0087] For example, in the case of manufacturing quality control, if the status indicates that the object 300 (a manufactured part) is ‘defective’, the processor 902 may determine control instructions to interact with the manufactured part to remove it from the manufacturing line. For example, in the case of manufacturing quality control, if the status indicates that the object 300 (a manufactured part) is ‘not defective’ the processor 902 may determine control instructions to ignore the manufactured part and move to inspect the next manufactured part.

[0088] In some embodiments, robotic system 900 uses a second trained machine learning model to determine the control instructions based on the status of the object 300. In some embodiments, the first trained machine learning model is also trained to determine the control instructions based on the determined status of the object 300. Interaction between the end effector 100 and gripper units 200 with the object 300 may be captured using the imaging devices 232 and subsequently used to train the respective machine learning model to optimise its interaction with objects. That is, the robotic system 900 may learn from its previous interactions with objects to better enable it to optimise how it interacts with said objects, for example.

[0089] At step 1014 of method 1000, processor 902 executing robotic control module 906 controls the robotic limb 199, end effector 100, and the at least two gripper units200, based on the control instructions determined at step 1012. This control may include manipulating the object 300 or moving to a new object 300, for example.

[0090] In some embodiments, image processing module 908 is further configured to receive and analyse a stream of image data 910, 912 from the imaging devices 232 while the end effector 100 and the gripper units 200 are manipulating the object 300. Analysing the image data 910, 912 during manipulation of the object 300 may assist the robotic system 900 in determining how to accurately and efficiently manipulate the object 300 for the desired outcome, for example. Analysing the image data 910, 912 during manipulation of the object 300 may assist the robotic system 900 in handling obstructions and foreign objects, for example. Analysing the image data 910, 912 during manipulation of the object 300 may assist the robotic system 900 to determine that slippage of the object 300 within the grasp of the gripper units 200 is occurring and sufficiently account for said slippage, for example. That is, image processing module 908 may determine that the object 300 is slipping from the grasp of the gripper units 200 and subsequently update the determined status, for example.

[0091] In some embodiments, image processing module 908 is further configured to determine deformation of the gripper body 210 based on the processed image data. Referring to Figures 11A and 11B, there are shown example gripper bodies 210 including a plurality of tracking points 1102, according to some embodiments. The gripper body 210 of Figure 11A is representative of a gripper unit 200 not in contact with an object. As shown, the gripper body 210 includes a plurality of tracking points 1102 marked on the ribbed portions 218. The plurality of tracking points 1102 are in an initial position when the gripper body 210 is not in contact with an object.

[0092] The plurality of tracking points 1102 may be physical points identified by the image processing module 908, such as the ends of the ribbed portions 218, for example. In some embodiments, the plurality of tracking points 1102 are software defined points determined by image processing module 908. The plurality of tracking points 1102 are configured to be disposed within the channel 216.

[0093] The gripper body 210 of Figure 11B is representative of a gripper unit 200 in contact with an object 300, wherein the gripper body 210 is deformed about the surface of object 300. The plurality of tracking points 1102 are displaced from their initial position shown in Figure 11A, where, in Figure 11B, there is shown increased distance between the plurality of tracking points 1102. The image processing module 908 may analyse the displacement of the tracking points 1102 to determine deformation of the gripper body 210. That is, movement or change in position of the plurality of tracking points 1102 may be monitored and analysed to determine deformation of the gripper body 210, for example. In some embodiments, the determined deformation is used to determine the force applied by the gripper unit 200 to an object. In some embodiments, determined deformation is used to determine contact with an object. In some embodiments, robotic control module 906 further determines the control instructions based on the determined deformation of the gripper unit 200.

[0094] Additional tracking points 1102 may be provided to enable the image processing module 908 to further determine complex deformations, such as twisting, compression, and a combination of bending and twisting. Different deformations may exhibit distinct movement patterns of the ribbed portions 218, which can be reflected by the plurality of tracking points 1102 and captured by the imaging devices 232 of the respective gripper unit 200. The first machine learning model may then be trained to identify these distinct movement patterns and to subsequently determine deformation of the gripper bodies 210.

[0095] It will be appreciated by persons skilled in the art that numerous variations and / or modifications may be made to the above-described embodiments, without departing from the broad general scope of the present disclosure. The present embodiments are, therefore, to be considered in all respects as illustrative and not restrictive.

Claims

CLAIMS:

1. A gripper unit configured to couple to an end effector to assist in robotic object interaction, the gripper unit comprising: a gripper body having proximal and distal ends, the gripper body defining a channel extending between the proximal and distal ends; a gripper base coupled to the proximal end of the gripper body and configured to couple to the end effector; and an imaging device positioned at the gripper base or the gripper body; wherein the channel includes an opening through which the imaging device sees.

2. The gripper unit of claim 1, further comprising a reflective surface extending within the channel between the proximal and distal ends of the gripper body, wherein the reflective surface is within a field of view of the imaging device.

3. The gripper unit of claim 2, wherein the reflective surface is configured to deform with deformation of the gripper body.

4. The gripper unit of claim 2 or 3, wherein the reflective surface is configured to provide the imaging device with additional imaging angles of an object, the object positioned near the distal end of the gripper unit.

5. The gripper unit of any one of claims 1 to 4, further comprising a plurality of tracking points disposed within the channel, wherein the plurality of tracking points are positioned within the field of view of the imaging device.

6. The gripper unit of claim 5, wherein deformation of the gripper body is determined based on the plurality of tracking points.

7. The gripper unit of claim 5 or claim 6, wherein the plurality of tracking points comprise one or more of: a plurality of physical points within the channel or a plurality of software defined points within the channel.

8. The gripper unit of any one of claims 1 to 7, wherein the gripper unit further comprises an actuator configured to pivot the gripper body relative to the end effector.

9. The gripper unit of any one of claims 1 to 8, further comprising at least one light source positioned at the gripper base and configured to emit light toward the second end of the gripper body.

10. An end effector for assisting in robotic object interaction, the end effector comprising: an end effector body having proximal and distal ends, the proximal end configured to couple to a robotic system; and at least two gripper units of any one of claims 1 to 9 coupled to the distal end of the end effector body, the at least two gripper units spaced apart from each other.

11. The end effector of claim 10, further comprising a second imaging device coupled to the distal end of the end effector and configured with a field of view (FOV), the at least two gripper units positioned within the FOV.

12. The end effector of claim 10 or claim 11, further comprising a mechanism for adjusting a distance between the at least two gripper units.

13. The end effector of any one of claims 10 to 12, comprising three gripper units coupled to the distal end of the end effector body, wherein the three gripper units are spaced apart from each other.

14. The end effector of any one of claims 10 to 12, comprising four gripper units coupled to the distal end of the end effector body, wherein the four gripper units are spaced apart from each other.

15. A robotic system for robotic object interaction, the robotic system comprising: a body housing a processor and a memory, the memory accessible to the processor; at least one robotic limb coupled to the body; and at least one end effector of any one of claims 10 to 14 coupled to the at least one robotic limb; wherein the processor is configured to execute instructions stored in the memory so as to cause the robotic system to interact with an object using at least one of: the robotic limb and the at least one end effector.

16. A method of assisting robotic object interaction, the method comprising: positioning a gripper unit proximal to an object, the gripper unit coupled to an end effector, wherein the gripper unit has a gripper body, and an imaging device positioned relative to the gripper body, wherein the imaging device has a view through an opening of the gripper body; and receiving a stream of imaging data of the object from the imaging device.

17. The method of claim 16, wherein the gripper body further includes a reflective surface, and the receiving comprises receiving imaging data relating to a direct view of the object; and receiving imaging data relating to an indirect view of the object via the reflective surface.

18. The method of claim 16 or claim 17, further comprising rotating the gripper unit about the object without gripping the object to inspect the object.

19. The method of claim 16 or claim 17, further comprising: determining, using a first machine learning model, based on the received stream of imaging data, a status of the object; and determining, using a second machine learning model, based on the status of the object, control instructions for at least one of the end effector and the gripper unit.

20. The method of claim 19, further comprising manipulating the object using at least one of the end effector and the gripper unit based on the control instructions.

21. The method of any one of claims 19 to 20, wherein the object is a fruit, and the status includes one or more of: ripe, unripe, defective, not defective, slipping, obstructed, and unobstructed.

22. The method of any one of claims 19 to 21, further comprising determining, based on the received imaging data, deformation of the gripper unit.

23. The method of claim 22, wherein the determining the control instructions is further based on the determined deformation of the gripper unit.

Images