Alignment system for automated object transfer

Abstract

Various object transfer systems and methods of automated alignment for object transfer systems are provided. An example method includes positioning at least one camera at a position in which the at least one camera is capable of capturing a digital image of an environment. The method also includes determining, based on a digital image from the at least one camera, a first location of a first object of the at least one object stored within the at least one shelf structure. The first location of the first object is determined based on an object identifier marker. The method further includes causing a transportation arm device to be moved to the first location of the first object. The method still further includes causing the transportation arm device to engage with the first object.

Description

ALIGNMENT SYSTEM FOR AUTOMATED OBJECT TRANSFERFIELD

[0001] An example embodiment relates generally to aligning object transfer systems and more particularly, to automatically aligning object transfer systems.BACKGROUND

[0002] Robotic arms periodically lose orientation and through failure or environmental conditions may have difficulty identifying a location within an object transfer system. Recalibration or re-teaching the robotic arm typically requires manual manipulation of the robotic arm (e.g., moving the robotic arm into the proper location), or manual configuration to a specific target. Manual teaching results in inefficient systems and also causes the robotic arm to be susceptible to mistakes by unskilled users during placement. Furthermore, laboratory environments are susceptible to change through human factors, such as cleaning, inspecting, and accidental involvement. As such, there exists a need for a system that can provide for an automated alignment of object transfer systems.SUMMARY

[0003] The following paragraphs present a summary of various embodiments of the present disclosure and are merely examples of potential embodiments. As such, the summary is not meant to limit the subject matter or variations of various embodiments discussed herein.

[0004] In some aspects, the techniques described herein relate to an automated object transfer system, the system including: a transportation arm device, wherein the transportation arm device is automated; at least one camera; and a controller including at least one processing device coupled to a memory device. The controller is configured to position the at least one camera at a position in which the at least one camera is capable of capturing a digital image of an environment, whereinthe environment includes at least one shelf structure and at least one object stored within the at least one shelf structure; determine, based on a digital image from the at least one camera, a first location of a first object of the at least one object stored within the at least one shelf structure, wherein the first location of the first object is determined based on an object identifier marker; cause the transportation arm device to be moved to the first location of the first object; and cause the transportation arm device to engage with the first object.

[0005] In some aspects, the techniques described herein relate to a system, wherein the controller is further configured to cause a disengagement of the transportation arm device and the first object upon movement of the first object to a destination location.

[0006] In some aspects, the techniques described herein relate to a system, wherein the controller is further configured to update the first location of the first object to the destination location.

[0007] In some aspects, the techniques described herein relate to a system, wherein the object identifier marker may be captured by the at least one camera.

[0008] In some aspects, the techniques described herein relate to a system, wherein the controller is further configured to determine a location of the transportation arm device based on one or more waypoint fiducial markers in the environment.

[0009] In some aspects, the techniques described herein relate to a system, wherein the at least one camera is positioned on the transportation arm device.

[0010] In some aspects, the techniques described herein relate to a system, wherein the controller is further configured to confirm the first object based on the object identifier marker upon movement of the transportation arm device to the first location of the first object.

[0011] In some aspects, the techniques described herein relate to a system, wherein the controller is further configured to determine, using one or more auto-teaching fiducial markers, an orientation of the digital image, wherein the digital image is analyzed in either landscape or portrait mode.

[0012] In some aspects, the techniques described herein relate to a system, wherein the controller is further configured to determine a size of the at least one shelf structure based on atleast one shelf identifier marker.

[0013] In some aspects, the techniques described herein relate to a system, wherein the controller is further configured to cause the transportation arm device to move to a second location of a second object based on the digital image from the at least one camera.

[0014] In some aspects, the techniques described herein relate to a method of operating an object transfer system, the method including: positioning at least one camera at a position in which the at least one camera is capable of capturing a digital image of an environment, wherein the environment includes at least one shelf structure and at least one object stored within the at least one shelf structure; determining, based on a digital image from the at least one camera, a first location of a first object of the at least one object stored within the at least one shelf structure, wherein the first location of the first object is determined based on an object identifier marker; causing a transportation arm device to be moved to the first location of the first object; and causing the transportation arm device to engage with the first object.

[0015] In some aspects, the techniques described herein relate to a method, further including causing a disengagement of the transportation arm device and the first object upon movement of the first object to a destination location.

[0016] In some aspects, the techniques described herein relate to a method, further including updating the first location of the first object to the destination location.

[0017] In some aspects, the techniques described herein relate to a method, wherein the object identifier marker may be captured by the at least one camera.

[0018] In some aspects, the techniques described herein relate to a method, further including determining a location of the transportation arm device based on one or more waypoint fiducial markers in the environment.

[0019] In some aspects, the techniques described herein relate to a method, wherein the at least one camera is positioned on the transportation arm device.

[0020] In some aspects, the techniques described herein relate to a method, further including confirming the first object based on the object identifier marker upon movement of the transportation arm device to the first location of the first object.

[0021] In some aspects, the techniques described herein relate to a method, further including determining, using one or more auto-teaching fiducial markers, an orientation of the digital image, wherein the digital image is analyzed in either landscape or portrait mode.

[0022] In some aspects, the techniques described herein relate to a method, further including determining a size of the at least one shelf structure based on at least one shelf identifier marker.

[0023] In some aspects, the techniques described herein relate to a method, further including causing the transportation arm device to move to a second location of a second object based on the digital image from the at least one camera.BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Many aspects of the present disclosure will be better understood with reference to the following drawings. The components in the drawings are not necessarily to scale, with emphasis instead being placed upon clearly illustrating the principles of the disclosure. Moreover, in the drawings, like reference numerals designate corresponding parts throughout the several views. It should be recognized that these implementations and embodiments are merely illustrative of the principles of the present disclosure. Therefore, in the drawings:

[0025] FIG. 1 is an illustration showing an example controller used to align an object transfer system, in accordance with various embodiments of the present disclosure;

[0026] FIG. 2A is an illustration of an example transportation arm device, in accordance with various embodiments of the present disclosure;

[0027] FIG. 2B is an illustration of another example transportation arm device, in accordance with various embodiments of the present disclosure;

[0028] FIGs. 3A and 3B illustrate example cameras that may be positioned on the transportation arm device, in accordance with various embodiments of the present disclosure;

[0029] FIG. 4 illustrates a lens cap used for a camera positioned on the transportation arm device, in accordance with various embodiments of the present disclosure;

[0030] FIG. 5 is a flowchart of an example method of operating an object transfer system, in accordance with various embodiments of the present disclosure; and

[0031] FIGs. 6A, 6B, and 6C illustrate different fiducial markers used with the object transfer system, in accordance with various embodiments of the present disclosure.DETAILED DESCRIPTION

[0032] The presently disclosed subject matter now will be described more fully hereinafter with reference to the accompanying drawings, in which some, but not all embodiments of the presently disclosed subject matter are shown. Like numbers refer to like elements throughout. The presently disclosed subject matter may be embodied in many different forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided so that this disclosure will satisfy applicable legal requirements.

[0033] Indeed, many modifications and other embodiments of the presently disclosed subject matter set forth herein will come to mind to one skilled in the art to which the presently disclosed subject matter pertains having the benefit of the teachings presented in the foregoing descriptions and the associated drawings. Therefore, it is to be understood that the presently disclosed subject matter is not to be limited to the specific embodiments disclosed and that modifications and other embodiments are intended to be included within the scope of the appended claims.

[0034] Throughout this specification and the claims, the terms “comprise,” “comprises”, and “comprising” are used in a non-exclusive sense, except where the context requires otherwise. Likewise, the term “includes” and its grammatical variants are intended to be non-limiting, such that recitation of items in a list is not to the exclusion of other like items that can be substituted or added to the listed items.

[0035] In various embodiments, the term “shelf structure” refers to any structure on which an object is placed to be retrieved via a transportation arm device. As such, the shelf structure need not necessarily be a shelf, but may be a structure with a surface that a object is positioned. The shelf structure may have indicators, as detailed herein.I. Example Use Case

[0036] In some aspects, the techniques described herein relate to an automated object transfersystem, the system including: a transportation arm device, wherein the transportation arm device is automated through the use of signals and data from cameras, mechanical hardware (load cells, inertial measurement units), and computer software. For example, in one aspect, at least one camera and a controller including at least one processing device coupled to a memory device are equipped and configured electronically and mechanically to a transportation arm device. Often times such equipment may be shielded in a housing or other aperture to protect from environmental harm, such as liquids or other contaminants that may harm electrical components. The controller is configured to position the at least one camera at a position in which the at least one camera is capable of capturing a digital image of an environment, wherein the environment includes at least one shelf structure and at least one object stored within the at least one shelf structure; determine, based on a digital image from the at least one camera, a first location of a first object of the at least one object stored within the at least one shelf structure, wherein the first location of the first object is determined based on an object identifier marker; cause the transportation arm device to be moved to the first location of the first object; and cause the transportation arm device to engage with the first object. In some aspects a shelf structure is a structure in which a labware object may rest or otherwise be placed, in other aspects the shelf structure comprises the exterior of a labware component, such as a labware device, wherein the labware device has an object identifier on the housing.

[0037] In some aspects, the techniques described herein relate to a system, wherein the controller is further configured to cause a disengagement of the transportation arm device and the first object upon movement of the first object to a destination location.

[0038] In some aspects, the techniques described herein relate to a system, wherein the controller is further configured to update the first location of the first object to the destination location.

[0039] In some aspects, the techniques described herein relate to a system, wherein the object identifier marker may be captured by the at least one camera.

[0040] In some aspects, the techniques described herein relate to a system, wherein the controller is further configured to determine a location of the transportation arm device based onone or more waypoint fiducial markers in the environment.

[0041] In some aspects, the techniques described herein relate to a system, wherein the at least one camera is positioned on the transportation arm device.

[0042] In some aspects, the techniques described herein relate to a system, wherein the controller is further configured to confirm the first object based on the object identifier marker upon movement of the transportation arm device to the first location of the first object.

[0043] In some aspects, the techniques described herein relate to a system, wherein the controller is further configured to determine, using one or more auto-teaching fiducial markers, an orientation of the digital image, wherein the digital image is analyzed in either landscape or portrait mode.

[0044] In some aspects, the techniques described herein relate to a system, wherein the controller is further configured to determine a size of the at least one shelf structure based on at least one shelf identifier marker.

[0045] In some aspects, the techniques described herein relate to a system, wherein the controller is further configured to cause the transportation arm device to move to a second location of a second object based on the digital image from the at least one camera.

[0046] In some aspects, the techniques described herein relate to a method of operating an object transfer system.II. Systems and Methods

[0047] Various embodiments of the present disclosure provide for an alignment system for use in object transfer, such as transporting labware or other objects. To do this, the system includes a camera system or camera array (plurality of cameras) attached to a transportation arm device designed to transport labware from one location to another. For example, the system may transport labware for life science research. The initial system would be designed to transport labware, consumables, carriers, and / or the like. Various embodiments may include the transportation of tubes or other lab and life science research formats. The camera system of various embodimentsallows for the transportation arm device to be moved to a specified transport location (for either pick-up or drop-off of labware) by using one or more waypoints. The controller of various embodiments may determine if the destination for the pick-up or drop-off associated with a particular object (e.g., instrument / hotel / other lab item) has moved since the last time the object was picked up or dropped off. Such a confirmation closes the loop on what would otherwise be an open loop move. To do this, the object may have an object identifier marker (e.g., a label, a barcode, a QR code, etc.) that may be scanned by the system. This object identifier marker in some aspects is designed to provide a calibration metric by having symbols or markings that contrast to provide a computing system, in coordination with cameras, to calibrate and align the transportation arm device. The transportation arm device would confirm the location of that identifier marker (e.g., via the camera(s)) and programmable instructions, and then move to a position to engage with the object (e.g., an identifier marker may be on at one edge of the object and the transportation arm device may be moved to the center of the object in order to engage with the object for transport).

[0048] Various embodiments of the present disclosure provide for four levels of automated alignment. The first level of automated alignment uses object identifier markers. Object identifier markers (e.g., Instrument ID stickers) may be markers affixed to each object in a system for identification. For example, each instrument in the system may have a barcode that may be used to identify the given instrument. Additionally, shelf systems that house the objects may also have one or more shelf identifier markers (e.g., a label, a barcode, a QR code, etc.). For example, each shelf system may have a barcode or other shelf identifier marker on each corner of the shelf system. Additionally, or alternatively, each shelf layer may also have a shelf identifier marker. As such, the system may determine the size / dimensions of the shelf system, as well as the height of each shelf layer. The object identifier marker(s) and / or the shelf identifier marker(s) may be sized such that a camera may read or otherwise process the marker. For example, the object identifier marker(s) and / or the shelf identifier marker(s) may be sized to be processed by a smartphone camera or the like (e.g., a smartphone camera may act as a barcode scanner for the markers).

[0049] The second level of automated alignment includes using waypoint fiducial marker(s).The waypoint fiducial marker(s) may be positioned on each object and / or any hotel stacks (e.g., shelf systems). The waypoint fiducial marker(s) may be three dimensional objects that include multiple fiducials. For example, the waypoint fiducial marker(s) may include three fiducials, two on a first plane and a third positioned closer or farther than the nominal camera position (e.g., as discussed herein). The waypoint fiducial marker(s) may be used for the transportation arm device to locate the fiducials and confirm the current position of the object with the last known good position. In an instance in which an object has moved within some tolerance, the transportation arm device should be able to use these fiducials to detect that and find the new waypoint location in x, y, z, and yaw (rotation about z). In an instance in which any of the fiducials become altered, moved, or fall off the object, then the fiducials will need to be reattached and the location of the object relative to the desired drop-off location(s) will need to be retaught. In various embodiments, any number of fiducial markers may be used. For example, two fiducial markers may be used in a two-dimensional plane (e.g., x by y, y by z, etc.). As such, the operations discussed herein may be completed using any number of fiducial markers. Additional fiducial markers may be used for additional dimensional planes.

[0050] The third level of automated alignment uses additional fiducial markers used to automatically teach the system. To do this, the system would include one or more sets of fiducial markers. For example, the system may include two to four sets of three fiducials. The auto-teaching fiducials may be similar or identical to the waypoint fiducial stickers discussed above. The autoteaching fiducials allow for the system to be analyzed using either portrait or landscape from the camera. As such, the camera on the transportation arm device may be placed at a nominal camera position (e.g., a location in which at least a portion of the shelf system(s) are viewable via the camera). The auto-teaching fiducials will be a known offset from the actual desired pick-up and drop-off locations. By understanding the positional and rotational differences between the coordinates from the “auto-teach” fiducials to the associated waypoint fiducial, the system can calculate and save the offsets from the waypoint stickers to the fiducial tool. In this way, the system can determine structure mapping and use the information during movement of the transportation arm device. In an example embodiment, to allow for auto-teaching, fiducialmarker(s) may be positioned on instrument(s) and / or object(s) used to auto-teach (e.g., plate(s) used to auto-teach). As such, the fiducial marker(s) may have known positioning and / or offsets, such that the system may be auto-taught and / or calibrated.

[0051] A fourth layer of automated alignment may use an additional camera to look forward at a special plate with fiducial markers as the plate is being placed in the drop-off location. In an instance in which there is any change detected between the transportation arm device holding the plate and after the transportation arm device dropped the plate off, adjustments may be made to the drop-off positions of the plate.

[0052] The system may be able to perform a no-stop alignment. To perform a no-stop alignment, the camera(s) may capture an image with the transportation arm device still moving and calculate the location of the fiducial relative to the transportation arm device during movement. Upon determination of the position of the fiducial marker relative to the transportation arm device the offset of the fiducial marker may be used to align the transportation arm device. An example no-stop alignment may be a one-shot alignment in which only a single image is required to determine the location of the transportation arm device. Any number of images may be taken and / or used to calibrate the transportation arm device.

[0053] In various embodiments, the system includes a transportation arm device equipped to a microprocessor and a camera system. The transportation arm device includes at least one camera, but may have two or more cameras to allow for increased precision and accuracy in calibration and alignment. One or more of the camera(s) may be a stereo camera. The transportation arm device is positioned such that the camera can take a digital image or video of the entire system (e.g., the objects within the shelf structures). As such, the camera may capture a digital image or video of the various markers (e.g., the object identifier marker(s), the shelf identifier marker(s), the waypoint fiducial marker(s), the auto-teaching fiducial marker(s), and / or the like). The system may perform filtering operations, transformations, and various tasks on the digital image utilizing libraries such as OpenCV (https: / / docs.opencv.Org / 4.x / index.html) to aid in processing the alignment routine. Based on the digital image or video captured by the at least one camera, the system may determine the position of each object within the environment.

[0054] Auto-teach fiducial marker(s) may be used at each pick-up and / or drop-off position on a given object. As such, the transportation arm device may determine the difference between object locations and store such information. As such, the location of the object within the environment is known and stored. The location may be updated upon movement of the object.

[0055] In various embodiments, the present disclosure allows the automatic delivery system to better sense the world around it, adjust to movements when offline or when labware objects are moved during procedures, and to prevent any accidents and streamline or eliminate the role of the automation engineer required to reconfigure the system.III. With Reference to the Figures

[0056] Referring now to FIG. 1, an example controller is shown used for the automation processes discussed herein. In various embodiments, the processing device may be a controller or microcontroller that is part of or in communication with the alignment system discussed herein. In various embodiments, the controller 100 may include a one or more processing devices 105, one or more memory devices 110, a communication interface 115, and / or the like.

[0057] As illustrated in FIG. 1 , in one embodiment, the controller 100 includes one or more processing devices 105 operatively coupled to a communication interface 115 and memory device(s) 110. It should be understood that the memory device(s) 110 may include one or more databases or other data structures / repositories. The memory device(s) 110 also includes computerexecutable program code that instructs the processing device(s) 105 to operate the network communication interface 115 to perform certain communication functions (e.g., communicating with the gripping finger(s) discussed herein) and / or perform other operations discussed herein. For example, in one embodiment of the controller, the memory device(s) 110 includes, but is not limited to, an alignment application 125 and a data repository 120 comprising data accessed, retrieved, and / or computed by the controller 100. The data repository 120 may store one or more algorithms used herein to process digital images from the at least camera, determine locations of one or more objects, cause the transportation arm device to move within the environment, transport objects from one location to another, and / or the like. The alignment application 125 may includecomputer-executable program code that upon execution, performs certain logic, data-extraction, and data-storing functions of the controller 100 described herein, as well as communication functions of the controller 100.

[0058] Referring now to FIGs. 2A and 2B, example transportation arm devices are shown in accordance with various embodiments. As such, the transportation arm device 200 (also referred to as a gripping device) of FIGs. 2A and / or 2B may be used in the method of FIG. 5. In various embodiments, the transportation arm device 200 may include the controller 100 shown in FIG. 1 and / or otherwise be in communication with the controller 100. A housing (such as the transportation arm device body 225 shown in FIG. 2B may be provided to protect components of the transportation arm device 200).

[0059] In various embodiments, the transportation arm device 200 includes a gripping mechanism (e.g., gripping finger(s) 205). The gripping mechanism may include any number of gripping fingers, projectiles, or gripping structures. As shown, in one example, the gripping mechanism may include two gripping fingers. In various embodiments, the gripping mechanism may include other engagement mechanisms (e.g., such as a platform to receive an object, a spatula delivery mechanism, a magnetic delivery system, or an air pressure delivery system). The gripping finger(s) 205 may have attachments (e.g., as shown in FIG. 2B).

[0060] In various embodiments, the gripping finger(s) 205 may be automatically moveable in relation to the rest of the transportation arm device 200. The movement of the gripping finger(s) may be along a first axis (e.g., as shown in FIGs. 2A and 2B, the gripping finger(s) may move in the direction of other and / or opposite the other gripping finger(s)). The movement of the gripping finger(s) may be along any number of other axes (e.g., the gripping finger(s) may have different functionality with the collision detection discussed herein). The gripping finger(s) 205 may be static in comparison to one another (e.g., the gripping finger(s) 205 may be moved together). Alternatively, the gripping finger(s) 205 may be moved independently (e.g., a first gripping finger may be moved in a first direction relative to the transportation arm device 200 and a second gripping finger may be moved in a second direction relative to the transportation arm device 200 and / or remain static relative to the transportation arm device 200). In various embodiments, oneof the gripping finger(s) may be stationary relative to the transportation arm device body 225 and one or more of the gripping finger(s) may be moveable relative to the transportation arm device body 225 (e.g., the gripping finger may move relative to the stationary gripping finger(s)).

[0061] The transportation arm device 200 may also include at least one camera 215. The at least one camera may be configured to take digital image(s) and / or video(s) of the environment. Any number of different camera designs and / or arrays may be used. FIGs. 3A and 3B illustrate two potential camera designs. As shown, the at least one camera may include multiple cameras and / or lens. For example, the camera 215 of FIG. 3 A illustrates a first camera 300A, a second camera 300B, and a camera array 305 (e.g., that includes three individual cameras), and the camera 215 of FIG. 3B includes a first camera 315A and a second camera 315B. Any number and / or type of cameras may be contemplated by the system. The at least one camera 215 may be positioned such that in an instance in which the transportation arm device 200 is at a nominal position (e.g., a predetermined position), the at least one camera is capable of taking a digital image and / or a video of the environment (e.g., the shelf structure(s) and / or object(s) within the shelf structure(s)).

[0062] As shown in FIG. 4, the camera may have a lens cap 400 that provides an aperture 410 for the camera(s). The lens cap 400 defines a sloped surface to provide a field of view for the camera(s). As shown, the camera(s) (e.g., the first camera 300A and the second camera 300B may be positioned to view through the aperture 410. The lens cap 400 provides protection and prevents obstructions to the camera(s). Any number of different lens caps may be used, such as the design shown in FIG. 3B.

[0063] FIG. 2B illustrates another example transportation arm device 200 that may be used in various embodiments. The transportation arm device 200 of FIG. 2B may share any of the components of the automated gripping device of FIG. 2A. In various embodiments, the transportation arm device body 225 may be larger to position more components within the transportation arm device body (e.g., cameras, processing device(s), memory device(s), motor(s), and / or the like).

[0064] The gripping fingers 205 may include one or more attachments to assist engagement with objects. For example, the attachment with gripping point(s) 220 is attached to the distal endof the gripping finger(s) 205 opposite the transportation arm device body 225. Attachment to the gripping finger(s) allows for different style attachments to be connected. In various embodiments, the gripping points used may be based on the type of objects being transported. For example, the size, shape, consistency, and / or the like of the object may affect the gripping points that are most efficient at securing the object. In various embodiments, the gripping points 220 may be removed from gripping finger(s) 205 and replaced with other gripping points and / or other tooling for which the automated gripping device may be used. The transportation arm device 200 may be calibrated (e.g., using auto-teach features) based on the design of any attachments. For example, the attachment with gripping points 220 of FIG. 2B still lower than the transportation arm device 200 of FIG. 2A, and the system may calibrate the device accordingly.

[0065] In various embodiments, one or more force detector(s) 210 may be positioned within the gripping finger(s). The force detector(s) 210 may be capable of determining forces applied to the gripping finger(s) and determine information relating to the detected force (e.g., weight of engaged object, untended collisions, etc.).

[0066] In various embodiments, the transportation arm device body 225 may house one or more cameras 215 that are positioned to view the gripping finger(s) 205 and any obj ects potentially carried by the gripping finger(s) 205. The camera(s) 230 may be used to determine an instance in which the gripping device is carrying an object (e.g., the front facing camera may detect a plate being carried by the gripping device). The camera may be used to determine the type of object being carried by the system. For example, the controller and / or other processing device(s) in communication with the automated gripping device(s) may use artificial intelligence and / or other machine learning models to determine the object type carried by the automated gripping device. In such an example, the size and / or shape may be used to determine the object type.

[0067] Referring now to FIG. 5, a flowchart 500 of an example method of controlling an automated gripping device. In various embodiments, one or more operations may be automated via a processing device, such as the controller 100 shown in FIG. 1. As such, the operations herein may be carried out by any of the embodiments herein unless otherwise stated. Additionally, the transportation arm devices shown in FIGs. 2A and 2B are merely an example transportation armdevice that may be used to carry out the operations of FIG. 5. As such, any number of different transportation arm device may be used, unless otherwise stated.

[0068] Referring now to Block 510 of FIG. 5, the method includes positioning at least one camera at a position in which the at least one camera is capable of capturing a digital image of an environment. The digital image may be a static image and / or video. In various embodiments, multiple images may be captured. The environment may include at least one object and at least one shelf structure housing the at least one object. The at least one camera 215 may be positioned on the transportation arm device 200. As such, positioning the at least one camera may be achieved via moving transportation arm device 200. In various embodiments, one or more of the at least one camera 215 may be independent from the transportation arm device 200 (e.g., a camera may be fixed within an environment, such that the camera may capture digital images of the environment).

[0069] In an instance in which the at least one camera 215 is positioned on the transportation arm device 200, the transportation arm device 200 may be moved to a nominal position, such that the camera is capable of capturing a digital image of the environment. For example, the nominal position may allow for the camera to capture the entirety of each shelf structure in the environment, or of any labware or other environment. The nominal position may be a known or “home” position of the transportation arm device 200. The movement of the transportation arm device 200 to the nominal position may be automated. In various embodiments, the transportation arm device 200 may be moved to the nominal position in-between movements (e.g., in an instance in which the transportation arm device 200 does not have an assignment to move any objects). The transportation arm device 200 may be controlled via a controller 100.

[0070] Referring now to Block 520 of FIG. 5, the method includes determining, based on a digital image from the at least one camera, a first location of a first object of the at least one object stored within the at least one shelf structure. While the operations herein discuss or otherwise use the term first object and first location, the operations herein may be carried out on any number of objects at different locations. As such, the term “first object” may refer to any object within an environment unless otherwise stated. The method may include determining the location for any number of objects (e.g., a first location of a first object, a second location of a second object, athird location of a third object, etc.) and is not limited to only determining a first location of a first object. Additionally, the digital image from the at least one camera may be any recording by the at least one camera, such as a digital image, multiple digital images, videos, and / or the like. Such images may be converted from analog to digital, or may otherwise be rendered, translated, and processed by a microprocessor or other general / special purpose computer equipped and electronically configured to the transportation arm device.

[0071] In various embodiments, the first location of the first object is determined based on an object identifier marker. Each of the at least one objects may include an object identifier marker (e.g., a label, a barcode, a QR code, etc.) as discussed herein. As such, the object (e.g., the first object) is identified via the object identifier marker and the first location is determined thereupon. In various embodiments, the location of a given object is determined based on the object identifier marker and one or more other markers (e.g., the shelf identifier marker(s), the waypoint fiducial marker(s), the auto-teaching fiducial marker(s), and / or the like). For example, the system may determine the location of a given object based on the object identifier marker of the object and the shelf identifier marker(s) nearby the given object. As such, the various markers may be used to determine the location of various objects within the environment. For example, the location of each object in the environment may be determined by the system. Example markers used for fiducial markers (e.g., the object identifier marker, the shelf identifier marker(s), the waypoint fiducial marker(s), the auto-teaching fiducial marker(s), and / or the like) are shown in FIGs. 6A- 6C.

[0072] In various embodiments, one or more physical characteristics of the shelf structure(s) may be determined based on the digital image from the at least one camera. For example, the size of the given shelf structure, the number of shelf layers (e.g., rows and / or columns), the shelf layer height, and / or the like may be determined via the digital image from the at least one camera 215.

[0073] Referring now to optional Block 530 of FIG. 5, the method includes determining a location of the transportation arm device based on one or more waypoint fiducial markers in the environment. Similar to the determination of the location of a given object, the system may use markers (e.g., the object identifier marker(s), the shelf identifier marker(s), the waypoint fiducialmarker(s), the auto-teaching fiducial marker(s), magnetic markers, RFID markers, and / or the like) to determine the location of the transportation arm device 200 during movement of the transportation arm device 200.

[0074] In various embodiments, the at least one camera 215 may monitor markers during the movement of the transportation arm device 200, such that the system is capable of determining the location of the transportation arm device 200. For example, the at least one camera 215 may scan a given marker and determine the location of the transportation arm device 200 based on the location of the given marker (e.g., determined via the digital image captured by the at least one camera 215, as discussed in Block 520 above). This capture may analyze the marker to prepare precise positional coordinates based on the image capture.

[0075] In various embodiments, the system may determine an orientation of the digital image from the at least one camera 215 using one or more markers (e.g., the object identifier marker(s), the shelf identifier marker(s), the waypoint fiducial marker(s), the auto-teaching fiducial marker(s), and / or the like). As such, the digital image may be analyzed in either landscape mode or portrait mode. The system, based on the given markers, determines the orientation of the digital image to avoid any issues with moving the transportation arm device 200 (e.g., the orientation is normalized such that the transportation arm device 200 may be moved in the correct direction). In various embodiments, the orientation of the at least one camera 215 may be normalized using one or more markers (e.g., the auto-teaching fiducial marker(s)). For example, the configuration of the markers may be known, such that the orientation of a given digital image can be known. In such an embodiment, the marker(s) used by the system may be fixed.

[0076] Referring now to Block 540 of FIG. 5, the method includes causing a transportation arm device to be moved to the first location of the first object. In various embodiments, the transportation arm device 200 may be automated, such that the transportation arm device 200 moves to the first location in order to engage with the first object.

[0077] In various embodiments, the transportation arm device 200 may be moved to a position adjacent to the first location that allows the at least one camera 215 to verify the object identifier marker on the first object. For example, the object identifier marker may be at a corner of the givenobject, such that the transportation arm device 200 moves to a position in which the at least one camera 215 can process the object identifier marker. As such, the method may also include confirming the first object based on the object identifier marker upon movement of the transportation arm device to the first location of the first object. Upon confirmation of the first object, the transportation arm device 200 may be moved to the first location (e.g., the position to engage with the first object).

[0078] Referring now to Block 550 of FIG. 5, the method includes causing the transportation arm device to engage with the first object. In various embodiments, the gripping mechanism of the transportation arm device (e.g., the gripping finger(s) 205) may be actuated (e.g., gripping finger(s) 205 moved or the transportation arm device 200 is moved such that the gripping finger(s) 205 engage with the object). Engaging with the object may include applying a force on the object such that the object moves with the transportation arm device 200 during movement of the transportation arm device 200.

[0079] Examples of engaging an object (e.g., a first object) may include contacting opposite sides of the given object with gripping fingers such that the object is held by the gripping fingers and / or placing the gripping fingers below the object and providing an upward force on the object via the transportation arm device 200. In various embodiments, engaging with the object may be any instance in which the transportation arm device 200 is in contact with the object such that moving the transportation arm device 200 causes the object to be moved with the transportation arm device 200.

[0080] In various embodiments, the at least one camera may be used to monitor the location of the transportation arm device 200 during movement and / or monitor the engagement of the transportation arm device 200 with the given object. For example, the at least one camera may monitor the first object (or any other objects engaged by the transportation arm device 200) for any movement during transport that may indicate that the engagement is insufficient or otherwise unstable. In such an example, the transportation arm device 200 may be adjusted to ensure an engagement and avoid any accidental drops.

[0081] The location of the transportation arm device 200 may be determined using one or moremarkers (e.g., the object identifier marker(s), the shelf identifier marker(s), the waypoint fiducial marker(s), the auto-teaching fiducial marker(s), and / or the like) within the environment. For example, the transportation arm device 200 (e.g., via the at least one camera 215) may process one or more markers (e.g., the object identifier marker(s), the shelf identifier marker(s), the waypoint fiducial marker(s), the auto-teaching fiducial marker(s), and / or the like) and based on a known location for the given marker, determine a location of the transportation arm device 200. For example, the location of a given marker may be determined during the initial analysis of the environment (e.g., via the digital image captured via the at least one camera 215). The location of various markers may be known from other processes (e.g., some markers, such as the shelf identifier markers may be stationary and consistent across extended periods of use).

[0082] Referring now to optional Block 560 of FIG. 5, the method includes causing a disengagement of the transportation arm device and the first object upon movement of the first object to a destination location. In various embodiments, the transportation arm device 200 may be automated, such that the transportation arm device 200 moves from the first location to a destination location of the first object. As detailed herein, the transportation arm device 200 may monitor the position of the transportation arm device 200 during movement to ensure that the transportation arm device 200 is moving in the direction of the destination location.

[0083] The transportation arm device 200 may disengage with the first object (or any other objects for which the transportation arm device 200 is engaged) in a reverse fashion to the engagement. For example, in an instance in which the transportation arm device 200 engages the first object by causing the gripping finger(s) 205 to contact the object, the transportation arm device 200 may disengage the first object by moving the gripping finger(s) 205 in the opposite direction such that the gripping finger(s) 205 no longer contact the object. Additional movements are applicable, and may comprise the steps of gripping in reverse, noting that the motion algorithms may vary depending on the environment, including the altitude and relative humidity.

[0084] Referring now to optional Block 570 of FIG. 5, the method includes updating the first location of the first object to the destination location. In various embodiments, the location of the object(s) in the environment may be stored (e.g., based on the digital image from the at least onecamera 215). The location of the objects may be used to retrieve objects and / or determine where to deposit objects within the environment. As such, the system may update the location for an object in an instance in which the object is moved. For example, the system may update the first location of the first object to the destination location in which the first object is deposited. The location of each object may be updated in real-time or near real-time, such that the system may determine the location of the objects, as well as occupied and unoccupied locations within the environment.

[0085] Referring now to optional Block 580 of FIG. 5, the method includes causing the transportation arm device to move to a second location of a second object based on the digital image from the at least one camera. As discussed herein, the operations herein may be completed on any number of objects. As such, the system may move any number of objects using the operations herein. In various embodiments, multiple objects may be moved based on the locations determined based on the digital image from the at least one camera 215. For example, the system may update the locations of object upon movement, but otherwise not require any environment digital images in order to move the objects. In various embodiments, the operations herein (e.g., the determination of locations of various objects) may be repeated periodically (e.g., based on time or number of objects moved) and / or in an instance an error occurs (e.g., an instance in which an object is not in the correct location stored by the system). Further, multiple gripping elements may be applied to a single operation, such as gripping a series of vials, trays or cassettes, in this fashion one camera may enable the gripping and movement of a plurality of objects.

[0086] Referring now to FIGs. 6A-6C, example fiducial markers used for the various markers discussed herein are shown. The fiducial markers of FIGs. 6A-6C may be used for any number of fiducial markers discussed herein, such as object identifier marker(s), shelf identifier marker(s), waypoint fiducial marker(s), auto-teaching fiducial marker(s), and / or the like. The markers may include various shape, patterns, and / or the like. In various embodiments, the markers may be monocolor (e.g., the entire marker may be black or any other color). Alternatively, the marker may have multiple colors (e.g., the system may be able to differentiate colors and use the different colors to identity the marker).

[0087] FIG. 6A illustrates an example marker 600. The fiducial marker may include some or all of the components shown in FIG. 6A. For example, the marker 600 includes a textual information marker 605 (“fast”), graphical markers (e.g., sub-markers 610A, 610B, 610C), numerical markers 615 (e.g., “1”), and / or the like. In various embodiments, the fiducial marker may be any individual component (e.g., textual information marker 605, graphical markers, or numerical marker 615) and not include the other components. For example, only graphical markers may be used instead of textual information and / or numerical markers. In various embodiments, any number of different components may be used. In various embodiments, the different components may be associated with one another. For example, the numerical marker of FIG. 6A corresponds to each of the graphical markers (e.g., sub-markers 610A, 610B, 610C). As such, different graphical markers may be relating to different numbers (e.g., the markers of FIG 6B is associated with the number five)

[0088] Any number of sub-markers (graphical markers) may be used in a fiducial marker. For example, the fiducial marker may include one sub-marker (e.g., the sub-marker may be the entirety of the fiducial marker and / or other types of markers may be used with a single sub-marker), two sub-markers (e.g. as shown in FIG. 6B), three sub-markers (as shown in FIGs. 6A and 6C), and / or any number of sub-markers.

[0089] The sub-marker be associated any object and / or position of the structure. In various embodiments, sub-marker may be identified based on the shape of the sub-marker (e.g., circular, square, etc.) and / or the graphical pattern. As shown in FIG. 6A, the sub-markers 610A and 610C include a square with white dots within creating a pattern with a circle around the square, while the sub-marker 610B includes a black square with white dots within creating a pattern that is different than the pattern of sub-markers 610A and 610C.

[0090] The different sub-markers on each fiducial marker may be analyzed together and / or individually for identification as discussed herein. For example, in some instances each sub-marker may include the same information (e.g., scanning one of the sub-markers may result in the same information, while in other instances, each sub-marker may include different information. As such, in various embodiments, the scanning of each sub-marker may also provide information (e.g., ascan of only one sub-marker in an instance in which the fiducial marker has more than one submarker may indicate incorrect alignment). As such, the combination of sub-markers may be used to determine alignment.

[0091] FIG. 6B is another example fiducial marker 625 that include a textual information marker 605 (“FLO”), graphical markers (e.g., sub-markers 610A, 610C), and numerical markers 615 (e.g., “5”). Compared to the fiducial marker 600 of FIG. 6A, the fiducial marker has different values for each component (e.g., different textual information marker value, different sub-marker patterns, different numerical marker values). As such, the fiducial markers may be differentiated (e.g., the system may determine position within the environment based on the scanning of different fiducial markers). The lack of sub-markers or other components may also be used to determine information. For example, the fiducial marker 625 of FIG. 6B does not include the second submarker that is included in the fiducial marker 600 of FIG. 6A (e.g., sub-marker 610B). The spacing of the sub-markers may be used to determine information relating to positioning.

[0092] FIG. 6C is still another example fiducial marker 650 that includes a textual information marker (“Fast”), graphical markers (e.g., sub-markers 605A, 605B, 605C), and a numerical marker (“2”). The position of the components within the fiducial marker 650 is different than the fiducial markers of FIGs. 6A and 6B. In various embodiments, the positioning of each component may be used for identification. The positioning of each component may be uniform across different types of markers (e.g., the system may expect a certain template of the components in order to analyze the markers). Sub-marker 610A and 610B includes multiple concentric circles. The spacing of the circles, number of circles, thickness of the circles, and / or the like may be used for identification, similar to the patterns shown in sub-marker 610B. During processing, the entirety of the circle may be captured via the camera (e.g., a stereo camera) and the transitions from dark to light may be used to determine information, such position. For example, the precises center of a marker may be determined and the three dimensional position of the transportation arm device may be determined via a stereo camera.

[0093] The systems discussed herein may be used with any of the embodiments discussed in reference to related PCT applications filed concurrently with the present disclosure. As such, the 1PCT applications titled “OBJECT TRANSPORTATION AND GRIPPING DEVICE” and “EXTENSIBLE VERTICALLY -EFFICIENT LABORATORY AUTOMATION SYSTEM” filed on May 20, 2025, are hereby incorporated by reference. The present disclosure will be updated to include application numbers after filing.

[0094] It should be emphasized that the above-described embodiments of the present . disclosure are merely possible examples of implementations set forth for a clear understanding of the principles of the disclosure. Many variations and modifications may be made to the abovedescribed embodiment(s) without departing substantially from the spirit and principles of the disclosure. All such modifications and variations are intended to be included herein within the scope of this disclosure and protected by the following claims.IV. Claim Clauses

[0095] Clause 1. An automated alignment object transfer system, comprising: a transportation arm device; at least one camera; and a controller comprising at least one processing device coupled to a memory device, wherein the controller is configured to: position the at least one camera at a position in which the at least one camera is capable of capturing a digital image of an environment, wherein the environment comprises at least one shelf structure and at least one object stored within the at least one shelf structure; determine, based on the digital image from the at least one camera, a first location of a first object of the at least one object stored within the at least one shelf structure, wherein the first location of the first object is determined based on an object identifier marker; cause the transportation arm device to be moved to the first location of the first object; and cause the transportation arm device to engage with the first object.

[0096] Clause 2. The system of Clause 1, wherein the controller is further configured to cause a disengagement of the transportation arm device and the first object upon movement of the first object to a destination location.

[0097] Clause 3. The system of Clause 2, wherein the controller is further configured to update the first location of the first object to the destination location.

[0098] Clause 4. The system of Clause 1, wherein the object identifier marker may be captured by the at least one camera.

[0099] Clause 5. The system of Clause 1, wherein the controller is further configured to determine a location of the transportation arm device based on one or more waypoint fiducial markers in the environment.

[0100] Clause 6. The system of Clause 1, wherein the at least one camera is positioned on the transportation arm device.

[0101] Clause 7. The system of Clause 1, wherein the controller is further configured to confirm the first object based on the object identifier marker upon movement of the transportation arm device to the first location of the first object.

[0102] Clause 8. The system of Clause 1, wherein the controller is further configured to determine, using one or more auto-teaching fiducial markers, an orientation of the digital image, wherein the digital image is analyzed in either landscape or portrait mode.

[0103] Clause 9. The system of Clause 1, wherein the controller is further configured to determine a size of the at least one shelf structure based on at least one shelf identifier marker.

[0104] Clause 10. The system of Clause 1 , wherein the controller is further configured to cause the transportation arm device to move to a second location of a second object based on the digital image from the at least one camera.

[0105] Clause 11. A method of operating an automated alignment object transfer system, comprising: positioning at least one camera at a position in which the at least one camera is capable of capturing a digital image of an environment, wherein the environment comprises at least one shelf structure and at least one object stored within the at least one shelf structure; determining, based on the digital image from the at least one camera, a first location of a first object of the at least one object stored within the at least one shelf structure, wherein the first location of the first object is determined based on an object identifier marker; causing a transportation arm device to be moved to the first location of the first object; and causing the transportation arm device to engage with the first object.

[0106] Clause 12. The method of Clause 11, further comprising causing a disengagement of the transportation arm device and the first object upon movement of the first object to a destination location.

[0107] Clause 13. The method of Clause 12, further comprising updating the first location of the first object to the destination location.

[0108] Clause 14. The method of Clause 11, wherein the object identifier marker may be captured by the at least one camera.

[0109] Clause 15. The method of Clause 11, further comprising determining a location of the transportation arm device based on one or more waypoint fiducial markers in the environment.

[0110] Clause 16. The method of Clause 11, wherein the at least one camera is positioned on the transportation arm device.

[0111] Clause 17. The method of Clause 11, further comprising confirming the first object based on the object identifier marker upon movement of the transportation arm device to the first location of the first object.

[0112] Clause 18. The method of Clause 11, further comprising determining, using one or more auto-teaching fiducial markers, an orientation of the digital image, wherein the digital image is analyzed in either landscape or portrait mode.

[0113] Clause 19. The method of Clause 11, further comprising determining a size of the at least one shelf structure based on at least one shelf identifier marker.

[0114] Clause 20. The method of Clause 11 , further comprising causing the transportation arm device to move to a second location of a second object based on the digital image from the at least one camera.

Claims

CLAIMSTherefore, the following is claimed:

1. An automated alignment object transfer system, comprising: a transportation arm device; at least one camera; and a controller comprising at least one processing device coupled to a memory device, wherein the controller is configured to: position the at least one camera at a position in which the at least one camera is capable of capturing a digital image of an environment, wherein the environment comprises at least one shelf structure and at least one object stored within the at least one shelf structure; determine, based on the digital image from the at least one camera, a first location of a first object of the at least one object stored within the at least one shelf structure, wherein the first location of the first object is determined based on an object identifier marker; cause the transportation arm device to be moved to the first location of the first object; and cause the transportation arm device to engage with the first object.

2. The system of Claim 1, wherein the controller is further configured to cause a disengagement of the transportation arm device and the first object upon movement of the first object to a destination location.

3. The system of Claim 2, wherein the controller is further configured to update the first location of the first object to the destination location.

4. The system of Claim 1, wherein the object identifier marker may be captured by the at least one camera.

5. The system of Claim 1, wherein the controller is further configured to determine a location of the transportation arm device based on one or more waypoint fiducial markers in the environment.

6. The system of Claim 1, wherein the at least one camera is positioned on the transportation arm device.

7. The system of Claim 1 , wherein the controller is further configured to confirm the first object based on the object identifier marker upon movement of the transportation arm device to the first location of the first object.

8. The system of Claim 1, wherein the controller is further configured to determine, using one or more auto-teaching fiducial markers, an orientation of the digital image, wherein the digital image is analyzed in either landscape or portrait mode.

9. The system of Claim 1, wherein the controller is further configured to determine a size of the at least one shelf structure based on at least one shelf identifier marker.

10. The system of Claim 1, wherein the controller is further configured to cause the transportation arm device to move to a second location of a second object based on the digital image from the at least one camera.

11. A method of operating an automated alignment object transfer system, comprising: positioning at least one camera at a position in which the at least one camera is capable of capturing a digital image of an environment, wherein the environment comprises at least one shelf structure and at least one object stored within the at least one shelf structure;determining, based on the digital image from the at least one camera, a first location of a first object of the at least one object stored within the at least one shelf structure, wherein the first location of the first object is determined based on an object identifier marker; causing a transportation arm device to be moved to the first location of the first object; and causing the transportation arm device to engage with the first object.

12. The method of Claim 11, further comprising causing a disengagement of the transportation arm device and the first object upon movement of the first object to a destination location.

13. The method of Claim 12, further comprising updating the first location of the first object to the destination location.

14. The method of Claim 11, wherein the object identifier marker may be captured by the at least one camera.

15. The method of Claim 11, further comprising determining a location of the transportation arm device based on one or more waypoint fiducial markers in the environment.

16. The method of Claim 11, wherein the at least one camera is positioned on the transportation arm device.

17. The method of Claim 11, further comprising confirming the first object based on the object identifier marker upon movement of the transportation arm device to the first location of the first object.

18. The method of Claim 11, further comprising determining, using one or more autoteaching fiducial markers, an orientation of the digital image, wherein the digital image is analyzed in either landscape or portrait mode.

19. The method of Claim 11, further comprising determining a size of the at least one shelf structure based on at least one shelf identifier marker.

20. The method of Claim 11, further comprising causing the transportation arm device to move to a second location of a second object based on the digital image from the at least one camera.

Images