Sensing boom for integrated mobile robotic hand

Abstract

A perception boom for a mobile robot is provided. The mobile robot includes a mobile base, a turntable operably coupled to the mobile base and configured to rotate about a first axis, an arm operably coupled to a first location on the turntable, and a perception boom operably coupled to a second location on the turntable, the perception boom configured to rotate about a second axis parallel to the first axis, wherein the perception boom includes a first perception module and a second perception module arranged thereon, the first perception module and the second perception module arranged between the first imaging module and the turntable.

Description

Background Technology

[0001] A robot is generally defined as a reprogrammable, multi-functional manipulator designed to perform tasks by moving materials, parts, tools, or specialized equipment through variable, programmable movements. A robot can be a physically anchored manipulator (such as an industrial robotic arm), a mobile robot that moves through an environment (e.g., using legs, wheels, or traction-based mechanisms), or some combination of manipulator and mobile robot. Robots are used in a wide range of industries, including manufacturing, warehousing and logistics, transportation, hazardous environments, exploration, and healthcare. Summary of the Invention

[0002] Some embodiments relate to a mobile robot including a mobile base, a turntable operably coupled to the mobile base and configured to rotate about a first axis, an arm operably coupled to a first position on the turntable, and a sensing rod operably coupled to a second position on the turntable, the sensing rod being configured to rotate about a second axis parallel to the first axis, wherein the sensing rod includes a first sensing module and a second sensing module disposed thereon, the first sensing module and the second sensing module being disposed between a first imaging module and the turntable.

[0003] In one aspect, the mobile robot further includes control circuitry configured to control the rotation of a sensing rod based at least in part on the rotation of a turntable and the distance between a first position and a second position. In another aspect, each of the first and second sensing modules includes a two-dimensional (2D) color camera, a depth sensor, and at least one light source. In another aspect, the depth sensor includes a time-of-flight (TOF) camera. In another aspect, the depth sensor includes a stereo camera. In another aspect, the two-dimensional camera includes a red-green-blue (RGB) monocular camera. In another aspect, the first and second sensing modules are arranged along the same side of the sensing rod. In another aspect, the first and second sensing modules are arranged such that the distance between a first imaging module and a second imaging module along the same side of the sensing rod is maximized. In another aspect, the fields of view of the first and second sensing modules do not overlap. In another aspect, the sensing rod includes a plurality of sides, including a first side identical to the side on which the first and second sensing modules are arranged, and the sensing rod further includes a third sensing module arranged on a side of the sensing rod different from the first side. In another aspect, the control circuit is also configured to control the operation of the arm based at least in part on the output of the first sensing module and / or the second sensing module.

[0004] In one aspect, the second position is located at the outer edge of the turntable. In another aspect, the mobile robot further includes control circuitry configured to simultaneously control the rotation of the sensing rod and the rotation of the turntable in opposite directions. In another aspect, the sensing rod further includes at least one antenna disposed thereon, configured to receive signals from a system external to the mobile robot. In another aspect, the sensing rod further includes a scanner configured to scan identifier tags on objects attached to the mobile robot's environment. In another aspect, the mobile robot further includes control circuitry configured to control the operation of a first sensing module and a second sensing module to capture one or more images while the mobile robot is moving. In another aspect, the control circuitry is also configured to control the operation of the mobile robot at least in part based on one or more images captured by the first sensing module and / or the second sensing module while the mobile robot is moving. In another aspect, the operation of the mobile robot includes the direction of travel of the mobile robot. In another aspect, the mobile robot further includes at least one camera disposed on the arm. In another aspect, the first sensing module is oriented at a first angle relative to the sensing rod, the second sensing module is oriented at a second angle relative to the sensing rod, and the first angle and the second angle are different.

[0005] Some embodiments relate to a method for capturing one or more images using a mobile robot. The method includes: controlling a sensing rod to rotate in a second direction opposite to the first direction while a turntable of the mobile robot rotates along a first direction, the sensing rod being operatively coupled to the turntable and having a plurality of sensing modules arranged on the sensing rod; and capturing one or more images through the plurality of sensing modules during rotation of the turntable and the sensing rod.

[0006] In one aspect, capturing one or more images includes capturing one or more images during movement of a movable base operatively coupled to the turntable.

[0007] It should be understood that the foregoing concepts, as well as other concepts discussed below, can be arranged in any suitable combination, as this disclosure is not limited in this respect. Furthermore, other advantages and novel features of this disclosure will become apparent from the following detailed description of various non-limiting embodiments when considered in conjunction with the accompanying drawings. Attached Figure Description

[0008] The accompanying drawings are not necessarily drawn to scale. In the drawings, each identical or nearly identical component shown in different figures may be indicated by similar reference numerals. For clarity, not every component is labeled in every drawing. In the drawings:

[0009] Figure 1A This is a perspective view of one embodiment of the robot;

[0010] Figure 1B yes Figure 1A Another perspective view of the robot;

[0011] Figure 2A The image shows a robot performing a task in a warehouse environment.

[0012] Figure 2B A robot is shown unloading boxes from a truck;

[0013] Figure 2C A robot is shown assembling pallets in a warehouse aisle;

[0014] Figure 3A This is a perspective view of one embodiment of the robot;

[0015] Figure 3B yes Figure 3A Another perspective view of the robot;

[0016] Figure 4 This is a top view schematic diagram of one embodiment of the robot's turntable; and

[0017] Figure 5 It is one example of a computer system that can be used to implement some of the embodiments. Detailed Implementation

[0018] Robots are typically configured to perform a variety of tasks in their environment. These tasks often involve interacting with objects and / or environmental elements. Notably, robots are gaining popularity in warehousing and logistics operations. Before the introduction of robots into these spaces, many tasks were performed manually. For example, one person manually unloads boxes from a truck onto one end of a conveyor belt, while another person at the other end organizes these boxes onto pallets. Then, a forklift operated by a third person picks up the pallet, drives the forklift to the warehouse's storage area, and places the pallet down. A fourth person removes the individual boxes from the pallet and places them on shelves in the storage area. Recently, robotic solutions have been developed to automate many of these functions. Such robots can be specialized robots (i.e., designed to perform a single task or a few closely related tasks) or general-purpose robots (i.e., designed to perform multiple tasks). To date, both specialized and general-purpose warehousing robots have significant limitations, as described below.

[0019] Specialized robots may be designed to perform a single task, such as unloading boxes from a truck onto a conveyor belt. While such specialized robots may be highly efficient at performing their designated tasks, they may lack the ability to perform other unrelated tasks. Therefore, a human or a separate robot (e.g., another specialized robot designed for a different task) may be needed to perform the next task in the sequence. Consequently, warehouses may need to invest in purchasing multiple specialized robots to perform a range of tasks, or they may need to rely on hybrid operations, in which there are often frequent robot-to-human or human-to-robot object transfers.

[0020] In contrast, general-purpose robots can be designed to perform a wide variety of tasks and are capable of handling boxes for most of their lifecycle, from truck to shelf (e.g., unloading, palletizing, transporting, depalletizing, storing). While such general-purpose robots can perform multiple tasks, they may not perform these tasks efficiently or with sufficient precision to guarantee successful integration into highly streamlined warehouse operations. For example, while mounting an off-the-shelf manipulator onto an off-the-shelf mobile robot might create a system theoretically capable of performing many warehouse tasks, this loosely integrated system may fail to perform the complex or dynamic movements requiring coordination between the manipulator and the mobile base, resulting in an inefficient and inflexible combined system. Typical operations of such a system in a warehouse environment may involve the mobile base and manipulator operating sequentially and (partially or completely) independently of each other. For example, in the event of a power outage to the manipulator, the mobile base might initially move towards a pile of boxes. The mobile base could stop upon reaching the pile, and while the base remains stationary, the manipulator could be powered on and begin manipulating the boxes. After the manipulator task is completed, the power can be cut off again, and the mobile base can move to another destination to perform the next task. As understood above, the mobile base and manipulator in this system are essentially two separate robots combined; therefore, the controller associated with the manipulator does not need to be configured to share information, send commands to, or receive commands from the separate controller associated with the mobile base. Consequently, when two separate controllers are trying to work together, this poorly integrated mobile manipulator robot may be forced to operate its manipulator and base at suboptimal speeds or along suboptimal trajectories. Furthermore, while there are some limitations from a purely engineering perspective, additional constraints must be imposed to comply with safety regulations. For example, if safety regulations require the mobile manipulator to be able to completely shut down within a certain time when a person enters an area within a certain distance of the robot, then a loosely integrated mobile manipulator robot may not be able to act quickly enough to ensure that the manipulator and mobile base do not pose a threat to personnel (individually and collectively). To ensure that such loosely integrated systems operate within necessary safety constraints, they must operate at slower speeds or follow more conservative trajectories than those already constrained by engineering problems. Therefore, the speed and efficiency of general-purpose robots performing tasks in warehouse environments have been limited to date.

[0021] In view of the above-mentioned problems, the inventors have recognized and realized that highly integrated mobile manipulator robots with system-level mechanical design and overall control strategy between the manipulator and the mobile base may be associated with certain benefits in warehousing and / or logistics operations. Such integrated mobile manipulator robots are capable of performing complex and / or dynamic movements that are impossible with conventional, loosely integrated mobile manipulator systems. Therefore, this type of robot may be well-suited for performing a variety of different tasks quickly, agilely, and efficiently (e.g., in warehouse environments).

[0022] Exemplary Robot Overview

[0023] This section outlines some components of one embodiment of a highly integrated mobile manipulator robot configured to perform various tasks, to explain the interactions and interdependencies of the robot's various subsystems. Each of the various subsystems and the control strategies used to operate them will be described in further detail in the following sections.

[0024] Figure 1A and 1B This is a perspective view of one embodiment of robot 100. Robot 100 includes a mobile base 110 and a robotic arm 130. The mobile base 110 includes an omnidirectional drive system that enables the mobile base to translate in any direction in a horizontal plane and rotate about a vertical axis perpendicular to that plane. Each wheel 112 of the mobile base 110 can be independently steered and driven. The mobile base 110 also includes multiple distance sensors 116 to assist the robot 100 in moving safely in its environment. The robotic arm 130 is a 6-DOF robotic arm, including three pitch joints and a 3-DOF wrist. An end effector 150 is disposed at the distal end of the robotic arm 130. The robotic arm 130 is operatively coupled to the mobile base 110 via a turntable 120 configured to rotate relative to the mobile base 110. In addition to the robotic arm 130, a sensing rod 140 is also coupled to a turntable 120, such that rotation of the turntable 120 relative to the movable base 110 causes rotation of both the robotic arm 130 and the sensing rod 140. The robotic arm 130 is kinematically constrained to avoid collision with the sensing rod 140. The sensing rod 140 is also configured to rotate relative to the turntable 120 and includes multiple sensing modules 142 configured to collect information about one or more objects in the robot's environment. The integrated structure and system-level design of the robot 100 enable rapid and efficient operation in many different applications, some of which will be illustrated below.

[0025] Figure 2ARobots 10a, 10b, and 10c are shown performing different tasks in a warehouse environment. The first robot 10a, located inside a truck (or container), moves boxes 11 from stacks inside the truck onto a conveyor belt 12 (this specific task will be referred to below). Figure 2B (To be discussed in more detail). At the opposite end of conveyor belt 12, the second robot 10b organizes boxes 11 onto pallets 13. In a separate area of ​​the warehouse, the third robot 10c picks boxes from the shelves to create orders on pallets (this specific task will be referred to below). Figure 2C (To be discussed in more detail). It should be understood that robots 10a, 10b, and 10c are different instances of the same type of robot (or highly similar robots). Therefore, the robots described herein can be understood as specialized multi-purpose robots, as they are designed to perform specific tasks precisely and efficiently, but are not limited to a single or a few specific tasks.

[0026] Figure 2B The diagram shows robot 20a unloading boxes 21 from truck 29 and placing them on conveyor belt 22. In this box-picking application (and in other box-picking applications), robot 20a repeatedly picks up boxes, rotates, places boxes, and rotates back to pick up the next box. Although Figure 2B Robot 20a is with Figure 1A and 1B 100 different embodiments of the robot, but reference is made to... Figure 1A and 1B The components of the robot 100 as indicated in the diagram are easy to explain. Figure 2B The operation of robot 20a. During operation, the sensing lever of robot 20a (similar to...) Figure 1A and 1BThe sensing rod 140 of robot 100 can be configured to rotate independently of the rotation of the turntable (similar to turntable 120) on which the sensing rod is mounted, allowing the sensing module (similar to sensing module 142) mounted on the sensing rod to capture an image of the environment. This enables robot 20a to plan its next movement while performing its current movement. For example, when robot 20a picks up the first box from a pile of boxes in truck 29, the sensing module on the sensing rod can point to the location where the first box will be placed (e.g., conveyor belt 22) and collect information about that location. Then, after the turntable rotates, as robot 20a places the first box on the conveyor belt, the sensing rod can rotate (relative to the turntable), so that the sensing module on the sensing rod points to the pile of boxes and collects information about the pile of boxes, which is used to determine the second box to be picked up. As the turntable rotates back to allow the robot to pick up the second box, the sensing rod can collect updated information about the area around the conveyor belt. In this way, robot 20a can perform tasks in parallel that would otherwise have to be performed sequentially, resulting in faster and more efficient operation.

[0027] exist Figure 2B It is also noteworthy that robot 20a works alongside humans (e.g., workers 27a and 27b). Given that robot 20a is configured to perform many tasks traditionally performed by humans, it is designed to have a small footprint, enabling it to access areas designed for human access, and minimizing the size of the safety zone around the robot that prevents human access.

[0028] Figure 2C A robot 30a is shown performing an order fulfillment task, in which it places a box 31 onto a pallet 33. Figure 2C In this example, pallet 33 is positioned above the head of autonomous mobile robot (AMR) 34; however, it should be understood that the capabilities of robot 30a illustrated in this example are adapted to construct pallets independent of AMRs. In this task, robot 30a picks up boxes 31 positioned above, below, or inside shelves 35 in the warehouse and places them on pallet 33. The position and orientation of some boxes relative to the shelves may require different box-picking strategies. For example, the robot can simply pick up a box located on a low shelf by gripping its top surface with the end effector of its robotic arm (thus performing "top picking"). However, if the box to be picked up is at the top of a stack of boxes, and the gap between the top of the box and the bottom of a horizontal shelf partition is limited, the robot may choose to pick up the box by gripping its side (thus performing "side picking").

[0029] To pick up boxes in a confined environment, a robot may need to carefully adjust the orientation of its arm to avoid touching other boxes or surrounding shelves. For example, in a typical "keyhole problem," a robot may only be able to approach a target box by guiding its arm through a small space or confined area (similar to a keyhole) defined by other boxes or surrounding shelves. In this case, coordination between the mobile base and the robot's arm can be beneficial. For instance, the ability to translate the base in any direction allows the robot to position itself as close to the shelf as possible, effectively extending the length of its arm (compared to conventional robots without omnidirectional actuation that may not be able to arbitrarily navigate to the shelf). Furthermore, the ability to translate the base backward allows the robot to retract its arm from the shelf after picking up the box without having to adjust joint angles (or to minimize the degree of joint angle adjustment), thus providing a simple solution to many keyhole problems.

[0030] Of course, it should be understood that, Figure 2A-2C The tasks illustrated are merely a few examples of applications that can utilize integrated mobile robotic arms, and this disclosure is not limited to robots configured to perform only these specific tasks. For example, the robots described herein can be adapted to perform a wide variety of tasks, including but not limited to unloading objects from trucks or containers, placing objects on conveyor belts, removing objects from conveyor belts, organizing objects into piles, organizing objects on pallets, placing objects on shelves, organizing objects on shelves, removing objects from shelves, picking up objects from the top (e.g., performing "top pick"), picking up objects from the side (e.g., performing "side pick"), coordinating with other mobile robotic arms, coordinating with other warehouse robots (e.g., coordinating with AMRs), coordinating with humans, and many other tasks.

[0031] Exemplary turntable and sensing lever

[0032] As mentioned above (for example, refer to...) Figure 1A and 1B The robotic arm of an integrated mobile manipulator robot can be coupled to a mobile base via a turntable. The turntable can rotate the robotic arm relative to the mobile base about a vertical axis (e.g., a deflection axis). In some embodiments, a sensing rod can also be coupled to the turntable, such that rotation of the turntable causes rotation (e.g., deflection) of both the robotic arm and the sensing rod. In some embodiments, the sensing rod may include additional degrees of freedom that allow rotation (e.g., deflection) of the sensing rod relative to the turntable. As described above (e.g., refer to...) Figure 2BThis independent control of the sensing stick relative to the turntable allows the robot to manipulate objects in the first area (e.g., by using an additional actuator on the sensing stick to point its orientation sensor towards the second area) while simultaneously using the robotic arm to manipulate objects in the first area (e.g., by repositioning the robotic arm using the turntable while it grasps an object). The rotational control of the sensing stick, independent of the turntable's rotation, also provides other advantages, which are further detailed below. The coordination between the turntable, the sensing stick, and other components of the robot is described below.

[0033] Please refer to Figure 1A and 1B As described above, robot 100 includes a mobile base 110, a turntable 120, a robotic arm 130 (with an end effector 150), and a sensing rod 140. The sensing rod 140 is implemented as a structural support coupled to a horizontal surface of the robot (e.g., the turntable 120) and includes a plurality of sensing modules 142 disposed thereon. In a box-picking application, robot 100 repeatedly picks up boxes, rotates, places boxes, and rotates back to pick up the next box. The sensing rod 140 can be configured to rotate independently of the rotation of the turntable 120 on which it is mounted, allowing cameras included in the sensing modules 142 to capture images of the environment. This enables robot 100 to plan its next move while performing its current move. For example, while robot 100 is picking up the first box, the sensing modules 142 on the sensing rod 140 can point to the location where the first box will be placed (e.g., a pallet, conveyor belt) and collect information about that location. Then, as robot 100 places the first box, sensing rod 140 can rotate so that sensing module 142 on sensing rod 140 points to the pile of boxes and collects information about the second box to be picked up. In this way, robot 100 can perform tasks in parallel that would otherwise have to be performed sequentially, thereby achieving faster and more efficient task execution.

[0034] It should be understood that capturing images of the robot's environment includes not only capturing images of the task the robot is performing (e.g., a pile of boxes or an area surrounding a pallet or conveyor belt), but also capturing images of the robot's environment that ensures the robot's safe operation. For example, when the robot is operating inside a container (e.g., a truck) to unload boxes or other objects from the container, the sensing module 142 arranged on the sensing rod 140 can be configured to image the walls and ceiling of the container to ensure that robot components (e.g., the mobile base and / or the robot arm) can operate safely and efficiently within the container.

[0035] In the above example, the sensing lever is moved away from the position where the robotic arm is operating to facilitate planning the next movement of the robotic arm while it is performing its current movement (e.g., planning the movement one frame in advance). However, the inventors have recognized that, in some cases, capturing images of the robotic arm itself can also be advantageous. For example, capturing images of the robotic arm while performing a calibration sequence can be useful for calibrating the robotic arm and / or one or more components of the sensing modules arranged on the sensing lever. Furthermore, capturing information about an object (e.g., a box) that the robotic arm has picked up can be used to determine one or more features of that object, which can help predict the robot's future actions. For example, the size of the box picked up by the robotic arm may help plan how to pick up a box of similar size from a pile of boxes being unloaded from a container (e.g., a truck). In some embodiments, the two-dimensional camera can be configured to capture visual identifiers (e.g., barcodes, QR codes) located on the object (e.g., a box) that the robotic arm is picking up. This visual identifier information can be used to identify the contents of the box, the manufacturer associated with the contents of the box, and any other information that can be used to understand the operation of the robot.

[0036] Figure 3A and 3B This is a perspective view of a robot 200 designed according to some embodiments. The robot 200 includes a mobile base 210 and a turntable 220 rotatably coupled to the mobile base. A robot arm 230 is operatively coupled to the turntable 220, as is a sensing stick 240. The sensing stick 240 includes an actuator 225 configured to allow rotation of the sensing stick relative to the turntable 220 and / or the mobile base 210. In some embodiments, the sensing stick 240 may include components not directly related to sensing the robot's environment, including but not limited to one or more communication systems and safety lights (e.g., ...). Figure 3B (Lamp 260 shown). For example, in some embodiments, the sensing stick 240 includes a communication module disposed at the top of the sensing stick. The communication module may include one or more antennas configured to receive signals from one or more systems (e.g., building safety systems, warehouse management systems) outside the mobile robot. In some embodiments, the sensing stick 240 also includes a scanner (e.g., a barcode reader, QR code reader) configured to scan one or more visual identifiers (e.g., visual markers) in the environment.

[0037] As shown in the figure, the sensing rod 240 includes a plurality of sensing modules 242 arranged vertically along the sensing rod. Each sensing module 242 includes a two-dimensional (2D) camera and a depth sensor. For example, the upper sensing module 242 includes an upper 2D camera 244A and an upper depth sensor 250A. The 2D camera and depth sensor included in the sensing module can be arranged in any suitable manner. For example, although the upper depth sensor 250A is shown as being arranged below the upper 2D camera 244A, it should be understood that the upper depth sensor 250A may also be arranged above the upper 2D camera 244A, or at the same height along the sensing rod 240 as the 2D camera 244A. In some embodiments, one or more of the sensing modules 242 may include only a 2D camera (e.g., an RGB camera or a monochrome camera) without a depth sensor, or only a 3D camera without a separate depth sensor, as this disclosure is not limited in this respect.

[0038] As shown in the figure, the sensing rod 240 also includes a lower sensing module comprising a lower 2D camera 244B and a lower depth sensor 250A. The lower sensing module is arranged along the same side of the sensing rod 240 as the upper sensing module and is located between the upper sensing module 242 and the actuator 255. The inventors have recognized that positioning multiple sensing modules at different locations on the sensing rod 240 (e.g., near the top and bottom of the sensing rod) provides the robot 200 with imaging capabilities that would be impossible with only a single sensing module. For example, the sensor within the upper sensing module can have a different field of view that does not overlap (or partially overlaps) with the field of view of the sensor within the lower sensing module, such that the combined field of view of the two sensing modules is larger than the field of view of each individual sensing module. This expanded field of view can be used to image a large number of boxes or other objects in the environment with which the robot will interact. In some embodiments, the vertical distance between the multiple sensing modules along the sensing rod is maximized to provide a wide vertical field of view. Furthermore, the image captured by the sensor of one sensing module can include features of objects in the environment that cannot be well captured by the sensor of another sensing module. For example, compared to the sensors in the lower sensing module, the sensors in the upper sensing module can capture more details about the features of objects in the environment at the same or similar height as the upper sensing module, while the sensors in the lower sensing module can capture one or more features of the same object, but at a larger angle. As another example, compared to the sensors in the upper sensing module, the sensors in the lower sensing module can capture more details about objects located near the robot's mobile base.

[0039] The inventors have recognized and realized that it is advantageous to maximize the height of the sensing rod while still allowing the mobile robot to fit into most spaces where it is expected to operate (e.g., unloading boxes inside containers, such as trucks). The vertical height of at least one of the sensing modules along the sensing rod may be important for certain sensing tasks, such as detecting the surface of an object that a robotic arm will interact with. For example, the detection of the box surface may be more accurate if the image captured by the sensing module is captured at a lower angle of incidence relative to the box surface. Furthermore, further spacing the sensing modules helps detect features of objects that may be occluded or partially occluded by one of the sensing modules. For example, the lower sensing module may have a field of view that enables it to capture images under low shelves in a warehouse, an area that may be at least partially occluded by a higher sensing module located on the sensing rod.

[0040] The sensing module 242 used in some embodiments may include one or more light sources (e.g., flash-based light sources) configured to provide active illumination to the environment during image capture. Some embodiments include one or more visible light sources arranged near the 2D camera to provide ambient illumination during image capture by the 2D camera. Examples of 2D cameras that may be used in some embodiments include, but are not limited to, red-green-blue (RGB) cameras, monochrome cameras, prism cameras, or any other type of 2D camera configured to capture a 2D image of the environment.

[0041] The one or more light sources can be configured to provide active illumination to the environment during image capture by the components of the sensing module. The light sources can include any suitable light-emitting element, including but not limited to light-emitting diodes (LEDs). In some embodiments, each sensing module includes two visible light sources arranged to at least partially surround a two-dimensional camera. This orientation is advantageous in ensuring that objects in the environment are uniformly illuminated and minimizing shadows during image capture by the two-dimensional camera. However, it should be understood that any suitable number of light sources arranged in any suitable manner can be used, and this disclosure is not limited in this respect. In some embodiments, each camera included in the sensing module has at least one associated light source. For example, a time-of-flight (TOF) camera for capturing depth information can be associated with one or more laser emitters to provide active ambient illumination. In embodiments using a stereo camera for depth sensing, a texture projector can be included in the sensing module to enable the stereo camera to operate in low-light conditions.

[0042] The inventors have recognized that including one or more onboard light sources enables mobile robotic arms to capture images of their environment without requiring adequate levels of ambient light in the area where the robot operates. Providing onboard light sources also helps eliminate shadows that may be present in the environment. This is particularly important for tasks where the robot is located inside a container (such as a truck), where there is typically not much ambient lighting.

[0043] As described above, the perception module may also include a depth sensor configured to capture depth information relating to objects in the environment. Examples of depth sensors include, but are not limited to, stereo cameras, time-of-flight (TOF) cameras, LiDAR, or any other depth sensor configured to capture depth information about the environment. In one embodiment, each perception module 242 includes two LED-based light sources, an RGB monocular camera, and a time-of-flight (TOF) camera. As described above, the arrangement of specific components within the perception module is not limiting, and these components can be arranged in any suitable manner. Preferably, the two-dimensional camera and the depth sensor are arranged to provide similar fields of view, which facilitates registration of information captured by the two-dimensional camera and the depth sensor.

[0044] In some embodiments, each of the at least one light source, the two-dimensional camera, and the depth sensor within the sensing module is electrically coupled to a control circuit configured to control the timing of operation of the respective components. For example, the sensing module may include hardware control circuitry electrically coupled to one or more components within the sensing module to achieve individual control of each component based on electrical signals provided by the control circuitry. In some embodiments, multiple components in the sensing module may be electrically connected to each other, such that a triggering operation of one component automatically triggers the operation of another component electrically connected to it, without having to separately send signals from the control circuitry to another component to control its operation.

[0045] In some embodiments, the sensing rod 240 may include control circuitry configured to control the timing of operation of sensors within each of a plurality of sensing modules (e.g., an upper sensing module and a lower sensing module). This centralized control circuitry enables coordinated control across sensing modules to allow information to be captured simultaneously or nearly simultaneously from all sensors located therein. In other cases, coordinated control across sensing modules can help reduce crosstalk between two sensing modules. For example, to detect distance information, time-of-flight cameras typically emit pulses of infrared (IR) radiation and detect reflections of the emitted infrared radiation from objects in the environment. In some embodiments, centralized control circuitry is used to stagger the timing of infrared radiation emitted by two time-of-flight cameras arranged on the sensing rod, so that a corresponding time-of-flight sensor senses only the reflection corresponding to its infrared emitter, and not the reflection from the infrared emitter of another time-of-flight sensor.

[0046] In some embodiments, one or both of the two-dimensional camera and depth sensor included in the sensing module may have a fixed orientation (e.g., they may not actively sway and / or pitch). Furthermore, sensors in the upper and lower sensing modules may be oriented at the same angle relative to the sensing rod 240, or they may be oriented at different angles relative to the sensing rod, to capture the desired field of view. For example, the sensors in the upper sensing module may be oriented to capture information about the environment at a 90° angle relative to the vertical axis of the sensing rod 240, while the sensors in the lower sensing module may be oriented to capture information about the environment at a 70° angle relative to the vertical axis of the sensing rod 240 (i.e., facing downwards toward the moving base), thereby enabling the capture of information near the moving base. As shown, in some embodiments, the lower sensing module may be positioned above the actuator 255 along the sensing rod 240, thereby enabling the capture of information near the moving base, but excluding the moving base itself (or only including a limited portion of the moving base) from the captured information.

[0047] The outputs of the plurality of sensing modules can be processed by one or more computing devices to determine features of one or more objects in the environment. Control circuitry located on the robot can be configured to perform actions (e.g., control the operation of the robot arm, change the path direction of the moving base) based at least in part on one or more determined features. For example, the outputs of the sensing modules can be used to determine the faces of stacked boxes, and, based on the determined faces, to determine which box to pick up next from the stack. In response to determining which box to pick up next, the control circuitry can control the robot arm to pick up the next box.

[0048] exist Figure 3A and 3BIn the illustrated embodiment, the sensing rod 240 includes two sensing modules located on the same side of the sensing rod. In some embodiments, the sensing rod 240 includes one or more additional sensing modules located on different (e.g., opposite) sides of the sensing rod. Arranging at least two sensing modules on different sides of the sensing rod enables the robot to capture images simultaneously in multiple directions, which may be advantageous for performing certain tasks.

[0049] Figure 4 This is a top-view schematic diagram of robot 300. This abstract view illustrates the relationship between the motion of turntable 320, sensing lever 340, and robot arm 330. Turntable 320 is configured relative to movable base 310 about a first vertical axis (in... Figure 4 In the top view, this axis is the axis that leads into the page, and this axis can be perpendicular to... Figure 4 The X and Y axes shown are parallel to the Z axis. The rotation of turntable 320 relative to movable base 310 is indicated by arrow 321 in the figure. Sensing rod 340 is configured to rotate relative to turntable 320 about a second vertical axis (again, in…). Figure 4 In the top view, the second vertical axis is the axis of entry into the page (and can rotate parallel to the Z-axis). The rotation of the sensing rod 340 relative to the turntable 320 is indicated by arrow 341 in the figure. It should be understood that since both the first and second axes are vertical, they are parallel.

[0050] Robotic arm 330 (for clarity, in) Figure 4 Only the area covered by the robot arm 330 on the turntable 320 is shown. The robot arm 330 is coupled to the turntable 320 such that the base of the robot arm 330 does not deflect relative to the turntable 320 (e.g., rotate about an axis parallel to the Z-axis). Importantly, the robot arm 330 is kinematically constrained such that a portion of the robot arm 330 is constrained to move within a vertical plane 331 defined in the coordinate system of the turntable 320. Brief reference. Figure 5 It shows a large degree of similarity Figure 4 The robotic arm 430 of arm 330 includes only pitch joints (i.e., joints 432, 434, and 436) in the portion near the wrist 438. That is, the proximal portion of arm 430 can only be positioned relative to turntable 420 around a parallel horizontal axis (i.e., ...). Figure 5 The axes 432a, 434a, and 436a in the center rotate. Thus, the proximal portion of arm 430 can only move within the vertical plane defined by the coordinate system of turntable 420. Return to... Figure 4The proximal portion of the robot arm 330 can only move within the vertical plane 331. Since the base of the arm 330 is rigidly mounted to the turntable 320 and thus rotates together with the turntable 320 about the vertical axis (as shown by arrow 321), the plane 331 in which the proximal portion of the arm 330 moves also rotates together with the turntable 320.

[0051] Importantly, the sensing rod 340 is mounted on the turntable 320 at a position spaced apart from the plane 331, ensuring that the arm 330 cannot physically collide with the sensing rod 340. In other words, since both the robotic arm 330 and the sensing rod 340 are mounted on the turntable 320, the turntable juxtaposes the arm and rod, thus defining their relative positions. Because the sensing rod 340 rotates only about a vertical deflection axis, and because the proximal portion of the robotic arm 330 is restricted to operating within a defined vertical plane, neither the robotic arm nor the sensing rod can move horizontally (in the turntable's coordinate system) to the extent that would cause a collision between the arm 330 and the mast 340.

[0052] In embodiments of a robotic arm that includes a 3-DOF wrist or typically includes joints other than a pitch joint, portions of the robotic arm may not be constrained to remain in the vertical plane as described above. However, certain relevant geometries (e.g., link lengths, end effector dimensions, sensor rod positions) can be selected to avoid collisions between the robotic arm and the sensor rod.

[0053] As described above, the position of the sensing rod on the turntable is at least in part based on the geometry, orientation, and movement of the robot limb to prevent collisions between the arm and the rod. However, other considerations also relate to the choice of the sensing rod's position on the turntable. From a sensing perspective, it may be desirable to place the sensing rod at its maximum distance from the robot arm to limit the arm's occlusion of the sensors on the sensing rod. From a safety perspective, it may be desirable to place the sensing rod within the coverage area of ​​the mobile base to avoid collisions between the sensing rod and the environment. Therefore, in some embodiments, the sensing rod may be located at the maximum radial range on the turntable relative to the turntable's axis of rotation. For example, if the turntable is circular and the sensing rod is circular, then the sensing rod may be located within the turntable's coverage area such that the sensing rod and the turntable's circumference are internally tangent.

[0054] In addition to sensing rods and / or robotic arms, other hardware can also be mounted on the robot's turntable. In robot embodiments where the end effector of the robotic arm is a vacuum-based end effector (e.g., a vacuum gripper or suction gripper), the robot may include an onboard vacuum source coupled to and providing vacuum to the end effector. In some such embodiments, the vacuum source may be coupled to the turntable so that the vacuum source rotates with the turntable as the turntable rotates relative to the moving base. While it may be advantageous (from a stability perspective) to place heavy components such as vacuum sources near the robot's base, configuring the vacuum source to rotate with the turntable may be associated with certain benefits related to the wiring and management of vacuum piping.

[0055] As previously referred to Figure 2B The robot's control circuitry allows for highly coordinated control of the robot's turntable, robotic arm, and sensing rod. The inventors have recognized and understand that controlling the rotation of the sensing rod independently of the turntable's rotation or the movement of the mobile base allows the sensing rod's sensing module to capture stable images even while the robot's turntable and / or mobile base are moving. For example, while the turntable rotates in a first direction (e.g., counterclockwise), the sensing rod can be controlled to rotate in a second direction opposite to the first direction (e.g., clockwise). By rotating the sensing rod in the opposite direction, the images captured by the sensing rod's sensing module can be almost completely blurred by the movement of the robot parts. The ability of the mobile manipulator robot to capture stable, clear images while moving allows for shorter cycle times for certain tasks, as no part of the robot needs to be completely stopped for any given duration. More specifically, it enables the capture of environmental images "instantaneously" while the robotic arm is working. The control circuitry can be configured to control the rotation speed of the sensing rod, at least in part, based on the turntable's rotation speed and the sensing rod's position on the turntable. In some embodiments, the rotational speed of the sensing rod can also be determined based on the position of the robotic arm on the turntable and / or the speed at which the moving base travels.

[0056] As mentioned above, the advantage of being able to control the sensing stick independently of the turntable and mobile base is that the robot can capture images while it is moving. In the box-picking example above, the robot's movement is manifested in the robotic arm repeatedly picking up boxes from a pile and placing them on a conveyor belt. However, when the mobile base moves around a building (such as a warehouse), controlling the sensing stick to capture images in different directions can also be beneficial, as the robot does not need to stop to capture images but can capture stable images while the robot is moving. For example, as the robot moves along a warehouse aisle, the sensing module can be oriented towards a shelf in the aisle to detect one or more labels placed thereon containing information that facilitates the robot's task. Images captured by the sensing module while the robot is moving can also be used to aid in the robot's safe operation. For example, the sensing module can be oriented to detect obstacles in the robot's path, and in response to obstacle detection, the robot's path can be changed to avoid the obstacles. Because the sensing stick can be controlled independently of the turntable, the turntable itself and the robotic arm operably coupled to it can remain stationary while the robot is moving, which improves safety, while the sensing stick can be rotated to capture images in any desired direction for any desired purpose (or for multiple purposes, such as reading tags and obstacle avoidance).

[0057] In the examples provided above, the sensing module is described only as being mounted on the sensing rod. However, in some embodiments, additional cameras or sensors may be arranged on other parts of the mobile robotic arm (e.g., on the mobile base or arm) to improve sensing for performing certain tasks, and this disclosure is not limited thereto.

[0058] Control of one or more of the robotic arm, mobile base, turntable, and sensing rod can be accomplished using one or more computing devices located on the mobile manipulator robot. For example, one or more computing devices may be located within a portion of the mobile base, with connections extending between the one or more computing devices and the sensing robotic components and the robotic components to be controlled. In some embodiments, the one or more computing devices may be coupled to dedicated hardware configured to send control signals to specific components of the robot to enable operation of various robotic systems. In some embodiments, the mobile manipulator robot may include dedicated safety-grade computing devices configured to integrate with a safety system that ensures safe operation of the robot.

[0059] exist Figure 5Exemplary embodiments of a computing system that can be used in conjunction with any of the embodiments disclosed herein are illustrated. For example, any of the computing devices described above can be implemented as computer system 500. Computer system 500 may include one or more computer hardware processors 502 and one or more articles of manufacture including non-transitory computer-readable storage media (e.g., memory 504 and one or more non-volatile storage devices 506). Processor 502 may control the writing of data to memory 504 and reading data from memory 504 and non-volatile storage devices 506 in any suitable manner. To perform any of the functions described herein, processor 502 may execute one or more processor-executable instructions stored in one or more non-transitory computer-readable storage media (e.g., memory 504), which may be used as a non-transitory computer-readable storage medium for storing processor-executable instructions executed by processor 502.

[0060] The computing devices and systems described and / or illustrated herein broadly refer to any type or form of computing device or system capable of executing computer-readable instructions (e.g., instructions contained in the modules described herein). In their most basic configuration, these computing devices may each include at least one memory device and at least one physical processor.

[0061] In some instances, the term "memory device" generally refers to any type or form of volatile or non-volatile storage device or medium capable of storing data and / or computer-readable instructions. In one instance, a memory device may store, load, and / or maintain one or more modules described herein. Examples of memory devices include, but are not limited to, random access memory (RAM), read-only memory (ROM), flash memory, hard disk drive (HDD), solid-state drive (SSD), optical disk drive, cache, variations or combinations of one or more of the same devices, or any other suitable memory.

[0062] In some instances, the term "physical processor" or "computer processor" generally refers to a processing unit of any type or form of hardware implementation capable of interpreting and / or executing computer-readable instructions. In one instance, a physical processor can access and / or modify one or more modules stored in the aforementioned memory devices. Examples of physical processors include, but are not limited to, microprocessors, microcontrollers, central processing units (CPUs), field-programmable gate arrays (FPGAs) implementing soft-core processors, application-specific integrated circuits (ASICs), portions of one or more of these devices, variations or combinations of one or more of these devices, or any other suitable physical processor.

[0063] While the modules illustrated and / or illustrated herein are shown as separate elements in the accompanying drawings, these modules may represent a portion of a single module or application. Furthermore, in some embodiments, one or more of these modules may represent one or more software applications or programs that, when executed by a computing device, enable the computing device to perform one or more tasks. For example, one or more of the modules illustrated and / or illustrated herein may represent modules stored and configured to operate on one or more of the computing devices or systems illustrated and / or illustrated herein. One or more of these modules may also represent all or part of one or more dedicated computers configured to perform one or more tasks.

[0064] Furthermore, one or more of the modules described herein can convert data, physical devices, and / or representations of physical devices from one form to another. Or, additionally, one or more of the modules described herein can convert a processor, volatile memory, non-volatile memory, and / or any other part of the physical computing device from one form to another by executing on a computing device, storing data on a computing device, and / or otherwise interacting with a computing device.

[0065] The above embodiments can be implemented in any of a variety of ways. For example, the embodiments can be implemented using hardware, software, or a combination thereof. When implemented in software, the software code can execute on any suitable processor or set of processors, whether these processors are provided in a single computer or distributed across multiple computers. It should be understood that any component or set of components performing the above functions can generally be considered as one or more controllers controlling the above functions. The one or more controllers can be implemented in a variety of ways, such as utilizing dedicated hardware or utilizing one or more processors programmed using microcode or software to perform the above functions.

[0066] In this regard, it should be understood that embodiments of the robot may include at least one non-transitory computer-readable storage medium (e.g., computer memory, portable memory, optical disc, etc.) encoded with a computer program (i.e., multiple instructions) that, when executed on a processor, performs one or more of the functions described above. These functions may, for example, include controlling the robot and / or driving its wheels or arms. The computer-readable storage medium may be portable, such that the program stored thereon can be loaded onto any computer resource to implement the various aspects of the invention discussed herein. Furthermore, it should be understood that references to computer programs that perform the functions described above when executed are not limited to applications running on a host computer. Rather, the term "computer program" is used herein in a general sense to refer to any type of computer code (e.g., software or microcode) that can be used to program a processor to implement the aforementioned aspects of the invention.

[0067] Various aspects of the present invention can be used alone, in combination, or in various arrangements not specifically discussed in the foregoing embodiments, and therefore their application is not limited to the details and arrangements of the components set forth in the foregoing description or shown in the drawings. For example, an aspect described in one embodiment can be combined in any way with aspects described in other embodiments.

[0068] Furthermore, embodiments of the invention can be implemented as one or more methods, examples of which have been provided herein. Actions performed as part of a method can be ordered in any suitable manner. Therefore, embodiments can be constructed that perform actions in a different order than those shown, which may include performing certain actions simultaneously, even if these actions are shown as sequential actions in the exemplary embodiments.

[0069] The use of ordinal numbers such as "first," "second," and "third" in claims to modify the claim elements themselves does not imply any priority, precedence, or order of one claim element relative to another, or the chronological order in which the actions of the method are performed. Ordinal numbers are merely labels used to distinguish one claim element with a certain name from another element with the same name (but using an ordinal number).

[0070] The wording and terminology used in this document are for illustrative purposes only and should not be considered restrictive. The use of words such as “comprising,” “including,” “having,” “containing,” “involving,” and their variations implies coverage of the items listed thereafter and any additional items.

[0071] Based on the detailed description of several embodiments of the present invention, various modifications and improvements will readily occur to those skilled in the art. Such modifications and improvements are all within the spirit and scope of the present invention. Therefore, the foregoing description is merely exemplary and not restrictive.

Claims

1. A mobile robot, comprising: Mobile base; A turntable operably coupled to a movable base, the turntable being configured to rotate about a first axis; An arm operably coupled to a first position on a turntable; as well as A sensing rod operably coupled to a second position on a turntable, the sensing rod being configured to rotate about a second axis parallel to a first axis, wherein the first axis and the second axis are spatially separated, and wherein the sensing rod includes a first sensing module and a second sensing module disposed thereon, the second sensing module being disposed between the first sensing module and the turntable.

2. The mobile robot as described in claim 1, further comprising: A control circuit configured to control the rotation of a sensing rod based at least in part on the rotation of a turntable and the distance between a first position and a second position.

3. The mobile robot of claim 1, wherein each of the first sensing module and the second sensing module comprises a two-dimensional (2D) color camera, a depth sensor and at least one light source.

4. The mobile robot of claim 3, wherein the depth sensor comprises a time-of-flight (TOF) camera.

5. The mobile robot of claim 3, wherein the depth sensor includes a stereo camera.

6. The mobile robot of claim 3, wherein the two-dimensional camera includes an RGB monocular camera.

7. The mobile robot of claim 1, wherein the first sensing module and the second sensing module are arranged along the same side of the sensing rod.

8. The mobile robot of claim 7, wherein the first sensing module and the second sensing module are arranged such that the distance between the first imaging module and the second imaging module along the same side of the sensing rod is maximized.

9. The mobile robot of claim 7, wherein the field of view of the first sensing module and the field of view of the second sensing module do not overlap.

10. The mobile robot of claim 7, wherein the sensing rod includes a plurality of sides, the plurality of sides including a first side that is the same as the side on which the first sensing module and the second sensing module are disposed, and wherein the sensing rod further includes a third sensing module disposed thereon on a side of the sensing rod that is different from the first side.

11. The mobile robot of claim 2, wherein the control circuitry is further configured to control the operation of the arm at least in part based on the output of the first sensing module and / or the second sensing module.

12. The mobile robot of claim 1, wherein the second position is at the outer edge of the turntable.

13. The mobile robot of claim 1, further comprising: A control circuit configured to simultaneously control the rotation of the sensing rod and the rotation of the turntable in opposite directions.

14. The mobile robot of claim 1, wherein the sensing rod further comprises at least one antenna disposed thereon and configured to receive signals from a system outside the mobile robot.

15. The mobile robot of claim 1, wherein the sensing rod further comprises a scanner configured to scan identifier tags on objects attached to the environment of the mobile robot.

16. The mobile robot of claim 1, further comprising: A control circuit configured to control the operation of a first sensing module and a second sensing module to capture one or more images as the mobile robot moves.

17. The mobile robot of claim 16, wherein the control circuitry is further configured to control the operation of the mobile robot based at least in part on one or more images captured by the first sensing module and / or the second sensing module during the movement of the mobile robot.

18. The mobile robot of claim 17, wherein the operation of the mobile robot includes the direction of travel of the mobile robot.

19. The mobile robot of claim 1, further comprising at least one camera disposed on the arm.

20. The mobile robot of claim 1, wherein the first sensing module is oriented at a first angle relative to the sensing rod, and the second sensing module is oriented at a second angle relative to the sensing rod, wherein the first angle and the second angle are different.

21. A method for capturing one or more images using a mobile robot, the method comprising: During the rotation of the mobile robot's turntable along a first direction, a control sensing rod is rotated along a second direction opposite to the first direction. The sensing rod is operatively coupled to the turntable, the sensing rod is configured to rotate about a first axis, the turntable is configured to rotate about a second axis, the first axis and the second axis are parallel and spatially separated, and the sensing rod includes a plurality of sensing modules disposed thereon. as well as During the rotation of the turntable and sensing rod, one or more images are captured by the plurality of sensing modules.

22. The method of claim 21, wherein capturing one or more images comprises capturing one or more images during movement of a movable base operatively coupled to the turntable.

Images