Clutter cleaning robot system

Abstract

The robot is operated to navigate its environment using cameras and to map the type, size, and location of objects. The system determines the type, size, and location of objects and categorizes them to be associated with specific containers. For each category of objects with a corresponding container, the robot selects a specific object within that category for pickup, performs path planning, and navigates to the object in that category to gather or pick it up. An actuated pusher arm moves other objects aside and manipulates the target object onto the front bucket for transport.

Description

[0001] Cross-references to related applications

[0002] This application claims priority and benefit under 35 USC 119(e) to U.S. Application Serial No. 63 / 119,533, filed November 30, 2020, entitled “Clutter Cleaning Robot System,” the entire contents of which are incorporated herein by reference. This application also claims priority and benefit under 35 USC 119(e) to U.S. Application Serial No. 63 / 253,867, filed October 8, 2021, entitled “Augmented Reality Robot Interface,” the entire contents of which are incorporated herein by reference. Background Technology

[0003] Not only are objects underfoot annoying, but they also pose safety hazards. Thousands of people fall and get injured at home every year. Loose items piled on the floor can be dangerous, but many people don't have enough time to deal with clutter in their homes. Automated cleaning or organizing robots can be an effective solution.

[0004] While a fully automated organizing robot with basic searching capabilities may be sufficient to pick up objects from the floor of any room, users may want more granular control over the robot's behavior and its sorting and destination of specific objects unique to their home. The robot may also need clear communication with the user when tasks are stalled or when situational parameters not considered in its programming are encountered. However, most users are unlikely to be experts in robotics and artificial intelligence.

[0005] Therefore, there is a need for a way for users to interact with tidying robots in an intuitive and powerful manner, without relying on in-depth programming knowledge. This interaction process would allow users to train the robot based on their specific tidying needs, instruct the robot to perform actions beyond preset routines, and receive instructions when the robot needs assistance to continue or complete assigned tasks. Attached Figure Description

[0006] To facilitate identification of any particular element or action being discussed, the highest significant digit in the reference numerals refers to the reference numeral in which the element was first introduced.

[0007] Figure 1 A robot system 100 according to one embodiment is shown.

[0008] Figure 2A A top view of a robot 200 according to one embodiment is shown, with the bucket in a downward position and the actuator arm (manipulator) in an open configuration.

[0009] Figure 2BA perspective view of a robot 200 according to one embodiment is shown, wherein the bucket is in a downward position and the manipulator is in an open configuration.

[0010] Figure 2C A front view of a robot 200 according to one embodiment is shown, with the bucket in a downward position and the manipulator in an open configuration.

[0011] Figure 2D A side view of a robot 200 according to one embodiment is shown, with the bucket in a downward position and the manipulator in an open configuration.

[0012] Figure 2E A top view of a robot 200 according to one embodiment is shown, with the bucket in an upward (lifted) position and the manipulator in an open configuration.

[0013] Figure 2F A perspective view of a robot 200 according to one embodiment is shown, wherein the bucket is in an upward (lifted) position and the manipulator is in an open configuration.

[0014] Figure 2G A front view of a robot 200 according to one embodiment is shown, with the bucket in an upward (lifted) position and the manipulator in an open configuration.

[0015] Figure 2H A side view of a robot 200 according to one embodiment is shown, with the bucket in an upward (lifted) position and the manipulator in an open configuration.

[0016] Figure 2I A top view of a robot 200 according to one embodiment is shown, with the bucket in a downward position and the manipulator in a closed configuration.

[0017] Figure 2J A perspective view of a robot 200 according to one embodiment is shown, wherein the bucket is in a downward position and the manipulator is in a closed configuration.

[0018] Figure 2K A perspective view of a robot 200 according to one embodiment is shown, wherein the bucket is in a downward position and the manipulator is in a closed configuration.

[0019] Figure 2L A side view of a robot 200 according to one embodiment is shown, with the bucket in a downward position and the manipulator in a closed configuration.

[0020] Figure 3 One aspect of the subject matter according to one embodiment is shown.

[0021] Figure 4AA front view of robot 400 in another embodiment is shown.

[0022] Figure 4B A 3D view of robot 400 is shown.

[0023] Figure 4C A side view of robot 400 is shown.

[0024] Figure 4D A robot 400 with a manipulator in an elevation orientation is shown.

[0025] Figure 5 A robot 500 according to yet another embodiment is shown.

[0026] Figures 6A to 6D A robot 600 according to yet another embodiment is shown.

[0027] Figures 7A-7D Aspects of a robot 700 according to one embodiment are shown.

[0028] Figure 8A A lowered bucket position and a lowered gripper position 800a of a robot 700 according to one embodiment are shown.

[0029] Figure 8B The diagram shows a robot 700 in a lowered bucket position and an elevated gripper position 800b according to one embodiment.

[0030] Figure 8C The diagram shows a robot 700 in an elevated bucket position and an elevated gripper position 800c according to one embodiment.

[0031] Figure 9 A robot 900 according to another embodiment is shown.

[0032] Figure 10 A robot process 100 according to one embodiment is shown.

[0033] Figure 11 Another robotic process 1100 according to one embodiment is shown.

[0034] Figure 12 A state-space mapping 1200 of a robot system according to one embodiment is shown.

[0035] Figure 13 An example of a robotic system in sleep mode is shown.

[0036] Figures 14A-14B An example of a robotic system in exploration mode is shown.

[0037] Figures 15A-15FAn example of a robotic system in pick-up mode is shown.

[0038] Figures 16A-16B An example of a robot system in delivery mode is shown.

[0039] Figure 17 A robot control algorithm 1700 for a robot system according to one embodiment is shown.

[0040] Figure 18 A robot control algorithm 1800 for a robot system according to one embodiment is shown.

[0041] Figure 19 A robot control algorithm 1900 for a robot system according to one embodiment is shown.

[0042] Figure 20 A robot control algorithm 2000 according to one embodiment is shown.

[0043] Figure 21 A robot control algorithm 2100 according to one embodiment is shown.

[0044] Figure 22 An tidying robot environment 2200 according to one embodiment is shown.

[0045] Figures 23A to 23E An augmented reality robot interface 2300 according to one embodiment is shown.

[0046] Figure 24 An embodiment of a robot control system 2400 is shown to implement the components and process steps of the system described herein. Detailed Implementation

[0047] An embodiment of a robotic system is disclosed, in which the robot uses cameras to navigate its environment to map the type, size, and location of toys, clothing, obstacles, and other objects. The robot includes a neural network to determine the type, size, and location of objects based on images from left / right stereo cameras. For each class of objects with a corresponding container, the robot selects a specific object to pick up within that class, performs path planning, and navigates to a nearby point facing the target object. An actuated pusher arm moves other objects aside and pushes the target object onto the front bucket for transport. The front bucket is slightly tilted upwards, and optionally, the actuated pusher arm can close in front to hold the object in place as the robot navigates to the next location in the planned path. This process is repeated to pick up other objects of the same class. Once the bucket is full, the robot performs additional path planning and drives itself toward a container of that class. The robot then navigates to the container, aligns with one side of the container, and raises the bucket to lift the target object above the edge of the container.

[0048] This robotic system can be used for the automated gathering of objects left on surfaces, where items left on the surface are automatically packed into containers according to rules. In one specific embodiment, the system can be used to automatically organize children's play areas (e.g., in homes, schools, or businesses), where toys and / or other items are automatically returned to containers specific to different object types after children have finished playing. In other specific embodiments, the system can be used to automatically pick up clothing from the floor and gather it into one or more laundry baskets for washing, or to automatically pick up trash from the floor and place it, for example, by type (plastic, cardboard, glass) into trash cans or one or more recycling bins. Typically, the system can be deployed to efficiently pick up a variety of different objects from surfaces and can learn to pick up new types of objects.

[0049] In some embodiments, the system may include a robotic arm to reach and grasp lifted objects and move them down into the bucket. In some embodiments, a paired “portable elevator” robot may also be used to lift the main robot onto a table, workbench, or other elevated surface and then lower it back to the floor. Some embodiments may utilize up / down vertical lifts (e.g., scissor lifts) to change the bucket height when dropping items into containers, shelves, or other high or elevated locations.

[0050] Another variant employs an up / down vertical lift (e.g., a scissor lift or telescopic lift) built into the robot, which raises the bucket and actuator arm to the height of a table or workbench to pick up objects from the table or workbench.

[0051] Some embodiments may utilize one or more of the following components:

[0052] • Left / right rotating brushes on the actuator arm push the object onto the bucket;

[0053] • An actuated gripper that grasps objects and moves them onto the bucket;

[0054] • A rotating wheel with baffles that pushes the object onto the bucket from above;

[0055] • A servo mechanism that lifts the front bucket into the air, and another separate servo mechanism that tilts the bucket forward to dump it into the container;

[0056] • A variation of a scissor lift that raises the bucket and gradually tilts it backward as the bucket height increases;

[0057] • The ramp on the container and the front bucket on the hinge allow the robot to push the item up the ramp and dump it into the container under the force of gravity at the top of the ramp.

[0058] • A storage bin on the robot increases its carrying capacity. Without using the front bucket, the target object is pushed up a ramp and into the storage bin. The storage bin tilts up and down like an automated dump truck to load items into it.

[0059] Embodiments of a robotic system are disclosed, which incorporates an augmented reality robotic interface presented to the user as an application to operate the sorting robot. The sorting robot can use cameras to navigate its environment to map the type, size, and location of toys, clothing, obstacles, and other objects. The robot may include neural networks to determine the type, size, and location of objects based on input from sensing systems, such as images from left / right stereo cameras. The robot can select specific objects to pick up, perform path planning, and navigate to a nearby point facing the target object. Actuated gripping pad arms can move other objects aside and manipulate the gripping pads to move the target object onto the bucket for handling. As the robot navigates to the next location in the planned path (e.g., a stacking destination), the bucket may tilt slightly upwards and close the gripping pads in front to hold the object in place.

[0060] In some embodiments, the system may include a robotic arm to reach and grasp elevated objects and move them down into the bucket. In some embodiments, a paired “portable elevator” robot is also used to lift the main robot onto a table, desk, or other elevated surface and then lower it back to the floor. Some embodiments may utilize an up / down vertical lift (e.g., a scissor lift) to change the bucket height when dropping items into containers, shelves, or other high or elevated locations.

[0061] Some embodiments may also utilize one or more of the following components:

[0062] • The left / right rotating brushes on the actuator arm push the object onto the bucket;

[0063] • An actuated gripper that grasps an object and moves it onto the bucket;

[0064] • A rotating wheel with baffles that pushes the object onto the bucket from above;

[0065] • A servo mechanism that lifts the front bucket into the air, and another separate servo mechanism that tilts the bucket forward to pour it into the container;

[0066] • A variation of a scissor lift that raises the bucket and gradually tilts it backward as the bucket height increases;

[0067] • The ramp on the container and the front bucket on the hinge allow the robot to push the item up the ramp and dump it into the container under the force of gravity at the top of the ramp.

[0068] • A storage bin on the robot increases its carrying capacity. Without using the front bucket, the target object is pushed up a ramp and into the storage bin. The storage bin tilts up and down like an automated dump truck to load items into it.

[0069] This robotic system can be used for the automated gathering of objects left on surfaces, where items left on the surface are automatically packed into containers according to rules. In one specific embodiment, the system can be used to automatically organize children's play areas (e.g., in homes, schools, or businesses), where toys and / or other items are automatically returned to containers specific to different object types after children have finished playing. In other specific embodiments, the system can be used to automatically pick up clothing from the floor and gather it into one or more laundry baskets for washing, or to automatically pick up trash from the floor and place it, for example, by type (plastic, cardboard, glass) into trash cans or one or more recycling bins. Typically, the system can be deployed to efficiently pick up a variety of different objects from surfaces and can learn to pick up new types of objects.

[0070] The disclosed solution is a method of using augmented reality to interact with a tidying robot to convey the following information to the user:

[0071] • The category / type of each object to be picked

[0072] How to group objects by type into boxes or other locations

[0073] What objects will be picked up next?

[0074] • Problematic objects that the robot cannot move (e.g., too large, too heavy, cluttered, dangerous, or fragile).

[0075] Obstacles blocking the robot's path

[0076] The tidying robot can use a combination of cameras, LiDAR, and other sensors to maintain a global area mapping of the environment and locate itself within it. Furthermore, the robot can perform object detection and classification, and generate a visual re-identification fingerprint for each object.

[0077] The robot can determine its relative distance and angle to each object. This distance and angle can then be used to locate the object on a global area map. This global area map, along with the object information, can then be wirelessly transmitted to an application. This application can be installed on a user's mobile device, a display mounted on the robot, or some other device that provides convenient display and acceptance of user input.

[0078] A mobile device's camera can be used to capture a camera frame. The augmented reality robot interface application can then perform object detection and classification, and generate a visual re-identification fingerprint for each object. A ground plane can also be detected in the camera frame. The smartphone can then be located on a global region map using a combination of the ground plane, object type, and re-identification fingerprint. The object can then be re-identified using its type, location, and re-identification fingerprint.

[0079] Information can then be overlaid on top of the camera view to indicate the object type and a user indicator requesting assistance. Users may be thanked if they help complete the task (e.g., moving obstacles or tidying up large objects). Users can also tap on objects to give the robot instructions to change the object's type or what to pick up next.

[0080] The disclosed algorithm improves the routine use of augmented reality as an application deployed in smart devices or mobile devices (such as smartphones or tablets) by providing ordinary users of the technology with a simple way to interact with consumer robots. The disclosed algorithm allows mobile devices and the organizing robot to reside in their independent workstations within the organizing area and communicate back and forth about the same objects detected in that area, regardless of the location and orientation of each device in the environment. In one embodiment, the augmented reality robot interface can be deployed as a computer or laptop device, a game console connected to a television, or an application on some other electronic system capable of presenting visual information to the user, accepting user input, and wirelessly connecting to the organizing robot described below.

[0081] The disclosed solution provides a way to visualize a tidying robot’s understanding of objects in its environment using a mobile device such as a smartphone, and then uses a mobile device application (the disclosed augmented reality robot interface) as an interface to allow the robot to request help and allow the user to give the robot instructions on how to tidy up.

[0082] In the consumer market, users don't need advanced computer coding backgrounds to guide the robots; instead, they can easily instruct the organizing robots on how to gather and place the objects they detect. The organizing robots can also clearly guide users on how to best assist with cleaning, such as how to help place problematic objects (i.e., those that are too large, too difficult to pick up, fragile, etc.). The disclosed solutions can leverage this difference by detecting and considering the different views and perspectives that the organizing robot and the user with their mobile device might have regarding the cleaning area. For example, the organizing robot can help users locate objects hidden from their view by furniture or other obstacles.

[0083] The publicly available interface visually shows the user which objects will be picked up by the sorting robot, the type or category detected for each object, and which available bins these objects can be grouped into. The user can change the object category so that objects can be placed in bins different from the robot's default selection. The robot can use the publicly available algorithm to request user assistance to clean up problematic objects it has identified, such as those that are too large, cluttered, fragile, dangerous, or obstructing its path in ways it cannot resolve on its own. The augmented reality robot interface allows the user to choose which objects to pick up next and which to skip.

[0084] The disclosed augmented reality robot interface combines a novel algorithm for wirelessly synchronizing a global region map of a tidying area between the robot and an application and re-identifying objects within the global region map, with an algorithm for locating a camera on the global region map. Since the augmented reality robot interface focuses on cleaning objects from the floor of the tidying area, the global region map, or the global region map used by the tidying robot, can be a two-dimensional map. In augmented reality, the ground plane and the object's starting position can be detected along with the object category and re-identified fingerprint. This information can be used to locate the mobile device on the global region map.

[0085] In one embodiment, the augmented reality robot interface can be used to gamify cleaning tasks. For example, the application can show the user items to tidy up and the robot items to tidy up, and hold a competition to see who can tidy up their items first. In one embodiment, the augmented reality robot interface can be used to instruct the tidying robot to retrieve a specific item and place it in its proper place or bring it to the user. For example, the tidying robot can be instructed to bring the user a TV remote control. In one embodiment, the augmented reality robot interface can be used without a tidying robot to help users organize objects in their homes.

[0086] In one embodiment, the augmented reality robot interface can be deployed on the organizing robot itself, eliminating the need for a separate mobile device. The robot can incorporate a speaker and provide audio messages and alerts to convey information such as object categories, expected pickup order, and problematic objects. The organizing robot may also include a microphone and voice recognition capabilities, allowing it to accept voice commands from a user. In one embodiment, the organizing robot may be configured with a touchscreen that can display a global area map and objects and accept user input. The organizing robot may include visual indicators that change color based on the type of object it picks up. The robot may include directional features such as colored lasers, allowing it to indicate specific objects in conjunction with audio messages.

[0087] In one embodiment, the mobile device may be able to map the tidying area and transmit that information to the tidying robot. In this way, the mobile device can update the global area mapping of the environment with areas the robot has not yet explored. In one embodiment, the augmented reality robot interface can map the tidying area and objects onto a view based on the mobile device's camera view, and in another embodiment, the global area mapping can be displayed in a top-down, two-dimensional view. In yet another embodiment, the mapping can be displayed from the robot's ground viewpoint, allowing users to see objects under furniture or other objects obscured by their mobile device's camera view.

[0088] Figure 1 A robot system 100 according to one embodiment is depicted. The robot system 100 receives input from one or more sensors 102 and one or more cameras 104, and provides these inputs for processing by localization logic 106, mapping logic 108, and cognitive logic 110. The outputs of the processing logic are provided to a path planner 112, a pickup planner 114, and a motion controller 116 that sequentially drives the system's motor and servo mechanism controllers 118.

[0089] One or more of the positioning logic 106, mapping logic 108, and cognitive logic 110 may be located on and / or executed on the mobile robot, or may be executed in a computing device that wirelessly communicates with the robot, such as a cellular phone, laptop computer, tablet computer, or desktop computer. In some embodiments, one or more of the positioning logic 106, mapping logic 108, and cognitive logic 110 may be located in and / or executed in the "cloud," that is, on a computer system connected to the robot via the Internet or other networks.

[0090] Cognitive logic 110 is coupled with image segmentation activation signal 144 and utilizes any one or more well-known image segmentation and object recognition algorithms to detect targets in the field of view of camera 104. Cognitive logic 110 may also provide calibration and object 120 signals for mapping targets. Localization logic 106 uses any one or more well-known algorithms to localize the mobile robot in its environment. Localization logic 106 outputs a reference frame transformation 122 from local to global, and mapping logic 108 combines it with the calibration and object 120 signal group to generate an environment mapping 124 for pickup planner 114 and an object tracking 126 signal for path planner 112.

[0091] In addition to the object tracking 126 signal from mapping logic 108, path planner 112 also utilizes the current system state 128 from system state setting 130, synchronization signal 132 from pickup planner 114, and motion feedback 134 from motion controller 116. Path planner 112 translates these inputs into navigation waypoints 136 that drive motion controller 116. Pickup planner 114 translates local cognitive input with image segmentation 138 from cognitive logic 110, 124 from mapping logic 108, and synchronization signal 132 from path planner 112 into maneuvering actions 140 (e.g., robot gripper, bucket) on motion controller 116. Embodiments of the algorithms used by path planner 112 and pickup planner 114 are described in more detail below.

[0092] In one embodiment, a Simultaneous Localization and Mapping (SLAM) algorithm can be used to generate a global map and simultaneously localize the robot on that map. Many SLAM algorithms are known in the art and are commercially available.

[0093] The motion controller 116 converts the navigation waypoint 136, the manipulation action 140, and the local cognition with the image segmentation 138 signal into the target motion 142 signal, which is then transmitted to the motor and servo mechanism controller 118.

[0094] Figure 2A-2LRobot 200 with various configurations including a lifting bucket and an actuator arm is depicted. Robot 200 can be configured to operate according to the algorithms disclosed herein. Robot 200 can utilize a front bucket (bucket 202) that is tightly sealed to a surface, and a pusher arm (manipulator actuator arm 204) capable of pushing objects on the surface and actuating them onto the bucket 202 for carrying. (The terms "bucket" and "shovel" are used interchangeably herein to refer to the robot's object-receiving container). For example, this allows the robot to simultaneously pick up several small objects, such as Lego bricks or marbles, without dropping them.

[0095] Robot 200 can utilize stereo camera 206 and machine learning / neural network software architecture (e.g., semi-supervised or supervised convolutional neural networks) to efficiently classify the type, size, and location of different objects on an environmental map. Robot 200 can use forward-facing and backward-facing cameras 206 to scan its front and rear. Bucket 202 can rotate upward and backward on a single actuator arm (bucket actuator arm 208) to allow a single servo mechanism / motor to slightly lift bucket 202 off the ground to carry an object, and then use the same servo mechanism / motor to lift bucket 202 above the top of the container for dumping. This method keeps robot 200 at a low height for picking up items (e.g., under a chair or sofa) but still enables it to lift items and dump them into a higher container.

[0096] The manipulator actuator arm 204 may include manipulator brushes 210 with bristles set at a slight angle (relative to the surface), allowing them to approach flat objects (e.g., flat Lego boards or puzzle pieces) and tumble / roll them onto the pickup bucket 202. Using angled manipulator brushes 210 can also aid in the pickup of high-friction objects such as rubber balls or toys. Providing horizontal pressure while positioning the manipulator actuator arm 204 slightly below the object helps in picking up items while preventing clogging.

[0097] In one embodiment, two motors are used to move the robot 200 around its environment, two servo mechanisms are used on the left / right manipulator actuator arms 204, and a servo mechanism is used to lift the bucket 202 up and back to dump items into a container. Using fewer motors / servo mechanisms reduces cost, lightens weight, and improves the reliability of the robot 200. The use of a bucket 202 with servo mechanisms allows the bucket 202 to tilt slightly up and back, raising it off the ground. This allows for carrying multiple items simultaneously without dropping them when navigating obstacles and crossing ridges. The manipulator actuator arms 204 can form a wedge-shaped "V" shape in front of the robot 200 to enable navigation in cluttered / messy environments where items are pushed left / right as they are driven, much like a plowshare. This allows the robot 200 to navigate in cluttered / messy environments without getting stuck or preventing items from getting under the robot 200 where they might become entangled in wheels or other drive mechanisms. In addition, this allows unwanted objects to reach the bucket 202.

[0098] Robot 200 can navigate its environment using manipulator actuator arms 204 to gather items on the floor into a pile before attempting to pick them up into bucket 202. This utilizes available floor space to group items by category. Manipulator actuator arms 204 are multi-purpose tools, as they can both push target objects onto bucket 202 for pickup and gather items into a pile on the floor.

[0099] In some embodiments, this can also be a "rapid cleanup mode," during which robot 200 picks up objects as quickly and indiscriminately as possible and places them into a single container, or even places them against a wall. In this mode, robot 200 can bypass many of the actions and states described later, such as aggregation, waypoint path planning, and packing.

[0100] Containers (as shown in other accompanying figures) can be placed in this environment, including category labels that are both machine-readable and human-readable. Replacing the labels on the containers changes the type of objects that will be gathered into them. This allows the robot 200 to work alongside humans to help clean up cluttered / messy rooms. It also facilitates changing the category of the gathered objects.

[0101] The bucket 202 may include: a unique shape (described hereby in relation to a bucket 202 oriented flat on the floor) that allows the bucket 202 to maintain a tight seal 302 against the floor, thereby allowing small objects to be pushed onto the bucket 202; and adjacent protrusions 304 and recesses 306 that retain objects within the bucket 202 and prevent round objects from rolling out, wherein the recess 306 transitions into an extended rear surface 308 (which may be curved), the extended rear surface being configured to function as a slide and funnel when inverted. The extended rear surface 308 may also be angled toward the back of the robot 200 (away from the bucket 202) to reduce the overall height of the robot 200.

[0102] Bucket 202 can utilize a hinge mechanism that allows it to tilt forward and dump unwanted items as the robot moves backward. This can be driven by the same servo mechanism that enables bucket 202 to be lifted and moved backward. This mechanism can also be used to drop items back onto the floor if they are accidentally picked up. A collection component (e.g., manipulator brush 210) on the manipulator actuator arm 204 can utilize a compressible foam pad that allows both manipulator actuator arms 204 to work together to grip and hold objects in front of the robot 200. This provides a useful way to manipulate specific objects, such as for grouping objects into clusters or for clearing paths. For example, a target object can be moved to a less cluttered area before being picked up, or an object can be moved to a pile on the floor with similar items.

[0103] Users can construct custom categories of objects via applications, such as those on mobile phones, to aggregate previously unlearned object types. For example, a user can use a mobile app to perform this construction to create custom labels and then take a photo of the custom objects they want to aggregate into a container with those labels.

[0104] In a particular embodiment, robot 200 may include some or all of the following components:

[0105] • 3D printed (or injection molded) plastic robot chassis.

[0106] • Two brushless planetary gear DC motors are used to drive the robot's wheels.

[0107] • Two small servo mechanisms for moving the left / right pusher arms.

[0108] • A medium-sized servo mechanism for lifting the bucket up and down.

[0109] • Brushes for the actuator arm and 3D-printed (or injection-molded) mounting components.

[0110] • 3D printed (or injection molded) front bucket.

[0111] • Two front wheels, as well as mounting hardware, ball bearings, metal shafts, and connecting connectors.

[0112] • Two rear casters that can rotate to allow the robot to move freely.

[0113] • Four RGB digital cameras are used to collect visual data.

[0114] • LED lights, used to provide illumination for robots in dark rooms.

[0115] • Microcontrollers with embedded GPUs (e.g., Jetson Nano) are used to run machine learning algorithms.

[0116] • A 24V lithium-ion battery that powers the robot.

[0117] • Custom-designed circuit boards, including battery charger circuitry, enable power management for the sensors and include an inertial measurement unit.

[0118] • Rigid metal beams, along with 3D-printed (or injection-molded) mounting components and ball bearings, can be lifted by a servo mechanism to raise and lower the front bucket.

[0119] • An infrared sensor installed at the front of the robot is used to detect stairs or places where a fall might occur.

[0120] The robot chassis includes mounting components for motors, servo mechanisms, batteries, microcontrollers, and other parts.

[0121] Figures 4A-4C Various views of the robot 400 in alternative embodiments are depicted, including a camera 402, a lifting actuator arm 404, a collecting component 406, a bucket 408, and a linear actuator 410. The robot 400 can be configured to operate according to the algorithms disclosed herein.

[0122] Figure 5 A robot 500 according to yet another embodiment is depicted, including a bucket 502, a camera 504, a manipulator actuator arm 506, a collection member 508, and a bucket actuator arm 510. The robot 500 can be configured to operate according to the algorithms disclosed herein.

[0123] Figures 6A to 6D A robot 600 according to yet another embodiment is shown. The robot 600 includes a bucket 602, a camera 604, a manipulator actuator arm 606, a collection member 608, a bucket actuator arm 610, and a manipulator lifting arm 612. The robot 600 can be configured to operate according to the algorithms disclosed herein.

[0124] Figures 7A to 7D Additional aspects of a robot 700 according to one embodiment are described. Figure 7A A side view of robot 700 is shown, and Figure 7B A top view is shown. The robot 700 may include a chassis 702, a mobility system 704, a sensing system 706, a control system 708, and a capture and containment system 710. The capture and containment system 710 may also include a bucket 712, a bucket lifting arm 714, a bucket lifting arm pivot point 716, two manipulator components 718, two manipulator arms 720, and two arm pivot points 722. Figure 7C and Figure 7D Side and top views of the chassis 702 are shown, along with the general connectivity of components of the sensing system 706, the mobility system 704, and the communication unit 724 with the control system 708. The sensing system 706 may also include cameras such as a front camera 726 and a rear camera 728, light detection and ranging (LIDAR) sensors such as a front lidar sensor 730 and a rear lidar sensor 732, and an inertial measurement unit (IMU) sensor such as an IMU sensor 734. In some embodiments, the front camera 726 may include a right front camera 736 and a left front camera 738. In some embodiments, the rear camera 728 may include a left rear camera 740 and a right rear camera 742.

[0125] The chassis 702 can support and house other components of the robot 700. The mobility system 704 may include wheels as shown, as well as tracks, conveyor belts, etc., as well as those well known in the art. The mobility system 704 may also include motors, servo mechanisms, or other rotational or kinetic energy sources to propel the robot 700 along its desired path. Mobility system components may be mounted on the chassis 702 for the purpose of moving the entire sorting robot without impeding or inhibiting the range of motion required by the capture and containment system 710. In at least some configurations of the chassis 702, bucket 712, manipulator component 718, and manipulator arm 720 relative to each other, elements of the sensing system 706, such as cameras, lidar sensors, or other components, may be mounted on the chassis 702 at locations that allow the robot 700 a clear line of sight to its environment.

[0126] The chassis 702 can house and protect the control system 708, and in some embodiments includes a processor, memory, and connections to the mobility system 704, sensing system 706, capture and containment system 710, and communication unit 724. The chassis 702 may include other electronic components, such as batteries, wireless communication devices, etc., as well as those well known in the field of robotics. These components can be used as part of the robot control system 2400, see reference 2400. Figure 24 To describe in more detail.

[0127] The capture and containment system 710 may include a bucket 712, a bucket lifting arm 714, a bucket lifting arm pivot point 716, a manipulator member 718, a manipulator arm 720, a manipulator pivot point 744, and an arm pivot point 722. The geometry of the bucket 712 and the configuration of the manipulator member 718 and the manipulator arm 720 relative to the bucket 712 describe a containment area in which an object can be safely carried. Servo mechanisms at the bucket lifting arm pivot point 716, the manipulator pivot point 744, and the arm pivot point 722 can be used to adjust the configuration of the bucket 712, the manipulator member 718, and the manipulator arm 720 between a fully lowered bucket and gripper position and a raised bucket and gripper position.

[0128] In some embodiments, clamping surfaces may be constructed on the sides of the manipulator member 718 facing the object to be lifted. These clamping surfaces may provide cushioning, abrasiveness, elasticity, or other features that increase friction between the manipulator member 718 and the object to be captured and contained. In some embodiments, the manipulator member 718 may be constructed with sweeping bristles. These sweeping bristles may assist in moving small objects upward from the floor onto the bucket 712. In some embodiments, the sweeping bristles may be angled downward and inward from the manipulator member 718 such that when the manipulator member 718 sweeps an object toward the bucket 712, the sweeping bristles form a ramp, allowing the foremost bristles to slide under the object and upward toward the manipulator member 718, facilitating capture of the object within the bucket and reducing the tendency for the object to be pressed against the floor, increasing its friction and making it more difficult to move.

[0129] Rubber objects are generally more difficult to pick up than less elastic objects because they tend to stick and get stuck. Forward-facing bristles are designed to reduce this friction so that the elastic object can rotate and thus not get stuck. Alternative embodiments may utilize low-friction materials such as PTFE (polytetrafluoroethylene) to hold the surface.

[0130] Figures 8A to 8C It shows things such as relative to Figures 7A to 7D The robot 700 described herein features a gripper arm controlled by a servo mechanism at the same contact point as the chassis and bucket. The robot 700 can be positioned in a lowered bucket position and a lowered gripper position 800a, a lowered bucket position and a raised gripper position 800b, and a raised bucket position and a raised gripper position 800c. The robot 700 can be configured to execute the algorithms disclosed herein.

[0131] Figure 9A robot 900 according to another embodiment is depicted, which includes a bucket 902, a bucket actuator arm 910, a camera 904, and a manipulator actuator arm 906 for a collection member 908 extending from and mounted on the bucket 902.

[0132] Figure 10 A robotic process 1000 in one embodiment is depicted. In box 1002, robotic process 1000 wakes a sleeping robot at a base station. In box 1004, robotic process 1000 navigates the robot around its environment using cameras to map the type, size, and location of toys, clothing, obstacles, and other objects. In box 1006, robotic process 1000 operates a neural network based on images from left / right stereo cameras to determine the type, size, and location of objects. In open-loop box 1008, robotic process 1000 performs this for each class of objects with corresponding containers. In box 1010, robotic process 1000 selects a specific object to pick up within a category. In box 1012, robotic process 1000 performs path planning. In box 1014, robotic process 1000 navigates to and from a target object. In box 1016, robotic process 1000 actuates the arm to move other objects aside and pushes the target object onto the front bucket. In box 1018, robot process 1000 tilts the front bucket upwards to hold them on the bucket (creating the bucket's "bowl-shaped" construction). In box 1020, robot process 1000 actuates the arm to close in front to keep objects off the wheels as the robot navigates to the next location. In box 1022, robot process 1000 performs path planning and navigates near the container used for collecting the current object classification. In box 1024, robot process 1000 aligns the robot with one side of the container. In box 1026, robot process 1000 raises and lowers the bucket upwards and backwards to lift the target object upwards and across the side of the container. In box 1028, robot process 1000 returns the robot to the base station.

[0133] In less complex operating modes, a robot might pick up objects within its field of view and place them into a container as needed, without first creating a global map of the environment. For example, the robot could simply explore until it finds the object to pick up, then explore again until it finds a matching container. This approach works effectively in unidirectional environments with limited exploration areas.

[0134] Figure 11 A robotic process 1000 in one embodiment is also depicted. An illustrative example of robotic process 1100 is depicted as follows: Figures 13-16B The graphic sequence in the image, where the robot system passes through in sequence as follows: Figure 12An example of state-space mapping 1200 is shown.

[0135] This sequence begins with the robot going to sleep (sleep state 1202) and charging at the base station. Figure 13 (and frame 1102). The robot, for example, activates as planned and enters exploration mode (environmental exploration state 1204, activation action 1206, scheduled start time 1208). Figure 14A and Figure 14B In environment exploration state 1204, the robot uses cameras (and other sensors) to scan the environment to update its environment map and locate its own position on the map (box 1104, exploration of the construction interval 1210). The robot can transition back to sleep state 1202 from environment exploration state 1204 if there are no more objects to pick up 1212 or the battery is low 1214.

[0136] From environment exploration state 1204, the robot can transition to object gathering state 1216, in which the robot operates to move items on the floor to gather them by category 1218. This transition can be triggered by the robot determining whether objects are too close together on the floor 1220 or whether paths to one or more objects are obstructed 1222. If neither of these triggering conditions is met, the robot can transition directly from environment exploration state 1204 to object picking state 1224, provided that the environment map includes at least one drop container 1226 for one type of object and that there is an unobstructed item 1228 for picking up within the container's category. Similarly, under the latter conditions, the robot can transition from object gathering state 1216 to object picking state 1224. If no object is ready to be picked up 1230, the robot can transition back from object gathering state 1216 to environment exploration state 1204.

[0137] In environment exploration state 1204 and / or object aggregation state 1216, image data from the camera is processed to identify different objects (box 1106). The robot selects a specific object type / category to pick up, determines the next waypoint to navigate to, and determines the target object and type location to pick up based on the environment map (boxes 1108, 1110, and 1112).

[0138] In object picking state 1224, the robot selects a target location (box 1114) that is adjacent to one or more target objects. It uses a path planning algorithm to navigate itself to the new location while avoiding obstacles. Figure 15AAs the robot moves forward, it actuates the left and right pusher arms to create an opening large enough for the target object to pass through, but not large enough to collect other unwanted objects (box 1116). The robot moves forward such that the target object is positioned between the left and right pusher arms, which work together to push the target object onto the collection bucket. Figure 15B , Figure 15C (and frame 1118).

[0139] The robot can remain in object-picking state 1224 to identify other target objects of the selected type to be picked up based on the environment map. If other such objects are detected, the robot selects a new target location that is adjacent to the target object. It uses a path planning algorithm to navigate itself to the new location while avoiding obstacles. Figure 15D The robot moves forward while carrying one or more previously collected target objects. As the robot moves forward, it actuates its left and right actuator arms to create an opening large enough for a target object to pass through, but not large enough to collect other unwanted objects. The robot moves forward such that the next target object(s) is positioned between the left and right actuator arms. Figure 15E Similarly, the left and right pusher arms work together to push the target object onto the collection bucket. Figure 15F ).

[0140] If all identified objects 1232 of the category have been picked up, or if the bucket is fully loaded 1234, the robot transitions to object delivery state 1236, uses environment mapping to select a target location near the box containing the collected object type, and uses a path planning algorithm to navigate itself to the new location while avoiding obstacles (box 1120). The robot then backs towards the box to a docking position where its back is aligned with the back of the box. Figure 16A (and box 1122). The robot lifts the bucket (box 1124) by rotating upwards and backwards on a rigid arm on its back. This lifts the target objects above the top of the box and dumps them into the box. Figure 16B ).

[0141] If there are more items to pick up (1238) or the robot has an incomplete environment mapping (1240), the robot can transition from the object delivery state (1236) back to the environment exploration state (1204). The robot resumes exploration and can repeat the process for each other type of object in the environment with an associated collection box (box 1126).

[0142] The robot can transition from object delivery state 1236 back to sleep state 1202 when there are no more objects to pick up 1212 or the battery is low 1214. Once the battery is fully charged, or when it is in the next activation or a predetermined pickup interval, the robot continues to explore and can repeat the process for each other type of object in the environment with an associated collection box (box 1126).

[0143] Figure 17 A robot control algorithm 1700 for a robotic system is depicted in one embodiment. The robot control algorithm 1700 begins by selecting one or more object categories to be clustered (box 1702). Within the selected one or more categories, clusters of target categories and starting positions for determining paths are identified (box 1704). Any of many well-known clustering algorithms can be used to identify object clusters within one or more categories.

[0144] A path is formed leading to the initial target location, comprising zero or more waypoints (box 1706). Motion feedback is provided back to the path planning algorithm. Waypoints can be selected to avoid static and / or dynamic (moving) obstacles (objects not in the target group and / or category). The robot's motion controller is engaged to follow the waypoints to the target group (box 1708). When the target group reaches the target location, it is evaluated, including determining its additional qualifications for safe aggregation (box 1710).

[0145] The robot's cognitive system is engaged (box 1712) to provide image segmentation, used to determine the activation sequence generated for the robot's manipulators (e.g., arms) and localization systems (e.g., wheels) to cluster the target group (box 1714). The activation sequence is repeated until the target group is clustered or fails to be clustered (failure results in a regression to box 1710). The engagement of the cognitive system can be triggered by proximity to the target group. Once the target group has been clustered, and if the robot has sufficient battery life and more groups need to be clustered in one or more categories, these actions are repeated (box 1716).

[0146] In response to low battery life, the robot navigates back to the docking station for recharging (box 1718). However, if sufficient battery life is available and one or more categories are clustered, the robot enters object-picking mode (box 1720) and picks up one of the clustered groups to return to the delivery container. Entering picking mode may also depend on an environmental map including at least one delivery container for the target object and the presence of unobstructed objects in the target group for picking. If no group of objects is ready to be picked up, the robot continues to explore the environment (box 1722).

[0147] Figure 18 A robot control algorithm 1800 for a robotic system is depicted in one embodiment. The robot control algorithm 1800 begins by selecting one or more object categories to be clustered (box 1802). Within the selected one or more categories, clusters are identified to determine the target category and the starting position of the path (box 1804). Any of many well-known clustering algorithms can be used to identify object clusters within one or more categories.

[0148] A path is formed leading to the initial target location, comprising zero or more waypoints (box 1806). Motion feedback is provided back to the path planning algorithm. Waypoints can be selected to avoid static and / or dynamic (moving) obstacles (objects not in the target group and / or category). The robot's motion controller is engaged to follow the waypoints to the target group (box 1808). When the target group reaches the target location, it is evaluated, including determining its additional qualifications for safe aggregation (box 1810).

[0149] The robot's cognitive system is engaged (box 1812) to provide image segmentation, used to determine the activation sequence generated for the robot's manipulators (e.g., arms) and localization systems (e.g., wheels) to cluster the target group (box 1814). The activation sequence is repeated until the target group is clustered or it fails to be clustered (failure results in a regression to box 1810). The engagement of the cognitive system can be triggered by proximity to the target group. Once the target group has been clustered, and if the robot has sufficient battery life and there are more groups to cluster in one or more categories, these actions are repeated (box 1816).

[0150] In response to low battery life, the robot navigates back to the docking station for recharging (box 1818). However, if sufficient battery life is available and one or more categories are clustered, the robot enters object-picking mode (box 1820) and picks up one of the clustered objects to return to the delivery container. Entering picking mode may also depend on an environmental map including at least one delivery container for the target object and the presence of unobstructed objects in the target group for picking. If no group of objects is ready to be picked up, the robot continues exploring the environment (box 1822).

[0151] Figure 19 A robot control algorithm 1900 for a robotic system is described in one embodiment. A target object in a selected object category is identified (item 1902), and the robot's target position is determined as the proximity of the target object (item 1904). A path to the target object is determined as a series of waypoints (item 1906), and the robot navigates along this path while avoiding obstacles (item 1908).

[0152] Once the target object is reached, it is assessed to determine if it can be safely manipulated (Project 1910). If the target object can be safely manipulated, the robot's manipulator, such as a bucket, is used to operate the robot to lift the object (Project 1912). At this point, the robot's cognitive module can be used to analyze the target object and nearby objects for better control of the manipulation (Project 1914).

[0153] Once the target object is on the bucket or other manipulator arm, it is secured (Item 1916). If the robot is unable to accommodate more objects, or if it is the last object of the selected (one or more) category, the object delivery mode is activated (Item 1918). Otherwise, the robot can restart the process (1902).

[0154] Figure 20 A robot control algorithm 2000 according to one embodiment is shown. At block 2002, such as regarding… Figures 7A to 7D The left and right cameras of the described cleaning robot, or some other configuration of the robot camera, can provide input that can be used to generate scale-invariant keypoints within the cleaning area.

[0155] In this disclosure, "scale-invariant keypoints" or "visual keypoints" refer to unique visual features that can be maintained across different viewpoints, such as photographs taken from different regions. This can be used to identify a feature or object within a region of an image of the captured area when the feature or object is captured from other images taken from different angles, at different scales, or using a different resolution than the original capture.

[0156] These scale-invariant key points can be detected using images captured by the robot's camera or the mobile device's camera, through an interface for a tidying robot or augmented reality robot installed on a mobile device. These scale-invariant key points can help the tidying robot or augmented reality robot interface on the mobile device determine the geometric transformations between camera views displaying matching content. This can help confirm or fine-tune estimates of the robot's or mobile device's position within the tidying area.

[0157] Keypoints with invariant scales can be detected, transformed, and matched using algorithms well-known in the field, such as (but not limited to) Invariant Scale Feature Transformation (SIFT), Speed-Up Robust Features (SURF), Oriented Robust Binary Features (ORB), and superpoints.

[0158] Objects located within the sorting area can be detected at box 2004 based on input from the left and right cameras, thus defining the starting position of the objects and classifying them into various categories. At box 2006, a re-identification fingerprint can be generated for the objects, where the re-identification fingerprint is used to determine the visual similarity between future detected objects and the objects. Future detected objects can be the same objects re-detected as part of an update or transformation of the global region mapping, or similar objects that will be similarly located in the future. The re-identification fingerprint can help classify objects more quickly.

[0159] At box 2008, the tidying robot can be positioned within the tidying area. Inputs from at least one of the left camera, right camera, LiDAR sensor, and inertial measurement unit (IMU) sensor can be used to determine the position of the tidying robot. The tidying area can be mapped to create a global region map that includes scale-invariant keypoints, objects, and their starting positions. At box 2010, objects within the tidying area can be re-identified based on at least one of starting position, category, and re-identification fingerprint. At box 2012, a persistent and unique identifier can be assigned to each object.

[0160] At box 2014, the tidying robot can receive camera feeds from an augmented reality robot interface installed as an application on a user-operated mobile device. Based on these feeds, it can update the global region map with keypoints whose starting positions and scales remain unchanged using a camera feed-to-global region map transformation. In this transformation, the global region map is searched to find a set of scale-invariant keypoints that match keypoints detected in the moving camera feed using a specific geometric transformation. This transformation maximizes the number of matching keypoints and minimizes the number of mismatched keypoints while maintaining geometric consistency.

[0161] At box 2016, user indicators can be generated for objects, where these indicators can include next target, target order, danger, too large, fragile, cluttered, and blocking the path. At box 2018, global region mapping and object details can be transferred to the mobile device, where these object details can include at least one of the object's visual snapshot, category, starting position, persistent and unique identifier, and user indicators. Figure 7C The communication 724 module introduced in the middle and Figure 24 The network interface 2402 introduced in the document supports the use of wireless signaling such as Bluetooth or Wi-Fi to transmit this information.

[0162] An augmented reality robot interface can be used to display updated global region maps, objects, starting positions, scale-invariant keypoints, and object details on mobile devices. The augmented reality robot interface accepts user input, where, at box 2020, user input indicates an object attribute overlay, including changing the object type, placing it next, not placing it, and modifying the user indicator. This object attribute overlay can be transferred from the mobile device to the organizing robot and can be used at box 2022 to update the global region map, user indicator, and object details. Returning to box 2018, the organizing robot can retransmit its updated global region map to the mobile device to resynchronize the information.

[0163] Figure 21 A robot control algorithm 2100 according to one embodiment is illustrated. At box 2102, the tidying robot can transmit a global region map representing the tidying area to an augmented reality robot interface installed as an application on a mobile device. The global region map may include object details related to objects disturbing the tidying area. The mobile device may include a camera. At box 2104, the augmented reality robot interface application can receive camera footage from the mobile device's camera. At box 2106, the camera footage can be used to detect objects in the tidying area and classify them into various categories.

[0164] At box 2108, a re-identification fingerprint can be generated for the object. This re-identification fingerprint can be used to determine the visual similarity between objects detected in the future. These future detected objects can be the same objects re-detected as part of an update or transformation of the global region mapping, or similar objects that will be similarly located in the future. In this case, the re-identification fingerprint can be used to help classify objects more quickly.

[0165] At box 2110, a ground plane can be detected in the camera view. At box 2112, a mobile device can be located on a global region map using at least one of the ground plane, object, object category, and re-identified fingerprint. At box 2114, an object can be re-identified based on at least one of the starting position, category, and re-identified fingerprint.

[0166] At box 2116, a user experience can be presented in an augmented reality robot interface, which displays a camera view, objects, and bounding boxes and user indicators overlaid on these objects. The augmented reality robot interface can accept user input based on user interaction with the user indicator. This user interaction can include clicking or tapping the user indicator. In one embodiment, the user indicator can include an action the user wants to take, and at box 2118, the augmented reality robot interface can detect actions taken by the user in the tidying area to complete those indicated tasks. In one embodiment, user actions can be detected through user confirmation from the augmented reality robot interface, changes recorded between previous and updated camera views, or based on information from the tidying robot's cameras and sensors. At box 2120, an indicator can be displayed to thank the user for their help.

[0167] At box 2122, the augmented reality robot interface can display available actions for an object based on user input (e.g., a user's click or tap on a user indicator). The augmented reality robot interface can then accept additional user input, selecting at least one of the available actions presented in box 2124. At box 2126, the augmented reality robot interface can send an object attribute overlay to the organizing robot based on the additional user input.

[0168] Figure 22 A tidying robot environment 2200 according to one embodiment is shown. The tidying robot environment 2200 may include a tidying area 2202, scale-invariant key points 2204, objects 2206 that scramble the area, a destination box 2208 where the objects 2206 can be placed, a mobile device 2210 that has an augmented reality robot interface application 2212 installed and is capable of providing camera views 2214, and a robot 700.

[0169] Robot 700 can be used Figure 7C and Figure 7D The sensors and cameras shown are used to detect features of the tidying area 2202. These features may include scale-invariant keypoints 2204, such as walls, corners, furniture, etc. The robot 700 can also detect objects 2206 and destination boxes 2208 on the floor of the tidying area 2202, where the objects 2206 can be placed based on categories, which the robot 700 can determine based on user input, recognition of similarity to previously processed objects, machine learning, or some combination of these. The robot 700 can also use its sensors and cameras to locate itself within the tidying area 2202. The robot 700 can synthesize all this data into a global area map 2216, such as regarding... Figure 20 As described.

[0170] The tidying robot's environment 2200 may include a user with a mobile device 2210, such as a tablet or smartphone. The mobile device 2210 may have an augmented reality robot interface application 2212 operating according to this disclosure installed. The augmented reality robot interface application 2212 may use a camera configured as part of the mobile device 2210 to provide a camera view 2214. The camera view 2214 may include a ground plane 2218 that can be identified and used to position the mobile device 2210 within the tidying robot's environment 2200, such that information about the tidying area 2202 detected by the robot 700 can be transformed according to a camera view to global area mapping transformation 2220 to allow the robot 700 and the mobile device 2210 to remain synchronized relative to objects 2206 in the tidying area 2202 and user indicators and object attribute overlays attached to those objects 2206, such as regarding... Figure 20 and Figure 21 As described, and about Figure 23A Further detailed description.

[0171] In one embodiment, the global region map 2216 may be a top-down two-dimensional representation of the sorting region 2202. The global region map 2216 may undergo a camera view to global region map transformation 2220, allowing information detected by the robot 700 to be represented from the user's perspective in the augmented reality robot interface application 2212. The global region map 2216 may be updated to include the mobile device position 2222, the sorting robot position 2224, and the object's starting position 2226. In one embodiment, the augmented reality robot interface application 2212 may also display the mobile device position 2222 and the sorting robot position 2224, which are not shown in this illustration.

[0172] Figure 23A An augmented reality robot interface 2300 according to one embodiment is shown at the start of a task. The augmented reality robot interface 2300 can be shown as follows: Figure 22 The sorting area 2202 introduced in the sorting robot environment 2200 includes objects 2206 to be picked up and destination bins 2208. As shown, bounding boxes 2302 and user indicators 2304 can be overlaid on the objects 2206. These can indicate the category 2306 of the objects 2206 and the actions that need to be taken by the user for items in the sorting area 2202. In one embodiment, indicators can be provided for the destination bins 2208 to indicate which category 2306 of objects 2206 are intended to be placed in each bin. The robot can make this decision dynamically or can be trained or instructed by the user on the bins for consistent use.

[0173] In the sorting area 2202 shown, objects 2206 and destination boxes 2208 can be categorized into types 2308 (generally unclassified), musical instruments 2310, stuffed animal toys 2312, plastic toys 2314, clothing 2316, and crafts 2318. The bounding box 2302 and other indicators associated with category 2306 may differ in color, line thickness and style, or other visually distinguishable features.

[0174] User indicator 2304 can identify the required actions before sorting begins, indicated by an exclamation mark. For example, if the destination box is overturned, lost, or misaligned, the user may need to put it back in its place. Unsorted objects, such as duffel bags and large, delicate objects like guitars, may need to be picked up by the user rather than the robot. Other user indicators 2320 can use numbers to indicate the order in which the robot 700 will begin picking up objects 2206 once the user has completed the required initial actions. For example, the user indicator 2320 shown indicates that the robot 700 intends to begin sorting by picking up the stuffed animal toy 2312 after the user's actions are completed. Users can clear the user indicator 2304 used for the initial actions by clicking it after they have completed their task. Users can adjust the order in which the robot 700 picks up objects 2206 by clicking the bounding box 2302 or the user indicator 2304 associated with the sorting order.

[0175] Figure 23B This indicates that user indicators 2304 and 2320 have been cleared. Figure 23A The augmented reality robot interface 2300 introduced in the code has completed the initial assistance actions requested by the robot. The box has been placed correctly, and the duffel bag (unsorted 2308) and guitar (instrument 2310) have been cleared from the sorting area 2202. The stuffed animal toys 2312 have been picked up by the robot 700 and stored in the destination boxes designated for them. The new user indicator 2322 indicates that the shirt (clothing 2316) is the next object to be picked up and placed in the designated box.

[0176] In one embodiment, when the user completes their task and observes the robot, the user can clear these user indicators 2304 and 2320. When the object 2206 is detected to have moved to an acceptable destination position in the camera view 2214, these user indicators 2304 and 2320 can be cleared through the augmented reality robot interface 2300. When the robot 700 no longer detects objects that the user is moving on its path, or when it picks up and stores objects that it intends to organize, or some combination thereof, these user indicators 2304 and 2320 can be cleared by the robot 700.

[0177] Figure 23C The augmented reality robot interface 2300 is shown after the user indicator 2322 for clothing 2316 has been cleared and clothing 2316 has been placed in its designated box. The next user indicator 2324 indicates the order in which plastic toys 2314 are picked up and placed in their designated boxes.

[0178] Figure 23D The augmented reality robot interface 2300 is shown after the user instruction 2324 for the plastic toy 2314 has been cleared and those items have been placed in their designated bins. The user instruction 2326 now instructs the robot 700 to pick up the craft supplies 2318 in the order they intend to stack them in the designated bins.

[0179] Figure 23E An augmented reality robot interface 2300 according to one embodiment is shown after all items have been picked up and placed in its box.

[0180] Figure 24 An embodiment of a robot control system 2400 is described to implement the components and process steps of the system described herein.

[0181] Input device 2404 (e.g., a robot or an accompanying device such as a mobile phone or personal computer) includes a transducer that converts physical phenomena into internal machine signals (typically electrical, optical, or magnetic signals). The signals can also be wireless, taking the form of electromagnetic radiation in the radio frequency (RF) range, but may also be electromagnetic radiation in the infrared or optical range. Examples of input device 2404 include a contact sensor that responds to touch or physical pressure from an object or proximity of an object to a surface; a mouse that responds to movement across space or across a plane; a microphone that converts vibrations in a medium (typically air) into device signals; and a scanner that converts optical patterns on a two-dimensional or three-dimensional object into device signals. Signals from input device 2404 are provided to memory 2406 via various machine signal conductors (e.g., a bus or network interface) and circuitry.

[0182] Memory 2406 is typically a known first or second-level memory device used to store (by means of the material's structure or state) signals received from input device 2404, instructions and information for controlling the operation of CPU 2408, and signals from storage device 2410. Memory 2406 and / or storage device 2410 may store computer-executable instructions, thus forming logic 2412 when applied to and executed by CPU 2408 implementing embodiments of the processes disclosed herein.

[0183] Information stored in memory 2406 is typically directly accessible by the device's CPU 2408. Signals input to the device cause a reconstruction of the internal material / energy state of memory 2406, essentially creating a new machine configuration, thereby influencing the behavior of the robot control system 2400 by constructing the CPU 2408 with control signals (instructions) and the data provided along with those control signals.

[0184] Secondary or tertiary storage devices 2410 can provide slower but higher capacity machine memory. Examples of storage devices 2410 are hard disks, optical disks, mass flash memory or other non-volatile memory technologies, and magnetic storage.

[0185] CPU 2408 can alter the configuration of memory 2406 via signals in storage device 2410. In other words, CPU 2408 can cause data and instructions to be read from storage device 2410 in memory 2406, potentially affecting the operation of CPU 2408 as instruction and data signals, and possibly providing data and instructions to output device 2414. CPU 2408 can alter the contents of memory 2406 by signaling to the machine interface of memory 2406 to change its internal configuration, and then relaying the signal to storage device 2410 to change its material internal configuration. In other words, data and instructions can be backed up from the generally volatile memory 2406 to the generally non-volatile storage device 2410.

[0186] The output device 2414 is a transducer that converts the signal received from the memory 2406 into physical phenomena, such as vibrations in the air, patterns of light on a machine display, or vibrations of ink or other materials (i.e., tactile devices) or patterns (i.e., printers and 3D printers).

[0187] Network interface 2402 receives signals from memory 2406 and converts them into electrical, optical, or wireless signals, which are typically transmitted to other machines via a machine network. Network interface 2402 also receives signals from the machine network and converts them into electrical, optical, or wireless signals for memory 2406.

[0188] List of reference numerals

[0189] 100 Robot Systems

[0190] 102 sensor

[0191] 104 camera

[0192] 106 Positioning Logic

[0193] 108 Mapping Logic

[0194] 110 Cognitive Logic

[0195] 112 Path Planner

[0196] 114 Pick Planner

[0197] 116 motion controller

[0198] 118 Motor and Servo Mechanism Controller

[0199] 120 Calibration and Object

[0200] 122 Local to Global Conversion

[0201] 124 Environment Mapping

[0202] 126 Object Tracking

[0203] 128 Current Status

[0204] 130 System Status Settings

[0205] 132 synchronization signal

[0206] 134 Motion Feedback

[0207] 136 navigation waypoints

[0208] 138 Local Cognition and Image Segmentation

[0209] 140 maneuvering actions

[0210] 142 Target Movement

[0211] 144 Image Segmentation Activation

[0212] 200 robots

[0213] 202 bucket

[0214] 204 manipulator actuator arm

[0215] 206 camera

[0216] 208 Bucket Actuator Arm

[0217] 210 manipulator brush

[0218] 302 seal

[0219] 304 convex part

[0220] 306 concave part

[0221] 308 extended back surface

[0222] 400 robots

[0223] 402 camera

[0224] 404 lifting actuator arm

[0225] 406 Collect Components

[0226] 408 bucket

[0227] 410 linear actuator

[0228] 500 robots

[0229] 502 bucket

[0230] 504 camera

[0231] 506 manipulator actuator arm

[0232] 508 collect components

[0233] 510 bucket actuator arm

[0234] 600 robots

[0235] 602 Bucket

[0236] 604 camera

[0237] 606 manipulator actuator arm

[0238] 608 collect components

[0239] 610 bucket actuator arm

[0240] 612 manipulator lifting arm

[0241] 700 robots

[0242] 702 chassis

[0243] 704 Mobile System

[0244] 706 Sensing System

[0245] 708 control system

[0246] 710 Capture and Containment System

[0247] 712 bucket

[0248] 714 Bucket Lifting Arm

[0249] 716 Bucket Lifting Arm Pivot Point

[0250] 718 control components

[0251] 720 manipulator arm

[0252] 722 Arm Pivot Point

[0253] 724 Communications Department

[0254] 726 front camera

[0255] 728 rear camera

[0256] 730 front lidar sensor

[0257] 732 rear lidar sensor

[0258] 734IMU sensor

[0259] 736 right front camera

[0260] 738 Front Left Camera

[0261] 740 rear left camera

[0262] 742 right rear camera

[0263] 744 Control Pivot Point

[0264] The 800a features a lowered bucket position and a lowered grabber position.

[0265] The 800b features a lowered bucket position and a raised grabber position.

[0266] The 800c raised bucket position and raised grabber position

[0267] 900 robots

[0268] 902 Bucket

[0269] 904 camera

[0270] 906 manipulator actuator arm

[0271] 908 collect components

[0272] 910 bucket actuator arm

[0273] 1000 Robot Process

[0274] 1002 frames

[0275] 1004 frames

[0276] 1006 frames

[0277] 1008 open-loop frame

[0278] 1010 frame

[0279] 1012 frames

[0280] 1014 frames

[0281] 1016 frames

[0282] 1018 frames

[0283] 1020 frame

[0284] 1022 frames

[0285] 1024 frames

[0286] 1026 frames

[0287] 1028 frames

[0288] 1100 Robot Process

[0289] Frame 1102

[0290] 1104 frame

[0291] 1106 frames

[0292] 1108 frames

[0293] 1110 frame

[0294] 1112 frame

[0295] 1114 frame

[0296] 1116 frame

[0297] 1118 frames

[0298] 1120 frames

[0299] 1122 frame

[0300] 1124 frames

[0301] 1126 frames

[0302] 1200 State Space Mapping

[0303] 1202 Sleep state

[0304] 1204 Environmental Exploration Status

[0305] 1206 Activation Action

[0306] 1208 scheduled start time

[0307] 1210 Exploration of Structural Intervals

[0308] 1212 No more objects to pick up

[0309] 1214 battery low power

[0310] 1216 Object aggregation state

[0311] 1218 Move items on the floor to group them by category

[0312] 1220 The object is too close to the floor.

[0313] 1222 The path of one or more objects is blocked.

[0314] 1224 Object Pickup Status

[0315] 1226 Environment mapping includes at least one delivery container for a class of objects.

[0316] 1228 There are accessible items for picking up in the container category.

[0317] 1230 No object ready to be picked up

[0318] 1232 picked up all the identified objects in the category.

[0319] The 1234 bucket is at full load.

[0320] 1236 Object Deployment Status

[0321] 1238 There are more items to pick up.

[0322] 1240 Incomplete environment mapping

[0323] 1700 Robot Control Algorithm

[0324] Frame 1702

[0325] 1704 frame

[0326] 1706 frame

[0327] 1708 frames

[0328] 1710 frame

[0329] 1712 frame

[0330] 1714 frame

[0331] 1716 frame

[0332] 1718 frame

[0333] 1720 frames

[0334] 1722 frames

[0335] 1800 Robot Control Algorithm

[0336] Frame 1802

[0337] 1804 frame

[0338] 1806 frame

[0339] 1808 frame

[0340] 1810 frame

[0341] 1812 frames

[0342] 1814 frames

[0343] 1816 frame

[0344] 1818 frame

[0345] 1820 frames

[0346] 1822 frames

[0347] 1900 Robot Control Algorithm

[0348] Project 1902

[0349] Project 1904

[0350] Project 1906

[0351] Project 1908

[0352] Project 1910

[0353] Project 1912

[0354] Project 1914

[0355] Project 1916

[0356] Project 1918

[0357] 2000 Robot Control Algorithm

[0358] 2002 frame

[0359] 2004 frame

[0360] 2006 frame

[0361] 2008 frame

[0362] 2010 frame

[0363] 2012 frame

[0364] 2014 frame

[0365] 2016 frame

[0366] 2018 frame

[0367] 2020 frame

[0368] 2022 frame

[0369] 2100 Robot Control Algorithm

[0370] 2102 frame

[0371] 2104 frame

[0372] 2106 frame

[0373] 2108 frame

[0374] 2110 frame

[0375] 2112 frame

[0376] 2114 frame

[0377] 2116 frame

[0378] 2118 frame

[0379] 2120 frame

[0380] 2122 frame

[0381] 2124 frames

[0382] 2126 frames

[0383] 2200 Cleaning Robot's Environment

[0384] 2202 Cleaning Area

[0385] The key point of keeping the 2204 ratio unchanged

[0386] 2206 objects

[0387] 2208 Destination Box

[0388] 2210 mobile devices

[0389] 2212 Augmented Reality Robot Interface Application

[0390] 2214 camera footage

[0391] 2216 Global Region Mapping

[0392] 2218 grounding plane

[0393] 2220 camera view to global area mapping conversion

[0394] 2222 mobile device locations

[0395] 2224 Organize robot positions

[0396] 2226 Initial position of the object

[0397] 2300 Augmented Reality Robot Interface

[0398] 2302 bounding box

[0399] 2304 User Indicator

[0400] Category 2306

[0401] 2308 Uncategorized

[0402] 2310 Musical Instruments

[0403] 2312 Stuffed Animal Toys

[0404] 2314 Plastic Toys

[0405] 2316 Clothing

[0406] 2318 Crafts and Supplies

[0407] 2320 User Indicator

[0408] 2322 User Indicator

[0409] 2324 User Indicator

[0410] 2326 User Indicator

[0411] 2400 Robot Control System

[0412] 2402 Network Interface

[0413] 2404 Input Device

[0414] 2406 memory

[0415] 2408 CPU

[0416] 2410 storage device

[0417] 2412 Logic

[0418] 2414 output device

[0419] The various functional operations described herein can be implemented in logic using nouns or noun phrases that reflect the operation or function. For example, an association operation can be performed by a “correlator” or “correlator”. Similarly, switching can be performed by a “switch”, selection by a “selector”, and so on. “Logic” refers to machine memory circuitry and non-transitory machine-readable media that includes machine-executable instructions (software and firmware) and / or circuitry (hardware), constructed by its material and / or material energy, including control and / or program signals, and / or settings and values ​​(such as resistance, impedance, capacitance, inductance, current / voltage ratings, etc.), which can be applied to affect the operation of a device. Magnetic media, electronic circuits, electrical and optical memories (including volatile and non-volatile) and firmware are examples of logic. Logic explicitly excludes pure signals or software itself (however, it does not exclude machine memory that includes software and thus forms a material structure).

[0420] In this disclosure, different entities (which may be referred to differently as “cells,” “circuits,” other group components, etc.) may be described or claimed to be “constructed” to perform one or more tasks or operations. The formula—[entity] constructed to [perform one or more tasks]—is used herein to refer to a structure (i.e., a physical thing such as an electronic circuit). More specifically, the formula is used to indicate that the structure is arranged to perform one or more tasks during operation. A structure may be said to be “constructed” to perform some tasks even if the structure is not currently being operated. “Credit allocation circuitry constructed to allocate credits to multiple processor cores” is intended to cover, for example, an integrated circuit having circuitry that performs that function during operation, even if the integrated circuit in question is not currently in use (e.g., power is not connected to it). Therefore, an entity described or stated as “constructed” to perform a task refers to a physical thing, such as a device, circuit, memory storing program instructions executable to perform that task, etc. This phrase is not used herein to refer to intangible things.

[0421] The term “constructed” does not mean “constructable”. For example, an unprogrammed FPGA is not considered “constructed” to perform certain functions, although it may be “constructable” to perform those functions after programming.

[0422] The statement in the appended claims that a structure is “constructed” to perform one or more tasks is expressly intended not to invoke 35 U.SC §112(f) for that claim element. Therefore, claims in this application that do not additionally include means “for” [performing the function] should not be interpreted in accordance with 35 U.SC §112(f).

[0423] As used herein, the term "based on" is used to describe one or more factors that influence a determination. This term does not exclude the possibility that other factors may influence the determination. That is, a determination may be based solely on the specified factors, or on the specified factors and other unspecified factors. Consider the phrase "A is determined based on B." This phrase specifies that B is a factor used to determine A or that influences the determination of A. This phrase does not exclude the possibility that A may also be determined based on some other factors such as C. This phrase is also intended to cover embodiments in which A is determined solely based on B. As used herein, the phrase "based on" is synonymous with the phrase "at least partially based on."

[0424] As used herein, the phrase "in response to" describes one or more factors that trigger an effect. This phrase does not exclude the possibility that other factors may influence or otherwise trigger the effect. That is, an effect may be a response to these factors alone, or it may be a response to a specific factor as well as other unspecified factors. Consider the phrase "A is executed in response to B." This phrase specifies that B is the factor that triggers the execution of A. This phrase does not exclude the possibility that A may also be executed in response to some other factor such as C. This phrase is also intended to cover embodiments in which A is executed only in response to B.

[0425] As used herein, the terms “first,” “second,” etc., are used as labels for the nouns preceding them and do not imply any type of ordering (e.g., spatial, temporal, logical, etc.) unless otherwise stated. For example, in a register file with eight registers, the terms “first register” and “second register” can be used to refer to any two of the eight registers, not just logical registers 0 and 1.

[0426] When used in the claims, the term "or" is used as an inclusive "or" rather than an exclusive "or". For example, the phrase "at least one of x, y, or z" means any one of x, y, and z, and any group combination thereof.

[0427] As used herein, the phrase “and / or” relating to two or more elements should be interpreted as referring to only one element or a group of elements. For example, “element A, element B, and / or element C” can include only element A, only element B, only element C, element A and element B, element A and element C, element B and element C, or element A, B, and C. Furthermore, “at least one of element A or element B” can include at least one of element A, at least one of element B, or at least one of element A and element B. Additionally, “at least one of element A and element B” can include at least one of element A and at least one of element B.

[0428] The subject matter of this disclosure is specifically described herein to satisfy legal requirements. However, this description itself is not intended to limit the scope of this disclosure. Rather, the inventors have considered that the claimed subject matter may also be embodied in other ways to incorporate other current or future techniques, including different steps or groups of steps similar to those described herein. Furthermore, although the terms “step” and / or “box” may be used herein to imply different elements of the method employed, these terms should not be construed as implying any particular order between the various steps disclosed herein, unless and only if the order of the various steps is explicitly described.

[0429] Having described the illustrative embodiments in such detail, it will be apparent that modifications and variations are possible without departing from the scope of the claimed invention. The scope of the subject matter is not limited to the depicted embodiments, but is set forth in the following claims.

Claims

1. A method for operating a robot, comprising: Associate each of the multiple object categories of an object in the environment with a corresponding container located in the environment; Activate the robot at the base station; The robot navigates around the environment using a camera to map the type, size, and location of the objects. For each object category: Select one or more objects from the category to pick; Perform path planning from the robot's current position to one or more of the objects to be picked up; Navigate to a point near one or more of the objects to be picked; An actuating manipulator, the manipulator being coupled to open and close at the front of the robot across the front of the bucket to remove obstacles and manipulate the one or more objects onto the bucket; The actuator includes a first actuator component and a second actuator component. The first actuator component is connected to a first actuator pivot point adjacent to a first front side of the bucket, and the second actuator component is connected to a second actuator pivot point adjacent to a second front side of the bucket. Wherein, the first operator pivot point allows the first operator member to open and close across the first front side of the bucket, while the second operator pivot point allows the second operator member to open and close across the second front side of the bucket; One or both of tilting or raising the bucket and actuating the manipulator to hold the object in the bucket; Navigate the robot to the corresponding container of the adjacent category; Align the rear end of the robot with the side of the corresponding container; and The bucket is raised above the robot and toward its rear end along an arc-shaped path above the robot's chassis, from the front end to the rear end, to store the held object in the corresponding container.

2. The method according to claim 1, characterized in that, The method further includes: The robot is operated to gather the objects in the environment into clusters, wherein each cluster comprises only objects from one of the categories.

3. The method according to claim 1, characterized in that, The method further includes: Operate at least one first arm to actuate the robot's manipulators to remove obstacles and manipulate the one or more objects onto the bucket; and Operate at least one second arm to tilt or raise the bucket.

4. The method according to claim 1, characterized in that, Each first arm is paired with a corresponding second arm, and also includes: Operate each pair of first and second arms from a common starting pivot point.

5. The method according to claim 1, characterized in that, Actuating the robot's manipulator to remove an obstacle includes actuating the manipulator to form a wedge in front of the bucket.

6. The method according to claim 5, characterized in that, Actuating the manipulator to hold the object in the bucket includes actuating the manipulator to form a barrier in front of the bucket.

7. The method according to claim 1, characterized in that, The method further includes: The neural network is operated to determine the type, size, and location of the object from images from the camera.

8. The method according to claim 1, characterized in that, The method further includes: Based on inputs from the left and right cameras, scale-invariant keypoints are generated within a clean area of ​​the environment. The position of the object in the sorting area is detected based on the input from the left camera and the right camera, thereby defining the starting position; The object is classified into the category; Generate a re-identification fingerprint of the object, wherein the re-identification fingerprint is used to determine the visual similarity between the objects; The robot is positioned within the sorting area based on inputs from at least one of the left camera, the right camera, a light detection and ranging (LIDAR) sensor, and an inertial measurement unit (IMU) sensor to determine the robot's position. Mapping the sorted area to create a global region map, the global region map including the scale-invariant keypoints, the object, and the starting position; and The object is re-identified based on at least one of the starting location, the category, and the re-identified fingerprint.

9. The method according to claim 8, characterized in that, The method further includes: Assign a persistent and unique identifier to the object; Receive camera feeds from the augmented reality robot interface installed as an application on the mobile device; Based on the camera view, a global region mapping transformation is used to convert the view to a global region, and the global region mapping is updated using the starting position and key points with unchanged scale; and Generate indicators for the object, wherein the indicators include one or more of the following: next target, target order, danger, too large, fragile, messy, and blocking the path of travel.

10. The method according to claim 9, characterized in that, The method further includes: The global region map and object details are transmitted to the mobile device, wherein the object details include at least one of the object's visual snapshot, the category, the starting position, the persistent and unique identifier, and the indicator; The augmented reality robot interface is used to display the updated global region map, the object, the starting position, the scale-invariant key points, and the object details on the mobile device. The system receives input from the augmented reality robot interface, where the input indicates object attribute overriding, including changing the object category, placing it next, not placing it, and modifying the user indicator. The object attribute overlay is sent from the mobile device to the robot; and The global region mapping, the indicator, and the object details are updated based on the object's attributes.

11. A robot system, comprising: robot; Base station; Multiple containers, each associated with one or more object categories; Mobile applications; as well as Logic, the logic being used for: The robot navigates around an environment containing multiple objects to map the type, size, and location of the objects; For each of the categories: Select one or more objects from the category to pick; Perform path planning on the object to be picked; Navigate to the point adjacent to each of the objects to be picked; An actuating manipulator, the manipulator being coupled to open and close at the front end of the robot across the front end of the bucket to remove obstacles and manipulate the one or more objects onto the bucket; The actuator includes a first actuator component and a second actuator component. The first actuator component is connected to a first actuator pivot point adjacent to a first front side of the bucket, and the second actuator component is connected to a second actuator pivot point adjacent to a second front side of the bucket. Wherein, the first operator pivot point allows the first operator member to open and close across the first front side of the bucket, while the second operator pivot point allows the second operator member to open and close across the second front side of the bucket; Tilting or raising the bucket and actuating the manipulator to hold the object to be picked up in the bucket, or both; Navigate the robot to the corresponding container of the adjacent category; Align the rear end of the robot with the side of the corresponding container; and The bucket is raised above the robot and toward its rear end along an arc-shaped path above the robot's chassis, from the front end to the rear end, to store the held object in the corresponding container.

12. The robot system according to claim 11, characterized in that, It also includes logic for operating the robot to cluster the objects in the environment into clusters, wherein each cluster includes only objects from one of the categories.

13. The robot system according to claim 11, characterized in that, The robot includes at least one first arm and at least one second arm, and the system further includes: The logic for actuating the manipulator of the robot to remove obstacles and push the one or more objects onto the bucket and operate at least one second arm to tilt or raise the bucket.

14. The robot system according to claim 11, characterized in that, Each first arm is paired with a corresponding second arm, and each pair of first and second arms has a common starting pivot point.

15. The robot system according to claim 11, characterized in that, It also includes logic for actuating the manipulator of the robot to form a wedge in front of the bucket.

16. The robot system according to claim 15, characterized in that, It also includes logic for actuating the actuator to form a closed barrier in front of the bucket.

17. The robot system according to claim 11, characterized in that, The system also includes: A neural network configured to determine the type, size, and location of an object from an image from a camera.

18. The robot system according to claim 11, characterized in that, Also includes for: Based on inputs from the left and right cameras, scale-invariant keypoints are generated within a clean area of ​​the environment. The position of the object in the sorting area is detected based on the input from the left camera and the right camera, thereby defining the starting position; The object is classified into the category; Generate a re-identification fingerprint of the object, wherein the re-identification fingerprint is used to determine the visual similarity between the objects; The robot is positioned within the sorting area to determine its location; and The sorted area is mapped to create a global region map, which includes the scale-invariant keypoints, the object, and the starting position.

19. The robot system according to claim 18, characterized in that, Also includes for: The object is re-identified based on at least one of the starting location, the category, and the re-identified fingerprint.

20. The robot system according to claim 19, characterized in that, Also includes for: The object is classified as one or more of the following: dangerous, too large, fragile, and messy.

Images