Mapping with static fraction for objects

Abstract

An exemplary system includes: a sensor for acquiring information about objects in an environment; and one or more processing devices configured to use the information when generating or updating a mapping of the environment. The mapping includes the objects and boundaries or landmarks in the environment. The mapping includes a static score associated with the object. The static score represents the probability that the object will remain stationary within the environment. This probability may lie between a specific degree of immobility and a specific degree of mobility.

Description

Technical Field

[0001] This specification relates in its entirety to exemplary systems configured to control autonomous vehicle operations (such as mapping, route planning, and navigation) using a score representing the possible mobility of elements. Background Technology

[0002] Autonomous vehicles, such as mobile robots, are configured to move within spaces such as warehouses. For example, an autonomous vehicle can create a mapping of the space and plan a route through it. During its movement along the route, the autonomous vehicle can determine its position within the space and use that position to control its future movements. Summary of the Invention

[0003] An exemplary system includes: a sensor for acquiring information about objects in an environment; and one or more processing devices configured to use the information when generating or updating a mapping of the environment. The mapping includes the objects and boundaries or landmarks in the environment. The mapping includes a static score associated with the object. The static score represents the probability that the object will remain immobile within the environment. This probability may lie between a specific immobility and a specific mobility. The exemplary system may include one or more of the following features (alone or in combination).

[0004] The one or more processing devices may be configured to use the mapping map to plan a route through the environment based at least in part on the static score. Planning the route may include determining a path through the environment, wherein the path avoids the object if the object has a static score below a predefined threshold. The route may include segments containing the object, and planning the route may include assigning weights to the segments at least in part based on the object's static score.

[0005] The system may include an autonomous vehicle configured to perform localization at least in part based on the static score while traversing the environment. At least one of the sensors or processing devices may be included on the autonomous vehicle. Alternatively, at least one of the sensors or processing devices may not be included on the autonomous vehicle. The static score may be associated with at least some of the boundaries or landmarks to indicate that the at least some of the boundaries or landmarks have a specific immobility. Performing localization may include using the static score of the at least some of the boundaries or landmarks to determine the position of the autonomous vehicle within the environment. Performing localization may also include using the static score of the object to determine the position of the autonomous vehicle within the environment.

[0006] The mapping map may include a plurality of objects, the plurality of objects including the objects. The mapping map may include static scores associated with each of the plurality of objects. The system may be or may include a mapping map generation system. The system may be or may include a route planning system. The static scores may be associated with at least some of the boundaries or landmarks representing a particular immobility. The system may include an autonomous vehicle configured to perform localization at least in part based on the static scores while traveling through the environment. Operations for performing localization may include using the static scores of the at least some of the boundaries or landmarks to determine first information about the location of the autonomous vehicle within the environment. Operations for performing localization may include using the static scores of the objects to determine second information about the location of the autonomous vehicle within the environment, if additional information is desired to establish the location of the autonomous vehicle within the environment. The second information may enhance the first information to establish the location of the autonomous vehicle within the environment. The additional information to establish the location of the autonomous vehicle within the environment may be desired because the first information is insufficient to establish the location. Operations for performing localization may include using a particle filter to generate particles in the environment. The particles may correspond to potential locations for the robot's future travel. Each of the particles can be weighted. Operations for performing positioning may include using the static score of the object to change the weight of at least one of the particles.

[0007] The static score of the object may be based at least in part on one or more times of the day in which the information is obtained. The static score of the object may be based at least in part on the location of the object within the environment. The static score of the object may be based at least in part on the pose of the object within the environment. The static score may be based at least in part on data about the object provided from an external source.

[0008] The one or more processing devices may be configured to implement a machine learning system, the machine learning system including a model capable of being trained to adjust the static score based on one or more conditions that change over time. The system may include a memory storing a database containing attributes of objects. The one or more processing devices may be configured to identify the object based on the attributes in the database and to assign the static score based on the identification of the object. The static score associated with the object may be an updated version of an earlier static score assigned to the object. The updated version of the earlier static score may be based on the sensor detecting the object at a slightly later time than when the earlier static score was assigned.

[0009] The system may include an autonomous vehicle configured to perform localization at least in part based on the static score while traversing the environment. The one or more processing devices may be on the autonomous vehicle and may include: a first processing device for receiving the information from the sensors; and a second processing device for identifying the object based on the information and assigning the static score to the object. The first processing device may be configured to generate the mapping map. The second processing device may have at least one of a higher clock frequency or a longer word size than the first processing device.

[0010] The sensor may be part of a system comprising multiple sensors. The multiple sensors may include: a light detection and ranging (LIDAR) sensor for detecting first data representing the boundary; and a camera for detecting second data representing the object. The information may include the second data. The one or more processing devices may be configured to use the first data to generate at least a portion of the mapping map. The multiple sensors may include: multiple LIDAR sensors, including the LIDAR sensor; and multiple cameras, such as multiple 2D cameras or multiple 3D cameras.

[0011] The system may include an autonomous vehicle configured to perform localization at least in part based on the static score while traveling through the environment. The LIDAR sensor and the camera may be mounted on the autonomous vehicle and guided to view at least a portion of the same area ahead of the autonomous vehicle in the direction of travel. At least one of the plurality of sensors may be mounted in the environment but not on the autonomous vehicle. At least one of the plurality of sensors may be mounted on a second autonomous vehicle, different from the first autonomous vehicle.

[0012] The static score of the object may change over time. The static score may have a value between a first value and a second value, where the first value represents a specific immobility of the object in the environment over a certain time period, and the second value represents a specific mobility of the object in the environment over the time period.

[0013] An exemplary method is used with an autonomous vehicle configured to operate within an environment. The method includes obtaining information about objects in the environment and using that information to generate or update a mapping of the environment. The mapping includes the objects and boundaries or landmarks within the environment. The mapping includes a static score associated with the object. The static score may represent the probability that the object will remain immobile within the environment. This probability may lie between a specific immobility and a specific mobility. One or more non-transitory machine-readable storage media may store instructions that can be executed by one or more processing devices on an autonomous vehicle configured to operate within the environment. The instructions are executable to perform the method. The exemplary method or instructions may include one or more of the preceding features (alone or in combination).

[0014] Any two or more of the features described in this specification (including this summary section) may be combined to form specific embodiments not specifically described herein.

[0015] The exemplary systems, vehicles, and processes described herein, or a portion thereof, may be implemented or controlled by a computer program product comprising instructions stored on one or more non-transitory machine-readable storage media, and such instructions may be executed on one or more processing devices to control (e.g., coordinate) the operations described herein. The exemplary systems, vehicles, and processes described herein, or a portion thereof, may be implemented using a device or electronic system, which may include one or more processing devices and memory storing executable instructions for implementing various operations.

[0016] Detailed descriptions of one or more specific embodiments are set forth in the accompanying drawings and the following detailed description. Other features and advantages will become apparent from the details, the drawings, and the claims. Attached Figure Description

[0017] Figure 1 This is a side view of an exemplary autonomous vehicle, showing possible internal and external components of an exemplary control system for an autonomous vehicle.

[0018] Figure 2 This is a perspective view of an exemplary autonomous vehicle showing an example of the placement of sensors and the sensor range provided by these sensors.

[0019] Figure 3 It is a top view of an exemplary mapping map that can be used for route planning and navigation of autonomous vehicles traveling through space.

[0020] Figure 4 This is a flowchart illustrating operations that may be included in an exemplary process for generating a mapping map.

[0021] Figure 5 yes Figure 3 A top view of a portion of an exemplary mapping map.

[0022] Similar reference numerals in different figures indicate similar elements. Detailed Implementation

[0023] This article describes an exemplary system configured to control autonomous vehicle operations (such as mapping, route planning, and navigation) using scores representing the mobility levels of environmental elements. Examples of such elements include environmental boundaries (such as walls or doorways), environmental landmarks (such as support pillars and stairs), and movable or immovable objects in the environment (such as boxes or furniture).

[0024] In this respect, in environments such as confined spaces, elements may be unable to move or may be able to move. Those elements that cannot be moved are highly static and are therefore assigned a high score. For example, a support column in a warehouse could be assigned a score indicating that the column is 100% static. In this example, the system could assign a "static score" of 1 to the column, indicating that the column is immovable and therefore 100% static. For objects that can move voluntarily or involuntarily (such as people or furniture), the system could assign a static score representing the probability that the object will remain stationary within the environment for at least a certain period of time.

[0025] In some examples, the static fractions assigned to elements in a space are not binary, but rather represent values ​​somewhere between a particular immobility and a particular mobility. For example, a container located on a warehouse floor could be assigned a static fraction indicating that there is a 60% chance the container will remain immobile for at least a certain period of time. In this example, the system could assign the container a static fraction of 0.6. Therefore, the static fraction has a value between a first value representing a particular immobility (such as 1) and a second value representing a particular or expected mobility (such as 0). The system assignment of static fractions can be based on external input or can be automatically generated by the vehicle's control system.

[0026] Static scores can be incorporated into the spatial mapping. For example, elements in the space (such as objects, boundaries, and landmarks) are assigned static scores in the mapping based on their probability of remaining stationary within the space. Thus, in the previous example, the pillar is assigned a static score of 1 in the mapping, and the container is assigned a static score of 0.6 within the space. These static scores can be associated with the representation of objects in the mapping.

[0027] The mapping map can be used by the system for route planning. Route planning may include determining routes through space to a destination. Static scores of elements in space can inform the system which routes are preferred. For example, if a container blocks a route for an autonomous vehicle (such as a mobile robot) to its destination, and its static score (0.8) indicates that the container is likely to remain on that route, the system can determine which different routes through space are preferred. Therefore, in this example, the system considers elements in space, their positions, and their static scores to determine the preferred routes.

[0028] Static scores are also used in space during localization (also known as navigation). In one example, after determining a preferred route, a mobile robot (“robot”) begins to move through space along a route located on a mapping map. During this movement, the robot periodically or intermittently determines its position, orientation, or both position and orientation in space. This information allows the robot to confirm it is on the route, determine its position on the route, and determine if route correction is needed to reach its destination. The robot uses elements in space to determine its position along the route by comparing elements detected by the robot using one or more sensors with the expected positions of those same elements on the mapping map. In some implementations, the robot assigns primacy (e.g., a score of 1) to elements considered completely static during localization based on their static scores. However, if additional information is needed, the robot can use other elements with static scores less than 1. For example, the robot can use elements with static scores less than 1 (such as 0.6) for localization. In this example, the robot can also identify elements in space, compare these elements with elements located on the mapping map, and determine its position based on those elements if they have static scores greater than 0.6.

[0029] The aforementioned operations can be performed by the robot's control system. The control system may include hardware, software, or both hardware and software to implement a mapping generation system, a route planning system, and a positioning system. In some implementations, all or part of the control system may be "onboard," meaning that all or part of the control system is located on the robot itself. In some implementations, at least a portion of the control system may be remote, meaning that at least a portion of the control system is not located on the robot itself.

[0030] An example of a robot configured to operate based on static scores is... Figure 1Robot 10. In this example, robot 10 is a mobile robot and is referred to as "robot 10" or "robot". Robot 10 includes a body 12 with wheels 13 to enable robot 10 to move on a surface 14 (such as the floor of a warehouse, factory, or other area). Robot 10 includes a support area 15 configured to support the weight of an object. In this example, robot 10 can be controlled to transport an object from one location to another. Robot 10 includes various detectors (also called sensors) for detecting elements near the robot. In some examples, elements may include living objects, inanimate objects, boundaries, or landmarks.

[0031] In this example, robot 10 includes different types of visual sensors, such as a 3D camera, a 2D camera, and a light detection and ranging (LIDAR) scanner. A 3D camera is also called an RGBD camera, where R corresponds to red, G to green, B to blue, and D to depth. The 3D camera can be configured to capture video, still images, or both. It is important to note that the robot is not limited to this configuration or the use of these specific types of sensors. For example, the robot may include a single sensor, a single type of sensor, or more than two types of sensors.

[0032] See Figure 2 The robot 10 includes a 3D camera 16 at its front 17. In this example, the front of the robot faces the direction of travel. The rear of the robot faces the terrain it has already traversed. The robot 10 also includes a LiDAR scanner 19 at its front. Since the LiDAR scanner is 2D, it will detect elements in a plane 20 in the space through which the robot is controlled. Since the camera is 3D, it will detect elements in a 3D volume 21 in the space through which the robot is controlled. The LiDAR scanner 19 is adjacent to the 3D camera 16 and points in the same general direction as the camera. Similarly, the 3D camera 16 is adjacent to the LiDAR scanner 19 and points in the same general direction as the LiDAR scanner. For example, the LiDAR scanner may be positioned just below the 3D camera, or the 3D camera may be positioned just below the LiDAR scanner, as... Figure 2 As shown in the example. In this configuration, both the 3D camera and the LIDAR scanner are configured to view at least a portion of the same area 22 in front of the robot during movement. The front of the robot may contain multiple 3D camera / LIDAR scanner assemblies, although only one is shown. The robot 10 may also include one or more 3D camera / LIDAR scanner assemblies 23 located at its rear 24. The robot 10 may also include one or more 3D camera / LIDAR scanner assemblies (not shown) located on its sides. Each 3D camera / LIDAR scanner may be configured to view a portion of the same area.

[0033] A 2D camera can be used as a replacement for or supplement to a 3D camera on robot 10. For example, in all the instances described herein, one or more 2D cameras may replace a 3D camera. To obtain 3D data of a region, two or more 2D cameras may be pointed at the same region and capture relevant 2D data to obtain 3D data. In the examples above, one or more 2D cameras and a LiDAR scanner may be configured to view at least a portion of the same region 22 in front of the robot during travel. Similarly, the 2D camera may be located at the rear or side of the robot.

[0034] In this regard, additional or alternative sensors may be used in some specific implementations. For example, the robot may include one or more one-dimensional (single-beam) optical sensors, one or more two-dimensional (2D) (scanning) laser rangefinders, one or more 3D high-definition LiDAR sensors, one or more 3D floodlight LiDAR sensors, one or more 2D or 3D sonar sensors, and / or one or more 2D cameras. Combinations of two or more of these types of sensors may be configured to detect both 3D and 2D information in the same area on the front, back, or side of the robot.

[0035] One or more sensors can be configured to continuously detect the distance between the robot and elements nearby. This allows for collision avoidance and safe guidance of the robot around or between detected objects. As the robot moves along the path, the onboard computing system continuously receives input from the sensors. If an obstacle obstructs the robot's trajectory, the onboard computing system is configured to plan a path around the obstacle. If an obstacle is predicted to obstruct the robot's trajectory, the onboard computing system is configured to plan a path around the obstacle.

[0036] A LIDAR scanner, a 3D camera, and / or any other sensors on the robot constitute the robot's vision system. Visual data acquired by the vision system can be used to create a mapping of space by moving the robot through the space. As described above, each mobile robot traversing space may include such a vision system and may contribute data (such as visual data) for the creation of the mapping.

[0037] Sensors (such as 3D cameras, LiDAR scanners, or other types described herein) can be mounted in areas where the mobile device will travel. For example, such sensors can be mounted on walls, floors, ceilings, or other structures within the area, and along the routes the robot may take between two or more locations. Figure 2In this example, sensor 26 is mounted on wall support beam 27. Information from these sensors, which are not located on the robot, can help create a mapping map. For example, information from these sensors can be wirelessly transmitted to the robot's control system and used by the control system to create a mapping map or to update the mapping map periodically, intermittently, sporadically, or in response to receiving new data. In some implementations, upon encountering a sensor, onboard components of the control system can query the sensor to obtain information about elements in its vicinity.

[0038] In some implementations, data from a LiDAR sensor is used to create a portion of the mapping map. In these implementations, at least a portion of the mapping map is 2D. In some implementations, a combination of data from a 3D camera and data from a LiDAR sensor is used to create a portion of the mapping map. In these implementations, at least a portion of the mapping map is 3D. Figure 3 An example of a mapping map containing 2D and 3D information is shown.

[0039] Exemplary control systems for robots implement operations associated with the robot, such as mapping generation, route planning, and localization. In some implementations, the control system stores spatial mappings in computer memory (“memory”). The mappings may be stored in the memory on each robot or at any location accessible to the control system and the robot. For example, the mappings may be stored at a remote computing system, such as a swarm management system. For instance, the mappings may be stored at a remote server accessible to the robot, the control system, and / or the swarm management system. In some examples, remote access may include wireless access, such as via a computer network or a direct wireless link.

[0040] refer to Figure 3 The mapping map 30 may define the boundaries of the space 31 traversed by the robot, such as walls 29 and doorways. The mapping map may include the locations of landmarks (such as pillars 12, corners, windows, poles, and other distinguishable permanent and non-permanent features of the space that serve as references for the robot during positioning). The mapping map may include objects within the space, such as goods, containers 33, humans, or animals 34. The mapping map may also include measurements indicating the dimensions of the space, measurements indicating the dimensions and locations of objects, boundaries, and landmarks, measurements indicating the distances between different objects, boundaries, and landmarks, and coordinate information identifying where objects, boundaries, and landmarks are located in the space.

[0041] As described above, in some embodiments, each robot may locally store its own copy of the mapping map. Information obtained from sensors on and / or off the robot can be used to adjust the static scores of elements on the mapping map. In some embodiments, a remote control system may transmit updates to the mapping map, a completely new mapping map containing the updates, or information identifying selected routes independent of the mapping map to the robot. Processing devices on the robot may receive the information and use it to update the mapping map and / or control its movement along the selected route.

[0042] refer to Figure 1 In some examples, the control system may include onboard components 32 (such as onboard circuitry, onboard computing systems, or onboard controllers) to perform mapping generation, route planning and localization, and other operations of the robot. Onboard components may include, for example, one or more processing devices, such as one or more microcontrollers, one or more microprocessors, programmable logic such as field-programmable gate arrays (FPGAs), one or more application-specific integrated circuits (ASICs), solid-state circuits, or any suitable combination of two or more of these types of electronic components.

[0043] In some embodiments, the onboard components of the control system may include a first processing device 34 for receiving information from sensors and generating a mapping map based at least in part on that information. The onboard components may also include a second processing device 35 for receiving sensor information from the first processing device to identify elements in the space / environment based on the information, generating static scores, and assigning the static scores to all or some of the elements. These static scores are then sent to the first processing device for inclusion in the mapping map. In this regard, the second processing device is configured to perform any machine learning or artificial intelligence process described herein to generate static scores. The first processing device may be a first microprocessor, and the second processing device may be a second microprocessor with a higher clock frequency or word size than the first microprocessor. More generally, in some embodiments, the second microprocessor may be more powerful than the first microprocessor. In some embodiments, the first and second processing devices may be two different cores within a single microprocessor. In some embodiments, the functions attributed to the first and second processing devices may be performed by a single microprocessor or other single-die device.

[0044] In some embodiments, the onboard components of the control system may communicate with a remote computing system 38 (such as a swarm management system). The computing system is remote, meaning it is not located on the robot. Components of the remote computing system may be located in the same geographical location or, for example, distributed across different geographical locations. Components of the remote computing system may be distributed among different robots in space. In one example, commands provided by the remote computing system can be transmitted for execution by the robot's onboard components. In some embodiments, the control system includes only onboard component 32. In some embodiments, the control system 40 includes a combination of onboard component 32 and remote computing system 38.

[0045] Onboard or remote components of a robot's control system can be configured to receive input from a user, either manually or wirelessly. In some implementations, the control system can be configured (e.g., programmed) to perform control functions, such as mapping generation, route planning, and localization, without local or remote input from a user. In some implementations, the control system can be configured to perform control functions, such as mapping generation, route planning, and localization, at least in part based on input from a user.

[0046] Therefore, to summarize, the robot's control system 40 can be located on the robot itself, distributed across various locations or devices, or positioned at a stationary location away from the robot. The control system can be implemented using one or more processing devices 34, 35 on the robot. The control system can be implemented using one or more processing devices on the robot and on one or more other robots (not shown) that are moving or have already moved in the same space as the robot. The control system can be implemented using one or more processing devices at a stationary location in space, separate from all other robots. The control system can be implemented using one or more processing devices on the robot, on one or more other robots, and / or at a stationary location.

[0047] A swarm management system, implementable on a remote computing system 38, can be configured to control one or more robots and perform at least some of the functions described herein. Each of the swarm management system and the robots may include a copy of the same spatial mapping map or have access to it. The swarm management system can be configured to receive updates on the actual position and operational status of each robot in the swarm and to receive prioritized operation and transport requests from the user. The swarm management system can be configured to perform global route planning for the entire swarm and to send driving or operation commands to all or some of the mobile robots within the swarm.

[0048] In some implementations, the control system may be configured to process commands from external sources, such as Enterprise Resource Planning (ERP) systems. In some implementations, the control system, robot, and sensors may communicate via wireless communication systems, such as local area networks (LANs) with Wi-Fi, ZigBee, or Z-wave. Other networks that can also be used for communication between the control system, robot, and sensors include, but are not limited to, LoRa, NB-IoT (Narrowband Internet of Things), and LTE (Long Term Evolution). The control system may include an application programming interface (API) through which other systems can interact with the control system.

[0049] As noted, the operations performed by the control system include mapping generation, route planning, and positioning. A mapping generation system, implemented at least in part by the hardware and / or software within the control system, can perform mapping generation. (See also...) Figure 4 In process 39, in some implementations, to create a mapping of a space (such as a warehouse or factory), a robot is controlled (41) to move throughout the space. Since the robot is initially unfamiliar with the space, it can be manually controlled to move through all potential routes across the space. For example, a user can remotely guide the robot. As the robot moves, its vision system collects (42) visual data representing its surrounding environment. The data from the robot's vision system is received by a control system, which uses the data to generate (43) a mapping of the space. During operation, if the robot cannot move around an element in its path, it can be controlled to move around the element or back along the original path, and appropriate information indicating this situation can be stored. If the robot can move around an object, it can be controlled to move around the object, but the information is still stored. In some implementations, stationary or moving sensors mounted throughout the space, not on the robot, can contribute visual data for the mapping by sending data from these sensors to the control system. In some implementations, user input data can also contribute to the content of the mapping.

[0050] In one example, as the robot moves through space, its onboard LiDAR system senses the distances to elements in the plane of space, such as objects, boundaries, and landmarks. Input from the LiDAR system is transferred to a control system configured (e.g., programmed) to create (45) a 2D mapping of the space. In this example, the mapping includes a point cloud representing the space. In some examples, the point cloud comprises a collection of data points in the space. The LiDAR scanner measures numerous points on the outer surface of objects it encounters and arranges these points relative to each other on the mapping.

[0051] In this example, the information about objects, boundaries, and landmarks is based on 2D LiDAR data, which may make it difficult to identify elements with sufficient specificity. However, operations can be performed to distinguish these elements. For example, to distinguish these elements, regions of the point cloud with a point density greater than a specified point density can be identified and considered to correspond to locations that are part of an element or the edge of an element. Different regions of the point cloud with the necessary density can be grouped by clustering (46). Clustering can be performed using properties such as the distance between points. For example, a sufficiently large gap in points indicates that two elements are not part of the same cluster. Clustering can also be performed using line segments. Line segments can be identified in the point cloud by recognizing linearly arranged points. Linearly arranged points can be considered to be part of the same cluster.

[0052] As previously explained, the robot may include a color video system comprising one or more 3D cameras arranged to capture images of the space as the robot moves. The 3D cameras may be configured to capture 3D information representing the space and to transmit that 3D information to a control system. The 3D cameras may detect one or more characteristics of elements. Exemplary characteristics include features attributable to a particular type or category of the element. For example, the 3D camera may be configured to detect one or more characteristics indicating that the object is a living object (such as a person or animal). For example, the 3D camera may be configured to detect one or more characteristics indicating that the object is an existing object (such as a robot). The control system may be configured to detect objects, boundaries, and landmarks in the space based at least in part on these characteristics. For all or some of these objects, boundaries, and landmarks, the control system may be configured to generate (47) a 3D visual bounding box representing the edges of the volume captured by the 3D camera.

[0053] The control system uses data representing point clusters and bounding boxes to identify (48) elements within the space. For example, point clusters and bounding boxes can be associated to identify locations where point clusters coincide with bounding boxes. These locations can represent the positions of elements in the space. 3D data defines the properties of those elements, such as shape and orientation. Other identifiable properties include, but are not limited to, characteristics or properties of elements such as their size, color, structure, weight, mass, density, location, environment, chemical composition, temperature, odor, gas emissions, opacity, reflectivity, radioactivity, manufacturer, distributor, country of origin, function, supported communication protocols, electronic signature, radio frequency identifier (RFID), compatibility with other devices, ability to exchange communication with other devices, mobility and markings (such as barcodes, quick response (QR) codes), and example-specific markings (such as scratches or other damage). The resulting location and attribute data can be compared with information in a repository of predefined properties that are general or specific to the space. When a match is detected between a combination of point clusters and bounding boxes and an element from the repository, the control system classifies (49) the element represented by the combination of point clusters and bounding boxes as a matching element from the repository. For example, if a point cluster and bounding box combination matches a support pillar in the library, the point cluster and bounding box combination is classified as a support pillar. Similarly, if a point cluster and bounding box combination matches a container in the library, the point cluster and bounding box combination is classified as a container. An exact match of features may be required, or only a partial match may be needed.

[0054] In some implementations, the library may include one or more lookup tables (LUTs) or other suitable data structures for implementing comparisons. For example, the library and rules may be stored in the form of machine learning models such as, but not limited to, fuzzy logic, neural networks, or deep learning. Stored attributes may include attributes of different types of elements (such as living objects, inanimate objects, boundaries, landmarks, and other structures). Attributes are compared with the stored attributes of different elements. The stored attribute that most closely matches the identified element indicates the type of the element. In some implementations, matching may require an exact match between a set of stored attributes and element attributes. In some implementations, a match may be sufficient if the element's attribute is within a predefined range of the stored attributes. For example, numerical values ​​can be assigned to element attributes and stored attributes. A match between an element attribute and a stored attribute can be declared if the numerical values ​​match exactly or if the numerical values ​​are within a specific percentage of each other. For example, a match can be declared if the numerical value of an element attribute deviates from the stored attribute by no more than, for example, 1%, 2%, 3%, 4%, 5%, or 10%. In some implementations, a match can be declared if a number or a recognized feature is present. For example, a match can exist if three or four of the five identifiable feature structures are present.

[0055] In some implementations, attributes can be weighted based on factors such as importance. For example, shape can be weighted more than other attributes, such as color. Therefore, when comparing element attributes with stored attributes, shape can have a greater impact on the comparison results than color.

[0056] After classification, the control system can assign initial static scores (50) to the classified elements (e.g., objects, boundaries, or landmarks). Static scores can be assigned based on user input or input from an external source (such as another robot or computing system). In one example, the user can be prompted to assign static scores to elements. For example, static scores can be assigned by the control system without user input by referencing a stored database of elements and expected or default static scores for such elements. The database may contain information about expected static scores for different types of elements. The control system can identify elements in the database and retrieve their static scores. Examples of static scores include: 1, which is for elements that are definitely not moving or almost certainly not moving (e.g., 100% static) (such as boundaries or landmarks); 0, which is for elements that are almost certainly moving over a period of time (e.g., 0% static) (such as people or animals); and values ​​between 0 and 1, which are for objects that may or may not move based on their probability of moving or not moving. For example, some elements (such as goods or containers) may be assigned a static score of 0.5, which indicates that the element has a 50% probability of moving. For example, some elements (such as parked cars) can be assigned a static score of 0.2, which indicates a 20% probability that the element will move within a certain time period. This time period can be predefined or configurable and can affect the assignment of the static score.

[0057] The static score assigned to an element from the database can be adjusted (50) based on the element's location in space, the time the element was detected, or a combination of these and other factors. For example, goods detected in a space area known as a storage area may be assigned a higher static score (compared to those identical goods detected in a space area that is not a known storage area). For example, goods detected in a warehouse during normal working hours may be assigned a lower static score (compared to those identical goods detected in the same location outside of standard working hours). The expectation here is that the goods have been placed for storage outside of standard working hours.

[0058] The static score assigned to an element from the database can also be adjusted based on the element's pose in space. In one example, the element's pose includes the element's orientation, position, or both orientation and position. For example, a rectangular box that is in a stack of rectangular boxes and aligned with other rectangular boxes in the stack may be assigned a higher static score (compared to the same rectangular box that is detected at the same time and is in the same or equivalent position but is not aligned with other rectangular boxes in the stack).

[0059] In some implementations, the control system may include machine learning or artificial intelligence (AI) software configured to identify and classify elements within a space. For example, the software may be configured to receive input static scores for objects in the space. This initial static score may be received from a previously described database or from user input. The software may be configured to adjust the static scores of objects based on observations of similar objects at similar or different locations, at similar or different time periods, or in similar or different poses. For example, a robot may detect a first box against a wall in a high-traffic area of ​​a factory. Initially, because the first box is in a high-traffic area, it is assigned a low static score, implying that the first box is likely to be moved. This assignment may be based on input from a database and an initial assessment of the condition of the first box's presence. Over time, the robot may continue to detect the first box in the same space and at different times (including during and after working hours). This information may be added to a model that is part of the software used to determine and assign static scores to elements in the space. The model may be trained to adjust the static scores based on one or more such conditions that change over time. Therefore, when a second box that is the same or similar to the first box in a high-traffic area but against a wall is detected, the software can assign a higher static score to the second box based on the behavior of the first box detected.

[0060] The detected elements and their static scores can be used to construct a (51) map. (Return to reference) Figure 3Map 40 includes elements such as objects 33 and 34 (box and cat, respectively), boundaries 29 and 60 (wall and doorway, respectively), and landmarks 32 and 61 (support post and light fixture, respectively). Each element in the map may include identification information (such as the type of element, the attributes of the element, etc.), which is stored in association with such identification information. Each element in the map is also associated with a static score 64. The static score of each element may be stored in association with the representation and / or location or expected location of the element on the map. In this example, the static score of each element is depicted in a circle next to the element's reference numeral. As mentioned above, a static score represents the probability that an element (such as an object, boundary, or landmark) will remain immobile within a space. The probability may lie between a specific (or explicit) immobility and a specific (or explicit) mobility, either permanently or for a certain period of time. In this example, boundaries 29 and 60 are assigned a static fraction of 1; in this example, marker 32 is assigned a static fraction of 1, and in this example, marker 61 is assigned a static fraction of 0.9; in this example, cat 34 is assigned a static fraction of 0, and in this example, box 33 is assigned a static fraction of 0.4. In some specific implementations, the range of static fractions may be within a continuum between 0 (e.g., cat) indicating a particular mobility and 1 (e.g., wall) indicating a particular immobility. In two other examples currently shown in the mapping diagram, a pallet located in a pallet stack may be assigned a static fraction of 0.9, and a box located in the middle of an aisle may be assigned a static fraction of 0.3.

[0061] In some implementations, the mapping map (including the static scores of elements on the mapping map) may be updated periodically, intermittently, sporadically, or in response to newly received data. For example, a robot may traverse a space at a first time to generate a mapping map. During this traversal, the robot identifies boxes on an aisle. For reasons previously described, the robot assigns a low static score to the box. However, at a second time (e.g., one month after the first traversal), the robot identifies the same box on the same aisle. Assuming the space remains in use during this period, the robot adjusts the static scores of the box and initiates a corresponding update to the mapping map. For example, at or after this later time, the robot may change the static score of the box from 0.3 to 0.7 and transmit this change to the control system, which updates the mapping map accordingly. The mapping map can then be assigned to or made available to the robot or other robots that can use the mapping map to traverse the space.

[0062] In some implementations, the mapping map can be transferred to a single robot or multiple (e.g., all) robots operating within a space defined by the mapping map and controlled by a control system. Therefore, in some examples, any behavior specified by the mapping map can be applied to multiple robots in the space.

[0063] In some implementations, the static scores of objects can be used to characterize segments of a mapping map. For example, if an object with a high static score is present in a segment of the mapping map, that segment can be weighted to indicate that it is potentially unattractive due to possible obstacles in a path that it is unlikely to move. In some implementations, segments with higher weights are less attractive, and in others, segments with lower weights are less attractive. The weighting can take into account the presence of multiple objects that can act as obstacles and be located along the path. To generate weights, the static scores of those objects can be combined (e.g., summed). When determining segment weights, the resulting combined static scores, along with the object's position and the likelihood that the robot can move around the object, can be considered.

[0064] Reference Figure 3 The mapping map 30 includes weights 64 on segments 65 of the route 66 traversing the mapping map 30. In some examples, the segments include straight paths through the route (e.g., without turns). In this example, the weight is "10". This weight can indicate the presence of an object with a high static score in the route. In this example, the weight is not a static score, but may be based on a static score. In this example, at least a portion of the weight can be attributed to the presence of a box 69 on segment 65. The weight of a segment can make the route containing that segment less attractive, and therefore less likely to be selected by the control system when planning robot movement between the origin and destination.

[0065] In this regard, as noted, the operations performed by the control system include route planning. Route planning can be performed by a route planning system that is at least partially implemented by the hardware and / or software in the control system.

[0066] The control system can use a mapping map containing static scores (such as mapping map 30) to perform route planning. For example, the control system can access the mapping map and use it to determine the distance and / or estimate the travel time of potential routes between the origin and the destination on the mapping map. The control system can determine which potential routes best satisfy preferred metrics, such as shortest distance, shortest travel time, or both, and select that route. The static scores of elements on the mapping map can influence which route is selected. For example, an object blocking the first route can be characterized as highly static (e.g., having a static score of 0.9), and an object blocking the second route can be characterized as potentially dynamic (e.g., having a static score of 0.2). All others being equal or within acceptable constraints, the control system will select the second route (preferably the first route).

[0067] In some implementations, route planning considers elements on the mapping map with static scores exceeding a threshold and ignores elements with static scores below the threshold. In this example, the control system is configured to select a route based on elements with static scores exceeding the threshold and also satisfying preferred distance and time constraints. If objects with static scores below the threshold exist on the route, the robot can be configured to avoid those objects while traveling along the route as described herein. For example, route planning may use only elements with static scores of 0.9 or greater and ignore all other elements. In addition to elements with static scores of 0.9 or greater, other factors unrelated to static scores may also be considered when planning a route on the mapping map between the origin and the destination.

[0068] In some implementations, route planning considers all elements on the mapping map, regardless of static scores. For example, a control system may be configured to consider all elements along a route, within an area that includes multiple routes, or within an area that is part of a route. The control system is configured to use the static scores of elements on the mapping map to determine whether to select a route on the mapping map based on, for example, the probability that the route will be blocked. In the example presented above, assuming all other factors (such as distance and travel time) are within acceptable limits, the control system will select the route containing an object with a static score of 0.2 (better than the route containing an object with a static score of 0.9). In some implementations, the static score of an object blocking a route can remove a route from consideration. For example, if a first route is shorter in terms of distance and travel time, but is blocked by an object with a high static score, the first route may be rejected, and a second route with a longer distance and travel time may be selected (better than the first route). If an object moves into the route that is ultimately selected and blocks the robot's path, the robot may be configured to avoid those objects while traveling along the route as described herein. If an object moves into the route that was ultimately selected and is blocking the robot's path, the robot can reselect a route during its journey as described in this article.

[0069] In some implementations, the control system may be configured to determine the probability that one or more potential routes are blocked and the consequences of choosing between those routes or different routes. In one example, the control system may be configured to select the shortest route through a tunnel. However, an object with a static score of 0.5 blocks the end of the tunnel. In this case, the shortest route would have a 50% probability of being available and a 50% probability of being blocked. If a route through the tunnel is selected and the tunnel is actually blocked, replanning and reselecting the route will be necessary. An alternative route might be longer but less likely to be blocked, or any blockage might be easier to avoid than the blockage at the end of the tunnel. The control system may be configured to perform a cost-benefit analysis to determine whether the benefit of attempting the shortest route through the tunnel outweighs the cost of replanning and reselecting the route if the shortest route is taken but blocked. If the benefit outweighs the cost, the robot may be instructed to proceed along the shortest route; otherwise, the robot may be instructed to proceed along the longer route. In other words, if the benefit outweighs the cost, the shortest route will be selected; otherwise, the longer route will be selected.

[0070] As described above, in some implementations, a route may have one or more weights stored on a mapping map. Each segment of the route may be weighted, or an overall weight may be generated for the route. In one example, the weight represents the level of difficulty associated with traversing the route. In one example, a larger weight may indicate a greater level of difficulty traversing the route. In another example, a lower weight may indicate a greater level of difficulty traversing the route. Weights may be based on various factors, such as the length of the route, the inclination of the route, and the materials included on the surface of the route. The weight of each route may also be based on the static score of objects located along the route, such as objects that can block the route. For example, objects with low static scores (such as scores less than 0.3) that block a route segment contribute less to the overall weight of the segment compared to objects with high static scores (such as scores greater than 0.7). The weights may be adjusted over time as the static scores of objects change and / or as objects move into or out of the path along the route.

[0071] In some examples, a local route planning system may have more complete or up-to-date information to make planning decisions compared to a global or cluster management system. Therefore, in some implementations, the local route planning system may take precedence over the cluster management system when performing route planning.

[0072] In some cases, routes may be temporarily occupied by other mobile robots, other vehicles, people, or static objects. To reduce the likelihood of objects obstructing robots, mapping maps may include information containing driving constraints, such as the width of doorways, gates, or paths. A fleet management system can be configured to plan and schedule individual routes for each mobile robot within the fleet in a manner that avoids or at least reduces the likelihood that two mobile robots in the same fleet must stagger each other along passageways that may be too narrow to accommodate two robots (e.g., side-by-side). This can be achieved using visual data from sensors in the robot vision system described earlier.

[0073] As described above, operations performed by the control system include positioning. Positioning can be performed by a positioning system, implemented at least in part by the hardware and / or software in the control system. Positioning involves the robot moving along a planned and selected route and determining its current position in order to continue along its current trajectory or to make path corrections.

[0074] As previously explained, in some implementations, a robot achieves localization by scanning its surroundings and comparing the elements detected in the space with the contents of a stored mapping. If an exact match or a match within a predefined tolerance exists between the detected element and the contents of the mapping, the robot can determine its current position and take further actions, such as continuing along its current trajectory or performing path corrections. In dynamic environments, it may be difficult to obtain a sufficient match between the elements in the environment and the contents of the mapping. In such cases, the robot may be unable to determine its current position in the space and may therefore be constrained from further autonomous movement.

[0075] In this regard, in some implementations, localization performed by the robot may include scanning space and comparing detected elements only with elements on a map that have a static score exceeding a predefined threshold (such as 0.9). If the information obtained from this comparison is sufficient for the robot to determine its current position relative to the map, no further action is required. For example, if the robot can perform triangulation using three identified elements in a map with a static score exceeding a predefined threshold, these identified elements may be sufficient information for the robot to determine its current position. However, if the information obtained from this comparison is insufficient for the robot to determine its current position relative to the map, the static score threshold can be lowered, and the robot may compare the detected elements only with elements on a map that have a static score exceeding the lowered threshold. For example, the robot may compare the detected elements with elements on a map that have a static score exceeding 0.7 (instead of 0.9). This process of comparing, determining, and lowering the threshold can continue if necessary until the robot obtains sufficient information to determine its current position relative to the map.

[0076] In some implementations, the robot may enter a region containing insufficient numbers of elements with static scores exceeding a predefined threshold (such as 0.9). In this case, the previous operation can be performed to decrease the threshold, determine if there is sufficient information to obtain using the decreased threshold, and if not, decrease the threshold again and repeat these operations. This process can be performed multiple times until the robot obtains enough information to determine its current position on the route relative to the mapping map.

[0077] In some implementations, robots can exhibit different behaviors towards different types of objects. Exemplary behaviors include, but are not limited to, selecting a new route, moving around objects in the path, or waiting for a certain period of time for an object to move out of the path before continuing along it. Behaviors towards specific types of objects can be learned using artificial intelligence, machine learning techniques, or obtained from user input. Different types of behaviors can also be implemented based on whether the system detects a known or unknown object.

[0078] In some implementations, during movement through space along a planned route, the control system may generate virtual particles (“particles”) arranged in a virtual representation of the environment. A particle filter executed on the control system may be configured to generate and arrange these particles. The particles correspond to potential locations for the robot’s future travel and are at least partially used to control the robot’s direction of travel. The particles may be positioned or not positioned along the selected route. Figure 5An example of a portion 70 of a mapping map 30 and a portion of a selected route 71 traversing that mapping map is shown. In this example, robot 68 has been planned to travel along route 71 in the direction of arrow 69. However, due to the planning, box 80 has moved into the robot's path along route 71. During localization, the robot's control system generates particles at potential travel points from the robot's current position. Each of these particles can be weighted. The weight can represent the desirability or availability of a point that the robot can travel to. For example, a high weight (e.g., 100) can indicate a desired point to travel to, and a low weight (e.g., 1) can indicate an undesirable point to travel to. For example, a particle 77 traveling along the route 71 planned for the robot will be weighted more than a particle 73 that is off-path and behind a wall. The static score of an object can be used to change the weight of one or more such particles and thus at least partially affect the robot's movement. For example, box 80 with a high static score can block the robot's path along route 71. Therefore, particle 74, positioned behind the box, becomes a less attractive point that the robot can directly reach (e.g., without avoiding obstacles, in this case, box 80). Thus, the weight of particle 74 can be reduced, thereby decreasing the likelihood of controlling the robot to reach that point. Conversely, particle 78 becomes a more attractive point to reach directly because it is not blocked by the box, is reachable by the robot, and represents a path around box 80. Therefore, the weight of particle 78 can be increased, thereby increasing the likelihood of controlling the robot to reach that point.

[0079] The exemplary robots and systems described herein may include control systems and / or the processes described herein may be implemented using such control systems, which consist of one or more computer systems comprising hardware or a combination of hardware and software. For example, the robot, control system, or both may include various controllers and / or processing devices located at different points in the system to control the operation of its components. A central computer may coordinate the operations in the various controllers or processing devices. The central computer, controllers, and processing devices may execute various software routines to achieve control and coordination of various automated components.

[0080] The exemplary robots and systems described herein can be controlled at least in part using one or more computer program products, such as one or more computer programs tangibly embodied in one or more information carriers (such as one or more non-transitory machine-readable media), for performing or controlling the operation of one or more data processing devices, such as programmable processors, computers, multiple computers and / or programmable logic components.

[0081] Computer programs can be written in any programming language, including compiled or interpreted languages, and can be deployed in any form, including as standalone programs or as modules, components, subroutines, or other units suitable for a computing environment. Computer programs can be deployed to execute on a single computer, at a single site, or distributed across multiple locations and interconnected via a network.

[0082] Actions associated with at least a portion of the robot can be performed by one or more programmable processors that execute one or more computer programs to perform the functions described herein. At least a portion of the robot can be implemented using dedicated logic circuitry, such as FPGAs (Field-Programmable Gate Arrays) and / or ASICs (Application-Specific Integrated Circuits).

[0083] Processors suitable for executing computer programs include, for example, both general-purpose and special-purpose microprocessors, and any one or more processors in any kind of digital computer. Typically, a processor receives instructions and data from read-only memory or random access memory, or both. The components of a computer include one or more processors for executing instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include (or be operatively coupled to receive data from or transfer data to, or both) one or more machine-readable storage media, such as mass storage devices for storing data, for example, magnetic disks, magneto-optical disks, or optical disks. Machine-readable storage media suitable for embodying computer program instructions and data include all forms of non-volatile memory, including, for example, semiconductor memory devices, such as EPROM, EEPROM, and flash memory devices; magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; and CD-ROMs and DVD-ROMs.

[0084] In the descriptions provided in this article, the adjectives “first,” “second,” “third,” etc., do not specify priority or order. Instead, these adjectives are used only to distinguish the nouns they modify.

[0085] Any mechanical or electrical connection described herein may include a direct physical connection or an indirect connection that includes intermediate components.

[0086] The components of the different specific embodiments described herein can be combined to form other embodiments not specifically illustrated above. Multiple components can be excluded from the structures described herein without adversely affecting their operation. Furthermore, individual components can be combined into one or more single components to perform the functions described herein.

Claims

1. A system comprising: Sensors, used to obtain information about objects in the environment; and One or more processing devices, the one or more processing devices being configured to The information is used when generating or updating a mapping of the environment, the mapping including... i. The object, ii. Boundaries or landmarks in the environment. iii. A static score associated with each of the objects, each static score having a value between a first value and a second value, the first value representing a specific immobility of the associated object within the environment over a certain time period, and the second value representing a specific mobility of the associated object within the environment over the time period, and iv. Weights of routes or segments of routes in the mapping map that include the respective objects in the objects, wherein each weight represents a difficulty level associated with a corresponding one traversing the route or segment of the route, wherein the weights are assigned at least in part based on the static scores of the objects on the respective routes; and Using the mapping map, a route through the environment is selected using the weights, which are at least partially based on the static scores.

2. The system of claim 1, further comprising an autonomous vehicle configured to perform localization at least in part based on the static score while traversing the environment.

3. The system of claim 2, wherein at least one of the sensor or the processing device of the one or more processing devices is included on the autonomous vehicle.

4. The system of claim 2, wherein the static score is associated with at least some of the boundaries or landmarks to indicate that the at least some of the boundaries or landmarks have a specific immobility; and The positioning process includes using the static fractions of at least some of the boundaries or landmarks to determine the location of the autonomous vehicle within the environment.

5. The system of claim 2, wherein the static score is associated with at least some of the boundaries or landmarks representing a particular immobility; and wherein the operation for performing positioning includes: Using the static fractions of at least some of the boundaries or landmarks, first information about the location of the autonomous vehicle within the environment is determined, and In cases where additional information is desired to establish the location of the autonomous vehicle within the environment, the static score of the object is used to determine second information about the location of the autonomous vehicle within the environment, the second information enhancing the first information to establish the location of the autonomous vehicle within the environment.

6. The system of claim 5, wherein the additional information is desired to establish the location of the autonomous vehicle within the environment, because the first information is insufficient to establish the location.

7. The system of claim 2, wherein the static fraction is associated with at least some of the boundaries or landmarks and represents a specific immobility; and The positioning includes: A particle filter is used to generate particles in the environment, which correspond to potential future locations of the robot's movement, and each of the particles is weighted. as well as Use the static score of the object to change the weight of at least one of the particles.

8. The system according to claim 2, The one or more processing devices are located on the autonomous vehicle, and the one or more processing devices include: A first processing device, configured to receive the information from the sensor, and A second processing device is configured to identify the object based on the information and assign the static score to the object; and The first processing device is configured to generate the mapping map, and the second processing device has at least one of a larger clock frequency or a longer word size than the first processing device.

9. The system of claim 1, wherein the static score of the object is at least partially based on i. One or more times of the day when the information is obtained; or ii. The location of the object within the environment; or iii. The pose of the object within the environment; or iv. Data about the object provided from an external source.

10. The system according to claim 1, wherein the system includes a mapping generation system or a route planning system.

11. The system of claim 1, wherein the one or more processing devices are configured to implement a machine learning system, the machine learning system comprising a model capable of being trained to adjust the static score based on one or more conditions that change over time.

12. The system according to claim 1, further comprising: A memory that stores a database containing the attributes of the object; The one or more processing devices are configured to identify the object based on the attributes in the database and to assign the static score based on the identification of the object.

13. The system of claim 1, wherein the static score associated with a first object in the objects is an updated version of an earlier static score assigned to the first object in the objects, the updated version of the earlier static score being based on the sensor detecting the first object in the objects at a slightly later time than when the earlier static score was assigned.

14. The system of claim 1, wherein the sensor is part of a system comprising a plurality of sensors, the plurality of sensors comprising: A light detection and ranging sensor (LIDAR sensor), the LIDAR sensor being used to detect first data representing the boundary; and A camera, the camera being used to detect second data representing the object, the information including the second data; and The one or more processing devices are configured to use the first data to generate at least a portion of the mapping map.

15. The system of claim 14, wherein the plurality of sensors comprises: Multiple LiDAR sensors, the multiple LiDAR sensors including the LiDAR sensor, and Multiple cameras, the multiple cameras including the camera.

16. The system of claim 14, further comprising an autonomous vehicle configured to perform localization at least in part based on the static score while traversing the environment; and in: i. The LIDAR sensor and the camera are mounted on the autonomous vehicle and guided to view at least a portion of the same area in front of the autonomous vehicle in the direction of travel; or ii. The autonomous vehicle is a first autonomous vehicle and at least one of the plurality of sensors is mounted on a second autonomous vehicle that is different from the first autonomous vehicle.

17. The system of claim 14, wherein at least one of the plurality of sensors is mounted in the environment but not on the autonomous vehicle.

18. The system of claim 1, wherein the static fraction of the object is capable of changing over time.

Images