Robot operating environment configuration including sensor layout

Abstract

Optimizing the layout of the robot's operating environment to reduce or minimize occlusion of at least one or more sensors improves the computational efficiency of motion planning and / or the efficiency and / or robustness of the generated motion plan. The corresponding sensor poses (i.e., localization and orientation) of one or more sensors can be determined, considering various robot poses and transitions between them. For example, a cost value representing occlusion can be determined, and candidate sensor poses can be evaluated based on this cost value or aggregate cost. The total number of sensors to be used may not be known before execution and may actually only be determined after several iterations, upon reaching a termination condition (e.g., task completion time). Sensors can take various forms, such as digital cameras, radar, LiDAR, sonar, and passive infrared (PIR) motion detectors.

Description

Technical Field

[0001] This disclosure generally relates to robotics technologies, such as multi-robot operating environments or shared workspaces, and particularly to operating environment layouts including the placement and orientation of sensors within the robot operating environment, and optionally motion planning and robot manipulation based on sensing aspects of the operating environment. Background Technology

[0002] Description of related technologies

[0003] Various applications employ one or more robots in a robotic operating environment. This environment can contain a single robot or take the form of a shared workspace or a common operating environment. For example, two or more robots can be used in a common operating environment to perform tasks on one or more objects or workpieces, such as tightening bolts to a chassis, where the robots can overlap within their range of motion.

[0004] Motion planning is typically a fundamental problem in robot control and robotics. It specifies the path a robot can follow from an initial state to a target state, usually to complete a task without colliding with any obstacles in the operating environment, or to reduce the likelihood of such collisions. Challenges in motion planning include the ability to execute motion plans quickly, potentially in response to environmental changes (e.g., alterations in the position or orientation of obstacles). Further challenges include performing motion planning using relatively low-cost equipment, with relatively low power consumption, and / or utilizing limited storage (e.g., memory circuitry, such as memory circuitry on a processor chip). Summary of the Invention

[0005] When performing motion planning, it is desirable to generate motion plans that the robot can execute effectively. Efficiency can be measured by the time to perform or complete one or more tasks, the total number of movements required to complete one or more tasks, the total energy consumption to complete one or more tasks, and / or the total amount of time the robot is idle, such as waiting for another robot to clear a shared area or workspace. When performing motion planning, it is desirable to generate robust motion plans (e.g., motion plans that are guaranteed or highly likely to be fully executed to successfully complete one or more tasks, such as completing one or more tasks without collisions). When performing motion planning, it is desirable to generate motion plans quickly, allowing the system to adapt rapidly to changes in the operating environment, such as real-time adaptation. When performing motion planning, it is desirable to generate motion plans in a computationally efficient manner, thereby reducing the computational resources (e.g., hardware, computation cycles, energy usage) used in motion planning.

[0006] Many operating environments will include one or more robots and sensing systems to gather information about the environment, including information indicating which areas or volumes of the environment are occupied. It is desirable to configure the operating environment in a way that minimizes occlusion by the one or more sensors used by the sensing systems to monitor it. In motion planning, occlusion is often considered as occupied space to prevent collisions with obstacles that may or may not be located within the occluded space. However, the more space indicated as occupied, the more difficult it becomes to find an effective motion plan for the robot to perform a given task.

[0007] For example, occlusion can occur when a robot or part thereof is located (e.g., between) a sensor and a given location (e.g., a voxel) in the operating environment. In such a case, the given voxel is occluded by the sensor (i.e., invisible or undetectable). Occlusion can pose serious problems when the robot is capable of taking a considerable number of robot poses. There is a risk of occlusion for every robot pose, and also for transitions between robot poses. The problem is further exacerbated when there are multiple robots in the operating environment—each capable of moving between a relatively large number of robot poses—as the number of combinations or arrangements rapidly increases.

[0008] Therefore, it is desirable to optimize the sensor layout to minimize or reduce occlusion of the sensors by the robot as well as occlusion by other objects in the operating environment (e.g., obstacles).

[0009] Systems and methods are described that allow for the optimization of the layout of an operating environment in which one or more robots operate and where one or more sensors are pose-determined (i.e., localized and oriented) to at least reduce or minimize occlusion, thereby improving the computational efficiency of motion planning itself and / or improving the efficiency and / or robustness of the motion plans generated by motion planning. The corresponding positions and / or orientations of one or more sensors can be determined, taking into account various robot poses and transitions between robot poses, to enhance the sensor coverage of the operating environment, i.e., providing sensor coverage where occlusion occurs less frequently. For example, a cost value representing occlusion can be determined, and candidate sensor poses can be evaluated based on the cost value or aggregate cost. Reducing occlusion can significantly enhance robot motion planning, resulting in more robust motion plans (e.g., a higher probability of generating collision-free solutions) and / or executing more efficient motion plans (e.g., completing tasks faster, requiring fewer poses to complete tasks, and lower energy consumption to complete tasks), and / or reducing the computational resources (e.g., hardware, computation cycles, energy consumption) required to perform motion planning to generate feasible motion plans. This is referred to herein as optimizing sensor pose (i.e., sensor position and / or orientation), even though any given implementation may not lead to an optimal or best (i.e., objectively optimal) solution, but only to an optimized or improved solution. The total number of sensors to be placed can be defined before execution, or may not be defined before execution or may be known in other ways, and in fact may only be determined after several iterations when a termination condition is met. In many implementations, the sensors will take the form of digital cameras, each with a corresponding field of view that contains a portion of the operating environment determined by the position and orientation of the sensors. However, the various implementations described herein are not limited to digital cameras, but can be applied to virtually any type of sensor that can be used to sense conditions in the operating environment in which one or more robots operate, including but not limited to radar, LiDAR, sonar, passive infrared (PIR) motion detectors, etc.

[0010] Optimization can employ one or more of the following: knowledge of the environment, including the volume or region occupied by static objects; knowledge of the position and orientation of each robot in the environment (e.g., the base of each robot); knowledge of motion planning path maps (e.g., a graph representing possible robot poses and transitions between poses in the robot's configuration (C-space); and / or a set of tasks to be performed by the robot. Optimization can also employ a finite set of candidate sensor poses, each representing a tuple of possible positions and orientations of sensors in the environment. Attached Figure Description

[0011] In the accompanying drawings, the same reference numerals denote similar elements or actions. The size and relative positions of the elements in the drawings are not necessarily drawn to scale. For example, the shapes and angles of various elements are not drawn to scale, and some of these elements are arbitrarily enlarged and positioned to improve the readability of the drawings. Furthermore, the specific shapes of the depicted elements are not intended to convey any information about the actual shape of the particular element, and are chosen solely for ease of identification in the drawings.

[0012] Figure 1 This is a schematic diagram of an operating environment according to an embodiment shown, in which multiple robots operate to perform tasks, multiple sensors are located and oriented to monitor the operating environment, a perception computer system collects information from the sensors, a layout optimization system performs optimization to configure the layout of the operating environment, and an optional motion planner and robot control system generate motion plans and control the operation of the robots to complete the tasks.

[0013] Figure 2 It is based on an embodiment shown. Figure 1 Functional block diagrams of robots, sensors, perception computer systems, layout optimization systems, and optional motion planners and robot motion controllers, wherein each computer system includes one or more processors and storage of processor-executable instructions and one or more non-transitory processor-readable media.

[0014] Figure 3 This is a flowchart illustrating a high-level operation method of a processor-based system according to at least one of the illustrated embodiments, which is used to perform sensor pose (i.e., sensor position and orientation) optimization and optionally perform motion planning and control operations for one or more robots in an operating environment.

[0015] Figure 4 This is a flowchart illustrating an operational method of a processor-based system according to at least one of the illustrated embodiments, which is used to perform occlusion assessment and candidate sensor pose selection, and can be used as an execution... Figure 3 It is executed as part of the method.

[0016] Figure 5A This is a flowchart illustrating an operational method of a processor-based system according to at least one of the illustrated embodiments, used to determine a cost value representing occlusion, and can be used as an execution... Figure 3 It is executed as part of the method.

[0017] Figure 5B This is a flowchart illustrating an operational method of a processor-based system according to at least one of the illustrated embodiments, used to determine a cost value representing occlusion, and can be used as an execution... Figure 5A It is executed as part of the method.

[0018] Figure 6 This is a flowchart illustrating an operational method of a processor-based system according to at least one of the illustrated embodiments, used to determine voxel occlusion values, and can be used as an execution... Figure 5A and Figure 5B It is executed as part of the method.

[0019] Figure 7 This is a flowchart illustrating an operational method of a processor-based system according to at least one of the illustrated embodiments, which is used to aggregate the cost of candidate sensor attitudes and can be used as an execution... Figure 5A and Figure 5B It is executed as part of the method. Detailed Implementation

[0020] In the following description, certain specific details are set forth in order to provide a thorough understanding of the various disclosed embodiments. However, those skilled in the art will recognize that the embodiments can be practiced without one or more of these specific details or using other methods, components, materials, etc. In other instances, well-known structures associated with computer systems, actuator systems, and / or communication networks have not been shown or described in detail to avoid unnecessarily obscuring the description of the embodiments. In other instances, well-known computer vision methods and techniques for generating perceptual data and volumetric representations of one or more objects, etc., have not been described in detail to avoid unnecessarily obscuring the description of the embodiments.

[0021] Unless the context otherwise requires, throughout the specification and appended claims, the word “comprise” and its variations such as “comprises” and “comprising” shall be interpreted in an open, inclusive sense, meaning “including but not limited to”.

[0022] Throughout this specification, references to "one embodiment," "implementation," "an example," or "an embodiment" mean that a particular feature, structure, or characteristic described in connection with that embodiment is included in at least one embodiment or at least one example. Therefore, the phrases "one embodiment," "implementation," "in one example," or "in an example" appearing in various places throughout this specification do not necessarily all refer to the same embodiment or example. Furthermore, particular features, structures, or characteristics may be combined in any suitable manner in one or more embodiments or examples.

[0023] As used in this specification and the appended claims, unless otherwise expressly stated, the singular forms “a,” “an,” and “the” include plural indicators. It should also be noted that, unless otherwise expressly stated, the term “or” is generally used to mean “and / or.”

[0024] As used in this specification and the appended claims, the terms "optimize," "optimize," and "be optimized" mean that an improved result is being prepared, generated, or produced, or that an improved result has already been prepared, generated, or produced. Such terms are used in their relative sense and do not imply that an absolutely optimal value has been prepared, generated, or produced.

[0025] As used in this specification and the appended claims, the terms "operating environment" or "environment" or "workspace" are used to refer to an operating environment in which one or more robots operate. The operating environment may include obstacles and / or workpieces (i.e., items that the robot will interact with or act upon).

[0026] As used in this specification and the appended claims, the terms “shared operating environment” or “shared environment” or “shared workspace” are used to refer to an operating environment in which two or more robots operate, and one or more portions of the shared workspace are volumes in which the robots may collide with each other, and may therefore be referred to as interference areas.

[0027] As used in this specification and the appended claims, the term "task" refers to a robotic task in which the robot transitions from posture A to posture B, preferably without colliding with obstacles in its environment. The task may involve grasping or releasing an object, moving or placing an object, rotating an object, or retrieving or placing an object. The transition from posture A to posture B may optionally include transitions between one or more intermediate postures.

[0028] As used in this specification and the appended claims, the terms determine, perform determine, and be determined, when used in the context of whether a collision will occur or result, mean to assess or predict whether a given posture or movement between two postures via multiple intermediate postures will result in a collision between a part of the robot and an object (e.g., another part of the robot, a persistent obstacle, a temporary obstacle, or an object other than the current target object).

[0029] The titles and abstracts provided herein are for convenience only and do not explain the scope or meaning of the embodiments.

[0030] Figure 1A three-dimensional operating environment 100 is shown in which one or more robots 102a, 102b, 102c (three are shown, collectively referred to as 102) according to one of the illustrated embodiments can operate. For the sake of brevity, the three-dimensional operating environment 100 is referred to herein as environment 100. Environment 100 represents a three-dimensional space in which the robot 102 can operate and move. It should be noted that environment 100 is different from the “configuration space” or “C-space” of any given robot 102.

[0031] Environment 100 may include obstacles 106a to 106e (collectively referred to as 106) representing areas of potential collision. The terms one obstacle and multiple obstacles 106 are used to indicate objects representing the collision risk of a given robot 102 (e.g., inanimate objects including other robots, and living objects including humans and other animals).

[0032] Some of these obstacles 106a, 106b, and 106c may take the form of static obstacles or objects (i.e., obstacles or objects that do not move (i.e., do not translate, rotate) or change shape during the relevant time period or running time (e.g., buildings, trees, rocks, furniture, bases, supports, shelves). Some of these obstacles may take the form of dynamic obstacles or objects 106d and 106e in environment 100 (i.e., obstacles or objects that move (i.e., translate, rotate) or change shape during at least a portion of the relevant time period or running time (e.g., other robots, containers, vehicles, autonomous vehicles, people, animals, rolling or displaced items). For example, a set of bases, walls, and supports may be fixed or may not move or change shape during the relevant time period (e.g., running time) and are therefore considered static obstacles. Also, for example, a set of containers, workpieces, and another robot 102 may move or change shape (e.g., tilt) during the relevant time period (e.g., running time) and are therefore considered dynamic obstacles.

[0033] Some obstacles 106a to 106c occupy areas or volumes that do not change over time, for example, remaining fixed or unchanged during the robot's operation or movement. Such obstacles can therefore be referred to as static or persistent obstacles. The areas or volumes occupied by static or persistent obstacles (e.g., obstacles 106a to 106c) can be known at the time when the model is determined or the computational circuit is configured (referred to as the configuration time or pre-run time).

[0034] For other obstacles 106d and 106e, the corresponding area or volume occupied by the obstacle changes or is expected to change over time, for example, during robot operation or movement. Such obstacles can therefore be referred to as dynamic or temporary obstacles. The area or volume occupied by dynamic or temporary obstacles (e.g., obstacles 106d and 106e) is typically unknown during configuration time but is determined during robot operation or runtime.

[0035] Environment 100 may optionally include one or more target objects 108a, 108b (two shown, collectively referred to as 108), with robot 102 intended to interact with one or more target objects 108a, 108b, for example by grasping, moving, or otherwise engaging target objects 108, to perform some defined task or operation. Target objects 108 are not typically considered obstacles, but in some implementations they may constitute obstacles, for example, in the presence of multiple target objects 108 that one robot 102a will sequentially engage, or when considered relative to another robot 102b, 102c that does not target a particular target object 108. Some environments may not include any target objects, with robot 102 moving between various postures without interacting with or engaging any object.

[0036] Figure 1 A representative environment 100 with a finite number of obstacles 106 and target objects 108 is shown. Typical environments may include many additional obstacles 106 and target objects 108, including objects that are other robots and various other natural or artificial static and dynamic obstacles 106 or target objects 108. Some environments 100 may omit the target objects 108 entirely, or even the dynamic obstacles 106d, 106e. The concepts taught herein can be applied in a similar manner to more densely populated environments than those shown.

[0037] Robot 102 can take any of a variety of forms. Robot 102 can be any type of robot, including but not limited to: Cartesian robots, selectively compliant arm for robotic assembly (SCARA) robots, cylindrical coordinate robots, delta robots, and polar and vertically articulated robots. One or more robots 102 can also be mobile, for example, in the form of a car, an airplane, a drone, or any other vehicle capable of autonomous or semi-autonomous (i.e., at least partially autonomous) operation and movement in the space represented by environment 100.

[0038] In the illustrated embodiment, each robot 102 includes a base 110 and an attachment 111 (only one is shown) formed by a set of links 112a, 112b (only two are shown, collectively referred to as 112) and a set of joints 114 (only one is shown), each of the joints 114 being physically coupled to a corresponding pair of links 112. The robot 102 may also include one or more actuators 116 (only one is shown) that drive movement of one link 112a relative to another link 112b or relative to the base 110. The actuators 116 may take any of a variety of forms, such as electric motors, stepper motors, solenoids, electromagnets, pneumatic actuators, or hydraulic actuators, and are coupled and operable to move the linkage mechanism in response to a control or drive signal. For example, a pneumatic actuator may include one or more pistons, cylinders, valves, reservoirs of pressurized gas, and / or pressure sources (e.g., compressors, blowers). For example, a hydraulic actuator may include one or more pistons, cylinders, valves, a reservoir of pressurized fluid (e.g., a low-compressibility hydraulic fluid), and / or a pressure source (e.g., a compressor, a blower). Robot 102 may also include an end effector or end-effector tool 118. Figure 1 (Only one is shown in the image), such as a gripper with opposing fingers, hooks, or vacuum ports, to physically engage a target object 108 in environment 100. For example, one or more robots 102 may carry one or more sensors mounted on the robot (e.g., cameras, position or rotary encoders, proximity sensors, inertial sensors). In some embodiments, one or more robots 102 may take the form of, for example, autonomous vehicles.

[0039] The perception system 120 may include one or more environmental sensors 122a, 122b, 122c, 122d (four are shown, collectively referred to as 122) and an optional perception computer system 123. As described herein, the position and orientation of the sensors 122 in the environment 100, and even the total number of sensors 122 in the environment 100, can be optimized to reduce or even eliminate occlusion, thereby improving motion planning and motion design to drive the robot to complete one or more tasks.

[0040] The environmental sensor 122 can take any of a variety of forms or types, such as one or more digital cameras 122a, 122b (e.g., time-of-flight digital cameras, 3D cameras), one or more motion sensors (e.g., passive-infrared motion sensors) or radar 122c, one or more LiDAR sensors 122d, one or more microphones (not shown), one or more weight sensors or weighing sensors (not shown), one or more photoelectric sensors (e.g., passive infrared (IR) sensors, including IR light sources and IR sensors) (not shown), one or more encoders (e.g., position encoders, rotary encoders, reed switches) (not shown), one or more temperature sensors (not shown), humidity sensors (not shown), and / or one or more pressure sensors (not shown), to name just a few. The sensor 122 detects characteristics of the environment 100, including characteristics of obstacles, target objects, robots, and / or other objects in the environment 100 (e.g., position, orientation, shape, occupancy, movement, speed). Sensor 122 is capable of providing signals wired, optical, and / or wirelessly to an optional perception computer system 123, which collects and optionally preprocesses the collected sensor information or sensor data. The perception information or data may be raw data sensed via one or more sensors (e.g., cameras, stereo cameras, time-of-flight cameras, LiDAR) and / or converted by the perception computer system 123 into a digital representation of obstacles. This can generate a corresponding discretization of the environment in which the robot 102 will operate to perform tasks for various different scenarios. In at least some embodiments, the perception data may be in the form of point clouds.

[0041] Optional sensing computer system 123 may include circuitry, such as one or more processors and / or one or more non-transitory processor-readable media (e.g., non-volatile memory, volatile memory, rotating memory media), and may execute, for example, a set of one or more processor-executable instructions stored by the non-transitory processor-readable media. Sensing computer system 123 may be communicatively coupled (e.g., wired, optical, wirelessly coupled) to one or more sensors 122 to directly or indirectly receive sensing information, such as sensing data. Sensing computer system 123 may generate a digital or virtual model (referred to as an "environment representation") 132 of environment 100 based at least in part on the sensing data.

[0042] In particular, in at least some embodiments, execution of processor-executable instructions by one or more processors may enable the perception computer system 123 to process or preprocess some or all of the received sensor information. As described herein, the perception computer system 123 may generate one or more digital representations or models 132 of a three-dimensional operating environment 100, which includes any obstacles present in the environment 100 during a given period of time during runtime. As described herein, the digital representations or models 132 of the environment 100 may be advantageously used for optimization of the layout and / or motion planning for a given robot 102. The digital representations or models 132 may take any of a variety of forms, such as a placeholder grid.

[0043] The layout optimizer computer system 124 advantageously determines an optimized sensor pose (i.e., sensor position and orientation) for each of one or more sensors 122 in the environment 100, and may even determine the total number of sensors 122 in the environment 100. In particular, the layout optimizer computer system 124 can determine an optimized layout for at least the sensors, which advantageously reduces or even eliminates occlusion, such as occlusion caused by the robot as it moves through a series of robot poses to complete a task, thereby improving motion planning and motion schemes to drive the robot 102 to complete one or more tasks. In some embodiments, such as those described in U.S. Patent Application 62 / 991,487, the layout optimizer computer system 124 may additionally determine an optimized layout for the robot 102 in the environment 102 (e.g., the position and orientation of the robot's base), thereby improving motion planning and motion schemes to drive the robot 102 to complete one or more tasks.

[0044] The layout optimizer computer system 124 may include circuitry, such as one or more processors and / or one or more non-transitory processor-readable media (e.g., non-volatile memory, volatile memory, rotating memory media), and may execute, for example, a set of one or more processor-executable instructions stored by a non-transitory processor-readable medium. The layout optimizer computer system 124 may be communicatively coupled (e.g., wired, optical, wirelessly coupled) to the sensing computer system 123 to receive a digital or virtual model (referred to as an "environment representation") 132 of the environment 100, or alternatively communicatively coupled (e.g., wired, optical, wirelessly coupled) to one or more sensors 122 to directly or indirectly receive sensing information, such as sensing data.

[0045] Optimization can employ one or more of the following: knowledge of the environment, including the volume or region occupied by static objects; knowledge of the position and orientation of each robot in the environment (e.g., the base of each robot); knowledge of motion planning path maps (e.g., a graph representing possible robot poses and transitions between robot poses in the robot's C-space); and / or a set of tasks to be performed by the robot. Optimization can also employ a finite set of candidate sensor poses. Each candidate sensor pose represents a tuple of possible positions and orientations of sensors in the environment. For example, candidate sensor poses could include multiple poses at each of multiple locations across a two-dimensional surface corresponding to a surface in the environment (e.g., across a ceiling or wall suitable for mounting sensors).

[0046] In at least some embodiments, the input to the layout optimizer computer system 124 may include one or more static environment models that represent or characterize the operating environment or workspace 100, such as representing floors, walls, ceilings, columns, other static obstacles, etc. The operating environment or workspace 100 may be represented by one or more models, such as geometric models (e.g., point clouds) representing floors, walls, ceilings, obstacles, and other objects in the operating environment. For example, this may be represented in Cartesian coordinates.

[0047] In at least some embodiments, the input to the layout optimizer computer system 124 may include a set of candidate sensor poses. Alternatively, a processor-based system may include a candidate sensor pose swarm generator (not shown). The candidate sensor pose swarm generator generates a set of candidate sensor poses based on provided input (e.g., a physical model of the environment or room).

[0048] The input to the layout optimizer computer system 124 may include one or more robot models for each of the multiple robots operating in a multi-robot operation environment. The robot models may represent or characterize each robot 102, for example, specifying geometry and kinematics such as size or length, number of links, number of joints, joint type, link length, type of end effector, range of motion, limits on velocity, and limits on acceleration or jerk. Robot 102 may be represented by one or more robot geometric models that define the geometry of a given robot 102a-102c, for example, in terms of joints, degrees of freedom, dimensions (e.g., link lengths), and / or in aspects of the corresponding C-space of robots 102a-102c.

[0049] The input to the layout optimizer computer system 124 may include one or more sets of tasks to be performed, such as those represented as target objectives (e.g., robot position or configuration). Targets may represent various positions or configurations that each robot or part thereof must move to in sequence or at specific times to complete a set of tasks. For example, targets may be represented in the configuration space (C-space) of the respective robot. For example, tasks may be represented based on the final pose, final configuration, or final state and / or intermediate pose, intermediate configuration, or intermediate state of the respective robots 102a to 102c. For example, robot pose, configuration, or state may be defined based on the joint positions and joint angles / rotations (e.g., joint pose, joint coordinates) of the respective robots 102a to 102c. The input may optionally include one or more dwell durations, which specify the amount of time a robot or part thereof should remain at a given target to complete a task (e.g., tightening a screw or nut, picking up and placing an object). Dwell durations may advantageously reflect the expected amount of time that the robot's end effector needs to remain at a given target to complete the corresponding task.

[0050] For example, the processor-based system 124 can receive one or more models 132 of a multi-robot operating environment. Model 132 can represent the physical environment in which multiple robots will operate, such as representing floors, walls, and various objects in the environment.

[0051] A set of tasks can be designated as a task plan. A task plan can specify multiple (T) tasks to be performed by multiple robots. The task plan can be modeled as a vector for each robot, where the vector is an ordered list of tasks to be performed by the corresponding robot, e.g., {task 7, task 2, task 9}. The task vector can also optionally include a dwell duration specifying the duration for which the robot or a portion thereof should remain at a given configuration or objective. The task vector can also specify in-situ poses and / or other “functional poses” (e.g., “give way” or store poses) that are not directly related to solving the task. Robot poses can be specified in the robot's C-space.

[0052] Robot 102 typically performs tasks using motion planning, which in turn involves collision detection or evaluation. Therefore, one or more motion planners or motion planning systems 126 (only one shown) can be provided to perform motion planning, including collision detection or evaluation. The motion planner or motion planning system 126 is communicatively coupled (e.g., wired, optical, or wirelessly coupled) to a sensing computer system 123 or alternatively directly coupled to sensors 122 to receive information from them and generate motion plans based at least in part on the received information. The information may include a digital representation or model 132 of the environment 100 (represented as an environment representation or “ENVMT REP”), including any obstacles present in the environment 100 (including other robots 102). The input to the motion planner or motion planning system 126 may also include a kinematic model 130 of the robot 102 and a set of tasks, objectives, or goals 133 to be performed by each robot 102. The kinematic model 130 may, for example, take the form of a hierarchical data structure. The hierarchical data structure may, for example, take the form of one or more types of trees. For example, suitable hierarchical data structures may include octrees, axis-aligned bounding box (AABB) trees, oriented (non-axis-aligned) bounding box trees, sphere trees, and / or other tree-type data structures. For example, the kinematic model 130 may take the form of a non-hierarchical data structure (e.g., an Euclidean distance field).

[0053] The motion planner or motion planning system 126 may be an integral part of the robot 102, or it may be separate from and distinct from the robot 102. In some embodiments, one or more portions of the motion planner or motion planning system 126 may be on a corresponding robot within the robot 102, while one or more other portions may be separate from the robot 102 (i.e., outside the robot 102). The motion planner or motion planning system 126 may include circuitry, such as one or more processors and / or one or more non-transitory processor-readable media (e.g., non-volatile memory, volatile memory, rotating memory media), and may execute, for example, one or more sets of processor-executable instructions stored by the non-transitory processor-readable media. The motion planner or motion planning system 126 may generate a motion plan 127 to cause the robot 102 to perform a specific task, such as moving between a series of sequential postures, preferably moving between the series of sequential postures without colliding with or with a reduced probability of collision with obstacle 106. The motion planner or motion planning system 126 can be communicatively coupled (e.g., wired, optical, or wireless) to one or more robot control systems 138 to provide instructions to them, which cause a particular robot 102 to follow or execute a motion plan.

[0054] The motion planner or motion planning system 126 may include or access the collision detection system 140. The collision detection system 140 may include circuitry, such as one or more processors and / or one or more non-transitory processor-readable media (e.g., non-volatile memory, volatile memory, rotating memory), and may execute, for example, a set of one or more processor-executable instructions stored by the non-transitory processor-readable media. The collision detection system 140 advantageously employs a digital representation or model of the environment 132 and optionally a kinematic model 130 of the robot (e.g., a data structure representation of the kinematic model 130) to determine or detect or evaluate the probability of the robot colliding with obstacles 106 in the environment 100 as it moves in various postures or between postures. Those obstacles 106 may include other robots in the environment 100. The motion planner or motion planning system 126 and / or the collision detection system 140 may, for example, take the form of the motion planning system and collision detection system described in International (PCT) Patent Application PCT / US2019 / 045270, filed August 6, 2019.

[0055] Processor-based systems (e.g., motion planners or motion planning systems 126 and / or collision detection systems 140) can model the operational environment 100, including the movement of robot 102 therein, to determine certain attributes, such as the amount of time required to complete a task, the probability or collision rate of a collision when completing the task, the feasibility or infeasibility of completing the task, the number of moves or transitions required to complete the task, and / or the energy consumption when completing the task. Processor-based systems (e.g., motion planners or motion planning systems 126 and / or collision detection systems 140) can reflect this in terms of cost, generating costs via one or more cost functions.

[0056] The robot control system 138 may include several components, which are typically different but may be combined in some embodiments in a common circuit board, processor, or other circuitry. For example, a set of drivers may include circuitry communicatively coupled to actuator 116 to drive actuator 116 to cause robot 102 to adopt or move into a defined posture. For example, the drivers may include motor controllers and similar circuitry that drive any one or more of an electric motor, stepper motor, solenoid, electromagnet, hydraulic piston, pneumatic piston, hydraulic valve, pneumatic valve, vacuum system, hydraulic system, and / or pneumatic system pump or compressor.

[0057] Figure 2 A system 200 according to at least one illustrated embodiment is shown. The system 200 may include or be implemented in... Figure 1 Shown and about Figure 1 The various components or structures described.

[0058] System 200 may include one or more sensors 202, optionally one or more perception computer systems 203, one or more layout optimizer computer systems 204, one or more motion planners 206, and one or more robots 208. As described herein, the corresponding position and orientation of each sensor 202, and even the total number of sensors 202, can advantageously be determined by the operation of the layout optimizer computer system 204.

[0059] One or more sensing computer systems 203 may be communicatively coupled to one or more sensors 202 to receive sensing information or data therefrom. One or more sensing computer systems 203 may include one or more processors 205 and a memory 207. The memory 207 stores processor-executable instructions that, when executed by at least one processor 205, cause at least one processor 205 to collect information or data from the sensors and generate a digital representation or model of the three-dimensional operating environment. One or more sensing computer systems 203 may be communicatively coupled to provide, for example, a digital representation or model of the three-dimensional operating environment to one or more layout optimizer computer systems 204 and / or one or more motion planners 206. One or more motion planners 206 may be communicatively coupled to provide motion plans 127 to one or more robots 208 via one or more motion controllers 210. Figure 1 ).

[0060] As previously described, each robot 208 may include an attachment formed by a set of links and joints, wherein an end-effector or end effector is provided at the end of the attachment, and / or each robot 208 may include one or more actuators 211a, 211b, 211c capable of operating to move the links about the joints. Figure 2 Three are shown in the diagram, collectively referred to as 211. Each robot 208 may include one or more motion controllers (e.g., motor controllers) 210 (only one is shown), which receive control signals, for example, from a motion planner or motion planning system 206 and provide drive signals to drive actuators 211. Motion controllers 210 may be dedicated to controlling a specific actuator in actuators 211.

[0061] For illustrative purposes, an exemplary environment layout optimizer computer system 204 will be described in detail. Those skilled in the art will recognize that this description is exemplary and that changes may be made to the described and illustrated layout optimizer computer system 204.

[0062] The layout optimizer computer system 204 may include one or more processors 222 and one or more associated non-transitory computer- or processor-readable storage media, such as system memory 224, disk drive 225, and / or memory or registers of processor 222 (not shown). The non-transitory computer- or processor-readable storage media 224, 225 are communicatively coupled to processor 222 via one or more communication channels, such as system bus 229. System bus 229 may employ any known bus structure or architecture, including memory bus with memory controller, peripheral bus, and / or local bus. One or more of such components may also be or alternatively communicated via one or more other communication channels (e.g., one or more parallel cables, serial cables, or wireless network channels capable of high-speed communication, such as Universal Serial Bus (“USB”) 3.0, Peripheral Component Interconnect Express (PCIe)) or via... They communicate with each other.

[0063] The layout optimizer computer system 204 can also be communicatively coupled to one or more remote computer systems 212, such as server computers, desktop computers, laptop computers, ultra-portable computers, tablet computers, smartphones, wearable computers, and / or sensors. Figure 2 (Not shown in the diagram), one or more remote computer systems 212 are directly or indirectly coupled to various components of the layout optimizer computer system 204, for example, via a network interface (not shown). The remote computing system (e.g., a server computer) can be used to program, configure, control, or otherwise interface with or provide input data (e.g., robot models) to the layout optimizer computer system 204. Such connections can be made via one or more communication channels 214, such as one or more wide area networks (WANs) like Ethernet or the Internet using the Internet Protocol. In some embodiments, pre-runtime calculations or configuration time calculations (e.g., robot modeling) can be performed by a system separate from the layout optimizer computer system 204 (e.g., computer system 212). Runtime calculations can be performed by one or more layout optimizer computer systems 204 and / or motion planner 206.

[0064] As described, the layout optimizer computer system 204 may include one or more processors 222 (i.e., circuitry), non-transitory storage media 224, 225, and a system bus 229 coupling various system components. The processor 222 may be any logic processing unit, such as one or more central processing units (CPUs), digital signal processors (DSPs), graphics processing units (GPUs), field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), programmable logic controllers (PLCs), etc. Non-limiting examples of commercial computer systems include, but are not limited to, those in the United States. The company provides Celeron, Core, Core 2, Itanium, and Xeon series microprocessors; Advanced Micro Devices provides K8, K10, Bulldozer, and Bobcat series microprocessors; Apple Computer provides A5, A6, and A7 series microprocessors; Qualcomm provides Snapdragon series microprocessors; and Oracle provides SPARC series microprocessors. Figure 2The construction and operation of the various structures shown can be implemented or employ structures, techniques, and algorithms described in or similar to those described in the following patent applications: International Patent Application No. PCT / US2017 / 036880, filed June 9, 2017, entitled “MOTION PLANNING FORAUTONOMOUS VEHICLES AND RECONFIGURABLE MOTION PLANNING PROCESSORS”; International Patent Application Publication No. WO2016 / 122840, filed January 5, 2016, entitled “SPECIALIZED ROBOT MOTIONPLANNING HARDWARE AND METHODS OF MAKING AND USING SAME”; and APPARATUS, METHOD AND ARTICLE TOFACILITATE MOTION PLANNING OF AN AUTONOMOUS VEHICLE IN AN ENVIRONMENT HAVINGDYNAMIC, filed January 12, 2018. The U.S. Patent Application No. 62 / 616,783 entitled “OBJECTS”; U.S. Patent Application Serial No. 62 / 865,431 entitled “MOTION PLANNING FOR MULTIPLE ROBOTS IN SHARED WORKSPACE”, filed June 24, 2019; U.S. Patent Application No. 62 / 991,487 entitled “DIGITAL REPRESENTATIONS OF ROBOT OPERATIONAL ENVIRONMENT, USEVUL IN MOTION PLANNING FOR ROBOTS”, filed March 18, 2020; and / or the international (PCT) patent application PCT / US2019 / 045270 filed August 6, 2019.

[0065] System memory 224 may include read-only memory (“ROM”) 226, random access memory (“RAM”) 228, flash memory 230, and EEPROM (not shown). A basic input / output system (“BIOS”) 232, which can form part of ROM 226, contains basic routines that help transfer information between elements within the layout optimizer computer system 204, for example, during startup.

[0066] Drive 225 may be, for example, a hard disk drive for reading from and writing to a disk, a solid-state (e.g., flash memory) drive for reading from and writing to a solid-state memory, and / or an optical disk drive for reading from and writing to a removable optical disk. The layout optimizer computer system 204 may also include any combination of such drives in various different embodiments. Drive 225 may communicate with processor 222 via system bus 229. Drive 225 may include an interface or controller (not shown) coupled between such drive and system bus 229, as known to those skilled in the art. Drive 225 and associated computer-readable media provide the layout optimizer computer system 204 with non-volatile storage of computer- or processor-readable and / or executable instructions, data structures, program modules, and other data. Those skilled in the art will understand that other types of computer-readable media capable of storing computer-accessible data may be employed, such as WORM drives, RAID drives, magnetic tape, digital video disks (“DVDs”), Bernoulli cassette tapes, RAM, ROM, smart cards, etc.

[0067] Executable instructions and data can be stored in system memory 224, such as operating system 236, one or more application programs 238, other programs or modules 240, and program data 242. Application program 238 may include processor-executable instructions that cause processor 222 to perform one or more of the following: collecting or receiving sensor or sensing data, receiving or generating a three-dimensional environment 132. Figure 1 The representation or model of the receiving robot 208; the kinematic model 130 ( Figure 1 ); Optimize sensor poses, which reduce or eliminate occlusion of sensors by objects in the environment, including occlusion caused by the robot's movement while performing a task. To determine an optimized (e.g., not necessarily the best) layout of sensors in environment 100, processor-executable instructions can cause processor 222 to evaluate each of a plurality of candidate sensor poses, assign a cost value representing occlusion, aggregate the cost values, and analyze the cost values ​​to determine the optimized pose and / or the total number of optimized sensors. Execution of the processor-executable instructions can also cause processor 222 to optimize the robot pose (e.g., the position and / or orientation of the base) of the robot in the environment. Such operation can be as described herein (e.g., refer to Figure 3 and Figure 7Application 238 may include one or more machine-readable and machine-executable instructions that cause processor 222 to perform other operations, such as optionally processing sensed data (captured via sensors). The processor-executable instructions cause processor 222 to construct a representation or model based on the sensed data. In some embodiments, the processor-executable instructions may cause processor 222 to construct a representation or model including volumes that not only contain areas occupied by a given robot but also indicate that the volume is unoccupied, as described, for example, in U.S. Patent Application 62 / 991,487. This can be used, for example, to illustrate various structures (e.g., cables) extending from the robot. Application 238 may additionally include one or more machine-executable instructions that cause processor 222 to simulate, for example, the movement of a robot along various trajectories while performing a defined task. Application 238 may additionally include one or more machine-executable instructions that cause processor 222 to perform various other methods described herein and by reference incorporated herein.

[0068] Although Figure 2 The system is shown as being stored in system memory 224, but the operating system 236, application program 238, other applications, program / module 240, and program data 242 can be stored on other non-transitory computer or processor readable media such as drive 225.

[0069] Although not strictly necessary, many implementations will be described in the general context of computer-executable instructions, such as program application modules, objects, or macros stored on a computer or processor-readable medium and executed by one or more computers or processors. In various implementations, operations may be performed entirely in hardware circuitry or as software stored in a storage device such as system memory 224 and executed by one or more hardware processors 222, such as one or more microprocessors, digital signal processors (DSPs), field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), graphics processing unit (GPU) processors, programmable logic controllers (PLCs), electrically programmable read-only memory (EEPROMs), or as a combination of hardware circuitry and software stored in a storage device.

[0070] The layout optimizer computer system 204 may optionally include one or more input / output components, such as a monitor or touchscreen display 244, a keypad or keyboard 246, and / or a pointing device such as a computer mouse 248.

[0071] Those skilled in the art will understand that the illustrated and other embodiments can be practiced with other system architectures and arrangements and / or other computing system architectures and arrangements, including those of robots, handheld devices, multiprocessor systems, microprocessor-based or programmable consumer electronics, personal computers (“PCs”), networked PCs, minicomputers, mainframes, etc. Embodiments or examples, or portions thereof (e.g., at configuration and runtime), can be practiced in a distributed computing environment where tasks or modules are executed by remote processing devices linked via a communication network. In a distributed computing environment, program modules can reside in both local and remote memory storage devices or media. However, the location and manner in which certain types of information are stored can be important for helping to improve operations, including determining the optimal layout for sensor poses.

[0072] The motion planner or motion planning system 206 may include one or more processors 250, and one or more associated non-transitory computer- or processor-readable storage media, such as system memory 252, disk drives (not shown), and / or registers of processor 250 (not shown). The non-transitory computer- or processor-readable storage media (e.g., system memory 252) is communicatively coupled to processor 250 via one or more communication channels, such as system bus 254. System bus 254 may employ any known bus structure or architecture, including memory buses with memory controllers, peripheral buses, and / or local buses. One or more of such components may also be or alternatively communicated via one or more other communication channels (e.g., one or more parallel cables, serial cables, or wireless network channels capable of high-speed communication, such as Universal Serial Bus (“USB”) 3.0, High Speed ​​Serial Computer Extensions (PCIe) standard) or via... Communication between them. One or more communication ports (not shown) enable communication between the motion planner or motion planning system 206 and the sensing computer system 203 and / or the motion controller 210. The motion planner or motion planning system 206 may optionally be communicatively coupled (e.g., wired, optical, or wireless) to a computer or terminal (not shown) to allow user input (e.g., indications of motion subdivision granularity values ​​or specifications) and to provide user output.

[0073] The motion planner or motion planning system 206 may receive a representation or model of the environment, or otherwise based on perceived information collected via sensor 202. The motion planner or motion planning system 206 may also receive a robot model, task, objective, limits on the total number of robots, limits on the task of each robot, limits or constraints on variables or other parameters, and / or limits on iterations.

[0074] The motion planner or motion planning system 206 executes processor-executable instructions (application 256) that enable the motion planner or motion planning system 206 to perform motion planning, typically relying on collision evaluation to generate a motion plan for the robot to perform a task. The motion planner or motion planning system 206 may, for example, construct a motion plan by performing collision detection or evaluation, update the cost of edges in the motion planning graph based on collision detection or evaluation, and perform path search or evaluation.

[0075] In some implementations, sensor pose optimization can be an aspect of overall workspace optimization. Workspace optimization can also include optimizing the position and / or orientation of the respective base of each robot operating in the workspace, as described, for example, in U.S. Patent Application 62 / 964,405. This can include, for example, generating a population of candidate solutions (e.g., candidate positions and orientations of robot bases), modeling the candidate solutions, generating or determining costs associated with the respective candidate solutions based at least in part on the modeling, performing optimization on the population of candidate solutions via an optimization engine that collaboratively optimizes two or more of the following across a set of two or more non-homogeneous parameters: the respective base positions and orientations of the robots, task assignments to the respective robots within the robots, and the respective target sequences of the robots; and / or providing outputs capable of locating and orienting robots in a multi-robot operating environment and enabling the robots to perform tasks.

[0076] Figure 3 A method of operation 300 for a processor-based system according to at least one illustrated embodiment is shown, which is used to perform sensor pose optimization and optionally perform motion planning and control operations for one or more robots in an operating environment. The processor-based system may include at least one processor and at least one non-transitory processor-readable medium storing at least one of data and processor-executable instructions. When executed by at least one processor, the processor-executable instructions cause the at least one processor to perform various operations or actions of method 300.

[0077] Method 300 can be performed to determine the corresponding positions and / or orientations of one or more sensors to enhance the coverage of the operating environment, i.e., to provide coverage with reduced occlusion. As previously described in the summary of the invention, reducing occlusion can significantly enhance the robot's motion planning, resulting in more robust and / or more efficient motion plans and / or reducing the computational resources required to execute motion plans. This is referred to herein as optimizing sensor pose, even if the execution of method 300 may not produce an optimal (i.e., objectively optimal) solution, but only an optimized or improved one. The total number of sensors to be placed may or may not be known before performing the method, and in fact may only be determined after several iterations of the method when a termination condition (e.g., a specified or optimized task completion time) is met. In many embodiments, the sensors will take the form of digital cameras, each with a corresponding field of view containing a portion of the operating environment determined by the position and orientation of the sensors. However, the various implementations described herein are not limited to digital cameras, but can be applied to virtually any type of sensor that can be used to sense conditions in the operating environment in which one or more robots operate, including but not limited to radar, LIDAR, sonar, passive infrared (PIR) motion detectors, etc.

[0078] As previously mentioned, optimization may employ one or more of the following: knowledge of the environment (e.g., perceived data or information) of the volume or region occupied by static objects, knowledge of the position and orientation of each robot in the environment (e.g., the base of each robot), knowledge of the motion planning path map of each robot (e.g., a graph representing possible robot poses and transitions between poses in the robot's C-space), and / or a set of tasks to be performed by the robots.

[0079] Optimization can also employ a finite set of candidate sensor poses. Each candidate sensor pose represents a tuple of possible locations and orientations of the sensor in the environment. For example, candidate sensor poses can include multiple poses at each of multiple locations spanning a two-dimensional surface corresponding to a surface in the environment (e.g., across a ceiling or wall suitable for sensor mounting). The set of candidate sensor poses can be determined, generated, or refined in various ways. For example, the set of candidate sensor poses can be randomly generated or randomly selected from all locations and orientations available in environment 100. For example, the set of candidate sensor poses can be selected to be uniformly spaced across all available surfaces in environment 100. One or more heuristics can be applied to determine, generate, or refine the finite set of candidate sensor poses. For example, locations where only sensors can be mounted and / or where there is a power supply (e.g., a power outlet or junction box) can be included in the set of candidate sensor poses.

[0080] In some implementations, optimization of a group of C candidate sensor poses can be performed via an optimization engine. The optimization engine can select optimized candidate sensor poses, for example, based on occlusion analysis (e.g., occlusion analysis performed for task assignment to one or more robots and / or motion plans for one or more robots). The optimization engine can be implemented as one or more processors executing processor-executable instructions. In at least some implementations, the optimization engine can interact with a robot environment simulator during optimization, for example, analyzing occlusion at least in part based on simulations of robot movement through a series of robot poses during task execution. The robot environment simulator can characterize the efficiency (e.g., time to complete the task) and collision probability or collision rate of the corresponding motion plan. Efficiency can be used to determine termination conditions, such as determining when poses have been determined for a sufficient number of sensors. The optimization engine can select one of the candidate sensor poses at least in part based on the corresponding costs associated with the candidate sensor poses. The optimization engine provides optimized output. In particular, the optimization engine can provide the position and orientation of sensors for localizing and orienting in the environment as output.

[0081] Method 300 begins at 302, for example, in response to power-on of a processor-based system or its components, receipt of a command, or a call from a calling routine.

[0082] Optionally, at 304, the processor-based system or its components optimize the position and / or orientation of the respective base of each of one or more robots operating in the operating environment. Various methods and techniques can be employed to optimize the position and / or orientation of the robot's base, including but not limited to those taught in U.S. Patent Application 62 / 964,405. Whether the position and / or orientation of the robot's base is optimized, knowledge of the position and / or orientation of the robot's base, and knowledge of the robot's corresponding motion planning pathmap contributes to optimizing the sensor poses, which take into account various robot poses that can be used by the robot through its movement while performing a task. Knowledge of the task to be performed also facilitates the optimization of sensor poses and the total number of sensors.

[0083] At 306, the processor-based system or its components initialize the candidate sensor attitude counter I, for example, by setting it to zero. The candidate sensor attitude counter I is used to allow the processor-based system or its components to iterate over multiple candidate sensor attitudes as part of evaluating the candidate sensor attitudes used to determine the attitude of sensors (e.g., digital cameras) in an operating environment in which one or more robots will operate.

[0084] At 308, the processor-based system or its components increment the candidate sensor attitude counter I.

[0085] At 310, a processor-based system or its components perform an occlusion evaluation on the candidate sensor poses indicated by the current candidate sensor pose counter I. The occlusion evaluation assesses the occlusion effect (if any) of one or more robots in the operating environment as one or more robots move individually or transition through a plurality of corresponding robot poses J. In the case where the operating environment includes two or more robots, the occlusion evaluation assesses the occlusion effect of combinations of robot poses of the two or more robots. For example, the occlusion evaluation may assess the occlusion effect of a second robot in each of a plurality of corresponding robot poses when the first robot is in a corresponding first robot pose, and then assess the occlusion effect of the second robot in each of a plurality of corresponding robot poses when the first robot is in a corresponding second robot pose. The occlusion evaluation may assess the occlusion effect for one or more robots, for example, by selecting robot poses from a set of robot poses and analyzing the robot poses one by one. This set of robot poses may include the robot poses of one, two, or more robots. The occlusion evaluation may assess the occlusion effect of various combinations of robot poses for each of the two or more robots. Alternatively, a subset of combinations of robot poses can be defined (e.g., robot poses that can be used to perform one or more defined tasks), or an exploratory approach can be used to limit the total number of evaluations. An example implementation of occlusion evaluation is provided in a non-limiting example implementation of method 300 described below.

[0086] At 312, the processor-based system or its components determine whether the candidate sensor attitude counter I has reached N. If the candidate sensor attitude counter I has not yet reached N, control returns to 308, where the candidate sensor attitude counter I is incremented. If the candidate sensor attitude counter I has reached N, control proceeds to 314.

[0087] At 314, the processor-based system or its components select candidate sensor poses based on occlusion evaluation to determine the pose of sensor K. For example, this selection could be based on a cost value representing the occlusion effect of positioning and orienting the sensor in the corresponding candidate sensor pose. Examples of determining the cost value and making a cost-based selection are provided in a non-limiting example implementation of method 300 described below. Selected sensor poses can be removed from the set of candidate sensor poses so that the same sensor pose is not selected in subsequent iterations.

[0088] Optionally, at 316, the sensor K is localized and oriented based on the selected candidate pose. This can be done manually. Alternatively, this can even be done automatically, for example, via the movement of one or more robots that physically attach or pick up and place the sensors. In some implementations, the pose determination of the sensors can be modeled until an acceptable layout of all sensors is determined.

[0089] At 318, the processor-based system or its components determine whether the termination condition has been met. If the termination condition has been met, control is passed to 320. If the termination condition has not been met, control returns to 306, where candidate sensor poses in the candidate sensor pose set are analyzed to select another sensor pose.

[0090] Termination conditions can take any one or more forms. For example, a termination condition could be a measure of occlusion (e.g., the amount or total number of occlusions) falling below a defined threshold. Alternatively, a termination condition could be a measure of improvement in occlusion reduction reaching a limit (e.g., each iteration has a progressively smaller occlusion reduction until no further improvement is needed for a given application). Alternatively, a termination condition could be determining that further occlusion reduction would not significantly improve speed or reduce computational resources used for motion planning. Alternatively, a termination condition could be determining that further occlusion reduction would not significantly improve the robustness of the motion plan generated using information collected by sensors whose pose is determined, or the efficiency of the robot executing the motion plan. Alternatively, a termination condition could be based on a determined task completion time (i.e., the time it takes for the robot to complete one or more tasks), such as a determined task completion time within a specified threshold, or the amount or rate of improvement (e.g., reduction) in the determined task completion time over multiple cycles converging or not improving at the desired rate. For example, convergence can be considered to have occurred if the standard deviation of the cost among the current candidate sensor attitude groups is less than the epsilon value. Additionally or alternatively, a termination condition can be based on an evaluation of the collision-free paths available for a given motion plan. Additionally or alternatively, a termination condition can be the maximum number of allowed or available sensors. Additionally or alternatively, a termination condition can be the maximum number of allowed iterations. Additionally or alternatively, a termination condition can be the maximum allowed execution time or iteration time.

[0091] At 320, a processor-based system (e.g., perception computer system 123) or a component thereof may optionally receive sensor information collected via one or more sensors in corresponding sensor poses. For example, one, two, or more sensors in the form of a digital camera may collect images of the operating environment and provide the image information to the processor-based device for motion planning. The camera may operate in the human-visible or "white light" portion of the electromagnetic spectrum, the infrared portion of the electromagnetic spectrum, the ultraviolet portion of electromagnetic waves, or a combination thereof. Additionally or alternatively, one, two, or more sensors in the form of radar, LIDAR, sonar, or PIR motion detectors may collect information from the operating environment and provide the collected information to the processor-based device for motion planning. For example, the collected images or other information may represent the presence, position, shape, movement, trajectory, velocity, or acceleration of one or more objects in the operating environment. For example, objects may include workpieces, obstacles, and robots. Note that in some embodiments, a first robot may constitute an obstacle relative to a second robot.

[0092] At 324, a processor-based system (e.g., motion planner 126) or a component thereof optionally performs motion planning for one or more robots in an operating environment, at least in part, based on collected sensor information. Motion planning can take any of a variety of forms, at least some of which are described in the following: International (PCT) patent applications: PCT / US2019 / 023031, published as WO 2019 / 183141; PCT / US2016 / 012204, published as WO 2016 / 122840; PCT / US2017 / 036880, published as WO 2019 / 0217857; PCT / US2019 / 012209, published as WO2019 / 139815; PCT / US2019 / 016700, published as WO 2019 / 156984; PCT / US2019 / 045270, published as WO2020 / 040979; and PCT / US2019 / 064511 and U.S. patent applications: 62 / 856548; 62 / 865431; 62 / 890830; and 62 / 991487.

[0093] At 326, a processor-based system or component thereof (e.g., robot control system 138, motion controller 210) optionally causes one or more parts of one or more robots to move in an operating environment according to a generated motion plan. This can include providing the motion plan to one or more motion controllers, which are present at the robot or communicatively coupled to the robot. The motion controller controls one or more actuators of the robot (e.g., electric motors, solenoids, electromagnets, pistons and cylinders, associated valves and / or compressors) to induce motion. For example, the motion plan can cause robot attachments to transition between multiple robot poses, such as to perform one or more tasks. Movement of one or more parts of a robot can take any of a variety of forms, at least some of which are described below: International (PCT) patent applications: PCT / US2019 / 023031, published as WO 2019 / 183141; PCT / US2016 / 012204, published as WO 2016 / 122840; PCT / US2017 / 036880, published as WO2019 / 0217857; PCT / US2019 / 012209, published as WO 2019 / 139815; PCT / US2019 / 016700, published as WO 2019 / 156984; PCT / US2019 / 045270, published as WO 2020 / 040979; and PCT / US2019 / 064511 and U.S. patent applications: 62 / 856548; 62 / 865431; 62 / 890830; and 62 / 991487.

[0094] Once sufficient sensor coverage of the operating environment has been achieved, or some other termination condition has been met, method 300 terminates or ends at 328 until it is called again. In some implementations, method 300 is implemented each time the operating environment is initially set up, or in response to a substantial change in the operating environment, such as the addition of a robot, repositioning of the robot's base, or the introduction of a fixture or other substantial structure that may interfere with the collection of information by one or more sensors. For example, adding a table to environment 100 may interfere with the field of view of one or more cameras, thereby triggering the desire to determine a new set of sensor poses.

[0095] Method 300, or a portion thereof, can be executed at configuration time or pre-runtime, which can occur before runtime. This advantageously allows some of the most computationally intensive work to be performed before runtime when responsiveness is not a primary concern. Parts of Method 300, such as the robot's motion planning and movement, can be executed during runtime.

[0096] Although method 300 is described according to an ordered process, in many implementations various actions or operations will be performed simultaneously or in parallel.

[0097] The following is for reference. Figures 4 to 7 A non-limiting example implementation of method 300 is described.

[0098] Sensor placement optimization can employ any of a variety of optimization algorithms or techniques. Some implementations may use a cost function to apply cost to candidate sensor poses and evaluate various candidate sensor poses at least in part based on cost assessment.

[0099] For example, a processor-based system can sample candidate sensor poses from a finite set C of candidate sensor poses. For each of one or more robots, the processor-based system can sample robot poses from a set of robot poses RP. For example, each robot pose can be a vertex in the robot's motion planning path graph, or even a pose along an edge connecting a pair of vertices, where each edge represents a valid transition between the two poses represented by the vertices connected by the edge. In a multi-robot environment, the processor-based system can sample combinations of poses from each of the respective motion planning path graphs of the robots.

[0100] Example implementations may include initializing the cost value for each candidate sensor pose.

[0101] Then, the example implementation can enter an external iteration loop, which is performed for each voxel in the workspace.

[0102] Inside the outer iteration loop, the corresponding occlusion value for each voxel in the workspace is initialized (e.g., set to zero).

[0103] Within the outer iterative loop, the corresponding occlusion value for each voxel in the workspace is then determined or computed. This may involve executing the inner iterative loop to check for occlusion for each corresponding robot pose. For example, for each robot pose in the set of robot poses, the processor-based system determines whether a given voxel (i.e., the voxel being evaluated in the current iteration of the outer iterative loop) is occluded based on sensors whose poses have already been determined. Notably, every voxel is occluded in the first iteration because no sensor pose has been determined yet. If a voxel is determined to be occluded (i.e., invisible or undetectable by the sensor), the processor-based system increments the occlusion value of all voxels in the workspace occupied by one or more robots in a given pose (where the evaluation takes place), the occupied volume being called the robot voxel. The occlusion value is a representation of what the sensor hides for a given voxel. If a voxel is not occluded, the processor-based system processes the next robot pose, eventually exiting the inner iterative loop once occlusion has been checked for all robot poses.

[0104] Within the outer iterative loop, the processor-based system can aggregate the costs of individual candidate sensor poses. For example, for each candidate sensor pose in the set, the processor-based system can determine or compute the total cost to reach a given voxel. This can be understood by imagining a straight line from the candidate sensor pose to the given voxel. The processor-based system will sum the occlusion values ​​for each voxel along the line. The processor-based system can then add this total cost to the cost of the given candidate sensor pose.

[0105] Upon exiting the outer iterative loop, the processor-based system can then select candidate sensor poses based on cost, such as choosing the candidate sensor pose with the lowest associated cost. Various minimum-cost algorithms can be employed. Once selected, the sensor can be posed accordingly (e.g., physically or virtually). This process can then be repeated until a termination condition is met, such as poses being determined for a defined number of sensors or some other criterion (e.g., convergence of task completion time). It is noteworthy that each subsequent iteration should have fewer occluded voxels, resulting in fewer robot poses occluding each of these voxels. It is also worth noting that various algorithms can be employed, including "greedy" algorithms, where a solution may not be guaranteed, and any solution at optimization time may not be absolutely optimal.

[0106] Figures 4 to 7 This represents at least one implementation of optimization using a cost function. Other implementations may employ other algorithms and / or cost functions.

[0107] Figure 4A method 400 for operating a processor-based system for performing occlusion assessment and candidate sensor pose selection, according to at least one of the illustrated embodiments, is shown. Method 400 can be used as an execution of method 300. Figure 3 The processor-based system may be executed as part of at least one processor and at least one non-transitory processor-readable medium storing at least one of data and processor-executable instructions. When executed by at least one processor, the processor-executable instructions cause at least one processor to perform various operations or actions of method 400.

[0108] At 402, for each candidate sensor pose, the processor-based system or its components determine the corresponding cost value representing the occlusion of the sensor when it is positioned and oriented in the corresponding candidate sensor pose.

[0109] At 404, the processor-based system or its components select the candidate sensor pose with the lowest associated cost compared to other candidate sensor poses.

[0110] Figure 5A An operational method 500a for determining a cost value representing occlusion in a processor-based system according to at least one illustrated embodiment is shown. Method 500a can be used as an execution method 300 ( Figure 3 The processor-based system may be executed as part of at least one processor and at least one non-transitory processor-readable medium storing at least one of data and processor-executable instructions. When executed by at least one processor, the processor-executable instructions cause at least one processor to perform various operations or actions of method 500a.

[0111] Method 500a begins at 502, for example, in response to power-on of a processor-based system or its components, receipt of a command, or a call from a calling routine.

[0112] At point 504, the processor-based system or its components initialize the corresponding cost values ​​for all candidate sensor poses in the candidate sensor pose set. These cost values ​​can be stored in a data structure, such as as fields in records or in a cost value vector.

[0113] At position 506, initialize the voxel counter V used for the outer iteration loop. For example, the voxel counter V can be set to zero. Execute the outer iteration loop to evaluate the occlusion of a given voxel V in each iteration, iterating over all voxels in the workspace.

[0114] At 508, the voxel counter V used in the outer iteration loop is incremented.

[0115] At point 510, the processor-based system or its components initialize the corresponding occlusion values ​​for all voxels in the workspace. Occlusion values ​​can be stored in a data structure, such as as fields in records or in a vector of occlusion values.

[0116] At position 512, the processor-based system or its components determine the corresponding occlusion values ​​for all voxels in the workspace. These occlusion values ​​can be stored in a data structure, such as as a field in a record or in a vector of occlusion values. For example, this could be done as described in the reference... Figure 5B The method shown in 500b is described and performed as described below.

[0117] At point 514, the processor-based system or its components aggregate the cost of candidate sensor poses. The aggregated cost can be stored in a data structure, for example, as a field in a record or in a vector of occlusion values.

[0118] At 516, the processor-based system or its components determine whether each voxel V has been evaluated for occlusion, for example, by comparing the value of the voxel counter V with the total number of voxels in the workspace. If there are additional voxels to be evaluated, control returns to 508. If there are no additional voxels to be evaluated, control is passed to 518, where method 500a terminates until it is called again.

[0119] Figure 5B An operational method 500b for determining a cost value representing occlusion in a processor-based system according to at least one illustrated embodiment is shown. Method 500b can be used as an execution method 500a. Figure 5A To perform this as part of a larger task, for example, to determine the corresponding occlusion values ​​for all voxels in the workspace. Figure 5A (512). A processor-based system may include at least one processor and at least one non-transitory processor-readable medium storing at least one of data and processor-executable instructions. When executed by at least one processor, the processor-executable instructions cause at least one processor to perform various operations or actions of method 500b.

[0120] Method 500b begins at 520, for example, in response to power-on of a processor-based system or its components, receipt of a command, or a call from a calling routine.

[0121] At position 522, the processor-based system or its components initialize the robot pose counter R for the internal iterative loop. (Relative to reference...) Figure 5A The outer iterative loop is described, while the inner iterative loop can be considered as internal.

[0122] At position 524, the robot pose counter R used by the processor-based system or its components in the internal iterative loop is incremented.

[0123] At 526, the processor-based system or component determines whether a given voxel V is occluded, where the given voxel V is the voxel evaluated in the current iteration of the outer iterative loop. If the given voxel V is occluded, control is passed to 528, where the processor-based system or component increases the occlusion value of all voxels occupied by the robot in the current corresponding robot pose R. It is worth noting that the environment includes n robots, each capable of assuming p possible robot poses, and the total number of robot poses in the set RP to be considered in the inner iterative loop can be given by the product n*p. If the given voxel V is not occluded, control is passed directly to 530.

[0124] At 530, the processor-based system or its components determine whether each robot pose has been evaluated for occlusion, for example, by comparing the value of the robot pose counter R with the total number of robot poses available to the robot. If there are additional robot poses to be evaluated, control returns to 524. If there are no additional robot poses to be evaluated, control is passed to 532, where method 500b terminates until it is called again.

[0125] Figure 6 A method 600 for operating a processor-based system for determining voxel occlusion values, according to at least one illustrated embodiment, is shown. Method 600 can be used as an execution method 500a, 500b (…). Figure 5A and Figure 5B The processor-based system may be executed as part of at least one processor and at least one non-transitory processor-readable medium storing at least one of data and processor-executable instructions. When executed by at least one processor, the processor-executable instructions cause at least one processor to perform various operations or actions of method 600.

[0126] At 602, the processor-based system or its components determine whether a given voxel is occluded based on any sensors whose poses have already been determined and on a set of robot poses RP, which includes sampled robot poses of one or more robots that will operate in the environment. It is noteworthy that in environments involving multiple robots, a given voxel may not be occluded by any robot, or may be occluded by one, two, or more robots. Each robot pose can be evaluated one by one, for example, by iterating over the set of robot poses RP, e.g., from the first robot pose to the nth robot pose, where the robot poses in the set of robot poses RP can be used for one, two, or more robots.

[0127] At 604, if a voxel is occluded, the processor-based system or its components increase the occlusion value of any voxel occupied by the robot when the corresponding robot is in a specific robot pose where the corresponding robot or part thereof occludes the given voxel. As mentioned above, a given pixel may be occluded by two or more robot poses simultaneously.

[0128] Figure 7 An operational method 700 for a processor-based system for aggregating candidate sensor attitudes according to at least one of the illustrated embodiments is shown. Method 700 can be used as an execution method 500a, 500b (…). Figure 5A and Figure 5B The processor-based system may be executed as part of at least one processor and at least one non-transitory processor-readable medium storing at least one of data and processor-executable instructions. When executed by at least one processor, the processor-executable instructions cause at least one processor to perform various operations or actions of method 700.

[0129] At 702, for each candidate sensor pose, the processor-based system or its components sum the occlusion values ​​of each voxel between the sensor and the corresponding voxel.

[0130] At point 704, the processor-based system or its components add the sum of the occlusion values ​​to the cumulative cost sum of the corresponding candidate sensor poses. The cumulative cost sum can be stored in a data structure, for example, as a field in a record or in a vector of occlusion values.

[0131] Example

[0132] In at least some embodiments, the structures and algorithms described herein can operate without a camera or other sensing sensors. For example, method 300 ( Figure 1 The first iteration can be performed on any sensor already in environment 100 ( Figure 1 The coordination between robots is performed prior to positioning and / or orientation. In at least some embodiments, coordination between robots relies on the robot's geometry, the robots' ability to communicate their respective motion plans, and the geometry of the shared workspace. In other embodiments, vision or other perception may be optionally employed, for example, to avoid people or other dynamic obstacles that may enter or occupy portions of the shared workspace.

[0133] A wide variety of algorithms are used to solve motion planning problems. Each of these algorithms typically needs to be able to determine whether a given pose of the robot, or a motion from one pose to another, will result in a collision with the robot itself or with obstacles in the environment. Virtual collision evaluation or checking can be performed “in software” using processors that execute processor-executable instructions from a stored set of processor-executable instructions to execute the algorithm. Virtual collision evaluation or checking can also be performed “in hardware” using a set of dedicated hardware circuitry, such as collision checking circuitry implemented in a field-programmable gate array (FPGA) or application-specific integrated circuit (ASIC). For example, such circuitry can represent the volume swept by the robot / robot attachment or a portion thereof during a corresponding motion or transition between two states (i.e., the swept volume). For example, the circuitry can generate a Boolean evaluation indicating whether the motion will collide with any obstacles, where at least some obstacles represent volumes swept by other robots operating in a shared workspace when performing motions or transitions.

[0134] Example

[0135] Example 1: A method for operating a processor-based system to facilitate robot operation, the method comprising:

[0136] For each of a plurality of candidate sensor poses, perform an occlusion evaluation of the occlusion effect of the first robot as it moves through at least a plurality of robot poses in the operating environment; and

[0137] The pose of the first sensor is determined by selecting one of the candidate sensor poses based at least in part on the evaluation of occlusion.

[0138] Example 2, according to the method of Example 1, wherein the sensor is a camera having a corresponding field of view for the operating environment of each candidate sensor pose among candidate sensor poses, and selecting a candidate sensor pose from the candidate sensor poses to determine the pose of the first sensor based at least in part on an evaluation of occlusion comprises: selecting a candidate sensor pose from the candidate sensor poses to determine the pose of the first camera based at least in part on an evaluation of occlusion.

[0139] Example 3, according to the method of Example 1, wherein performing an occlusion assessment of the occlusion effect of the first robot as it moves through each of a plurality of robot poses of the first robot in the operating environment comprises: performing an occlusion assessment for each of a plurality of robot poses derived from a C-space route map of poses achievable by the first robot.

[0140] Example 4, according to the method of Example 1, wherein performing an occlusion assessment of the occlusion effect of the first robot as it moves through each of a plurality of robot poses of the first robot in the operating environment comprises: performing an occlusion assessment for each of a plurality of robot poses of a C-space route map selected from poses achievable by the first robot, based at least in part on the task to be performed by the first robot.

[0141] Example 5, according to the method of Example 1, wherein performing an occlusion assessment of the occlusion effect of the first robot as it moves through each of a plurality of robot poses of the first robot in an operating environment comprises: performing an occlusion assessment for each of a plurality of robot poses sampled from a roadmap of poses achievable by the first robot, the roadmap being in graphical form comprising a set of nodes representing each robot pose in the robot's C-space and a plurality of edges of coupled node pairs, and the plurality of edges representing transitions between corresponding robot poses represented by nodes coupled to the respective edges.

[0142] Example 6. According to the method of Example 1, wherein performing an occlusion evaluation of the occlusion effect of the first robot when at least a first robot moves through each of a plurality of robot poses of the first robot in the operating environment includes performing an occlusion evaluation of the occlusion effect of the first robot and the second robot when the first robot and at least a second robot in the operating environment move through each of a plurality of robot poses of the first robot and the second robot.

[0143] Example 7, according to the method of Example 6, wherein performing an occlusion assessment of the occlusion effect of the first robot and the second robot as the first robot and at least the second robot in the operating environment move through each of the respective plurality of robot poses of the first robot and the second robot includes: performing an occlusion assessment based on a set of robot poses representing the plurality of robot poses of the first robot and the plurality of robot poses of the second robot.

[0144] Example 8, the method described in Example 1 further includes:

[0145] For each candidate sensor pose in the candidate sensor poses, a corresponding cost value is determined, where the corresponding cost value represents occlusion.

[0146] Example 9, the method described in Example 1 further includes:

[0147] In the outer iteration loop performed for each voxel in the workspace,

[0148] Determine the corresponding occlusion values ​​of at least other voxels in the workspace, wherein the occlusion value indicates whether the voxel of the current iteration of the outer iteration loop is occluded by an object relative to one or more sensors, the object being positioned to at least partially occupy the corresponding voxel whose occlusion value is being determined; and

[0149] Cost of aggregating candidate sensor attitudes.

[0150] Example 10, the method described in Example 9 further includes:

[0151] Before determining the corresponding occlusion value for the voxel,

[0152] For each candidate sensor pose, initialize the corresponding cost value for the candidate sensor pose;

[0153] In the outer iteration loop, initialize the corresponding occlusion values ​​for all voxels in the workspace.

[0154] Example 11, according to the method described in Example 9, wherein determining the corresponding occlusion value of the voxel in the current iteration of the outer iteration loop includes:

[0155] In the inner iterative loop, it is determined whether the voxel of the current iteration of the outer iterative loop is occluded based on any sensors that have been located.

[0156] In the inner iteration loop, if the voxel of the current iteration of the outer iteration loop is occluded, the corresponding occlusion value of any voxel occupied by the object in one of the robot poses, where the object occludes the voxel of the current iteration of the outer iteration loop in the corresponding robot pose, is increased.

[0157] Example 12, according to the method in Example 9, wherein the cost of aggregating candidate sensor poses includes:

[0158] For each candidate sensor pose, sum the occlusion values ​​for each voxel between the sensor and the voxel in the current iteration of the external iteration loop where occlusion is being evaluated; and

[0159] Add the sum to the cumulative cost sum of the corresponding candidate sensor poses.

[0160] Example 13, the method according to any one of Examples 1 to 12, wherein selecting a candidate sensor pose from among the candidate sensor poses to determine the pose of the first sensor, at least in part based on an evaluation of occlusion, comprises: selecting the candidate sensor pose based on the associated cost of the candidate sensor pose relative to the associated costs of other candidate sensor poses among the candidate sensor poses.

[0161] Example 14. The method according to any one of Examples 1 to 12, wherein selecting a candidate sensor pose from among the candidate sensor poses to determine the pose of the first sensor is based at least in part on an evaluation of occlusion, comprises: selecting the candidate sensor pose that has the lowest associated cost of the candidate sensor pose relative to the associated costs of the other candidate sensor poses among the candidate sensor poses.

[0162] Example 15, the method described in Example 1 further includes:

[0163] After selecting a candidate sensor pose from among the candidate sensor poses based at least in part on an evaluation of occlusion to determine the pose of the first sensor,

[0164] For each of a plurality of candidate sensor poses, perform an occlusion evaluation of at least the occlusion effect of the first robot as it moves through each of a plurality of robot poses of the first robot in the operating environment; and

[0165] The pose of the second sensor is determined by selecting one of the candidate sensor poses based at least in part on the evaluation of occlusion.

[0166] Example 16, according to the method of Example 15, wherein performing an occlusion assessment of the occlusion effect of the first robot as it moves through each of a plurality of robot poses of the first robot in the operating environment includes performing an occlusion assessment further for the first sensor, which is localized and oriented according to a candidate sensor pose selected for the first sensor from among the candidate sensor poses.

[0167] Example 17: The method according to Example 15, wherein the sensor is a camera, and the method further includes:

[0168] Receive sensor information collected by a first camera in a camera at a first pose corresponding to a sensor pose selected for a first sensor and by a second camera in a camera at a second pose corresponding to a sensor pose selected for a second sensor; and

[0169] Motion planning is performed for at least the first robot based on the received sensor information.

[0170] Example 18. The method described in Example 15 further includes:

[0171] After selecting a candidate sensor pose from among the candidate sensor poses based at least in part on an evaluation of occlusion to determine the pose of the second sensor,

[0172] Repeat the following until the termination condition is met: For each of the multiple candidate sensor poses, perform an occlusion evaluation of the occlusion effect of at least the first robot when it moves through each of the multiple robot poses of the first robot in the operating environment; and

[0173] The candidate sensor pose is selected from among the candidate sensor poses, at least in part, based on an assessment of occlusion, to determine the pose of the additional sensor.

[0174] Example 19. A system for facilitating the operation of a processor-based system for robot operation, the system comprising:

[0175] At least one processor;

[0176] At least one processor-readable medium communicatively coupled to at least one processor and storing processor-executable instructions, which, when executed by at least one processor, cause at least one processor to:

[0177] For each of a plurality of candidate sensor poses, perform an occlusion evaluation of the occlusion effect of the first robot as it moves through at least a plurality of robot poses in the operating environment; and

[0178] The pose of the first sensor is determined by selecting one of the candidate sensor poses based at least in part on the evaluation of occlusion.

[0179] Example 20: The system according to Example 19, wherein the sensor is a camera having a corresponding field of view for the operating environment of each candidate sensor pose among the candidate sensor poses.

[0180] Example 21, the system according to Example 19, wherein, in order to perform occlusion evaluation, the instructions, when executed, cause the at least one processor to perform occlusion evaluation for each of a plurality of robot poses from a C-space roadmap of poses achievable by the first robot.

[0181] Example 22, the system according to Example 19, wherein, in order to perform occlusion assessment, the instructions, when executed, cause the at least one processor to perform occlusion assessment for each of a plurality of robot poses selected from a C-space roadmap of poses achievable by the first robot, based at least in part on a task to be performed by the first robot.

[0182] Example 23. The system according to Example 19, wherein, in order to perform occlusion evaluation, the instructions, when executed, cause the at least one processor to perform occlusion evaluation for each of a plurality of robot poses sampled from a roadmap of poses achievable by the first robot, the roadmap being in graphical form including a set of nodes representing each robot pose in the robot's C-space and a plurality of edges of coupled node pairs, and the plurality of edges representing transitions between corresponding robot poses represented by nodes coupled by the respective edges.

[0183] Example 24. The system according to Example 19, wherein, in order to perform an occlusion assessment, the instructions, when executed, cause the at least one processor to perform an occlusion assessment of the occlusion effect of the first robot and the second robot as the first robot and at least a second robot in the operating environment move through each of a plurality of corresponding robot poses of the first robot and the second robot.

[0184] Example 25. The system according to Example 24, wherein, in order to perform occlusion evaluation, the instructions, when executed, cause the at least one processor to perform occlusion evaluation based on a set of robot poses representing a plurality of robot poses of a first robot and a plurality of robot poses of a second robot.

[0185] Example 26. The system according to Example 19, wherein, when executed, the processor-executable instructions cause the at least one processor to further:

[0186] For each candidate sensor pose in the candidate sensor poses, a corresponding cost value is determined, where the corresponding cost value represents occlusion.

[0187] Example 27. The system according to Example 19, characterized in that, when executed, the processor-executable instructions cause at least one processor to further:

[0188] In the outer iteration loop performed for each voxel in the workspace,

[0189] Determine the corresponding occlusion values ​​of at least other voxels in the workspace, wherein the occlusion value indicates whether the voxel of the current iteration of an external iteration loop is occluded by an object relative to one or more sensors, the object being positioned to at least partially occupy the corresponding voxel whose occlusion value is being determined; and

[0190] Cost of aggregating candidate sensor attitudes.

[0191] Example 28. The system according to Example 27, wherein, when executed, a processor-executable instruction causes at least one processor to:

[0192] Before determining the corresponding occlusion value for the voxel,

[0193] For each candidate sensor pose, initialize the corresponding cost value for the candidate sensor pose;

[0194] In the outer iteration loop, initialize the corresponding occlusion values ​​for all voxels in the workspace.

[0195] Example 29. According to the system described in Example 28, in order to determine the corresponding occlusion value of the voxel in the current iteration of the outer iterative loop, the processor may execute instructions, when executed, to cause at least one processor to:

[0196] In the inner iterative loop, it is determined whether the voxel of the current iteration of the outer iterative loop is occluded based on any sensors that have been located.

[0197] In the inner iteration loop, if the voxel of the current iteration of the outer iteration loop is occluded, the corresponding occlusion value of any voxel occupied by the object in one of the robot poses, where the object occludes the voxel of the current iteration of the outer iteration loop in the corresponding robot pose, is increased.

[0198] Example 30, the system according to Example 27, wherein, in order to aggregate the cost of candidate sensor poses, the processor can execute instructions, when executed, to cause at least one processor to:

[0199] For each candidate sensor pose, sum the occlusion values ​​of each voxel between the sensor and the voxel being evaluated in the current iteration of the outer iteration loop; and

[0200] Add the sum to the cumulative cost sum of the corresponding candidate sensor poses.

[0201] Example 31, the system according to any one of Examples 19 to 30, wherein, in order to select a candidate sensor pose from among the candidate sensor poses based at least in part on an evaluation of occlusion to determine the pose of a first sensor, a processor-executable instruction, when executed, causes at least one processor to: select the candidate sensor pose based on the associated cost of the candidate sensor pose relative to the associated cost of the other candidate sensor poses among the candidate sensor poses.

[0202] Example 32, the system according to any one of Examples 19 to 30, wherein, in order to select a candidate sensor pose from among the candidate sensor poses based at least in part on an evaluation of occlusion to determine the pose of a first sensor, a processor executable instruction, when executed, causes at least one processor to further: select the candidate sensor pose that has the lowest associated cost of the candidate sensor pose relative to the associated costs of the other candidate sensor poses among the candidate sensor poses.

[0203] Example 33. The system according to Example 19, wherein, when executed, a processor-executable instruction causes at least one processor to:

[0204] After selecting a candidate sensor pose from among the candidate sensor poses based at least in part on an evaluation of occlusion to determine the pose of the first sensor,

[0205] For each of a plurality of candidate sensor poses, perform an occlusion evaluation of the occlusion effect of the first robot as it moves through at least a plurality of robot poses in the operating environment; and

[0206] The pose of the second sensor is determined by selecting one of the candidate sensor poses from the candidate sensor poses, based at least in part on the evaluation of occlusion.

[0207] Example 34. The system according to Example 33, wherein, in order to perform occlusion assessment, the processor may execute instructions, when executed, to cause at least one processor to also: perform an occlusion assessment for a first sensor that is localized and oriented according to a candidate sensor pose selected for the first sensor from among the candidate sensor poses.

[0208] Example 35. A system according to Example 33, wherein the sensor is a camera, and wherein the processor is executable, when executed, causes at least one processor to:

[0209] Receive sensor information collected by a first camera in a camera at a first pose corresponding to a sensor pose selected for a first sensor and by a second camera in a camera at a second pose corresponding to a sensor pose selected for a second sensor; and

[0210] Motion planning is performed for at least the first robot based on the received sensor information.

[0211] in conclusion

[0212] The foregoing detailed description has illustrated various embodiments of the device and / or process using block diagrams, schematic diagrams, and examples. Where such block diagrams, schematic diagrams, and examples contain one or more functions and / or operations, those skilled in the art will understand that each function and / or operation in such block diagrams, flowcharts, or examples can be implemented individually and / or collectively by various hardware, software, firmware, or virtually any combination thereof. In one embodiment, this subject matter can be implemented using Boolean circuits, application-specific integrated circuits (ASICs), and / or FPGAs. However, those skilled in the art will recognize that all or part of the embodiments disclosed herein can be implemented in standard integrated circuits in various different ways as one or more computer programs running on one or more computers (e.g., as one or more programs running on one or more computer systems), as one or more programs running on one or more controllers (e.g., microcontrollers), as one or more programs running on one or more processors (e.g., microprocessors), as firmware, or as virtually any combination thereof, and that designing circuits and / or writing software and / or firmware code according to this disclosure will be within the skill of those skilled in the art.

[0213] Those skilled in the art will recognize that many of the methods or algorithms described herein may employ additional actions, omit some actions, and / or perform actions in a different order than specified.

[0214] Furthermore, those skilled in the art will understand that the mechanisms taught herein can be implemented in hardware, such as with one or more FPGAs or ASICs.

[0215] The various embodiments described above can be combined to provide other embodiments. All commonly assigned U.S. patent applications, U.S. patent applications, foreign patents, and foreign patent applications mentioned in and / or listed in the application data sheet are incorporated herein by reference in their entirety, including but not limited to: International Patent Application No. PCT / US2017 / 036880, filed June 9, 2017, entitled “MOTION PLANNING FORAUTONOMOUS VEHICLES AND RECONFIGURABLE MOTION PLANNING PROCESSORS”; International Patent Application Publication No. WO2016 / 122840, filed January 5, 2016, entitled “SPECIALIZED ROBOT MOTION PLANNING HARDWARE AND METHODS OF MAKING AND USING SAME”; and International Patent Application No. WO2016 / 122840, filed January 12, 2018, entitled “APPARATUS, METHOD AND ARTICLE TOFACILITATE MOTION PLANNING OF AN AUTONOMOUS VEHICLE IN AN ENVIRONMENT HAVINGDYNAMIC”. U.S. Patent Application No. 62 / 616,783, entitled “OBJECTS”; U.S. Patent Application No. 62 / 626,939, entitled “MOTION PLANNING OF A ROBOT STORING ADISCRETIZED ENVIRONMENT ON ONE OR MORE PROCESSORS AND IMPROVED OPERATION OF SAME”, filed February 6, 2018; U.S. Patent Application No. 62 / 856,548, entitled “APPARATUS, METHODS AND ARTICLES TO FACILITATE MOTION PLANNING IN ENVIRONMENTS HAVING DYNAMIC OBSTACLES”, filed June 3, 2019; and U.S. Patent Application No. 62 / 856,548, entitled “MOTION PLANNING FOR MULTIPLE ROBOTS IN SHARED”, filed June 24, 2019. U.S. Patent Application No. 62 / 865,431 entitled “WORKSPACE”; U.S. Patent Application No. 62 / 964,405 entitled “CONFIGURATION OF ROBOTS INMULTI-ROBOT OPERATIONAL ENVIRONMENT” filed on January 22, 2020;And U.S. Patent Application No. 63 / 009,583, filed April 14, 2020, entitled "CONFIGURATION OF ROBOT OPERATIONAL ENVIRONMENT INCLUDINGLAYOUT OF SENSORS". These and other changes can be made to the embodiments based on the above detailed description. Generally, the terminology used in the appended claims should not be construed as limiting the claims to the specific embodiments disclosed in the specification and claims, but should be construed as including all possible embodiments and the full scope of equivalents conferred by such claims. Therefore, the claims are not limited to this disclosure.

Claims

1. A method for operating a processor-based system to facilitate robot operation, the method comprising: For each of a plurality of candidate sensor poses, perform an occlusion assessment of the occlusion effect of the first robot as it moves through at least one of a plurality of robot poses in the operating environment. as well as The method further comprises selecting a candidate sensor pose from among the candidate sensor poses, at least in part, based on an evaluation of occlusion, to determine the pose of a first sensor, such that the first sensor is localized and oriented according to the candidate sensor pose selected from the candidate sensor poses, wherein the method further comprises: In the outer iteration loop performed for each voxel in the workspace, Determine the corresponding occlusion value for each voxel among the other voxels in the workspace, wherein the occlusion value indicates whether the voxel of the current iteration of the external iteration loop is occluded by an object relative to one or more sensors, the object being positioned to at least partially occupy the corresponding voxel whose occlusion value is being determined; and The cost of aggregating candidate sensor poses, wherein determining the corresponding occlusion value of the voxel in the current iteration of the external iteration loop includes: In the inner iterative loop, it is determined whether the voxel of the current iteration of the outer iterative loop is occluded based on any sensors that have been located. In the inner iteration loop, if a voxel of the current iteration of the outer iteration loop is occluded, the corresponding occlusion value of any voxel occupied by an object in a robot pose of one of the robot poses is increased, wherein the object occludes the voxel of the current iteration of the outer iteration loop in the corresponding robot pose.

2. The method according to claim 1, wherein, The first sensor is a first camera, the first camera having a corresponding field of view for the operating environment of each candidate sensor pose among the candidate sensor poses, and determining the pose of the first sensor by selecting one of the candidate sensor poses based at least in part on an evaluation of occlusion includes: selecting one of the candidate sensor poses based at least in part on an evaluation of occlusion to determine the pose of the first camera.

3. The method according to claim 1, wherein, Performing an occlusion assessment of the occlusion effect of the first robot as it moves through each of a plurality of robot poses in an operating environment includes performing an occlusion assessment for each of a plurality of robot poses derived from a C-space route map of poses achievable by the first robot.

4. The method according to claim 1, wherein, Performing an occlusion assessment of the occlusion effect of the first robot as it moves through each of a plurality of robot poses in an operating environment includes: performing an occlusion assessment for each of a plurality of robot poses selected from poses achievable by the first robot, based at least in part on the task to be performed by the first robot.

5. The method according to claim 1, wherein, Performing an occlusion assessment of the occlusion effect of the first robot as it moves through each of a plurality of robot poses in an operating environment includes: performing an occlusion assessment for each of a plurality of robot poses sampled from a roadmap of poses achievable by the first robot, the roadmap being in graphical form including a set of nodes representing each robot pose in the robot's C-space and a plurality of edges of coupled node pairs, and the plurality of edges representing transitions between corresponding robot poses represented by nodes coupled to the respective edges.

6. The method according to claim 1, wherein, Performing an occlusion evaluation of the occlusion effect of at least a first robot in an operating environment as the first robot moves through each of a plurality of robot poses of the first robot includes performing an occlusion evaluation of the occlusion effect of the first robot and the second robot as the first robot and at least a second robot in an operating environment move through each of a plurality of robot poses of the first robot and the second robot.

7. The method according to claim 6, wherein, Performing an occlusion assessment of the occlusion effect of the first robot and the second robot as they move in the first robot and the operating environment via each of a plurality of robot poses of the first robot and the second robot includes: performing an occlusion assessment based on a set of robot poses representing a plurality of robot poses of the first robot and a plurality of robot poses of the second robot.

8. The method according to claim 1, further comprising: For each candidate sensor pose in the candidate sensor poses, a corresponding cost value is determined, where the corresponding cost value represents occlusion.

9. The method according to claim 1, further comprising: Before determining the corresponding occlusion value for the voxel, For each candidate sensor pose, initialize the corresponding cost value for the candidate sensor pose; In the outer iteration loop, the corresponding occlusion values ​​of all voxels in the workspace are initialized.

10. The method according to claim 1, wherein, The cost of aggregating candidate sensor poses includes: For each candidate sensor pose, sum the occlusion values ​​for each voxel between the sensor and the voxel in the current iteration of the external iteration loop where occlusion is being evaluated; and Add the sum to the cumulative cost sum of the corresponding candidate sensor poses.

11. The method according to any one of claims 1 to 10, wherein, Selecting a candidate sensor pose from among the candidate sensor poses to determine the pose of the first sensor, at least in part based on an assessment of occlusion, includes: selecting the candidate sensor pose based on the associated cost of the candidate sensor pose relative to the associated costs of other candidate sensor poses among the candidate sensor poses.

12. The method according to any one of claims 1 to 10, wherein, Selecting a candidate sensor pose from among the candidate sensor poses to determine the pose of the first sensor, at least in part based on an evaluation of occlusion, includes: selecting the candidate sensor pose that has the lowest associated cost relative to the associated costs of the other candidate sensor poses among the candidate sensor poses.

13. The method according to claim 1, further comprising: After selecting a candidate sensor pose from among the candidate sensor poses based at least in part on an evaluation of occlusion to determine the pose of the first sensor, For each of a plurality of candidate sensor poses, perform an occlusion assessment of at least the occlusion effect of the first robot as it moves through each of a plurality of robot poses of the first robot in the operating environment. as well as The pose of the second sensor is determined by selecting one of the candidate sensor poses based at least in part on the evaluation of occlusion.

14. The method according to claim 13, wherein, Performing an occlusion assessment of the occlusion effect of the first robot as it moves through each of a plurality of robot poses in the operating environment includes performing an occlusion assessment of the first sensor as specified by the localization and orientation of a candidate sensor pose selected for the first sensor from among the candidate sensor poses.

15. The method according to claim 13, wherein, The sensor is a camera, and the method further includes: Receive sensor information collected by a first camera in a camera body in a first pose corresponding to a sensor pose selected for the first sensor and by a second camera in a camera body in a second pose corresponding to a sensor pose selected for the second sensor; and Motion planning is performed for at least the first robot based on the received sensor information.

16. The method of claim 13, further comprising: After selecting a candidate sensor pose from among the candidate sensor poses based at least in part on an evaluation of occlusion to determine the pose of the second sensor, Repeat the following until a termination condition is met: For each of a plurality of candidate sensor poses, perform an occlusion evaluation of at least the occlusion effect of the first robot when the first robot moves through each of a plurality of robot poses of the first robot in the operating environment; and The candidate sensor pose is selected from among the candidate sensor poses, at least in part, based on an assessment of occlusion, to determine the pose of the additional sensor.

17. A system for facilitating the operation of a processor-based system for robot operation, the system comprising: At least one processor; At least one processor-readable medium communicatively coupled to the at least one processor and storing processor-executable instructions that, when executed by the at least one processor, cause the at least one processor to: For each of a plurality of candidate sensor poses, perform an occlusion evaluation of the occlusion effect of the first robot as it moves through at least one of a plurality of robot poses in the operating environment; and The pose of the first sensor is determined by selecting one of the candidate sensor poses from the candidate sensor poses, at least in part, based on an evaluation of occlusion. Wherein, when the processor-executable instructions are executed, the at least one processor also: In the outer iteration loop performed for each voxel in the workspace, Determine the corresponding occlusion values ​​of at least other voxels in the workspace, wherein the occlusion value indicates whether the voxel of the current iteration of the external iteration loop is occluded by an object relative to one or more sensors, the object being positioned to at least partially occupy the corresponding voxel whose occlusion value is being determined; and The cost of aggregating candidate sensor poses; In the inner iterative loop, it is determined whether the voxel of the current iteration of the outer iterative loop is occluded based on any sensors that have been located. In the inner iteration loop, if a voxel of the current iteration of the outer iteration loop is occluded, the corresponding occlusion value of any voxel occupied by an object in a robot pose of one of the robot poses is increased, wherein the object occludes the voxel of the current iteration of the outer iteration loop in the corresponding robot pose.

18. The system according to claim 17, wherein, The sensor is a camera, which has a corresponding field of view for the operating environment of each candidate sensor pose.

19. The system according to claim 17, wherein, In order to perform occlusion assessment, the instructions, when executed, cause the at least one processor to perform occlusion assessment for each of a plurality of robot poses derived from a C-space roadmap of poses achievable by the first robot.

20. The system according to claim 17, wherein, In order to perform occlusion assessment, the instructions, when executed, cause the at least one processor to perform occlusion assessment for each of a plurality of robot poses selected from a C-space roadmap of poses achievable by the first robot, based at least in part on a task to be performed by the first robot.

21. The system according to claim 17, wherein, In order to perform occlusion evaluation, the instructions, when executed, cause the at least one processor to perform occlusion evaluation for each of a plurality of robot poses sampled from a roadmap of poses achievable by the first robot. The roadmap, in graphical form, includes a set of nodes representing each robot pose in the robot's C-space and a plurality of edges of coupled node pairs, and the plurality of edges represent transitions between corresponding robot poses represented by nodes coupled by the respective edges.

22. The system according to claim 17, wherein, In order to perform an occlusion assessment, the instructions, when executed, cause the at least one processor to perform an occlusion assessment of the occlusion effect of the first robot and the second robot as the first robot and at least a second robot in the operating environment move through each of a plurality of corresponding robot poses of the first robot and the second robot.

23. The system according to claim 22, wherein, In order to perform occlusion assessment, the instructions, when executed, cause the at least one processor to perform occlusion assessment based on a set of robot poses representing multiple robot poses of the first robot and multiple robot poses of the second robot.

24. The system according to claim 17, wherein, When executed, the processor-executable instructions cause the at least one processor to also: For each candidate sensor pose in the candidate sensor poses, a corresponding cost value is determined, where the corresponding cost value represents occlusion.

25. The system according to claim 17, wherein, When executed, the processor-executable instructions cause the at least one processor to also: Before determining the corresponding occlusion value for the voxel, For each candidate sensor pose, initialize the corresponding cost value for the candidate sensor pose; In the outer iteration loop, the corresponding occlusion values ​​of all voxels in the workspace are initialized.

26. The system according to claim 17, wherein, To aggregate the cost of candidate sensor poses, the processor can execute instructions, when executed, to cause the at least one processor to: For each candidate sensor pose, sum the occlusion values ​​of each voxel between the sensor and the voxel being evaluated in the current iteration of the external iteration loop. as well as Add the sum to the cumulative cost sum of the corresponding candidate sensor poses.

27. The system according to any one of claims 17 to 26, wherein, In order to select a candidate sensor pose from among the candidate sensor poses to determine the pose of a first sensor based at least in part on an evaluation of occlusion, the processor executable instructions, when executed, cause the at least one processor to select the candidate sensor pose based on the associated cost of the candidate sensor pose relative to the associated cost of other candidate sensor poses among the candidate sensor poses.

28. The system according to any one of claims 17 to 26, wherein, In order to select a candidate sensor pose from among the candidate sensor poses to determine the pose of the first sensor based at least in part on an evaluation of occlusion, the processor may execute instructions, when executed, to cause the at least one processor to further: select the candidate sensor pose that has the lowest associated cost of the candidate sensor pose relative to the associated costs of the other candidate sensor poses among the candidate sensor poses.

29. The system according to claim 17, wherein, When executed, the processor-executable instructions cause the at least one processor to also: After selecting a candidate sensor pose from among the candidate sensor poses based at least in part on an evaluation of occlusion to determine the pose of the first sensor, For each of a plurality of candidate sensor poses, perform an occlusion evaluation of the occlusion effect of the first robot as it moves through at least one of a plurality of robot poses in the operating environment; and The pose of the second sensor is determined by selecting one of the candidate sensor poses based at least in part on the evaluation of occlusion.

30. The system according to claim 29, wherein, In order to perform occlusion assessment, the processor may execute instructions, when executed, to cause the at least one processor to also: perform an occlusion assessment for the first sensor that is localized and oriented according to a candidate sensor pose selected for the first sensor from among the candidate sensor poses.

31. The system according to claim 29, wherein, The sensor is a camera, and wherein the processor can execute instructions, when executed, cause the at least one processor to also: Receive sensor information collected by a first camera in a camera body in a first pose corresponding to a sensor pose selected for the first sensor and by a second camera in a camera body in a second pose corresponding to a sensor pose selected for the second sensor; and Motion planning is performed for at least the first robot based on the received sensor information.

Images