Safety systems and methods employed in robotic operations

Abstract

A safety system for robotics includes: multiple sensors, preferably a heterogeneous set of commercially available off-the-shelf sensors; and at least one processor that evaluates the operational states of the sensors, verifies the system state based on the evaluated sensor operational states to determine whether there are sufficient sensors operable to provide a safety-certified system, and monitors the operating environment for violations of safety rules that specify rules regarding human-robot proximity. A control system for robotics includes at least one processor that performs motion planning taking into account safety monitoring rules implemented by the safety system, thereby reducing the triggering of stop, mitigation, or preventative occlusion by the safety system.

Description

Technical Field

[0001] This disclosure generally relates to robots, and more particularly to safety systems and methods used in robot operation, which, for example, incorporate a robot control system that itself can employ motion planning to generate motion plans to drive the robot in the operating environment. Background Technology

[0002] Description of related technologies

[0003] Robots are becoming increasingly common in a variety of applications and environments.

[0004] Typically, a robot control system performs motion planning and / or control of one or more robots. A robot control system can take the form of, for example, a processor-based system, which typically has one or more sensors (e.g., cameras, contact sensors, force sensors, encoders). The robot control system can determine and / or execute motion planning to enable the robot to perform a series of tasks. Motion planning is a fundamental problem in robot control and robotics. Motion planning specifies the path that a robot can follow from an initial state to a target state, typically to complete the task without encountering or minimizing the likelihood of encountering any obstacles in the operating environment, including humans. The challenges of motion planning involve the ability to execute motion planning very quickly, even as environmental characteristics change. For example, the characteristics of one or more obstacles in the environment, such as their position or orientation, may change over time. Challenges also include performing motion planning using relatively low-cost equipment, with relatively low energy consumption and limited storage (e.g., memory circuitry, such as memory circuitry on a processor chip).

[0005] The safety of robot operation, especially the safe movement of the robot or its parts, is often a major concern when humans (e.g., robot operators) enter or may enter an operating environment in which one or more robots are operating.

[0006] When safety is a primary concern, a dedicated safety system can be employed. This dedicated safety system can complement the robot control system that performs motion planning and / or control of one or more robots. Such a system can take the form, for example, a processor-based cell safety system, which typically incorporates one or more sensors (e.g., cameras). Processor-based cell safety systems monitor the operating environment for hazards, particularly the presence of humans or objects that may be human.

[0007] Safety systems used in robotics can be safety-certified, in which case they typically employ multiple safety-certified sensors. Increasing the number of such safety-certified sensors generally reduces the likelihood of occlusion, which refers to areas obscured from the sensor's field of view. However, safety-certified sensors are significantly more expensive than the more common commercial off-the-shelf (COTS) sensors. Therefore, a balance is often struck between the desire to add safety-certified sensors to reduce occlusion and the substantial cost of adding more safety-certified sensors to a safety-certified system.

[0008] Processor-based cell safety systems typically operate by triggering safety-related stops or decelerations of robot operations when the safety system detects certain safety-related conditions, such as detecting a human approaching the robot or its trajectory. While this helps prevent accidents, unnecessary stops or decelerations can adversely affect the overall performance of one or more robots. Summary of the Invention

[0009] Using multiple COTS sensors to implement a safety-certified system would be advantageous, as it is typically much less expensive than using safety-certified sensors alone. Reducing or even eliminating the total number and / or duration of robot stoppages or slowdowns would also be beneficial.

[0010] A processor-based work unit safety system can be viewed as consisting of two parts: sensors located and oriented to monitor at least a portion of the operating environment or work unit; and a processor-based system communicatively coupled to the sensors and processing the sensor data provided by the sensors. Some implementations may include additional types of sensors that detect when a human has entered the operating environment, but not necessarily detect the human's location or orientation within the work unit. Such sensors may include, for example, one or more of the following: an RFID interrogation system that detects radio frequency identification (RFID) transponders worn by the human body, a laser scanner, a pressure sensor, and / or a passive infrared (PIR) motion detector that detects the presence of a human in the work unit.

[0011] In most cases, the position and orientation of one or more robots are known to the processor, for example, based on the known joint angles of one or more robots, or this information can be obtained in a way that allows for safe authentication. If a processor-based cell-of-work safety system loses tracking of a static object (e.g., a table), this is generally not considered a safety hazard because static objects typically do not collide with humans. Therefore, for safety authentication, the primary issue is tracking the position of one or more humans within an operating environment in which one or more robots are operating.

[0012] Losing tracking of one or more humans for a very short period (i.e., less than the time that could lead to a conflict between humans and robots) is acceptable for a safety system. In other words, there will always be some uncertainty about the location of a human (e.g., due to sensor noise, sampling rate, occlusion). The longer the human's location remains unknown to the system, the larger the area of ​​uncertainty (i.e., the area where the human might be) grows over time. For example, if the human's location is unknown for a period of time, the human might be in an unknown location, for example, within the range of approximately 2 m / s multiplied by the amount of time since the human's location was last known. For safety purposes, a processor-based cell safety system might cause the entire cell or operating environment to be treated as if someone were in the area. While this would be overly cautious, such treatment could adversely affect motion planning and robot operation. Alternatively, if the human's location is lost for a period of time, thus increasing the area of ​​uncertainty, a processor-based cell safety system could provide an indication that the area of ​​uncertainty should be treated as occlusion during the motion planning and / or movement execution of one or more robots.

[0013] To ensure security, both functional aspects of a processor-based cell security system (i.e., sensing and processing) must be considered secure.

[0014] Various methods can be used to address this issue. For example, the system can employ dual modular redundancy (DMR). DMR is sufficient because if the two modules are inconsistent, it is considered a detected problem, and the robot operation stops, slows down, or introduces the occluded area into the motion plan. As another example, a processor-based cell-based safety system can employ triple modular redundancy (TMR), where the system uses the output of the majority of modules (e.g., if two of the three modules are consistent, the system uses the output of the consistent module).

[0015] Regarding sensing, it would be advantageous to mitigate failure modes in sensor data, for example, by employing one or more failure mode and effects analysis (FMEA) processes or techniques. For illustrative purposes, some example failure mode and effects analysis processes or techniques are described below.

[0016] One possible failure mode effect analysis (FMEA) method or technique is to verify that the security system is receiving sensor data (e.g., images) as expected, for example, when the security system's processor should be receiving sensor data from the sensor, it is indeed receiving sensor data. For instance, if the sensor is an image sensor that samples or captures images at 30Hz, the security system's processor should receive an image every 1 / 30th of a second. This can be checked or verified, for example, via a watchdog mechanism. It is important to note that this check will not detect when the sensor is stuck (e.g., erroneously repeating or continuously sending the same stale sensor data, even if the sensed portion of the operating environment has changed).

[0017] Another possible approach to failure mode effect analysis involves shutting down the loop on sensor data, for example, determining whether the sensor data is meaningful or consistent with known conditions (e.g., whether the sensor data is consistent with known conditions regarding the position, orientation, and / or movement of various objects in an operating environment, which includes one or more robots and / or references and includes sensors). Any one or more of the following methods can be used to determine whether sensor data is consistent with or meaningful under given known conditions.

[0018] Processor-based cell safety systems can advantageously employ multiple heterogeneous sensors to monitor the operating environment. Heterogeneous sensors can include sensors with different sensor modes (e.g., different operating modes), located in different advantageous positions or otherwise having different fields of view, having different sampling rates, and / or from different manufacturers or different models. For example, a processor-based cell safety system can employ multiple sensors with different sensing modes (e.g., 1D laser scanners, 2D cameras, 3D cameras, time-of-flight (TOF) cameras, thermal sensors). Each of these sensors is not inherently reliable or safety-certified. Each sensor has a sampling rate (e.g., frame rate, image capture rate) at which it captures samples in some format (e.g., capturing images, capturing distance measurements, capturing 3D representations). Each sample depends on the sensing mode. While the use of heterogeneous sensors may hinder maintenance and is therefore often avoided, heterogeneous sensors (e.g., diversity in sensor modes, spatial (different advantageous locations), temporal (different times), manufacturers, and models) are particularly advantageous when employing COTS sensors to implement safety-certified processor-based cell-of-work safety systems. Diversity prevents common-mode failures (e.g., two sensors in the same advantageous location missing the same voxel, or two sensors of the same mode failing in the same way or due to the same conditions in the same operating environment).

[0019] If two or more sensors capture overlapping areas of the operating environment (also known as the cell of work), a processor-based cell of work safety system can compare sensor data from the two or more sensors to infer the likelihood of a fault. For example, images captured by two or more image sensors can be compared. With only two sensors, the processor-based cell of work safety system will not know which sensor is faulty, or even if both sensors are faulty. However, it is clear that these two sensors are untrusted. In response, the processor-based cell of work safety system may stop or slow down robot operation or movement to ensure safety. Alternatively, the processor-based cell of work safety system may indicate that an area or region monitored by the two sensors is occluded for motion planning purposes, thereby achieving a higher level of safety without completely stopping the operation or movement of one or more robots. If three or more sensors capture overlapping areas of the operating environment, the processor-based cell of work safety system is able to compare sensor data from the three or more sensors to determine whether the sensor data from the majority of sensors is consistent with each other. The processor-based cell of work safety system can then infer that the minority of sensors are untrusted and take action based on the consistent sensor data from the majority of sensors.

[0020] Some implementations can advantageously use one or more references that have moved in a known or perceptible (e.g., sensed) manner to determine whether sensor data received from one or more sensors is meaningful. For example, a processor-based cell safety system can know the position, location, and / or movement (e.g., direction and / or magnitude) of a reference. For example, a processor-based cell safety system might know that a given reference has moved 1 cm to the right over a given time period. The processor-based cell safety system can then know or determine what effect the known reference movement should have on the perception of that reference by any given sensor. For example, a processor-based cell safety system can compare an “before movement” image with a “after movement” image to detect certain malfunctions in a given sensor. For example, such a comparison can advantageously allow the processor-based cell safety system to determine that a given sensor is stuck, erroneously repeating the same stale image data (e.g., images) over and over again, even if the position of an object in the sensor’s field of view has changed over the relevant time period. For example, this might be indicated by a sensor that always perceives something in the same location (e.g., one square centimeter in the upper right corner of the sensor’s field of view) over a period of time, during which a part of the operating environment monitored by the sensor changes. If there is safe-certifiable knowledge about the robot's state or configuration (e.g., if the robot provides joint angles in a safe-certified manner or through some other mechanism), the robot or a part thereof can be used as a reference. Because the processor-based cell-of-work safety system knows where in space the robot or a part thereof, including or carrying the reference, should be at each moment when acquiring sensor data, the processor-based cell-of-work safety system is able to verify that the sensors observing the robot are functioning correctly.

[0021] Some implementations may employ one or more fixed references and one or more sensors that move in a known or identifiable manner in a safety-certified way. If the movement of one or more sensors is known (e.g., the joint positions of the robot carrying the sensors are known or can be queried in a safety-certified manner, or the number of rotations of the motor moving the sensors is known or can be queried in a safety-certified manner), then a processor-based cell-of-work safety system can compare sensor data collected before and after sensor movement to detect faults, for example, by comparing an image captured after movement with an image captured before sensor movement.

[0022] In some implementations, a processor-based work unit safety system may employ a default state indicating that the entire work unit or operating environment is occluded, relaxing this assumption only when there is consistent or mutually consistent sensor data from at least two sensors, meaning the area is not occluded. This default assumption is obviously pessimistic, but it ensures safety. Other implementations may indicate an area or region of the work unit or operating environment as occluded in response to determining that sensor data from at least two sensors covering an area or region of the work unit or operating environment is inconsistent or non-consistent. As noted, some implementations may operate based on determining that sensor data from a majority of sensors is consistent or mutually consistent.

[0023] A multifaceted FMEA approach includes sensor diversity (e.g., time, optics, geometry, sensor modality, manufacturer, model), checking the consistency of sensor data with known or known information that can be safely certified, and / or consistency or uniformity between sensors. This multifaceted FMEA approach may advantageously promote the use of COTS sensors while ensuring that a) the probability of the robot colliding with a person is low enough from all sensors, b) simultaneously, and c) for a sustained period of time due to missed human detection in the operating environment, to pass a sufficiently low hazard risk and thus qualify the safety system for safety certification.

[0024] A safety-certified operating environment or work cell can be decomposed into: i) a functional system operating the robot (i.e., the robot control system); and ii) a processor-based work cell safety system ensuring safety. The functional system can include one or more sensors and a processor-based system including one or more processors communicatively coupled to the sensors and performing motion planning and / or control of the robot. Similarly, the processor-based work cell safety system can include one or more sensors and a processor-based system including one or more processors communicatively coupled to the sensors and performing safety analysis. This operational separation is useful for several reasons, most notably because it allows the design of the functional robot motion planning and / or control system to be independent of the design of the processor-based work cell safety system.

[0025] In a safety-certified operating environment or cell of work, a processor-based cell of work safety system will trigger one or more robots to stop or slow down whenever a functional system causes a robot to get too close to a human, as defined by a set of safety rules. The concept of “too close” typically depends on how the safety system is configured and operates. For example, a processor-based cell of work safety system might use a laser scanner to divide the floor into an 8x8 grid, and the safety system would be triggered to interrupt robot operation whenever a robot is within one of the grid areas designated for a human.

[0026] A functional system can be designed or configured to know and consider how a processor-based cell safety system operates, in order to reduce or even avoid triggering safety-triggered stops, decelerations, or preventative occlusions by operating one or more robots in a manner that knows what will trigger the safety system. In other words, if the functional system knows how the processor-based cell safety system works and what triggers a stop, deceleration, or preventative occlusion, it can operate one or more robots to make it less likely to trigger the processor-based cell safety system. In the mesh-generating laser sensor example above, the functional system would know not to place a robot within a human's mesh area, even if the initial distance between the human and the robot is not necessarily dangerous. For example, the functional system could access a set of safety rules and conditions that the processor-based cell safety system enforces, or that the processor-based cell safety system relies on, to detect violations of safety rules.

[0027] Furthermore, the functional system can be optimized to consider anticipated or predicted human movement when performing motion planning, while reducing the likelihood of triggering safety systems. For example, the functional system can access human behavior models. Additionally or alternatively, the functional system can rely on logic that reflects humans entering the operating environment, trained according to a set of defined guidelines, thus anticipating that the human will remain within a fairly predictable segment of the operating environment or work cell, or otherwise move in a predictable manner (e.g., at a predictable or maximum speed). The functional system can take this information into account to generate motion planning in a safety-system-aware manner, allowing these plans to be optionally enhanced by predicted or anticipated human behavior. For example, if it is predicted that a human will enter a grid area next to the robot, the functional system can proactively move the robot away to avoid triggering safety systems, thereby avoiding stopping or slowing it down. Attached Figure Description

[0028] In the accompanying drawings, the same reference numerals identify similar elements or actions. The dimensions and relative positions of the elements in the drawings are not necessarily drawn to scale. For example, the shapes and angles of various elements are not drawn to scale, and some of these elements are arbitrarily enlarged and positioned to improve the readability of the drawing. Furthermore, the specific shapes of the elements drawn are not intended to convey any information about the actual shape of the particular element and are only selected for ease of identification in the accompanying drawings.

[0029] Figure 1 This is a schematic diagram of a robot system according to an embodiment shown, which includes multiple robots operating in an operating environment to perform tasks, and includes one or more robot control systems having motion planners that generate motion plans for robot dynamics, and one or more processor-based cell safety systems that monitor hazards in the operating environment, such as humans entering the robot path.

[0030] Figure 2 This is a functional block diagram of a processor-based cell safety system according to an embodiment shown. The processor-based cell safety system includes multiple sensors and at least one processor, which is communicatively coupled to the sensors and operable to evaluate the operational status of the sensors and the system status of the safety system to determine whether there is an abnormal system status and to take appropriate measures based on the system status, as well as to monitor whether an unsafe condition occurs in the operating environment.

[0031] Figure 3 This is a functional block diagram of a first robot and a robot control system with a motion planner according to an embodiment shown, the motion planner generating motion plans to control the operation of at least the first robot.

[0032] Figure 4 This is an example motion planning diagram of a robot operating in an operating environment or work cell according to an embodiment shown.

[0033] Figure 5 This is a flowchart illustrating an advanced operation method in a processor-based work cell safety system according to at least one of the illustrated embodiments, the operation method being used to implement safety monitoring of the operating environment to control the operation of a robot in the operating environment and to verify the safety monitoring system.

[0034] Figure 6 This is a flowchart illustrating a low-level operation method in a processor-based work unit safety system according to at least one of the illustrated embodiments. This low-level operation method is used to implement safety monitoring of the operating environment to control robot operation within the operating environment and to verify the safety monitoring system. The low-level method can be used as an execution... Figure 5This is part of the execution of the advanced method shown.

[0035] Figure 7 This is a flowchart illustrating a low-level operation method in a processor-based work unit safety system according to at least one of the illustrated embodiments. This low-level operation method is used to implement safety monitoring of the operating environment to control robot operation within the operating environment and to verify the safety monitoring system. The low-level method can be used as an execution... Figure 5 This is part of the execution of the advanced method shown.

[0036] Figure 8 This is a flowchart illustrating a low-level operation method in a processor-based cell safety system according to at least one of the illustrated embodiments, the low-level operation method being used to control robot operations in an operating environment thereby reducing the triggering of the processor-based cell safety system. Detailed Implementation

[0037] In the following description, certain specific details are set forth in order to provide a comprehensive understanding of the various disclosed embodiments. However, those skilled in the art will recognize that embodiments can be practiced without one or more of these specific details or by other methods, components, materials, etc. In other instances, well-known structures associated with computer systems, actuator systems, and / or communication networks have not been shown or described in detail to avoid unnecessarily obscuring the description of the embodiments. In other instances, well-known computer vision methods and techniques for generating perceptual data and volumetric representations of one or more objects have not been described in detail to avoid unnecessarily obscuring the description of the embodiments.

[0038] Unless the context otherwise requires, throughout the specification and appended claims, the word “comprising” and its variations, such as “including” and “constituting”, shall be interpreted in an open, inclusive sense as “including, but not limited to”.

[0039] Throughout this specification, references to "an implementation," "an embodiment," or "an example" or "an example" mean that a particular feature, structure, or characteristic described in connection with that example is included in at least one implementation or in at least one implementation. Therefore, the phrases "an implementation," "an embodiment," or "an example" appearing throughout this specification do not necessarily refer to the same implementation or example. Furthermore, a particular feature, structure, or characteristic may be combined in any suitable manner in one or more implementations or examples.

[0040] As used in this specification and the appended claims, unless otherwise expressly specified, the singular forms “a,” “an,” and “the” include the plural objects. It should also be noted that, unless otherwise expressly specified, the term “or” is generally used in its sense encompassing “and / or.”

[0041] As used in this specification and the appended claims, the terms determine, determining, and determined, when used in the context of whether a conflict will occur or arise, mean assessing or predicting whether a given posture or movement between two postures via multiple intermediate postures will result in a conflict between a part of the robot and some object (e.g., another part of the robot, a part of another robot, a persistent obstacle, a transient obstacle, such as a person).

[0042] As used in this specification and the appended claims, reference to one or more robots means one or more robots and / or parts of one or more robots.

[0043] As used in this specification and the appended claims, the term "reference point" means a standard of reference, such as an object and / or mark or set of marks appearing in the field of view of one or more sensors (e.g., one or more image sensors of an imaging system) in sensor data (e.g., images) generated by one or more sensors, to serve as a reference point or measurement point. One or more reference points may be placed within or on one or more robots, or may be mounted to move independently of one or more robots.

[0044] As used in this specification and the appended claims, the term "path" means a set of points or a trajectory of points in two- or three-dimensional space, and the term "trajectory" means a path that includes the time it takes to reach some of these points, and may also include velocity and / or acceleration values.

[0045] The headings and abstracts of this disclosure provided herein are for convenience only and do not define the scope or meaning of the embodiments.

[0046] Figure 1 A robot system 100 according to one illustrated embodiment is shown, which includes one or more robots 102a, 102b (the two robots shown are collectively referred to as 102), which operate in an operating environment 104 (also referred to as a work cell) to perform tasks.

[0047] Robot 102 can take any of a variety of forms. Typically, robot 102 will take the form of one or more robotic appendages or have one or more robotic appendages. Robot 102 may include one or more links with one or more joints, and actuators (e.g., electric motors, stepper motors, solenoids, pneumatic actuators, or hydraulic actuators) coupled to the links and operable to move the links in response to control or drive signals. Pneumatic actuators may, for example, include one or more pistons, cylinders, valves, air tanks, and / or pressure sources (e.g., compressors, blowers). Hydraulic actuators may, for example, include one or more pistons, cylinders, valves, fluid (e.g., low-compressibility hydraulic fluid) reservoirs, and / or pressure sources (e.g., compressors, blowers). Robot system 100 may take other forms of robot 102, such as autonomous vehicles with or without movable appendages.

[0048] Although in some limited embodiments the operating environment 104 may represent a two-dimensional space, it typically represents a three-dimensional space in which robots 102a and 102b can operate and move. The operating environment 104 is a volume or region in which at least a portion of robots 102a and 102b may overlap spatially and temporally or otherwise transmit conflict if motion is not controlled to avoid collisions. It should be noted that the work cell or operating environment 104 differs from the corresponding “configuration space” or “C-space” of robots 102a and 102b.

[0049] As described herein, robot 102a or a portion thereof may constitute an obstacle when considered from the perspective of another robot 102b (i.e., when motion planning is performed on another robot 102b). The operating environment 104 may additionally include other obstacles, such as mechanical components (e.g., conveyor 106), pillars, columns, walls, ceilings, floors, tables, humans, and / or animals. The operating environment 104 may additionally include one or more work items or workpieces 108 manipulated by robot 102 as part of performing a task, such as one or more packages, packs, fasteners, tools, articles, or other objects. In at least some embodiments, the operating environment 104 may additionally include one or more references 111a, 111b (the two references shown are collectively referred to as 111). As described in detail herein, references 111a, 111b may help determine whether one or more sensors are functioning correctly. One or more references 111a may be unique parts of robot 102b or unique parts carried by a portion of robot 102b, and move with that part of the robot in a known or discernible manner that can be safely authenticated (e.g., a known or discernible trajectory over time, such as a trajectory known or discernible based on joint rotation angles). One or more references 111b may be separate from and different from robots 102a and 102b, and are mounted for movement (e.g., movement on track or guide rail 113) and driven by actuators (e.g., motors, solenoids) to move in a known or discernible manner that can be safely authenticated (e.g., a known or discernible trajectory over time, such as a trajectory known or discernible based on the rotational speed of a motor-driven shaft captured by a rotary encoder).

[0050] Robot system 100 may include one or more robot control systems 109a, 109b (the two robot control systems shown are collectively referred to as 109), and the one or more robot control systems 109a, 109b include one or more motion planners, such as corresponding motion planners 110a, 110b for each of robots 102a, 102b respectively (the two motion planners shown are collectively referred to as 110). In at least some embodiments, a single motion planner 110 may be used to generate motion plans for two, more, or all robots 102. The motion planner 110 is communicatively coupled to control the corresponding robot in robot 102. The motion planner 110 is also communicatively coupled to receive various types of input, such as robot geometry models 112a, 112b (also referred to as kinematic models, collectively referred to as 112), tasks 114a, 114b (collectively referred to as 114), and motion plans 116a, 116b (collectively referred to as 116), or other representations of the motion of other robots 102 operating in the operating environment 104. The robot geometry model 112 defines the geometry of a given robot 102, for example, based on joints, degrees of freedom, dimensions (e.g., link lengths), and / or based on the corresponding C-space of the robot 102. The transformation from the robot geometry model 112 to the motion planning graph can occur at runtime or, for example, be performed by a processor-based server system (not included in the original text). Figure 1 Prior to the execution of the task (shown in the diagram). Alternatively, for example, the motion planning diagram may be generated by one or more processor-based robot control systems 109a, 109b using any of a variety of techniques. Task 114 specifies the task to be performed, for example, based on the final pose, final configuration, or final state and / or intermediate pose, intermediate configuration, or intermediate state of the corresponding robot 102. For example, the pose, configuration, or state may be defined based on the joint positions and joint angles / rotations (e.g., joint poses, joint coordinates) of the corresponding robot 102.

[0051] Motion planners 110a and 110b are optionally communicatively coupled to receive static object data 118a and 118b (collectively referred to as 118) as input. Static object data 118 represents static objects (e.g., size, shape, location, space occupied) within the work cell or operating environment 104, which may be, for example, known a priori. Static objects may include, for example, one or more fixed structures within the work cell or operating environment, such as columns, pillars, walls, ceilings, floors, or conveyors 106. Since robots 102 operate within a shared work cell or operating environment 104, the static objects are typically the same for each robot. Therefore, in at least some embodiments, the static object data 118a and 118b supplied to motion planner 110 will be identical. In other embodiments, the static object data 118a and 118b supplied to motion planner 110 may be different for each robot, for example, based on the robot 102's position or orientation in the environment or the robot 102's environmental perspective. Additionally, as described above, in some embodiments, a single motion planner 110 can generate motion plans for two or more robots 102.

[0052] The motion planner 110 is optionally communicatively coupled to receive, for example, sensing data 120 provided by the sensing subsystem 124 as input. The sensing data 120 represents static and / or dynamic objects in the work unit or operating environment 104 that are not known a priori. The sensing data 120 may be raw data sensed via one or more sensors (e.g., two-dimensional or three-dimensional cameras 122a, 122b, time-of-flight cameras, laser scanners, LIDAR, LED-based photoelectric sensors, laser-based sensors, ultrasonic sensors, sonar sensors) and / or raw data converted by the sensing subsystem 124 into a digital representation of the obstacle. Such sensors may take the form of COTS sensors and may or may not be used as part of a safety-certified safety system.

[0053] The optional perception subsystem 124 may include one or more processors that can execute one or more machine-readable instructions that enable the perception subsystem 124 to generate a corresponding discretized representation of the environment in which the robot 102 will operate to perform tasks for a variety of different scenarios.

[0054] Optional sensing sensors (e.g., cameras 122a, 122b) provide raw sensing information (e.g., point clouds) to the sensing subsystem 124. The optional sensing subsystem 124 can process the raw sensing information and can provide the resulting sensing data as a point cloud, an occupancy mesh, a box (e.g., a bounding box), or other geometric object, or a voxel stream representing obstacles present in the environment (i.e., a “voxel” is equivalent to a 3D or volumetric pixel). The representation of obstacles can optionally be stored in the on-chip memory of one or more processors, such as any one or more processors of the optional sensing subsystem 124. The sensing data 120 can indicate which voxels or sub-volumes (e.g., boxes) are occupied in the environment at the current time (e.g., during runtime). In some embodiments, when representing a robot or another obstacle in the environment, the corresponding surface of the robot or obstacle (e.g., including other robots) can be represented as a voxel or a polygon mesh (typically triangles). In some cases, it is advantageous to alternatively represent objects as boxes (rectangular prisms, bounding boxes) or other geometric objects. Because the shape of an object is not random, there can be a great deal of structure in the organization of voxels; in 3D space, many voxels within an object are adjacent to each other. Therefore, representing an object as a box may require fewer bits (i.e., perhaps only the x, y, z Cartesian coordinates of the two opposite corners of the box). Furthermore, the complexity of performing an intersection test on a box is similar to that of performing an intersection test on a voxel.

[0055] At least some implementations can combine the outputs of multiple sensors, and the sensors can provide very fine-grained voxelization. However, in order for the motion planner to perform motion planning effectively, coarser voxels (i.e., "processor voxels") can be used to represent the environment and volume in 3D space swept by the robot 102 or a portion thereof as transitions occur between various states, configurations, or poses. Therefore, the optional perception subsystem 124 can correspondingly translate the outputs of the sensors (e.g., cameras 122a, 122b). For example, the outputs of cameras 122a, 122b can use 10-bit precision on each axis, so each voxel directly derived from cameras 122a, 122b has a 30-bit ID, and there are 2 30 Each sensor voxel. The robot control systems 109a and 109b can use 6-bit precision for the 18-bit processor voxel ID on each axis, and there will be 2... 18 There are 2 processor voxels. Therefore, for example, for each processor voxel, there might be 2 12 Each sensor voxel. During operation, if the system determines that any of the sensor voxels within the processor voxel are occupied, the robot control systems 109a and 109b assume that the processor voxel will be occupied and generate an occupancy grid accordingly.

[0056] The robot system 100 may include one or more processor-based cell safety systems 130 (one cell safety system is shown), which includes multiple sensors, such as a first sensor 132a, a second sensor 132b, a third sensor 132c, and a fourth sensor 132d (the four sensors shown are collectively referred to as 132), and one or more processors 134 communicatively coupled to the sensors 132 of the safety system 130.

[0057] Sensors 132 are positioned and oriented to jointly sense or monitor most or even all of the operating environment 104. Preferably, at least pairs of sensors 132 overlap within the coverage area of ​​various parts of the operating environment to facilitate safety-certified operation by applying FMEA methods or techniques. Although four sensors 132 are shown, fewer or even possibly more sensors 132 may be employed. The total number of sensors 132 used by the safety system 130 will generally depend in part on the size and configuration of the operating environment, the type of sensors 132, the required or specified level of security, and / or the level or extent of conflict considered acceptable. As described herein, sensors 132 may advantageously take the form of COTS sensors, but the entire processor-based cell safety system 130 is safety-certified by applying FMEA methods or techniques (at least some of which are described herein).

[0058] Sensor 132 preferably includes a set of heterogeneous sensors.

[0059] The heterogeneous sensor 132 may, for example, take the form of a first sensor having a first operating mode and a second sensor having a second operating mode. The second operating mode may advantageously differ from the first operating mode. In this embodiment, the processor-based system advantageously receives information from the first sensor in a first modal format and from the second sensor in a second modal format, which differs from the first modal format. For example, the first sensor may be in the form of an image sensor and the first modal format is a digital image. As another example, the second sensor may be in the form of a laser scanner, a passive infrared (PIR) motion sensor, ultrasound, sonar, LiDAR, or a thermal sensor, and the second modal format is an analog signal or a digital signal, neither of which is a digital image format. In any given embodiment, there may be a third sensor, a fourth sensor, or even more sensors, each with its own corresponding operating mode, thereby increasing diversity and heterogeneity.

[0060] The heterogeneous sensor 132 may, for example, take the form of a first sensor with a first field of view of the operating environment and a second sensor with a second field of view of the operating environment, the second field of view being different from the first field of view. In this implementation, the processor-based system advantageously receives information from the first sensor with the first field of view and from the second sensor with the second field of view. In any given implementation, there may be a third sensor, a fourth sensor, or even more sensors, each with its own corresponding field of view, thereby increasing diversity and heterogeneity. In some cases, the fields of view of two or more sensors may partially or completely overlap, and some fields of view of two or more sensors may be almost identical in all respects.

[0061] The heterogeneous sensor 132 may, for example, take the form of a first sensor having a first sensor brand (i.e., manufacturer) and model number, and a second sensor having a second sensor brand and model number, wherein at least one of the second brand or model number of the second sensor differs from the corresponding first brand and model number of the first sensor. In this implementation, the processor-based system advantageously receives information from the first sensor in a first format and from the second sensor in a second format, the first format being specific to the first sensor brand and / or model number, and the second format being specific to the second sensor brand and / or model number. In any given implementation, a third sensor, a fourth sensor, or even more sensors may be present, each with its own corresponding brand and model number, thereby increasing diversity and heterogeneity.

[0062] Heterogeneous sensors can, for example, take the form of a first sensor with a first sampling rate and a second sensor with a second sampling rate different from the first sampling rate. In this implementation, the processor-based system advantageously receives information from the first sensor capturing data at the first sampling rate and from the second sensor capturing data at the second sampling rate. In any given implementation, there may be a third, fourth, or even more sensors, each with its own corresponding sampling rate, thereby increasing diversity and heterogeneity.

[0063] Any combination of heterogeneous sensors can be used. Generally, although increasing the heterogeneity of sensor groups may adversely increase maintenance costs and is therefore usually avoided, increasing the heterogeneity of sensor groups can be advantageously used to achieve security certification of the entire security system.

[0064] Sensor 132 may be separate from and distinct from cameras 122a and 122b of the sensing subsystem 124. Alternatively, one or more sensors 132 may be part of the sensing subsystem 124. Sensor 132 may take any of a variety of forms capable of sensing objects in the operating environment 104 and, in particular, sensing the operating environment 104 to detect the presence, location, and / or movement or trajectory of one or more humans in the operating environment 104. Sensor 132 may take the form of, by way of non-limiting example, a two-dimensional digital camera, a three-dimensional digital camera, a time-of-flight camera, a laser scanner, a laser-based sensor, an ultrasonic sensor, sonar, a passive infrared sensor, a LiDAR, and / or a thermal sensor. As used herein, the term “sensor” includes a sensor or transducer that detects physical characteristics of the operating environment 104, and any transducer or other energy source associated with such a sensor, such as a light-emitting diode, other light source, laser and laser diode, loudspeaker, haptic engine, ultrasonic energy, etc.

[0065] Although not explicitly stated, some implementations may include other types of sensors that detect when a human enters the operating environment; for example, RFID interrogation systems that detect radio frequency identification (RFID) transponders worn by the human body, laser scanners, pressure sensors, and passive infrared (PIR) motion detectors that detect the presence of a human in the work unit, but not necessarily the human's position in the work unit.

[0066] One or more processors 134 of the processor-based cell security system 130 and other components (e.g., communication ports, radio devices, analog-to-digital converters) are communicatively coupled to sensor 132 to receive sensor data from sensor 132. One or more processors 134 of the processor-based cell security system 130 execute logic, which is stored, for example, as processor-executable instructions in a non-transitory processor-readable medium (e.g., read-only memory, random access memory, flash memory, solid-state drive, magnetic hard disk drive).

[0067] For example, the processor-based cell safety system 130 may store one or more sets of sensor state rules 125a on at least one non-transitory processor-readable medium. The sensor state rules 125a specify rules, operating conditions, values ​​or ranges of values ​​for various parameters, and / or other criteria for a given sensor 132 or sensor type. The processor-based cell safety system 130 may apply the sensor state rules 125a to evaluate or otherwise determine the operating state of any given sensor 132, i.e., whether the corresponding sensor 132 is operating within normal or acceptable limits (i.e., fault-free condition, operable condition), or to identify a faulty or potentially faulty sensor 132 or other unacceptable conditions (i.e., fault condition, inoperable condition). The evaluation may assess one, two, or more operating states for each of the sensors 132. The sensor operating state can be based on an evaluation of any one or more of the following: ON or OFF state; the sensor providing sensor information; the sensor providing sensor information at its nominal sampling rate; the sensor not being stuck (i.e., the sensor information provided by the sensor is changing; changing in an expected manner relative to known predefined environmental conditions; and / or changing in a manner consistent with changes sensed by one or more other sensors, such as movement of another robot or other fiduciary). The evaluation can, for example, assess the operating state of a given sensor by comparing two or more of the sensors 132 (e.g., comparing the outputs of two or more sensors 132), examples of which are described herein. As an example, each sensor 132 can be associated with a corresponding sampling rate. Sensor state rule 125a can define a corresponding acceptable sampling range or a percentage of sampling rate error considered acceptable, or conversely, define a similar value considered unacceptable. Also as an example, sensor state rule 125a can define a corresponding amount of time a sensor 132 might be stuck, or a frequency used to confirm that a sensor 132 is not stuck, which is considered acceptable, or conversely, define a similar value considered unacceptable. An assessment of the operating condition or operational status of a sensor can indicate whether one or more sensors 132 are operating as expected and / or within a set of defined performance parameters or conditions, and thus enable reliance on individual sensors to provide a safe operating unit or operating environment 104 or to determine whether a faulty or inoperable condition or a potential faulty or inoperable condition exists.

[0068] Sensor status rules 125a can be stored by sensor type or even searched by individual sensor identity.

[0069] For example, the processor-based cell-of-work safety system 130 may store one or more sets of system verification rules 125b on at least one non-transitory processor-readable medium. System verification rules 125b specify rules, operating conditions, parameter values, and / or other criteria for verifying the operational status of the processor-based cell-of-work safety system 130. Verification may be based on, for example, the determined operational status of sensors 132. System verification rules 125b may specify, for example, rules for selecting sensors 132 and / or selecting one or more sets of sensors 132 (e.g., all sensors must be operational; sensors identified as necessary must be operational, while other sensors may be operational or not; a majority of sensors in a set must be consistent). The processor-based cell-of-work safety system 130 may evaluate or otherwise apply system verification rules 125b to determine whether there are sufficient sensors 132 operating within normal or acceptable limits to rely on the safety system 130 to ensure secure certified operation. When there are enough sensors 132 operating normally or within acceptable limits to rely on the security system 130 for secure authentication operations, the processor-based cell-based security system 130 can identify or indicate the presence of a non-abnormal system state. Conversely, if there are not enough sensors 132 operating normally or within acceptable limits to rely on the security system 130 for secure authentication operations, the processor-based cell-based security system 130 can identify or indicate the presence of an abnormal system state.

[0070] As also described herein, the processor-based cell safety system 130 may optionally determine whether the results of system verification indicate an anomalous or non-anomalous system state for the processor-based cell safety system 130, which would render the entire processor-based cell safety system 130 unreliable. This may be based at least in part on the evaluation of a first sensor, a second sensor, and possibly more sensors. The system state of the processor-based cell safety system 130 can be defined via a set of system verification rules 125b, which specifies how many sensors 132 and / or which sensors 132 can be considered operational or reliable due to the presence of a non-anomalous state, or conversely, how many sensors 132 and / or which sensors 132 can be considered inoperable or unreliable due to the presence of an anomalous system state. System verification rules 125b may specify that an error or fault indication or operational state defined in any single specific sensor (i.e., a necessary or required sensor) of the sensor 132 constitutes an anomalous system state of the processor-based cell safety system 130. System verification rule 125b may specify that an error or fault indication or operational state defined in a set of two or more specific sensors 132 constitutes an abnormal system state of the processor-based cell safety system 130. For example, the detection of a fault condition or faulty operational state in any single sensor in a set of sensors, or the detection of a fault condition or faulty operational state in all sensors in a set of sensors, or the detection of a fault condition or faulty operational state in most sensors in a set of sensors constitutes an abnormal system state of the processor-based cell safety system 130. Alternatively, system verification rule 125b may define an abnormal system state of the processor-based cell safety system 130 when there is inconsistency among most of the sensors 132. In some embodiments, when there is consistency among most of the sensors 132, at least one processor may determine that the sensors 132 are sufficiently reliable to provide safe operation within or in parts of the operating environment.

[0071] For example, the processor-based cell safety system 130 may store one or more sets of safety monitoring rules 125c on at least one non-transitory processor-readable medium. Safety monitoring rules 125c specify rules, conditions, parameter values, and / or other standards for assessing whether the operating environment violates specified safety standards. For example, safety monitoring rule 125c may specify rules or standards requiring the maintenance of specific conditions between a robot or a part thereof and an object that is or may be human. For example, safety monitoring rule 125c may specify the existence of at least one defined measurement unit (e.g., a grid area) between an object (e.g., a human) and a part of the robot or the robot's path or trajectory, along which the robot will move over time. The processor-based cell safety system 130 may evaluate sensor data provided by one or more sensors 132 to determine the location of an object and / or assess whether the object is or may be human. The processor-based cell safety system 130 can evaluate sensor data provided by one or more sensors 132, sensor data provided by the perception subsystem 124, and / or information (e.g., joint angles) from the robot control systems 109a, 109b or from the robots 102a, 102b themselves to determine the position and orientation and / or trajectory of the robots 102a, 102b over a given time. The processor-based cell safety system 130 can determine whether the position, path, or trajectory of a human and the position, path, or trajectory of one or more robots 102a, 102b will violate one or more safety monitoring rules 125c. In response to detecting a violation of safety monitoring rule 125c, the processor-based cell safety system 130 can provide one or more signals that result in stopping, mitigation, preventative occlusion, or otherwise inhibiting the operation of one or more robots 102a, 102b.

[0072] Various communication paths in Figure 1The arrows are shown in the diagram. Communication paths can take the form of one or more wired communication paths (e.g., electrical conductors, signal buses, or optical fibers) and / or one or more wireless communication paths (e.g., via RF or microwave radios and antennas, infrared transceivers). Notably, each of the motion planners 110a, 110b is directly or indirectly communicatively coupled to each other to provide motion plans for the respective robots in robots 102a, 102b to the other motion planners in motion planners 110a, 110b. For example, motion planners 110a, 110b can be communicatively coupled to each other via network infrastructure such as a non-proprietary network infrastructure (e.g., Ethernet network infrastructure) 126. Also noteworthy is that the processor-based cell safety system 130 is optionally communicatively coupled to the robot control systems 109a, 109b to provide signals to them. For example, the processor-based cell safety system 130 can provide signals to stop or slow the movement of one or more robots 102, for example, in response to determining the presence of an abnormal system state. For example, the processor-based cell safety system 130 can signal, for instance, robot control systems 109a, 109b to cause motion planners 110a, 110b to identify one or more zones or areas of the operating environment as occluded. For example, in response to determining that one or more corresponding sensors 132 are operating outside a set of expected conditions (e.g., a faulty operating state of one or more sensors), a zone or area monitored by one or more sensors can be identified as occluded. For example, the processor-based cell safety system 130 can provide access to one or more sets of safety monitoring rules 125c to robot control systems 109a, 109b and / or motion planners 110a, 110b. This allows the robot control portion of robot system 100 (e.g., robot control systems 109a, 109b, motion planners 110a, 110b) to advantageously consider the configuration of the safety system when developing and / or executing motion planning, particularly conditions that would trigger inhibition of robot operation, as described herein. Therefore, although the functional parts of the robot system 100 can generally be configured independently of the processor-based cell safety system 130, the robot control systems 109a, 109b can advantageously take into account the operation of the processor-based cell safety system 130 to reduce the use of stopping, slowing down and / or preventive occlusion.

[0073] The term "environment" refers to the robot's current work cell, which is the operational environment in which one, two, or more robots operate within the same workspace. The environment can include obstacles and / or workpieces (i.e., objects that the robot interacts with, acts upon, or acts upon). The term "task" refers to a robot task in which the robot transitions from posture A to posture B without interfering with obstacles in its environment. A task may involve grasping or releasing an object, moving or placing an object, rotating an object, or retrieving or placing an object. The transition from posture A to posture B may optionally include transitions between one or more intermediate postures. The term "scenario" refers to a class of environment / task pairs. For example, a scenario could be "a pick-and-place task in an environment with a 3-foot table or conveyor and between x- and y-obstacles of a given size and shape." There may be many different task / environment pairs that meet such criteria, depending on the location of the target and the size and shape of the obstacles.

[0074] Motion planner 110 is operable to dynamically generate motion plan 116 to enable robot 102 to perform tasks in the environment, while taking into account the planned motions of other robots in robot 102 (e.g., represented by the corresponding motion plan 116 or the resulting sweep volume) and / or optionally, the rules and conditions adopted by processor-based cell safety system 130. Motion planner 110 may optionally consider representations of prior static objects and / or perception data 120 represented by static object data 118 when generating motion plan 116. Optionally, motion planner 110 may consider safety monitoring rules 125c implemented by processor-based cell safety system 130 when generating motion plan. Optionally, motion planner 110 may consider the motion states of other robots 102 at a given time, such as whether another robot 102 has already completed a given motion or task, and allow recalculation of motion plan based on the motion or task of one of the other robots being completed, thereby making previously excluded paths or trajectories available. Optionally, the motion planner 110 may take into account the operating conditions of the robot 102, such as the occurrence or detection of fault conditions, the occurrence or detection of obstruction conditions and / or the occurrence or detection of acceleration requests, or alternatively delay or skip motion planning requests.

[0075] Figure 2 A processor-based cell security system 200 according to one illustrated embodiment is shown. The processor-based cell security system 200 can implement the processor-based cell security system 130. Figure 1 ).

[0076] Processor-based work unit security system 200 may include multiple sensors 232, preferably a set of heterogeneous sensors, one or more processors 222, and one or more associated non-transitory computer- or processor-readable storage media, such as system memory 224a, disk drive 224b, and / or memory or registers of processor 222 (not shown). The non-transitory computer- or processor-readable storage media are communicatively coupled to one or more processors 222 via one or more communication channels such as system bus 227. System bus 227 can employ any known bus structure or architecture, including memory bus with memory controller, peripheral bus, and / or local bus. One or more of these components may also, or alternatively, communicate with each other via one or more other communication channels, such as via one or more parallel cables, serial cables, or wireless network channels capable of high-speed communication, such as Universal Serial Bus (“USB”) 3.0, Peripheral Component Interconnect Express (PCIe), or via... They communicate with each other.

[0077] As described above, the processor-based work unit security system 200 may include one or more processors 222 (i.e., circuits), non-transitory storage media, and a system bus 227 coupling various system components. The processors 222 may be any logic processing unit, such as one or more central processing units (CPUs), digital signal processors (DSPs), graphics processing units (GPUs), field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), programmable logic controllers (PLCs), etc. The system memory 224a may include read-only memory (“ROM”) 226, random access memory (“RAM”) 228, flash memory 230, and EEPROM (not shown). A basic input / output system (“BIOS”) 232, capable of forming a portion of the ROM 226, contains basic routines that facilitate the transfer of information between components within the processor-based work unit security system 200, for example, during startup.

[0078] Disk drive 224b may be, for example, a hard disk drive for reading from and writing to a disk, a solid-state (e.g., flash memory) drive for reading from and writing to a solid-state memory, and / or an optical disk drive for reading from and writing to a removable optical disk. The processor-based unit-of-work security system 200 may also include any combination of such drives in various different embodiments. Disk drive 224b may communicate with one or more processors 222 via system bus 227. One or more disk drives 224b may include an interface or controller (not shown) coupled between such a drive and system bus 227, as is known to those skilled in the art. Disk drive 224b and its associated computer-readable medium provide the processor-based unit-of-work security system 200 with non-volatile storage of computer- or processor-readable and / or executable instructions, data structures, program modules, and other data. Those skilled in the art will understand that other types of computer-readable media capable of storing computer-accessible data may be employed, such as WORM drives, RAID drives, magnetic tape cassettes, digital video discs (“DVDs”), Bernoulli cassette tapes, RAM, ROM, smart cards, etc.

[0079] Executable instructions and data can be stored in system memory 224a, such as operating system 236, one or more application programs 238, other programs or modules 240, and data 242. Application program 238 may include processor-executable instructions that cause one or more processors 222 to perform one or more of the following operations: at least in part based on sensor state rule 125a (… Figure 1 Assess the sensor's operational status; at least in part based on system verification rule 125b. Figure 1 The system's operational status is assessed to determine whether the processor-based work unit safety system 200 exhibits a non-abnormal or abnormal system state based on the sensor status of specific sensors 232 and / or groups of sensors 232, and at least in part based on safety monitoring rule 125c. Figure 1 The system performs security monitoring of the operating environment. One or more processors 222 can execute instructions that perform various algorithms listed herein, such as method 500, method 600, method 700, and method 800 (respectively...). Figure 5 , Figure 6 , Figure 7 and Figure 8The application 238 may additionally include one or more machine-readable and machine-executable instructions that cause one or more processors 222 to perform other operations, such as optionally processing sensor data captured via sensor 232. The application 238 may additionally include one or more machine-executable instructions that cause one or more processors 222 to perform various other methods described herein and by reference incorporated herein.

[0080] Data 242 may, for example, include one or more sets of sensor state rules 125a stored on at least one non-transitory processor-readable medium. Figure 1 Data 242 may, for example, include one or more sets of system verification rules 125b on at least one non-transitory processor-readable medium. Figure 1 Data 242 may, for example, be included on at least one set or more sets of security monitoring rules 125c on a non-transitory processor-readable medium. Figure 1 ).

[0081] In various implementations, one or more of the above operations may be performed by one or more remote processing devices or computers linked via a communication network through a network interface.

[0082] Although Figure 2 The system is shown to be stored in system memory 224a, but the operating system 236, application program 238, other programs / modules 240 and program data 242 can be stored on other non-transitory computer or processor readable media such as disk drive 224b.

[0083] One or more processors 222 may be or may include any logic processing unit, such as one or more central processing units (CPUs), digital signal processors (DSPs), graphics processing units (GPUs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), programmable logic controllers (PLCs), etc. Non-limiting examples of commercial computer systems include, but are not limited to, those from Intel Corporation (U.S.). Microprocessors from: Corporation of America (Celron, Core, Core 2, Itanium and Xeon series); Advanced Micro Devices (K8, K10, Bulldozer and Bobcat series); Apple Computer (A5, A6 and A7 series); Qualcomm (Snapdragon series); and Oracle Corp. (SPARC series). Figure 2 The construction and operation of the various structures shown can be implemented or adopted using international patent application No. PCT / US2017 / 036880, filed on June 9, 2017, entitled "MOTION PLANNING FOR AUTONOMOUS VEHICLES AND RECONFIGURABLE MOTION PLANNING PROCESSORS", international patent application No. WO2016 / 122840, filed on January 5, 2016, entitled "SPECIALIZED ROBOT MOTION PLANNING HARDWARE AND METHODS OF MAKING AND USING SAME", and / or the patent application No. APPARATUS, METHOD AND ARTICLE TO FACILITATE MOTION PLANNING OF AN AUTONOMOUS VEHICLE IN AN ENVIRONMENT HAVING DYNAMIC, filed on January 12, 2018. The structures, techniques and algorithms described in U.S. Patent Application No. 62 / 616,783 for “OBJECTS”, or those structures, techniques and algorithms similar to those described.

[0084] Although not required, many implementations will be described in the general context of computer-executable instructions, which are, for example, program application modules, objects, or macros stored on a computer or processor-readable medium and executed by one or more computers or processors, capable of performing obstacle representation, collision assessment, and other motion planning operations.

[0085] Figure 3 A first robot control system 300 and a first robot 302 according to at least the illustrated embodiments are shown. The first robot control system 300 includes a first motion planner 304 that generates a first motion plan 306 to control the operation of the first robot 302.

[0086] Similarly, other motion planners in other robot control systems generate other motion plans to control other robots. Figure 3 (The operation is not shown in the image).

[0087] One or more robot control systems 300 may be communicatively coupled, for example, via at least one communication channel (e.g., transmitter, receiver, transceiver, radio, router, Ethernet), to receive motion planning maps and / or sweep volume representations from one or more sources of motion planning maps and / or sweep volume representations. According to one illustrated embodiment, one or more sources of motion planning maps and / or sweep volumes may be separate from and distinct from motion planner 304. One or more sources of motion planning maps and / or sweep volumes may be, for example, one or more processor-based computing systems (e.g., server computers), which may be operated or controlled by the respective manufacturer of robot 302 or by some other entity. Motion planning maps may include a set of nodes representing the state, configuration, or pose of the respective robot, and a set of edges coupling the nodes in the respective node pairs and representing a legal or valid transition between states, configurations, or poses. For example, a state, configuration, or pose may represent the joint position, orientation, pose, or set of coordinates of each joint in the joints of the respective robot 302. Therefore, each node can represent the pose of robot 302 or a portion thereof, which is entirely defined by the pose of the joints comprising robot 302. The motion planning graph can be determined, set, or defined before runtime (i.e., before task execution), for example, during pre-runtime or configuration time. The sweep volume represents the corresponding volume that robot 302 or a portion thereof will occupy when performing a motion or transformation corresponding to the corresponding edge of the motion planning graph. The sweep volume can be represented in any of a variety of forms, such as voxels, Euclidean distance fields, or hierarchical structures of geometric objects. This advantageously allows some of the most computationally intensive tasks to be performed before runtime, when the response is not of particular concern.

[0088] One or more robot control systems 300 may be optionally communicatively coupled, for example, via at least one communication channel (e.g., transmitter, receiver, transceiver, radio, router, Ethernet), to the processor-based work cell safety system 130. Figure 1 ) or processor-based work unit security system 200 ( Figure 2 ) Receive signals and / or data, such as signals to stop robot operation, signals to slow down robot operation, signals indicating obstructed areas or regions, and / or access safety monitoring rules 125c ( Figure 1 The signal is from ). Alternative site, safety monitoring rule 125 ( Figure 1It can optionally be stored at one or more robot control systems 300.

[0089] Each robot 302 may include a set of links, joints, end-effectors or end effectors and / or actuators 318a, 318b, 318c (the three actuators shown are collectively referred to as 318), which are operable to move the links about the joints. Each robot 302 may include one or more motion controllers (e.g., motor controllers) 320 (only one shown) that receive control signals, such as control signals of the form of motion planning 306, and provide drive signals to drive the actuators 318.

[0090] Each robot 302 may have a corresponding robot control system 300, or alternatively, a robot control system 300 may perform motion planning for two or more robots 302. For illustrative purposes, a robot control system 300 will be described in detail. Those skilled in the art will recognize that this description can be applied to similar or even identical additional instances of other robot control systems.

[0091] The robot control system 300 may include one or more processors 322 and one or more associated non-transitory computer- or processor-readable storage media, such as system memory 324a, disk drive 324b, and / or memory or registers of processor 322 (not shown). The non-transitory computer- or processor-readable storage media are communicatively coupled to one or more processors 322 via one or more communication channels (e.g., system bus 327). The system bus 327 can employ any known bus structure or architecture, including a memory bus with a memory controller, a peripheral bus, and / or a local bus. One or more of these components may also, or alternatively, communicate with each other via one or more other communication channels, such as via one or more parallel cables, serial cables, or wireless network channels capable of high-speed communication, such as Universal Serial Bus (“USB”) 3.0, Peripheral Component Interconnect (PCIe), or via... They communicate with each other.

[0092] The robot control system 300 can also be communicatively coupled to one or more remote computer systems, such as server computers (e.g., sources of motion planning maps), desktop computers, laptops, ultra-portable computers, tablets, smartphones, wearable computers, and / or sensors. Figure 3(Not shown in the diagram), one or more remote computer systems are directly or indirectly coupled to various components of the robot control system 300, for example, via a network interface (not shown). The remote computing system (e.g., a server computer (e.g., a source of motion plans)) can be used to program, configure, control, or otherwise interact with data (e.g., motion plans, sweep volumes, task specifications 315) or input said data to the robot control system 300 and various components within it. Such connections can be made using Internet protocols over one or more communication channels, such as one or more wide area networks (WANs), such as Ethernet or the Internet. As described above, pre-run calculations (e.g., generation of motion plan family) can be performed by a system separate from the robot control system 300 or the robot 302, while runtime calculations can be performed by one or more processors 322 of the robot control system 300, which in some embodiments may be mounted on the robot 302.

[0093] As previously described, the robot control system 300 may include one or more processors 322 (i.e., circuitry), non-transitory storage media (e.g., system memory 324a, one or more disk drives 324b), and a system bus 327 coupling various system components. The processors 322 may be any logic processing unit, such as one or more central processing units (CPUs), digital signal processors (DSPs), graphics processing units (GPUs), field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), programmable logic controllers (PLCs), etc. The system memory 324a may include read-only memory (“ROM”) 326, random access memory (“RAM”) 328, flash memory 330, and EEPROM (not shown). A basic input / output system (“BIOS”) 332, capable of forming a portion of the ROM 326, contains basic routines that facilitate the transfer of information between components within the robot control system 300, for example, during startup.

[0094] Drive 324b may be, for example, a hard disk drive for reading from and writing to a disk, a solid-state (e.g., flash memory) drive for reading from and writing to a solid-state memory, and / or an optical disk drive for reading from and writing to a removable optical disk. In various embodiments, the robot control system 300 may also include any combination of such drives. Drive 324b may communicate with one or more processors 322 via system bus 327. One or more drives 324b may include an interface or controller (not shown) coupled between such a drive and system bus 327, as is known to those skilled in the art. Drive 324b and its associated computer-readable medium provide the robot control system 300 with non-volatile storage of computer- or processor-readable and / or executable instructions, data structures, program modules, and other data. Those skilled in the art will understand that other types of computer-readable media capable of storing computer-accessible data may be used, such as WORM drives, RAID drives, magnetic tape cassettes, digital video discs (“DVDs”), Bernoulli cassette tapes, RAM, ROM, smart cards, etc.

[0095] Executable instructions and data can be stored in system memory 324a, such as operating system 336, one or more application programs 338, other programs or modules 340, and program data 342. Application program 338 may include processor-executable instructions that cause one or more processors 322 to perform one or more of the following operations: generate a discrete representation of the environment in which robot 302 will operate, including obstacles and / or target objects or workpieces in the environment, where planned movements of other robots may be represented as obstacles; generate motion plans or roadmaps, including invoking or otherwise obtaining the results of conflict assessments, setting cost values ​​for edges in the motion planning graph, and estimating available paths in the motion planning graph; optionally storing multiple determined motion plans or roadmaps; and / or optionally identifying situations that may trigger processor-based cell safety systems 130, 200 and associating costs with corresponding transitions to postpone such transitions, thereby potentially avoiding the introduction of stopping, mitigation, or preventative occlusion. Motion planning construction (e.g., conflict detection or assessment, cost of updating edges in the motion planning graph based on conflict detection or assessment and / or triggering rules and conditions of a processor-based cell-of-work safety system, and path search or estimation) can be performed as described herein and in references incorporated herein by reference. Conflict detection or assessment can be performed using various structures and techniques described elsewhere herein. Application 338 may additionally include one or more machine-readable and machine-executable instructions that cause one or more processors 322 to perform other operations, such as optionally processing sensed data (captured via sensors). Application 338 may additionally include one or more machine-executable instructions that cause one or more processors 322 to perform various other methods described herein and in references incorporated herein by reference.

[0096] Optionally, safety monitoring rule 125c ( Figure 1 This information can be stored in the robot control system 300, for example, in the system memory 324a.

[0097] In various embodiments, one or more of the above operations may be performed by one or more remote processing devices or computers linked via a communication network (e.g., a network) through a network interface.

[0098] Although Figure 3 The system is shown to be stored in system memory 324a, but the operating system 336, application program 338, other programs / modules 340 and program data 342 can be stored on other non-transitory computer or processor readable media, such as one or more drives 324b.

[0099] The motion planner 304 of the robot control system 300 may include dedicated motion planner hardware or may be implemented, in whole or in part, via one or more processors 322 and processor-executable instructions stored in system memory 324a and / or driver 324b.

[0100] The motion planner 304 may include or implement a motion converter 350, a conflict detector 352, a rule analyzer 359, a cost setter 354, and a path analyzer 356.

[0101] Motion converter 350 converts the motion of other robots in the robot into a representation of obstacles. Motion converter 350 receives motion plans or other representations of motion from other motion planners. Motion converter 350 then determines the area or volume corresponding to one or more motions. For example, the motion converter is capable of converting a motion into a corresponding sweep volume, i.e., the volume swept by the corresponding robot or part thereof as it moves or transforms between postures as represented by the motion plan. Advantageously, motion planner 304 may simply queue obstacles (e.g., sweep volumes) and may not need to determine, track, or indicate the timing of corresponding motions or sweep volumes. While described as a motion converter 350 of a given robot 302 converting the motion of other robots into obstacles, in some embodiments, other robots 302b may provide the given robot 302 with an obstacle representation (e.g., a sweep volume) for a specific motion.

[0102] The collision detector 352 performs collision detection or analysis to determine whether a transformation or movement of a given robot 302 or a part thereof will result in a collision with an obstacle. As previously mentioned, the movement of other robots can advantageously be represented as an obstacle. Therefore, the collision detector 352 is able to determine whether the movement of one robot will result in a collision with another robot moving through the work cell or operating environment 104.

[0103] In some implementations, the collision detector 352 performs software-based collision detection and evaluation, such as performing bounding box-to-bounding box collision evaluation or evaluation based on a hierarchy represented by the geometry (e.g., a sphere) of the volume swept by the robot 302 or a portion thereof during movement. In some implementations, the collision detector 352 performs hardware-based collision detection or evaluation, such as employing a set of dedicated hardware logic circuits to represent obstacles and to stream a representation of motion via the dedicated hardware logic circuits. In hardware-based collision detection or evaluation, the collision detector can employ one or more configurable arrays of circuits, such as one or more FPGAs 358, and can optionally generate Boolean collision evaluations.

[0104] Rule analyzer 359 determines or evaluates the likelihood or probability that a motion or transition (represented by edges in the graph) will cause the processor-based cell safety system to trigger a stop, mitigation, or preventative occlusion, or other inhibition of robot operation. For example, rule analyzer 359 may evaluate or simulate one or more more robot motion plans or portions thereof (e.g., edges) to determine whether any transition will violate a safety rule (e.g., causing a violation of safety monitoring rule 125c, as enforced by the processor-based cell safety system). Figure 1 As defined by [the relevant rules], one or more robots or parts thereof are too close to a human. For example, rule analyzer 359 can evaluate or simulate the position and / or path or trajectory of an object (e.g., a human) or a part thereof to determine whether any position or movement of the object would violate a safety rule (e.g., leading to safety monitoring rule 125c as implemented by a processor-based cell safety system). Figure 1 (As defined by the rule that a human or a portion thereof is too close to one or more robots). For example, in a processor-based cell safety system employing a laser scanner that divides a portion of the operating environment into a grid, and where rules enforced by processor-based cell safety cause stopping, mitigation, or preventative occlusion when a human is within a grid location where a portion of the robot is located, rule analyzer 359 can identify transitions within a grid that bring a portion of the robot to the location of the human or the predicted location of the human, enabling the adjustment (e.g., increase) of weights associated with edges corresponding to these identified transitions.

[0105] The cost setter 354 is capable of being based at least in part on conflict detection or assessment and optionally on the processor-based work cell security system 130 provided by the rule analyzer 359. Figure 1 Processor-based work unit security system 200 Figure 2 The cost of edges in the motion planning graph is set or adjusted based on the analysis of the rules and conditions applied. For example, the cost setter 354 can set a relatively high cost value for edges representing transitions between states or movements between poses that cause or may cause conflicts and / or may trigger processor-based cell safety systems 130, 200, potentially preventing stopping, mitigation, or the introduction of preventative occlusion. Alternatively, the cost setter 354 can set a relatively low cost value for edges representing transitions between states or movements between poses that do not cause or may not cause conflicts and / or may not trigger processor-based cell safety systems 130, 200, potentially preventing stopping, mitigation, or the introduction of preventative occlusion. Setting costs can include setting cost values ​​that are logically associated with the corresponding edges via some data structure (e.g., fields, pointers, tables).

[0106] Path analyzer 356 can use a motion planning graph and cost values ​​to determine a path (e.g., optimal or optimized). For example, path analyzer 356 can constitute a minimum-cost path optimizer, which determines the minimum or relatively low-cost path between two states, configurations, or poses, represented by corresponding nodes in the motion planning graph. Path analyzer 356 can use or execute any kind of pathfinding algorithm, such as a minimum-cost pathfinding algorithm, while considering the cost value associated with each edge representing the probability of a conflict and / or the probability of triggering a safety system.

[0107] Various algorithms and structures can be used to determine the lowest-cost path, including those that implement the Bellman-Ford algorithm, but other algorithms and structures can also be used, including but not limited to any process that determines the lowest-cost path as the path between two nodes in a motion planning graph such that the sum of the costs or weights of their constituent edges is minimized. This process improves the motion planning technique of robots 102 and 302 by using motion planning graphs that represent the motion of other robots as obstacles and by collision detection, thereby improving the efficiency and response time in finding the “best” path to perform the task without collisions.

[0108] The motion planner 304 may optionally include a modifier 360. The modifier 360 may receive information indicating that another robot has completed a motion, referred to herein as a motion completion message. Alternatively, a flag may be set to indicate completion. In response, the modifier 360 may remove an obstacle or part of an obstacle representing a now completed motion. This may allow the generation of new motion plans for a given robot, which may be more efficient or allow the given robot to participate in performing a task previously blocked by the motion of another robot. This method advantageously allows the motion converter 350 to ignore the timing of motion when generating obstacle representations for motion, while still achieving better throughput than using other techniques. The motion planner 304 may additionally cause the collision detector 352 to perform new collision detection or evaluation in the event that an obstacle has been modified, to produce an updated motion plan graph in which the weights of edges or the costs associated with edges have been modified, and cause the cost setter 354 and the path analyzer 356 to update cost values ​​and determine new or modified motion plans accordingly.

[0109] The motion planner 304 may optionally include an environment converter 363 that converts the output (e.g., a digital representation of the environment) from an optional sensor 362 (e.g., a digital camera) into a representation of obstacles. Therefore, the motion planner 304 is capable of performing motion planning that takes into account transient objects in the environment (e.g., people, animals, etc.).

[0110] One or more processors 322 and / or motion planners 304 may be or may include any logic processing unit, such as one or more central processing units (CPUs), digital signal processors (DSPs), graphics processing units (GPUs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), programmable logic controllers (PLCs), etc. Non-limiting examples of commercial computer systems include, but are not limited to, the Celeron, Core, Core 2, Itanium, and Xeon series microprocessors provided by Intel Corporation; the K8, K10, Bulldozer, and Bobcat series microprocessors provided by Advanced Micro Devices, Inc.; the A5, A6, and A7 series microprocessors provided by Apple Inc.; the Snapdragon series microprocessors provided by Qualcomm Incorporated; and the SPARC series microprocessors provided by Oracle Corporation. Figure 3 The construction and operation of the various structures shown can be implemented or adopted using international patent application No. PCT / US2017 / 036880, filed on June 9, 2017, entitled "MOTION PLANNING FOR AUTONOMOUS VEHICLE AND RECONFIGURABLE MOTION PLANNING PROCESSORS", international patent application No. WO 2016 / 122840, filed on January 5, 2016, entitled "SPECIALIZED ROBOT MOTION PLANNING HARDWARE AND METHODS OF MAKING AND USING SAME", and / or the patent application No. APPARATUS, METHOD AND ARTICLE TOFACILITATE MOTION PLANNING OF AN AUTONOMOUS VEHICLE IN AN ENVIRONMENT HAVING DYNAMIC, filed on January 12, 2018. The structures, techniques and algorithms described in U.S. Patent Application No. 62 / 616,783, “OBJECTS”, or similar structures, techniques and algorithms.

[0111] Although not required, many implementations will be described in the general context of computer-executable instructions, such as program application modules, objects, or macros stored on a computer or processor-readable medium and executed by one or more computers or processors capable of performing obstacle representation, collision assessment, and other motion planning operations.

[0112] Motion planning operations may include, but are not limited to, generating one, more, or all of the following: based on the robot's geometric model 112 ( Figure 1 Representation of robot geometry in Task 114 Figure 1 ) and the volume occupied by the robot in various states or poses and / or during movement between states or poses (e.g., sweep volume), or converting one, more, or all of the above into digital form, such as point clouds, Euclidean distance fields, data structure formats (e.g., hierarchical formats, non-hierarchical formats), and / or curves (e.g., polynomial or spline representations). Motion planning operations may optionally include, but are not limited to, generating one, more, or all of the following: static object data 118 representing static or transient obstacles ( Figure 1 ) and / or perceived data 120 ( Figure 1 A representation of a static or persistent obstacle, or a conversion of one, more, or all of the above into a digital form, such as a point cloud, Euclidean distance field, data structure format (e.g., hierarchical format, non-hierarchical format), and / or curve (e.g., polynomial or spline representation).

[0113] Motion planning operations may include, but are not limited to, using various conflict assessment techniques or algorithms (e.g., software-based, hardware-based) to determine, detect, or predict conflicts for various states or postures of a robot or for the robot’s motion between states or postures.

[0114] In some implementations, motion planning operations may include, but are not limited to, determining one or more motion plans, motion plans, or route maps; storing the determined one or more plans, motion plans, or route maps; and / or providing one or more plans, motion plans, or route maps to control the operation of the robot.

[0115] In one implementation, collision detection or evaluation is performed in response to a function call or similar procedure and returns a Boolean value. The collision detector 352 can be implemented via one or more field-programmable gate arrays (FPGAs) and / or one or more application-specific integrated circuits (ASICs) to perform collision detection while achieving low latency, relatively low power consumption, and progressively increasing amounts of processable information.

[0116] In various implementations, this operation may be performed entirely in hardware circuitry or as software stored in memory such as system memory 324a and executed by one or more hardware processors 322, such as one or more microprocessors, digital signal processors (DSPs), field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), graphics processing unit (GPU) processors, programmable logic controllers (PLCs), electrically programmable read-only memory (EEPROMs), or as a combination of hardware circuitry and software stored in memory.

[0117] International patent application No. PCT / US2017 / 036880, filed on June 9, 2017, entitled "MOTION PLANNING FOR AUTONOMOUS VEHICLES AND RECONFIGURABLE MOTION PLANNING PROCESSORS"; International patent application No. WO 2016 / 122840, filed on January 5, 2016, entitled "SPECIALIZED ROBOT MOTION PLANNING HARDWARE AND METHODS OF MAKING AND USING SAME"; and International patent application No. WO 2016 / 122840, filed on January 12, 2018, entitled "APPARATUS, METHOD AND ARTICLE TO FACILITATE MOTION PLANNING OF AN AUTONOMOUS VEHICLE IN AN ENVIRONMENT HAVING DYNAMIC". U.S. Patent Application No. 62 / 616,783, entitled “OBJECTS,” and U.S. Patent Application No. 62 / 856,548, filed June 3, 2019, entitled “APPARATUS, METHODS ANDARTICLES TO FACILITATE MOTION PLANNING IN ENVIRONMENTS HAVING DYNAMICOBSTACLES,” also describe aspects of perception, mapping, conflict detection, and pathfinding that can be adopted in whole or in part. Those skilled in the art will understand that the illustrated and other embodiments can be practiced with other system architectures and arrangements and / or other computing system architectures and arrangements, including those of robots, handheld devices, multiprocessor systems, microprocessor-based or programmable consumer electronics, personal computers (“PCs”), networked PCs, minicomputers, mainframes, etc. These embodiments or examples, or portions thereof (e.g., at configuration time and runtime), can be practiced in a distributed computing environment where tasks or modules are executed by remote processing devices linked via a communication network. In a distributed computing environment, program modules may reside on local and remote storage devices or media. However, where and how certain types of information are stored is crucial for improving motion planning.

[0118] For example, various motion planning solutions "bake in" a roadmap (i.e., a motion planning graph) into a processor (e.g., an FPGA), and each edge in the roadmap corresponds to a non-reconfigurable Boolean circuit in the processor. The design of "bake in" the planning graph into the processor raises the problem of storing multiple or a large number of planning graphs within a limited processor circuit, and the design is often non-reconfigurable for use with different robots.

[0119] One solution provides a reconfigurable design that stores planning information in a storage device. This approach stores the information in memory rather than embedding it into the circuitry. Another approach uses templated reconfigurable circuitry instead of memory.

[0120] As described above, some information (e.g., robot geometry) can be captured, received, input, or provided during the configuration time prior to runtime. The received information can be processed during configuration time to generate processed information (e.g., motion planning graphs) to accelerate operation or reduce computational complexity during runtime.

[0121] During runtime, collision detection can be performed across the entire environment, including determining whether any part of the robot will or is expected to collide with another part of the robot itself, with other robots or parts thereof, with persistent or static obstacles in the environment, or with transient obstacles in the environment with unknown trajectories (e.g., people or humans).

[0122] Figure 4 This illustrates a scenario where the objective of robots 102 and 302 is to perform a task while avoiding collisions with both static and dynamic obstacles. Robot 102 ( Figure 1 ), 302 Figure 3 Example planning diagram 400, the obstacle can include other robots operating in the work cell or operating environment 104.

[0123] Planning diagram 400 includes multiple nodes 408a to 408i (represented as hollow circles in the diagram) connected by edges 410a to 410h (represented as straight lines between node pairs in the diagram). Each node implicitly or explicitly represents the time and variables characterizing the state of robots 102 and 302 in the configuration space of robots 102 and 302. The configuration space is often referred to as C-space and is the space of the state, configuration, or pose of robots 102 and 302 represented in planning diagram 400. For example, each node may represent the state, configuration, or pose of robots 102 and 302, which may include, but is not limited to, position, orientation, or a combination of position and orientation. The state, configuration, or pose may be represented, for example, by a set of joint positions and joint angles / rotations (e.g., joint pose, joint coordinates) of the joints of robots 102 and 302.

[0124] The edges in planning graph 400 represent valid or permitted transitions between these states, configurations, or poses of robots 102 and 302. The edges in planning graph 400 do not represent actual movement in Cartesian coordinates, but rather transitions between states, configurations, or poses in C-space. Each edge in planning graph 400 represents a transition of robots 102 and 302 between corresponding node pairs. For example, edge 410a represents a transition of robots 102 and 302 between two nodes. Specifically, edge 410a represents a transition between the states of robots 102 and 302 in a specific configuration associated with node 408b and the states of robots 102 and 302 in a specific configuration associated with node 408c. Although the nodes are shown at different distances from each other, this is for illustrative purposes only and is not related to any physical distance. There is no limit to the number of nodes or edges in the planning graph 400. However, the more nodes and edges used in the planning graph 400, the more accurately and precisely the motion planner can determine the optimal path to perform the task based on one or more states, configurations, or poses of the robots 102 and 302, because there are more paths to choose from among them, the one with the lowest cost.

[0125] Each edge is assigned a cost value or associated with a cost value; for example, this assignment can be updated at runtime. The cost value can represent a conflict assessment for the motion represented by the corresponding edge. The cost value can also represent an assessment of the motion probability, which is indicated by the corresponding edge that would trigger a processor-based work unit safety system, thereby causing a stop, mitigation, or preventative occlusion. As described herein, safety monitoring rule 125c ( Figure 1 This can be used to determine what conditions or circumstances will trigger a processor-based cell-of-work safety system. For those edges corresponding to transitions that are considered likely to trigger a processor-based cell-of-work safety system, the cost value (e.g., weight) assigned to the edges can be increased to reduce the tendency to select paths that include those transitions.

[0126] Typically, it is desirable for robots 102 and 302 to avoid certain obstacles, such as shared work cells or other robots in the operating environment. In some cases, it may be desirable for robots 102 and 302 to contact or approach certain objects in the shared work cell or operating environment, such as to grasp or move objects or workpieces. Figure 4 A planning diagram 400 is shown, which is used by a motion planner to identify the path of robots 102 and 302 when the goal of robots 102 and 302 is to avoid conflict with one or more obstacles while moving through multiple poses while performing a task (e.g., picking up and placing objects).

[0127] Obstacles can be represented digitally as, for example, bounding boxes, oriented bounding boxes, curves (e.g., splines), Euclidean distance fields, or hierarchical structures of geometric entities. Whichever numerical representation best suits the obstacle type and the type of collision detection to be performed may depend on the specific hardware circuitry employed. In some implementations, the sweep volume in the route map is pre-calculated. Examples of conflict assessments are described in international patent application PCT / US2017 / 036880, filed June 9, 2017, entitled “MOTION PLANNING FOR AUTONOMOUSVEHICLES AND RECONFIGURABLE MOTION PLANNING PROCESSORS”; U.S. patent application 62 / 722,067, filed August 23, 2018, entitled “COLLISION DETECTION USEFUL IN MOTIONPLANNING FOR ROBOTICS”; and international patent application publication WO 2016 / 122840, filed January 5, 2016, entitled “SPECIALIZED ROBOT MOTION PLANNING HARDWARE AND METHODS OF MAKING AND USINGSAME”.

[0128] Motion planner or a part thereof (e.g., Figure 3 The collision detector 352 in the model determines or evaluates the likelihood or probability that a movement or transformation (represented by an edge) will result in a collision with an obstacle. In some cases, this determination produces a Boolean value, while in others, it can be expressed as a probability.

[0129] For nodes in planning diagram 400, where direct transitions between nodes may lead to conflicts with obstacles, the motion planner (e.g., Figure 3The cost setter 354 in the planning graph 400 assigns cost values ​​or weights to the edges that transition between those nodes (e.g., edges 410a, 410b, 410c, 410d, 410e, 410f, 410g, 410h), with the cost values ​​or weights indicating the probability of a collision with an obstacle. Figure 4 In the example shown, the region with a relatively high probability in C-space is represented as graphical part 414, but does not correspond to the physical region.

[0130] For example, the motion planner can assign a cost value or weight equal to or close to zero to each of the multiple edges of the planning graph 400 whose corresponding probability of colliding with an obstacle is below a defined threshold conflict probability. In this example, the motion planner has assigned a cost value or weight of zero to those edges of the planning graph 400 that represent transitions or movements of robots 102, 302 and have no or almost no probability of colliding with obstacles. For each of the multiple edges of the planning graph 400 whose corresponding probability of colliding with obstacles in the environment is above a defined threshold conflict probability, the motion planner assigns a cost value or weight with a value significantly greater than zero. In this example, the motion planner has assigned a cost value or weight greater than zero to those edges of the planning graph 400 with a relatively high probability of colliding with obstacles. The specific threshold used for the conflict probability may vary. For example, the threshold may be 40%, 50%, 60%, or a lower or higher conflict probability. Furthermore, assigning a cost value or weight with a value greater than zero may include assigning a cost value or weight with a magnitude greater than zero corresponding to the corresponding conflict probability. In other implementations, cost values ​​or weights may present a binary choice between conflicting and non-conflicting options, with only two cost values ​​or weights available when assigning cost values ​​or weights to edges.

[0131] Motion planner or parts thereof (e.g., Figure 3 The rule analyzer 359 in the system determines or evaluates the likelihood or probability that a motion or transition (represented by edges) will cause a processor-based cell safety system to trigger a stop, mitigation, or preventative occlusion. For example, a motion planner or a portion thereof (e.g., Figure 3 The rule analyzer 359 in the system can simulate motion planning to determine whether any change will violate safety rules (e.g., causing the robot or parts thereof to get too close to a human, as per safety monitoring rule 125c implemented by a processor-based cell safety system). Figure 1 (as defined by the definition). In addition to the probability of conflict, such as the probability that causes a processor-based work unit safety system to trigger a stop, mitigation, or preventative occlusion, the motion planner can assign, set, or adjust the cost value or weight of each edge based on factors or parameters.

[0132] For example, as shown in planning diagram 400, the motion planner has assigned a cost value or weight of 5 to edges 410b, 410e, and 410f that have a higher probability of conflict and / or a higher probability of triggering stop, mitigation, or preventive occlusion, but has assigned a cost value or weight of 0 to edge 410a, and a cost value or weight of 1 to edges 410c and 410g. The motion planner determines that edges 410a, 410c, and 410g have a much lower probability of conflict and / or a much lower probability of triggering stop, mitigation, or preventive occlusion.

[0133] After the motion planner sets a cost value or weight representing the probability of robots 102, 302 colliding with obstacles, based at least in part on conflict assessment, optionally on the probability of causing a processor-based work cell safety system to trigger a stop, mitigation, or preventative occlusion, and / or optionally on other factors (e.g., latency, power consumption), the motion planner (e.g., Figure 3 The path analyzer 356 performs optimization to identify path 412 (indicated by thick lines) in the generated planning map 400. Path 412 provides motion planning for robots 102, 302 as specified by the path, which is unlikely to conflict with obstacles of other robots operating in the work cell or operating environment, or has a relatively low probability of conflicting with such obstacles, and / or is unlikely to cause the processor-based work cell safety system to trigger stop, mitigation or preventive occlusion, or has a relatively low probability of causing the processor-based work cell safety system to trigger stop, mitigation or preventive occlusion.

[0134] In one implementation, once all edge costs of the planning graph 400 have been assigned or set, the motion planner (e.g., Figure 3 The path analyzer 356 in the model can perform calculations to determine the lowest cost path to or toward the target state represented by the target dielectric. For example, the path analyzer 356 ( Figure 3 The system can execute a minimum-cost path algorithm from the current state of robots 102 and 302 in planning graph 400 to possible states, configurations, or poses. The motion planner then selects the path with the lowest cost (closest to zero) in planning graph 400. As mentioned above, cost can reflect not only the probability of conflict and / or the probability of causing the processor-based cell safety system to trigger a stop, mitigation, or preventative occlusion, but also other factors or parameters. In this example, the current state, configuration, or pose of robots 102 and 302 in planning graph 400 is at node 408a, and the path is depicted in planning graph 400 as path 412 (the thick path includes segments extending from node 408a to node 408i).

[0135] Although shown as a path with many sharp turns in planning diagram 400, these turns do not represent physical turns in the routine, but rather logical transitions between the states, configurations, or postures of robots 102 and 302. For example, each edge in the identified path 412 may represent a change in the state of the physical configuration of robots 102 and 302 relative to the environment, but not necessarily a change in the state of the physical configuration of robots 102 and 302. Figure 4 The angle of path 412 shown corresponds to the directional changes of robots 102 and 302.

[0136] Figure 5 A high-level method 500 for operating a processor-based system according to at least one illustrated embodiment is shown for implementing safety monitoring of the operating environment to control robot operation in the operating environment and for verifying the safety monitoring system. Method 500 can, for example, be implemented by a processor-based safety system 130 (… Figure 1 One or more processors 134 ( Figure 1 ) is executed, for example by a processor-based work unit security system 200 ( Figure 2 One or more processors 222 () Figure 2 The processor-based work unit safety system 200 can, for example, optionally integrate with the robot control system 300. Figure 3 ) Communication coupling, robot control system 300 ( Figure 3 Generate motion planning and / or control one or more robots 102a, 102b Figure 1 Operations within an operating environment. As used herein and in the claims, the operation or movement of a robot includes the operation or movement of the entire robot or parts thereof (e.g., robot appendages, end-effectors, end effectors). While generally discussed in relation to a robot, various operations and actions apply to operating environments in which one, two, or even more robots operate.

[0137] Method 500 begins at 502. For example, method 500 may begin in response to power-on of the processor-based cell safety system 200, the robot control system 300, and / or the robot 102, or a call or activation from a calling routine. Method 500 may execute continuously or even continuously, for example, during the operation of one or more robots 102.

[0138] At 504, one or more processors 222 of the processor-based work unit security system 200 ( Figure 2 From the first sensor 132a ( Figure 1 Upon receiving information, the first sensor 132a is positioned and oriented to detect the location of a human (if present) in at least a first part of the operating environment.

[0139] At 506, one or more processors 222 of the processor-based work unit security system 200 ( Figure 2 From at least the second sensor 132b ( Figure 1 Upon receiving information, the second sensor 132b is positioned and oriented to detect the location of a human (if any) in at least a second portion of the operating environment. The second portion of the operating environment at least partially overlaps with the first portion of the operating environment. The second sensor 132b is advantageously heterogeneous relative to the first sensor 132a. Notably, at 506, the processor-based work unit safety system 200 can receive information from a third, fourth, or even more sensors 132, each associated with a corresponding position and orientation or field of view, which are positioned and oriented to monitor at least a portion of the operating environment to detect the location of a human (if any) in at least that portion. While in some embodiments two or more sensors 132 may share certain operational characteristics (e.g., sensor operating modes, brand and model, sampling rate), diversity of operational characteristics among various sensors is not intuitively desirable for increasing overall operational safety.

[0140] The first sensor, the second sensor, and any additional sensors 132 can be sensors dedicated to safety monitoring and can form part of a dedicated processor-based work unit safety system 200. Alternatively, sensor 122 for motion planning ( Figure 1 It can also provide sensor data to the processor-based work unit safety system 200 to perform safety monitoring. Robot control systems 109a and 109b ( Figure 1 The processor-based cell security system 200 may differ from and optionally be communicatively coupled to it. The first sensor, the second sensor, and any additional sensors 132 can advantageously be readily available, low-cost sensors. As described above, the group comprising the first sensor, the second sensor, and any additional sensors 132 can be a heterogeneous sensor group, where two, more, or even all sensors of the processor-based cell security system 200 have operating characteristics distinct from each other. This can advantageously achieve the desired safety margin of common, readily available sensors, for example, the safety margin typically associated with a substantially wider range of safety-certified sensors.

[0141] At 508, at least one processor 222 of the processor-based work unit security system 200 ( Figure 2 The system performs an evaluation of one or more operational states of the first sensor and at least the second sensor, as well as other sensors 132 (if present) of the processor-based work unit safety system 200. The evaluation may be based at least in part on one or more sets of sensor state rules 125a. Figure 1The system may apply one or more sets of sensor state rules 125a to evaluate one, two or more operating states or conditions of each of the sensors 132, or to evaluate the operating states or conditions among the sensors 132 of the processor-based work unit safety system 200, examples of which are described herein. The evaluation of operating states or conditions may indicate whether one or more sensors 132 are operating as expected and / or within a set of defined performance parameters, states or conditions, and thus can be relied upon to provide a safe operating environment.

[0142] The evaluation can be based on one or more sets of sensor state rules 125a, which specify one or more of various factors, operating states, conditions, parameters, criteria, and / or rules. For example, at least one processor 222 of the processor-based cell safety system 200 ( Figure 2 This can advantageously assess whether sensor 132 is operating correctly. For example, at least one processor 222 of the processor-based work unit safety system 200 ( Figure 2 It is advantageous to assess whether the information received from the first sensor and at least the second sensor 132 indicates that the sensors 132 of the processor-based cell safety system 200 are providing sensing information in the expected manner (e.g., at a defined or nominal sampling rate; two or more sensors 132 consistently sensing the same event), and / or that no sensor 132 is stuck (i.e., incorrectly providing the same stale information over and over again, when conditions in the operating environment have changed and different information should be provided within the relevant time period). Some of the assessments in the evaluation (e.g., sampling rate) can be performed individually for each sensor 132, while other assessments (e.g., comparing the information sensed by two or more sensors 132) can be performed jointly for two or more sensors 132 of the processor-based cell safety system 200.

[0143] For example, each sensor 132 can be associated with a corresponding sampling rate. Rules can define a corresponding acceptable sampling range or a percentage of sampling rate error considered acceptable, or conversely, a similar value considered unacceptable. As another example, rules can define a corresponding amount of time a sensor might be stuck, or a frequency used to confirm that a sensor is not stuck, which is considered acceptable, or conversely, a similar value considered unacceptable.

[0144] At 510, at least one processor 222 of the processor-based work unit security system 200 performs system state verification, at least in part based on one or more sets of system verification rules 125b. Figure 1The processor-based cell safety system 200 is used to verify the state (i.e., system state) of the processor-based cell safety system 200. Verification can be based, for example, on the determined operational state of the sensors 132. System verification rule 125b can specify, for example, rules applied to the selection of sensors 132 and / or the selection of one or more groups of sensors 132 (e.g., all sensors must be operational; all sensors identified as necessary must be operational, while other sensors may be operational or may not be operational; a majority of sensors in a group of sensors must be consistent). One or more processors 222 of the processor-based cell safety system 200 can evaluate or otherwise apply system verification rule 125b to determine whether there are sufficient sensors 132 operating within normal or acceptable limits (i.e., fault-free conditions, operational states) to rely on the processor-based cell safety system 200 to ensure safe certified operation. When there are sufficient sensors 132 operating within normal or acceptable limits to rely on the processor-based cell safety system 200 to ensure safe certified operation, one or more processors 222 of the processor-based cell safety system 200 can identify or indicate the presence of a non-abnormal system state. Conversely, if there are not enough sensors 132 operating within normal or acceptable limits (i.e., failure conditions, inoperability) to rely on the processor-based cell safety system 200 to ensure secure authentication operations, one or more processors 222 of the processor-based cell safety system 200 can identify or indicate the presence of an abnormal system state.

[0145] For example, system verification rule 125b may specify how many and / or which sensors 132 can be considered inoperable or unreliable due to the presence of anomalous system conditions. System verification rule 125b may specify that the inoperability or default sensor state of any individual sensor 132 constitutes or indicates an anomalous system state. Additionally or alternatively, system verification rule 125b may specify that a combination of specific sensors 132 in a group of two or more specific sensors 132, or the inoperability or default sensor state of one sensor, constitutes or indicates an anomalous system state. For example, an anomalous system state may exist if one, two, more, or even all of the group of sensors 132 are faulty, inoperable, or potentially faulty or potentially inoperable. Alternatively, when there is no consistency among the majority of sensors 132, system verification rule 125b may define an anomalous system state in the processor-based cell-of-work safety system 200. In the case of consistency among the majority of sensors 132, at least one processor 222 may determine that the sensors 132 as a group or set are sufficiently reliable to provide safe operation within the operating environment or some portion thereof.

[0146] At 512, at least one processor 222 of the processor-based cell security system 200 determines whether the evaluation result based on system verification rule 125b indicates that the processor-based cell security system 200 has an abnormal system state.

[0147] In response to verification indicating that an abnormal system state does exist in the processor-based work cell safety system 200 (e.g., not all sensors 132 are operating within the defined operating parameters, the number of sensors 132 operating within the defined operating parameters is insufficient, or most sensors 132 are operating inconsistently with each other within the defined operating parameters), at 514, at least one processor 222 provides a signal to at least partially control one or more robots 102. Figure 1 The processor 222 can operate to stop movement, slow movement, add preventative occlusion, or otherwise inhibit the movement of one or more robots 102. For example, at least one processor 222 can provide a signal to stop or slow the movement of one or more robots 102 at least until the abnormal system state is mitigated, such as by sending a signal to one or more robots 102 ( Figure 1 The robot control systems 109a and 109b Figure 1 ) or motion controller 320 ( Figure 3 ) provides signals. For example, at least one processor 222 may provide the following signal: this signal indicates the operating environment 104 ( Figure 1 Areas considered to be preventively occluded for motion planning, such as areas covered by one or more sensors 132 in a faulty or inoperable operating state, for example, providing signals to the robot control systems 109a, 109b or motion controllers 320 of one or more robots 102. Method 500 then terminates at 524.

[0148] In response to verification indicating that the processor-based cell-of-work safety system 200 does not have an abnormal system state (e.g., all sensors 132 operate within defined operating parameters, a sufficient number of sensors 132 operate within defined operating parameters, and most sensors 132 operate consistent with each other within defined operating parameters), at 516, the processor-based cell-of-work safety system 200 ( Figure 2 At least one processor 222 () Figure 2 Monitoring operating environment 104 ( Figure 1 Has a violation of safety monitoring rule 125c occurred? Figure 1 To monitor for violations of security rules in the operating environment, one or more processors 222 may employ a representation of the operating environment 104. Figure 1Sensor data of objects in the environment. One or more processors 222 can identify objects that are human or appear to be human. One or more processors 222 can determine the current location of one or more humans and / or the three-dimensional region occupied by one or more humans in the operating environment. One or more processors 222 can optionally predict the path or trajectory of humans over a period of time and / or the three-dimensional region occupied by one or more humans during that period of time. For example, one or more processors 222 can determine the path or trajectory or three-dimensional region based on the current location of one or more humans and based on the previous movement of one or more humans and / or based on the predicted behavior or training of one or more humans. One or more processors 222 can employ artificial intelligence or machine learning to predict the path or trajectory of humans. One or more processors 222 can determine the current location of one or more robots and / or the three-dimensional region occupied by one or more robots over a period of time. For example, one or more processors 222 can determine the path or trajectory or three-dimensional region based on the current location of one or more robots and the robot's motion planning.

[0149] For example, one or more processors 222 can determine whether the position of one or more humans relative to the position and / or path or trajectory of one or more robots will violate one or more safety monitoring rules 125c. Figure 1 For example, a violation of one or more safety monitoring rules 125c can be determined if the movement of one or more humans and / or one or more robots results in the distance between one or more humans and one or more robots falling within a defined threshold safe distance. This can be determined based on straight-line distance calculations, but may also be based on certain sensors 132. Figure 1 The operational characteristics of the operating environment (e.g., the resolution or granularity of the sensors) can be considered, for example, in situations where safety monitoring rule 125c requires maintaining a separation of at least a predefined number of separate areas between one or more humans and one or more robots to avoid triggering stop, mitigation, or preventative occlusion, thus defining the operating environment 104 (e.g., the operating environment 104). Figure 1 (or a portion thereof) is considered to be divided into separate regions (e.g., equal or unequal in size).

[0150] At location 518, processor-based work unit security system 200 ( Figure 2 At least one processor 222 () Figure 2 Determine whether safety rule 125c has been violated. Figure 1One or more of the following. Safety rules may be violated, for example, if a human is too close to robot 102, or if the human's path or trajectory is too close to robot 102 or too close to the path or trajectory of robot 102, or if they are a certain number of units apart. Here, closeness or proximity is defined as straight-line distance.

[0151] In response to the determination that no violation of safety monitoring rule 125c ( Figure 1 At point 520, at least one processor 222 provides signals to at least partially control the operation of one or more robots 102, allowing the operation or movement of one or more robots 102. For example, at least one processor 222 may provide signals allowing one or more robots 102 to move, for example, to robot control systems 109a, 109b of one or more robots 102. Figure 1 ) or motion controller 320 ( Figure 3 The processor 222 can provide a signal indicating that an area of ​​the operating environment is not represented as occluded for motion planning. In some implementations, the default condition may be an indication of the entire work unit or operating environment 104. Figure 1 The entire work unit or operating environment 104 is occluded, and at least one processor 222 can therefore provide a signal that allows relaxation of the assumption that the entire work unit or operating environment 104 is occluded, in response to determining that the system state of the processor-based work unit safety system 200 is a non-abnormal system state (e.g., sensor 132 with a non-faulty or operational sensor state provides sufficient sensor coverage). Control can then return to 504, where portions of method 500 are repeated.

[0152] In response to the detection of a violation of security monitoring rule 125c ( Figure 1 At 522, at least one processor 222 provides a signal to stop, slow down, or otherwise inhibit the operation (e.g., movement) of one or more robots. For example, one or more processors 222 may send a signal to the robot control system 300. Figure 3 ) provides a signal to stop or slow down the movement, or to the motion planners 110a, 110b ( Figure 1 The method provides a signal to identify one or more areas or regions that are occluded for motion planning purposes. Method 500 then terminates at 524, for example, until it is called again. In some implementations, such as when the robot or a part thereof is powered, method 500 may operate continuously or even periodically.

[0153] Figure 6A low-level method 600 for operating a processor-based system according to at least one illustrated embodiment is shown for implementing safety monitoring of the operating environment, thereby controlling robot operation in the operating environment and verifying the safety monitoring system. Method 600 can, for example, be implemented by a processor-based cell safety system 200 (…). Figure 2 ) one or more processors 222 ( Figure 2 ) to execute. For example, the processor-based cell safety system 200 may optionally be integrated with the robot control system 300 ( Figure 3 ) Communicationally coupled, the robot control system 300 generates motion planning and / or controls one or more robots 102 ( Figure 1 Operations in operating environment 104 Figure 1 Method 600 can, for example, be used as part of evaluating one or more operating states of sensor 508. Figure 5 ) to execute.

[0154] At 602, at least one processor 222 determines whether the information received from the first sensor and at least the second sensor indicates that one or both of the first sensor or the second sensor are stuck (i.e., the same stale data or information is being mistakenly sent repeatedly in a situation where the activity in the area or region covered by the sensor has changed over time).

[0155] For example, at least one processor 222 can determine a reference 111 (represented in the information received from the first sensor and at least the second sensor). Figure 1 Whether movement has occurred over a period of time. For example, at least one processor can determine whether the movement of reference 111 represented in the information received from the first sensor and at least the second sensor is consistent with the expected movement of reference 111 over a period of time.

[0156] In at least some embodiments, reference 111a is part of or carried by a portion of robot 102. In such embodiments, at least one processor 222 may, for example, determine whether the movement of reference 111a, as represented in information received from the first and second sensors 132, corresponds to an expected movement of reference 111a over a period of time. This may include, for example, determining whether the movement of reference 111a matches the movement of a portion of robot 102a during that time period. This may be performed, for example, using known joint angles of robot 102a during transitions or movements.

[0157] In at least some embodiments, reference 111b is separate from and distinct from robot 102, and moves independently of robot 102. In such embodiments, at least one processor 222 may, for example, determine whether the movement of reference 111b, as represented in information received from the first and second sensors 132, is consistent with the expected movement of reference 111b over a period of time. For example, this may include determining whether the movement of reference 111b matches the expected movement of reference 111b over that period of time.

[0158] In at least some embodiments, at least one of the first sensor or the second sensor 132 moves in a defined pattern over a period of time. In such embodiments, at least one processor 222 may, for example, determine whether the apparent motion of a reference 111 represented in the information received from the first sensor and the second sensor 132 during that period of time is consistent with the expected apparent motion of the reference 111 during that period of time, based on the movement of the first sensor or the second sensor 132.

[0159] At 604, at least one processor 222 determines whether the information received from sensor 132 is consistent with the sensor's corresponding sampling rate. For example, the first sensor 132 may be in the form of a digital camera that captures images at a rate of 30 frames per second. Therefore, the information received from sensor 132 is expected to be thirty frames per second. A laser scanner may acquire information at 120 samples per second, so the information received from the sensor is expected to be 120 sets of data per second.

[0160] At 606, at least one processor 222 compares data from the operating environment 104 ( Figure 1The processor 222 receives information from the first sensor and at least the second sensor 132, whose second part at least partially overlaps with the first part, to determine whether there is a discrepancy. For example, there may be a stationary or moving object (e.g., a part of robot 102) occupying space in the field of view of two or more sensors 132. At least one processor 222 analyzes the sensed information from each of those sensors 132 to determine whether the pose (i.e., position and / or orientation) of the object detected by each of the sensors 132 is consistent with the pose of the object captured by the other sensors 132. In evaluating the pose, at least one processor may take into account the different corresponding field of view of the sensors 132, for example, normalizing one or more field of view relative to another or relative to a defined reference frame. For example, the image captured by the first image sensor 132 may be manipulated (e.g., translated and / or rotated in three dimensions) based on the field of view of the first image sensor relative to the second image sensor, for example, via a graphics processing unit (GPU). At least one processor 222 may then compare the image captured by the second sensor 132 with the manipulated image from the first sensor 132 to determine whether the two sensors have captured the object consistent with each other. Although described in light of an image-based sensor 132 for the sake of illustration, and although the term "field of view" is used, sensor 132 is not limited to image-based sensor 132. Comparison of information provided by two or more sensors 132 is also not limited to sensors 132 having the same operating mode (e.g., information collected by a PIR motion sensor and a laser sensor can be compared).

[0161] Figure 7 A low-level method 700 for operating a processor-based system according to at least one illustrated embodiment is shown for implementing safety monitoring of the operating environment to control the operation of a robot in the operating environment and for verifying the safety monitoring system. Method 700 may, for example, be provided by a processor-based cell safety system 200 (…). Figure 2 ) one or more processors 222 ( Figure 2 ) Execution. The processor-based work unit safety system 200 may optionally be integrated with, for example, the robot control system 300 ( Figure 3 Through communicative coupling, the robot control system 300 generates motion plans and / or controls the operation of one or more robots in the operating environment. For example, method 700 can serve as a result of determining system validation. Figure 5 Does method 500(510) indicate an abnormal system state in the security system? Figure 5 It is performed as part of method 500(512).

[0162] At 702, at least one processor 222 of the processor-based work unit security system 200 determines any sensor 132 that is identified as necessary. Figure 1 (If any) Whether it is determined to be an operating state with a fault or potential fault. The determination of the presence or absence of a fault or potential fault operating state can be performed as part of method 600. Figure 6 This is part of the process, for example, by verifying whether sensor 132 is stuck, whether sensor 132 is providing sampling at the nominal sampling rate, whether the output of sensor 132 is meaningful or consistent with the expected output or the output of other sensors 132.

[0163] In response to determining that one or more sensors 132 identified as necessary have a faulty or potentially faulty operating state, at least one processor 222 provides a signal at 704 that i) causes the robot operation to stop; ii) causes the robot operation to slow down; and / or iii) indicates an area or region to be identified as occluded. Method 700 can then terminate at 706 until the fault is resolved, and method 700 is invoked again. Alternatively, in response to determining that one or more sensors identified as necessary do not have a faulty or potentially faulty operating state, control is passed to 708.

[0164] At 708, at least one processor 222 of the processor-based work unit safety system 200 determines whether any group or combination (if any) of the sensors 132 identified as necessary is determined to be in a faulty or potentially faulty operating state. The determination of the presence or absence of a faulty or potentially faulty operating state can be performed as part of execution method 600. Figure 6 This is part of the process, for example, by verifying whether sensor 132 is stuck, whether sensor 132 is providing sampling at the nominal sampling rate, and whether the output of sensor 132 is meaningful or consistent with the expected output or the output of other sensors 132.

[0165] In response to determining that one or more of the group or combination of sensors 132 identified as necessary have a faulty or potentially faulty operating state, at least one processor 222 provides a signal at 704 that i) causes the robot operation to stop; ii) causes the robot operation to slow down; and / or iii) indicates an area or region to be identified as occluded. Method 700 can then terminate at 706 until the fault is resolved and method 700 is invoked again. Alternatively, in response to determining that one or more of the group or combination of sensors 132 identified as necessary do not have a faulty or potentially faulty operating state, control is transferred to 710.

[0166] At 710, at least one processor 222 of the processor-based work unit safety system 200 determines whether each zone or area of ​​the operating environment has sufficient sensor coverage by sensors 132 that are determined to be in an operating state without faults or potential faults. The determination of the absence or presence of a faulty or potential faulty operating state can be performed as an execution method 600. Figure 6 This is part of the process, for example, by verifying whether sensor 132 is stuck, whether sensor 132 is providing sampling at the nominal sampling rate, and whether the output of sensor 132 is meaningful or consistent with the expected output or the output of other sensors 132.

[0167] In response to determining the operating environment 104 ( Figure 1 If one or more zones or areas are determined to have insufficient sensor coverage by sensor 132 that is determined to be in a fault-free or potentially fault-free operating state, at least one processor 222 provides a signal at 704 that i) causes robot operation to stop; ii) causes robot operation to slow down; and / or iii) indicates the corresponding zone or area to be identified as occluded. Method 700 can then terminate at 706 until the fault is resolved and method 700 is invoked again. Alternatively, in response to determining that a zone or area has sufficient sensor coverage by sensor 132 that is determined to be in a fault-free and potentially fault-free operating state, at 712, processor 222 allows robot motion planning and / or robot operation to continue uninterruptedly.

[0168] Figure 8 A method 800 for operating a processor-based system according to at least one illustrated embodiment is shown for controlling robot operations in an operating environment, thereby reducing triggering of processor-based cell safety systems. For example, method 800 may be provided by a robot control system 300 (…). Figure 3 One or more processors 322 ( Figure 3 ) execution, robot control system 300 ( Figure 3 Generate motion planning and / or control one or more robots 102 ( Figure 1 In operating environment 104 ( Figure 1 The robot control system 300 can optionally communicate with, for example, a processor-based work cell safety system.

[0169] The processor-based work unit security system 200 is based on a set of security monitoring rules 125c. Figure 1The safety monitoring rule 125c assesses safety conditions, including a number of conditions under which the processor-based cell safety system 200 triggers at least one of slowing down or stopping the operation of at least one robot 102 operating in the cell or operating environment 104. For example, the processor-based cell safety system 200 may trigger stopping or slowing down, or even indicate a portion of the operating environment 104 as preventative occlusion, in response to detecting a transient object (e.g., a human or possibly a human) located within a defined distance of a portion of one or more robots 102 or within a defined distance of the projected trajectory of a portion of one or more robots 102. This distance may or may not be a straight-line distance, and may, for example, be considered in relation to a specific sensor 132 (…). Figure 1 The resolution of the work cell. For example, the processor-based work cell safety system 200 may trigger a stop or slowdown, or even indicate a portion of the operating environment 104 as preventative occlusion, in response to the detection of a collision or proximity between the trajectory of a transient object (e.g., a human or possibly a human) and the predicted trajectory of a portion of one or more robots 102.

[0170] Stopping, mitigating, and preventative occlusion can impede robot operation, and limiting or even avoiding such operation is advantageous if possible. To mitigate this stopping, mitigating, and preventative occlusion, the processor-based robot control system 300 (…) performs motion planning. Figure 3 ) It is advantageous to consider triggering a processor-based work unit security system 200 ( Figure 2 Safety monitoring rule 125c Figure 1 ).

[0171] Method 800 begins at 802. For example, method 800 may begin in response to power-on of a processor-based system (e.g., processor-based robot control system 300; processor-based cell safety system 200), power-on of one or more robots 102, or a call or enable from a calling routine. Method 800 may execute continuously, for example, during the operation of one or more robots 102.

[0172] At 804, at least one processor 322 of the processor-based robot control system 300 ( Figure 3 Access to processor-based work unit security system 200 ( Figure 2 A set of storage security monitoring rules 125c implemented. Figure 1 This group's safety monitoring rule 125c ( Figure 1The data can be stored locally in the processor-based robot control system 300, but is preferably stored at and retrieved from the processor-based cell safety system 200 to ensure the use of the latest set of rules and conditions.

[0173] Optionally, at 806, at least one processor 322 of the processor-based robot control system 300 determines the predicted behavior of a human (e.g., an operator) or a human who appears likely to enter the work unit or operating environment 104. The at least one processor 322 may, for example, use machine learning or artificial intelligence trained on a dataset of similar operating environments and robot scenarios to determine the predicted behavior of the human in the work unit or operating environment 104. The at least one processor 322 may, for example, determine the predicted behavior of the human in the work unit or operating environment 104 based at least in part on a set of operator training guidelines specifying the position or orientation of the operator and other humans when they are present in the operating environment 104, as well as the time and / or speed of movement. At least one processor 222 may, for example, determine the predicted trajectory (e.g., path, speed) of the human at least in part through the work unit or operating environment 104.

[0174] Optionally, at 808, at least one processor 322 of the processor-based robot control system 300 can, for example, determine whether the human's behavior is consistent with the predicted behavior. In response to determining that the human's behavior is inconsistent with the predicted behavior, at least one processor can, for example, provide a signal at 810 that causes the movement of one or more robots 102 to slow down and / or causes another action that reduces the likelihood or probability of one or more robots 102 colliding with an unpredictable human, such as causing one or more robots 102 to move away from the human's current position. Control is then passed to 812. In response to determining that the human's behavior is consistent with the predicted behavior, control is directly passed to 812.

[0175] At 812, at least one processor 322 of the processor-based robot control system 300 is at least partially based on the processor-based safety system 200. Figure 2 Safety monitoring rule 125c Figure 1 And optionally, at least one robot 102 is determined in part based on predicted human behavior (if any) in the operating environment 104. Figure 1 The motion planning of at least one robot 102 can, for example, determine the motion planning of at least one robot 102, which at least reduces the risk of damage to the processor-based work cell safety system 200. Figure 2The probability of triggering at least one robot 102 to slow down or stop operation, or to reduce or even eliminate the use of preventative occlusion.

[0176] At least one processor 322 may determine motion planning, for example, based on the resolution or granularity of at least one component (e.g., sensor 132) of the processor-based cell safety system 200. At least one processor 322 may determine motion planning, for example, based on the resolution or granularity of at least one sensor 132 of the processor-based cell safety system 200. For example, at least one processor 322 may determine motion planning based on a set of dimensions of a region grid (e.g., wedge-shaped or triangular regions, rectangular regions, hexagonal regions), where, for example, sensor 132 (e.g., a laser-based sensor) divides the operating environment or a portion thereof into a partial grid or array. In cases where predicted human behavior has been determined, at least one processor 322 of the processor-based robot control system 300 may, for example, partially follow the safety monitoring rule 125c of the processor-based safety system 200. Figure 1 ), and at least one robot 102 is determined based at least in part on predicted behaviors determined by humans in the operating space or work unit 104. Figure 1 Motion planning for at least one robot 102. At least one processor 322 may determine motion planning for at least one robot 102 based, for example, at least in part, on predicted human behavior (e.g., position / location, time, velocity, trajectory).

[0177] At least one processor 322 of the processor-based robot control system 300 can employ various techniques to determine motion planning that advantageously reduces or even eliminates the possibility that the processor-based cell safety system 200 will trigger at least one of slowing down or stopping the operation of at least one robot 102, or reduces or even eliminates the use of preventative occlusion. For example, at least one processor 322 can adjust the cost value or weight associated with an edge representing a transition between robot configurations that would violate a set of safety monitoring rules 125c enforced by the processor-based cell safety system 200. Figure 1One or more safety rules or conditions, or otherwise, may be specified to trigger the processor-based cell safety system 200 to cause a stop, mitigation, or preventative occlusion. Cost values ​​or weights may be adjusted (e.g., increased) to reduce the probability of selecting an associated transition during the lowest-cost path analysis of the planning map performed by the processor-based robot control system 300. Weights may be adjusted even if the transition does not necessarily lead to a conflict between a part of the robot and a human, but rather if a given transition necessarily or likely to trigger and intervene in the processor-based cell safety system 200 (e.g., causing a stop, mitigation, or preventative occlusion).

[0178] Once the motion plan is determined, control can be passed to 814.

[0179] At point 814, at least one processor 322 of the processor-based robot control system 300 causes at least one robot 102 to move according to a determined motion plan. For example, at least one processor 322 of the processor-based robot control system 300 can send inputs to one or more motion controllers 320 (…). Figure 3 ) provides signals, for example to control one or more robots 102 ( Figure 1 The motor controller provides signals to the movement of (e.g., controlling the motor).

[0180] Method 800 terminates at 816, for example, until it is reactivated. In some implementations, such as when the robot or a part thereof is powered, method 800 may operate continuously or even periodically.

[0181] The foregoing detailed description has illustrated various embodiments of the device and / or process using block diagrams, schematic diagrams, and examples. Wherever such block diagrams, schematic diagrams, and examples contain one or more functions and / or operations, those skilled in the art will understand that each function and / or operation within such block diagrams, flowcharts, or examples can be implemented individually and / or collectively by a wide range of hardware, software, firmware, or virtually any combination thereof. In one embodiment, this subject matter can be implemented using Boolean circuits, application-specific integrated circuits (ASICs), and / or FPGAs. However, those skilled in the art will recognize that the embodiments disclosed herein, in whole or in part, can be implemented in standard integrated circuits in a variety of different ways as one or more computer programs running on one or more computers (e.g., one or more programs running on one or more computer systems), one or more programs running on one or more controllers (e.g., microcontrollers), one or more programs running on one or more processors (e.g., microprocessors), firmware, or virtually any combination of one or more programs, and that designing circuits and / or writing software and / or firmware code according to this disclosure will be entirely within the skill of those skilled in the art.

[0182] Those skilled in the art will recognize that many of the methods or algorithms described herein may employ additional actions, omit some actions, and / or perform actions in an order different from the specified order.

[0183] Furthermore, those skilled in the art will understand that the mechanisms taught herein can be implemented in hardware, such as in one or more FPGAs or ASICs.

[0184] The various embodiments described above can be combined to provide further embodiments. All commonly assigned U.S. patent application publications, U.S. patent applications, foreign patents, and foreign patent applications cited in this specification and / or listed in the application data sheet, including but not limited to International Patent Application Publication No. PCT / US2017 / 036880 entitled "MOTION PLANNING FOR AUTONOMOUS VEHICLES AND RECONFIGURABLE MOTION PLANNING PROCESSORS" filed June 9, 2017; International Patent Application Publication No. WO 2016 / 122840 entitled "SPECIALIZED ROBOT MOTION PLANNING HARDWARE AND METHODS OF MAKING AND USING SAME" filed January 5, 2016; and International Patent Application Publication No. WO 2016 / 122840 entitled "APPARATUS, METHOD AND ARTICLE TO FACILITATE MOTION PLANNING OF AN AUTONOMOUS VEHICLE IN AN ENVIRONMENT HAVING DYNAMIC" filed January 12, 2018. U.S. Patent Application No. 62 / 616,783 entitled “OBJECTS”; U.S. Patent Application No. 62 / 626,939 entitled “MOTION PLANNING OF A ROBOTSTORING A DISCRETIZED ENVIRONMENT ON ONE OR MORE PROCESSORS AND IMPROVEDOPERATION OF SAME” filed on February 6, 2018; U.S. Patent Application No. 62 / 856,548 entitled “APPARATUS, METHODS AND ARTICLES TO FACILITATE MOTION PLANNING IN ENVIRONMENTSHAVING DYNAMIC OBSTACLES” filed on June 3, 2019; and U.S. Patent Application No. 62 / 865,431 entitled “MOTION PLANNING FOR MULTIPLE ROBOTS IN SHARED WORKSPACE” filed on June 24, 2019.U.S. Patent Application No. 63 / 105,542, entitled "SAFETY SYSTEMS AND METHODS EMPLOYED INROBOT OPERATIONS," filed October 26, 2020, and International Patent Application PCT / US2020 / 039193, entitled "MOTION PLANNING FOR MULTIPLE ROBOTS IN SHARED WORKSPACE," filed June 23, 2020, are both incorporated herein by reference in their entirety. These and other changes may be made to the embodiments based on the above detailed description. Generally, the terminology used in the appended claims should not be construed as limiting the claims to the specific embodiments disclosed in the specification and claims, but should be construed as including all possible embodiments and the full scope of equivalents conferred by such claims. Therefore, the claims are not limited to this disclosure.

Claims

1. A method for operating a processor-based system for monitoring the operating environment in which at least one robot operates, the method comprising: Information is received from a first sensor, which is positioned and oriented to detect the location of a human in a first part of the operating environment in the presence of a human. Information is received from a second sensor, which is positioned and oriented to detect the location of a human in a second part of the operating environment in the presence of a human, the second part of the operating environment overlapping at least partially with the first part of the operating environment, and the second sensor is heterogeneous relative to the first sensor; For each of the first sensor and the second sensor, an evaluation of the corresponding operating state of the first sensor and the second sensor is performed by at least one processor, wherein performing the evaluation of the corresponding operating state of the first sensor and the second sensor includes: determining by at least one processor whether information received from the first sensor and the second sensor indicates that one or both of the first sensor and the second sensor are erroneously retransmitting stale information. The system state is verified at least in part based on a set of rules, which specify the system state at least in part based on the corresponding operating states of the first and second sensors being evaluated. At least once, it was determined that an abnormal system state existed; and In response to determining the existence of the abnormal system state, the at least one processor provides a signal to at least partially control the operation of the at least one robot.

2. The method according to claim 1, wherein, Providing a signal to at least partially control the operation of the at least one robot in response to determining the presence of the abnormal system state includes: providing a signal to prevent or slow down the movement of the at least one robot, at least until the abnormal system state is mitigated.

3. The method according to claim 1, wherein, Providing a signal to at least partially control the operation of the at least one robot in response to determining the presence of the abnormal system state includes: providing a signal indicating an area of ​​the operating environment as occluded for motion planning.

4. The method according to claim 1, wherein, Determining whether the information received from the first sensor and the second sensor indicates that one or both of the first sensor and the second sensor are erroneously retransmitting stale information includes: determining whether the reference represented in the information received from the first sensor and the second sensor has moved over a period of time.

5. The method according to claim 1, wherein, Determining whether information received from the first sensor and the second sensor indicates that one or both of the first sensor and the second sensor are erroneously retransmitting stale information includes: determining whether the movement of a reference represented in the information received from the first sensor and the second sensor is consistent with the expected movement of the reference over a period of time.

6. The method according to claim 5, wherein, The reference is part of or carried by the at least one robot, and determining whether the movement of the reference, as indicated by information received from the first and second sensors, is consistent with the expected movement of the reference over a period of time includes: determining whether the movement of the reference matches the movement of the at least one robot's part over the period of time.

7. The method according to claim 5, wherein, The reference is independent of the movement of the at least one robot, and determining whether the movement of the reference, as indicated by information received from the first and second sensors, is consistent with the expected movement of the reference over a period of time includes: determining whether the movement of the reference matches the expected movement of the reference over the period of time.

8. The method according to claim 1, wherein, At least one of the first sensor and the second sensor moves in a defined pattern over a period of time, and determining whether the information received from the first sensor and the second sensor indicates that one or both of the first sensor and the second sensor are erroneously retransmitting stale information includes: determining, based on the movement of the first sensor or the second sensor over a period of time, whether the apparent motion of a reference represented in the information received from the first sensor and the second sensor is consistent with the expected apparent motion of the reference over the period of time.

9. The method according to any one of claims 1 to 8, wherein, The first sensor has a first operating mode, the second sensor has a second operating mode, the second operating mode is different from the first operating mode, and receiving information from the first sensor includes receiving information from the first sensor in the first modal format, and receiving information from the second sensor includes receiving information from the second sensor in the second modal format, the second modal format is different from the first modal format.

10. The method according to claim 9, wherein, The first operating mode of the first sensor is an image sensor and the first mode format is a digital image, and the second operating mode of the second sensor is at least one of a laser scanner, a passive infrared motion sensor, and a thermal sensor, and the second mode format is a non-image digital signal.

11. The method according to any one of claims 1 to 8, wherein, The first sensor has a first field of view of the operating environment and the second sensor has a second field of view of the operating environment, the second field of view being different from the first field of view, and receiving information from the first sensor includes receiving information from the first sensor having the first field of view and receiving information from the second sensor includes receiving information from the second sensor having the second field of view.

12. The method according to any one of claims 1 to 8, wherein, The first sensor is a first sensor brand and model, the second sensor is a second sensor brand and model, at least one of the second sensor brand and model is different from the corresponding one of the first sensor brand and model, and receiving information from the first sensor includes receiving information from the first sensor brand and model, and receiving information from the second sensor includes receiving information from the second sensor brand and model.

13. The method according to claim 12, wherein, The first sensor has a first operating mode and the second sensor has a second operating mode, the second operating mode being different from the first operating mode.

14. The method according to claim 12, wherein, The first sensor has a first operating mode and the second sensor has a second operating mode, the second operating mode being the same as the first operating mode.

15. The method according to any one of claims 1 to 8, wherein, The first sensor has a first sampling rate, the second sensor has a second sampling rate, the second sampling rate is different from the first sampling rate, and receiving information from the first sensor includes receiving information captured at the first sampling rate from the first sensor, and receiving information from the second sensor includes receiving information captured at the second sampling rate from the second sensor.

16. The method according to claim 15, wherein, The evaluation of the corresponding operating states of the first and second sensors includes: determining whether the information received from the first sensor is consistent with the first sampling rate of the first sensor, and determining whether the information received from the second sensor is consistent with the second sampling rate of the second sensor.

17. The method according to any one of claims 1 to 8, wherein, Performing an assessment of the corresponding operational states of the first and second sensors includes comparing information received from the first and second sensors, which at least partially overlap with the first part of the second part used in the operating environment, to determine whether there are any discrepancies.

18. The method according to claim 17, wherein, In response to determining that a discrepancy exists, the system determines that the abnormal system state exists and prevents the at least one robot from moving until the discrepancy is resolved.

19. The method of claim 17, wherein, In response to determining that a difference exists, the existence of the abnormal system state is determined and the presence of a portion of the difference in the operating environment is occluded during motion planning until the difference is resolved.

20. The method according to any one of claims 1 to 8, further comprising: Information is received from a third sensor, which is located and oriented to detect the position of a human in a third part of the operating environment in the presence of a human, the third part of the operating environment overlapping at least partially with the first and second parts of the operating environment, and the third sensor being heterogeneous relative to at least one of the first and second sensors. and The evaluation of the corresponding operating states of the first and second sensors includes: evaluating the corresponding operating states of the first, second, and third sensors based at least in part on information received from the first, second, and third sensors.

21. The method according to claim 20, wherein, The evaluation of the corresponding operating states of the first, second, and third sensors is performed, based at least in part on the consistency among most of the first, second, and third sensors.

22. The method according to any one of claims 1 to 8, further comprising: At least once confirm that a non-abnormal system state exists; as well as In response to determining the existence of the non-abnormal system state, the at least one processor provides a signal to at least partially control the operation of the robot.

23. The method according to claim 22, wherein, Providing a signal to at least partially control the operation of the at least one robot in response to determining the existence of the non-abnormal system state includes: providing a signal that allows the assumption that the entire work unit is occluded to be relaxed in response to determining that at least two of the sensors are consistent with each other.

24. The method according to claim 22, wherein, Providing a signal to at least partially control the operation of the at least one robot in response to determining the existence of the non-abnormal system state includes: providing a signal that allows the at least one robot to move or indicates that an area of ​​the operating environment is not represented as occluded.

25. A system for monitoring the operating environment in which at least one robot operates, comprising: The first sensor is positioned and oriented to detect the location of a human in the first part of the operating environment in the presence of a human. A second sensor is positioned and oriented to detect the location of a human in the presence of a human in a second part of the operating environment, the second part of the operating environment at least partially overlapping the first part of the operating environment, and the second sensor is heterogeneous relative to the first sensor. as well as At least one processor communicatively coupled to receive information from a first sensor and a second sensor, the at least one processor being operable to execute processor-executable instructions, which, when executed by the at least one processor, cause the at least one processor to perform the method according to any one of claims 1 to 24.

26. A processor-readable medium storing processor-executable instructions, which, when executed, cause one or more processors to perform the method according to any one of claims 1 to 24.

Images