Work space safety monitoring and device control

Abstract

Systems and methods for monitoring a workspace for safety purposes using sensors distributed around it. The sensors are registered relative to each other, and this registration is monitored over time. Occluded and occupied spaces are identified, and this mapping is frequently updated.

Description

Cross-references to related applications

[0001] This application claims priority and benefit to U.S. Provisional Patent Application Nos. 62 / 455,828 and 62 / 455,834, filed on February 7, 2017. Technical Field

[0002] The field of this invention generally relates to monitoring industrial environments in which people and machinery interact or are in proximity, and more particularly to systems and methods for detecting unsafe conditions in monitored workspaces. Background Technology

[0003] Industrial machinery is generally dangerous to humans. Some machinery is dangerous unless completely shut down, while others may have multiple operating states, some dangerous and some not. In some cases, the level of danger may depend on a person's position relative to the machinery. Therefore, many "protective" methods have been developed to prevent injury from machinery. One very simple and common type of protection is a enclosure surrounding the machinery, constructed such that opening the enclosure's door shuts down the circuitry. This ensures that a person can never approach the machinery while it is running. Of course, this prevents all interaction between people and the machine and severely limits the use of the workspace.

[0004] More sophisticated types of protection involve optical sensors. Examples include light curtains, which determine if any object has intruded into an area monitored by one or more light emitters and detectors, and 2D LiDAR sensors, which use active optical sensing to detect the minimum distance to an obstacle along a series of light rays emitted from the sensor, and can therefore be configured to detect approach or intrusion into a pre-configured two-dimensional (2D) area. Recently, systems have begun to use, for example, 3D time-of-flight cameras, 3D LiDAR, and stereo vision cameras to incorporate 3D depth information. These sensors have the ability to detect and locate intrusions into the area surrounding industrial machinery in 3D, offering several advantages. For example, a 2D LiDAR system protecting an industrial robot would have to stop the robot if an intrusion is detected at a distance far exceeding the robot's arm length, because if the intrusion represents a person's leg, that person's arm might be closer and would not be detected by a planar LiDAR. However, a 3D system can allow the robot to continue operating until the person actually extends his or her arm toward the robot. This allows for a tighter interlock between machine and human actions, which is beneficial for many applications and saves space in the factory floor, which is always very important.

[0005] Because they endanger personal safety, protective equipment must typically meet stringent industry standards. These standards may specify failure rates for hardware components and rigorous development practices for both hardware and software components. A compliant system must ensure a high probability of detecting hazardous situations, detecting system malfunctions, and responding to detected malfunctions by converting the equipment to a controlled state in a safe condition. Designing protective equipment becomes particularly challenging in tasks involving human-machine collaboration. While machines may be more powerful, faster, more precise, and more repeatable than humans, they lack human dexterity, skill, and judgment. An example of collaborative applications is installing a dashboard in a car—the dashboard is heavy and difficult for a person to manipulate, but connecting it requires various connectors and fasteners that necessitate manual handling. Simply separating humans from machines means that protective measures are much simpler, and then detecting unsafe conditions when a person actively uses a machine that could harm them. Traditional protective systems lack sufficient operational granularity to reliably monitor such environments.

[0006] 3D sensor systems offer the possibility of improving the granularity of protection systems. However, compared to 2D sensor systems, 3D data systems can be more difficult to configure. First, specific areas must be designed and configured for each use case, taking into account the potential movements of machinery and people in the workspace, the workspace layout, and the specific hazards posed by the position and field of view of each sensor. Calculating the optimal shape of the exclusion zone can be difficult, especially when trying to optimize ground space and system throughput while maintaining safety, where an object may occlude relative to a sensor, and where light levels differ relative to different sensors. Misconfigurations can lead to serious safety hazards, requiring significant design and testing expenditures. If any changes are made to the workspace, all of this work must be completely redone. The additional degrees of freedom offered by 3D systems result in a wider range of possible configurations and hazards. Therefore, there is a need for improved and computationally tractable techniques for monitoring workspaces with high granularity.

[0007] Even with precise mapping and monitoring of the workspace, maintaining safety in a rapid and uneven manner is crucial in dynamic environments where robots and humans can move—that is, change positions and configurations. Typical industrial robots are stationary but still possess powerful robotic arms that can cause injury within a large “envelope” of possible motion trajectories. Typically, robotic arms consist of numerous mechanical links connected by precisely controllable rotary joints, with a controller coordinating all joints to achieve a trajectory determined by industrial engineers for a specific application.

[0008] A single robot application may only use a portion of the robot's entire range of motion. However, the software controlling the robot's trajectory is typically not considered or developed as part of a robot safety system. Therefore, while the robot may only use a small portion of its trajectory envelope, protective systems (e.g., enclosures) are configured to cover the robot's entire range of motion. Like other types of protection, such devices have evolved from simple mechanical solutions to electronic sensors and software controls. In recent years, robot manufacturers have also introduced so-called "soft" axis and rate limiting systems—safety-grade software that restricts the robot to certain portions of its range of motion and specific speeds. This constraint is then enforced within the safety-grade software—an emergency stop is initiated if the robot is detected violating the soft axis and rate limit settings at any time. This approach increases the effective safety zone around the robot and supports collaborative applications.

[0009] However, these systems exhibit at least two significant drawbacks. First, given the robot's trajectory, potential human movements within the workspace, workspace layout, and the position and field of view of each sensor, specific areas typically must be programmed by industrial engineers for each use case. Regardless of the precision with which these areas can be characterized and monitored, calculating their optimal shape is challenging. Configuration errors can lead to serious safety hazards, requiring reconfiguration of the safety areas should any changes be made to the robot program or the workspace. Second, these areas and speed limits are discrete—there is often no way to scale down the robot's speed to the precise distance required between the robot and detected obstacles; therefore, they must be highly conservative. Another complexity lies in the need to plan anticipated and probable robot trajectories, including workpieces already picked up by the robot or already associated with it. Therefore, new methods for dynamically configuring and reconfiguring safety areas are needed as workspace occupancy and robot tasks evolve. Summary of the Invention

[0010] On one hand, embodiments of the present invention provide systems and methods for monitoring a workspace for safety purposes using sensors distributed around the workspace. The workspace may contain one or more devices that may pose a hazard to humans, as well as industrial robots and auxiliary equipment, such as part feeders, guide rails, fixtures, or other machines. The sensors are registered relative to each other and this registration is monitored over time. Occluded spaces and occupied spaces are identified, and this mapping is updated frequently.

[0011] Areas within the monitored space may be marked as occupied, unoccupied, or unknown; only empty spaces can ultimately be considered safe, and only if any other safety criteria are met—for example, a minimum distance from a controlled piece of machinery. Typically, raw data from each sensor is analyzed to determine whether objects or boundaries in the 3D-mapped space have been definitively detected within the entire coverage area corresponding to the sensor.

[0012] When a person moves in 3D space, he or she will typically obscure certain areas, hindering certain sensors and resulting in temporarily unknown areas in the space. Additionally, mobile mechanical devices such as industrial robotic arms may also temporarily obscure certain areas. When the person or mechanical device moves to a different location, one or more sensors will be able to observe the unknown space again and return it to a state of confirmation that it is empty, thus ensuring safety for the robot or machine to operate in that space. Therefore, in some embodiments, spaces can also be classified as "potentially occupied." An unknown space is considered potentially occupied when there is a situation where it may be occupied. This can occur when an unknown space is adjacent to the entrance point of the workspace, or if an unknown space is adjacent to an occupied or potentially occupied space. Potentially occupied spaces "infect" unknown spaces at a rate representing the rate of movement of a person in the workspace. Potentially occupied spaces remain in a potentially occupied state until they are observed to be empty. For safety purposes, potentially occupied spaces are treated as identical to occupied spaces.

[0013] For certain sensor models, such as those relying on active light signals, the sensor's ability to definitively detect objects or boundaries decreases rapidly with increasing distance; that is, beyond a certain distance, the sensor may be unable to distinguish between objects and empty space because the relevant light levels are too similar. Points or areas at such locations are marked as "unknown" relative to the relevant sensor, and such marked areas cannot be identified as empty by the sensor.

[0014] On the other hand, embodiments of the present invention provide systems and methods for determining safe zones in a workspace, wherein safe actions are calculated in real time based on all sensed relevant objects and the current state of mechanical equipment (e.g., robots) in the workspace. These embodiments may, but must not, utilize the workspace monitoring methods described in the detailed description below. Embodiments of the present invention use, for example, models of human motion and other forms of control to perform dynamic modeling of robot geometry and predict the future trajectory of the robot and / or human. In some embodiments, robot modeling and prediction may utilize data provided by a robot controller that may or may not include safety assurances. However, in either case, embodiments of the present invention are able to provide safety assurances through independent verification of the data and the use of safety-level stop functions.

[0015] Embodiments of this invention can predict the movement of mechanical equipment and potential human movement within a space in real time, and continuously update the predictions as the equipment operates and people move within the workspace. As the system tracks and predicts, it may encounter unknown, potentially occupied, occupied, or unidentified volumes. The system treats such volumes as currently occupied. Our method overcomes the need for programming specific areas, operates without discrete speed limits, and maintains robot movement within a wider range of human actions within the workspace, thereby reducing workspace areas designated as off-limits to personnel even as the robot continues to operate.

[0016] On another front, embodiments of the invention determine the configuration of workpieces and whether they are actually handled by monitored mechanical equipment such as robots. The problems addressed by these embodiments are particularly challenging in real-world factory environments because many objects (most of which are not workpieces) may be close to mechanical equipment. Therefore, such embodiments can leverage semantic understanding to distinguish workpieces that may be associated with mechanical equipment and other objects (and people) in the workspace, which may or may not be detected, for example, when a robot is handling a workpiece. In this case, to establish the envelope of possible robot trajectories, the workpiece is considered part of the robot. The envelope is tracked as the robot and workpiece move together within the work cell, and the corresponding occupied space is dynamically marked as non-empty and unsafe. Unless independent verification of emptiness can be obtained from other sensors, the 3D space occluded by the robot-workpiece combination will be marked as non-empty.

[0017] In various embodiments, the system includes multiple sensors distributed around a workspace. Each sensor includes or is associated with a pixel grid for recording a representation of a portion of the workspace within the sensor's field of view; multiple portions of the workspace collectively cover the entire workspace. Computer memory stores (i) a series of images from the sensors, (ii) a model of the robot and its permitted movements, and (iii) safety protocols specifying speed limits for the robot when approaching a human and minimum distances between the robot and the human. The processor is configured to generate a spatial representation of the workspace from the stored images (e.g., as a volume, which may correspond to voxels, i.e., 3D pixels). The processor identifies and monitors the space occupied by the robot within the workspace over time as a representation of the robot region in the volume. The processor generates an envelope region around the robot region based on the stored model, the envelope region spanning the robot's permitted movements.

[0018] The processor also identifies and monitors the volumes representing workpieces. This identification can be aided by information about the physical shape of the workpieces determined during the configuration process, which may include CAD models, 3D scans, or 3D models learned by the system during the learning phase. These workpiece volumes are then characterized to determine that they are unoccupied, thus allowing the robot to approach according to safety protocols. Additionally, the processor identifies interactions between the robot and the workpieces within the workspace and, in response to the identified interactions, updates the robot region to include the workpieces and updates the envelope region based on the stored model and the updated robot region. The processor generates a safety zone around the robot region according to the safety protocols as it is updated.

[0019] Generally, as used herein, the term "substantially" means ±10%, and in some embodiments, ±5%. Additionally, references to "an example," "example," "an embodiment," or "an embodiment" in the specification indicate that a particular feature, structure, or characteristic relating to the description of that example is included in at least one instance of the present technology. Therefore, the phrases "in an example," "in a sample," "an embodiment," or "an embodiment" appearing in various places throughout this specification do not necessarily refer to the same example. Furthermore, specific features, structures, procedures, steps, or characteristics may be combined in any suitable manner in one or more examples of the present technology. The headings provided herein are for convenience only and are not intended to limit or interpret the scope or meaning of the claimed technology. Attached Figure Description

[0020] In the accompanying drawings, the same reference numerals generally refer to the same parts in different views. Furthermore, the drawings are not necessarily drawn to scale, but generally focus on illustrating the principles of the invention. In the following description, various embodiments of the invention are described with reference to the following drawings, wherein: Figure 1This is a perspective view of the monitored workspace according to an embodiment of the present invention.

[0021] Figure 2 The classification of areas within a monitored workspace according to an embodiment of the present invention is illustrated schematically.

[0022] Figure 3 A control system according to an embodiment of the present invention is illustrated schematically.

[0023] Figure 4 An object monitoring system according to an embodiment of the present invention is illustrated schematically.

[0024] Figure 5 The diagram schematically illustrates the limitation of a progressive safety envelope near an industrial machine. Detailed Implementation

[0025] In the following discussion, we describe an integrated system for monitoring a workspace, classifying areas therein for safety purposes, and dynamically identifying safety conditions. In some cases, the latter functions involve semantic analysis of robots within the workspace and identification of workpieces interacting with them. However, it should be understood that these various elements can be implemented individually or in desired combinations; not all of the elements described are required for the inventive aspects discussed herein, but they are presented together only for ease of presentation and illustration of their interoperability. The described system represents only one embodiment.

[0026] 1. Workspace monitoring First refer to Figure 1 It shows a 3D workspace 100 monitored by multiple sensors, represented by 1021, 1022, and 1023. 。 Sensor 102 can be a conventional optical sensor, such as a camera, like a 3D time-of-flight camera, stereo vision camera, or 3D LiDAR sensor, or a radar-based sensor, ideally with a high frame rate (e.g., between 30Hz and 100Hz). The operating mode of sensor 102 is not critical, as long as a 3D representation of workspace 100 can be obtained from images or other data acquired by sensor 102. As shown, sensors 102 collectively cover and monitor workspace 100, which includes a robot 106 controlled by a conventional robot controller 108. The robot interacts with various workpieces W, and a person P in workspace 100 can interact with both the workpieces and robot 108. Workspace 100 may also contain various auxiliary devices 110, which can complicate workspace analysis due to the obstruction of various parts of the workspace by sensors. In reality, any realistic sensor arrangement often cannot "see" at least some parts of the active workspace. This is in Figure 1As shown in the simplified arrangement, due to the presence of person P, at least some parts of the robot controller 108 may be obscured from all the sensors. In an environment where people move back and forth and even stationary objects may move from time to time, areas that cannot be observed will shift and change.

[0027] like Figure 2 As shown, embodiments of the present invention classify workspace areas as occupied, unoccupied (or empty), or unknown. For ease of explanation, Figure 2 Two sensors 2021 and 2022 are shown, along with their coverage areas 2051 and 2052 in two dimensions within the workspace 200; similarly, only the 2D coverage area 210 of the 3D object is shown. The portion of the coverage area 205 between the object boundary and the sensors 200 is marked as unoccupied because neither sensor has detected any obstruction in this intermediate space. The space at the object boundary is marked as occupied. In the coverage area 205 outside the object boundary, all space is marked as unknown; the corresponding sensors are configured to sense occupancy in this area, but cannot do so due to the presence of the object 210.

[0028] Re-reference Figure 1 Data from each sensor 102 is received by the control system 112. The volume of space covered by each sensor—typically a solid cone—can be represented in any suitable manner; for example, the space can be divided into small (e.g., 5 cm) cubes or a 3D mesh of “voxels” or other suitable forms of volume representation. For example, workspace 100 can be represented using 2D or 3D ray tracing, where the intersection of 2D or 3D rays emanating from sensor 102 is used as the volume coordinates of workspace 100. This ray tracing can be performed dynamically or using a pre-calculated volume, where objects in workspace 100 have been identified and captured in advance by the control system 112. For ease of representation, the following discussion assumes the use of voxel representation; the control system 112 maintains the internal representation of workspace 100 at the voxel level, where voxels are labeled as occupied, unoccupied, or unknown.

[0029] Figure 3A representative embodiment of the control system 112, which can be implemented on a general-purpose computer, is shown in more detail. The control system 112 includes a central processing unit (CPU) 305, system memory 310, and one or more non-volatile mass storage devices (such as one or more hard disks and / or optical storage units) 312. The system 112 also includes a bidirectional system bus 315 through which the CPU 305, memory 310, and storage devices 312 communicate with each other and with internal or external input / output (I / O) devices such as a display 320 and peripheral devices 322, which may include conventional input devices such as a keyboard or mouse. The control system 112 also includes a wireless transceiver 325 and one or more I / O ports 327. The transceiver 325 and I / O ports 327 provide a network interface. The term "network" is used broadly herein to refer to a wired or wireless network of a computer or telecommunications device (e.g., a wired or wireless telephone, tablet computer, etc.). For example, a computer network may be a local area network (LAN) or a wide area network (WAN). When used in a LAN network environment, a computer can connect to the LAN via a network interface or adapter; for example, an administrator can use a wirelessly connected tablet to establish communication with the control system 112. When used in a WAN network environment, the computer typically includes a modem or other communication mechanism. The modem can be internal or external and can be connected to the system bus via a user input interface or other suitable mechanism. Networked computers can connect via the Internet, intranet, extranet, Ethernet, or any other system that provides communication. Some suitable communication protocols include, for example, TCP / IP, UDP, or OSI. For wireless communication, communication protocols can include IEEE 802.11x (“Wi-Fi”), Bluetooth, ZigBee, IrDa, Near Field Communication (NFC), or other suitable protocols. Furthermore, components of the system can communicate via a combination of wired or wireless paths, and the communication can involve both the computer and telecommunications networks.

[0030] CPU 305 is typically a microprocessor, but in various embodiments it may be a microcontroller, peripheral integrated circuit element, CSIC (customer application-specific integrated circuit), ASIC (application-specific integrated circuit), logic circuit, digital signal processor, programmable logic device such as FPGA (field-programmable gate array), PLD (programmable logic device), PLA (programmable logic array), RFID processor, graphics processing unit (GPU), smart chip, or any other device or device setup capable of implementing the steps of the process of the present invention.

[0031] System memory 310 includes a series of frame buffers 335, which are partitions that store images acquired by sensor 102 in digital form (e.g., as pixels or voxels or as depth maps); as described above, the data can actually be reached via I / O port 327 and / or transceiver 325. System memory 310 contains instructions conceptually illustrated as a set of modules that control the operation of CPU 305 and its interaction with other hardware components. Operating system 340 (e.g., Windows or Linux) directs the execution of low-level basic system functions, such as memory allocation, file management, and the operation of mass storage device 312. At a higher level, and as described in more detail below, analysis module 342 registers images in frame buffers 335 and analyzes them to classify regions of the monitored workspace 100. The results of the classification can be stored in a space map 345, which contains a volumetric representation of workspace 100, wherein each voxel (or other representation unit) in the space map is labeled as described herein. Alternatively, spatial graph 345 can simply be a 3D array of voxels, with voxel labels stored in a separate database (in memory 310 or in mass storage 312).

[0032] The control system 112 can also use common control routines, collectively indicated by 350, to control the operating or mechanical equipment in the workspace 100. As explained below, the configuration of the workspace and therefore the classification associated with its voxel representation may change over time as people and / or machines move, and the control routines 350 can respond to these changes in the operating mechanical equipment to achieve a high level of safety. All modules in the system memory 310 can be programmed in any suitable programming language, including but not limited to high-level languages ​​such as C, C++, C#, Ada, Basic, Cobra, Fortran, Java, Lisp, Perl, Python, Ruby, or low-level assembly language.

[0033] 1.1 Sensor registration In a typical multi-sensor system, the precise position of each sensor 102 relative to all other sensors is established during setup. Sensor registration is typically performed automatically and should be as simple as possible to simplify setup and reconfiguration. For simplicity, assuming each frame buffer 335 stores images from a specific sensor 102 (which can be refreshed periodically), the analysis module 342 can register the sensor 102 by comparing all or part of the images from each sensor with images from other sensors in the frame buffer 335 and using conventional computer vision techniques to identify correspondences in those images. Suitable global-registration algorithms that do not require an initial registration approximation generally fall into two categories: feature-based methods and intensity-based methods. Feature-based methods identify correspondences between image features (such as edges), while intensity-based methods use correlation metrics between intensity patterns. Once an approximate registration is identified, the iterative nearest point (ICP) algorithm or a suitable variant thereof can be used to fine-tune the registration.

[0034] If there is sufficient overlap between the fields of view of the individual sensors 102, and enough detail in the workspace 100 to provide different sensor images, it may be sufficient to compare images of a static workspace. If this is not the case, a “registration object” with a unique 3D signature can be placed within the workspace 100 at a location detectable by all sensors. Alternatively, registration can be achieved by having the sensors 102 record images of one or more people standing or walking in the workspace over a period of time, combining a sufficient number of partially matching images until accurate registration is achieved.

[0035] In some cases, registration of mechanical equipment within workspace 100 can be performed without any additional instruments, particularly if the equipment has a unique 3D shape (e.g., a robotic arm), provided the equipment is visible to at least one sensor registered relative to the others. Alternatively, registration objects can be used, or a user interface showing and displaying the scene observed by the sensors on display 320 can allow the user to designate portions of an image as key elements of the controlled mechanical equipment. In some embodiments, the interface provides an interactive 3D display showing the coverage of all sensors to aid in configuration. If the system is configured with some level of advanced information about the controlled mechanical equipment (e.g., for the purposes of control routine 350)—such as the location of one or more hazardous parts of the equipment and their stopping times and / or distances—analysis module 342 can be configured to provide intelligent feedback on whether the sensors provide sufficient coverage and suggest placement of additional sensors.

[0036] For example, given a conservative estimate of walking speed, the analysis module 342 can be programmed to determine the minimum distance from the observed mechanical device that must detect a person so that the device stops when the person approaches (or a safe area around it). (Alternatively, the required detection distance can be directly input into the system via the display 320.) Optionally, the analysis module 342 can then analyze the field of view of all sensors to determine if the space is sufficiently covered to detect all approaches. If sensor coverage is insufficient, the analysis module 342 can suggest new positions for existing sensors or for additional sensors to compensate for the deficiency. Otherwise, the control system will default to a safe state, and control routine 350 will not allow the mechanical device to operate unless the analysis module 342 confirms that all approaches can be effectively monitored. Machine learning and genetic or evolutionary algorithms can be used to determine the optimal sensor placement within the cell. Parameters to be optimized include, but are not limited to, minimizing occlusion around the robot during operation and the observability of the robot and workpiece.

[0037] If necessary, this static analysis may include "background" subtraction. During the initial startup phase, when it can be safely assumed that no objects have intruded into workspace 100, analysis module 342 identifies all voxels occupied by static elements. These elements can then be subtracted from future measurements, not considered potential intruders. Nevertheless, continuous monitoring is performed to ensure that the observed background image is consistent with the space map 345 stored during startup. The background may also be updated if stationary objects are removed or added to the workspace.

[0038] There may be areas that sensor 102 cannot adequately observe to provide safety, but these areas are protected by other methods (such as enclosure). In this case, the user interface can allow the user to designate these areas as safe, overriding sensor-based safety analysis. Safety-rated soft 4-axis and rate limiting can also be used to limit the robot's envelope to improve system performance.

[0039] Once registration is achieved, sensors 102 should remain in the same position and orientation while monitoring workspace 100. If one or more sensors 102 move unexpectedly, the resulting control output will be invalid and could potentially cause safety hazards. Analysis module 342 can extend the algorithm used for initial registration to monitor the continuous accuracy of registration. For example, during initial registration, analysis module 342 can calculate an index of the fitting accuracy between the observed data and the model of the static elements of the work cells captured during the registration process. As the system operates, the same index can be recalculated. If the index exceeds a specified threshold at any time, the registration is considered invalid, and an error condition is triggered; in response, if any machinery is operating, control routine 350 can stop it or switch the machinery to a safe state.

[0040] 1.2 Identify occupied and potentially occupied areas Once the sensors are registered, the control system 112 periodically updates the spatial map 345 at a high, fixed frequency (e.g., every analysis cycle) to identify any intrusions into the workspace 100. The spatial map 345 reflects data fusion from some or all of the sensors 102. However, given the nature of the 3D data, depending on the location of the sensors 102 and the configuration of the workspace 100, an object in one location may obstruct the sensor's field of view of objects in other locations, including objects closer to hazardous machinery (potentially including people or parts of people, such as arms). Therefore, to provide a reliable safety system, the system monitors both obstructed and occupied spaces.

[0041] In one embodiment, the space map 345 is a voxel grid. Typically, each voxel may be marked as occupied, unoccupied, or unknown; only empty spaces can ultimately be considered safe, and only if any other safety criterion—e.g., a minimum distance from a controlled mechanical device—is met. Raw data from each sensor is analyzed to determine whether an object or boundary of the 3D mapped space has been definitively detected for each voxel in the volume corresponding to that voxel. To enhance safety, the analysis module 342 may designate only voxels that are observed to be empty by the multiple sensors 102 as empty. Similarly, all spaces that cannot be confirmed as empty are marked as unknown. Thus, only the space between the sensor 102 and the detected object or the boundary of the mapped 3D space along the ray can be marked as empty.

[0042] If a sensor detects anything in a given voxel, all voxels located on a ray starting from the sensor's focal point and passing through the occupied voxel, as well as those located between the focal point and the occupied voxel, are classified as unoccupied, while all voxels located outside the occupied voxels on that ray are classified as occluded for the sensor; all these occluded voxels are considered "unknown." Information from all sensors can be combined to determine which areas are occluded for all sensors; these areas are considered unknown and therefore unsafe. Analysis module 342 may ultimately only mark voxels or workspace volumes that have been marked "unoccupied" at least once (or, in some embodiments, at least twice) as "unoccupied." Based on the markings associated with voxels or discrete volumes within the workspace, analysis module 342 may map one or more safe volume regions within space map 345. These safe regions are outside the safe areas of the mechanical equipment and include only voxels or workspace volumes marked as unoccupied.

[0043] A common failure mode of reflectivity-dependent active optical sensors (such as LiDAR and Time-of-Flight cameras) is that they do not return any signal from surfaces with insufficient reflection and / or when the angle of incidence between the sensor and the surface is too small. This can lead to dangerous failures because the signal may be indistinguishable from the returned result if no obstacle is encountered; in other words, the sensor will report an empty voxel even though an obstacle may be present. This is why ISO standards specify minimum reflectivity for objects that must be detected for, for example, 2D LiDAR sensors; however, these reflectivity standards may be difficult to meet for some 3D sensor forms (such as ToF). To mitigate this form of failure, the analysis module 342 marks the space as empty only when some obstacle is definitively detected at a greater distance along the same ray. By pointing the sensor slightly downwards so that most rays will encounter the floor in the absence of obstacles, most of the workspace 100 can be definitively analyzed. However, if the level of light sensed in a given voxel is insufficient to definitively determine that it is empty or that a boundary exists, the voxel is marked as unknown. The signal and threshold can vary depending on the type of sensor used. In the case of intensity-based 3D sensors (e.g., time-of-flight cameras), the threshold can be the signal strength, which may be attenuated by objects with low reflectivity in the workspace. In the case of stereo vision systems, the threshold can be the ability to distinguish individual objects within the field of view. Depending on the type of sensor used, other combinations of signals and thresholds can be utilized.

[0044] A security system can be created by treating all unknown spaces as occupied. However, this can be overly conservative in some situations, leading to poor performance. Therefore, it is desirable to further categorize unknown spaces based on whether they are likely to be occupied. When a person moves within 3D space, he or she typically obscures certain areas of some sensors, resulting in temporarily unknown areas in the space (see...). Figure 1 Furthermore, moving mechanical devices such as industrial robotic arms may temporarily obscure certain areas. When a person or mechanical device moves to a different location, one or more sensors will be able to observe the unknown space again and return it to a confirmed empty state, in which the operation of the robot or machine is safe. Therefore, in some embodiments, spaces can also be classified as "potentially occupied." An unknown space is considered potentially occupied when there is a situation where it may be occupied. This may occur when an unknown space is adjacent to the entrance point of the workspace, or if an unknown space is adjacent to an occupied or potentially occupied space. Potentially occupied spaces "infect" unknown spaces at a rate representing the rate at which a person moves within the workspace. Potentially occupied spaces remain in a potentially occupied state until they are observed to be empty. For safety purposes, potentially occupied spaces are treated the same as occupied spaces. Probabilistic techniques such as Bayesian filtering may be needed to determine the state of each voxel, allowing the system to combine data from multiple samples to provide higher confidence in the results. Suitable models of human motion, including predicted speeds (e.g., the speed at which an arm is raised may be faster than the speed at which a person walks), are readily available.

[0045] 2. Classify objects For many applications, the classification of areas within the workspace described above may be sufficient—for example, if the control system 112 is monitoring a space where there should be absolutely no objects during normal operation. However, in many cases, it is necessary to monitor areas where at least some objects are present during normal operation, such as one or more machines and the workpieces running on them. In these cases, the analysis module 342 can be configured to identify accidental or potentially human intrusive objects. A suitable approach to this classification is to cluster the individual occupied voxels into objects that can be analyzed at a higher level.

[0046] To achieve this, the analysis module 342 can implement any of several common, well-known clustering techniques, such as Euclidean clustering, K-means clustering, and Gibbs-sampling clustering. Any of these or similar algorithms can be used to identify clusters of occupied voxels from 3D point cloud data. Mesh techniques can also be used, which determine a grid best suited to the point cloud data, and then the optimal clustering can be determined using the grid shape. Once identified, these clusters can be used in various ways.

[0047] One simple clustering method that can be used is to eliminate groups of occupied or potentially occupied voxels that are too small to accommodate a person. As mentioned above, such small clusters may arise from occupancy and occlusion analysis, and otherwise may cause the control system 112 to incorrectly identify hazards. Clusters can be tracked over time by simply associating identified clusters in each image frame with nearby clusters in previous frames, or by using more sophisticated image processing techniques. The shape, size, or other features of clusters can be identified and tracked from one frame to the next. Such features can be used to confirm associations between clusters between frames, or to identify the movement of clusters. This information can be used to enhance or enable certain classification techniques described below. Additionally, clustering of tracking points can be used to identify errors and thus identify potential hazards. For example, clusters that are not present in previous frames and are not close to known boundaries of the field of view can indicate errors.

[0048] In some cases, it may be sufficient to filter out clusters below a certain size and identify cluster transitions indicating erroneous states. However, in other cases, it may be necessary to further classify objects into one or more of four categories: (1) components of the mechanical equipment controlled by system 112, (2) one or more workpieces on which the mechanical equipment operates, and (3) other foreign objects, including people, that may move in unpredictable ways and may be harmed by the mechanical equipment. Final classification of people and other unknown foreign objects may or may not be necessary. It may be necessary to definitively identify components of the mechanical equipment because, by definition, these components will always be in a state of "collision" with the mechanical equipment itself, and therefore, if these components are detected and incorrectly classified, it will cause the system to erroneously stop the mechanical equipment. Similarly, mechanical equipment typically comes into contact with workpieces, but contact between mechanical equipment and people is generally dangerous. Therefore, analysis module 342 should be able to distinguish between workpieces and unknown foreign objects, especially people.

[0049] The components of the mechanical equipment itself can be classified through the optional background subtraction calibration steps described above. When the mechanical equipment changes shape, the components can be identified and classified, for example, by providing the analysis module 342 with information about these components (e.g., as a scalable 3D representation) and, in some cases (e.g., an industrial robot), by providing an immediate source of information about the state of the mechanical equipment. The analysis module 342 can be "trained" under the observation of the sensor 102 by operating the mechanical equipment, conveyor, etc., in isolation, allowing the analysis module 342 to learn the precise operating area resulting from performing all actions and postures. The analysis module 342 can then classify the obtained spatial area as occupied.

[0050] Conventional computer vision techniques can be used to enable the analysis module 342 to distinguish between workpieces and people. These include deep learning, a branch of machine learning designed to use higher levels of data abstraction. Among the most successful deep learning algorithms are convolutional neural networks (CNNs) and, more recently, recurrent neural networks (RNNs). However, this technique is typically used in situations where accidental misidentification of a human as a non-human does not pose a safety hazard. To use such a technique in the current environment, several modifications may be necessary. First, machine learning algorithms can often be tuned to be biased towards error affirmation or error negation (e.g., logistic regression can be tuned for high specificity and low sensitivity). In this case, error affirmation does not pose a safety hazard—if the robot misidentifies a workpiece as a human, it will react conservatively. Furthermore, multiple algorithms or neural networks based on different image attributes can be used to improve the discriminative power, which is crucial for achieving a sufficient level of safety reliability. A particularly valuable source of discriminative power is obtained by using sensors that provide both 3D and 2D image data of the same object. If any technique identifies an object as a human, that object will be considered a human. Sufficient reliability can be achieved by using multiple techniques or machine learning algorithms, all tuned to be biased towards error affirmation rather than error negation. Furthermore, multiple images can be tracked over time to further improve reliability—again, each object can be treated as human until there are enough recognitions to identify it as non-human to achieve the reliability metric. Essentially, this differential algorithm identifies things that are definitively not human rather than humans.

[0051] Besides combining classification techniques, artifacts can also be identified entirely without relying on any type of human classification. One approach is to configure the system by providing a model of the artifacts. For example, the "learning" step in system configuration could simply provide an image or key features of the artifact to analysis module 342, which searches for matching configurations in spatial graph 345, or alternatively, it could involve training a neural network to automatically classify artifacts in the spatial graph in the same way. In either case, only objects that precisely match the stored model are considered artifacts, while all other objects are considered humans.

[0052] Another suitable approach is to designate a specific area within the workspace, as shown in spatial diagram 345, into which the workpiece will enter (e.g., the top of a conveyor belt). Only objects entering the workspace at this location meet the criteria for being considered workpieces. The workpiece can then be modeled and tracked from its entry into the workspace to its exit. When a monitored machine, such as a robot, is processing the workpiece, the control system 112 ensures that the workpiece moves only in a manner consistent with the expected motion of the robot's end effector. Known devices such as conveyor belts can also be modeled in this way. Humans can be prevented from entering the work cell in a way that resembles a workpiece, such as sitting on a conveyor belt.

[0053] All these technologies can be used individually or in combination, depending on design requirements and environmental constraints. However, in all cases, there may be instances where the analysis module 342 loses its ability to track whether an identified object is a workpiece. In these situations, the system should return to a safe state. An interlock can then be placed in a safe area of ​​the workspace where workers can confirm the absence of foreign objects, allowing the system to resume operation.

[0054] In some cases, foreign objects enter the workspace but should subsequently be ignored or treated as workpieces. For example, a stack of boxes that were not present in the workspace at the time of configuration may later be placed there. This type of situation will become more common as flexible systems replace fixed protective devices and can be addressed by providing a user interface (e.g., displayed on display 320 or on a device that wirelessly communicates with control system 112) that allows workers to designate new objects as safe for future interactions. Of course, analysis module 342 and control routines 350 can still prevent collisions between machinery and new objects, but the new objects will not be treated as potentially human-like objects that may move toward the machinery, thus allowing the system to handle them in a less conservative manner.

[0055] 3. Generate control output At this stage, the analysis module 342 has identified all objects in the monitored area 100 that must be considered for safety purposes. Given this data, various actions can be taken and control outputs generated. During static calibration or in the default configuration with the workspace unattended, the spatial map 345 can be used by humans to assess sensor coverage, the configuration of deployed machinery, and the potential for unwanted human-machine interaction. Even without enclosures or fixed protection, the overall workspace layout can be improved by guiding or encouraging people through areas marked as safe zones as described above and away from areas with poor sensor coverage.

[0056] When certain conditions are detected, in response to analysis module 342, control routine 350 can generate control signals for mechanical devices (such as robots) operating within workspace 100. This control can be binary, indicating safe or unsafe conditions, or more complex, indicating which actions are safe and unsafe. The simplest type of control signal is a binary signal indicating whether an intrusion into an occupied or potentially occupied volume has been detected in a specific area. In the simplest case, a single intrusion area exists, and control system 112 provides a single output indicating the intrusion. This output can be transmitted, for example, via I / O port 327 to a complementary port on the controlled mechanical device to stop or limit its operation. In more complex cases, multiple areas are monitored separately, and control routine 350 sends digital outputs to the target mechanical device via I / O port 327 or transceiver 325 over a network (e.g., using Internet Protocol or other suitable addressing schemes).

[0057] Another condition that can be monitored is the distance between any object in the workspace and the machine, comparable to the output of a 2D proximity sensor. This can be converted to a binary output by establishing a proximity threshold, below which the output should be declared. The system may also need to record and provide the position and extent of the object closest to the machine. In other applications, such as safety systems for collaborative industrial robots, the required control output may include the position, shape, and extent of all objects observed within the area covered by sensor 102.

[0058] 4. Safety action constraints and dynamic determination of safe zones ISO 10218 and ISO / TS 15066 describe speed and spacing monitoring as a safety function that enables collaboration between industrial robots and workers. Risk reduction is achieved by maintaining a protective spacing between the worker and the robot during robot movement. This protective spacing is calculated using information including the position and movement of both the robot and worker, the robot's stopping distance, measurement uncertainty, system delay, and system control frequency. The robot system stops when the calculated spacing decreases below the protective spacing value. This methodology can be extended from industrial robots to the field of mechanical equipment.

[0059] For simplicity, the following discussion focuses on dynamically defining the safety zone around a robot operating within workspace 100. However, it should be understood that the techniques described herein are applicable not only to multiple robots but also to any form of mechanical device that may pose a danger when too close and has a minimum safe distance that may vary over time and by the specific actions performed by the machine. As described above, the sensor array acquires sufficient image information to characterize the position and extent of the robot and all relevant objects in the area surrounding the robot in 3D form during each analysis cycle. (Each analysis cycle includes image capture, frame buffer refresh, and computational analysis; therefore, although the time period of the analysis or control cycle is short enough for effective real-time monitoring, it involves many computer clock cycles.) The analysis module 342 uses this information, along with instantaneous information about the robot's current state in each cycle, to determine instantaneous, current safe action constraints for the robot's motion. These constraints can be communicated to the robot directly through the analysis module 342 or through control routine 350, or through transceiver 325 or I / O port 327.

[0060] refer to Figure 4A conceptual explanation of the system organization and operation best facilitates understanding of the system's operation. As described above, sensor array 102 monitors the workspace 400 including robot 402. The robot's movement is controlled by a conventional robot controller 407, which may be part of the robot itself or separate from it; for example, a single robot controller may issue commands to multiple robots. The robot's activities may primarily involve a manipulator, whose movement is coordinated by the robot controller 407 using joint commands to manipulate the manipulator joints to achieve the desired movement. An object monitoring system (OMS) 410 acquires information about objects from sensor 102 and uses this information to identify relevant objects in workspace 400. OMS 410 communicates with robot controller 407 via any suitable wired or wireless protocol. (In industrial robots, control electronics are typically located in an external control box. However, for robots with built-in controllers, OMS 410 communicates directly with the robot's onboard controller.) Using information obtained from the robot (typically sensor 102), OMS 410 determines the robot's current state. Therefore, OMS 410 determines the safety motion constraints for robot 402 given the robot's current state and all identified relevant objects. Finally, OMS 410 communicates the safety motion constraints to robot 407. (For further understanding, see [link to documentation]). Figure 3 The functions of OMS 410 are performed in the control system 112 by the analysis module 342, and in some cases by the control routine 350. 4.1 Identify relevant objects Sensor 102 provides real-time image information analyzed by object analysis module 415 at a fixed frequency in the manner described above; specifically, in each cycle, object analysis module 415 identifies the precise 3D position and extent of all objects in workspace 400 that are within the robot's reach or can move within the robot's reach at a conservatively expected speed. If not all relevant volumes are within the common field of view of sensor 102, OMS 410 can be configured to determine and indicate the position and extent of all stationary objects (or a conservative superset of these objects) within that area and / or verify whether other protective techniques have been used to prevent entry into unmonitored areas.

[0061] 4.2 Determine robot status The Robot State Determination Module (RSDM) 420 responds to data from sensor 102 and signals from robot 402 and / or robot controller 407 to determine the robot's instantaneous state. Specifically, RSDM 420 determines the pose and position of robot 402 within workspace 400; this can be achieved using data from sensor 102, signals from the robot and / or its controller, or some combination of these sources. RSDM 420 can also determine the instantaneous velocity of robot 402 or any of its attachments; furthermore, it may be necessary to know the robot's instantaneous joint accelerations or torques, or planned future trajectories, in order to determine safe motion constraints for subsequent cycles, as described below. Typically, this information comes from robot controller 407, but in some cases, it can be inferred directly from images recorded by sensor 102, as described below.

[0062] For example, this data can be provided by robot 402 or robot controller 407 via a security-level communication protocol that provides access to safety-level data. The robot's 3D pose can then be determined by combining the provided joint positions with a static 3D model of each link to obtain the 3D shape of the entire robot 402.

[0063] In some cases, the robot can provide an interface to obtain non-safety-grade joint positions, in which case the joint positions can be verified against images from sensor 102 (e.g., using safety-grade software). For example, the received joint positions can be combined with a static 3D model of each link to generate a 3D model of the entire robot 402. This 3D image can be used to remove any objects belonging to the robot itself from the sensing data. If the joint positions are correct, this will completely eliminate all object data attributable to the robot 402. However, if the joint positions are incorrect, the true position of the robot 402 will differ from the model, and some detected parts of the robot will not be removed. These points will then appear as foreign objects in the new cycle. In the previous cycle, the joint positions can be assumed to be correct, because otherwise the robot 402 would have been stopped. Since the robot's base joints do not move, at least one branch point must be close to the robot. The detection of an unexpected object close to the robot 402 can then be used to trigger an error condition, which will cause the control system 112 (see...) Figure 1The robot 402 is then switched to a safe state. Alternatively, sensor data can be used with relevant algorithms to identify the robot's position, such as those described above in the registration section, and this detected position can be compared with the joint positions reported by the robot. If the joint position information provided by robot 402 has been verified in this way, it can be used to verify joint velocity information, which can then be used to predict future joint positions. If these positions are inconsistent with the previously verified actual joint positions, the program can similarly trigger error conditions. These techniques allow data to be generated using non-safety-grade interfaces, which can then be used to perform other safety functions.

[0064] Finally, the RSDM 420 can be configured to determine the robot's joint states using only the image information provided by sensor 102, without any information provided by sensor 102 to robot 402 or controller 407. Given a model of all the robot's links, the RSDM 420 can use any of several common, well-known computer vision techniques to register the model with the sensor data, thereby determining the position of the modeled object in the image. For example, the ICP algorithm (as described above) minimizes the difference between two 3D point clouds. ICP can often efficiently provide a locally optimal solution, so if the approximate position is known, ICP can be used accurately. This is the case if the algorithm runs every cycle, because robot 402 cannot move away from its previous position. Therefore, a globally optimal registration technique is not required, and it may not be efficient enough to run in real time. Given the joint positions identified by the registration algorithm, instantaneous joint velocities can then be determined using digital filters such as Kalman filters or particle filters.

[0065] These image-based monitoring technologies typically rely on running in each system cycle and assuming the system was in a safe state in the previous cycle. Therefore, tests can be performed when the robot 402 starts—for example, to confirm that the robot is in a known, pre-configured "original" position and that all joint velocities are zero. Automated equipment often has a set of tests performed by the operator at fixed intervals, such as when the equipment starts or when a shift changes. Reliable condition analysis typically requires an accurate model of each robot link. This model can be obtained a priori, for example, from 3D CAD files provided by the robot manufacturer or 3D CAD files generated by industrial engineers for a specific project. However, such models may not be available, at least not for the robot and all possible attachments it may have.

[0066] In this scenario, the RSDM 420 can, for example, create the model itself using sensor 102. This can be done in a separate training mode, where robot 402 performs a set of movements, such as movements intended for use in a given application and / or a set of movements designed to provide sensor 102 with an appropriate view of each link. Some basic information about the robot, such as the length and axis of rotation of each link, can be provided a priori, but is not required. During this training mode, the RSDM 420 generates a 3D model of each link, including all necessary attachments. The RSDM 420 can then use this model in conjunction with the sensor images to determine the robot's state.

[0067] 4.3 Determine safety action constraints In traditional axis-limiting and speed-limiting applications, industrial engineers calculate which actions are safe for the robot given its planned trajectory and workspace layout—completely prohibiting certain areas of the robot's range of motion and limiting its speed in other areas. These limitations assume a fixed, static working environment. Here, we focus on a dynamic environment where objects and people move around and change positions; therefore, the Safe Motion Determination Module (SADM) 425 calculates safe actions in real time based on all sensed relevant objects and the current state of the robot 402, and these safe actions can be updated every cycle. To be considered safe, actions should ensure that the robot 402 does not collide with any stationary objects and also ensures that the robot 402 does not come into contact with a person who may be moving towards the robot. Since the robot 402 has a certain maximum possible deceleration, the controller 407 should be instructed to begin sufficiently reducing the robot's speed in advance to ensure that it can reach a complete stop before contact.

[0068] One way to achieve this is to proportionally adjust the robot's maximum speed (i.e., the speed of the robot itself or any of its attachments) to the minimum distance between any point on the robot and any point on a sensed object in the relevant group to be avoided. The robot is allowed to run at its maximum speed when the nearest object is further away from a certain threshold distance, beyond which collisions are not a concern, and the robot comes to a complete stop if the object is within a certain minimum distance. Sufficient margin can be added to the specified distance to accommodate the movement of relevant objects or people toward the robot at a certain maximum practical speed. This is in... Figure 5As shown in the diagram, the SADM 425 generates an outer envelope, or 3D region 502, around the robot 504 through calculation. Outside of this region 502, all movements of the human P are considered safe because they do not bring the human close enough to the robot 504 to cause danger during the operating cycle. Detection of any part of the human P's body within a second 3D region 508 defined within region 502 is recorded by the SADM 425, but the robot 504 is allowed to continue operating at full speed. If any part of the human P exceeds a threshold in region 508 but remains outside the inner danger zone 510, the robot 504 is signaled to operate at a slower speed. If any part of the human P enters danger zone 510—or if a model based on the human's movements predicts entry into danger zone 510 in the next cycle—the robot 504 stops operating. These regions can be updated as the robot 504 moves within the environment.

[0069] An improvement to the SADM 425 is that the maximum speed is controlled proportionally to the square root of the minimum distance traveled. This reflects the fact that, with constant deceleration, a speed that varies proportionally to the square root of the distance traveled results in smoother and more efficient operation while remaining equally safe. A further improvement to the SADM 425 is that the maximum speed is adjusted proportionally to the shortest possible collision time—that is, timely projection of the robot's current state forward, projection of intrusions onto the robot's trajectory, and identification of the nearest potential collision. The advantage of this improvement is that the robot moves away from obstacles faster than it moves towards them, maximizing throughput while still maintaining safety correctly. Since the robot's future trajectory depends not only on its current speed but also on subsequent commands, the SADM 425 can consider all points that the robot 402 can reach within a certain reaction time, given the robot's current joint position and speed, and generate control signals to be issued based on the minimum collision time of any of these states. Another further improvement to the SADM 425 is that the entire planned trajectory of the robot is considered when performing this calculation, rather than just instantaneous joint velocities. Furthermore, the SADM 425 can alter the robot's trajectory via the robot controller 407, rather than simply changing the maximum speed along the trajectory. It can select a trajectory from a set of fixed trajectories that reduces or eliminates potential collisions, and can even generate new trajectories in real time.

[0070] While not necessarily violating safety regulations, collisions with static elements in the workspace are generally undesirable. The relevant object group can include all objects in the workspace, including static backgrounds (e.g., walls and tables) and moving objects (e.g., workpieces and workers). Based on prior configuration or runtime detection, sensor 102 and analysis module 342 may be able to infer which objects might be moving. In this case, any of the algorithms described above can be improved to allow for additional margins to account for potentially moving objects, but those margins for objects known to be static should be eliminated to avoid unnecessarily reducing throughput while still automatically eliminating the possibility of collisions with static parts of the work cell.

[0071] In addition to simply allowing margin to account for the maximum speed of a potential moving object, state estimation techniques based on information detected by the sensor system can also be used to project the motion of people and other objects forward in a timely manner, thereby expanding the control options available in control routine 350. For example, skeletal tracking technology can be used to identify the moving limbs of a detected person and limit potential collisions based on human attributes and estimated motion, such as that of a person's arm rather than the entire person.

[0072] 4.4 Communicating safety action constraints to robots Safety motion constraints identified by the SADM 425 can be communicated to the robot controller 407 via the robot communication module 430 by the OMS 410 in each cycle. As mentioned above, the communication module can correspond to the I / O port 327 interface with a complementary port on the robot controller 407, or it can correspond to the transceiver 325. Most industrial robots provide various interfaces for use with external devices. A suitable interface should operate with low latency at least at the system's control frequency. The interface can be configured to allow programming of the robot and operation as usual, with maximum speed transmitted via the interface. Alternatively, some interfaces allow the transmission of trajectories in the form of waypoints. Using this type of interface, the expected trajectory of the robot 402 can be received and stored within the OMS 410, and then waypoints closer or further away can be generated based on the safety motion constraints. Similarly, interfaces that allow input of target joint torques can be used to drive trajectories calculated accordingly. These types of interfaces can also be used when the SADM 425 selects a new trajectory or modifies a trajectory based on safety motion constraints.

[0073] Similar to the interface used to determine the robot's state, it might be sufficient if the robot 402 supports a safety-level protocol that provides real-time access to relevant safety-level control inputs. However, if a safety-level protocol is unavailable, other safety-level software on the system can be used to ensure the entire system remains safe. For example, if the robot operates according to a communicated safety action, the SADM 425 can determine the robot's expected speed and position. The SADM 425 then determines the robot's actual state as described above. If the robot's actions do not correspond to the expected actions, the SADM 425 typically uses an emergency stop signal to transition the robot to a safe state. This effectively implements a real-time safety-level control scheme without requiring a real-time safety-level interface beyond a safety-level stop mechanism.

[0074] In some situations, a hybrid system may be optimal—many robots have digital inputs that can be used to maintain a safety-monitored stop. For example, a communication protocol for variable speeds might be needed when an intruder is relatively far from the robot, but a digital safety-monitored stop could be used when the robot must come to a complete stop, such as when an intruder approaches the robot.

[0075] Some embodiments of the present invention have been described above. However, it is clearly stated that the present invention is not limited to these embodiments; rather, additions and modifications to the content explicitly described herein are also included within the scope of the present invention.

Claims

1. A safety system for enforcing the safe operation of mechanical equipment, said mechanical equipment performing activities in a three-dimensional (3D) workspace, said system comprising: Multiple sensors distributed around a workspace, each associated with a pixel grid, are used to record images of a portion of the workspace within the sensor's field of view, the multiple portions of which together cover the entire workspace; Computer memory for storing (i) multiple images from sensors, (ii) a model of the mechanical device and its permitted movements during the activity, and (iii) safety protocols specifying speed limits for the mechanical device approaching a person and minimum distances between the mechanical device and the person. as well as The processor is configured as follows: A 3D spatial representation of the workspace is computationally generated from stored images; Identify a first 3D region of the workspace, which corresponds to the space occupied by the mechanical equipment within the workspace, the space being expanded by a 3D envelope surrounding the mechanical equipment, the envelope spanning permissible movements according to a stored model; A second 3D region of the workspace is identified, which corresponds to the space that has been occupied and may be occupied by a person in the workspace. This space is expanded by a 3D envelope around the person, which corresponds to the person's expected movement in the workspace within a predetermined future time. The space that may be occupied includes any space that is obscured from the sensor's field of view when there is a situation where obscured space may be occupied by a person. and Based on the proximity between the first and second 3D regions, the movement of mechanical equipment is restricted according to a safety protocol.

2. The security system according to claim 1, wherein, The workspace is computationally represented as multiple voxels.

3. The security system according to claim 1, wherein, The processor is configured to identify the region corresponding to the mechanical device based at least in part on state data provided by the mechanical device.

4. The security system according to claim 3, wherein, The status data is security-grade and provided through a security-grade communication protocol.

5. The security system according to claim 3, wherein, The status data is not security-grade; it is verified by information received from the sensor.

6. The security system according to claim 3, wherein, The state data is verified by constructing a robot model, removing objects detected within the model, and stopping the machine if any remaining objects are adjacent to it.

7. The security system according to claim 3, wherein, The status data is verified by using computer vision to identify the position of the mechanical equipment and comparing it with the reported position.

8. The security system according to claim 3, wherein, The status data is determined by the sensor without any interface with the mechanical equipment.

9. The security system according to claim 1, wherein, The first 3D region is divided into multiple nested, spatially distinct 3D sub-regions.

10. The security system according to claim 9, wherein, The overlap between the second 3D region and each sub-region alters the operation of the mechanical equipment to varying degrees.

11. The security system according to claim 1, wherein, The processor is also configured to identify the workpiece being processed by the mechanical equipment, and to regard the workpiece as part of the mechanical equipment when identifying the first 3D region.

12. The security system according to claim 1, wherein, The processor is configured to dynamically control the maximum speed of the mechanical equipment to prevent contact between the mechanical equipment and a person except when the mechanical equipment is stopped.

13. The security system according to claim 1, wherein, The processor is configured to calculate the minimum possible time of collision based on the proximity.

14. The security system according to claim 1, wherein, The processor responds to the sensor’s real-time monitoring of the workspace and is configured to change the operation of the mechanical equipment in response to an intrusion into the workspace detected by the sensor.

15. The safety system according to claim 1, wherein the mechanical device is at least one robot.

16. A method for safely operating mechanical equipment in a three-dimensional (3D) workspace, the method comprising the following steps: The workspace is monitored by multiple sensors distributed around it, each sensor being associated with a pixel grid to record an image of a portion of the workspace within the sensor's field of view, with multiple portions of the workspace partially overlapping each other; The sensors are registered with each other so that the images acquired by the sensors collectively represent the workspace; The computer memory stores (i) multiple images from sensors, (ii) a model of the mechanical equipment and its permitted movements during the activity, and (iii) a safety protocol specifying the speed limit of the mechanical equipment approaching a person and the minimum distance between the mechanical equipment and the person. A 3D spatial representation of the workspace is computationally generated from stored images; The first 3D region of the workspace is computationally identified, which corresponds to the space occupied by the mechanical equipment within the workspace. This space is expanded by a 3D envelope around the mechanical equipment, which spans permissible movements according to a stored model. The second 3D region of the workspace is computationally identified, which corresponds to the space that has been occupied and may be occupied by a person in the workspace. This space is expanded by a 3D envelope around the person, which corresponds to the person's expected movement in the workspace within a predetermined future time. The space that may be occupied includes any space that is obscured from the sensor's field of view when there is a situation where obscured space may be occupied by a person. and Based on the proximity between the first and second 3D regions, the movement of mechanical equipment is restricted according to a safety protocol.

17. The method according to claim 16, wherein, The workspace is computationally represented as multiple voxels.

18. The method according to claim 16, wherein, The first 3D region is divided into multiple nested, spatially distinct 3D sub-regions.

19. The method according to claim 18, wherein, The overlap between the second 3D region and each sub-region alters the operation of the mechanical equipment to varying degrees.

20. The method of claim 16, further comprising the step of: The workpiece being processed by the mechanical equipment is identified, and the workpiece is processed as part of the mechanical equipment when the first 3D region is computationally identified.

21. The method according to claim 16, wherein, Limiting the activity of the mechanical equipment includes controlling the maximum speed of the mechanical equipment in proportion to the square root of the proximity.

22. The method according to claim 16, wherein, Limiting the movement of the mechanical equipment includes calculating the minimum possible time of collision based on the proximity.

23. The method of claim 16, further comprising the step of: The operation of the mechanical equipment is altered in response to an intrusion into the workspace detected by the sensor.

24. The method of claim 16, wherein the mechanical device is at least one robot.

25. A safety system for identifying safe areas in a three-dimensional (3D) workspace, the workspace including mechanical equipment performing activities, the system comprising: Multiple sensors distributed around a workspace, each sensor including a pixel grid, are used to record an image of a portion of the workspace within the sensor's field of view, the multiple portions of the workspace collectively covering the entire workspace; Computer memory for storing (i) multiple images from sensors, (ii) a model of the mechanical device and its permitted movements during the activity, and (iii) safety protocols specifying speed limits for the mechanical device approaching a person and minimum distances between the mechanical device and the person. as well as The processor is configured as follows: A 3D spatial representation of the workspace is computationally generated from stored images; Over time, the space occupied by mechanical equipment within the workspace is identified and monitored as a representation of the 3D mechanical equipment region, and a 3D envelope region spanning the mechanical equipment that allows movement is generated around the mechanical equipment region based on the stored model. Semantically distinguish between workpieces that may be associated with mechanical equipment and other objects in the workspace that are not associated with them, and identify the interaction between mechanical equipment and workpieces in the workspace; In response to the identified interactions, the 3D mechanical equipment region is updated to include the workpiece, and the 3D envelope region is updated based on the stored model and the updated mechanical equipment region. and According to the security protocol, a 3D security zone is computationally generated around the 3D mechanical equipment area with each update.

26. The security system of claim 25, wherein the processor is further configured to: Identify the areas within the volume that correspond to the space already occupied or potentially occupied by people within the workspace; and Based on the proximity between the robot's area and areas already occupied by humans, the robot's activities are restricted according to safety protocols.

27. The security system according to claim 26, wherein, The area occupied by the person is expanded by a 3D envelope around the person, the 3D envelope corresponding to the person's expected movement within the workspace at a predetermined future time.

28. The security system according to claim 25, wherein, The processor is also configured to identify items in the workspace other than the robot and the workpiece in the image, and the processor recognizes, like a human, the detected items as not being part of the robot or the workpiece and not being otherwise identified items.

29. The security system according to claim 28, wherein, The processor is configured to detect items within the workspace in the image and receive externally provided identifiers for those items. The processor recognizes, like a human, that the detected items are not part of the robot or workpiece and that no externally provided identifiers have been received for the detected items.

30. The security system according to claim 25, wherein, The workspace is computationally represented as multiple voxels.

31. The safety system of claim 25, wherein the mechanical device is at least one robot.

32. A method for safely operating mechanical equipment in a three-dimensional (3D) workspace, the method comprising the following steps: The workspace is monitored by multiple sensors distributed around it, each sensor including a pixel grid for recording an image of a portion of the workspace within the sensor's field of view, with multiple portions of the workspace partially overlapping each other; The sensors are registered with each other so that the images acquired by the sensors collectively represent the workspace; The computer memory stores (i) multiple images from sensors, (ii) a model of the mechanical equipment and its permitted movements during the activity, and (iii) a safety protocol specifying the speed limit of the mechanical equipment approaching a person and the minimum distance between the mechanical equipment and the person. A 3D spatial representation of the workspace is computationally generated from stored images; The space occupied by mechanical equipment in the workspace is identified and monitored over time as a representation of the 3D mechanical equipment area, and a 3D envelope region spanning the mechanical equipment that allows movement is generated around the mechanical equipment area based on the stored model. Semantically distinguish between workpieces that may be associated with mechanical equipment and other objects in the workspace that are not associated with them, and identify the interaction between mechanical equipment and workpieces in the workspace; In response to the recognized interaction, the 3D mechanical equipment region is computationally updated to include the workpiece based on the stored model and the updated mechanical equipment region, and the 3D envelope region is computationally updated. and According to the security protocol, a 3D security zone is computationally generated around the 3D mechanical equipment area with each update.

33. The method of claim 32, further comprising the following step: Identify the area within the volume that corresponds to the space already occupied by people in the workspace; and Based on the proximity between the robot's area and areas already occupied by humans, the robot's activities are restricted according to safety protocols.

34. The method of claim 33, further comprising the following steps: The area occupied by a person is expanded by a 3D envelope around the person, which corresponds to the person's expected movement within the workspace at a predetermined future time.

35. The method of claim 32, further comprising the following step: (i) Identify items in the workspace other than machinery and workpieces in an image, and (ii) identify, like a person, items that are not part of machinery or workpieces and are not otherwise identified.

36. The method of claim 35, further comprising the step of: (i) Detecting items in the workspace in an image and receiving externally provided identification, and (ii) identifying, like a human, that the detected items are not part of machinery or workpieces and that no externally provided identification has been received for the detected items.

37. The method according to claim 32, wherein, The workspace is computationally represented as multiple voxels.

38. The method of claim 32, wherein the mechanical device is at least one robot.

Images