Assurance-driven architecture for multi-modal sensing in human- cyber-physical systems

Abstract

Embodiments include systems and methods for managing safety in a multi-sensor environment. Aspects include receiving data from a first localization system and a second localization system to identify a last known consistent location or an inconsistent area for each of a set of objects detected in an environment, identifying one or more hazards within the environment, and determining a maximum speed for each of the set of objects. Aspect also includes evaluating whether any of the set of objects are capable of entering a hazardous area of the one or more hazards within a threshold time and based on a determination that an object of the set of objects is capable of entering the hazardous area of the one or more hazards within the threshold time, performing a safety action.

Description

ASSURANCE-DRIVEN ARCHITECTURE FOR MULTI -MODAL SENSING IN HUMAN-CYBER-PHYSICAL SYSTEMSBACKGROUND

[0001] The present disclosure relates to human-cyber-physical systems, specifically to assurance-driven architectures for multi-modal sensing to ensure safety and reliability in environments where humans and cyber-physical systems interact.

[0002] In modern manufacturing environments, it is necessary to ensure worker safety in settings where human-cyber-physical systems (HCPS) operate. These systems, which include manufacturing robots and mobile ground robots, share physical spaces with human participants, necessitating robust safety measures to prevent harm. Traditionally, safety has been maintained by physically fencing off dangerous zones, such as areas with robots and heavy machinery. However, this approach can limit the flexibility and efficiency of HCPS, as it restricts human access to certain areas and does not fully leverage the potential of these dynamic systems.

[0003] Current systems with significant safety requirements often rely on multiple sensor modalities to prevent a single point of failure. However, existing solutions face substantial challenges, such as sensor heterogeneity, calibration, synchronization, and computational complexity. These challenges can lead to uncertainties and potential system failures, particularly when sensor data is processed asynchronously or when sensor failures occur. Moreover, traditional sensor fusion techniques may not adequately address safety assurance levels or resilience to outliers and missing data. As a result, there is a pressing need for a more reliable and comprehensive approach to ensure the safety and reliability of HCPS in manufacturing environments.SUMMARY

[0004] Embodiments of the present disclosure are directed to computer- implemented methods for managing safety in a multi-sensor environment. According to an aspect, a computer-implemented method includes receiving data from a first localization system and a second localization system to identify a last known consistent location or an inconsistent area for each of a set of objects detected in an environment, identifying one or more hazards within theenvironment, and determining a maximum speed for each of the set of objects. The method also includes evaluating whether any of the set of objects are capable of entering a hazardous area of the one or more hazards within a threshold time and based on a determination that an object of the set of objects is capable of entering the hazardous area of the one or more hazards within the threshold time, performing a safety action.

[0005] Embodiments also include computer systems and computer program products for managing safety in a multi-sensor environment.

[0006] Additional technical features and benefits are realized through the techniques of the present disclosure. Embodiments and aspects of the disclosure are described in detail herein and are considered a part of the claimed subject matter. For a better understanding, refer to the detailed description and to the drawings.BRIEF DESCRIPTION OF THE DRAWINGS

[0007] FIG. 1 A depicts a schematic block diagram illustrating an environment with a plurality of objects and sensors in accordance with an embodiment.

[0008] FIG. IB depicts a block diagram of a system for ensuring localization consistency in multi-sensor environment in accordance with an embodiment.

[0009] FIG. 2 depicts an illustration of a correspondence between a first identification and second identification of a set of objects in accordance with an embodiment.

[0010] FIG. 3 depicts a graph illustrating the spatial relationship between consistent and inconsistent object locations in a multi-sensor environment in accordance with an embodiment.

[0011] FIG. 4 depicts a flow chart diagram depicting a method for ensuring localization consistency and safety in a multi-sensor environment in accordance with an embodiment.

[0012] FIG. 5 depicts illustration of a scenario for determining when an object is capable of entering a hazardous area in accordance with an embodiment.

[0013] FIG. 6 depicts a flow chart diagram depicting a method for managing safety in a multi-sensor environment in accordance with an embodiment.

[0014] FIG. 7 depicts a block diagram of a processing system in accordance with an embodiment.

[0015] In the accompanying figures and following detailed description of the disclosed embodiments, the various elements illustrated in the figures are provided with three digit reference numbers. In some instances, the leftmost digits of each reference number corresponds to the figure in which its element is first illustrated.DETAILED DESCRIPTION

[0016] In modern manufacturing environments, ensuring worker safety is important, particularly in environments where human-cyber-physical systems (HCPS) operate. These systems, which include manufacturing robots and mobile ground robots, share physical spaces with human participants, necessitating robust safety measures to prevent harm. Traditionally, safety has been maintained by physically fencing off dangerous zones, such as areas with robots and heavy machinery. However, this approach can limit the flexibility and efficiency of HCPS, as it restricts human access to certain areas and does not fully leverage the potential of these dynamic systems.

[0017] Current systems with significant safety requirements often rely on multiple sensor modalities to prevent a single point of failure. However, existing solutions face significant challenges, such as sensor heterogeneity, calibration, synchronization, and computational complexity. These challenges can lead to uncertainties and potential system failures, particularly when sensor data is processed asynchronously or when sensor failures occur. Moreover, traditional sensor fusion techniques may not adequately address safety assurance levels or resilience to outliers and missing data.

[0018] Disclosed herein is an assurance-driven architecture for multi-modal sensing in human-cyber-physical systems that prioritizes safety assurance by integrating multiple sensor modalities to provide robust and reliable localization and classification of objects within the environment. By leveraging diverse sensor technologies, the system enhances the accuracy andprecision of object detection and tracking, while also ensuring resilience to sensor failures and environmental variations. The architecture includes components such as a consistency checker and a hazard interpreter, which work together to continuously monitor and evaluate the safety of the environment, enabling dynamic responses to potential hazards. This novel approach not only improves safety but also enhances the operational efficiency and flexibility of HCPS in manufacturing settings.

[0019] Referring now to FIG. 1A, an environment 100 having a plurality of objects 102- 1, 102-2, 102-3, 102-4, and 102-5 (referred to collectively herein as objects 102), as well as multiple sensors 112 and sensors 122 is shown. The environment 100 is designed to illustrate the spatial arrangement and interaction between the objects 102 and sensors 112, 122. In exemplary embodiments, sensors 112 are a first type of sensor that is configured to monitor the location of the objects 102 in the environment. For example, the first sensors 112 may be video cameras that are configured to monitor a portion of the environment 100. Likewise, sensors 122 are a second type of sensor that is configured to monitor the location of the objects 102 in the environment. For example, the second sensor 122 may be radio frequency sensors that are configured to detect the presence of an object 102 in the environment 100.

[0020] Referring now to FIG. IB, a system 110 is designed to ensure localization consistency in a multi-sensor environment in accordance with an embodiment is shown. In exemplary embodiments, the system 110 integrates various components to achieve robust and reliable localization and safety assurance in environments where human-cyber-physical systems (HCPS) operate.

[0021] In exemplary embodiments, the sensors 112 and 122 are configured to monitor an environment 100 and provide the necessary data for localization of objects 102 in the environment 100. The sensors 112 are typically visual sensors, such as video cameras 112, that capture high-resolution images or video feeds of the environment. These sensors are strategically placed to cover important areas and provide comprehensive visual data of the environment 100. The sensors 122, on the other hand, are non- visual sensors, such as radio frequency sensors 122, that detect the presence and location of objects through alternativemeans. This diversity in sensor types allows the system to gather a wide range of data, enhancing the robustness of the localization process.

[0022] In exemplary embodiments, the first localization system 114 processes localization data from the sensors 112. The first localization system 114 is configured to analyze the visual data to identify and track objects within the environment, generating a first set of location data. In exemplary embodiments, the first localization system 114 is adept at handling the specific characteristics of visual data, extracting meaningful information regarding the position and movement of objects. In one embodiment, the movement of an object is only tracked by the changes of a position of an object over time, rather than real-time movement tracking. In exemplary embodiments, the first localization system 114 is also configured to classify each object based on the captured visual data. For instance, the system can distinguish between different types of machinery, such as identifying a robotic arm versus a conveyor belt, by analyzing their shapes, sizes, and movement patterns. It can also classify human participants based on learned features which distinguish humans from other classes such as robots or ground vehicles. Additionally, the system can differentiate between stationary objects, like storage racks, and mobile objects, such as forklifts, by assessing their motion characteristics over time. This classification capability enhances the system's ability to monitor and manage the environment effectively, ensuring that each object is accurately identified according to its specific attributes.

[0023] In exemplary embodiments, the second localization system 124 complements the first localization system 114 by processing data from the sensors 122. The design of the second localization system 124 focuses on determining the location of objects using non- visual data, such as radio frequency signals. Techniques like triangulation and / or Ultra- Wideband (UWB) technology, which uses Time Difference of Arrival (TDoA), are employed by the second localization system 124 to achieve precise location tracking. This system offers an alternative perspective, particularly useful in scenarios where visual data may be obstructed or insufficient or where one of the two sensor modalities fail.

[0024] In exemplary embodiments, the consistency verification system 130 is configured to receive data streams from both the first localization system 114 and the second localizationsystem 124. The consistency verification system 130 compares the location data from both systems to ensure consistency. The consistency verification system 130 conducts detailed analyses to identify discrepancies between the reported locations, calculating the spatial distance between objects as represented in the different data streams. In exemplary embodiments, the consistency verification system 130 determines the last known consistent location for each object, providing a consistency status that indicates whether the location data is consistent or inconsistent.

[0025] In exemplary embodiments, the consistency verification specifications 132 are specifications that guide the consistency verification system 130 in the analysis process. These specifications define the logic and associated thresholds for determining location consistency, allowing the system to adapt to different operational requirements and environmental conditions. By adjusting these specifications, the system can fine-tune sensitivity to discrepancies and ensure that the localization process meets the desired accuracy and reliability standards. For example, the consistency verification specifications 132 may provide different threshold distances based on the classification of the object. For high-risk objects, such as mobile robots or heavy machinery, a stricter threshold may be set to ensure precise localization and minimize safety risks. Conversely, for low-risk objects, such as stationary storage racks, a more lenient threshold might be acceptable, allowing for minor discrepancies without triggering alerts. Additionally, the system can adjust thresholds based on environmental factors, such as the presence of obstacles or varying lighting conditions, to maintain consistent performance across diverse scenarios. These tailored preferences enable the system to maintain a balance between safety and operational efficiency, ensuring that the localization process is both reliable and adaptable to the specific needs of the environment.

[0026] In exemplary embodiments, the hazard detection system 140 utilizes the output from the consistency verification system 130 to assess potential safety risks. The hazard detection system 140 analyzes the consistency status and location data to identify hazardous situations, such as objects entering restricted areas or discrepancies that could indicate sensor failures. The hazard detection system 140 is equipped with algorithms that evaluate the potential impact of these hazards and determine appropriate actions to mitigate risks.

[0027] In exemplary embodiments, the hazard detection specifications 142 provide the hazard detection system 140 with guidelines for evaluating safety risks. These specifications include logic and associated thresholds that define acceptable risk levels, response times, and specific conditions that trigger safety actions. By configuring these specifications, the hazard detection system 140 can tailor its capabilities to align with the safety requirements of the environment. The environmental data 144 is additional information that the hazard detection system 140 uses to enhance the analysis. This data includes factors such as environmental conditions, operational parameters, and historical data on object movements. By incorporating this data, the system can improve the understanding of the environment and make more informed decisions regarding safety.

[0028] In exemplary embodiments, the hazard detection specification 142 and the consistency verification specification 132 are parameters that can be set by a safety engineer responsible for overseeing the environment. These preferences are designed to tailor the system's response to specific safety requirements and operational conditions within the environment. By configuring these preferences, the safety engineer can establish the criteria for identifying potential hazards and ensuring location consistency, thereby optimizing the system's performance. In exemplary embodiments, these preferences are not static, and they can be dynamically adjusted over time based on observed data and evolving conditions within the environment. As the system collects and analyzes data, the safety engineer can refine the preferences to enhance accuracy and reliability, ensuring that the system remains responsive to changes and continues to meet the desired safety standards. This adaptability allows the system to maintain a high level of safety assurance while accommodating the unique characteristics and demands of the environment.

[0029] In exemplary embodiments, the safety control system 160 is responsible for executing safety actions based on the analysis from the hazard detection system 140. The safety control system 160 may provide commands to disable operations or initiate safety protocols when hazardous situations are detected. The safety control system 160 is equipped with multiple safety protocols that can be activated to prevent unwanted interactions between objects, such as humans and machines or machines and other machines. One of the actions includes sounding an audio or visual alarm to alert personnel of a potential safety risk, thereby enabling them to takeprecautionary measures. Additionally, the safety control system 160 can disable machinery or change the speed of operation of machinery to prevent accidents or injuries. In scenarios where altering the operation of machines is possible, the safety control system 160 can change the movement path or adjust the speed of a machine. Such changes to the speed or path of a machine can be used to avoid collisions or other hazardous interactions, ensuring that machines operate within safe parameters. By dynamically adjusting these operational aspects, the safety control system 160 helps maintain a safe environment, minimizing risks associated with the interaction of various objects within the space. The safety control system 160 ability to implement these diverse safety actions ensures comprehensive protection and enhances the overall safety and efficiency of the human-cyber-physical system.

[0030] FIG. 2 shows an illustration 200 depicting the correspondence 206 between a first identification 204-1 and a second identification 204-2 of a set of objects in accordance with an embodiment. The illustration 200 includes a first data structure (FDS) 202-1, a second data structure (SDS) 202-2, a first identification 204-1, a second identification 204-2, and a correspondence 206 between the first identification 204-1 and a second identification 204-2. The illustration 200 provides a visual representation of how objects 102 are identified and matched across different data structures within a multi-sensor environment.

[0031] In exemplary embodiments, the first data structure FDS 202-1 includes a series of objects identified by the first identification 204-1. Each object within the FDS 202-1 is associated with a specific identifier, which is used to track and manage the object's location and status within the environment. In exemplary embodiments, the FDS 202-1 is periodically generated by the first localization system 114 and provided by the first localization system 114 to the consistency verification system 130.

[0032] In exemplary embodiments, the first data structure (FDS) 202-1 is a comprehensive repository of information related to a set of objects within a multi-sensor environment. Each entry in the FDS 202-1 corresponds to an individual object and contains several pieces of data that are used for accurate localization and tracking. For each object, the FDS 202-1 includes a unique identifier, referred to as the first identification 204-1. This identifier is used to distinguish the object from others within the data structure and is used for tracking theobject's movements and status over time. In addition to the identifier, the FDS 202-1 records the location of each object using a set of coordinates, typically represented as an (x, y) pair. These coordinates provide a precise spatial reference for the object's position within the environment, allowing for accurate mapping and analysis of its movements. Furthermore, the FDS 202-1 includes a timestamp for each object entry. This timestamp indicates the exact time at which the location data was captured, providing a temporal context for the object's position. The inclusion of timestamps is used for synchronizing data across different sensors and ensuring that the localization process accounts for any temporal discrepancies.

[0033] In exemplary embodiments, the second data structure SDS 202-2 includes a series of objects identified by the second identification 204-2. Similar to the FDS 202-1, the SDS 202- 2 provides identification and location data for objects 102 within the environment 100, but this data is sourced from a different localization system. In exemplary embodiments, the SDS 202-2 is periodically generated by the second localization system 124 and provided by the second localization system 124 to the consistency verification system 130. The SDS 202-2 complements the FDS 202-1 by offering an alternative perspective on the objects' positions.

[0034] In exemplary embodiments, the second data structure (SDS) 202-2 is a comprehensive repository of information related to a set of objects within a multi-sensor environment. Each entry in the SDS 202-2 corresponds to an individual object and contains several pieces of data that are used for accurate localization and tracking. For each object, the SDS 202-2 includes a unique identifier, referred to as the second identification 204-2. This identifier is used to distinguish the object from others within the data structure and is used for tracking the object's movements and status over time. In addition to the identifier, the SDS 202- 2 records the location of each object using a set of coordinates, typically represented as an (x, y) pair. These coordinates provide a precise spatial reference for the object's position within the environment, allowing for accurate mapping and analysis of its movements. Furthermore, the SDS 202-2 includes a timestamp for each object entry. This timestamp indicates the exact time at which the location data was captured, providing a temporal context for the object's position. The inclusion of timestamps is used for synchronizing data across different sensors and ensuring that the localization process accounts for any temporal discrepancies.

[0035] The first identification 204-1 and the second identification 204-2 are used to establish a correspondence 206 between the objects in the two data structures. As used herein, the first identification 204-1 is considered to correspond with the second identification 204-2 based on a determination that the first identification 204-1 and the second identification 204-2 both relate to the same object. The correspondence 206 represents the process of aligning and verifying the consistency of object identifications across different data streams, ensuring that the objects are accurately tracked and managed within the multi-sensor environment.

[0036] FIG. 3 shows a graph 300 illustrating the spatial relationship between various locations of objects within a multi-sensor environment. The graph 300 includes an illustration of an object having a consistent location 302 and an object having an inconsistent location 308. The graph 300 is used to depict how different data points from multiple sensors are analyzed to determine the consistency of object localization. In one embodiment, the first location 304 is derived from the data provided by the first localization system. The first location 304 is based on the initial sensor readings and serves as a baseline for comparison with other data points. The second location 306 is obtained from the second localization system, offering an alternative perspective on the object's position. The second location 306 is compared against the first location 304 to assess the consistency of the data.

[0037] In exemplary embodiments, an object is determined to have a consistent location 302 when the data from different sensors agree on the position of an object. For example, when the first location 304 and a second location 306 are within a threshold maximum difference, which is specified by the consistent verification preferences 132, from one another, the location of the object is classified as being a consistent location 302. Likewise, an object is determined to have an inconsistent location 308 when there is a discrepancy between the first location 304 and the second location 306 of an object. For example, when the first location 304 and a second location 306 are separated by a distance that is greater than the threshold maximum difference, the location of the object is classified as being an inconsistent location 308.

[0038] Referring now to FIG. 4, a flow chart of a computer- implemented method 400 for ensuring localization consistency and safety in a multi- sensor environment according to one or more embodiments is shown. In exemplary embodiments, the method 400 implemented by asystem 110, as shown in FIG. IB, that is designed to process data from multiple localization systems to identify and manage hazardous areas within an environment.

[0039] As shown at block 402, the method 400 involves receiving a first data set from a first localization system. This data set is received as part of a data stream that collects and transmits the data at a first frequency. The data set includes the identification of a set of objects in an environment and the location of each of these objects. In one embodiment, the first localization system typically processes visual data to generate this information, ensuring that the data stream provides a continuous and timely update of the objects' positions within the environment. As shown at block 404, the method involves receiving a second data set from a second localization system. This data set is received as part of a data stream that collects and transmits the data at a second frequency, which may be different from the first frequency. Similar to the first data set, this second data set includes the identification and location of the same set of objects in the environment. In one embodiment, the second localization system uses non- visual data to provide an alternative perspective on the objects' positions.

[0040] Next, as shown at block 406, the method 400 classifies each object in the set as having a consistent or inconsistent location. This classification is based on the difference between the reported locations from the first and second data sets and is guided by one or more consistency verification preferences 132. These preferences define the criteria for determining location consistency. In exemplary embodiments, the calculation of the difference between the first location and the second location for each object is performed by accessing the location data from both the first and second localization systems. For example, each localization system provides a set of coordinates, typically in the form of (x, y) pairs, for each object at a given timestamp. The data points are aligned based on their corresponding identifiers and timestamps to ensure that the comparison is made between the correct instances of each object. Once aligned, the Euclidean distance between the two sets of coordinates for each object is calculated.represent the coordinates from the first and second localization systems, respectively. The resulting distance value quantifies the spatial discrepancy between the two reported locations.

[0041] As shown at block 408, the method 400 includes calculating the last known consistent location for each object classified as having a consistent location. The last known consistent location is the most recent position of an object within the environment that has been classified as having a consistent location between the multiple data sources. This location is determined by comparing data from multiple localization systems and ensuring that the reported positions from these systems align within a predefined threshold of consistency. The last known consistent location serves as a reference point for tracking the object's movements and is used to assess the object's current and future positions.

[0042] In one embodiment, the last known consistent location can be calculated by averaging the coordinates from both localization systems. This involves computing the midpoint of the two sets of coordinates:Xi + x2+ y22 ’ 2JThis averaged position provides a balanced and reliable estimate of the object's location, taking into account the data from both systems.

[0043] In another embodiment, the last known consistent location can also be calculated using a weighted average approach, where the weights are determined based on the error rates of the different types of sensors involved. This method provides a more nuanced estimate of the object's position by giving more influence to the data from sensors with lower error rates. In this approach, each localization system provides a set of coordinates, (%i> yi) and (*2^2), along with an associated error rate, e and e2, which reflects the reliability of the data from each sensor type. The weights for the averaging process are inversely proportional to these error rates, meaning that a sensor with a lower error rate will have a higher weight in the calculation. The weighted average for the last known consistent location is calculated using the following formula:where the weights Wj and w2are defined as:1 1Wi = — , w2= — ei e2This calculation ensures that the most reliable data contributes more significantly to the final location estimate. By using a weighted average, the system can account for the varying accuracy of different sensor types, resulting in a more precise and dependable last known consistent location. This method is particularly beneficial in environments where sensor performance may vary due to factors such as interference or environmental conditions, allowing the system to maintain high localization accuracy and reliability.

[0044] For objects classified as having an inconsistent location, the method calculates an inconsistent location area in the environment. In exemplary embodiments, when an object is classified as having an inconsistent location, the method 400 calculates an inconsistent location area in the environment to better understand the potential range of error in the object's reported position. To calculate the inconsistent location area, the method considers the potential error margin around the object's reported positions. This area can be represented as a circular region centered at the midpoint of the two inconsistent locations, with a radius equal to the calculated Euclidean distance. The midpoint is determined by averaging the coordinates from both localization systems:Xi + x2+ y22 ’ 2JThe resulting circle defines the inconsistent location area, representing the possible range within which the object might actually be located. Furthermore, the size of this area is dependent on a length of time of a time period used for capturing the location data. This area is used for assessing potential safety risks and determining appropriate actions to mitigate hazards in the environment. By identifying and analyzing inconsistent location areas, the system can enhance its understanding of localization discrepancies and improve overall safety assurance.

[0045] Next, as shown at block 410, the method 400 identifies one or more hazards in the environment. The one or more hazards may include hazardous areas and / or hazardous situations. In exemplary embodiments, the hazardous areas are specified in the hazard detection specifications 142 shown in FIG. IB. Additionally, the hazard detection specifications 142 includes logic that is used to identify hazardous situations. In exemplary embodiments, hazardsare determined based on the analysis of consistent and inconsistent locations, ensuring that potential safety risks are recognized. In exemplary embodiments, hazardous areas are specific zones within the environment that pose potential safety risks to human participants or equipment. These areas can be identified through a detailed analysis of both consistent and inconsistent object locations. For example, a hazardous area might be a region where heavy machinery operates, such as near a robotic arm or a conveyor belt, where the risk of collision or injury is high. Another example could be a zone where mobile robots frequently navigate, especially if their paths intersect with human walkways, creating a potential for accidents. The identification of hazards involves monitoring the movement and location data of objects within the environment. If an object, such as a human or a piece of equipment, is detected moving towards a zone with a high concentration of machinery or where inconsistent location data suggests potential sensor errors, the system can flag this as a hazardous situation. Additionally, areas with frequent inconsistencies in object localization might indicate sensor blind spots or interference, which could lead to undetected hazards. In exemplary embodiments, once identified, these hazardous areas are continuously monitored to ensure that any object entering or approaching these zones is tracked with high precision.

[0046] As shown at block 412, the method 400 involves automatically modifying the operation of a piece of equipment in the one or more hazards. In exemplary embodiments, this action is based on an evaluation of the last known consistent locations, the inconsistent location areas, and the hazards ensuring that safety protocols are enforced to prevent accidents or injuries. In exemplary embodiments, the modifying the operation of the machinery or equipment can include disabling the machinery or equipment or changing an operation speed of the machinery or equipment. In addition, the system may also implement other safety protocols, rerouting mobile robots, or alerting personnel through audio or visual alarms, to mitigate risks and maintain a safe operational environment.

[0047] In one embodiment, for an object with a consistent location, the system relies on the last known consistent location to determine its proximity to a hazardous area. If the object's last known consistent location indicates that it is approaching a hazardous area, the system assesses the potential risk based on predefined safety thresholds. For example, if a mobile robot with a consistent location is detected moving towards an area where heavy machinery operates,and the distance to the machinery falls below a critical threshold, the system may decide to disable the machinery to prevent a collision. This decision is based on the reliable data provided by the consistent location, ensuring that the response is both timely and accurate. Alternately, the system may decide to alter the speed or path of the mobile robot to avoid the heavy machinery.

[0048] In another embodiment, for an object with an inconsistent location, the system must account for the uncertainty in its position. The inconsistent location area, represented as a potential range of error around the object's reported position, is used to evaluate the risk. If this inconsistent location area overlaps with a hazardous zone, the system considers the worst-case scenario where the object might be closer to the hazard than indicated by the inconsistent data. For instance, if a human is detected with an inconsistent location near a conveyor belt, and the inconsistent location area suggests a possibility of the person being within the belt's operational range, the system may choose to disable the conveyor belt as a precautionary measure. This decision is made to error on the side of safety, acknowledging the potential inaccuracies in the object's reported position.

[0049] Referring now to FIG. 5, an illustration of a scenario 500 for determining when an object is capable of entering a hazardous area are shown. In the illustrated scenario 500, an object 502 is shown having a location. As illustrated, based on the maximum speed of the object 502 and the frequency of the monitoring period, concentric circles 506 that represent the range of possible locations of the object 502 are created. In the case where one of the circles 506 overlaps a hazardous area 504, it is determined that the object is capable of entering the hazardous area 504 within the time period associated with the corresponding circle 506. In the illustrated scenario 500, the object 502 is determined to be capable of entering the hazardous area 504 within a time period that is greater than 2 times the duration of the monitoring window and less than 3 times the duration of the monitoring window. In exemplary embodiments, this time period is compared to a threshold time period specified in the hazard detection specification 142 to determine whether a corrective action should be taken.

[0050] Referring now to FIG. 6, a flow chart of a computer-implemented method 600 for managing safety in a multi-sensor environment according to one or more embodiments is shown. In exemplary embodiments, the method 600 is performed by the system 110, as depicted in FIG.IB, which is designed to process data from multiple localization systems to identify and manage hazardous areas within an environment.

[0051] As shown at block 602, the method 600 begins by identifying a last known consistent location or an inconsistent location area for each of a set of objects detected in the environment. This step involves analyzing data from both the first and second localization systems to determine the reliability of the reported positions. For objects with consistent locations, the system calculates the last known consistent location, while for those with inconsistent locations, it defines an inconsistent location area to account for potential errors in localization.

[0052] At block 604, the method involves identifying hazards in the environment. In exemplary embodiments, the hazards can include one or more hazardous areas and hazardous situations that are determined based on a comprehensive analysis of the environment, taking into account both consistent and inconsistent object locations. The identification of a hazardous area is based on one or more hazard detection preferences that include a threshold proximity to heavy machinery and high-traffic zones. For instance, a hazardous area can be identified in regions where heavy machinery operates, such as near a robotic arm or a conveyor belt. These areas are inherently risky due to the potential for collisions or injuries if humans or other objects inadvertently enter the area. The system can flag these areas as hazardous by monitoring the proximity of objects to the machinery and assessing the likelihood of interaction. Another example of identifying hazards involves zones where mobile robots frequently navigate. These areas can become hazardous if the paths of the robots intersect with human walkways or other critical zones. The system can analyze the movement patterns and speeds of the robots, along with the layout of the environment, to determine potential points of conflict. If an object, such as a human or another piece of equipment, is detected moving towards these intersecting paths, the system can designate the area as hazardous. Additionally, areas with frequent inconsistencies in object localization might indicate sensor blind spots or interference, which could lead to undetected hazards. By continuously monitoring these factors, the system can dynamically identify and manage hazardous areas, ensuring that safety risks are promptly recognized and mitigated.

[0053] Next, as shown at block 606, the method identifies a maximum speed for each of the set of objects. One way to determine the maximum speed is by analyzing historical movement data of the objects. By examining past trajectories and speeds, the system can establish a maximum speed for each object type, such as humans, mobile robots, or machinery. This historical analysis allows the system to predict future movements based on established patterns. Another approach involves real-time monitoring of the objects' current speeds using data from the localization systems. By continuously tracking the position changes of each object over time, the system can determine the maximum speed of the object. This method is useful in dynamic environments where objects may frequently change speed due to operational demands or environmental conditions. By combining both historical data and real-time monitoring, the system can accurately determine the maximum speed for each object, enhancing its ability to predict potential interactions with hazardous areas.

[0054] At decision block 608, the method evaluates whether any of the objects are capable of entering a hazardous area within a threshold time period. The threshold time period is a predefined duration specified in the hazard detection specifications 142 that used to assess whether an object is capable of entering a hazardous area within a set timeframe. For example, a hazard detection specification 142 may set a threshold time period at 5 seconds and a determination of whether an object is capable of entering a hazardous area within the threshold time period includes creating a circle with a center based at the objects current location with a radius equal to the maximum speed of the object per second times 5 seconds. If this circle overlaps the hazardous area, the object is capable of entering a hazardous area within a threshold time period. In exemplary embodiments, the determination that the object is capable of entering a hazardous area within a threshold time period is based on a comprehensive analysis of the object's current location, speed, and the proximity to the identified hazardous zones. In addition, this decision can also be based on a frequency of the monitoring windows of the localization systems. If the frequency is high enough to receive a new data set before the object is capable of reaching the hazardous area, the decision can be delayed to the outcome of the next consistency checks for the new data set.

[0055] In one embodiment, to determine whether any of the objects are capable of entering a hazardous area within a threshold time period, the system employs a geometricapproach by identifying the potential area that the object can travel to within the threshold time period. This involves drawing a circle centered at the object's last known consistent location, with the radius determined by the product of the object's speed and the duration of the threshold time period. For instance, consider an object, such as a mobile robot, with a consistent location at coordinates (x, y) and a speed of 2 meters per second. If the threshold time period for assessing potential entry into a hazardous area is set at 5 seconds, the radius of the circle would be calculated as:Radius = Speed X Time = 2m / s X 5s = lOmetersThe system then draws a circle with a center at (x, y) and a radius of 10 meters. This circle represents the area that the robot is capable of reaching within the 5-second timeframe. Next, the system checks for any overlap between this circle and the boundaries of identified hazardous areas. If the circle intersects with a hazardous zone, it indicates that the object is capable of entering the hazardous area within the threshold time. For example, if the circle overlaps with a region where heavy machinery operates, the system determines that the robot poses a potential safety risk.

[0056] In another embodiment, to determine whether any of the objects are capable of entering a hazardous area within a threshold time, the system uses a geometric approach by expanding the object's inconsistent location area. This involves enlarging the existing inconsistent area in each direction by a distance determined by the product of the object's speed and the duration of the threshold time period. For example, consider an object, such as a mobile robot, with an inconsistent location area represented as a rectangle with dimensions w by h. The robot has a speed of 1.5 meters per second. If the threshold time period for assessing potential entry into a hazardous area is set at 4 seconds, the expansion distance in each direction would be calculated as:Expansion Distance = Speed x Time = 1.5m / s x 4s = 6metersThe system then expands the inconsistent location area by 6 meters in each direction, effectively creating a larger rectangle that represents the potential area the robot could reach within the 4- second timeframe. Next, the system checks for any overlap between this expanded area and theboundaries of identified hazardous areas. If the expanded area intersects with a hazardous zone, it indicates that the object is capable of entering the hazardous area within the threshold time period. For instance, if the expanded area overlaps with a region where heavy machinery operates, the system determines that the robot poses a potential safety risk.

[0057] In exemplary embodiments, the threshold time period for determining whether any of the objects are capable of entering a hazardous area is obtained from the hazard detection preferences. The threshold time periods may be different for objects that have consistent locations and inconsistent locations. In addition, the threshold time periods may be different for different classes of objects.

[0058] Next, as shown at block 610, the method involves disabling the operation of a piece of equipment in the hazardous area. This action is performed to prevent potential accidents or injuries when an object is determined to be capable of entering a hazardous zone within the threshold time period. In addition to disabling machinery or equipment, the system may also implement other safety protocols, rerouting mobile robots, or alerting personnel through audio or visual alarms, to mitigate risks and maintain a safe operational environment. One such action is rerouting mobile robots to avoid potential collisions. By dynamically adjusting the path of a robot, the system can steer it away from hazardous zones, ensuring that it continues its operations without compromising safety. Another action involves alerting personnel through audio or visual alarms. These alerts serve as immediate warnings to human operators, enabling them to take precautionary measures or evacuate the area if necessary. The alarms can be tailored to the severity of the risk, with more urgent situations triggering louder or more conspicuous alerts.

[0059] In exemplary embodiments, the system can also adjust the speed of moving objects, such as slowing down a robot or conveyor belt as it approaches a hazardous area. This reduction in speed provides additional time for corrective actions and reduces the likelihood of accidents. Furthermore, the system may activate physical barriers or safety gates to restrict access to hazardous areas temporarily. These barriers can prevent unauthorized entry and protect both personnel and equipment from potential harm. By employing a combination of theseactions, the system ensures a comprehensive approach to safety management, effectively reducing risks and enhancing the overall security of the environment.

[0060] Finally, the method continues to monitor the reported location of each object from both the first and second localization systems, as shown at block 612. The system updates the last known consistent location or the inconsistent location area for each object, maintaining an ongoing assessment of safety risks and ensuring that the environment remains secure. This continuous monitoring allows the system to dynamically respond to changes and maintain high safety standards.

[0061] FIG. 7 illustrates an example of a processing system 700 that can be used to implement the computer-based components described herein. The processing system 700 includes an exemplary computing device (“computer”) 702 configured for performing various aspects of the operations described herein in accordance aspects of the invention. In addition to computer 702, exemplary processing system 700 includes network 714, which connects computer 702 to additional systems (not depicted) and can include one or more wide area networks (WANs) and / or local area networks (LANs) such as the Internet, intranet(s), and / or wireless communication network(s). Computer 702 and additional system are in communication via network 714, e.g., to communicate data between them. In exemplary embodiments, one or more of the first localization system 114, the second localization system 124, the consistency verification system 130, the hazard detection system 140, and the safety control system may be embodied in a processing system 700. In exemplary embodiments, the first localization system 114 and the second localization system 124 are configured to run on separate hardware systems to ensure that a single point of failure does exist between the first localization system 114 and the second localization system 124.

[0062] Exemplary computer 702 includes processor cores 704, main memory (“memory”) 710, and input / output component(s) 712, which are in communication via bus 703. Processor cores 704 includes cache memory (“cache”) 706 and controls 708, which include branch prediction structures and associated search, hit, detect and update logic, which will be described in more detail below. Cache 706 can include multiple cache levels (not depicted) that are on or off-chip from processor 704. Memory 710 can include various data stored therein, e.g.,instructions, software, routines, etc., which, e.g., can be transferred to / from cache 706 by controls 708 for execution by processor 704. Input / output component(s) 712 can include one or more components that facilitate local and / or remote input / output operations to / from computer 702, such as a display, keyboard, modem, network adapter, etc. (not depicted).

[0063] A cloud computing system 720 is in wired or wireless electronic communication with the processing system 700. The cloud computing system 720 can supplement, support or replace some or all of the functionality (in any combination) of the processing system 700. Additionally, some or all of the functionality of the processing system 700 can be implemented as a node of the cloud computing system 720.

[0064] For the sake of brevity, conventional techniques related to making and using the disclosed embodiments may or may not be described in detail herein. In particular, various aspects of computing systems and specific computer programs to implement the various technical features described herein are well known. Accordingly, in the interest of brevity, many conventional implementation details are only mentioned briefly or are omitted entirely without providing the well-known system and / or process details.

[0065] The various components / modules / models of the systems illustrated herein are depicted separately for ease of illustration and explanation. In embodiments of the invention, the functions performed by the various components / modules / models can be distributed differently than shown without departing from the scope of the various embodiments of the invention describe herein unless it is specifically stated otherwise.

[0066] Aspects of the invention can be embodied as a system, a method, and / or a computer program product at any possible technical detail level of integration. The computer program product may include a computer readable storage medium (or media) having computer readable program instructions thereon for causing a processor to carry out aspects of the present invention.

[0067] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the present invention. As used herein, the singular forms “a”, “an” and “the” are intended to include the plural forms as well, unless thecontext clearly indicates otherwise. It will be further understood that the terms “comprises” and / or “comprising,” when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, element components, and / or groups thereof.

[0068] While the present invention has been described with reference to an exemplary embodiment or embodiments, it will be understood by those skilled in the art that various changes may be made and equivalents may be substituted for elements thereof without departing from the scope of the present invention. In addition, many modifications may be made to adapt a particular situation or material to the teachings of the present invention without departing from the essential scope thereof. Therefore, it is intended that the present invention not be limited to the particular embodiment disclosed as the best mode contemplated for carrying out this present invention, but that the present invention will include all embodiments falling within the scope of the claims. 1

Claims

CLAIMSWhat is claimed is:

1. A computer-implemented method for managing safety in a multi-sensor environment, the method comprising: receiving data from a first localization system and a second localization system to identify a last known consistent location or an inconsistent area for each of a set of objects detected in an environment; identifying one or more hazards within the environment; determining a maximum speed for each of the set of objects; evaluating whether any of the set of objects are capable of entering a hazardous area of the one or more hazards within a threshold time; and based on a determination that an object of the set of objects is capable of entering the hazardous area of the one or more hazards within the threshold time, performing a safety action.

2. The computer-implemented method of claim 1, wherein the safety action includes disabling an operation of a piece of equipment in the hazardous area.

3. The computer-implemented method of claim 1, further comprising continuously monitoring the location of each object in the set of objects from both the first and second localization systems and updating the last known consistent location or the inconsistent area for each object.

4. The computer-implemented method of claim 1 , wherein the identification of the hazards is based on one or more hazard detection preferences that include a threshold proximity to heavy machinery and high-traffic zones.

5. The computer-implemented method of claim 4, further comprising dynamically updating the one or more hazards based on real-time changes in the environment.

6. The computer-implemented method of claim 1 , wherein the maximum speed for each object is determined using a combination of historical movement data and real-time monitoring.

7. The computer-implemented method of claim 6, further comprising adjusting the maximum speed based on environmental conditions or a classification of the object.

8. The computer-implemented method of claim 1, wherein the safety action includes one of altering a path of a mobile robot or activating an alarm to prevent entry of the object into the hazardous area.

9. A system for managing safety in a multi-sensor environment, the system comprising: a processor; a memory coupled to the processor; and one or more computer readable storage media coupled to the processor, the one or more computer readable storage media collectively containing instructions that are executed by the processor via the memory to cause the processor to perform operations comprising: receiving data from a first localization system and a second localization system to identify a last known consistent location or an inconsistent area for each of a set of objects detected in an environment; identifying one or more hazards within the environment; determining a maximum speed for each of the set of objects; evaluating whether any of the set of objects are capable of entering a hazardous area of the one or more hazards within a threshold time; andbased on a determination that an object of the set of objects is capable of entering the hazardous area of the one or more hazards within the threshold time, performing a safety action.

10. The system of claim 9, wherein the safety action includes disabling an operation of a piece of equipment in the hazardous area.

11. The system of claim 9, wherein the operations further comprise continuously monitoring the location of each object in the set of objects from both the first and second localization systems and updating the last known consistent location or the inconsistent area for each object.

12. The system of claim 9, wherein the identification of the hazards is based on one or more hazard detection preferences that include a threshold proximity to heavy machinery and high-traffic zones.

13. The system of claim 12, wherein the operations further comprise dynamically updating the one or more hazards based on real-time changes in the environment.

14. The system of claim 9, wherein the maximum speed for each object is determined using a combination of historical movement data and real-time monitoring.

15. The system of claim 14, wherein the operations further comprise adjusting the maximum speed based on environmental conditions or a classification of the object.

16. The system of claim 9, wherein the safety action includes one of altering a path of a mobile robot or activating an alarm to prevent entry of the object into the hazardous area.

17. A computer program product, the computer program product comprising a non- transitory tangible storage device having program code embodied therewith, the program code executable by a processing system to perform operations comprising: receiving data from a first localization system and a second localization system to identify a last known consistent location or an inconsistent area for each of a set of objects detected in an environment;identifying one or more hazards within the environment; determining a maximum speed for each of the set of objects; evaluating whether any of the set of objects are capable of entering a hazardous area of the one or more hazards within a threshold time; and based on a determination that an object of the set of objects is capable of entering the hazardous area of the one or more hazards within the threshold time, performing a safety action.

18. The computer program product of claim 17, wherein the safety action includes disabling an operation of a piece of equipment in the hazardous area.

19. The computer program product of claim 17, wherein the operations further comprise continuously monitoring the location of each object in the set of objects from both the first and second localization systems and updating the last known consistent location or the inconsistent area for each object.

20. The computer program product of claim 17, wherein the identification of the hazards is based on one or more hazard detection preferences that include a threshold proximity to heavy machinery and high-traffic zones.

Images