Ensuring localization consistency in multi-sensor systems

Abstract

Embodiments ensure localization consistency in multi-sensor environments by processing data from multiple localization systems. Aspects include receiving a first data stream from a first localization system, which includes data structures with object identifications, locations, and timestamps. Similarly, a second data stream from a second localization system provides corresponding data. The system identifies correspondences between the object identifications from both streams. For each object, it calculates the difference between the first and second locations, classifying them as consistent or inconsistent based on a predefined threshold. Aspects also include calculating the last known consistent location for each object, ensuring accurate tracking and management within the environment.

Description

ENSURING LOCALIZATION CONSISTENCY IN MULTI-SENSOR SYSTEMSBACKGROUND

[0001] The present disclosure relates to human-cyber-physical systems, specifically to methods and systems for ensuring localization consistency in multi-sensor environment.

[0002] Ensuring the safety of human participants is a primary concern in systems involving dynamic interactions between humans and automated systems, such as manufacturing robots or mobile ground robots. These systems rely on accurate positioning to prevent harm or injury to humans sharing the same physical space. However, achieving reliable positioning in environments with multiple sensors presents significant challenges. Current methods often utilize sensor fusion to estimate system states by combining data from various sensors. While this approach provides a high-level framework for state estimation, it frequently falls short in addressing the complexities of safety assurance, particularly under varying noise models and sensor modalities.

[0003] Existing solutions often lack resilience to sensor failures, outliers, and asynchronous data, which can lead to inaccuracies in state estimation and potential safety risks. These methods typically do not provide deterministic or probabilistic guarantees on the error between estimated states and actual conditions, particularly when sensor failures or missing data occur. As a result, there is a pressing need for a more robust and reliable approach to ensure accurate positioning in systems with multiple sensors, one that can offer enhanced safety assurances and effectively manage the uncertainties and variabilities present in sensor data.SUMMARY

[0004] Embodiments of the present disclosure are directed to computer- implemented methods for ensuring localization consistency in multi-sensor environments. According to an aspect, a computer-implemented method includes receiving, from a first localization system, a first data stream that includes a plurality of first data structures, each of the first data structures include a first identification of a set of objects detected in an environment, a first location of each of the set of objects detected in the environment, and a timestamp corresponding to when the first location was captured and receiving, from second first localization system, a second data stream that includes a plurality of a second data structures, each of the second data structuresinclude a second identification of the set of objects detected in the environment, a second location of each of the set of objects detected in the environment, and a timestamp corresponding to when the second location was captured. The method also includes identifying a correspondence between the first identification of the set of objects and the second identification of the set of objects and for each of the set of objects, calculating a difference between the first location and the second location of each of the set of objects. The method further includes classifying each of the set of objects as having a consistent location or inconsistent location based on the difference between the first location and the second location and based on a threshold maximum difference and calculating a last known consistent location in the environment for each of the set of objects.

[0005] Embodiments also include computer systems and computer program products for ensuring localization consistency in multi-sensor environments.

[0006] Additional technical features and benefits are realized through the techniques of the present disclosure. Embodiments and aspects of the disclosure are described in detail herein and are considered a part of the claimed subject matter. For a better understanding, refer to the detailed description and to the drawings.BRIEF DESCRIPTION OF THE DRAWINGS

[0007] FIG. 1 A depicts a schematic block diagram illustrating an environment with a plurality of objects and sensors in accordance with an embodiment.

[0008] FIG. IB depicts a block diagram of a system for ensuring localization consistency in multi-sensor environment in accordance with an embodiment.

[0009] FIG. 2 depicts an illustration of a correspondence between a first identification and second identification of a set of objects in accordance with an embodiment.

[0010] FIG. 3 A depicts a comparison between first and second data structures during a monitoring period in accordance with an embodiment.

[0011] FIG. 3B depicts the use of monitoring windows within a multi-sensor environment in accordance with an embodiment.

[0012] FIG. 4 depicts a graph illustrating the spatial relationship between consistent and inconsistent object locations in a multi-sensor environment in accordance with an embodiment.

[0013] FIG. 5 depicts a flow chart diagram depicting a method for ensuring localization consistency in multi-sensor environment in accordance with an embodiment.

[0014] FIG. 6 depicts a block diagram of a processing system in accordance with an embodiment.

[0015] In the accompanying figures and following detailed description of the disclosed embodiments, the various elements illustrated in the figures are provided with three digit reference numbers. In some instances, the leftmost digits of each reference number corresponds to the figure in which its element is first illustrated.DETAILED DESCRIPTION

[0016] Automated systems may be used to ensure the safety of human participants in a human-cyber-physical systems (HCPS), particularly in environments where dynamic interactions occur between humans and automated systems, such as manufacturing robots or mobile ground robots. These systems traditionally rely on accurate localization of objects in the environment to prevent harm or injury to humans sharing the same physical space. However, achieving reliable localization in environments with multiple sensors presents significant challenges.

[0017] Existing solutions in the field of HCPS, such as those used in autonomous vehicles and unmanned aerial vehicles, have utilized sensor fusion to perform state estimation. Sensor fusion algorithms typically combine information from different noisy sensors to estimate the state, reasoning about the uncertainty of sensor measurement noise using probability distributions. However, these algorithms have notable limitations and often do not explicitly address safety assurance levels under different noise models for individual sensor modalities. Furthermore, existing systems may not be resilient to intermittent sensor failures, outliers, missing data, or asynchronous observations. These shortcomings can lead to inaccuracies in state estimation and potential safety risks, as they do not provide deterministic or probabilistic guarantees on the error between estimated states and actual conditions, particularly in the presence of sensor failures or missing data.

[0018] The methods, systems, and computer program products disclosed herein address these challenges using methods for ensuring localization consistency in multi-sensor environments. The disclosed method provides a robust framework for reliable localization, applicable to any type of sensor tasked with person localization. The approach leverages a multimodal system to address distinct failure root causes and manages multiple sensor signals operating at different rates to produce a single observation signal. The method introduces a consistency verification system that determines the spatial distance between objects represented in two or more disparate input streams. The consistency verification system is configured to determine a last known consistent location for each object in the environment and to identify any objects in the environment that have inconsistent locations.

[0019] Referring now to FIG. 1A, an environment 100 having a plurality of objects 102- 1, 102-2, 102-3, 102-4, and 102-5 (referred to collectively herein as objects 102), as well as multiple sensors 112 and sensors 122 is shown. The environment 100 is designed to illustrate the spatial arrangement and interaction between the objects 102 and sensors 112, 122. In exemplary embodiments, sensors 112 are a first type of sensor that is configured to monitor the location of the objects 102 in the environment. For example, the first sensors 112 may be video cameras that are configured to monitor a portion of the environment 100. Likewise, sensors 122 are a second type of sensor that is configured to monitor the location of the objects 102 in the environment. For example, the second sensor 122 may be radio frequency sensors that are configured to detect the presence of an object 102 in the environment 100.

[0020] Referring now to FIG. IB, a block diagram of system 110 for ensuring localization consistency in multi-sensor environment in accordance with an embodiment is shown. In exemplary embodiments, the system includes a first localization system 114 and a second localization system 124 that respectively receive data from sensors 112 and sensors 122 that are disposed in the environment 100. The sensors 112, 122 are configured to capture environmental data, which is then processed to determine the location of objects within the monitored space. The sensors 112 are typically of a first type, such as video cameras, which provide visual data about the environment. On the other hand, the sensors 122 are of a second type, such as radio frequency sensors, which detect the presence of objects through non- visualmeans. The diversity in sensor types allows the system 110 to capture a wide range of data, enhancing the robustness of the localization process.

[0021] In exemplary embodiments, the first localization system 114 receives data from the sensors 112 and is configured to processing the data received from these sensors 112. The first localization system 114 is configured to analyze the received data to identify and track objects 102 within the environment 100, generating a first set of location data for the identified objects 102. The first localization system 114 is designed to handle the specific characteristics of the data provided by the sensors 112, such as high-resolution images or video feeds, and to extract meaningful information regarding the position and movement of objects.

[0022] In exemplary embodiments, the second localization system 124 receives data from the sensors 122 and processes the data from these sensors. The second localization system 124 is configured to determine the location of objects 102 within the environment. In one embodiment, each of the objects 102 includes an ultra- wideband (UWB) tag and the sensors 122 are configured to ready the UWB tags and to determine a position of the object 102 in the environment 100. The second localization system 124 complements the first localization system 114 by providing an alternative perspective on the environment, which is particularly useful in scenarios where visual data may be obstructed or insufficient.

[0023] In exemplary embodiments, the second localization system 124 utilizes UWB technology in conjunction with Time Difference of Arrival (TDoA) techniques to determine the location of objects 102 within the environment. UWB is a radio technology that uses a wide frequency spectrum to enable precise location tracking and high-resolution ranging capabilities. The second localization system 124 receives data from the sensors 122, which are equipped with UWB capabilities. These sensors emit UWB signals that interact with UWB tags attached to the objects 102. The UWB tags respond to the signals, allowing the sensors 122 to capture the time- of-flight (ToF) of the signals. The ToF data is crucial for calculating the distance between each sensor and the UWB tag on the object. To determine the precise location of an object 102, the second localization system 124 employs triangulation, a method that involves using the known positions of multiple sensors 122 and the measured distances to the object. By analyzing the ToF data from at least three different sensors, the system can calculate the object's position throughgeometric triangulation. This process involves solving a set of equations that represent the distances from the sensors to the object, allowing the system to pinpoint the object's location within the environment. The use of UWB and triangulation provides several advantages, including high accuracy and robustness in environments with potential signal obstructions or interference. This method is particularly effective in dynamic environments where objects may move rapidly or where visual data may be insufficient. By leveraging UWB technology, the second localization system 124 enhances the overall reliability and precision of the localization process, ensuring that the system can maintain accurate tracking of objects 102 within the environment.

[0024] In exemplary embodiments, the consistency verification system 130 is configured to receive data streams from both the first localization system 114 and the second localization system 124. The consistency verification system 130 is configured to compare location data received from the first localization system 114 and the second localization system 124 to ensure that both sets of the location data for objects within the environment are consistent with one another. In exemplary embodiments, the consistency verification system 130 performs a detailed analysis of the data streams, identifying any discrepancies between the locations reported by the two systems. The consistency verification system 130 calculates the spatial distance between objects as represented in the different data streams and determines the last known consistent location for each object.

[0025] In exemplary embodiments, the output of the consistency verification system 130 may include various elements. The consistency verification system 130 is configured to provide a consistency status, indicating whether the location data for each object is consistent or inconsistent. This status is determined by comparing the spatial distance between the locations reported by the first and second localization systems. If the difference between the two locations is within a predefined threshold, the system classifies the location as consistent; otherwise, it is marked as inconsistent. Additionally, the consistency verification system 130 outputs the last known consistent location for each object. In one embodiment, the last known consistent location is calculated as a weighted average of the most recent consistent data points from both localization systems, with weights based on the error distribution statistics to ensure that the most reliable data contributes more significantly to the final location estimate.

[0026] In cases where inconsistencies are detected, the consistency verification system 130 may be configured to generate discrepancy alerts, highlighting the discrepancies. These alerts can be used to trigger further investigation or corrective actions to address potential issues in the sensor data or the localization process. Furthermore, the consistency verification system 130 may provide probabilistic guarantees on the correctness of the consistent state estimates, calculating the likelihood that the reported consistent locations are accurate based on the observed data and the known error characteristics of the sensors. In exemplary embodiments, the consistency verification system 130 may output a temporal analysis of the data streams, indicating the time periods during which the data was consistent or inconsistent. This analysis helps in understanding the dynamics of the environment and the performance of the localization systems over time.

[0027] FIG. 2 shows an illustration 200 depicting the correspondence 206 between a first identification 204-1 and a second identification 204-2 of a set of objects in accordance with an embodiment. The illustration 200 includes a first data structure (FDS) 202-1, a second data structure (SDS) 202-2, a first identification 204-1, a second identification 204-2, and a correspondence 206 between the first identification 204-1 and a second identification 204-2. The illustration 200 provides a visual representation of how objects 102 are identified and matched across different data structures within a multi-sensor environment.

[0028] In exemplary embodiments, the first data structure FDS 202-1 includes a series of objects identified by the first identification 204-1. Each object within the FDS 202-1 is associated with a specific identifier, which is used to track and manage the object's location and status within the environment. In exemplary embodiments, the FDS 202-1 is periodically generated by the first localization system 114 and provided by the first localization system 114 to the consistency verification system 130.

[0029] In exemplary embodiments, the first data structure (FDS) 202-1 is a comprehensive repository of information related to a set of objects within a multi-sensor environment. Each entry in the FDS 202-1 corresponds to an individual object and contains several pieces of data that are used for accurate localization and tracking. For each object, the FDS 202-1 includes a unique identifier, referred to as the first identification 204-1. This identifieris used to distinguish the object from others within the data structure and is used for tracking the object's movements and status over time. In addition to the identifier, the FDS 202-1 records the location of each object using a set of coordinates, typically represented as an (x, y) pair. These coordinates provide a precise spatial reference for the object's position within the environment, allowing for accurate mapping and analysis of its movements. Furthermore, the FDS 202-1 includes a timestamp for each object entry. This timestamp indicates the exact time at which the location data was captured, providing a temporal context for the object's position. The inclusion of timestamps is used for synchronizing data across different sensors and ensuring that the localization process accounts for any temporal discrepancies.

[0030] In exemplary embodiments, the second data structure SDS 202-2 includes a series of objects identified by the second identification 204-2. Similar to the FDS 202-1, the SDS 202- 2 provides identification and location data for objects 102 within the environment 100, but this data is sourced from a different localization system. In one embodiment, a camera system provides the classification and location where the classification determines what an object it is (human, robot, etc.). The spatial distance between the two localizations can be used to assign the corresponding object and identification. In exemplary embodiments, the SDS 202-2 is periodically generated by the second localization system 124 and provided by the second localization system 124 to the consistency verification system 130. The SDS 202-2 complements the FDS 202-1 by offering an alternative perspective on the objects' positions.

[0031] In exemplary embodiments, the second data structure (SDS) 202-2 is a comprehensive repository of information related to a set of objects within a multi-sensor environment. Each entry in the SDS 202-2 corresponds to an individual object and contains several pieces of data that are used for accurate localization and tracking. For each object, the SDS 202-2 includes a unique identifier, referred to as the second identification 204-2. This identifier is used to distinguish the object from others within the data structure and is used for tracking the object's movements and status over time. In addition to the identifier, the SDS 202- 2 records the location of each object using a set of coordinates, typically represented as an (x, y) pair. These coordinates provide a precise spatial reference for the object's position within the environment, allowing for accurate mapping and analysis of its movements. Furthermore, the SDS 202-2 includes a timestamp for each object entry. This timestamp indicates the exact timeat which the location data was captured, providing a temporal context for the object's position.The inclusion of timestamps is used for synchronizing data across different sensors and ensuring that the localization process accounts for any temporal discrepancies.

[0032] The first identification 204-1 and the second identification 204-2 are used to establish a correspondence 206 between the objects in the two data structures. As used herein, the first identification 204-1 is considered to correspond with the second identification 204-2 based on a determination that the first identification 204-1 and the second identification 204-2 both relate to the same object. The correspondence 206 represents the process of aligning and verifying the consistency of object identifications across different data streams, ensuring that the objects are accurately tracked and managed within the multi-sensor environment. In some cases, for example when a signal from an object is missing, a direct correspondence can not be made as there may be a different number of objects in the FDS 202-1 and the SDD 202-2.

[0033] FIGS. 3A and 3B illustrate the process of monitoring and comparing data structures within a multi-sensor environment to ensure localization consistency. FIG. 3A depicts a comparison between a first data structure (FDS) 202-1 and a second data structure (SDS) 202- 2. The FDS 202-1 and SDS 202-2 represent data streams from two different localization systems. Each data structure contains a series of entries corresponding to objects detected in the environment. The comparison 302 is performed to evaluate the consistency of the location between the corresponding entries in the FDS 202-1 and the SDS 202-2. This process involves aligning the data based on identifiers and timestamps to ensure that the same objects are being compared across both data structures. The monitoring period 300 indicates the time frame over which this comparison is conducted, allowing for the assessment of spatial consistency between the two data streams. In one embodiment, a longer the time frame would allow more checks to be performed and the system would become relatively safer. In exemplary embodiment the monitoring period should be long enough to receive signals from both of the localization systems. In exemplary embodiments, the location of the objects identified in the most recently received FDS 202-1 and SDS 202-2 VI are compared with one another. For example, the location of the objects identified in FDS 202-1 R3 is compared the location of the objects identified in SDS 202-2 VI until SDS 202-2 V2 is received, at which point the location of theobjects identified in FDS 202-1 R3 is compared the location of the objects identified in SDS 202- 2 V2.

[0034] FIG. 3B illustrates the use of monitoring windows, also referred to herein as monitoring periods, within the multi-sensor environment. FIG. 3B shows three monitoring windows, a first monitoring window 310-1, a second monitoring window 310-2, and a third monitoring window 310-3. Each monitoring window represents a fixed period during which data from the FDS 202-1 and SDS 202-2 is aggregated and analyzed for consistency. The monitoring windows are designed to capture multiple, timestamped messages from each data stream, providing a comprehensive view of the objects' locations over time. The overlapping nature of the monitoring windows allows for continuous assessment of localization consistency, ensuring that any discrepancies are promptly identified and addressed.

[0035] In one embodiment, the monitoring window is configured to dynamically adjust the size based on the variability of the data streams, allowing for more flexible and adaptive monitoring of object locations. This embodiment could utilize machine learning algorithms to predict optimal window sizes based on historical data patterns, thereby enhancing the accuracy of location consistency checks. In another embodiment, the monitoring window is implemented with a fixed size but incorporates a buffer system to handle asynchronous data arrivals, ensuring that all relevant data is considered within the window despite timing discrepancies. This buffer system could be designed to temporarily store incoming data until processing can occur in sync with other data streams. Additionally, the monitoring window could be equipped with a realtime visualization tool that provides a graphical representation of the spatial consistency checks, allowing operators to quickly identify and address inconsistencies. Furthermore, the monitoring window could be integrated with a notification system that alerts users when a significant inconsistency is detected, enabling prompt corrective actions.

[0036] FIG. 4 shows a graph 400 illustrating the spatial relationship between various locations of objects within a multi-sensor environment. The graph 400 includes an illustration of an object having a consistent location 402 and an object having an inconsistent location 408. The graph 400 is used to depict how different data points from multiple sensors are analyzed to determine the consistency of object localization. In one embodiment, the first location 404 isderived from the data provided by the first localization system. The first location 404 is based on the initial sensor readings and serves as a baseline for comparison with other data points. The second location 406 is obtained from the second localization system, offering an alternative perspective on the object's position. The second location 406 is compared against the first location 404 to assess the consistency of the data.

[0037] In exemplary embodiments, an object is determined to have a consistent location 402 when the data from different sensors agree on the position of an object. For example, when the first location 404 and a second location 406 are within a threshold maximum difference from one another, the location of the object is classified as being a consistent location 402. Likewise, an object is determined to have an inconsistent location 408 when there is a discrepancy between the first location 404 and the second location 406 of an object. For example, when the first location 404 and a second location 406 are separated by a distance that is greater than the threshold maximum difference, the location of the object is classified as being an inconsistent location 408. Furthermore, an object may be classified as having an inconsistent location if only one of the two localization systems reports a location for an object.

[0038] Referring now to FIG. 5, a flow chart of a computer- implemented method 500 for ensuring localization consistency in a multi-sensor environment according to one or more embodiments is shown. In exemplary embodiments, the method 500 is performed by the consistency verification system 130 shown in FIG. IB. The method 500 begins at block 502 with receiving, from a first localization system, a first data stream that includes a plurality of first data structures. Each of these data structures contains a first identification of a set of objects detected in an environment, a first location of each object, and a timestamp corresponding to when the first location was captured. In exemplary embodiments, the first data stream is received from a first localization system that is configured to track the locations of objects in a multi-sensor environment using sensors of a first type.

[0039] Next, as shown at block 504, the method 500 involves receiving, from a second localization system, a second data stream that includes a plurality of second data structures. Each of these data structures includes a second identification of the set of objects, a second location of each object, and a timestamp corresponding to when the second location was captured. Inexemplary embodiments, the second data stream is received from a second localization system that is configured to track the locations of objects in a multi-sensor environment using sensors of a second type, that is different than the first type.

[0040] In various embodiments, the system for ensuring localization consistency in multi-sensor environments can accommodate different configurations of data streams that provide data structures at varying frequencies. For instance, in one embodiment, the first data stream may be generated by a high-frequency radar system, while the second data stream is produced by a lower-frequency camera-based vision system. This configuration allows the system to leverage the high temporal resolution of radar data for dynamic object tracking, while using the detailed spatial information from the camera system to enhance localization accuracy. In another embodiment, the first data stream could be sourced from a LiDAR system operating at a medium frequency, providing precise distance measurements, whereas the second data stream might originate from an ultrasonic sensor array that operates at a different frequency, offering complementary proximity data. Additionally, the system can be adapted to integrate data streams from wearable sensors on individuals within the environment, such as GPS-enabled devices, which may transmit location data at yet another frequency. This flexibility in sensor integration enables the system to maintain robust localization consistency across diverse operational scenarios, including indoor environments where GPS signals are weak or unavailable, by relying on alternative sensor modalities.

[0041] As shown at block 506, the method 500 proceeds with identifying a correspondence between the first identification of the set of objects and the second identification of the set of objects. This step aligns the data from the two different localization systems. In exemplary embodiments, a matching algorithm is implemented to compare the identifiers from the first data stream with those from the second data stream. This algorithm can employ various techniques, such as direct matching for identifiers expected to be the same across both systems, fuzzy matching to accommodate slight variations due to sensor discrepancies or data entry errors, and proximity matching, which considers the spatial proximity of objects when identifiers are not directly comparable. If two objects are located within a predefined distance threshold, they may be considered a match. Once the matching process is complete, a correspondence map is created, linking each identifier from the first data stream to its matching identifier in the seconddata stream. This map serves as a reference for aligning the data from localization systems. To ensure accuracy, the correspondence map is verified by cross-referencing additional attributes, such as object size, shape, or movement patterns, to confirm that the matched identifiers truly represent the same object.

[0042] Following this, at block 508, the method 500 involves calculating a difference between the first location and the second location for each of the set of objects. In exemplary embodiments, the calculation of the difference between the first location and the second location for each object is performed each time new data is received from a data stream, at this time the new data is then compared with the last received data from the other localization system. This calculation is used for assessing the consistency of the location data. In exemplary embodiments, the calculation of the difference between the first location and the second location for each object is performed by accessing the location data from both the first and second localization systems. For example, each localization system provides a set of coordinates, typically in the form of (x, y) pairs, for each object at a given timestamp. The data points are analyzed based on their corresponding identifiers and timestamps to ensure that the comparison is made between the correct instances of each object. Once two data points have been determined to correspond to the same object, the Euclidean distance between the two sets of coordinates for the object is calculated. This involves applying the formula y (x2— Xi)2+ (y2— yi)2, where (x^y ) and (x2, y2) represent the coordinates from the first and second localization systems, respectively. The resulting distance value quantifies the spatial discrepancy between the two reported locations.

[0043] At block 510, the method 500 includes classifying each of the set of objects as having a consistent location or an inconsistent location based on the calculated difference and a predefined threshold maximum difference. In exemplary embodiments, the classification is performed by comparing the calculated difference in location to the predefined threshold maximum difference to determine whether the locations are consistent. If the difference is within the acceptable range, the locations are deemed consistent; otherwise, they are marked as inconsistent. This process is repeated for each object in the set, providing a comprehensive assessment of the consistency of the location data across the multi-sensor environment. By systematically calculating and evaluating these differences, the method ensures that anysignificant discrepancies are identified and addressed, thereby enhancing the reliability and accuracy of the localization system.

[0044] In one embodiment, the steps of calculating a difference between the first location and the second location for each of the set of objects, as shown at block 508, and classifying each of the set of objects as having a consistent location or an inconsistent location based on the calculated difference and a predefined threshold maximum difference, as shown at block 510, are repeated each time new data is received from one of the first or second localization systems. In another embodiment, the calculating a difference between the first location and the second location for each of the set of objects, as shown at block 508 is repeated each time new data is received from one of the first or second localization systems and the step of classifying each of the set of objects as having a consistent location or an inconsistent location based on the calculated difference and a predefined threshold maximum difference, as shown at block 510, is repeated once per monitoring window.

[0045] In exemplary embodiments, the predefined threshold maximum difference is a parameter used in determining the consistency of location data across a multi-sensor environment. This threshold can be established through various methods, each tailored to the specific requirements and conditions of the system. One approach involves conducting empirical studies to analyze historical data from the sensors, identifying typical variations in location measurements under normal operating conditions. By statistically analyzing this data, a threshold can be set at a level that accounts for expected measurement noise while still flagging significant discrepancies. Another method involves simulation-based testing, where the system is subjected to various scenarios that mimic real- world conditions, including potential sources of error such as sensor drift or environmental interference. The results of these simulations can help define a threshold that balances sensitivity to inconsistencies with tolerance for minor deviations. Additionally, the threshold can be dynamically adjusted based on real-time feedback from the system, using machine learning algorithms to continuously refine the threshold as more data is collected and analyzed. This adaptive approach allows the system to respond to changing conditions and maintain high accuracy in localization. By employing these methods, the predefined threshold maximum difference is carefully calibrated to ensure that the classificationof object locations as consistent or inconsistent is both reliable and robust, enhancing the overall performance of the localization system.

[0046] Finally, the method 500 concludes at block 512 by calculating a last known consistent location in the environment for each of the set of objects. In one embodiment, the last known consistent location of objects in a multi- sensor environment is calculated based on a weighted average calculation of two or more real-time localization systems. The weighted average calculation of the last known consistent location can be adjusted based on the reliability of each sensor type, with more weight given to the sensor with historically lower error rates. This step ensures that the most reliable location data is used for further processing and analysis. In exemplary embodiments, the last known consistent location is used as the location of an object to enhance the accuracy and safety of the multi-sensor system.

[0047] In various embodiments, the system for ensuring localization consistency in multi-sensor environments can be adapted to accommodate different types of sensors and configurations. For instance, the first data stream could be generated by a Real-time Localization System (RTLS) using UWB tags, while the second data stream might be produced by an Al-based Vision Detection and Localization system utilizing cameras and image processing algorithms. The system can be configured to classify objects based on their movement characteristics, such as stationary, slow-moving, or fast-moving, and assign a maximum travel speed for each classification. This classification can be dynamically adjusted based on environmental conditions or specific application requirements, such as indoor versus outdoor settings. A threshold maximum travel distance can be determined by considering the frequency of data updates from each sensor type, allowing for flexibility in environments where sensor data is received at varying rates. Additionally, the system can incorporate machine learning algorithms to refine the classification and threshold parameters over time, improving accuracy and adaptability. The monitoring window can be adjusted in size to balance between computational efficiency and the need for detailed temporal analysis, ensuring that the system remains responsive to changes in object movement patterns while maintaining high safety assurance levels.

[0048] In one embodiment, the system for calculating the difference between the first location and the second location of each object utilizes a real-time processing unit that continuously receives and processes data from both the first and second localization systems. This processing unit can be implemented using a high-performance microcontroller or a dedicated digital signal processor (DSP) to ensure rapid computation and minimal latency. In another embodiment, the system employs a distributed computing architecture where data from the localization systems is processed in parallel across multiple nodes in a network, allowing for scalability and redundancy. This setup can be particularly beneficial in environments with a large number of objects to track, such as in a factory or warehouse setting. Additionally, the system may incorporate machine learning algorithms to dynamically adjust the threshold maximum difference based on historical data and environmental conditions, thereby improving accuracy and adaptability. The system can also be configured to operate in various environmental conditions, with sensors capable of functioning in low-light or high-interference scenarios, ensuring robust performance across different settings. Furthermore, the system may include a user interface that provides real-time feedback on the consistency status of object locations, allowing operators to make informed decisions quickly.

[0049] FIG. 6 illustrates an example of a processing system 600 that can be used to implement the computer-based components described herein. The processing system 600 includes an exemplary computing device (“computer”) 602 configured for performing various aspects of the operations described herein in accordance with aspects of the invention. In addition to computer 602, exemplary processing system 600 includes network 614, which connects computer 602 to additional systems (not depicted) and can include one or more wide area networks (WANs) and / or local area networks (LANs) such as the Internet, intranet(s), and / or wireless communication network(s). Computer 602 and the additional system are in communication via network 614, e.g., to communicate data between them. In exemplary embodiments, one or more of the first localization system 114, the second localization system 124, and the consistency verification system 130 may be embodied in a processing system 600.

[0050] Exemplary computer 602 includes processor cores 604, main memory (“memory”) 610, and input / output component(s) 612, which are in communication via bus 603. Processor cores 604 includes cache memory (“cache”) 606 and controls 608, which includebranch prediction structures and associated search, hit, detect and update logic, which will be described in more detail below. Cache 606 can include multiple cache levels (not depicted) that are on or off-chip from processor 604. Memory 610 can include various data stored therein, e.g., instructions, software, routines, etc., which, e.g., can be transferred to / from cache 606 by controls 608 for execution by processor 604. Input / output component(s) 612 can include one or more components that facilitate local and / or remote input / output operations to / from computer 602, such as a display, keyboard, modem, network adapter, etc. (not depicted).

[0051] A cloud computing system 620 is in wired or wireless electronic communication with the processing system 600. The cloud computing system 620 can supplement, support or replace some or all of the functionality (in any combination) of the processing system 600. Additionally, some or all of the functionality of the processing system 600 can be implemented as a node of the cloud computing system 620.

[0052] For the sake of brevity, conventional techniques related to making and using the disclosed embodiments may or may not be described in detail herein. In particular, various aspects of computing systems and specific computer programs to implement the various technical features described herein are well known. Accordingly, in the interest of brevity, many conventional implementation details are only mentioned briefly or are omitted entirely without providing the well-known system and / or process details.

[0053] The various components / modules / models of the systems illustrated herein are depicted separately for ease of illustration and explanation. In embodiments of the invention, the functions performed by the various components / modules / models can be distributed differently than shown without departing from the scope of the various embodiments of the invention describe herein unless it is specifically stated otherwise.

[0054] Aspects of the invention can be embodied as a system, a method, and / or a computer program product at any possible technical detail level of integration. The computer program product may include a computer readable storage medium (or media) having computer readable program instructions thereon for causing a processor to carry out aspects of the present invention.

[0055] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the present invention. As used herein, the singular forms “a”, “an” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms “comprises” and / or “comprising,” when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, element components, and / or groups thereof.

[0056] While the present invention has been described with reference to an exemplary embodiment or embodiments, it will be understood by those skilled in the art that various changes may be made and equivalents may be substituted for elements thereof without departing from the scope of the present invention. In addition, many modifications may be made to adapt a particular situation or material to the teachings of the present invention without departing from the essential scope thereof. Therefore, it is intended that the present invention not be limited to the particular embodiment disclosed as the best mode contemplated for carrying out this present invention, but that the present invention will include all embodiments falling within the scope of the claims.

Claims

CLAIMSWhat is claimed is:

1. A computer-implemented method for ensuring localization consistency in multisensor environments, the method comprising: receiving, from a first localization system, a first data stream that includes a plurality of first data structures, each of the first data structures include a first identification of a set of objects detected in an environment, a first location of each of the set of objects detected in the environment, and a timestamp corresponding to when the first location was captured; receiving, from second first localization system, a second data stream that includes a plurality of a second data structures, each of the second data structures include a second identification of the set of objects detected in the environment, a second location of each of the set of objects detected in the environment, and a timestamp corresponding to when the second location was captured; identifying a correspondence between the first identification of the set of objects and the second identification of the set of objects; for each of the set of objects, calculating a difference between the first location and the second location of each of the set of objects; classifying each of the set of objects as having a consistent location or inconsistent location based on the difference between the first location and the second location and based on a threshold maximum difference; and calculating a last known consistent location in the environment for each of the set of objects.

2. The computer-implemented method of claim 1, wherein the first data stream and the second data stream provide the first data structures and the second data structures at a different frequency.

3. The computer-implemented method of claim 1, wherein the first location of each of the set of objects detected in the environment is determined by a first type of sensor and wherein the second location of each of the set of objects detected in the environment is determined by a second type of sensor, that is different from the first type of sensor.

4. The computer-implemented method of claim 1, further comprising: identifying a monitoring window that includes two or more of the plurality of first data structures and two or more of the plurality of second data structures; identifying a location error in the first data stream based on a determination that a difference between the first location of one of the set of objects in a first of the two or more of the plurality of first data structures and the first location of the one of the set of objects in a second of the two or more of the plurality of first data structures is greater than a threshold maximum travel distance.

5. The computer-implemented method of claim 4, further comprising classifying each of the set of objects and identifying a maximum travel speed for each classification of objects, wherein the threshold maximum travel distance is determined based on a frequency of the first data stream and the classification of the one of the set of objects.

6. The computer-implemented method of claim 1, wherein calculating the difference between the first location and the second location of each of the set of objects includes comparing a most recently received first location to a most recently received second location of each of the set of objects.

7. The computer-implemented method of claim 1, wherein the last known consistent location for each of the set of objects is calculated as a weighted average of a most recently received first location and second location that were classified as having a consistent location.

8. A system for ensuring localization consistency in multi-sensor environments, the system comprising: a processor; a memory coupled to the processor; and one or more computer readable storage media coupled to the processor, the one or more computer readable storage media collectively containing instructions that are executed by the processor via the memory to cause the processor to perform operations comprising: receiving, from a first localization system, a first data stream that includes a plurality of first data structures, each of the first data structures include a first identification of a set of objects detected in an environment, a first location of each of the set of objects detected in the environment, and a timestamp corresponding to when the first location was captured; receiving, from second first localization system, a second data stream that includes a plurality of a second data structures, each of the second data structures include a second identification of the set of objects detected in the environment, a second location of each of the set of objects detected in the environment, and a timestamp corresponding to when the second location was captured; identifying a correspondence between the first identification of the set of objects and the second identification of the set of objects; for each of the set of objects, calculating a difference between the first location and the second location of each of the set of objects; classifying each of the set of objects as having a consistent location or inconsistent location based on the difference between the first location and the second location and based on a threshold maximum difference; and calculating a last known consistent location in the environment for each of the set of objects.

9. The system of claim 8, wherein the first data stream and the second data stream provide the first data structures and the second data structures at a different frequency.

10. The system of claim 8, wherein the first location of each of the set of objects detected in the environment is determined by a first type of sensor and wherein the second location of each of the set of objects detected in the environment is determined by a second type of sensor, that is different from the first type of sensor.

11. The system of claim 8, wherein the operations further comprise: identifying a monitoring window that includes two or more of the plurality of first data structures and two or more of the plurality of second data structures; identifying a location error in the first data stream based on a determination that a difference between the first location of one of the set of objects in a first of the two or more of the plurality of first data structures and the first location of the one of the set of objects in a second of the two or more of the plurality of first data structures is greater than a threshold maximum travel distance.

12. The system of claim 11, wherein the operations further comprise classifying each of the set of objects and identifying a maximum travel speed for each classification of objects, wherein the threshold maximum travel distance is determined based on a frequency of the first data stream and the classification of the one of the set of objects.

13. The system of claim 8, wherein calculating the difference between the first location and the second location of each of the set of objects includes comparing a most recently received first location to a most recently received second location of each of the set of objects.

14. The system of claim 8, wherein the last known consistent location for each of the set of objects is calculated as a weighted average of a most recently received first location and second location that were classified as having a consistent location.

115. A computer program product, the computer program product comprising a non- transitory tangible storage device having program code embodied therewith, the program code executable by a processing system to perform operations comprising: receiving, from a first localization system, a first data stream that includes a plurality of first data structures, each of the first data structures include a first identification of a set of objects detected in an environment, a first location of each of the set of objects detected in the environment, and a timestamp corresponding to when the first location was captured; receiving, from second first localization system, a second data stream that includes a plurality of a second data structures, each of the second data structures include a second identification of the set of objects detected in the environment, a second location of each of the set of objects detected in the environment, and a timestamp corresponding to when the second location was captured; identifying a correspondence between the first identification of the set of objects and the second identification of the set of objects; for each of the set of objects, calculating a difference between the first location and the second location of each of the set of objects; classifying each of the set of objects as having a consistent location or inconsistent location based on the difference between the first location and the second location and based on a threshold maximum difference; and calculating a last known consistent location in the environment for each of the set of objects.

16. The computer program product of claim 16, wherein the first data stream and the second data stream provide the first data structures and the second data structures at a different frequency.

17. The computer program product of claim 16, wherein the first location of each of the set of objects detected in the environment is determined by a first type of sensor and whereinthe second location of each of the set of objects detected in the environment is determined by a second type of sensor, that is different from the first type of sensor.

18. The computer program product of claim 16, wherein the operations further comprise: identifying a monitoring window that includes two or more of the plurality of first data structures and two or more of the plurality of second data structures; identifying a location error in the first data stream based on a determination that a difference between the first location of one of the set of objects in a first of the two or more of the plurality of first data structures and the first location of the one of the set of objects in a second of the two or more of the plurality of first data structures is greater than a threshold maximum travel distance.

19. The computer program product of claim 19, wherein the operations further comprise classifying each of the set of objects and identifying a maximum travel speed for each classification of objects, wherein the threshold maximum travel distance is determined based on a frequency of the first data stream and the classification of the one of the set of objects.

Images