Annotating base map

Abstract

To provide a method of translating between a base map of an industrial facility and a robot specific map utilized by a robot to navigate about the industrial facility.SOLUTION: In the method, first translation data is generated based on comparing a base map and a first robot map, and second translation data is generated based on comparing the base map and a second robot map. Using the first and second translation data, a pose of a first environmental feature detected by the first robot in the first robot map is translated and graphically rendered in a corresponding pose in the base map, and a pose of a second environmental feature in the second robot map is translated and graphically rendered in a corresponding pose in the base map. Association between industrial data received for industrial components observed in the base is associated with detection of an environmental feature in a robot-specific map.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present disclosure relates generally to base map annotation, and more particularly to conversion between a base map of an industrial facility and a robot-specific map utilized by a robot to navigate around the industrial facility. [Background technology]

[0002] Multiple robots may be utilized and deployed in different areas of an industrial facility to perform different missions, such as navigating to different points of interest (POIs), capturing images of the POIs, measuring gas readings or other measurements, monitoring for anomalies, and / or repairing leaks. For example, at a given time, a first robot (e.g., a wheel-driven robot carrying a visual sensor) may be performing a first mission that involves navigating to a first POI (e.g., an open space to monitor for anomalies), capturing corresponding images of the first POI for anomaly detection, and / or monitoring (e.g., for an oil leak). Continuing with this example, at a given time (or a different time), a second robot (e.g., a drone carrying a gas sensor) may be performing a second mission that involves navigating to a second POI (e.g., the enclosed space of a large container) and detecting the presence of a gas based on gas readings of the gas sensor for the second POI.

[0003] When performing a first mission about, or navigating, as the case may be, the industrial facility, a first robot utilizes a first robot map that is specific to the first robot. When performing a second mission about, or navigating, as the case may be, the industrial facility, a second robot utilizes a second robot map that is specific to the second robot. In other words, when performing a mission about, or navigating, as the case may be, the industrial facility, each robot utilizes a map that is specific to that robot. That is, a map may be utilized only by that robot or only by a subset of robots located in the environment, and other robots utilize maps that are specific to those other robots.

[0004] For example, a first robot may be a first model robot from a first manufacturer and may utilize a first robot map generated based on past observations from the first robot, and a second robot may also be a first model robot from the first manufacturer but may utilize a second robot map generated based on past observations from the second robot. Although both robots are the same first model from the same first manufacturer, they may utilize separate maps because they do not utilize any "global" map but instead each generates and utilizes its own unique map.

[0005] As another example, the first robot may be a first model robot from a first manufacturer and may utilize a first robot map generated based on past observations from the first robot, and the second robot may be a second model robot from a second manufacturer and may utilize a second robot map generated based on past observations from the second robot.

[0006] The robot-specific map may be generated in various ways and may take various forms, for example, the robot-specific map may be generated from lidar readings and stored as a point cloud generated based on the lidar readings. Summary of the Invention [Problem to be solved by the invention]

[0007] As mentioned above, in an industrial environment (e.g., an industrial facility) with multiple robots, each robot utilizes a map that is unique to that robot. This can present various challenges. As an example, visual data from a first robot may be processed to detect a first object (e.g., a dynamic object moving around, such as a forklift or a human) in the environment where the first robot resides and / or to detect a first pose of the first object. The first pose of the first object is a first-robot-map eigenpose relative to a first-robot-reference-frame ("first frame") of a first-robot-map utilized by the first robot. Furthermore, visual data from a second robot may be processed to detect a second object in the environment where the second robot resides and / or to detect a second pose of the second object. The second pose of the second object is a second-robot-map eigenpose relative to a second-robot-reference-frame ("second frame") of a second-robot-map utilized by the second robot. The environment in which the first robot resides may or may not overlap with the environment in which the second robot resides. The environment in which the first robot resides and the environment in which the second robot resides may each correspond to a portion of an industrial environment.

[0008] However, because the poses of the first object and the second object are detected in different robot maps having different reference frames, it is not possible to determine the relative pose between the first object and the second object using only the first pose and the second pose. Furthermore, it is not possible to determine a first base map pose for the first object on a base map (e.g., a 3D model of an industrial environment) using only the first pose, or to determine a second base map pose for the second object on the base map using only the second pose. Therefore, the effectiveness of detecting the first object and first pose and the second object and second pose is limited.

[0009] For example, the first object and the second object cannot be correlated with each other and / or rendered in the appropriate location on the representation of the base map (e.g., to allow an operator to observe the positions of the first object and the second object within the industrial facility). As another example, the first object and the second object cannot be correlated to industrial automation data (e.g., automated sensor readings, etc.) defined relative to the base map reference frame ("base frame") of the base map. The base map may be, for example, a base map utilized by humans in monitoring and / or controlling aspects of the industrial facility, robots deployed within the industrial facility, and / or objects or other features detected by such robots. The base map may be generated, for example, from a 3D model of the facility and / or from a point cloud generated based on lidar readings (and / or other readings) from within the facility (e.g., from a lidar-equipped backpack worn by a user around the facility for mapping purposes). The base map may optionally include rich information about various industrial components within the industrial facility, such as the type of component, the installation date of the component, current or recent readings for the component, etc. [Means for solving the problem]

[0010] Implementations disclosed herein relate to translating between each of a plurality of robot-specific maps and a base map of an industrial facility and using those translations to convert robot-specific map poses of detected environmental features (e.g., the first object and / or second object) to base map poses of the detected environmental features. Each of the plurality of robot-specific maps is specific to or utilized by a subset of robots (e.g., a single robot) deployed within the industrial facility.

[0011] For example, a comparison of a base map point cloud (e.g., used to form or generate the base map) and a first robot-specific map point cloud (e.g., used to form or generate the first robot map) may enable (a) determination of a first correspondence between points in the base map and points in the first robot map (sometimes referred to as the “first robot-specific map”), and / or (b) determination of a first transformation function for transforming between points in the base map and points in the first robot map.

[0012] As another example, a comparison of the base map point cloud and a second robot-specific map point cloud (e.g., used to form or generate a second robot map) may enable the determination of (a) a second correspondence between points in the base map and points in the second robot map (sometimes referred to as a “second robot-specific map”), and / or (b) a second transformation function for transforming between points in the base map and points in the second robot map.

[0013] Transforming the robot map pose of robot-detected environmental features to a base map pose may, for example, allow the detected environmental features to be rendered within the representation of the base map at the appropriate location (and / or with the appropriate orientation) within the representation of the base map. Transforming the robot map pose of robot-detected environmental features to a base map pose may additionally or alternatively allow, for example, features from multiple disparate robots to be rendered simultaneously and / or correlated with each other within the representation of the base map.

[0014] Converting the robot map pose of the robot-detected environmental features to a base map pose may additionally or alternatively allow, for example, the detected environmental features to be correlated to industrial automation data that is defined relative to the base map but not to any robot-specific map. For example, the techniques described herein may allow the occurrence of a detection of some equipment near an industrial component to be correlated with the occurrence of an anomalous sensor reading associated with the industrial component. The anomalous sensor reading may be defined relative to a base map of the industrial environment (e.g., the pose of the industrial component to which the sensor reading corresponds may be defined within the base map).

[0015] In various implementations, a method implemented using one or more processors is provided, the method including identifying a base map for an industrial environment and a first robot map utilized by a first robot deployed within the industrial environment, the base map being different from the first robot map. The industrial environment may be or may include an industrial facility in which the first robot is deployed. In some implementations, the first robot map may be generated based on observations of the industrial environment (or a portion thereof) by the first robot. For example, the first robot map may be generated based on observations by the first robot along one or more planned paths within the industrial environment.

[0016] In some implementations, the base map may be generated based on observations of the industrial environment (e.g., in its entirety) using sensors (e.g., Lidar sensors) carried around the industrial environment. For example, the base map may be a 3D base map point cloud rendered using sensor readings of the sensors. In some implementations, the base map may be a 3D model of the industrial environment, such as a 3D floor plan. In some implementations, the base map may be a 2D model of the industrial environment that has been converted from the 3D model.

[0017] In some implementations, identifying the base map and the first robot map for the industrial environment may optionally be in response to receiving user input from a user requesting to monitor the industrial environment.

[0018] In some implementations, the first robot map is generated from sensor readings of one or more sensors of the first robot. The first robot map may be, for example, a first robot map point cloud (e.g., 3D) collected using one or more sensors (e.g., which may be movably mounted to the first robot). In some implementations, the first robot map is stored locally on the first robot. In some implementations, additionally or alternatively, the first robot map is stored on one or more server devices in communication with the first robot.

[0019] In various implementations, the method may further include generating first transformation data (and / or first correspondence data) based on comparing the base map and the first robot map. As a non-limiting example, the base map may take the form of a 3D base map point cloud (sometimes simply referred to as a "base map point cloud"), and the first robot map may take the form of a 3D first robot-specific map point cloud (sometimes simply referred to as a "first robot-specific map point cloud" or a "first robot map point cloud"). In this non-limiting example, points in the base map point cloud may be compared to points in the first robot-specific map point cloud to determine first transformation data (and / or first correspondence data) between points in the first robot-specific map and points in the base map.

[0020] The first transformation data may include, for example, a first transformation function applicable to transform any robot point in the first robot map to a corresponding base point in the base map. The first correspondence data may include, for example, a mapping relationship between one or more points in the first robot map and one or more corresponding points in the base map.

[0021] For example, given a point (X1, Y1, Z1) in the first robot map, the first transformation data may indicate that a point (X1+4.5, Y1+3.5, Z1) in the base map corresponds to a point (X1, Y1, Z1) in the first robot map. For example, given a point (X1, Y1, Z1) in the first robot map, the first correspondence data may indicate that a point (X1', Y1', Z1') in the base map corresponds to or maps to a point (X1, Y1, Z1) in the first robot map.

[0022] In various implementations, the method may further include receiving a first environmental feature and a first robot map pose about the first environmental feature within a first frame of the first robot map. In these implementations, the first feature and the first robot map pose may be determined based on processing first sensor data that is within the first frame and detected by the first robot. The first environmental feature may be or correspond to a first object in an industrial environment, such as a forklift or a human, for example. The first environmental feature may be detected by the first robot based on processing the first sensor data (e.g., lidar data) within the first frame of the first robot map. In some implementations, the first sensor data (e.g., lidar data) may be further processed to determine a first robot map pose (location and / or orientation) about the first environmental feature within the first frame of the first robot map.

[0023] In some implementations, the first feature and the first robot map pose can be determined based on processing the first sensor data using a machine learning model. In some implementations, additional first sensor data different from the first sensor data can be applied instead of the first sensor data to detect the first environmental feature or to determine the first robot map pose for the first environmental feature in the first frame. In other words, different or partially different sensor data can be optionally captured and utilized to detect the first environmental feature and to determine the first robot map pose for the first environmental feature in the first frame.

[0024] As a non-limiting example, the first frame may include a reference point at the origin of the first landmark and may include three reference points, each at one unit distance along a coordinate axis (e.g., X, Y, and Z). In this non-limiting example, a first robot pose (e.g., X1, Y1, Z1) of the first robot may be determined with respect to the first landmark, and a first robot map pose (e.g., X1+m, Y1+n, Z1) with respect to a first environmental feature may be determined based on the relative pose between the first robot and the first environmental feature in the first frame.

[0025] In various implementations, the method may further include using the first transformation data to convert a first robot map pose of a first environmental feature (i.e., in a first frame of the first robot map) to a first base map pose of the first environmental feature (i.e., in the base map). Continuing with the non-limiting example above, given a first robot map pose (e.g., X1+m, Y1+n, Z1) for a first environmental feature in the first frame, the first transformation data may be utilized to determine that a first base map pose (e.g., X1+m+4.5, Y1+n+3.5, Z1) in the base map corresponds to the first robot map pose (e.g., X1+m, Y1+n, Z1) in the first frame.

[0026] In some implementations, the first environmental feature corresponds to a movable object that is movable within the industrial environment. For example, the first environmental feature may correspond to a movable object (e.g., a forklift or a human operator) that is movable relative to a fixed component within the industrial environment.

[0027] In various implementations, the method may further include receiving industrial data and a given map attitude (e.g., in a base map) corresponding to the industrial data. In some implementations, the industrial data may be received from a fixed component (or an industrial component that remains unchanged or fixed for a period of time) in the industrial environment. In some implementations, the industrial data includes, for example, an abnormal sensor reading. The abnormal sensor reading may be provided, for example, by a fixed component (e.g., a fixed sensor) in the industrial environment. In some implementations, the industrial data including the abnormal sensor reading may include, or may be received in, an alert message. In some implementations, the given map attitude corresponding to the industrial data may include, for example, a location and / or orientation that is defined / determined in the base map relative to the fixed component (e.g., a fixed sensor).

[0028] The industrial data (e.g., anomaly sensor readings) and / or the given map pose may be transmitted, for example, over one or more networks. Alternatively or additionally, the industrial data may be transmitted over one or more networks along with an identifier of a fixed component (e.g., a fixed sensor) with which the industrial data is associated. The identifier of the fixed component (e.g., a fixed sensor) may allow the given map pose to be derived when the industrial data is captured using the fixed sensor.

[0029] In various implementations, the method may further include determining that the industrial data corresponds to a first environmental feature. In some implementations, determining that the industrial data corresponds to the first environmental feature may be based on determining that the industrial data corresponds in time to a detection of the first environmental feature and determining that a first base map pose for the first environmental feature corresponds in position to a given map pose for the industrial data.

[0030] In some implementations, determining that the industrial data corresponds in time to the detection of the first environmental feature includes determining that an industrial data timestamp for the industrial data satisfies a temporal threshold for the environmental feature timestamp for the first environmental feature, the temporal threshold being based on one or more times associated with the first sensor data. In some implementations, the temporal threshold may be determined based on an industrial data type of the industrial data and / or an environmental feature type of the first environmental feature. For example, a first temporal threshold (e.g., 0.1 seconds) may be determined when the first environmental feature is a forklift and the industrial data is of a first sensor type. A second temporal threshold (different from the first threshold, e.g., 0.2 seconds) may be determined when the first environmental feature is a forklift and the industrial data is of a second sensor type (different from the first sensor type). A third threshold (different from the first threshold and / or the second threshold, e.g., 2 seconds) may be determined when the first environmental feature is a maintenance worker and the industrial data is of a third sensor type.

[0031] In some implementations, determining that the first base map pose for the first environmental feature positionally corresponds to the given map pose for the industrial data may include determining that the first base map pose and the given map pose satisfy a distance threshold. The distance threshold may be different or may be determined based on an industrial data type of the industrial data and / or an environmental feature type of the first environmental feature. For example, a first distance threshold may be determined when the first environmental feature is a maintenance worker, and a second distance threshold may be determined when the first environmental feature is a forklift, where the first threshold may be less than the second threshold.

[0032] In various implementations, the method may further include using the determined correspondence of the industrial data to the first characteristic in adapting the industrial process. In some implementations, using the determined correspondence of the industrial data to the first characteristic in adapting the industrial process may include adapting one or more parameters for operating a component used in the industrial process (e.g., to which a fixed sensor is attached or an anomalous sensor reading is obtained using a fixed sensor) based on the type of the first environmental characteristic and / or the type of industrial data. As a non-limiting example, adapting the industrial process may include pausing the industrial process if the type of the first environmental characteristic indicates that the first environmental characteristic is a liquid leak and the type of industrial data is low pressure. In some implementations, adapting the industrial process may include, for example, adapting the industrial process via the first environmental characteristic if, for example, the type of the first environmental characteristic indicates that the first environmental feature is a maintenance crew or a repair tool / robot.

[0033] In various implementations, an additional method implemented using one or more processors is provided, the method including identifying a base map for an industrial environment, a first robot map utilized by a first robot located within the industrial environment, and a second robot map utilized by a second robot located within the industrial environment, wherein the base map, the first robot map, and the second robot map may all be different from one another.

[0034] The industrial environment may be or may include an industrial facility in which the first robot and the second robot are located. The first robot and the second robot may be of the same type or different types. The first robot and the second robot may be manufactured by the same manufacturer or different manufacturers. In some implementations, even if the first robot and the second robot are of the same type and manufactured by the same manufacturer, the first robot and the second robot may generate and utilize different robot-specific maps (i.e., a first robot map generated based on observations of the industrial environment by the first robot along a first path, and a second robot map generated based on observations of the industrial environment by the second robot along a second path different from the first path).

[0035] In some implementations, the first robot map may be for a first region of an industrial facility (e.g., including one or more landmarks in the first region from which the first robot determines its pose within the first region) and may correspond to a first portion of the base map. The second robot map may be for a second region of the industrial facility and may correspond to a second portion of the base map. The first portion may or may not overlap with the second portion. Note that even if the first region coincides with the second region, the first robot map may still differ from the second robot map because the first and second robot maps may have different origins and / or coordinate systems. In this case, the relative pose between the first robot and the second robot cannot be determined given only the first robot pose determined for the first robot in the first robot map and the second robot pose determined for the second robot in the second robot map. Furthermore, since the correspondence between the first (or second) robot and the base map remains undetermined, a first robot base pose cannot be determined for the first robot within the base map, and a second robot base pose cannot be determined for the second robot within the base map.

[0036] To address the above concerns, in various implementations, the additional method further includes generating first transformation data (and / or first correspondence data) based on comparing the base map and the first robot map, and generating second transformation data (and / or second correspondence data) based on comparing the base map and the second robot map.

[0037] In some implementations, the first transformation data includes a first transformation function that can be applied to transform any point in the first robot map to a corresponding point in the base map. In some implementations, the second transformation data includes a second transformation function that is separate from the first transformation function and that can be applied to transform any point in the second robot map to a corresponding point in the base map.

[0038] As a non-limiting example, the base map may take the form of a 3D base map point cloud (sometimes referred to as a "base map point cloud") and the first robot map may take the form of a robot map point cloud. In this non-limiting example, points in the base map point cloud may be compared to points in the first robot-specific map point cloud to determine first transformation data (and / or first correspondence data) between points in the first robot-specific map and points in the base map. Similarly, points in the base map point cloud may be compared to points in a second robot-specific map point cloud to determine second transformation data (and / or second correspondence data) between points in the second robot map and points in the base map.

[0039] The first transformation data may include, for example, a first transformation function that may be applied to transform any robot point in the first robot map to a corresponding base point in the base map. The first correspondence data may include an explicit mapping relationship between one or more points in the first robot map and one or more corresponding points in the base map. The second transformation data may include a second transformation function that is different from the first transformation function and that may be applied to transform any robot point in the second robot map to a corresponding base point in the base map. The second correspondence data may include an explicit mapping relationship (e.g., a mapping matrix) between one or more points in the second robot map and one or more corresponding points in the base map. It should be noted that using the first (or second) transformation function and / or the first (or second) correspondence data, a specified point in the base map can also be transformed to a specific point in the first (or second) robot map.

[0040] In various implementations, the additional method further includes receiving a first environmental feature and a first robot map pose for the first environmental feature in a first frame of a first robot map, where the first environmental feature and the first robot map pose are determined based on processing first sensor data in the first frame and sensed by the first robot; and receiving a second environmental feature and a second robot map pose for the second environmental feature in a second frame of a second robot map, where the second environmental feature and the second robot map pose are determined based on processing second sensor data in the second frame and determined by the second robot.

[0041] In some implementations, the first environmental feature corresponds to a first object in the industrial environment. In some implementations, the second environmental feature corresponds to a second object in the industrial environment.

[0042] In some implementations, the first features and the first robot map pose are determined based on processing the first sensor data using a first machine learning model. In some implementations, the second features and the second robot map pose are determined based on processing the second sensor data using a second machine learning model different from the first machine learning model. The first sensor data may be lidar data detected using a first lidar sensor carried by the first robot. The second sensor data may be additional lidar data detected using a second lidar sensor carried by the second robot.

[0043] In some implementations, the first robot map is generated from sensor readings of one or more sensors (e.g., including the first Lidar sensor described above) of the first robot. In some implementations, the second robot map is generated from sensor readings of one or more sensors (e.g., including the second Lidar sensor described above) of the second robot.

[0044] In various implementations, the additional method further includes converting the first robot map pose to a first base map pose (i.e., in the base map) using the first transformation data, and converting the second robot map pose to a second base map pose (e.g., in the base map) using the second transformation data.

[0045] In various implementations, the additional method further includes causing the base map to be rendered at a first base map pose with a first graphical representation of the first environmental feature, and at a second base map pose with a second graphical representation of the second environmental feature.

[0046] In some implementations, the base map is further rendered with a graphical representation of industrial data included within the base map but absent from the first robot map and absent from the second robot map. The graphical representation of the industrial data can indicate the type of component associated with the industrial data, installation data for the component, current or recent readings related to the component, etc.

[0047] In various implementations, a further method implemented using one or more processors is provided, the method including identifying a base map for an industrial environment and a first robot map utilized by a first robot positioned within the industrial environment, wherein the base map is different from the first robot map.

[0048] In various implementations, a further method further includes generating first transformation data based on comparing the base map and the first robot map; receiving a first environmental feature associated with equipment in an industrial environment and a first robot map pose for the first environmental feature within a first frame of the first robot map, where the first environmental feature and the first robot map pose are determined based on processing first sensor data within the first frame and sensed by the first robot; converting the first robot map pose into a first base map pose using the first transformation data; receiving industrial data indicative of an abnormal condition and a given map pose corresponding to the industrial data; determining that the industrial data corresponds to the first environmental feature; and storing the determined correspondence of the industrial data to the first feature in a database accessible within the industrial environment.

[0049] Additionally, some implementations include one or more processors of one or more computing devices, the one or more processors operable to execute instructions stored in associated memory, the instructions configured to cause any of the aforementioned methods to be performed. Some implementations also include one or more non-transitory computer-readable storage media having stored thereon computer instructions executable by the one or more processors to perform any of the aforementioned methods.

[0050] It should be appreciated that all combinations of the foregoing concepts and additional concepts described in more detail herein are contemplated as being part of the presently disclosed subject matter, for example, all combinations of claimed subject matter appearing at the end of this disclosure are contemplated as being part of the presently disclosed subject matter. [Brief explanation of the drawings]

[0051] [Figure 1A] FIG. 1 illustrates a schematic diagram of an exemplary environment in which selected aspects of the present disclosure may be implemented, according to various implementations. [Figure 1B] FIG. 1 illustrates a schematic diagram of another exemplary environment in which selected aspects of the present disclosure may be implemented, according to various implementations. [Figure 1C] 1A-1C illustrate example user interfaces showing a base map without representations of objects detected by a corresponding robot, according to various implementations. [Figure 1D] 1A-1C illustrate example user interfaces showing a base map rendered with corresponding robot-detected objects according to various implementations. [Figure 1E] FIG. 10 illustrates another example of a user interface showing a base map rendered with corresponding robot-detected objects, according to various implementations. [Figure 2] 1A-1C illustrate exemplary methods for carrying out selected aspects of the present disclosure, according to various implementations. [Figure 3]FIG. 1 illustrates another exemplary method for carrying out selected aspects of the present disclosure, according to various implementations. [Figure 4] FIG. 1 illustrates a schematic diagram of an exemplary computer architecture in which selected aspects of the present disclosure may be implemented. DETAILED DESCRIPTION OF THE INVENTION

[0052] Implementations disclosed herein relate to transforming between each of one or more robot-specific maps and a base map of an industrial facility and using those transformations to transform robot-specific map poses of detected environmental features to base map poses of detected environmental features. The one or more robot-specific maps may each be specific to and utilized by a subset of robots (e.g., a single robot) deployed in the industrial facility. For example, the one or more robot-specific maps may include a first robot-specific map (sometimes referred to as a “first robot map”) utilized by a first robot deployed in the industrial facility and / or a second robot-specific map (sometimes referred to as a “second robot map”) utilized by a second robot deployed in the industrial facility.

[0053] In some implementations, the base map may be, or may in some cases be generated from, a base map point cloud. In some implementations, the first robot map may be (or may in some cases be generated from) a first robot-specific map point cloud (sometimes referred to as a “first robot map point cloud”). A comparison of the base map point cloud and the first robot-specific map point cloud may enable determination of a first correspondence (e.g., an explicit mapping) between points in the base map and points in the first robot-specific map. Alternatively or additionally, a comparison of the base map point cloud and the first robot-specific map point cloud may enable determination of a first transformation function for transforming between points in the base map and points in the first robot map.

[0054] In some implementations, the second robot map may be (or may in some cases be generated from) a second robot-specific map point cloud (sometimes referred to as a "second robot map point cloud"). A comparison of the base map point cloud and the second robot map point cloud may enable a determination of a second correspondence between points in the base map and points in the second robot map. Additionally or alternatively, a comparison of the base map point cloud and the second robot map point cloud may enable a determination of a second transformation function for transforming between points in the base map and points in the second robot map.

[0055] Using a first correspondence and / or a first transformation function, a robot map pose of a first object (e.g., a human operator or a movable object such as a forklift) detected by a first robot may be transformed from the first robot map to the base map. Using a second correspondence and / or a second transformation function, a robot map pose of a second object (e.g., a human operator or a movable object such as a forklift) detected by a second robot may be transformed from the second robot map to the base map. Transforming the robot map pose of a robot-detected environmental feature (e.g., a first object or a second object) to a base map pose may, for example, enable the detected environmental feature to be rendered in the representation of the base map at the appropriate location (and / or appropriate orientation) within the representation of the base map.

[0056] Converting the robot map pose of the robot-detected environmental features to a base map pose may additionally or alternatively allow, for example, different environmental features (e.g., a first object and a second object) detected by multiple heterogeneous robots to be rendered simultaneously and / or correlated with each other within a representation of the base map, in which case the relative pose (e.g., position and / or orientation) between the first object and the second object may be accurately determined or visually perceived.

[0057] Converting the robot map pose of a robot-detected environmental feature to a base map pose may additionally or alternatively allow the robot-detected environmental feature to be correlated to industrial automation data (e.g., low pressure) defined for the base map but not for any robot-specific map. For example, the occurrence of a detection of a piece of equipment by a robot (e.g., a first robot) near an industrial component in a first robot map may be correlated with the occurrence of an anomalous sensor reading associated with the industrial component in the base map. As another example, the occurrence of a detection of an abnormal condition (e.g., a leak) by a particular robot (e.g., a first robot navigating the interior or exterior of a pipe to monitor for leaks) near an industrial component in a first robot map may be correlated with the occurrence of an anomalous sensor reading (e.g., low pressure) associated with the industrial component in the base map. In this latter example, if a subsequent anomalous sensor reading related to the same industrial component (in the base map) is received, the particular robot may be sent to the pose at which the leak was previously detected to inspect any current leaks, and / or additional robots may be sent to poses to repair the leaks.

[0058] By utilizing the transformation or correspondence determined between the base map and the first robot map (or the second robot map), representations of dynamic objects detected by one or more robots located in the industrial facility can be dynamically rendered in the base map. The transformation or correspondence determined between the base map and the first robot map also enables the occurrence of an object or event (e.g., a leak) detected by a robot at a particular pose in the robot-specific map to be associated in the base map with industrial data (e.g., low pressure) automatically received from fixed sensors for industrial structural elements (fixed in the industrial facility). Because the first robot map may not include the industrial components, but the base map shows the poses of the industrial components, the industrial data is reflected in the base map but not in the first robot map.

[0059] FIG. 1A schematically illustrates an exemplary environment in which selected aspects of the present disclosure may be implemented, according to various implementations. Referring now to FIG. 1A, an exemplary environment 100 is shown in which various aspects of the present disclosure may be implemented. The exemplary environment 100 may be or include an industrial facility 130, which may take many forms. The exemplary environment 100 may optionally be designed to implement any number of at least partially automated processes. The industrial facility 130 may take the form of a chemical processing plant, an industrial office environment, an oil or natural gas refinery, a catalyst plant, a manufacturing plant, an offshore oil platform, or any other applicable facility.

[0060] The exemplary environment 100 may include one or more client devices (e.g., local client devices 103-A and 103-B) operably coupled to a process automation network 106 within an industrial facility. Client device 103-A or 103-B may be implemented as a computer (e.g., laptop, desktop, notebook), tablet, robot, smart appliance (e.g., smartphone), messaging device, wearable device (e.g., watch), or any other applicable device. The process automation network 106 may be implemented using various wired and / or wireless communication technologies, including, but not limited to, Institute of Electrical and Electronics Engineers (IEEE) Standard 802.3 (Ethernet), IEEE 802.11 (Wi-Fi), cellular networks such as 3GPP® Long Term Evolution (“LTE”), or other wireless protocols designated as 3G, 4G, 5G, and beyond, and / or other types of communication networks of various types of topologies (e.g., mesh).

[0061] In various implementations, the example environment 100 may include a base map 140, which may be rendered via a display on the client device 103-A (or another device, such as the client device 103-B or the server device 105). The base map 140 may be or may be generated from a 3D model or a 3D point cloud collected using a Lidar sensor carried around the industrial facility 130. For example, the base map 140 may be generated based on Lidar sensor observations for the industrial facility 130 (e.g., the entirety thereof) when there are no mobile robots in the industrial facility 130 (or prior to any robots being deployed in the industrial facility 130). In this case, the base map 140 may lack any representation of any robots in robot pose (i.e., the position and / or orientation of any mobile robots or movable objects detected by the mobile robots). In some implementations, the base map 140 may depict various static components of the industrial facility 130 (e.g., industrial components such as doors, open spaces, windows, walls, tanks, equipment, fixed sensors, etc.) For example, as shown in FIG. 1A , the base map 140 may include or depict a structure A′ and an industrial component 180 (e.g., a pressure gauge).

[0062] Client device 103-A or 103-B may each include an input device and / or an output device for user interaction with base map 140. For example, user input received via the input device of client device 103-B (e.g., a click on a graphical representation in base map 140 representing industrial component 180) may cause industrial data related to industrial component 180 to be rendered graphically in base map 140. The industrial data may include, for example, an ID of industrial component 180 (e.g., a device ID of 555), a status of industrial component 180 (e.g., normal or out of order), a maintenance history of industrial component 180, measurements for industrial component 180, etc.

[0063] The exemplary environment 100 may further include one or more mobile robots. For example, the exemplary environment 100 may include a robot fleet having a first robot 111. The first robot 111 may be a four-legged robot (e.g., a robot dog), a wheel-driven robot, an unmanned aerial vehicle (e.g., a drone), a crawler robot, or any other applicable robot movable in or around an industrial facility. The different robots may be of different types and manufactured by different manufacturers. The different robots may utilize different robot-specific maps (which may, however, all be in the form of 3D point clouds, for example) for navigation and to perform tasks / missions. In some implementations, the robot fleet may include subsets of robots of the same type and / or manufactured by the same manufacturer. In these implementations, for example, when different robots from the subset are deployed or programmed to observe an industrial facility (e.g., 130) along different planned paths, the robot-specific maps generated and / or respectively utilized by corresponding robots from the subset of robots may still differ from one another. This is because, for example, a robot-specific map may be generated based on past observations of the robot, which may vary depending on the path the robot has previously traveled.

[0064] As non-limiting examples, the first robot 111 may be a wall-climbing robot (e.g., to inspect the exterior of a vessel or wall for the industrial facility 130), a crawler robot (e.g., to operate and manipulate certain components of the industrial facility), or a drone (e.g., to inspect the stacks and infrastructure of the industrial facility, or to inspect the interior of a vessel of the industrial facility, etc.) In some implementations, instead of a wall-climbing robot, the first robot 111 may instead be a robot for transporting supplies and products, a robot dog for patrolling and monitoring the industrial facility for anomalies, a spider robot for inspecting the exterior of a pipeline, a snake robot for inspecting the interior of a pipeline, or other type of robot.

[0065] In some implementations, the first robot 111 may include one or more sensors for performing one or more missions. For example, the first robot 111 may include (or, in some cases, be equipped with) a light detection and ranging (lidar) sensor for imaging objects by creating a 3D model of the imaged object. Additionally or alternatively, the first robot 111 may include other sensors, such as a visual sensor, ultrasonic testing immersion transducers for detecting surface irregularities and flaws (e.g., corrosion), one or more gas sensors for detecting the presence and concentration of harmful gases or vapors, and / or a temperature sensor for measuring temperature. The visual sensor may be a monographic camera, a stereographic camera, a thermal camera, or any other applicable visual sensor for capturing one or more images of one or more specific components of the industrial facility 130. The visual sensor may be removably coupled to the first robot 111 or may be integrated with the first robot 111. In some implementations, the visual sensor may change location and / or orientation relative to the first robot 111, for example, by rotation or other movement. As a non-limiting example, the first robot 111 may include front and rear high-resolution cameras removably coupled to the first robot 111.

[0066] In some implementations, the first robot 111 may be optionally positioned in one or more areas of the industrial facility 130. For example, the first robot 111 may be positioned to navigate around the first area of ​​the industrial facility 130. While positioned to navigate around the first area, the first robot 111 may generate a plurality of points reflecting one or more components and / or objects of the first area. For example, the first robot 111 may detect one or more points corresponding to structure A of the industrial facility 130 within a first robot map 141 specific to the first robot 111, and may further detect a moving object 121 (e.g., a human) within the first robot map 141. In some implementations, the first robot map 141 may have a first reference frame 171 (sometimes referred to as the “first frame,” which may or may not be visually rendered), and the robot map pose of the moving object 121 may be determined within the first frame of the first robot map 141.

[0067] The exemplary environment 100 may further include a server computing device 105 (sometimes simply referred to as a "server device"). The server computing device 105 may include a map support engine 1051, a pose conversion engine 1052, a rendering engine 1053, and / or storage 15. In some implementations, the base map 140 may be stored in storage 15. Optionally, in some implementations, the exemplary environment 100 may include two or more base maps (e.g., base maps determined in different years or for different industrial facilities, etc.).

[0068] In various implementations, the map enablement engine 1051 may request access to a first robot map (e.g., stored in the first robot 111) to compare the first robot map to the base map 140. The map enablement engine 1051 may compare the first robot map to the base map 140 and determine that structure A in the first robot map 141 is the same structure as structure A′ in the base map 140. In this case, one or more points corresponding to structure A in the first robot map 141 may be compared to one or more points of structure A′ in the base map 140 to determine first correspondence data (e.g., the first correspondence discussed above) and / or first transformation data.

[0069] The first correspondence data may include, for example, an explicit mapping relationship between one or more 3D points in the first robot map and one or more 3D points in the 3D point cloud used to form the base map 140. The first transformation data may include, for example, a transformation function that transforms one or more points in the first robot map 141 to one or more corresponding points in the 3D point cloud that forms the base map 140, or vice versa.

[0070] In various implementations, the server device 105 (or a client device, e.g., 103-B) may receive a first robot map pose for a first environmental feature (e.g., the aforementioned moving object 121, which may be a walking human operator) in a first frame of a first robot map. The pose transformation engine 1052 may use the first correspondence data and / or the first transformation data to convert the first robot map pose of the first environmental feature (e.g., the moving object 121) in the first robot map 141 to a first base map pose in the base map 140. The rendering engine 1053 may further render a first graphical representation 131 of the first environmental feature (e.g., the moving object 121) in the base map 140 at the first base map pose.

[0071] Note that the first graphical representation 131 of the first environmental feature in the base map 140 may differ in the way that it corresponds to the first environmental feature in the first robot map 141. The first graphical representation 131 may be, for example, an image of the first environmental feature, a symbol characterizing the first environmental feature, etc. In some implementations, the representation of the object in the base map being captured by the first robot may optionally have a particular form (e.g., color, size, bounding box, etc.) specific to the first robot. Optionally, based on the tracking pose of the first environmental feature in the first robot map 141 at different times, the rendering engine 1053 may render the first graphical representation 131 of the first environmental feature (e.g., the moving object 121) in a pose corresponding to the base map 140 at different times, such that the base map 140 is a “dynamic” map for a user to observe and track the movement of the first environmental feature (e.g., the moving object 121) in the industrial facility 130 via the base map 140.

[0072] In some implementations, the server computing device 105 may further include an occurrence association engine 1054. The occurrence association engine 1054 may determine, based on the transformation and / or correspondence between the base map and the first robot map, whether an occurrence of an object or event (e.g., a moving object 121) detected by the first robot 111 (or other robot) at a particular pose (e.g., first robot map pose) in the first robot map 141 is associated with automatically received industrial data (e.g., low-pressure sensor readings) for an industrial component (e.g., a fixed instrument 180) in the base map, such as industrial data that is correlated to the base map but not the first robot map. Note that the base map 140 may be a 2D map (or a 3D map) having a coordinate system 170 defined by an origin, an X′ axis, and a Y′ axis.

[0073] 1B schematically illustrates another exemplary environment 100′ in which selected aspects of the present disclosure may be implemented, according to various implementation forms. As shown in FIG. 1B , environment 100′ may include a robot fleet having a first robot 111 and a second robot 112. Similar to environment 100, environment 100′ may further include one or more client devices (e.g., client device 103-A, client device 103-B) and one or more server devices 105. For clarity, repeated descriptions of the client devices and server devices will be omitted herein.

[0074] 1B , the base map 140 may be rendered via a computing device (e.g., client device 103-B) for users or personnel of the industrial facility 130 to observe or inspect different areas of the industrial facility. The base map 140 may include and may depict various components of the industrial facility 130, such as one or more walls, one or more rooms, and one or more industrial components, such as industrial component 180 (which may often remain fixed or unmoving for a period of time, e.g., weeks or months).

[0075] In some implementations, the first robot 111 may be positioned to navigate around the industrial facility 130 utilizing the first robot map 141, and the second robot 112 may be positioned to navigate around the industrial facility 130 utilizing the second robot map 142. As a non-limiting example, the first robot map 141 may include a representation of structure A within the industrial facility 130. Additionally or alternatively, the first robot map 141 may include a representation of the first robot 111 that can change its pose while navigating around the industrial facility 130. In other words, at different times, the representation of the first robot 111 may be at different locations on the first robot map 141 and may have different orientations. Additionally or alternatively, the first robot map 141 may detect a first environmental feature (e.g., a moving human operator 121) and include a representation of the first environmental feature within the first robot map 141.

[0076] The second robot map 142 may include a representation of structure B within the industrial facility 130. Additionally or alternatively, the second robot map 142 may include a representation (not shown) of the second robot 112, which may change its pose while navigating around the industrial facility 130. In other words, at different times, the representation of the second robot 112 may be at different locations on the second robot map 142 and may have different orientations. Additionally or alternatively, the second robot map 142 may detect a second environmental feature (e.g., a moving forklift 122) and include a representation of the second environmental feature within the second robot map 142.

[0077] In some implementations, the first robot 111 may receive a request providing an update to the first robot detection (e.g., an update to the first robot's 111 current pose and / or the current pose of one or more environmental features detected by the first robot 111 in the first robot map 141), and the second robot 112 may receive another request providing an update to the second robot detection (e.g., an update to the second robot's 112 current pose and / or the current pose of one or more environmental features in the second robot map 142). The request received by the first robot 111 and the other request received by the second robot 112 may be generated based on user input or may be generated automatically (e.g., periodically or at other regular or irregular intervals) by the server device 105. In some implementations, the first robot 111 and / or the second robot 112 may proactively push their current pose to the server device 105 without necessarily first receiving any request from the server device 105. For example, the first robot 111 may actively push its current pose to the server device 105 at least periodically when the first robot 111 is connected to a local area network to which the server device 105 is also connected.

[0078] In response to receiving the request, the first robot 111 may provide the current pose of the first robot 111 and / or the current poses of one or more environmental features (all in the first robot map 141), for example, to the server device 105. In response to receiving a separate request, the second robot 112 may provide the current pose of the second robot 112 and / or the current poses of one or more environments (all in the second robot map 142), for example, to the server device 105. The pose transformation engine 1052 may then access the first correspondence data (and / or the first transformation data) to transform the current pose of the first robot 111 and / or the current poses of the one or more environmental features in the first robot map 141 into corresponding poses in the base map 140. Alternatively or additionally, the pose transformation engine 1052 may access second correspondence data (and / or second transformation data) to transform the current pose of the second robot 112 and / or the current pose of one or more environmental features in the second robot map 142 into a corresponding pose in the base map 140.

[0079] The first transformation data and / or the first correspondence data may be determined, for example, based on identifying that structure A in the first robot map 141 and structure A' in the base map 140 are the same structure. The second transformation data and / or the second correspondence data may be determined based on identifying that structure B in the second robot map 142 and structure B' in the base map 140 are the same structure.

[0080] The rendering engine 1053 may then generate a representation of the first robot 111 and / or one or more environmental features detected in the first robot map 141 in the base map 140. The rendering engine 1053 may then generate a representation of the second robot 112 and / or one or more environmental features detected in the second robot map 142 in the base map 140. In some implementations, the representations of the first robot 111 and the second robot 112 may not be rendered in the base map 140. For example, the user may select or configure not to display the first robot 111 and / or the second robot 112 in the base map 140.

[0081] 1B , representation 132 of the second environmental feature (e.g., the moving forklift 122 detected by the second robot 112) and representation 131 of the first environmental feature (e.g., the moving human operator 121) may be visually rendered within base map 140. In some implementations, representation 131 and representation 132 may be rendered simultaneously. For example, when first robot 111 detects the first environmental feature and second robot 112 detects the second environmental feature, they may be rendered simultaneously. In some implementations, representation 131 and representation 132 may be rendered at different times.

[0082] For example, at a first moment in time, a first environmental feature is detected by the first robot 111, and no environmental features are detected by the second robot 112. At this first moment in time, representation 131 may be rendered in base map 140, e.g., with instruments 180, without representation 132 of the second environmental feature being rendered in base map 140. At a second moment in time, a second environmental feature is detected by the second robot 112, and no environmental features are detected by the first robot 111. At this second moment in time, representation 132 may be rendered in base map 140, e.g., with instruments 180, without representation 131 of the first environmental feature being rendered in base map 140. Note that the representation of the first environmental feature in first robot map 141 may differ from the representation of the first environmental feature in base map 140 in terms of aspects such as orientation, size, color, shape, etc. Alternatively or additionally, the representation of the second environmental feature in the second robot map 142 may differ from the representation of the second environmental feature in the base map 140 in one or more aspects, such as orientation, size, color, shape, etc.

[0083] In some implementations, representation 131 may be rendered in response to the first robot 111 detecting a first environmental feature without the first robot 111 receiving such a request. In some implementations, representation 132 may be rendered in response to the second robot 112 detecting a second environmental feature without the second robot 112 receiving such another request.

[0084] Using the base map 140 rendered in various representations (e.g., representations of the first robot 111 and / or the second robot 112, representations of the first environmental feature and / or the second environmental feature), a relative pose between different environmental features may be determined. Alternatively or additionally, a relative pose between the first (or second) environmental feature and an industrial component (or other structure) of the industrial facility 130 may be determined.

[0085] 1C illustrates a non-limiting example of a user interface 190 showing an unrendered base map 140 without any representations of robots and objects detected by the robots for implementing selected aspects of the present disclosure, according to various implementations. Such a base map may be rendered, for example, when an application providing access to the base map is just launched via the display device 109, or when a user selects not to render any representations of robots located in the environment or objects detected by the robots. As shown in FIG. 1C, the base map 140 may be a “raw” map showing static or fixed components within an industrial environment (e.g., industrial facility 130), including, but not limited to, open spaces E, walls F, doors H, rooms G, and / or fixed sensors (e.g., instruments 180).

[0086] 1D illustrates a non-limiting example of a user interface 191 showing a base map rendered / annotated with objects detected by corresponding robots for practicing selected aspects of the present disclosure, according to various implementations. As shown in FIG. 1D , the base map 140 may include fixed components 180 and may further be dynamically rendered with representations 131 for a moving human 121 detected by a first robot 111 (a non-limiting example of a first environmental feature mentioned above) and representations 132 for a moving forklift 122 detected by a second robot 112 (a non-limiting example of a second environmental feature mentioned above).

[0087] The representation 131 of the moving human 121 detected by the first robot 111 may be, for example, an RGB image captured by a camera of the first robot 111, or may be a symbol (with or without a text description) representing the moving human 121. The representation 132 for the moving forklift 122 detected by the second robot 112 may be, for example, an RGB image captured by a camera of the second robot 112, or may be a symbol (with or without a text description) representing the moving forklift 122. The representation 131 for the moving human 121 may be rendered at a corresponding base map pose (the aforementioned first base map pose for the first environmental feature) determined using techniques described in this disclosure. The representation 131 for the moving human 121 may be rendered at a corresponding base map pose (the aforementioned second base map pose for the second environmental feature) determined using techniques described in this disclosure. In this case, the representations of the first robot 111 and the second robot 112 may not be rendered in the base map 140 for purposes such as saving computing resources and reducing latency.

[0088] 1E illustrates another non-limiting example of a user interface 192 showing a base map 140 rendered with objects detected by corresponding robots for carrying out selected aspects of the present disclosure, according to various implementations. As shown in FIG. 1E, at a particular instant t, the base map 140 may be dynamically rendered with a representation 131 for a moving human 121 detected by a first robot 111. At a particular instant t, the base map 140 may further be rendered with a representation 132 for a moving forklift 122 detected by a second robot 112. The representation 131 may be rendered in response to the first robot 111 detecting the moving human 121 in the first robot map, and the representation 132 may be rendered in response to the second robot 112 detecting the moving forklift 122 in the second robot map.

[0089] In some implementations, base map 140 may include a representation for industrial component 180 at any given moment because component 180 was observed in the point cloud that forms (or is used to generate) base map 140. In some implementations, industrial component 180 may be an instrument, and base map 140 may show industrial data, including, but not limited to, readings of instrument 180, where readings of instrument 180 may be different at different moments.

[0090] As a non-limiting example, at a particular instant t, the base map 140 may indicate an anomalous reading for an instrument 180. The occurrence association engine 1054 may determine that the anomalous reading for the instrument 180 is associated with the moving object 121, which is represented in the base map 140 by the representation 131. Such a determination by the occurrence association engine 1054 may be based on determining that the industrial data (i.e., the anomalous reading) corresponds in time to the detection of a first environmental feature (e.g., both the anomalous reading and the moving object 121 are detected at the particular instant t or within a time threshold, e.g., 0.5 seconds), and determining that a first base map pose for the first environmental feature corresponds in position to a given map pose for the industrial data (e.g., the relative distance between the moving object 121 and the instrument 180 is within a distance threshold, e.g., 0.8 m). The association may then be stored, for example, in storage 15, for later use. For example, based on the stored association, the next time an abnormal reading occurs, the moving object 121 may be identified and a human operator 121 may be sent, for example, to inspect the abnormal reading and adapt the industrial process affected by the abnormal reading.

[0091] In some implementations, alternatively, the representation for the human 121 moving in the first base map pose may be a symbol 151 that, when selected, causes the representation 131 to be rendered (e.g., as an overlay) on the base map 140. In some implementations, the representation for the moving forklift 122 may be a symbol 152 that, when selected, may cause a symbol for the moving human 121 to be rendered (e.g., as an overlay) on the base map 140. In some implementations, optionally, the base map 140 may further include a representation 161 (e.g., a symbol) for the first robot 111 and / or a representation 162 (e.g., a symbol) for the second robot 112, where the symbol 161 is different from the symbol 162. The first robot 111 may be, for example, a drone. The second robot 112 may be, for example, a robotic dog. In some implementations, the representation 161 (eg, a symbol) for the first robot 111 and the representation 162 (eg, a symbol) for the second robot 112 may be omitted from the base map 140.

[0092] It should be noted that the base map 140 may be referred to as a "true map" when it represents the entire industrial environment / facility. The true map, when annotated with objects detected by the robots and / or corresponding robots, may reflect a real-time representation of static and dynamic objects within the industrial facility.

[0093] 2 illustrates an exemplary method 200 for carrying out selected aspects of the present disclosure, according to various implementations. For convenience, the operations of the flowchart are described with reference to a system that performs those operations. This system may include various components of various computer systems, such as one or more components of server computing device 105 (and / or additional computing devices, such as client device 103-A or 103-B). Moreover, although the operations of method 200 are shown in a particular order, this is not meant to be limiting. One or more operations may be reordered, omitted, or added.

[0094] In various implementations, in block 202, the system may identify, for example, by a server such as server computing device 105, a base map for the industrial environment and a first robot map to be utilized by a first robot to be deployed within the industrial environment, where the base map is different from the first robot map.

[0095] The industrial environment may be or may include an industrial facility in which the first robot is deployed. In some implementations, the first robot map may be generated based on observations of the industrial environment or a portion thereof by the first robot. For example, the first robot map may be generated based on observations by the first robot along one or more planned paths in the industrial environment. In some implementations, the base map may be generated based on observations of the industrial environment (e.g., its entirety) using sensors (e.g., Lidar sensors) carried around the industrial environment. The base map may be a base map 3D point cloud (or a 3D model) rendered using sensor readings of the sensors, or a 2D graph (e.g., a floor plan) generated and / or validated using the base map 3D point cloud (or using the 3D model) from the sensor readings.

[0096] In some implementations, the first robot map is generated from sensor readings of one or more sensors of the first robot. The first robot map may be, for example, a first robot map of 3D points collected using one or more sensors movably mounted on the first robot. In some implementations, the first robot map is stored as a point cloud locally on the first robot.

[0097] In various implementations, in block 204, the system may generate, for example, by a server such as server computing device 105, first transformation data (and / or first correspondence data) based on comparing the base map and the first robot map. As a non-limiting example, the base map may take the form of a 3D base map point cloud (sometimes simply referred to as a "base map point cloud"), and the first robot map may take the form of a 3D first robot-specific map point cloud (sometimes simply referred to as a "first robot-specific map point cloud" or "first robot map point cloud"). In this non-limiting example, points in the base map point cloud may be compared to points in the first robot-specific map point cloud to determine first transformation data (and / or first correspondence data) between points in the first robot-specific map and a portion of points in the base map.

[0098] The first transformation data may include, for example, a first transformation function that may be applied to transform any robot point in the first robot map to a corresponding base point in the base map. The first correspondence data may include, for example, a mapping relationship between one or more points in the first robot map and one or more corresponding points in the base map.

[0099] For example, given a point (X1, Y1, Z1) in the first robot map, the first transformation data may indicate that a point (X1+4.5, Y1+3.5, Z1) in the base map corresponds to a point (X1, Y1, Z1) in the first robot map. For example, given a point (X1, Y1, Z1) in the first robot map, the first correspondence data may indicate that a point (X1', Y1', Z1) in the base map corresponds to (e.g., maps to) a point (X1, Y1, Z1) in the first robot map.

[0100] In various implementations, in block 206, the system may receive, for example, by a server, such as the server computing device 105, a first environmental feature and a first robot map pose for the first environmental feature within a first frame of the first robot map. In these implementations, the first feature and the first robot map pose may be determined based on processing first sensor data that is within the first frame and detected by the first robot. The first environmental feature may correspond to a first object in an industrial environment, such as a forklift or a human. The first environmental feature may be detected by the first robot based on processing the first sensor data (e.g., Lidar data) within the first frame of the first robot map. In some implementations, the first sensor data may be further processed to determine a first robot map pose (location and / or orientation) for the first environmental feature within the first frame of the first robot map.

[0101] In some implementations, the first feature and the first robot map pose may be determined based on processing the first sensor data using a machine learning model. In some implementations, instead of the first sensor data, additional first sensor data different from the first sensor data may be applied to detect the first feature or to determine the first robot map pose for the first environmental feature in the first frame. In other words, different or partially different sensor data may be captured and utilized to detect the first feature or to determine the first robot map pose for the first environmental feature in the first frame.

[0102] The first frame may include, for example, a reference point at the origin of the first landmark and three reference points each at one unit distance along a coordinate axis, such as the X-axis, the Y-axis, and the Z-axis. In this example, a first robot pose (e.g., X1, Y1, Z1) of the first robot may be determined with respect to the first landmark, and a first robot map pose (e.g., X1+m, Y1+n, Z1) with respect to the first environmental feature may be determined based on the relative pose between the first robot and the first environmental feature in the first frame.

[0103] In various implementations, in block 208, the system, e.g., by a server such as server computing device 105, may use the first transformation data to convert the first robot map pose (i.e., in the first frame of the first robot map) to a first base map pose (i.e., in the base map). For example, given a first robot map pose (e.g., X1+m, Y1+n, Z1) for a first environmental feature in the first frame, the first transformation data may be utilized to determine that a first base map pose (e.g., X1+m+4.5, Y1+n+3.5, Z1) in the base map corresponds to the first robot map pose (e.g., X1+m, Y1+n, Z1) in the first frame.

[0104] In various implementations, in block 210, the system may receive, by a server such as the server computing device 105, industrial data and a given map pose corresponding to the industrial data. In some implementations, the first environmental feature corresponds to a movable object that is movable relative to fixed components in the industrial environment. In these implementations, the industrial data includes, for example, an anomaly sensor reading for the fixed components in the industrial environment. The given map pose corresponding to the industrial data includes a location and / or orientation defined / determined within the base map.

[0105] The industrial data (e.g., anomalous sensor readings) and / or the given map pose may be transmitted, for example, over one or more networks. Alternatively or additionally, the industrial data may be transmitted over one or more networks along with an identifier of the fixed sensor providing the anomalous sensor reading, where the identifier allows the given map pose to be derived when the industrial data is being captured using the fixed sensor.

[0106] In various implementations, in block 212, the system may determine, for example by a server such as server computing device 105, that the industrial data corresponds to the first environmental feature. In some implementations, determining that the industrial data corresponds to the first environmental feature may be based on determining that the industrial data corresponds in time to a detection of the first environmental feature and determining that a first base map pose for the first environmental feature corresponds in position to a given map pose for the industrial data.

[0107] In some implementations, determining that the industrial data corresponds in time to the detection of the first environmental feature includes determining that an industrial data timestamp for the industrial data satisfies a temporal threshold for an environmental feature timestamp for the first environmental feature, the timestamp being based on one or more times associated with the first sensor data. In some implementations, the temporal threshold may be determined based on an industrial data type of the industrial data and / or an environmental feature type of the first environmental feature. For example, a first threshold may be determined when the first environmental feature is a forklift and the industrial data is of a first sensor type. A second threshold (different from the first threshold) may be determined when the first environmental feature is a forklift and the industrial data is of a second sensor type (different from the first sensor type). A third threshold (different from the first and / or second threshold) may be determined when the first environmental feature is a maintenance worker and the industrial data is of a third sensor type (different from the first and / or second sensor type).

[0108] In some implementations, determining that the first base map pose for the first environmental feature positionally corresponds to the given map pose for the industrial data may include determining that the first base map pose and the given map pose satisfy a distance threshold. The distance threshold may differ or be determined based on an industrial data type of the industrial data and / or an environmental feature type of the first environmental feature. For example, a first distance threshold may be determined when the first environmental feature is a maintenance worker, and a second threshold may be determined when the first environmental feature is a forklift, where the first threshold may be less than the second threshold.

[0109] In various implementations, at block 214, the system may use the determined correspondence of the industrial data to the first characteristic in adapting the industrial process, for example, by a server, such as the server computing device 105. In some implementations, using the determined correspondence of the industrial data to the first characteristic in adapting the industrial process may include adapting one or more parameters for operating components (e.g., the aforementioned fixed components) used in the industrial process.

[0110] 3 illustrates another exemplary method 300 for carrying out selected aspects of the present disclosure, according to various implementations. For convenience, the operations of the flowchart are described with reference to a system that performs those operations. This system may include various components of various computer systems, such as one or more components of server computing device 105 (and / or additional computing devices, such as client device 103-A or 103-B). Moreover, while the operations of method 300 are shown in a particular order, this is not meant to be limiting. One or more operations may be reordered, omitted, or added.

[0111] In various implementations, in block 302, the system may identify, by a server such as server computing device 105, a base map for the industrial environment, a first robot map utilized by a first robot located in the industrial environment, and a second robot map utilized by a second robot located in the industrial environment. The base map, the first robot map, and the second robot map may all be different from one another.

[0112] The industrial environment may be or may include an industrial facility in which the first robot and the second robot are located. The first robot and the second robot may be of the same type or different types. The first robot and the second robot may be manufactured by the same manufacturer or different manufacturers. In some implementations, even if the first robot and the second robot are of the same type and manufactured by the same manufacturer, the first robot and the second robot may generate and utilize different robot-specific maps (i.e., a first robot map generated based on observations of the industrial environment by the first robot and a second robot map generated based on observations of the industrial environment by the second robot).

[0113] In some implementations, the first robot map may be for a first region of the industrial facility (e.g., including one or more landmarks in the first region from which the first robot determines its pose within the first region) and may correspond to a first portion of the base map. The second robot map may be for a second region of the industrial facility and may correspond to a second portion of the base map. The first region may not overlap with the second portion, or may only partially overlap with the second portion. The first portion may not overlap with the second portion, or may only partially overlap with the second portion. Note that even if the first region coincides with the second region, the first robot map may differ from the second robot map because the first and second robot maps have different origins and / or coordinate systems. In this case, the relative pose between the first robot and the second robot cannot be determined given only the first robot pose determined for the first robot in the first robot map and the second robot pose determined for the second robot in the second robot map. Furthermore, because the correspondence between the first (or second) robot and the base map has not been determined, a first robot base pose cannot be determined for the first robot in the base map and a second robot base pose cannot be determined for the second robot in the base map.

[0114] In various implementations, in block 304, the system may generate, by a server, such as the server computing device 105, first transformation data (and / or first correspondence data) based on comparing the base map and the first robot map, and may generate second transformation data (and / or second correspondence data) based on comparing the base map and the second robot map.

[0115] In some implementations, the first transformation data includes a first transformation function that can be applied to transform any point in the first robot map to a corresponding point in the base map, and in some implementations, the second transformation data includes a second transformation function that is separate from the first transformation function and that can be applied to transform any point in the second robot map to a corresponding point in the base map.

[0116] As a non-limiting example, the base map may take the form of a 3D base map point cloud (sometimes simply referred to as the "base map point cloud") and the first robot map may take the form of a 3D first robot-specific map point cloud (sometimes simply referred to as the "first robot-specific map point cloud" or "first robot map point cloud"). In this non-limiting example, points in the base map point cloud may be compared to points in the first robot-specific map point cloud to determine first transformation data (and / or first correspondence data) between points in the first robot-specific map and a portion of points in the base map. Similarly, points in the base map point cloud may be compared to points in the second robot-specific map point cloud to determine second transformation data (and / or second correspondence data) between points in the first robot-specific map and a portion of points in the base map.

[0117] The first transformation data may include, for example, a first transformation function that may be applied to transform any robot point in the first robot map to a corresponding base point in the base map. The first correspondence data may include a mapping relationship between one or more points in the first robot map and one or more corresponding points in the base map. The second transformation data may include a second transformation function that is different from the first transformation function and that may be applied to transform any robot point in the second robot map to a corresponding base point in the base map. The second correspondence data may include a mapping relationship between one or more points in the second robot map and one or more corresponding points in the base map. It should be noted that using the first (or second) transformation function and / or the first (or second) correspondence data, a specified point in the base map may be transformed to a specific point in the first (or second) robot map.

[0118] In various implementations, in block 306, the system may receive, by a server such as the server computing device 105, a first environmental feature and a first robot map pose for the first environmental feature in a first frame of the first robot map, the first environmental feature and the first robot map pose being determined based on processing first sensor data in the first frame and detected by the first robot, and may receive a second environmental feature and a second robot map pose for the second environmental feature in a second frame of the second robot map, the second environmental feature and the second robot map pose being determined based on processing second sensor data in the second frame and detected by the second robot.

[0119] In some implementations, the first environmental feature corresponds to a first object in the industrial environment. In some implementations, the second environmental feature corresponds to a second object in the industrial environment.

[0120] In some implementations, the first feature and the first robot map pose are determined based on processing the first sensor data with a first machine learning model, hi some implementations, the second feature and the second robot map pose are determined based on processing the second sensor data with a second machine learning model different from the first machine learning model.

[0121] In some implementations, the first robot map is generated from sensor readings of one or more first sensors of the first robot, and in some implementations, the second robot map is generated from sensor readings of one or more second sensors of the second robot.

[0122] In various implementations, in block 308, the system may, for example, by a server such as the server computing device 105, convert the first robot map pose to a first base map pose using the first transformation data and convert the second robot map pose to a second base map pose using the second transformation data.

[0123] In various implementations, in block 310, the system may cause a server, such as server computing device 105, to render a base map at a first base map pose with a first graphical representation of a first environmental feature, and at a second base map pose with a second graphical representation of a second environmental feature.

[0124] In some implementations, the base map is further rendered with a graphical representation of industrial data that is included within the base map but is absent from the first robot map and is absent from the second robot map. The graphical representation of the industrial data may indicate the type of component, the installation date of the component, current or recent readings for the component, etc.

[0125] In various implementations, a further method implemented using one or more processors is provided, the method including identifying a base map for an industrial environment and a first robot map utilized by a first robot positioned within the industrial environment, wherein the base map is different from the first robot map.

[0126] In various implementations, a further method further includes generating first transformation data based on comparing the base map and the first robot map; receiving a first environmental feature associated with equipment in an industrial environment and a first robot map pose for the first environmental feature within a first frame of the first robot map, wherein the first environmental feature and the first robot map pose are determined by processing first sensor data within the first frame and sensed by the first robot; converting the first robot map pose into a first base map pose using the first transformation data; receiving industrial data indicative of an abnormal condition and a given map pose corresponding to the industrial data; determining that the industrial data corresponds to the first environmental feature; and storing the determined correspondence of the industrial data to the first feature in a database accessible within the industrial environment.

[0127] 4 is a block diagram of an exemplary computing device 410 that may optionally be utilized to implement one or more aspects of the techniques described herein. The computing device 410 generally includes at least one processor 414 that communicates with several peripheral devices via a bus subsystem 412. These peripheral devices may include, for example, a storage subsystem 424 including a memory subsystem 425 and a file storage subsystem 426, a user interface output device 420, a user interface input device 422, and a network interface subsystem 416. The input and output devices enable user interaction with the computing device 410. The network interface subsystem 416 provides an interface to external networks and is coupled to corresponding interface devices in other computing devices.

[0128] The user interface input devices 422 may include pointing devices such as a keyboard, a mouse, a trackball, a touchpad, or a graphics tablet, a scanner, a touchscreen integrated into a display, a voice input device such as a voice recognition system, a microphone, and / or other types of input devices. In general, use of the term "input device" is intended to include all possible types of devices and methods for inputting information into the computing device 410 or onto a communications network.

[0129] The user interface output devices 420 may include a display subsystem, a printer, a fax machine, or a non-visual display such as an audio output device. The display subsystem may include a cathode ray tube (CRT), a flat panel device such as a liquid crystal display (LCD), a projection device, or some other mechanism for creating a visual image. The display subsystem may also provide a non-visual representation, such as via an audio output device. In general, use of the term "output device" is intended to include all possible types of devices and methods for outputting information from the computing device 410 to a user or to another machine or computing device.

[0130] Storage subsystem 424 stores programming and data structures that provide the functionality of some or all of the modules described herein. For example, storage subsystem 424 may include logic for performing selected aspects of the method of Figure 3 and for implementing the various components shown in Figures 1-2.

[0131] These software modules are generally executed by the processor 414 alone or in combination with other processors. The memory 425 used within the storage subsystem 424 may include several memories, including a main random access memory (RAM) 430 for storing instructions and data during program execution and a read-only memory (ROM) 432 in which fixed instructions are stored. The file storage subsystem 426 may provide persistent storage for program files and data files and may include a hard disk drive, a floppy disk drive with associated removable media, a CD-ROM drive, an optical drive, or a removable media cartridge. Modules that implement the functionality of certain implementations may be stored within the storage subsystem 424 by the file storage subsystem 426 or within other machines accessible by the processor 414.

[0132] The bus subsystem 412 provides a mechanism for allowing the various components and subsystems of the computing device 410 to communicate with each other as intended. Although the bus subsystem 412 is shown schematically as a single bus, alternative implementations of the bus subsystem may use multiple buses.

[0133] Computing device 410 may be of various types, including a workstation, a server, a computing cluster, a blade server, a server farm, or any other data processing system or computing device. Due to the ever-changing nature of computers and networks, the description of computing device 410 shown in Figure 4 is intended only as a specific example to illustrate some implementations. Many other configurations of computing device 410 are possible, having more or fewer components than the computing device shown in Figure 4.

[0134] While several implementations have been described and illustrated herein, various other means and / or structures may be utilized to perform the functions and / or obtain one or more of the results and / or advantages described herein, and each such variation and / or modification is considered to be within the scope of the implementations described herein. More generally, all parameters, dimensions, materials, and configurations described herein are meant to be exemplary, and the actual parameters, dimensions, materials, and / or configurations will depend on the particular application or applications in which the present teachings are used. Those skilled in the art will recognize, or be able to ascertain using no more than routine experimentation, many equivalents to the specific implementations described herein. Accordingly, it should be understood that the foregoing implementations are presented by way of example only, and that, within the scope of the appended claims and equivalents thereto, implementations other than those specifically described and claimed may be practiced. Implementations of the present disclosure are directed to each individual feature, system, article, material, kit, and / or method described herein. In addition, any combination of two or more such features, systems, articles, materials, kits, and / or methods is included within the scope of the present disclosure, if such features, systems, articles, materials, kits, and / or methods are not mutually inconsistent. [Explanation of symbols]

[0135] 15. Storage 100 Environment 100' environment 103-A Local Client Device, Client Device 103-B Local Client Device, Client Device 105 Server devices, server computing devices 106 Process Automation Network 109 Display Devices 111 The First Robot 112 The Second Robot 121 Object, Human Operator, Human 122 Forklift 130 Industrial Facilities 131 First Graphical Representation, Representation 132 Expression 140 basemaps 141 First Robot Map 142 Second Robot Map 151 Symbols 152 symbols 161 Expressions, Symbols 162 Expressions, Symbols 170 Coordinate Systems 171 First Reference Frame 180 Industrial Components, Instruments 190 User Interface 191 User Interface 192 User Interface 200 ways 300 ways 410 Computing Devices 412 Bus Subsystem 414 processor 416 Network Interface Subsystem 420 User Interface Output Device 422 User Interface Input Devices 424 Storage Subsystem 425 Memory Subsystem, Memory 426 File Storage Subsystem 430 Main Random Access Memory (RAM) 432 Read-Only Memory (ROM) 1051 Map Engine 1052 Attitude Change Engine 1053 Rendering Engine 1054 Occurrence Correlation Engine

Claims

1. 1. A method implemented by one or more processors, comprising: identifying a base map for an industrial environment and a first robot map utilized by a first robot located within the industrial environment; the base map is different from the first robot map; generating first transformation data based on comparing the base map and the first robot map; receiving a first environmental feature and a first robot map pose for the first environmental feature within a first frame of the first robot map, the first feature and the first robot map pose being determined based on processing first sensor data within the first frame and sensed by the first robot; converting the first robot map pose to a first base map pose using the first transformation data; receiving industrial data and a given map pose corresponding to the industrial data; determining that the industrial data corresponds to the first environmental characteristic, determining that the industrial data corresponds in time to the detection of the first environmental feature; and determining that the first base map pose for the first environmental feature positionally corresponds to the given map pose for the industrial data; determining that the industrial data corresponds to the first environmental characteristic based on the using the determined correspondence of the industrial data to the first characteristic in adapting an industrial process; and A method comprising:

2. The method of claim 1 , wherein the first environmental feature corresponds to a movable object approaching a fixed component in the industrial environment.

3. The method of claim 2 , wherein the industrial data comprises anomalous sensor readings for the fixed components within the industrial environment.

4. using the determined correspondence of the industrial data to the first characteristic in adapting the industrial process; performing an action on the industrial process involving the fixed component based on a type of the first environmental characteristic.

2. The method of claim 1, comprising:

5. The method of claim 1 , wherein the first feature and the first robot map pose are determined based on processing the first sensor data using a machine learning model.

6. The method of claim 1 , wherein the given map pose is within the base map.

7. The method of claim 1 , wherein the industrial data is transmitted along with the given map pose over one or more networks.

8. The method of claim 1 , wherein the base map includes components at the given map pose, and the industrial data is associated with the components.

9. The method of claim 8 , wherein the industry data includes a type of the component, an installation date of the component, and / or a current or recent reading for the component.

10. 1. A method implemented by one or more processors, comprising: identifying a base map for an industrial environment, a first robot map utilized by a first robot located within the industrial environment, and a second robot map utilized by a second robot located within the industrial environment; the base map, the first robot map, and the second robot map are all different from one another; generating first transformation data based on comparing the base map to the first robot map; generating second transformation data based on comparing the base map to the second robot map; receiving a first environmental feature and a first robot map pose for the first environmental feature within a first frame of the first robot map, the first environmental feature and the first robot map pose being determined based on processing first sensor data within the first frame and sensed by the first robot; receiving a second environmental feature and a second robot map pose for the second environmental feature within a second frame of the second robot map, the second environmental feature and the second robot map pose being determined based on processing second sensor data within the second frame and sensed by the second robot; converting the first robot map pose to a first base map pose using the first transformation data; converting the second robot map pose to a second base map pose using the second transformation data; rendering the base map with a first graphical representation of the first environmental feature at the first base map pose and with a second graphical representation of the second environmental feature at the second base map pose; A method comprising:

11. 11. The method of claim 10, wherein the base map is further rendered with a graphical representation of industrial data included within the base map but absent from the first robot map and absent from the second robot map.

12. The method of claim 10 , wherein the first environmental feature corresponds to a first object within the industrial environment.

13. The method of claim 10 , wherein the second environmental feature corresponds to a second object within the industrial environment.

14. 11. The method of claim 10, wherein the first transformation data includes a first transformation function that can be applied to transform any point in the first robot map to a corresponding point in the base map.

15. 11. The method of claim 10, wherein the second transformation data includes a second transformation function that is separate from the first transformation function and that is applied to transform any point in the second robot map to a corresponding point in the base map.

16. The method of claim 10 , wherein the first feature and the first robot map pose are determined based on processing first sensor data using a first machine learning model.

17. 17. The method of claim 16, wherein the second features and the second robot map pose are determined based on processing the second sensor data using a second machine learning model different from the first machine learning model.

18. The method of claim 10 , wherein the first robot map is generated from sensor readings of one or more first sensors of the first robot.

19. The method of claim 10 , wherein the second robot map is generated from sensor readings of one or more second sensors of the second robot.

20. 1. A method implemented by one or more processors, comprising: identifying a base map for an industrial environment and a first robot map utilized by a first robot located within the industrial environment; the base map is different from the first robot map; generating first transformation data based on comparing the base map and the first robot map; receiving a first environmental feature associated with equipment in the industrial environment and a first robot map pose for the first environmental feature within a first frame of the first robot map, the first environmental feature and the first robot map pose being determined based on processing first sensor data within the first frame and sensed by the first robot; converting the first robot map pose to a first base map pose using the first transformation data; receiving industrial data indicative of an abnormal condition and a given map pose corresponding to the industrial data; determining that the industrial data corresponds to the first environmental characteristic, determining that the industrial data corresponds in time to the detection of the first environmental feature; and determining that the first base map pose for the first environmental feature positionally corresponds to the given map pose for the industrial data; determining that the industrial data corresponds to the first environmental characteristic based on the storing the determined correspondence of the industrial data to the first characteristic in a database accessible within the industrial environment; A method comprising:

Images