Annotation of base map

Abstract

To provide a method of translating between a base map of an industrial facility and a robot specific map utilized by a robot to navigate about the industrial facility.SOLUTION: In the method, first translation data is generated based on comparing a base map and a first robot map, and second translation data is generated based on comparing the base map and a second robot map. Using the first and second translation data, a pose of a first environmental feature detected by the first robot in the first robot map is translated and graphically rendered in a corresponding pose in the base map, and a pose of a second environmental feature in the second robot map is translated and graphically rendered in a corresponding pose in the base map. Association between industrial data received for industrial components observed in the base is associated with detection of an environmental feature in a robot-specific map.SELECTED DRAWING: Figure 3

Description

Technical Field

[0001] The present disclosure generally relates to annotation of base maps, and more specifically, to conversion between a base map of an industrial facility and a robot-specific map utilized by a robot to navigate around the industrial facility.

Background Art

[0002] Multiple robots can be utilized and deployed in different areas of an industrial facility to perform different missions such as navigating to different points of interest (POIs), capturing images of POIs, measuring gas readings or other measurements, monitoring for anomalies, and / or repairing leaks. For example, at a given time, a first robot (e.g., a wheeled robot carrying a vision sensor) may be performing a first mission involving navigating to a first POI (e.g., an empty space for monitoring anomalies), capturing a corresponding image of the first POI for anomaly detection, and / or monitoring (e.g., for oil leaks). Continuing with this example, at a given time (or a different time), a second robot (e.g., a drone carrying a gas sensor) may be performing a second mission involving navigating to a second POI (e.g., an enclosed space of a large container) and detecting the presence of a gas based on the gas reading of the gas sensor for the second POI.

[0003] When performing a first mission about an industrial facility, or, if applicable, navigating its surroundings, the first robot uses a first robot map specific to that first robot. When performing a second mission about an industrial facility, or, if applicable, navigating its surroundings, the second robot uses a second robot map specific to that second robot. In other words, when performing a mission about an industrial facility, or, if applicable, navigating it, each robot uses a map specific to that robot. That is, the map can only be used by that robot, or by a subset of robots placed in that environment, and other robots use maps specific to those other robots.

[0004] For example, the first robot may be a first model robot from a first manufacturer and can utilize a first robot map generated based on past observations of the first robot, and the second robot may also be a first model robot from a first manufacturer but can utilize a second robot map generated based on past observations of the second robot. Both robots are the same first model from the same first manufacturer, but they do not utilize any "global" map; instead, each generates and utilizes its own unique map, so these robots can utilize separate maps.

[0005] As another example, the first robot may be a first model robot from a first manufacturer and can utilize a first robot map generated based on past observations of the first robot, and the second robot may be a second model robot from a second manufacturer and can utilize a second robot map generated based on past observations of the second robot.

[0006] The robot-specific map may be generated in various ways and may take various forms. For example, the robot-specific map may be generated from Lidar readings, or it may be stored as a point cloud generated based on Lidar readings. [Overview of the project] [Problems that the invention aims to solve]

[0007] As mentioned above, in an industrial environment with multiple robots (e.g., an industrial facility), each robot utilizes a map specific to that robot. This can present various challenges. For example, visual data from a first robot may be processed to detect a first object in the environment where the first robot is located (e.g., a moving, dynamic object such as a forklift or a person) and / or to detect the first pose of the first object. The first pose of the first object is the first robot map-specific pose related to the first robot reference frame ("first frame") of the first robot map used by the first robot. Furthermore, visual data from a second robot may be processed to detect a second object in the environment where the second robot is located and / or to detect the second pose of the second object. The second pose of the second object is the second robot map-specific pose related to the second robot reference frame ("second frame") of the second robot map used by the second robot. The environment in which the first robot resides may or may not overlap with the environment in which the second robot resides. The environments in which the first robot resides and the environments in which the second robot resides may each correspond to a part of an industrial environment.

[0008] However, since the poses of the first and second objects are detected within different robot maps having different reference frames, it is not possible to determine the relative pose between the first and second objects using only the first and second poses. Furthermore, it is not possible to determine the first basemap pose of the first object on the base map (e.g., a 3D model of an industrial environment) using only the first pose, or to determine the second basemap pose of the second object on the base map using only the second pose. Therefore, the effectiveness of detecting the first object and its first pose, as well as the second object and its second pose, is limited.

[0009] For example, the first object and the second object cannot be correlated with each other and / or rendered within appropriate locations on the base map representation (for example, to allow an operator to observe the locations of the first object and the second object within the industrial facility). As another example, the first object and the second object cannot be correlated with industrial automation data (e.g., automated sensor readings) defined against the base map reference frame ("base frame") of the base map. The base map may be, for example, a base map used by humans when monitoring and / or controlling the appearance of an industrial facility, robots placed within the industrial facility, and / or objects or other features detected by such robots. The base map may be generated, for example, from a 3D model of the facility and / or from a point cloud generated from within the facility based on Lidar readings (and / or other readings) (e.g., from a Lidar-equipped backpack worn by a user around the facility for mapping purposes). The base map may optionally contain rich information about various industrial components within the industrial facility, such as the type of component, the installation date of the component, and current or recent readings for the component. [Means for solving the problem]

[0010] The implementations disclosed herein relate to translating between each of a plurality of robot-specific maps and a base map of an industrial facility, and using those translations to convert the robot-specific map orientation of a detected environmental feature (e.g., the first and / or second object described above) to the base map orientation of the detected environmental feature. Each of the plurality of robot-specific maps is specific to or utilized by a subset of robots (e.g., a single robot) deployed within the industrial facility.

[0011] For example, comparing a basemap point cloud (used, for example, to form or generate a basemap) with a first robot-specific map point cloud (used, for example, to form or generate a first robot map) may enable (a) determination of a first correspondence between points in the basemap and points in the first robot map (sometimes referred to as the "first robot-specific map"), and / or (b) determination of a first transformation function for transforming between points in the basemap and points in the first robot map.

[0012] As another example, comparing a base map point cloud with a second robot-specific map point cloud (for example, used to form or generate a second robot map) may allow for the determination of (a) a second correspondence between points in the base map and points in the second robot map (sometimes referred to as the "second robot-specific map"), and / or (b) a second transformation function for transforming between points in the base map and points in the second robot map.

[0013] Converting robot map poses of detected environment features to base map poses may, for example, allow detected environment features to be rendered within the base map representation in the appropriate location (and / or orientation) within the base map representation. Converting robot map poses of detected environment features to base map poses may, as an addition or alternative, allow features from multiple heterogeneous robots to be rendered simultaneously within the base map representation and / or correlated with each other.

[0014] Converting robot map poses of detected environmental features to base map poses may, as an addition or alternative, allow for correlation of detected environmental features with industrial automation data that is defined for the base map but not for any robot-specific map. For example, the technique described herein may allow correlation of the occurrence of detection of a device near an industrial component with the occurrence of an anomaly sensor reading associated with that industrial component. The anomaly sensor reading may be defined for the base map of the industrial environment (for example, the pose of the industrial component to which the sensor reading corresponds may be defined within the base map).

[0015] Various implementations provide a method implemented using one or more processors, which includes the step of identifying a base map of an industrial environment and a first robot map used by a first robot placed within the industrial environment, wherein the base map is different from the first robot map. The industrial environment may be or include the industrial facility where the first robot is located. In some implementations, the first robot map may be generated based on observations of the industrial environment (or a portion thereof) by the first robot. For example, the first robot map may be generated based on observations by the first robot along one or more planned paths within the industrial environment.

[0016] In some implementations, the base map may be generated based on observations of the industrial environment (e.g., the entire environment) using portable sensors (e.g., Lidar sensors) around the industrial environment. For example, the base map may be a 3D base map point cloud rendered using the sensor readings of the sensors. In some implementations, the base map may be a 3D model of the industrial environment, such as a 3D floor plan. In some implementations, the base map may be a 2D model of the industrial environment converted from the 3D model.

[0017] In some implementations, the step of identifying a base map and a first robot map for the industrial environment may optionally be in response to receiving user input from a user requesting to monitor the industrial environment.

[0018] In some implementations, the first robot map is generated from sensor readings of one or more sensors on the first robot. The first robot map may be, for example, a first robot map point cloud (e.g., 3D) collected using one or more sensors (e.g., which may be movably mounted on the first robot). In some implementations, the first robot map is stored locally on the first robot. In some implementations, as an addition or alternative, the first robot map is stored on one or more server devices communicating with the first robot.

[0019] In various implementations, this method may further include the step of generating first transformation data (and / or first correspondence data) based on a comparison of a base map and a first robot map. In a non-limiting example, the base map may take the form of a 3D base map point cloud (sometimes simply called the “base map point cloud”), and the first robot map may take the form of a 3D first robot-specific map point cloud (sometimes simply called the “first robot-specific map point cloud”) or “first robot map point cloud”). In this non-limiting example, points in the base map point cloud may be compared to points in the first robot-specific map point cloud to determine first transformation data (and / or first correspondence data) between points in the first robot-specific map and points in the base map.

[0020] The first transformation data may include, for example, a first transformation function applicable to transform any robot point in the first robot map into a corresponding base point in the base map. The first correspondence data may include, for example, mapping relationships between one or more points in the first robot map and one or more corresponding points in the base map.

[0021] For example, assuming a point (X1, Y1, Z1) in the first robot map, the first transformation data may indicate that a point (X1+4.5, Y1+3.5, Z1) in the base map corresponds to a point (X1, Y1, Z1) in the first robot map. For example, assuming a point (X1, Y1, Z1) in the first robot map, the first correspondence data may indicate that a point (X1', Y1', Z1') in the base map corresponds to or is mapped to a point (X1, Y1, Z1) in the first robot map.

[0022] In various implementations, this method may further include the step of receiving a first environmental feature and a first robot map orientation for the first environmental feature, which is located within a first frame of the first robot map. In these implementations, the first feature and the first robot map orientation may be determined based on processing first sensor data, which is located within a first frame and detected by the first robot. The first environmental feature may be a first object in an industrial environment, such as a forklift or a person, or may correspond to a first object. The first environmental feature may be detected by the first robot based on processing first sensor data (e.g., Lidar data) located within a first frame of the first robot map. In some implementations, the first sensor data (e.g., Lidar data) may be further processed to determine the first robot map orientation (location and / or orientation) for the first environmental feature in a first frame of the first robot map.

[0023] In some implementations, the first feature and the first robot map pose may be determined based on processing the first sensor data using a machine learning model. In some implementations, additional first sensor data different from the first sensor data may be applied instead of the first sensor data to detect the first environmental feature or to determine the first robot map pose for the first environmental feature in the first frame. In other words, different or partially different sensor data may be optionally captured and used to detect the first environmental feature and to determine the first robot map pose for the first environmental feature in the first frame.

[0024] As a non-limiting example, the first frame may include a reference point at the origin of the first landmark and may include three reference points each at a unit distance along a coordinate axis (e.g., the X-axis, the Y-axis, and the Z-axis). In this non-limiting example, the first robot pose (e.g., X1, Y1, Z1) of the first robot can be determined for the first landmark, and the first robot map pose (e.g., X1+m, Y1+n, Z1) for the first environmental feature can be determined based on the relative pose between the first robot and the first environmental feature within the first frame.

[0025] In various implementations, this method may further include converting, using the first conversion data, the first robot map pose of the first environmental feature (i.e., within the first frame of the first robot map) to the first base map pose of the first environmental feature (i.e., within the base map). Continuing with the above non-limiting example, assuming a first robot map pose (e.g., X1+m, Y1+n, Z1) for the first environmental feature within the first frame, the first conversion data can be utilized to determine that a first base map pose (e.g., X1+m+4.5, Y1+n+3.5, Z1) within the base map corresponds to the first robot map pose (e.g., X1+m, Y1+n, Z1) within the first frame.

[0026] In some implementations, the first environmental feature corresponds to a movable object that is movable within the industrial environment. For example, the first environmental feature can correspond to a movable object (e.g., a forklift or a human operator) that is movable with respect to a fixed component within the industrial environment.

[0027] In various implementations, this method may further include receiving industrial data and a given map pose corresponding to the industrial data (e.g., within a base map). In some implementations, the industrial data may be received from the aforementioned fixed components within the industrial environment (or industrial components that remain in a non-changing state or a fixed state over a certain time period). In some implementations, the industrial data may include, for example, abnormal sensor readings. The abnormal sensor readings may be provided by, for example, fixed components (e.g., fixed sensors) within the industrial environment. In some implementations, the industrial data including abnormal sensor readings may include a warning message and may be received therein. In some implementations, a given map pose corresponding to the industrial data may include, for example, a location and / or orientation defined / judged within the base map with respect to a fixed component (e.g., a fixed sensor).

[0028] Industrial data (e.g., abnormal sensor readings) and / or a given map pose may be transmitted via, for example, one or more networks. As an alternative or addition, the industrial data may be transmitted via one or more networks along with an identifier of a fixed component (e.g., a fixed sensor) to which the industrial data is associated. The identifier of the fixed component (e.g., a fixed sensor) may enable a given map pose to be derived when the industrial data is captured using the fixed sensor.

[0029] In various implementations, this method may further include determining that the industrial data corresponds to a first environmental feature. In some implementations, the step of determining that the industrial data corresponds to a first environmental feature may be based on the steps of determining that the industrial data temporally corresponds to the detection of the first environmental feature and determining that a first base map pose for the first environmental feature spatially corresponds to a given map pose for the industrial data.

[0030] In some implementations, the step of determining that industrial data is temporally corresponding to the detection of a first environmental feature includes determining that the industrial data timestamp for the industrial data satisfies a temporal threshold for the environmental feature timestamp for the first environmental feature, based on one or more times related to the first sensor data. In some implementations, the temporal threshold may be determined based on the industrial data type of the industrial data and / or the environmental feature type of the first environmental feature. For example, a first temporal threshold (e.g., 0.1 seconds) may be determined when the first environmental feature is a forklift and the industrial data is of a first sensor type. A second temporal threshold (different from the first threshold, e.g., 0.2 seconds) may be determined when the first environmental feature is a forklift and the industrial data is of a second sensor type (different from the first sensor type). A third threshold (different from the first threshold and / or the second threshold, e.g., 2 seconds) may be determined when the first environmental feature is a maintenance worker and the industrial data is of a third sensor type.

[0031] In some implementations, the step of determining that a first base map orientation for a first environmental feature corresponds positionally to a given map orientation for industrial data may include the step of determining that the first base map orientation and the given map orientation satisfy a distance threshold. The distance threshold may be different or may be determined based on the industrial data type of the industrial data and / or the environmental feature type of the first environmental feature. For example, the first distance threshold may be determined when the first environmental feature is a maintenance worker, and the second distance threshold may be determined when the first environmental feature is a forklift, where the first threshold may be less than the second threshold.

[0032] In various implementations, this method may further include a step of using a determined correspondence of industrial data to a first feature when adapting an industrial process. In some implementations, the step of using a determined correspondence of industrial data to a first feature when adapting an industrial process may include a step of adapting one or more parameters to operate components used in the industrial process (e.g., fixed sensors are attached, or abnormal sensor readings are obtained using fixed sensors) based on the type of the first environmental feature and / or the type of industrial data. As a non-limiting example, the step of adapting an industrial process may include a step of pausing the industrial process if the type of the first environmental feature indicates that the first environmental feature is a liquid leak and the type of industrial data indicates low pressure. In some implementations, the step of adapting an industrial process may include a step of adapting the industrial process via the first environmental feature, for example, if the type of the first environmental feature indicates that the first environmental feature is a maintenance worker or repair tool / robot.

[0033] Various implementations provide additional methods implemented using one or more processors, which include the step of identifying a base map for the industrial environment, a first robot map used by a first robot located within the industrial environment, and a second robot map used by a second robot located within the industrial environment. The base map, the first robot map, and the second robot map may all be different from one another.

[0034] The industrial environment may be or may include an industrial facility where the first robot and the second robot are located. The first robot and the second robot may be of the same type or of different types. The first robot and the second robot may be manufactured by the same manufacturer or by different manufacturers. In some implementations, even if the first robot and the second robot are of the same type and manufactured by the same manufacturer, the first robot and the second robot can generate and utilize different robot-specific maps (i.e., a first robot map generated based on observations of the industrial environment by the first robot along a first path, and a second robot map generated based on observations of the industrial environment by the second robot along a second path different from the first path).

[0035] In some implementations, the first robot map may be for a first region of an industrial facility (for example, including one or more landmarks within the first region for the first robot to determine its orientation within the first region) and may correspond to a first portion of the base map. The second robot map may be for a second region of an industrial facility and may correspond to a second portion of the base map. The first portion may or may not overlap with the second portion. Note that even when the first region coincides with the second region, the first robot map and the second robot map may have different origins and / or coordinate systems, and therefore the first robot map may still be different from the second robot map. In this case, the relative orientation between the first robot and the second robot cannot be determined by assuming only the first robot orientation determined for the first robot in the first robot map and the second robot orientation determined for the second robot in the second robot map. Furthermore, since the correspondence between the first (or second) robot and the base map remains undetermined, the first robot base pose for the first robot in the base map cannot be determined, and the second robot base pose for the second robot in the base map cannot be determined.

[0036] To address the above concerns, various implementations may further include the steps of generating first transformation data (and / or first corresponding data) based on comparing a base map with a first robot map, and generating second transformation data (and / or second corresponding data) based on comparing a base map with a second robot map.

[0037] In some implementations, the first transformation data includes a first transformation function that can be applied to transform any point in the first robot map into a corresponding point in the base map. In some implementations, the second transformation data is separate from the first transformation function and includes a second transformation function that can be applied to transform any point in the second robot map into a corresponding point in the base map.

[0038] As an unrestricted example, the base map may take the form of a 3D base map point cloud (sometimes called a "base map point cloud"), and the first robot map may take the form of a robot map point cloud. In this unrestricted example, points in the base map point cloud may be compared to points in the first robot-specific map point cloud to determine a first transformation data (and / or first correspondence data) between points in the first robot-specific map and points in the base map. Similarly, points in the base map point cloud may be compared to points in the second robot-specific map point cloud to determine a second transformation data (and / or second correspondence data) between points in the second robot map and points in the base map.

[0039] The first transformation data may include, for example, a first transformation function that can be applied to transform any robot point in the first robot map into a corresponding base point in the base map. The first correspondence data may include explicit mapping relationships between one or more points in the first robot map and one or more corresponding points in the base map. The second transformation data may include a second transformation function, unlike the first transformation function, that can be applied to transform any robot point in the second robot map into a corresponding base point in the base map. The second correspondence data may include explicit mapping relationships (e.g., a mapping matrix) between one or more points in the second robot map and one or more corresponding points in the base map. Note that using the first (or second) transformation function and / or the first (or second) correspondence data, a specified point in the base map can also be transformed into a specific point in the first (or second) robot map.

[0040] In various implementations, the additional method further includes the steps of receiving a first environmental feature and a first robot map posture for the first environmental feature, which is located in a first frame of the first robot map, and determining that the first environmental feature and the first robot map posture are located in a first frame and detected by the first robot; and receiving a second robot map posture for a second environmental feature and a second environmental feature, which is located in a second frame of the second robot map, and determining that the second environmental feature and the second robot map posture are located in a second frame and detected by the second robot.

[0041] In some implementations, the first environmental feature corresponds to a first object within the industrial environment. In some implementations, the second environmental feature corresponds to a second object within the industrial environment.

[0042] In some implementations, the first feature and the first robot map pose are determined based on processing the first sensor data using a first machine learning model. In some implementations, the second feature and the second robot map pose are determined based on processing the second sensor data using a second machine learning model different from the first machine learning model. The first sensor data may be Lidar data detected using a first Lidar sensor carried by the first robot. The second sensor data may be additional Lidar data detected using a second Lidar sensor carried by the second robot.

[0043] In some implementations, the first robot map is generated from the sensor readings of one or more sensors of the first robot (including, for example, the first Lidar sensor mentioned above). In some implementations, the second robot map is generated from the sensor readings of one or more sensors of the second robot (including, for example, the second Lidar sensor mentioned above).

[0044] In various implementations, the additional method further includes the steps of using the first transformation data to convert the first robot map pose to the first base map pose (i.e., in the base map) and using the second transformation data to convert the second robot map pose to the second base map pose (for example, in the base map).

[0045] In various implementations, the additional method further includes the step of rendering the basemap in a first basemap pose with a first graphical representation of the first environment features, and in a second basemap pose with a second graphical representation of the second environment features.

[0046] In some implementations, the base map is further rendered in a graphical representation of the industrial data, which is included within the base map but absent from both the first robot map and the second robot map. The graphical representation of the industrial data can indicate the type of components associated with the industrial data, the installation data of the components, and current or recent readings related to the components.

[0047] Various implementations provide further methods that are implemented using one or more processors, each including the step of identifying a base map for an industrial environment and a first robot map used by a first robot placed within the industrial environment, wherein the base map is different from the first robot map.

[0048] In various implementations, further methods include the steps of: generating first transformation data based on a comparison of a base map and a first robot map; receiving first environmental features related to equipment in an industrial environment and a first robot map posture for the first environmental features, which are located within a first frame of the first robot map, and determining that the first environmental features and the first robot map posture are located within a first frame and detected by the first robot; converting the first robot map posture to a first base map posture using the first transformation data; receiving industrial data indicating an abnormal condition and a given map posture corresponding to the industrial data; determining that the industrial data corresponds to a first environmental feature; and storing the determined correspondence of the industrial data to the first feature in a database accessible within the industrial environment.

[0049] In addition, some implementations include one or more processors in one or more computing devices, the one or more processors being operable to execute instructions stored in associated memory, and these instructions being configured to perform one of the methods described above. Some implementations also include one or more non-temporary computer-readable storage media storing computer instructions that can be executed by one or more processors to perform one of the methods described above.

[0050] Please understand that all combinations of the aforementioned and additional concepts described in more detail herein are intended to be part of the subject matter disclosed herein. For example, all combinations of claimed subject matter appearing at the end of this disclosure are intended to be part of the subject matter disclosed herein. [Brief explanation of the drawing]

[0051] [Figure 1A] This diagram schematically illustrates exemplary environments in which selected embodiments of this disclosure can be implemented through various implementation methods. [Figure 1B] This diagram schematically illustrates another exemplary environment in which a selected aspect of this disclosure may be implemented in various implementation forms. [Figure 1C] This figure shows examples of user interfaces that display base maps without representations of objects detected by the corresponding robot, based on various implementation configurations. [Figure 1D] This figure shows examples of user interfaces that display base maps rendered with objects detected by the corresponding robots, using various implementation methods. [Figure 1E] This figure shows another example of a user interface that displays a base map rendered with objects detected by the corresponding robot, using various implementation methods. [Figure 2] This figure shows exemplary methods for carrying out a selected aspect of this disclosure using various implementation forms. [Figure 3]This figure shows another exemplary method for carrying out a selected aspect of the present disclosure in various implementation forms. [Figure 4] This figure schematically illustrates an exemplary computer architecture in which selected aspects of the present disclosure may be implemented. [Modes for carrying out the invention]

[0052] The implementations disclosed herein relate to transformations between each of one or more robot-specific maps and a base map of an industrial facility, and to using these transformations to convert robot-specific map orientations of detected environmental features to base map orientations of detected environmental features. Each of the one or more robot-specific maps may be specific to and utilized by a subset of robots (e.g., a single robot) located within an industrial facility. For example, one or more robot-specific maps may include a first robot-specific map (sometimes referred to as the "first robot map") utilized by a first robot located within the industrial facility, and / or a second robot-specific map (sometimes referred to as the "second robot map") utilized by a second robot located within the industrial facility.

[0053] In some implementations, the base map may be a base map point cloud, or may be generated from it. In some implementations, the first robot map may be a first robot-specific map point cloud (sometimes also called the "first robot map point cloud"), or may be generated from it. A comparison of the base map point cloud and the first robot-specific map point cloud may allow for the determination of a first correspondence (e.g., an explicit mapping) between points in the base map and points in the first robot-specific map. Alternatively or additionally, a comparison of the base map point cloud and the first robot-specific map point cloud may allow for the determination of a first transformation function for transforming between points in the base map and points in the first robot map.

[0054] In some implementations, the second robot map may be (or may be generated from) a second robot-specific map point cloud (sometimes called the "second robot map point cloud"). Comparing the base map point cloud with the second robot map point cloud may allow for the determination of a second correspondence between points in the base map and points in the second robot map. As an addition or alternative, comparing the base map point cloud with the second robot map point cloud may allow for the determination of a second transformation function for converting between points in the base map and points in the second robot map.

[0055] Using a first correspondence and / or a first transformation function, the robot map pose of a first object detected by a first robot (e.g., a human operator or a movable object such as a forklift) can be converted from the first robot map to a base map. Using a second correspondence and / or a second transformation function, the robot map pose of a second object detected by a second robot (e.g., a human operator or a movable object such as a forklift) can be converted from the second robot map to a base map. Converting the robot map pose of a robot-detected environmental feature (e.g., a first or second object) to a base map pose may, for example, allow the detected environmental feature to be rendered within the base map representation in the appropriate location (and / or orientation) within the base map representation.

[0056] Converting robot map poses of robot-detected environmental features to base map poses may, as an addition or alternative, allow for the simultaneous rendering and / or correlation of different environmental features detected by multiple heterogeneous robots (e.g., a first object and a second object) within the base map representation. In this case, the relative poses (e.g., position and / or orientation) between the first and second objects can be precisely determined or perceived visually.

[0057] Converting robot map postures of robot detection environment features to base map postures may, as an addition or alternative, allow for correlation of robot detection environment features with industrial automation data (e.g., low pressure) that is defined for the base map but not for any robot-specific map. For example, the occurrence of detection of a piece of equipment by a robot (e.g., robot 1) near an industrial component in a first robot map may correlate with the occurrence of an anomaly sensor reading associated with the industrial component in the base map. As another example, the occurrence of detection of an anomaly condition (e.g., a leak) by a particular robot (e.g., robot 1 navigating inside or outside a pipe to monitor for leaks) near an industrial component in a first robot map may correlate with the occurrence of an anomaly sensor reading (e.g., low pressure) associated with the industrial component in the base map. In this latter example, if an anomaly sensor reading associated with the same industrial component (in the base map) is subsequently received, the particular robot may be sent to the posture at the time the leak was previously detected to inspect any current leaks, and / or an additional robot may be sent to a posture to repair the leaks.

[0058] By utilizing the transformation or correspondence determined between the base map and the first robot map (or second robot map), representations of dynamic objects detected by one or more robots deployed in an industrial facility can be dynamically rendered within the base map. The transformation or correspondence determined between the base map and the first robot map also makes it possible to associate the occurrence of objects or events (e.g., leaks) detected by the robot in a specific orientation within the robot-specific map with industrial data (e.g., low pressure) automatically received from fixed sensors for industrial structural elements (fixed within the industrial facility) within the base map. The first robot map may not contain industrial components, but since the base map shows the orientation of industrial components, the industrial data is reflected in the base map but not in the first robot map.

[0059] Figure 1A schematically illustrates an exemplary environment in which a selected aspect of the Disclosure may be implemented in various implementation forms. Referring now to Figure 1A, an exemplary environment 100 in which a variety of aspects of the Disclosure may be implemented is schematically shown. The exemplary environment 100 may be an industrial facility 130, or may include an industrial facility 130, which may take on numerous forms. The exemplary environment 100 may optionally be designed to implement any number of at least partially automated processes. The industrial facility 130 may take the form of a chemical processing plant, an industrial office environment, an oil or natural gas refinery, a catalyst plant, a manufacturing plant, an offshore oil platform, or any other applicable facility.

[0060] An exemplary environment 100 may include one or more client devices (e.g., local client devices 103-A and 103-B) operably coupled to a process automation network 106 within an industrial facility. Client device 103-A or 103-B may be implemented as a computer (e.g., laptop, desktop, notebook), tablet, robot, smart appliance (e.g., smartphone), messaging device, wearable device (e.g., watch), or any other applicable device. The process automation network 106 may be implemented using a variety of wired and / or wireless communication technologies, including, but not limited to, cellular networks such as the IEEE 802.3 standard (Ethernet), IEEE 802.11 (Wi-Fi), 3GPP® Long-Term Evolution ("LTE"), or other wireless protocols designated as 3G, 4G, 5G, and later, and / or other types of communication networks in various types of topologies (e.g., mesh).

[0061] In various implementations, the exemplary environment 100 may include a base map 140, which can be rendered via a display on a client device 103-A (or another device, such as client device 103-B, or server device 105). The base map 140 may be a 3D model, or a 3D point cloud collected using Lidar sensors carried around the industrial facility 130, or generated from them. For example, the base map 140 may be generated based on observations by Lidar sensors for the industrial facility 130 (e.g., the entire facility) when there are no mobile robots in the industrial facility 130 (or prior to any robots being placed in the industrial facility 130). In this case, the base map 140 may lack any representation of any robot in terms of robot posture (i.e., the position and / or orientation of any mobile robot or any movable object detected by the mobile robot). In some implementations, the base map 140 may represent various static components of the industrial facility 130 (e.g., industrial components such as doors, open spaces, windows, walls, tanks, equipment, and fixed sensors). For example, as shown in Figure 1A, the base map 140 may include and represent structure A' and industrial components 180 (e.g., pressure gauges).

[0062] Client devices 103-A or 103-B may each include an input device and / or output device for user interaction with the base map 140. For example, user input received via the input device of client device 103-B (e.g., a click in the graphical display in the base map 140 representing an industrial component 180) can cause industrial data related to the industrial component 180 to be graphically rendered in the base map 140. The industrial data may include, for example, the ID of the industrial component 180 (e.g., device ID 555), the status of the industrial component 180 (e.g., normal or faulty), the service history of the industrial component 180, and measurements of the industrial component 180.

[0063] An exemplary environment 100 may further include one or more mobile robots. For example, an exemplary environment 100 may include a robot fleet having a first robot 111. The first robot 111 may be a quadruped robot (e.g., a robotic dog), a wheeled robot, an unmanned aerial vehicle (e.g., a drone), a crawler robot, or any other applicable robot that is mobile within or around an industrial facility. Different robots may be of different types and may be manufactured by different manufacturers. Different robots may utilize different (however, all may be in the form of 3D point clouds, for example) robot-specific maps for navigation and to perform tasks / missions. In some implementations, a robot fleet may include a subset of robots of the same type and / or manufactured by the same manufacturer. In these implementations, for example, when different robots from the subset are positioned or programmed to observe an industrial facility (e.g., 130) along different planned routes, the robot-specific maps generated by and / or utilized by the corresponding robots from the subset of robots can still be different from one another. This is because, for example, a robot-specific map may be generated based on past observations of the robot, and past observations of the robot may differ depending on the path the robot has previously traveled.

[0064] As a non-limiting example, the first robot 111 may be a wall-climbing robot (for example, for inspecting the outside of a container or wall for an industrial facility 130), a crawler robot (for example, for operating and manipulating a component of an industrial facility), or a drone (for example, for inspecting chimneys and infrastructure of an industrial facility, or for inspecting the inside of a container of an industrial facility, etc.). In some implementations, instead of a wall-climbing robot, the first robot 111 may instead be a robot for transporting supplies and products, a robotic dog for patrolling and monitoring an industrial facility for anomalies, a spider-like robot for inspecting the outside of a pipeline, a snake-like robot for inspecting the inside of a pipeline, or other types of robots.

[0065] In some implementations, the first robot may include one or more sensors for performing one or more missions. For example, the first robot 111 may include (or be equipped with) a light-detecting and ranging (Lidar) sensor for imaging an object by creating a 3D model of the imaged object. Additional or alternative, the first robot 111 may include other sensors, such as a vision sensor, ultrasonic testing immersion transducers for detecting surface irregularities and defects (e.g., corrosion), one or more gas sensors for detecting the presence and concentration of harmful gases or vapors, and / or a temperature sensor for measuring temperature. The vision sensor may be a monographic camera, a stereographic camera, a thermal camera, or any other applicable vision sensor for capturing one or more images of one or more specific components of one or more industrial facilities 130. The vision sensor may be detachably coupled to or integrated with the first robot 111. In some implementations, the vision sensor may change its location and / or orientation relative to the first robot 111, for example, by rotation or other movement. In a non-limiting example, the first robot 111 may include front and rear high-resolution cameras that are detachably coupled to the first robot 111.

[0066] In some implementations, the first robot 111 may optionally be positioned in one or more areas of the industrial facility 130. For example, the first robot 111 may be positioned to navigate around a first area of ​​the industrial facility 130. While positioned to navigate around the first area, the first robot 111 may generate a number of points that reflect one or more components and / or objects of the first area. For example, the first robot 111 may detect one or more points in a first robot map 141 specific to the first robot 111 that correspond to structure A of the industrial facility 130, and may further detect moving objects 121 (e.g., a human) in the first robot map 141. In some implementations, the first robot map 141 may have a first reference frame 171 (sometimes called the "first frame," which may or may not be visually rendered), and the robot map orientation of the moving object 121 may be determined within the first frame of the first robot map 141.

[0067] The exemplary environment 100 may further include a server computing device 105 (sometimes simply referred to as the “server device”). The server computing device 105 may include a map-aware engine 1051, an attitude transformation engine 1052, a rendering engine 1053, and / or storage 15. In some implementations, the base map 140 may be stored in the storage 15. In some implementations, optionally, the exemplary environment 100 may include two or more base maps (for example, base maps determined for different years or for different industrial facilities).

[0068] In various implementations, the map-aware engine 1051 may request access to the first robot map (for example, stored in the first robot 111) in order to compare the first robot map with the base map 140. The map-aware engine 1051 may compare the first robot map with the base map 140 to determine that structure A in the first robot map 141 has the same structure as structure A' in the base map 140. In this case, one or more points in the first robot map 141 corresponding to structure A may be compared with one or more points in the base map 140 corresponding to structure A' in order to determine the first correspondence data (for example, the first correspondence described above) and / or the first transformation data.

[0069] The first correspondence data may include, for example, an explicit mapping relationship between one or more 3D points in the first robot map and one or more 3D points in a 3D point cloud used to form the base map 140. The first transformation data may include, for example, a transformation function that transforms one or more points in the first robot map 141 to one or more corresponding points in a 3D point cloud forming the base map 140, or vice versa.

[0070] In various implementations, the server device 105 (or a client device, e.g., 103-B) may receive a first robot map pose for a first environmental feature (e.g., the aforementioned moving object 121, which may be a walking human operator) within a first frame of the first robot map. The pose conversion engine 1052 may use the first correspondence data and / or the first conversion data to convert the first robot map pose of the first environmental feature (e.g., the moving object 121) in the first robot map 141 to a first base map pose in the base map 140. The rendering engine 1053 may further render the first graphical representation 131 of the first environmental feature (e.g., the moving object 121) in the base map 140 with the first base map pose.

[0071] It should be noted that the first graphical representation 131 of the first environmental feature in the base map 140 may differ from the representation of the first environmental feature in the first robot map 141. The first graphical representation 131 may be, for example, an image of the first environmental feature, a symbol characterizing the first environmental feature, etc. In some implementations, the representation of an object in the base map captured by the first robot may optionally be a specific form unique to the first robot (e.g., color, size, bounding box, etc.). Optionally, based on the tracking pose of the first environmental feature in the first robot map 141 at different points in time, the rendering engine 1053 may render the first graphical representation 131 of the first environmental feature (e.g., a moving object 121) in a pose corresponding to the base map 140 at different points in time, so that the base map 140 is a “dynamic” map for the user to observe and track the movement of the first environmental feature (e.g., a moving object 121) in the industrial facility 130 via the base map 140.

[0072] In some implementations, the server computing device 105 may further include an occurrence association engine 1054. Based on the transformation and / or correspondence between the base map and the first robot map, the occurrence association engine 1054 may determine whether the occurrence of an object or event (e.g., a moving object 121) detected by the first robot 111 (or another robot) in a particular orientation (e.g., first robot map orientation) within the first robot map 141 is associated with industrial data (e.g., low-pressure sensor readings) automatically received for industrial components within the base map (e.g., a fixed instrument 180), such as industrial data that is correlated with the base map but not with the first robot map. Note that the base map 140 may be a 2D map (or a 3D map) having a coordinate system 170, X' axis and Y' axis defined by the origin.

[0073] Figure 1B schematically shows another exemplary environment 100' in which a selected embodiment of the present disclosure may be implemented in various implementation forms. As shown in Figure 1B, environment 100' may include a robot fleet having a first robot 111 and a second robot 112. Similar to environment 100, environment 100' may further include one or more client devices (e.g., client device 103-A, client device 103-B) and one or more server devices 105. For clarity, repetition of the descriptions of client devices and server devices is omitted in this specification.

[0074] As shown in Figure 1B, the base map 140 can be rendered via a computing device (e.g., client device 103-B) for users or staff of the industrial facility 130 to observe or inspect different areas of the industrial facility. The base map 140 may include and show various components of the industrial facility 130, such as one or more walls, one or more rooms, and one or more industrial components 180 (which often remain fixed or immobile for a period of time, for example, several weeks or months).

[0075] In some implementations, the first robot 111 may be positioned to navigate around the industrial facility 130 using the first robot map 141, and the second robot 112 may be positioned to navigate around the industrial facility 130 using the second robot map 142. As a non-limiting example, the first robot map 141 may include a representation of structure A within the industrial facility 130. Additional or alternative, the first robot map 141 may include a representation of the first robot 111 that can change its orientation while navigating around the industrial facility 130. In other words, at different points in time, the representation of the first robot 111 may be at different locations in the first robot map 141 and may have different orientations. Additional or alternative, the first robot map 141 may detect a first environmental feature (e.g., a moving human operator 121) and include a representation of the first environmental feature within the first robot map 141.

[0076] The second robot map 142 may include a representation of structure B within the industrial facility 130. Alternatively, the second robot map 142 may include a representation (not shown) of a second robot 112 that can change its orientation while navigating around the industrial facility 130. In other words, at different points in time, the representation of the second robot 112 may be at different locations in the second robot map 142 and may have different orientations. Alternatively, the second robot map 142 may detect a second environmental feature (e.g., a moving forklift 122) and include a representation of the second environmental feature within the second robot map 142.

[0077] In some implementations, the first robot 111 may receive a request to provide an update to the first robot detection (e.g., an update to the current pose of the first robot 111 and / or an update to the current pose of one or more environmental features detected by the first robot 111 in the first robot map 141), and the second robot 112 may receive another request to provide an update to the second robot detection (e.g., an update to the current pose of the second robot 112 and / or an update to the current pose of one or more environmental features in the second robot map 142). The requests received by the first robot 111 and the other requests received by the second robot 112 may be generated based on user input or automatically generated by the server device 105 (e.g., periodically or at other regular or irregular intervals). In some implementations, the first robot 111 and / or the second robot 112 may proactively push their current poses to the server device 105 without necessarily receiving any requests from the server device 105 first. For example, when the first robot 111 is connected to a local area network to which the server device 105 is also connected, the first robot 111 may, at least periodically, actively push its current posture to the server device 105.

[0078] In response to a request, the first robot 111 may provide, for example, the server device 105, with the current pose of the first robot 111 and / or the current pose of one or more environmental features (all within the first robot map 141). In response to a separate request, the second robot 112 may provide, for example, the server device 105, with the current pose of the second robot 112 and / or the current pose of one or more environments (all within the second robot map 142). The pose conversion engine 1052 may then access the first corresponding data (and / or the first conversion data) to convert the current pose of the first robot 111 and / or the current pose of one or more environmental features in the first robot map 141 into the corresponding pose in the base map 140. Alternatively or additionally, the attitude change engine 1052 may access second corresponding data (and / or second transformation data) to convert the current attitude of the second robot 112 and / or the current attitude of one or more environmental features in the second robot map 142 to the corresponding attitude in the base map 140.

[0079] The first conversion data and / or the first correspondence data may be determined, for example, based on identifying that structure A in the first robot map 141 and structure A' in the base map 140 are the same structure. The second conversion data and / or the second correspondence data may be determined based on identifying that structure B in the second robot map 142 and structure B' in the base map 140 are the same structure.

[0080] The rendering engine 1053 may then generate a representation of the first robot 111 and / or a representation of one or more environmental features detected in the first robot map 141 within the base map 140. The rendering engine 1053 may also generate a representation of the second robot 112 and / or a representation of one or more environmental features detected in the second robot map 142 within the base map 140. In some implementations, the representations of the first robot 111 and the second robot 112 do not need to be rendered within the base map 140. For example, the user may choose not to display the first robot 111 and / or the second robot 112 within the base map 140, or may configure it to do so.

[0081] Referring to Figure 1B, the representation 132 of the aforementioned second environmental feature (e.g., a moving forklift 122 detected by the second robot 112) and the representation 131 of the aforementioned first environmental feature (e.g., a moving human operator 121) can be visually rendered within the base map 140. In some implementations, representations 131 and 132 may be rendered simultaneously. For example, when the first robot 111 detects the first environmental feature and the second robot 112 detects the second environmental feature, these may be rendered simultaneously. In some implementations, representations 131 and 132 may be rendered at different points in time.

[0082] For example, at a first moment, a first environmental feature is detected by the first robot 111, and no environmental features are detected by the second robot 112. At this first moment, representation 131 may be rendered in the base map 140 together with the instrument 180, for example, without the representation 132 of the second environmental feature being rendered in the base map 140. At a second moment, a second environmental feature is detected by the second robot 112, and no environmental features are detected by the first robot 111. At this second moment, representation 132 may be rendered in the base map 140 together with the instrument 180, for example, without the representation 131 of the first environmental feature being rendered in the base map 140. Note that the representation of the first environmental feature in the first robot map 141 may differ from the representation of the first environmental feature in the base map 140 in terms of aspects such as orientation, size, color, and shape. The representation of the second environmental feature in the second robot map 142 may differ from the representation of the second environmental feature in the base map 140 in one or more aspects, such as orientation, size, color, or shape, as an alternative or addition.

[0083] In some implementations, representation 131 may be rendered in response to the detection of a first environmental feature by the first robot 111 without the first robot 111 receiving the aforementioned request. In some implementations, representation 132 may be rendered in response to the detection of a second environmental feature by the second robot 112 without the second robot 112 receiving the aforementioned request.

[0084] Using the base map 140 rendered in various representations (for example, representations of the first robot 111 and / or the second robot 112, and representations of the first and / or second environmental features), the relative orientation between different environmental features can be determined. Alternatively or additionally, the relative orientation between the first (or second) environmental features and the industrial components (or other structures) of the industrial facility 130 can be determined.

[0085] Figure 1C shows a non-limiting example of a user interface 190 showing an unrendered base map 140 without any representation of robots or objects detected by robots, for implementing a selected aspect of the present disclosure in various implementation forms. Such a base map may be rendered, for example, when an application providing access to the base map is just launched via a display device 109, or when the user chooses not to render any representation of robots or objects detected by robots placed in the environment. As shown in Figure 1C, the base map 140 may be a “raw” map showing static or fixed components within an industrial environment (e.g., industrial facility 130), including, but not limited to, empty spaces E, walls F, doors H, rooms G, and / or fixed sensors (e.g., instrument 180).

[0086] Figure 1D shows a non-limiting example of a user interface 191 that displays a base map rendered / annotated with objects detected by a corresponding robot for implementing a selected aspect of the present disclosure in various implementation forms. As shown in Figure 1D, the base map 140 may include fixed components 180 and may be further dynamically rendered with a representation 131 for a moving human 121 (a non-limiting example of the first environmental feature described above) detected by a first robot 111 and a representation 132 for a moving forklift 122 (a non-limiting example of the second environmental feature described above) detected by a second robot 112.

[0087] The representation 131 of a moving human 121 detected by the first robot 111 may be, for example, an RGB image captured by the camera of the first robot 111, or a symbol (with or without text description) representing the moving human 121. The representation 132 of a moving forklift 122 detected by the second robot 112 may be, for example, an RGB image captured by the camera of the second robot 112, or a symbol (with or without text description) representing the moving forklift 122. The representation 131 of the moving human 121 may be rendered in the corresponding basemap pose (the aforementioned first basemap pose for the first environmental feature) determined using the techniques described in this disclosure. The representation 131 of the moving human 121 may be rendered in the corresponding basemap pose (the aforementioned second basemap pose for the second environmental feature) determined using the techniques described in this disclosure. In this case, the representations of the first robot 111 and the second robot 112 do not need to be rendered within the base map 140 for purposes such as saving computing resources and reducing latency.

[0088] Figure 1E shows another non-limiting example of a user interface 192 showing a base map 140 rendered with an object detected by a corresponding robot in order to implement a selected aspect of the present disclosure in various implementation forms. As shown in Figure 1E, at a particular moment t, the base map 140 may be dynamically rendered with a representation 131 for a moving human 121 detected by a first robot 111. At a particular moment t, the base map 140 may be further rendered with a representation 132 for a moving forklift 122 detected by a second robot 112. Representation 131 may be rendered in response to the detection of a moving human 121 in a first robot map by the first robot 111, and representation 132 may be rendered in response to the detection of a moving forklift 122 in a second robot map by the second robot 112.

[0089] In some implementations, the component 180 is observed within a point cloud that forms (or is used to generate) the base map 140, so the base map 140 may contain a representation of the industrial component 180 at any given moment. In some implementations, the industrial component 180 may be an instrument, and the base map 140 may show industrial data, including, but not limited to, the readings of the instrument 180, where the readings of the instrument 180 may differ at different moments.

[0090] As a non-limiting example, at a particular moment t, the base map 140 may indicate an anomaly reading for instrument 180. The occurrence association engine 1054 may determine that the anomaly reading for instrument 180 is associated with a moving object 121, which is represented in the base map 140 by representation 131. Such determination by the occurrence association engine 1054 may be based on determining that the industrial data (i.e., the anomaly reading) corresponds temporally to the detection of a first environmental feature (e.g., both the anomaly reading and the moving object 121 are detected at a particular moment t, or within a temporal threshold, e.g., 0.5 seconds), and that the first base map orientation for the first environmental feature corresponds spatially to a given map orientation for the industrial data (e.g., the relative distance between the moving object 121 and instrument 180 is within a distance threshold, e.g., 0.8 m). The association may then be stored, for example, in storage 15, for later use. For example, based on stored associations, when an abnormal reading occurs next, the moving object 121 may be identified, and a human operator 121 may be dispatched, for example, to inspect the abnormal reading and adapt the industrial process affected by the abnormal reading.

[0091] In some implementations, the representation for a human 121 moving in a first base map pose may, when selected, be a symbol 151 that causes the representation 131 to be rendered (for example, as an overlay) on the base map 140. In some implementations, the representation for a moving forklift 122 may, when selected, be a symbol 152 that can cause the symbol for the moving human 121 to be rendered (for example, as an overlay) on the base map 140. In some implementations, the base map 140 may optionally further include a representation 161 (for example, a symbol) for a first robot 111 and / or a representation 162 (for example, a symbol) for a second robot 112, where symbol 161 is different from symbol 162. The first robot 111 may, for example, be a drone. The second robot 112 may, for example, be a robot dog. In some implementations, the representation 161 (e.g., a symbol) for the first robot 111 and the representation 162 (e.g., a symbol) for the second robot 112 may be omitted from the base map 140.

[0092] It should be noted that base map 140 is sometimes referred to as the “true map” when it represents the entire industrial environment / facility. The true map can reflect a real-time representation of static and dynamic objects within the industrial facility when annotated with objects detected by robots and / or corresponding robots.

[0093] Figure 2 shows exemplary methods 200 for carrying out selected embodiments of the present disclosure in various implementation forms. For convenience, the operations of the flowchart will be described with reference to a system that carries out those operations. This system may include various components of various computer systems, such as one or more components of a server computing device 105 (and / or additional computing devices, such as client devices 103-A or 103-B). Furthermore, although the operations of Method 200 are shown in a particular order, this is not intended to be limiting. One or more operations may be rearranged, omitted, or added.

[0094] In various implementations, in block 202, the system can identify a base map of the industrial environment and a first robot map used by a first robot deployed within the industrial environment, for example, by a server such as a server computing device 105, where the base map is different from the first robot map.

[0095] The industrial environment may be or may include an industrial facility where the first robot is located. In some implementations, the first robot map may be generated based on observations of the industrial environment or a portion thereof by the first robot. For example, the first robot map may be generated based on observations of the first robot along one or more planned paths within the industrial environment. In some implementations, the base map may be generated based on observations of the industrial environment (e.g., the entire environment) using sensors (e.g., Lidar sensors) carried around the industrial environment. The base map may be a base map 3D point cloud (or 3D model) rendered using sensor readings from the sensors, or a 2D graph (e.g., floor plan) generated and / or validated using a base map 3D point cloud (or 3D model) from sensor readings.

[0096] In some implementations, the first robot map is generated from sensor readings of one or more sensors on the first robot. The first robot map may be, for example, a first robot map 3D point collected using one or more sensors that are movably mounted on the first robot. In some implementations, the first robot map is stored locally as a point cloud on the first robot.

[0097] In various implementations, in block 204, the system may generate first transformation data (and / or first correspondence data) based on a comparison between the base map and a first robot map, for example, by a server such as a server computing device 105. In a non-limiting example, the base map may take the form of a 3D base map point cloud (sometimes simply called the “base map point cloud”), and the first robot map may take the form of a 3D first robot-specific map point cloud (sometimes simply called the “first robot-specific map point cloud”) or the “first robot map point cloud”). In this non-limiting example, points in the base map point cloud may be compared to points in the first robot-specific map point cloud to determine first transformation data (and / or first correspondence data) between points in the first robot-specific map and a portion of points in the base map.

[0098] The first transformation data may include, for example, a first transformation function that can be applied to transform any robot point in the first robot map into a corresponding base point in the base map. The first correspondence data may include, for example, a mapping relationship between one or more points in the first robot map and one or more corresponding points in the base map.

[0099] For example, assuming a point (X1, Y1, Z1) in the first robot map, the first transformation data may show that a point (X1+4.5, Y1+3.5, Z1) in the base map corresponds to a point (X1, Y1, Z1) in the first robot map. For example, assuming a point (X1, Y1, Z1) in the first robot map, the first correspondence data may show that a point (X1', Y1', Z1) in the base map corresponds to (for example, is mapped to) a point (X1, Y1, Z1) in the first robot map.

[0100] In various implementations, in block 206, the system may receive a first environmental feature and a first robot map orientation for the first environmental feature, located within a first frame of the first robot map, from a server, for example, a server computing device 105. In these implementations, the first feature and the first robot map orientation may be determined based on processing first sensor data, located within a first frame and detected by the first robot. The first environmental feature may correspond to a first object in an industrial environment, such as a forklift or a human. The first environmental feature may be detected by the first robot based on processing first sensor data (e.g., Lidar data) located within a first frame of the first robot map. In some implementations, the first sensor data may be further processed to determine the first robot map orientation (location and / or orientation) for the first environmental feature within a first frame of the first robot map.

[0101] In some implementations, the first feature and the first robot map pose may be determined based on processing the first sensor data using a machine learning model. In some implementations, instead of the first sensor data, additional first sensor data different from the first sensor data may be applied to detect the first feature or to determine the first robot map pose for a first environmental feature within a first frame. In other words, different or partially different sensor data may be captured and used to detect the first feature or to determine the first robot map pose for a first environmental feature within a first frame.

[0102] The first frame may include, for example, a reference point at the origin of the first landmark, and three reference points, each one unit away along a coordinate axis such as the X, Y, and Z axes. In this example, the first robot pose of the first robot (e.g., X1, Y1, Z1) may be determined with respect to the first landmark, and the first robot map pose with respect to the first environmental feature (e.g., X1+m, Y1+n, Z1) may be determined based on the relative pose between the first robot and the first environmental feature in the first frame.

[0103] In various implementations, in block 208, the system may use the first transformation data to convert the first robot map pose (i.e., in the first frame of the first robot map) to the first base map pose (i.e., in the base map) using a server, for example, a server computing device 105. For example, assuming a first robot map pose (e.g., X1+m, Y1+n, Z1) for a first environmental feature in the first frame, the first transformation data may be used to determine that the first base map pose (e.g., X1+m+4.5, Y1+n+3.5, Z1) in the base map corresponds to the first robot map pose (e.g., X1+m, Y1+n, Z1) in the first frame.

[0104] In various implementations, in block 210, the system may receive industrial data and a given map orientation corresponding to the industrial data from a server, such as a server computing device 105. In some implementations, the first environmental feature corresponds to a movable object that is movable relative to a fixed component in the industrial environment. In these implementations, the industrial data includes, for example, anomaly sensor readings for a fixed component in the industrial environment. The given map orientation corresponding to the industrial data includes location and / or orientation defined / determined in the base map.

[0105] Industrial data (e.g., anomaly sensor readings) and / or a given map orientation may be transmitted, for example, over one or more networks. Alternatively or additionally, the industrial data may be transmitted over one or more networks along with an identifier for a fixed sensor that provides anomaly sensor readings, where the identifier enables the derivation of a given map orientation when the industrial data is captured using the fixed sensor.

[0106] In various implementations, in block 212, the system may determine, for example, by a server such as server computing device 105, that the industrial data corresponds to a first environmental feature. In some implementations, determining that the industrial data corresponds to a first environmental feature may be based on determining that the industrial data corresponds temporally to the detection of the first environmental feature, and determining that a first base map pose for the first environmental feature corresponds spatially to a given map pose for the industrial data.

[0107] In some implementations, determining that industrial data is temporally corresponding to the detection of a first environmental feature involves determining that the industrial data timestamp for the industrial data satisfies a temporal threshold for the environmental feature timestamp for the first environmental feature, based on one or more times related to the first sensor data. In some implementations, the temporal threshold may be determined based on the industrial data type of the industrial data and / or the environmental feature type of the first environmental feature. For example, the first threshold may be determined when the first environmental feature is a forklift and the industrial data is of a first sensor type. A second threshold (different from the first threshold) may be determined when the first environmental feature is a forklift and the industrial data is of a second sensor type (different from the first sensor type). A third threshold (different from the first and / or second thresholds) may be determined when the first environmental feature is a maintenance worker and the industrial data is of a third sensor type (different from the first and / or second sensor type).

[0108] In some implementations, determining that a first base map orientation for a first environmental feature corresponds positionally to a given map orientation for industrial data may include determining that the first base map orientation and the given map orientation satisfy a distance threshold. The distance threshold may differ or be determined based on the industrial data type of the industrial data and / or the environmental feature type of the first environmental feature. For example, the first distance threshold may be determined when the first environmental feature is a maintenance worker, and the second threshold may be determined when the first environmental feature is a forklift, where the first threshold may be less than the second threshold.

[0109] In various implementations, in block 214, the system may use the determined correspondence of industrial data to the first feature when adapting the industrial process, for example, by a server such as server computing device 105. In some implementations, using the determined correspondence of industrial data to the first feature when adapting the industrial process may include adapting one or more parameters to operate the components used in the industrial process (for example, the fixed components mentioned above).

[0110] Figure 3 shows another exemplary method 300 for carrying out a selected aspect of the present disclosure in various implementation forms. For convenience, the operations of the flowchart will be described with reference to the system that carries out those operations. This system may include various components of various computer systems, such as one or more components of the server computing device 105 (and / or additional computing devices, such as client devices 103-A or 103-B). Furthermore, although the operations of method 300 are shown in a particular order, this is not to say that it is limiting. One or more operations may be rearranged, omitted, or added.

[0111] In various implementations, in block 302, the system may, for example, use a server such as a server computing device 105 to identify a base map of the industrial environment, a first robot map used by a first robot located within the industrial environment, and a second robot map used by a second robot located within the industrial environment. The base map, the first robot map, and the second robot map may all be different from one another.

[0112] The industrial environment may be, or include, the industrial facility where the first robot and the second robot are located. The first robot and the second robot may be of the same type or of different types. The first robot and the second robot may be manufactured by the same manufacturer or by different manufacturers. In some implementations, even if the first robot and the second robot are of the same type and manufactured by the same manufacturer, the first robot and the second robot can generate and utilize different robot-specific maps (i.e., a first robot map generated based on observations of the industrial environment by the first robot, and a second robot map generated based on observations of the industrial environment by the second robot).

[0113] In some implementations, the first robot map may be for a first area of ​​the industrial facility (for example, including one or more landmarks within the first area for the first robot to determine its orientation within the first area) and may correspond to a first portion of the base map. The second robot map may be for a second area of ​​the industrial facility and may correspond to a second portion of the base map. The first area may not overlap with the second portion, or may only partially overlap with the second portion. Even when the first area coincides with the second area, the first robot map and the second robot map may differ from the second robot map because they have different origins and / or coordinate systems. In this case, the relative attitude between the first robot and the second robot cannot be determined by assuming only the first robot attitude determined for the first robot in the first robot map and the second robot attitude determined for the second robot in the second robot map. Furthermore, since the correspondence between the first (or second) robot and the base map is not determined, the first robot base attitude for the first robot in the base map cannot be determined, and the second robot base attitude for the second robot in the base map cannot be determined.

[0114] In various implementations, in block 304, the system can generate first transformation data (and / or first correspondence data) based on a comparison between the base map and a first robot map by a server, for example, a server computing device 105, and generate second transformation data (and / or second correspondence data) based on a comparison between the base map and a second robot map.

[0115] In some implementations, the first transformation data includes a first transformation function that can be applied to transform any point in the first robot map into a corresponding point in the base map. In some implementations, the second transformation data is separate from the first transformation function and includes a second transformation function that can be applied to transform any point in the second robot map into a corresponding point in the base map.

[0116] As an unrestricted example, the base map may take the form of a 3D base map point cloud (sometimes simply called the “base map point cloud”), and the first robot map may take the form of a 3D first robot-specific map point cloud (sometimes simply called the “first robot-specific map point cloud”) or “first robot map point cloud”). In this unrestricted example, points in the base map point cloud may be compared to points in the first robot-specific map point cloud to determine first transformation data (and / or first correspondence data) between points in the first robot-specific map and a portion of points in the base map. Similarly, points in the base map point cloud may be compared to points in the second robot-specific map point cloud to determine second transformation data (and / or second correspondence data) between points in the first robot-specific map and a portion of points in the base map.

[0117] The first transformation data may include, for example, a first transformation function that can be applied to transform any robot point in the first robot map into a corresponding base point in the base map. The first correspondence data may include mapping relationships between one or more points in the first robot map and one or more corresponding points in the base map. The second transformation data may include a second transformation function, unlike the first transformation function, that can be applied to transform any robot point in the second robot map into a corresponding base point in the base map. The second correspondence data may include mapping relationships between one or more points in the second robot map and one or more corresponding points in the base map. Note that using the first (or second) transformation function and / or the first (or second) correspondence data, a specified point in the base map can be transformed into a specific point in the first (or second) robot map.

[0118] In various implementations, in block 306, the system can receive, for example, a first environment feature and a first robot map posture for the first environment feature located within a first frame of the first robot map, from a server such as a server computing device 105, and the first environment feature and the first robot map posture are determined based on processing first sensor data located within a first frame and detected by the first robot; and the system can receive a second robot map posture for a second environment feature and a second environment feature located within a second frame of the second robot map, and the second environment feature and the second robot map posture are determined based on processing second sensor data located within a second frame and detected by the second robot.

[0119] In some implementations, the first environmental feature corresponds to a first object within the industrial environment. In some implementations, the second environmental feature corresponds to a second object within the industrial environment.

[0120] In some implementations, the first feature and the first robot map pose are determined based on processing the first sensor data using a first machine learning model. In some implementations, the second feature and the second robot map pose are determined based on processing the second sensor data using a second machine learning model different from the first machine learning model.

[0121] In some implementations, the first robot map is generated from the sensor readings of one or more first sensors of the first robot. In some implementations, the second robot map is generated from the sensor readings of one or more second sensors of the second robot.

[0122] In various implementations, in block 308, the system can, for example, use a server such as a server computing device 105 to convert a first robot map pose to a first base map pose using first conversion data, and use second conversion data to convert a second robot map pose to a second base map pose.

[0123] In various implementations, in block 310, the system may have a server, such as a server computing device 105, render the basemap in a first basemap orientation with a first graphical representation of the first environment features, and in a second basemap orientation with a second graphical representation of the second environment features.

[0124] In some implementations, the base map is further rendered in a graphical representation of industrial data, which is included within the base map but absent from both the first robot map and the second robot map. The graphical representation of industrial data may indicate the type of component, the installation date of the component, and current or recent indications regarding the component.

[0125] Various implementations provide further methods that are implemented using one or more processors, each including the step of identifying a base map of an industrial environment and a first robot map used by a first robot placed within the industrial environment, wherein the base map is different from the first robot map.

[0126] In various implementations, further methods include the steps of: generating first transformation data based on a comparison of a base map and a first robot map; receiving first environmental features related to equipment in an industrial environment and a first robot map posture for the first environmental features, which are located within a first frame of the first robot map, wherein the first environmental features and the first robot map posture are determined by processing first sensor data, which are located within a first frame and detected by the first robot; converting the first robot map posture to a first base map posture using the first transformation data; receiving industrial data indicating an abnormal condition and a given map posture corresponding to the industrial data; determining that the industrial data corresponds to a first environmental feature; and storing the determined correspondence of the industrial data to the first feature in a database accessible within the industrial environment.

[0127] Figure 4 is a block diagram of an exemplary computing device 410, which may optionally be used to implement one or more embodiments of the techniques described herein. The computing device 410 generally includes at least one processor 414 that communicates with several peripheral devices via a bus subsystem 412. These peripheral devices may include, for example, a storage subsystem 424, which includes a memory subsystem 425 and a file storage subsystem 426, a user interface output device 420, a user interface input device 422, and a network interface subsystem 416. The input and output devices enable user interaction with the computing device 410. The network interface subsystem 416 provides an interface to an external network and is coupled to a corresponding interface device in another computing device.

[0128] The user interface input device 422 may include pointing devices such as keyboards, mice, trackballs, touchpads, or graphics tablets, scanners, touchscreens integrated into displays, voice input devices such as voice recognition systems and microphones, and / or other types of input devices. In general, the use of the term “input device” is intended to include all possible types of devices and methods for inputting information within the computing device 410 or onto a communication network.

[0129] The user interface output device 420 may include non-visual displays such as a display subsystem, printer, fax machine, or audio output device. The display subsystem may include flat panel devices such as cathode ray tubes (CRTs), liquid crystal displays (LCDs), projection devices, or several other mechanisms for creating visual images. The display subsystem may also provide non-visual representations, such as via an audio output device. In general, the use of the term “output device” is intended to include all possible types of devices and methods for outputting information from the computing device 410 to the user or to another machine or computing device.

[0130] The storage subsystem 424 stores programming structures and data structures that provide some or all of the functions of the modules described herein. For example, the storage subsystem 424 may include logic for carrying out a selected embodiment of the method in Figure 3, as well as for implementing the various components shown in Figures 1 and 2.

[0131] These software modules are generally executed by processor 414, either alone or in combination with other processors. The memory 425 used within the storage subsystem 424 may include several memories, including main random access memory (RAM) 430 for storing instructions and data during program execution, and read-only memory (ROM) 432 for storing fixed instructions. The file storage subsystem 426 can provide persistent storage for program files and data files and may include a hard disk drive, a floppy disk drive, a CD-ROM drive, an optical drive, or a removable media cartridge, along with associated removable media. Modules implementing certain functionalities may be stored by the file storage subsystem 426 within the storage subsystem 424 or in other machines accessible by processor 414.

[0132] The bus subsystem 412 provides a mechanism for various components and subsystems of the computing device 410 to communicate with each other as intended. Although the bus subsystem 412 is schematically shown as a single bus, alternative implementations of the bus subsystem may use multiple buses.

[0133] The computing device 410 may be of various types, including workstations, servers, computing clusters, blade servers, server farms, or any other data processing system or computing device. Due to the ever-changing nature of computers and networks, the description of the computing device 410 shown in Figure 4 is intended to be merely a specific example to illustrate several implementation forms. Many other configurations of the computing device 410 are possible, having more or fewer components than the computing device shown in Figure 4.

[0134] While several implementations have been described and illustrated herein, various other means and / or structures may be used to perform the functions described herein and / or to obtain one or more of the results and / or benefits, and each such variant and / or modification shall be considered within the scope of the implementations described herein. More generally, all parameters, dimensions, materials and configurations described herein are illustrative, and the actual parameters, dimensions, materials and / or configurations will depend on the specific one or more applications in which this teaching is used. A person skilled in the art will be able to recognize or verify many equivalents to the particular implementations described herein simply by using routine experiments. Therefore, it should be understood that the aforementioned implementations are presented only as examples, and implementations other than those specifically described and claimed may be practiced within the scope of the appended claims and their equivalents. The implementations of this disclosure cover each individual feature, system, article, material, kit and / or method described herein. In addition, any combination of two or more such features, systems, articles, materials, kits, and / or methods is included within the scope of this disclosure, provided that such features, systems, articles, materials, kits, and / or methods are not mutually inconsistent. [Explanation of Symbols]

[0135] 15 Storage 100 Environment 100' environment 103-A Local client device, client device 103-B Local client device, client device 105 Server devices, server computing devices 106 Process Automation Network 109 Display Devices 111 The First Robot 112 The Second Robot 121 Object, human operator, human 122 Forklifts 130 Industrial Facilities 131 First graphical representation, representation 132 Expression 140 Base Map 141 The first robot map 142 Second Robot Map 151 Symbols 152 Symbols 161 Expression, Symbol 162 Expressions, Symbols 170 Coordinate System 171 First Reference Frame 180 Industrial Components, Instruments 190 User Interface 191 User Interface 192 User Interface 200 ways 300 ways 410 Computing Devices 412 Bus subsystem 414 processors 416 Network Interface Subsystem 420 User Interface Output Devices 422 User Interface Input Devices 424 Storage subsystems 425 Memory subsystem, memory 426 File Storage Subsystem 430 Main Random Access Memory (RAM) 432 Read-only memory (ROM) 1051 Map-compatible engine 1052 Attitude change engine 1053 Rendering Engine 1054 Occurrence association engine

Claims

1. A method implemented by one or more processors, A step of identifying a base map of an industrial environment and a first robot map used by a first robot placed within the industrial environment, The base map differs from the first robot map in terms of steps, A step of generating first conversion data based on comparing the base map and the first robot map, A step of receiving a first environmental feature and a first robot map posture for the first environmental feature, which is located within a first frame of the first robot map, wherein the first feature and the first robot map posture are determined based on processing first sensor data located within the first frame and detected by the first robot. Using the first conversion data, the first robot map pose is converted to a first base map pose. The steps include receiving industrial data and a given map orientation corresponding to the industrial data, A step of determining that the aforementioned industrial data corresponds to the first environmental characteristic, The determination that the aforementioned industrial data corresponds in time to the detection of the first environmental feature, and Determining that the first base map orientation for the first environmental feature corresponds positionally to the given map orientation for the industrial data. Based on this, the steps include determining that the industrial data corresponds to the first environmental characteristic, A step of using the determined correspondence of the industrial data to the first feature when adapting an industrial process. Methods that include...

2. The method according to claim 1, wherein the first environmental feature corresponds to a movable object approaching a fixed component within the industrial environment.

3. The method according to claim 2, wherein the industrial data includes abnormal sensor readings for the fixed components in the industrial environment.

4. The step of using the determined correspondence of the industrial data to the first feature when adapting the industrial process is: A step of taking action on the industrial process, which involves fixed components within the industrial environment, based on the type of the first environmental characteristic. The method according to claim 1, including the method described in claim 1.

5. The method according to claim 1, wherein the first feature and the first robot map posture are determined based on processing the first sensor data using a machine learning model.

6. The method according to claim 1, wherein the given map orientation is located within the base map.

7. The method according to claim 1, wherein the industrial data is transmitted together with the given map orientation over one or more networks.

8. The method according to claim 1, wherein the base map includes components in the given map orientation, and the industrial data is associated with the components.

9. The method according to claim 8, wherein the industrial data includes the type of the component, the installation date of the component, and / or current or recent indications relating to the component.

10. A method implemented by one or more processors, A step of identifying a base map of an industrial environment, a first robot map used by a first robot located within the industrial environment, and a second robot map used by a second robot located within the industrial environment, The base map, the first robot map, and the second robot map are all different from each other in terms of steps and, The steps include generating first conversion data based on comparing the base map with the first robot map, The steps include generating second transformation data based on comparing the base map with the second robot map, A step of receiving a first environmental feature and a first robot map posture for the first environmental feature, which is located within a first frame of the first robot map, wherein the first environmental feature and the first robot map posture are determined based on processing first sensor data located within the first frame and detected by the first robot. A step of receiving a second environmental feature and a second robot map posture for the second environmental feature, which is located within a second frame of the second robot map, wherein the second environmental feature and the second robot map posture are determined based on processing second sensor data located within the second frame and detected by the second robot. Using the first conversion data, the first robot map pose is converted to a first base map pose. Using the second conversion data, the steps include converting the second robot map pose to the second base map pose, A step of rendering the base map in the first base map orientation with a first graphical representation of the first environment feature, and in the second base map orientation with a second graphical feature of the second environment feature. Methods that include...

11. The method according to claim 10, wherein the base map is further rendered in a graphical representation of industrial data, which is included in the base map but is absent from the first robot map and also absent from the second robot map.

12. The method according to claim 10, wherein the first environmental characteristic corresponds to a first object in the industrial environment.

13. The method according to claim 10, wherein the second environmental characteristic corresponds to a second object in the industrial environment.

14. The method according to claim 10, wherein the first transformation data includes a first transformation function that can be applied to transform any point in the first robot map into a corresponding point in the base map.

15. The method according to claim 10, wherein the second transformation data is separate from the first transformation function and includes a second transformation function applied to transform any point in the second robot map into a corresponding point in the base map.

16. The method according to claim 10, wherein the first feature and the first robot map posture are determined based on processing first sensor data using a first machine learning model.

17. The method according to claim 16, wherein the second feature and the second robot map posture are determined based on processing the second sensor data using a second machine learning model different from the first machine learning model.

18. The method according to claim 10, wherein the first robot map is generated from sensor readings of one or more first sensors of the first robot.

19. The method according to claim 10, wherein the second robot map is generated from sensor readings of one or more second sensors of the second robot.

20. A method implemented by one or more processors, A step of identifying a base map of an industrial environment and a first robot map used by a first robot placed within the industrial environment, The base map differs from the first robot map in terms of steps, A step of generating first conversion data based on comparing the base map and the first robot map, A step of receiving a first environmental feature related to equipment in the industrial environment and a first robot map posture for the first environmental feature, which is located within a first frame of the first robot map, wherein the first environmental feature and the first robot map posture are determined based on processing first sensor data located within the first frame and detected by the first robot. Using the first conversion data, the first robot map pose is converted to a first base map pose. The steps include receiving industrial data indicating an abnormal state and a given map orientation corresponding to the industrial data, A step of determining that the aforementioned industrial data corresponds to the first environmental characteristic, The determination that the aforementioned industrial data corresponds in time to the detection of the first environmental feature, and Determining that the first base map orientation for the first environmental characteristics corresponds positionally to the given map orientation for the industrial data. Based on this, the steps include determining that the industrial data corresponds to the first environmental characteristic, The steps include storing the determined correspondence of the industrial data to the first feature in a database accessible within the industrial environment. Methods that include...

Images