A method for analyzing an agent's environment in multidimensional space.

Abstract

A computer-implemented method and system for analyzing an environment of an agent in a multidimensional space is provided, the method including acquiring first sensor data at a first location of the agent, acquiring stored second sensor data from a second location, acquiring a plurality of first sub-regions of the first sensor data, acquiring a plurality of second sub-regions of the second sensor data, comparing the second sub-regions with each first sub-region using a similarity comparison measure to determine a first sub-region that is most similar to the second sub-region, determining associated relative rotations, and aggregating the relative rotations of the plurality of second sub-regions to obtain an action vector that indicates a probable direction of the agent from the first location to the second location.

Description

[Technical Field]

[0001] A system and computer-implemented method for understanding multidimensional space based on comparison of spatial observations with previously stored data, for the purposes of navigation in unknown environments and visual classification of objects and images. [Background technology]

[0002] Simultaneous Map and Localization (SLAM) is the problem and process of computationally constructing or updating a map of an unknown environment while simultaneously tracking the position of an agent within that environment. SLAM algorithms are based on concepts from computational geometry and computer vision and are used in robot navigation, robotic mapping, and odometry measurements in virtual or augmented reality.

[0003] SLAM algorithms are also used in autonomous vehicle control to map the unknown environment around the vehicle, and the resulting map information can be used to perform tasks such as path planning and obstacle avoidance.

[0004] Visual SLAM uses an approach consisting of a front-end that processes images from a camera and performs "feature extraction." This feature extraction process detects small regions of an image that meet certain visually distinguishable criteria, such as high local spatial frequency or contrast. These regions can then be saved for the purpose of finding those same regions in subsequent camera images taken at short intervals with the same camera. A camera model that accounts for camera lens distortion is then applied to map the locations of these features to real-world orientations. These are provided to a back-end algorithm called a bundle adjuster. The bundle adjuster compares the predicted location in 3D space of the features based on previous camera images with their new orientation from the current camera image, simultaneously determining the location of the features relative to each other in 3D space and the camera's position relative to the features. By repeating this process, the bundle adjuster tracks the moving camera's position and the location of the feature map over time.

[0005] This approach has several drawbacks that limit its effectiveness in real-world robotics applications. First, feature extraction and bundle adjustment are both computationally expensive processes. While feature extraction can be easily parallelized for efficiency, bundle adjustment is a strictly sequential process.

[0006] Second, bundle adjustment requires that new feature orientations and predicted feature locations converge to a stable solution, which is heavily affected by sensor noise and environmental changes such as wind blowing leaves off a bush. If a feature is incorrectly detected in the wrong location on the camera image, bundle adjustment will not be able to converge to a stable solution, and the system will become "lost" and unable to locate it.

[0007] To address convergence failures in bundle adjusters, the state of the art employs two approaches: one that mitigates the failure and the other that degrades performance if convergence fails, allowing the system to continue.

[0008] The first approach is outlier removal. A set of constraints is used to assess the likelihood that when a feature is found, it is the same as one found previously. For example, if a feature moves from one side of the camera to the other, it is unlikely to be the same unless the camera is moving very fast, and is rejected as a match. If it is close to its previous location, it is more likely to be the same feature and is accepted. Accepted features are passed to the bundle adjuster, while rejected features are not. Outlier removal is another process that is very computationally expensive, and is the largest computational component in some SLMA systems.

[0009] The second approach is visual inertial odometry, a system that uses the features identified by the feature extractor in a different way. A second source of motion information, linear acceleration measurements from an inertial sensor and tracking functions, is used to predict the camera's velocity rather than its position. This velocity measurement is integrated over time and can be used in conjunction with a bundle adjuster to correct for convergence failures. However, integrating VIO causes position errors to accumulate over time, and this approach to SLMA relies on a metric measurement of position, leading to failures at long integration times. Because the same features are used by VIO and bundle adjustment, feature extraction becomes a single point of failure for the entire system.

[0010] Another weakness of the feature extraction front end is that the feature size results in a relatively low dimensionality of the data for a single feature, which increases the likelihood that multiple parts of the image will match the feature, especially in aliased environments such as a hotel hallway where identical doors are evenly spaced. In such environments, the bundle adjuster may converge to the wrong position.

[0011] Loop closure is another weakness of the current technology. When a moving camera traverses a complete loop and returns to the same location from a different direction, the system needs to recognize that the camera has returned to the same location. Because the system is metric, it also needs to ensure that the map matches the same position at the same location. Due to errors accumulated in the bundle adjustment process, this is rarely the case; there is usually a metric error between the representations of the positions at both ends of the loop. The system then needs to go back and adjust all the features stored throughout the loop to correct for and remove this error. This loop closure process is computationally expensive and prone to error.

[0012] The computational complexity of algorithms that can handle such environments and problems is very high, limiting the platforms they can be deployed on. Furthermore, the memory consumption of the generated maps is very large, ranging from hundreds of megabytes to several gigabytes.

[0013] It is therefore appreciated that there is a need for a localization and mapping method and system that allows an agent to understand the structure and identity of a space and the objects therein, that is less prone to error, and that is less computationally and memory intensive. Summary of the Invention

[0014] This summary is provided to introduce some concepts in a simplified form that are further described in the detailed description.

[0015] According to a first aspect of the present disclosure, a computer-implemented method for analyzing an environment of an agent in a multidimensional space is provided, the method including: acquiring first sensor data describing an environment around the agent at a first location of the agent; searching stored second sensor data describing an environment around a second location exhibiting a characteristic in the multidimensional space for comparison with the first sensor data; acquiring a plurality of first sub-regions of the first sensor data, each first sub-region describing a respective first portion of the environment around the agent at the first location, each first portion being associated with a respective first direction from the first location; and acquiring a plurality of second sub-regions of the second sensor data, each second sub-region describing a respective first portion of the environment around the second location. describing respective second portions of the environment, each second portion being associated with a respective second direction from the second location; for each second sub-region, comparing the second sub-region with each first sub-region using a similarity comparison measure to determine a first sub-region that is most similar to the second sub-region; and determining a relative rotation between the second direction associated with the second sub-region and the first direction associated with the most similar first sub-region, the method further including aggregating the relative rotations of the multiple second sub-regions to obtain an action vector indicating an estimated direction from the first location of the agent to the second location that characterizes the multidimensional space.

[0016] The first and second subregions are directional portions of the first and second sensor data, formed from contiguous subsets of the first and second sensor data. Comparing these subregions, rather than the entire first and second sensor data, results in directional variations, such that subregions may closely match one another in one directional comparison but may not match in another. This concept allows multiple sets of subregions to be compared to effectively determine a direction indicating a second location in 2D or 3D space, thereby forming the direction of an action vector. Therefore, this method provides a way to relocalize to a previously visited location (second location) based purely on a comparison of the first and second sensor data.

[0017] The first sensor data and the second sensor data are first image data and second image data, respectively, and each of the first sub-regions includes a contiguous subset of pixels of the first image data and each of the second sub-regions includes a contiguous subset of pixels of the second image data. The image data is modified to reduce size and improve computational efficiency. The first and second sensor data may be any data capable of geometrically representing the environment.

[0018] The first and second sensor data may be arranged into a first vector and a second vector concatenated from the first and second image data, respectively. The image data may initially be an array with multiple channels per pixel. Converting this to a vector allows for computational efficiency in terms of performing sub-region comparisons.

[0019] The similarity measure may be the dot product between the first subregion and the second subregion, which occurs for each comparison of the subregions.

[0020] Obtaining the first sensor data includes obtaining original first sensor data of the agent's environment at a first location, processing the original first sensor data to reduce a size of the original first sensor data, and obtaining first sensor data (the first sensor data is in a reduced size format relative to the original first sensor data), wherein the saved second sensor data is also saved in a reduced size format, and the method further includes obtaining original second sensor data of the agent's environment at a second location, processing the original second sensor data to reduce a size of the original second sensor data, and obtaining second sensor data (the second sensor data is in a reduced size format relative to the original second sensor data).

[0021] The original first sensor data and the original second sensor data may be the original first image data and the original second image data, respectively. Thus, the first and second sensor data are reduced-size forms of the original first and second image data, respectively. The processing may include several operations, including, but not limited to, filtering, masking, blurring, etc.

[0022] Processing the original first sensor data and the original second sensor data may include applying one or more filters and / or masks to the original first sensor data and the original second sensor data to reduce the size of each dimension of the original first sensor data and the original second sensor data, respectively.

[0023] The original first sensor data and the original second sensor data may be original first image data and original second image data, respectively.

[0024] Obtaining a plurality of first sub-regions of the first sensor data includes iteratively applying a mask to the first sensor data to extract each of the plurality of first sub-regions, where in each iteration the mask or the first sensor data is sorted by at least one data entry / cell in one dimension of the first sensor data; and obtaining a plurality of second sub-regions of the second sensor data includes iteratively applying a mask to the second sensor data to extract each of the plurality of second sub-regions, where in each iteration the mask or the second sensor data is sorted by at least one data entry / cell in one dimension of the second sensor data.

[0025] There are at least four repeats, such that there are at least four first subregions and at least four second subregions.

[0026] The mask may be smaller than the first and second sensor data, with a major dimension of the mask being 50% or less the size of the corresponding major dimension of the first and second sensor data, or 25% or less the size of the corresponding major dimension of the first and second sensor data.

[0027] The first and second sensor data are arranged in a first array and a second array, respectively, the first array and the second array having dimensions of X×Y cells, and the mask having dimensions of (Xm)×Y cells, where m is a positive integer.

[0028] The first array and the second array have a plurality of channels, and the dimensions of the array are X x Y x Z (Z is a positive integer).

[0029] The method may further include moving the agent from the first location according to the action vector.

[0030] The method may further include, after moving the agent to a new position according to the action vector, acquiring third sensor data at the agent's new position, where the third sensor data describes an environment surrounding the agent; acquiring a plurality of third sub-regions of the third sensor data, where each third sub-region describes a respective third portion of the environment surrounding the agent at the third location, where each third portion is associated with a respective third direction from the third location; and, for each second sub-region, comparing the second sub-region with each third sub-region using a similarity comparison measure to determine a third sub-region that is most similar to the second sub-region; determining a relative rotation between the second direction associated with the second sub-region and the third direction associated with the most similar third sub-region, and the method further includes aggregating the relative rotations of the plurality of second sub-regions to obtain an updated action vector, where the updated action vector indicates an estimated direction of the agent from the third location to the second location that is indicative of the multidimensional spatial characteristic. The third sensor data may be of the same type as the first sensor data, and thus may be image data or other data capable of providing a geometric representation of the environment.

[0031] The method further includes confirming the presence of the environmental feature in the multidimensional space when the observation comparison measure meets or exceeds a confirmation threshold level, wherein the observation comparison measure is associated with the similarity measure and / or the action vector.

[0032] Determining the relative rotation may include determining an offset angle between the first direction and the second direction.

[0033] The method further includes determining a magnitude of the action vector for the pair of opposing second sub-regions, determining an average offset angle from the offset angles associated with the pair of opposing second sub-regions, and assigning a magnitude based on the average offset angle, wherein the size of the average offset angle is proportional to the magnitude.

[0034] The method may further include determining a respective magnitude for each of a plurality of pairs of offset angles associated with opposing pairs of second sub-regions and aggregating / averaging the respective magnitudes to form a magnitude of the action vector.

[0035] The first sensor data may represent a substantially 360-degree field of view around the agent, and the second sensor data may represent a substantially 360-degree field of view around the second location, where each first portion described by each first sub-region is a portion of the 360-degree field of view around the agent, and each second portion described by each second sub-region is a portion of the 360-degree field of view around the second location. Thus, the first and second sensor data may be obtained from a visual panorama around the agent.

[0036] Each of the plurality of first sub-regions can overlap with at least an adjacent first sub-region, and each of the plurality of second sub-regions overlaps with at least an adjacent second sub-region.

[0037] A feature of the environment can be either a specific location in multidimensional space, an image or part thereof, or an object or part thereof.

[0038] The agent may be virtual, the multidimensional space is a two-dimensional or three-dimensional virtual space, and the first sensor data and the second sensor data are acquired using virtual sensors.

[0039] Physical entities include, for example, robots and vehicles.

[0040] The second sensor data may describe an environment indicating a target location in multidimensional space, such that features of the environment described by the second sensor data are associated with the target location. The method further includes navigating the agent from the first location to the target location according to the action vector.

[0041] The second sensor data forms part of a set of second sensor data, the set including a plurality of instances of the second sensor data, each instance describing an environment indicating a respective location in a multidimensional space, and the method further includes iteratively navigating the agent from the first location to a target location at each location indicated by the plurality of instances of the second sensor data, and in each iteration obtaining an action vector for one of the instances of the second data, and moving from the agent's location to the respective location indicated by the one instance of the second data according to the action vector, and iteratively navigating the agent until the target location is reached.

[0042] Navigating the agent from the first location to the goal location according to the action vector may form a primary navigation process, and the method further includes navigating the agent from an initial position in the multidimensional space to the first location using a secondary navigation process, and switching from the secondary navigation process to the primary navigation process at or near the first location.

[0043] The secondary navigation process may be configured to use a positioning system.

[0044] The positioning system is a satellite-based radio navigation system.

[0045] The second sensor data may be stored in a remotely accessible database, so that the agent can remotely retrieve the second sensor data when needed to perform the comparison.

[0046] The method may further include recording second sensor data at the second location with the recording agent and storing the second sensor data in a remotely accessible database.

[0047] The method further includes recording metadata corresponding to the second sensor data and associating the metadata with the second sensor data, the metadata including at least one of metric information regarding the second location in multidimensional space, information regarding variability of one or more portions of an environment surrounding the second location captured in the second sensor data, information regarding one or more fixed features of the environment surrounding the second location captured in the second sensor data, time information regarding the capture of the second sensor data, an availability metric indicating a predicted or predetermined availability of the signal at the second location, and information regarding the particular feature captured in the second sensor data. The method further includes storing the metadata along with the corresponding second sensor data in a remotely accessible database.

[0048] The metadata may include metric information about a second location within the multidimensional space, and the method further includes obtaining metric information about a first location of the agent within the multidimensional space; determining from the metric information about the first location and the metric information about the second location that the second location is near the first location; and obtaining second sensor data from a remotely accessible database based on a determination that the second location is near the first location.

[0049] The metadata may include information regarding variability of one or more portions of the environment surrounding the second location captured in the original second sensor data, and processing the original second sensor data by applying one or more filters and / or masks to the original second sensor data to reduce the size of each dimension of the original second sensor data includes filtering or masking one or more portions of the environment surrounding the second location captured in the original second sensor data based on the metadata so that the reduced form of the second sensor data does not include one or more portions of the environment indicated by information in the metadata.

[0050] The metadata may include information regarding one or more fixed features of the environment surrounding the second location captured in the original second sensor data, and processing the original second sensor data by applying one or more filters and / or masks to the original second sensor data to reduce the size of each dimension of the original second sensor data includes maintaining the one or more fixed features of the environment surrounding the second location captured in the original second sensor data based on the metadata, such that the reduced form of the second sensor data includes the one or more fixed features of the environment indicated by the information in the metadata.

[0051] The metadata may include an availability metric indicating a predicted or predetermined availability of a positioning system signal and / or a data connection signal at the second location, and the method further includes navigating the agent based on the availability metric according to the positioning system signal and / or determining when to download information from the remotely accessible database based on the availability metric according to the data connection signal.

[0052] The method further includes selecting, by the recording agent, a second location for recording the second sensor data, the selection being based on location selection criteria including at least one of a search status associated with the second location and a determination whether the second location is traversable or impassable by the agent.

[0053] According to a second aspect, there is provided a system including a processor, a memory, and a sensor or virtual sensor configured to acquire spatial data describing a multi-dimensional spatial environment in a local vicinity of the sensor or virtual sensor, the memory having instructions stored thereon that, when executed by the processor, cause the system to perform the method of the first aspect above.

[0054] The multidimensional space may be a physical space, and the system may be a robot or vehicle equipped with sensors, the robot or vehicle being an agent, and further comprising a controllable locomotion module configured to move the robot or vehicle within the physical space.

[0055] The sensors may include one or more of an ultraviolet imager, a camera, a LIDAR sensor, an infrared sensor, a radar, a tactile sensor, or other sensors configured to provide spatial information.

[0056] The system comprises a plurality of devices, the plurality of devices including a robot or vehicle and an additional computing device, the additional computing device having a processor, memory, and sensors, the additional computing device being a recording agent, and the system is configured to perform any of the method steps described above in the first aspect.

[0057] The system is a computer system including a virtual sensor, the computer system configured to operate in a virtual space, and the agents are represented by points in the virtual space.

[0058] According to a third aspect, there is provided a computer program stored on a non-transitory computer readable medium, the computer program being configured, when executed by a processor, to cause the processor to perform a method according to any of the first aspect above.

[0059] According to a fourth aspect, there is provided a computer-implemented method for analyzing an environment in a multidimensional space, the method comprising: recording, by a recording agent in the multidimensional space, sensor data describing an environment surrounding the recording agent; processing the sensor data to identify one or more features present in the environment captured in the sensor data; and associating metadata with the identified one or more features, wherein the features comprise at least one of a position of the recording agent at the time the sensor data was captured in the multidimensional space, variability of one or more portions of the environment surrounding the recording agent, and one or more fixed features of the environment surrounding the recording agent. The method further comprises storing the sensor data and associated metadata in a remotely accessible database.

[0060] The method may further include retrieving, by a navigation agent in the multidimensional space, the stored sensor data and associated metadata from a remotely accessible database, and navigating the multidimensional space using the sensor data and associated metadata by comparing the sensor data retrieved by the navigation agent with the stored sensor data.

[0061] Using the sensor data and associated metadata to navigate the multidimensional space further includes reducing the stored sensor data according to the associated metadata, wherein the reduced stored sensor data includes sensor data corresponding to non-variable or fixed features identified in the environment of the recording agent.

[0062] The method further includes reducing the sensor data according to associated metadata before storing the sensor data in the remotely accessible database, where storing the sensor data includes storing sensor data corresponding to non-variable or fixed features identified in the environment and discarding sensor data corresponding to variable or non-fixed features identified in the environment.

[0063] The sensor data may form a panorama around the recording agent.

[0064] According to a fifth aspect, there is provided a system including a first computing device and a server, wherein the first computing device includes a processor, a memory, and a sensor or virtual sensor and is configured to acquire spatial data describing a multidimensional spatial environment in a local vicinity of the sensor or virtual sensor, and the server includes a memory having a remotely accessible database stored thereon, and the system is configured to perform the method of the fourth aspect.

[0065] The system further comprises a robot or vehicle or virtual robot comprising a processor, a memory, and a sensor or virtual sensor configured to acquire spatial data describing an environment in a multidimensional space in a local vicinity of the sensor or virtual sensor, the robot or vehicle or virtual robot being a navigation agent, and the system configured to perform the method of the fourth aspect.

[0066] According to a sixth aspect, there is provided a computer-implemented method for analyzing a multidimensional space. The method explores attributes of the space based on a series of observations and displacements performed by an agent within the space. The method includes: making a first observation from a first location of the agent within the space, where the first observation includes data describing a first portion of the space in a local neighborhood of the first location; comparing the first observation to a set of stored observations and displacements to identify a first stored observation from the set of stored observations and displacements that is most similar to the first observation; determining a hypothesis of an attribute of the space based on the first stored observation, where the first stored observation is associated with the hypothesis; obtaining a hypothesis subset of the set of stored observations and displacements, where the hypothesis subset includes at least the first stored observation, a predicted observation, and a predicted displacement required to arrive at the predicted observation, where the predicted observation and predicted displacement are also associated with the hypothesis; and testing the hypothesis. The hypothesis is verified by moving the agent to a second location in the space based on a movement function, the movement function depending on the predicted displacement of the hypothesis subset, making a second observation from the agent's second location in the space, the second observation including data describing a second portion of the space in a local neighborhood of the second location, comparing the second observation with the predicted observation to obtain an observation comparison measure, and / or comparing the actual displacement between the first location and the second location with the predicted displacement to obtain a displacement comparison measure, and adjusting, maintaining, or confirming the hypothesis based on the observation comparison measure and / or the displacement comparison measure, and confirming the hypothesis includes determining that a hypothesis confirmation condition is met and exploring attributes of the space.

[0067] The movement function includes a first movement component that depends on the predicted displacement of the hypothesis subset and a second movement component that depends on the predicted observation of the hypothesis subset, the first movement component being weighted according to the remaining distance of the predicted displacement, the first movement component weakening as the agent moves along the predicted displacement, and the second movement component being weighted according to the predicted observation, the second movement component becoming stronger as the agent approaches the predicted observation.

[0068] The method may include measuring an actual displacement traveled from a first location to a second location.

[0069] When the agent reaches the end of the predicted displacement or within a displacement threshold distance of the end of the predicted displacement, the first movement component of the movement function is weighted as zero and the first movement component no longer contributes to the movement function.

[0070] The second movement component includes an expected direction and magnitude relative to the predicted observation, the direction and magnitude being obtained by making a transition observation away from the first location after or during the agent's movement and performing a transition comparison of the transition observation with the predicted observation to obtain the direction and magnitude relative to the predicted observation, and the second movement component being weighted according to the transition comparison, the stronger the comparison with the predicted observation, the stronger the weighting of the second movement component.

[0071] The method may include repeatedly updating a second movement component including a direction and a magnitude as the agent moves away from the first location, the method including making a plurality of transition observations.

[0072] The method includes using the movement function to stop the agent at a second location, the second location being a location of one of the transition observations of a second movement component of the movement function, and a transition comparison of one of the transition observations indicating a best match with a predicted observation from the plurality of transition observations, and / or a location where the magnitude of the second movement component function is equal to or below a magnitude threshold. One of the transition observations at the second location can be a second observation, and the transition comparison indicating a best match can be the second observation.

[0073] The observation comparison measure can include one or more of a first observation comparison measure component, a second observation comparison measure component, a third observation comparison measure component, and a fourth observation comparison measure component, where the first observation comparison measure component is a similarity measure indicating the similarity between the second observation and the predicted observation, the second observation comparison measure component is a measure indicating the direction of the predicted observation relative to the second observation, the third observation comparison measure component is a vector pointing from the second observation to the predicted observation, and the fourth observation comparison measure component is a second similarity measure indicating the similarity between the second observation and the predicted observation.

[0074] The observation comparison measure may include first, second, and third observation comparison measure components, which effectively provide a method for determining the similarity between the compared observations, the directional change between the compared observations, and the direction and magnitude (e.g., in the form of a vector) from the agent's location to the location of the compared observation.

[0075] The observational comparison scale may further include a fourth observational comparison scale component, which provides redundancy and can be advantageously used to distinguish the first observational comparison scale component from environmental factors and characteristics.

[0076] The first observation comparison measure component may be formed by an observation similarity function that takes the dot product of the second observation and the predicted observation, where the second observation and the predicted observation are vectors, and in particular unit vectors.

[0077] The third observation comparison measure component can be formed by dividing the predicted observation into a first plurality of sub-regions, obtaining first and second observation comparison measure components for a comparison of the first plurality of sub-regions with a second plurality of sub-regions of the second observation, such that each of the first plurality of sub-regions is associated with a first and second observation comparison measure component, and aggregating the first and second observation comparison measure components to form a vector from the second observation to the predicted observation.

[0078] Adjusting, maintaining, or confirming the hypothesis based on the hypothesis testing may include adjusting the hypothesis when the observed comparison measure is below a first observation threshold level, maintaining the hypothesis when the observed comparison measure is above a first observation threshold level, and confirming the hypothesis when the observed comparison measure is above a hypothesis-confirming observation threshold level (wherein the hypothesis-confirming observation threshold level is a hypothesis-confirming condition), or adjusting the hypothesis when the variance comparison measure is below a first variance threshold level, maintaining the hypothesis when the variance comparison measure is above the first variance threshold level, and confirming the hypothesis when the variance comparison measure is above a hypothesis-confirming threshold level. confirming the hypothesis if the two-dimensional similarity measure exceeds a first two-dimensional similarity measure threshold level (wherein the hypothesis-confirming threshold level is a hypothesis-confirming condition); or combining the observation comparison measure and the displacement comparison measure to form a two-dimensional similarity measure, adjusting the hypothesis if the two-dimensional similarity measure is below a first two-dimensional similarity measure threshold level, maintaining the hypothesis if the two-dimensional similarity measure exceeds the first two-dimensional similarity measure threshold level, and confirming the hypothesis if the two-dimensional similarity measure exceeds a hypothesis-confirming two-dimensional similarity measure threshold level (wherein the hypothesis-confirming two-dimensional similarity measure threshold level is a hypothesis-confirming condition).

[0079] The set of stored observations is associated with a plurality of hypotheses and includes a plurality of hypothesis subsets, each including at least one predicted observation and predicted displacement associated with a particular hypothesis.

[0080] Adjusting the hypothesis may include rejecting the hypothesis, moving the agent from the second location to the first location according to a negative displacement of the actual displacement, selecting a new hypothesis from among the plurality of hypotheses based on a comparison of the first observation with the set of stored observations and displacements, and identifying a next stored observation from the set of stored observations and displacements that is next-similar to the first observation.

[0081] The hypothesis subset may include multiple predicted observations and multiple predicted displacements predicted to be required to travel between the predicted observations.

[0082] Maintaining the hypotheses may include obtaining a next predicted observation and a next predicted displacement from the hypothesis subset, and repeating the testing of the hypotheses with respect to the next predicted observation and the next predicted displacement.

[0083] The method may include repeating the testing of the hypothesis for multiple predicted observations and multiple predicted displacements of the hypothesis subset until the hypothesis is adjusted or confirmed.

[0084] The attributes of the space to be explored are objects or images in the space, the hypotheses indicate predicted objects or images, predicted observations and predicted displacements associated with the hypotheses are associated with the predicted objects or images, and exploring the attributes of the space includes identifying the objects or images in the space as the predicted objects.

[0085] The attribute of the space to be explored is a destination within the space, the hypotheses indicate routes to the destination, predicted observations and predicted displacements associated with the hypotheses are associated with expected routes to the destination, and exploring the attribute of the space includes reaching the destination within the space.

[0086] The space may be physical or logical, and may be two or three dimensional.

[0087] The method further includes extracting a set of low-level spatial features from the physical or logical space, wherein determining a hypothesis based on the set of stored predicted observations and displacements includes processing the first observations and the low-level spatial features to determine the hypothesis, wherein the set of low-level spatial features are assigned weighting coefficients in determining the hypothesis.

[0088] The method may include processing the second observation and / or the nth observation to verify a hypothesis regarding the physical or logical space; extracting a second or nth set of low-level spatial features from the physical or logical space; and processing the second or nth observation and the second or nth set of low-level spatial features to verify the hypothesis, where n is a positive real number.

[0089] According to a seventh aspect of the present disclosure, there is provided a system including a processor, a memory, and a sensor or virtual sensor configured to acquire spatial data describing a portion of a space in a local vicinity of the sensor or virtual sensor, the memory having instructions stored thereon that, when executed by the processor, cause the system to perform the method outlined in the sixth aspect above.

[0090] The system is a robot or vehicle equipped with sensors, the robot or vehicle being an agent configured to operate within a physical space, and further comprising a controllable locomotion module configured to move the robot or vehicle within the physical space, the sensors being cameras, LIDAR sensors, infrared sensors, radar, tactile sensors, or other sensors configured to provide spatial information.

[0091] The system is a computer system that includes virtual sensors, the computer system is configured to operate on a logical space, and the agents are represented by points in the logical space.

[0092] The logical space is an image, the points in the logical space are pixels of the image, and the virtual sensor is configured to capture a portion of the image data in the vicinity of the pixel.

[0093] According to an eighth aspect of the present disclosure there is provided a computer program stored on a non-transitory computer readable medium, the computer program being configured to, when executed by a processor, cause the processor to perform a method according to the sixth aspect above.

[0094] According to a ninth aspect of the present disclosure, there is provided a computer-implemented method for analyzing a physical or logical space based on a series of observations and displacements performed by an agent in the physical or logical space, and determining attributes of the physical or logical space.The method includes selecting a hypothesis for an attribute of the physical or logical space based on a comparison of a current observation of the agent in the physical or logical space with a current predicted observation of a stored set of predicted observations and displacements associated with the hypothesis, where the current observation is recorded from the agent's current position in the physical or logical space and includes data describing a current portion of the logical or physical space in a local vicinity of the current position, obtaining a next predicted displacement and a next predicted observation from the set of predicted observations and displacements associated with the hypothesis for the current predicted observation, iteratively comparing the next predicted observation associated with the hypothesis with the current observation to obtain an observation comparison measure indicative of a similarity between the next predicted observation and the current observation, determining a direction and magnitude of movement required to reach the predicted observation based on the observation comparison measure, moving the agent according to the determined movement and next predicted displacement such that the agent is moved from its current location to a next location in the physical or logical space, and taking a next observation from a next location of the agent, where the next observation includes data describing a next portion of the logical or physical space in a local vicinity of the next location; setting the next observation as a current observation and setting the next location as the current location; repeating the method until an observation comparison condition is reached in which the agent is at a final location in the physical or logical space; measuring an actual displacement from the original current location to the final location in the physical or logical space and comparing the actual displacement with the next predicted displacement to obtain a displacement comparison measure; determining whether a hypothesis confirmation condition is met;

[0095] According to a tenth aspect of the present disclosure, there is provided a computer-implemented method for analyzing a physical or logical space based on a series of observations and displacements performed by an agent within the physical or logical space and determining attributes of the physical or logical space, the method including: recording a first observation from a first location of the agent within the logical or physical space, wherein the first observation includes data describing a first portion of the logical or physical space in a local vicinity of the first location; processing the first observation to determine hypotheses based on a set of stored predicted observations and predicted displacements, wherein each of the stored sets of predicted observations and predicted displacements is associated with one or more hypotheses; determining a direction and magnitude of a required movement based on predicted observations associated with the hypothesis; moving the agent according to the required movement such that the agent is moved from a first location in the physical or logical space to a second location; recording second observations from the agent's second location in the logical or physical space, where the second observations include data describing a second portion of the logical or physical space in a local vicinity of the second location; processing the second observations to verify the hypothesis related to the physical or logical space; adjusting or maintaining the hypothesis based on the processing of the second observations; and determining whether a hypothesis confirmation condition is met.

[0096] Processing the first observation to determine a hypothesis may include comparing the first observation with a predicted observation from the set of stored predicted observations to determine a similarity measure between the first observation and the predicted observation from the set of stored predicted observations, and obtaining a hypothesis corresponding to the predicted observation from the set of stored predicted observations having the strongest similarity measure.

[0097] According to an eleventh aspect of the present disclosure, there is provided a computer-implemented method for generating a map of a physical or logical space based on a series of observations and displacements performed by an agent in the physical or logical space. The method includes recording a first observation from a first location of an agent in logical or physical space, the first observation including data describing a first portion of the logical or physical space in a local vicinity of the first location; moving the agent by a first displacement to displace the agent from the first location in the physical or logical space to a second location; generating a second observation from the agent's second location in the logical or physical space, the second observation including data describing the second portion of the logical or physical space in a local vicinity of the second location; recording the second observation, recording the first displacement from the first observation to the second observation, and recording a negative first displacement from the second observation to the first observation; and repeating this process for at least n-1 displacements, n observations, and n-1 negative displacements to generate a set of connected n-1 displacements and n observations that define a map of the physical or logical space, where n is a positive real number.

[0098] Recording observations and displacements can be seen as making observations and displacements by an agent.

[0099] Before recording the second or nth observation, the method compares the nth observation with the previous n-1th observation to determine an observation similarity measure between the nth observation and the n-1th observation; compares the observation similarity measure with a predetermined maximum similarity threshold; if the similarity measure does not exceed the maximum similarity threshold, record the nth observation; record an n-1th displacement from the n-1th observation to the nth observation; record a negative n-1th displacement from the nth observation to the n-1th observation; or if the similarity measure exceeds the maximum similarity threshold, move the agent by a further displacement to move the agent in physical or logical space. generating an n+1 observation from the agent's n+1 position in the logical or physical space (where the n+1 observation includes data describing the n+1 portion of the logical or physical space that is in a local vicinity of the n+1 position); comparing the n+1 observation with a previous n-1 observation to determine an observation similarity measure between the n+1 observation and the n-1 observation; comparing the observation similarity measure to a predefined maximum similarity threshold; and repeating this process until the predefined maximum similarity threshold is not exceeded.

[0100] The method may include forming an identity from the n observations and the n-1 displacements, where the identity includes at least a subset of the n observations and the n-1 displacements, and storing the identity.

[0101] The method may further include associating or labeling the identity and the corresponding subset of observations and displacements that form it with an attribute of the space.

[0102] Multiple identities may be formed from n observations and n-1 displacements.

[0103] A map and / or identity may include more or fewer than n-1 displacements, since an observation may have more than one displacement relative to other observations, such as imaging a set of three observations, with each observation connected to the other two to form a triangle with three observations and three displacements. It is understood that if the number of observations is large, the number of displacements may be greater than the number of observations.

[0104] According to a twelfth aspect of the present disclosure, there is provided a method of navigating a map of a physical or logical space generated according to the eleventh aspect above, the method comprising: obtaining a global target observation (wherein the target observation is an observation of a set of n observations and n-1 displacements); making a first observation from a first location of an agent in the logical or physical space (wherein the first observation includes data describing a first portion of the logical or physical space in a local vicinity of the first location); comparing the first observation with the set of n observations and n-1 displacements to obtain a global target observation of the logical or physical space; The method includes identifying a first target observation of a set of displacements (where the first target observation is an observation connected to an overall target observation in a map of physical or logical space that is most similar to the first observation); determining a first movement from the first observation to the first target observation based on a comparison between the first observation and the first target observation; relocalizing the agent to the first target observation by moving the agent according to the first movement; and then iteratively moving the agent toward the overall target observation in the map through n-1 displacements and n observations by moving the agent according to the n-1 displacements, making transition observations at the n observations, comparing the transition observations to the n observations, comparing the measured displacements to the n-1 displacements, and adjusting the agent's movement based on the comparison until the overall target observation is reached.

[0105] According to a thirteenth aspect of the present disclosure, there is provided a computer-implemented method for determining similarity between a portion of a physical or logical space navigable by an agent and stored spatial data for identifying the agent's location in the physical or logical space, the method including, in order, recording a series of observations from respective positions of the agent in the physical or logical space, where each observation includes data describing a respective portion of the logical or physical space in a local neighborhood of the agent, moving the agent between the respective positions and recording a series of displacements between the respective positions, comparing each of the series of observations with a set of previously stored observations to obtain an observation similarity measure for each observation with respect to one or more of the previously stored observations, comparing each of the displacements with the set of previously stored displacements to obtain a displacement similarity measure for each displacement with respect to one or more of the previously stored displacements, and determining that the series of observations and / or displacements are similar to the previously stored series of observations and / or displacements if the observation similarity measure and / or the displacement similarity measure exceed a confidence threshold.

[0106] The method may include combining the observation similarity measure and the displacement similarity measure to generate a two-dimensional identity comparison measure, and processing the identity comparison measure to determine a type of similarity between the observation and displacement and a previously stored observation and displacement.

[0107] According to a fourteenth aspect of the present disclosure, there is provided a computer-implemented method for performing simultaneous localization and mapping in logical or physical space, the method including: recording a first observation from a first location in the logical or physical space, wherein the first observation includes data describing a first portion within the logical or physical space; performing a first displacement from the first location in the logical or physical space to an unknown second location; recording a second observation from the second location, wherein the second observation includes data describing the second portion within the logical or physical space; repeating the method for n observations and n displacements, where n is a positive real number, to form a series of sets of successive observations and corresponding displacements; and comparing the set of successive observations and corresponding sets of displacements with a saved set of observations and a saved set of displacements, wherein the saved sets are captured in a closed loop, and the comparing includes comparing using at least one of a displacement comparison measure and one or more observation comparison measures to determine whether a feature corresponding to the saved set is present in the space.

[0108] The method is effectively used to analyze a multidimensional space based on a series of observations and displacements performed by an agent within the space and to explore attributes of the space. The method includes identifying hypotheses about attributes of the space and verifying the hypotheses by making sequential observations and displacements from the agent's position within the space, comparing the observations and displacements with a set of stored observations and displacements to obtain an observation comparison measure and / or a displacement comparison measure, and adjusting, maintaining, or confirming the hypotheses based on the observation comparison measure and / or the displacement comparison measure.

[0109] Each of the above aspects pertains to an agent in a space configured to move and observe the space in a minimal way so that a network of connected observations and displacements describes the space in a map. Current observations and displacements made by the agent are compared to stored observations and displacements on the map and may influence the behavior of the agent in terms of movement and function. In this way, the agent can move and / or explore the space while simultaneously observing the space for purposes of image and object recognition.

[0110] An agent may be a real object such as a robot, a vehicle, an autonomous vehicle, or an unmanned aerial vehicle, but is not limited to such possibilities.

[0111] A group of agents can be used together to communicate with a computer system or the like to update a global map or resources based on the spaces in which different agents reside.

[0112] Embodiments of the present invention will now be described, by way of example only, with reference to the following drawings, in which: [Brief explanation of the drawings]

[0113] [Figure 1] FIG. 1 is a schematic diagram of a physical space analyzed by an agent. [Figure 2] 1 is a schematic diagram of the physical / logical space analyzed by the agent. [Figure 3] FIG. 1 is a flow diagram outlining a method for searching attributes of a space. [Figure 4] FIG. 1 is a detailed flow diagram outlining a portion of a method for searching attributes. [Figure 5] FIG. 1 is a schematic diagram showing an overview of a system for carrying out the method. [Figure 6] FIG. 2 is a schematic diagram of the functions performed by the agent in various ways. [Figure 7] FIG. 1 is a schematic diagram illustrating various ways in which an agent can move according to a method. [Figure 8]FIG. 1 is a schematic diagram illustrating identities that can be used to generate a map of a space. [Figure 9] 1 shows various schematic diagrams of vectors and matrices obtained in various ways. [Figure 10] 1 shows various schematic diagrams of vectors and matrices obtained in various ways. [Figure 11] FIG. 10 is a flow diagram illustrating a process for forming subregions from observation vectors in accordance with various embodiments. [Figure 12] 1 illustrates a schematic diagram of a spatial set of sub-regions corresponding to a current observation vector, according to various embodiments. [Figure 13A] 1 illustrates a schematic diagram of a comparison of sub-regions of a saved observation vector with a sub-region of a current observation vector, according to various embodiments. [Figure 13B] 1 illustrates a schematic diagram of a comparison of sub-regions of a saved observation vector with a sub-region of a current observation vector, according to various embodiments. [Figure 14] 1 illustrates a schematic diagram of a comparison of a saved observation vector with a current observation vector performed by an agent, according to various embodiments; [Figure 15A] 10A-10C show schematic diagrams of an agent's axes overlaid with an offset determined from a comparison of a current sub-region and a saved sub-region, according to various embodiments. [Figure 15B] 10A-10C show schematic diagrams of an agent's axes overlaid with an offset determined from a comparison of a current sub-region and a saved sub-region, according to various embodiments. [Figure 15C] 10A-10C show schematic diagrams of an agent's axes overlaid with action vectors determined from a comparison of a current sub-region and a saved sub-region, according to various embodiments. [Figure 15D] 10A-10C show schematic diagrams of an agent's axes overlaid with action vectors determined from a comparison of a current sub-region and a saved sub-region, according to various embodiments. [Figure 16] 1 illustrates a flow diagram of a method for using two navigation processes according to various embodiments. [Figure 17]1 illustrates a schematic diagram of the results of a method for using metadata to associate sensor data, according to various embodiments. [Figure 18] 1 illustrates a schematic diagram of the results of a method for using metadata to associate sensor data, according to various embodiments. DETAILED DESCRIPTION OF THE INVENTION

[0114] Common reference numbers are used throughout the drawings to denote like features.

[0115] The examples described herein relate to computer-implemented methods and systems for analyzing a multidimensional space and determining the structure of that space or one or more attributes of that space. The approach taken in the examples described herein treats the problem of understanding space as a closed-loop problem, where sensor data from sensors or virtual sensors that describe an agent's environment directly drives changes in the agent's behavior.

[0116] The term "agent" is used to refer to either or both physical agents that exist in the physical space of the real world, such as vehicles or robots, and non-physical agents that exist in logical space, such as points or pixels in an image or map.

[0117] Agents can move within their respective spaces. In physical spaces, agents move by activating movement modules, such as actuators and motors. In logical spaces, agents move from a first point or pixel to a second point or pixel. Agents can be real or virtual. A real agent, such as a robot, occupies an area in a space. A virtual agent is a projection onto a space and does not actually exist within the space. For example, a virtual agent may be a projection of the center of view of a camera observing a two-dimensional scene. The camera can move and / or rotate to change the center of view on the scene, changing the position of the virtual agent. This is explained in more detail below.

[0118] Physical space refers to the three-dimensional space that a real-world object, vehicle, robot, or other agent can physically reach or traverse. Three-dimensional space may consist of one or more other physical or material objects. Three-dimensional space may also include the distance (or time) interval between two points, objects, or events that an agent may interact with as it moves through physical space.

[0119] A logical space is a space with an arbitrary set of points, such as a mathematical space or a topological space. A logical space can therefore be, for example, an image, a map, or a concept map. A set of points may satisfy a set of axioms or constraints. A logical space may correspond to a physical space, as described above.

[0120] Data is captured regarding the position of an agent in space, and the captured data describes the spatial environment in the local vicinity of the agent. Data capture is performed by sensors in the physical space, such as cameras, radar, tactile sensors, LIDAR sensors, infrared sensors, etc. Data capture is performed by virtual sensors in the logical space that obtain data of the logical space in the vicinity of the agent.

[0121] In physical space, there is the issue of observability, where an agent may not be able to observe all of the physical space at any given time: the physical space may be too large for sensors to observe, or the physical space may contain objects or terrain that obstruct regions of the physical space.

[0122] In a logical space, all of the logical space may be observable at once. For example, if the logical space is an image, the entire image may be stored in memory accessible to the agent.

[0123] However, in logical space, it is more efficient for an agent not to observe the entire logical space at once. For this reason, virtual sensors only capture a portion of the logical space, simulating sensors in physical space in that only the agent's vicinity in the logical space is captured and / or acquired by the virtual sensors.

[0124] Therefore, in both scenarios with agents in logical space and agents in physical space, it is computationally advantageous for agents to observe only a portion of their neighborhood in each space, rather than requiring the agents to observe the entirety of their respective spaces.

[0125] If the agent is a non-physical agent in a logical space, the agent's neighborhood does not need to be directly observed by a sensor or virtual sensor. For example, the agent's neighborhood could be a portion or region of a pre-recorded image that defines the overall logical space.

[0126] We define observation data as data captured by sensors or virtual sensors that observe the space near an agent. This observation data is used to make an "observation." Thus, an observation is the result of processing the observation data in a way that succinctly describes the environment near the agent.

[0127] In physical space, observations may be represented by matrices or vectors that provide spatial information derived from observational data, such as images or point cloud data of the agent's surroundings.

[0128] In logical space, an observation may be represented by a matrix or vector that provides spatial information in the agent's local neighborhood. The observation data from which this observation is derived may be a set of pixels or points from the logical space surrounding the agent. For example, the observation data could be a matrix of pixels that are within a threshold observable distance from the agent.

[0129] An agent may make multiple different observations at different locations in space. The agent makes observations by moving between these locations. The vector between two observations is defined as a "displacement" that has both magnitude and direction. Thus, an agent makes an observation by moving between two specific locations and moving along the displacement between those two specific locations.

[0130] Each observation and displacement made by an agent in a space may be stored in a memory associated with the agent. When an agent makes a series or sequence of observations and displacements in a space, the series or sequence of observations and displacements may be stored together as a set of observations and displacements. A series of observations and displacements may form an "identity." An identity may describe a particular feature or attribute of the space in which it is found. The terms attribute and feature are used interchangeably hereinafter.

[0131] For example, in a physical space, an identity may describe attributes such as a particular object or a path to a particular destination in the space. The observations in the set that form the identity indicate how the agent perceives the space and therefore how a particular feature is perceived. For example, if there is a particular object, such as a chair, in the space, the identity of the chair may include multiple observations taken from various locations on parts or sides of the chair. Similarly, if there is a particular destination in the space, the identity may include multiple observations taken from locations on the path to the destination. The displacements in the set that form the identity indicate how the agent moves between observation locations. For example, a first observation of the chair in the space may be made from a position near the back legs of the chair. A second observation of the chair in the space may be made from a position near the front legs of the chair. The first displacement of the identity is the displacement required to move from a position near the back legs of the chair to a position near the front legs of the chair.

[0132] An exemplary identity is shown in Figure 1. Figure 1 shows a series of observations 101 connected by a series of displacements 102 in an exemplary indoor space containing a door and a central obstacle. This exemplary room space can be a physical space such as a real-world apartment. The observations 101 may be captured using a camera on a robot. The displacements 102 may be made by moving the robot. In this example, the identity represents a path to a door, which is considered a destination. Each observation 101 provides data describing the environment from where the observation was made.

[0133] Although the displacements are shown in Figure 1 as being unidirectional, a corresponding set of negative displacements is also recorded, with the understanding that the identity consists of an observation 101, a set of displacements 102, and the negative displacement that links the observations.

[0134] The spatial attribute of the identity shown in Figure 1 may be a destination, such as a door. Thus, this identity can be used to navigate through a room, avoid an obstacle in the middle, and navigate to the door.

[0135] A second example of identity is shown in Figure 2. Figure 2 shows a series of observations 201 connected by a series of displacements 202 on a chair. The chair may be three-dimensional in a physical space, such as the real world, or two-dimensional in a logical space, such as an image. As in Figure 1, displacements 202 may also include opposite, negative displacements.

[0136] In the physical space, for example, the camera is responsible for capturing observations 201 and performing displacements 202 to obtain identities as shown in Figure 2. In the logical space, this may be performed by a virtual agent.

[0137] In the case of a camera in a physical space, the camera may or may not be in a fixed position. If the camera is not in a fixed position relative to the space, it may move within the space. For example, the camera may be attached to a robot. In this case, the agent may be the robot (or camera), and observation data is recorded at the agent's location by simply recording images of the robot's or camera's surroundings. If in a fixed position, the camera may rotate or zoom to focus on different parts of a chair. In this case, the camera's focus and gaze point (i.e., direction of gaze) from the perspective of the camera's field of view may represent the agent. In this case, the agent may be defined, for example, as the center point or projection of another specific pixel in the camera's field of view. Although the camera does not move translationally, the agent "moves" when the camera zooms or rotates. This is because such actions change the center point or the location of a specific pixel in the camera's field of view relative to the scene / image being observed. Therefore, the agent's position does not need to match the camera's position. If fixed, the camera may not displace between observations to obtain displacement 202. Instead, displacement may be created and stored as the distance between two fixed points in the camera's field of view across two observations. Displacement can be performed by rotating the camera to "move" the agent and move the center point of the camera's field of view.

[0138] For example, the camera may first capture an image centered on the legs of a chair. This image is then processed in a first observation. The agent's position is considered to be the center point of the camera's field of view for the first image. The camera is then rotated and panned upwards to capture and process a second image of the chair back. The second observation is processed from the image of the chair back. The agent's position in this second observation is considered to be the center point of the camera's field of view for the second image, and the displacement between the first and second observations is the distance in pixels between the center point of the field of view for the first image and the center point of the field of view for the second image and / or a distance related to the size of the rotation. This can be estimated using, for example, visual inertial odometry.

[0139] If the camera is not fixed, the agent is the camera itself (or the device on which the camera is attached). In this example, the camera / device moves to a specific location, makes observations at that location, and performs movements. Displacements are recorded, for example, using odometry.

[0140] In the logical space, Figure 2 can represent an image of a chair. In this example, the entire image is stored, so there is no observability issue and the entire image can be observed at once. However, to improve computational efficiency, the logical space is treated similarly to the physical space, and instead of processing the entire image at once, each portion of the image is observed in detail. Specifically, a portion of the image plane, for example, the portion containing the legs of a chair, is observed. The size of the portion may be fixed or may vary depending on the size of the image. Similar to the example of a fixed camera in physical space, the location of the agent is considered to be a specific point in the portion of the image, for example, a specific pixel, such as the center pixel of the image portion.

[0141] The "agent" or virtual agent in this case can "move" a number of pixels on the image plane to perform a displacement before making a second observation. In practice, this simply involves retrieving a second portion of the image plane from memory, which is then centered around the agent's new position (the pixel to which the virtual agent has "moved"). Thus, in logical space, the agent is a tool for selecting different parts of the image based on a fixed point in the image. The movement of the virtual agent involves a translation from the pixel where the virtual agent was previously located to another pixel.

[0142] Therefore, when referring to an "agent," it is important to understand that an agent can be three things. First, an agent can be a mobile physical device, whereby the agent physically moves to a location in physical space and makes observations at the device's location. In this case, the agent performs the movement between observations by moving the physical device, and the observation data corresponds to the data captured by the agent's current location. Second, an agent can be a projection of a fixed physical device. In this case, observations are made at various orientations of the fixed physical device, and the agent is a projection of a point in the physical device's field of view relative to the physical device's orientation. For example, the agent is placed at the center point of the fixed physical device's field of view. Displacement in this case is measured based on the distance between projections of points in the physical device's field of view caused by changes in the physical device's orientation. Third, an agent can be a virtual agent in logical space. In this case, the agent is placed at a point or pixel in logical space. The point or pixel specifies the portion of the surrounding space that corresponds to the observation data. For example, the agent may be placed at the center pixel of the portion of space that corresponds to the observation data of the observations. Displacement is measured as the distance between pixels or points in logical space.

[0143] Therefore, when referring to an agent, and in particular to the "location of an agent" in the preceding description, it should be noted that any of the above definitions of "agent" may apply. The location of an agent need not be the location of a physical device. Similarly, when referring to an agent that performs an action, such as moving or making an observation, the above definitions apply. That is, the physical device need not move, and the observation may be made by a physical device or computing device that is not necessarily the agent itself.

[0144] Returning to FIG. 2, the attribute of space defined by the identity formed by a series of displacements 202 and observations 201 may be the identity of a feature (in this case, a chair).

[0145] Thus, Figures 1 and 2 show that identities can be formed in both two-dimensional or three-dimensional physical and logical space for the purposes of identifying an object such as a three-dimensional chair, identifying an image or feature of an image such as a two-dimensional chair, or navigating three-dimensional space to reach a destination. It should be understood that there are other uses for identities, and object / image recognition and navigation are merely examples.

[0146] Multiple identities, such as those in Figures 1 and 2, may be stored in a set of saved observations and displacements from previous experience of one or more spaces. Identities need not be labeled; that is, in the example above, the identity corresponding to a chair need not be labeled a chair or even known to correspond to a chair. Rather, once a set of observations and displacements is saved as an identity, the agent can use this information to locate and identify matches to the identity in other regions of the space or across other spaces. The system or method does not need to understand the real-world features to which the identities relate.

[0147] In some examples described below, identities may be labeled according to features or attributes of the space they represent. In the above example, an identity corresponding to a chair may be labeled as a chair. The process of labeling identities in a set of stored observations and displacements may be performed as part of a general training process, which may be performed according to existing techniques understood by skilled artisans. For example, various machine learning techniques may be used, such as supervised or unsupervised learning processes, the use of neural networks, or classifiers.

[0148] For example, the set of stored observations and displacements may be obtained from a validation space, such as a validation image that includes or exhibits one or more known features. For example, a validation image of a chair is presented to the agent, and an identity formed from that image and stored in the set of stored observations and displacements is found to correspond to a chair and is labeled as such. The process of obtaining the set of stored observations is described in more detail below.

[0149] Using a set of observations and displacements to effectively describe spatial features, rather than an entire map or image of the space, is computationally efficient because the observations and displacements contain only the respective portions of the feature, requiring less memory and computation to identify it in a portion of space. Furthermore, the observations and displacements can be computed using a variety of data, such as vectors encoded from the output of a series of spatial filters applied to the visual data of the observations, or, in the case of displacements, the local temporal correlations of low-level features. This eliminates common failure modes and provides versatility by allowing the two parts of the system (observations and displacements) to compensate for the weaknesses of the other. In the application domain of 3D spatial SLMA, the system inherently performs not only localization and mapping but also navigation and path planning, eliminating the need to add additional components to achieve a complete real-world implementation.

[0150] Compared to standard SLAM, the methods and systems described herein are more robust to environmental factors and occlusions. The methods and systems described herein also have the advantage of storing a minimal set of measurements necessary to navigate and observe a particular space. Observations are made within a space depending on the complexity and / or changes within that space, allowing the methods and systems to adapt to work optimally in any space. For example, in a vast field or area with little information or features, fewer observations are made and stored compared to a feature-rich environment. Because the methods do not involve tracking features or objects that occupy a small portion of the field of view but instead fully utilize the characteristics of a large portion of the full solid angle sphere, occlusions or changes to portions of the visual scene have less impact on the system's ability to complete its task compared to traditional systems and approaches.

[0151] The features or attributes to which the stored observations and displacement sets relate may be identified in other regions of the same space, or in entirely new spaces. For example, in a physical space such as a living room or dining room, a stored observation and displacement set associated with a chair can be used to identify identical or similar chairs in other parts of the living room or dining room. Furthermore, a stored observation and displacement set can be used to identify identical or similar chairs in an entirely different space, such as another building.

[0152] Similarly, a destination in space, such as a store or landmark, may be navigated from an initial known location in the world or a previously visited location based on a stored set of observations and displacements connecting the starting point and the destination. Additionally, a destination in space may be navigated from an unknown region of the world or from an unknown starting point from a set of observations and displacements.

[0153] Identifying attributes of a space in this way relies on comparing a set of stored observations and displacements with current observations and displacements made in a particular space. The next part of the description will explain this process in more detail. In the next part of the description, stored observations and displacements will be referred to as predicted observations and displacements, and target observations and displacements. It is important to understand that these terms define the same technical features and are simply different to provide context as to why and when they are used. Similarly, current observations from a space will be referred to as first observations, second observations, and transition observations. Again, these terms define the same technical features—observations made by an agent within the space in which the agent resides—but are labeled differently to provide context as to why and when they are made or used. The same terminology applies to displacements.

[0154] How this process of identifying / determining attributes of a space using existing knowledge in the form of a set of stored observations and displacements is carried out is described in detail below with reference to Figure 1. The advantage of the present method is that only a portion of the space is needed in terms of the current observations and displacements to identify the attributes.

[0155] FIG. 3 illustrates a flow diagram of a method 300 for determining attributes of a space that an agent can explore or navigate, according to various examples.

[0156] In the initial conditions of method 300, the agent is in a space and can observe parts of the space through sensors, virtual sensors, etc. This space may have been previously explored by the agent or may be completely unknown to the agent. The starting location may be known to the agent or may be completely unknown to the agent.

[0157] In a first step 301, the agent makes an observation in space from the agent's current location in space. This observation may be considered a current observation or a first observation from a current location or initial location, although it should be understood that this observation need not be the very first observation in the space. Rather, "first" is used here simply to distinguish between a current observation and a later observation.

[0158] The first observation includes data describing a first portion of space in the agent's local neighborhood. For example, if the agent is a robot or vehicle, the first observation is made using sensor data from sensors such as cameras, radar, and LIDAR, and the sensor data describes a view of the world from the robot's current location. The agent's local neighborhood is the local neighborhood of the robot's current location. In physical space, the extent of the local neighborhood is determined by the sensor constraints and the robot's environment. For example, if a sensor has a limited range, the observed sensor data is obtained only for the spatial environment within that range. Similarly, the robot's environment may include one or more features that limit the observable surroundings, such as walls or other obstacles that block the sensor's view. Therefore, the agent's neighborhood is not fixed, and there may be a maximum distance from the agent's location based on the sensor constraints. In logical space, the agent's local neighborhood is the portion of the surroundings around the agent's location, which is smaller than the entire space. Because the entire space can be stored, for example, if the space is an image, data does not need to be obtained from the sensor to make the observation. Rather, the portion of the space around the agent's location is obtained from the entire stored space. This "virtual sensor" simulates the same effect as using a sensor in physical space, but only considers a portion of the space in its observations, rather than the entire space. Thus, an agent's neighborhood in logical space may be set based on distance from the agent's location. In the example where the space is an image and the agent is located at a pixel in the image, the neighborhood may be set, for example, by the number of pixels away from the agent's pixel.

[0159] In a first step 301, a first observation is made and in a second step 302, it is compared to a stored observation from a set of stored observations and displacements.

[0160] The comparison of the first observation with the stored observations generates an observation comparison measure that indicates how similar each of the stored observations is to the first observation. The observation comparison measure for each stored observation is ranked, the highest observation comparison measure is identified, and the corresponding stored observation is obtained. In this manner, the stored observation that is most similar to the first observation is determined and selected. The process of comparing observations is described in more detail below.

[0161] In a third step 303, a hypothesis is determined regarding the stored observations that are most similar to the first observation. In particular, the set of stored observations and displacements includes one or more identities that form a subset of the set of stored observations and displacements. Each subset includes one or more observations and one or more displacements, each associated with a particular attribute or feature to which the identity is associated. In the third step 303, the attribute or feature with which the selected stored observation is associated is determined. This attribute forms the basis of a hypothesis, which in effect becomes a prediction of what the agent is observing in space. The prediction can be, for example, a two-dimensional or three-dimensional object or image, or a navigable destination.

[0162] The set of stored observations and displacements may include multiple subsets of observations and displacements, each subset associated with a particular attribute and therefore a particular hypothesis. In one example, there is a subset associated with the attribute "chair," a subset associated with the attribute "table," and a subset associated with the attribute "door." The comparison in the second step 302 is performed for all observations in the set of stored observations and displacements, and the observation comparison measure may be highest or strongest for a particular observation, and that particular observation is part of the subset associated with the attribute "door." Thus, in the third step 303, a hypothesis is made that the first observation is part of a door, i.e., the attribute of the space observed by the agent is a door in the space. The subset of stored observations and displacements associated with an attribute is called a hypothesis subset.

[0163] In a fourth step 304, a hypothesis subset of observations and displacements is obtained. In particular, the stored observation most similar to the first observation is part of the hypothesis subset of stored observations and displacements, as described above. Furthermore, the observations and displacements within the hypothesis subset are linked, forming a network of observations separated by displacements. The links between the observations and displacements within the hypothesis subset are stored in a set of stored observations and displacements, and the network arrangement is stored in memory. When obtaining the hypothesis subset of observations and displacements, the method includes obtaining the observation and displacement associated with the stored observation most similar to the first observation. In other words, the neighboring observations and displacements required to reach the first observation from the stored observation most similar to the first observation are obtained from the hypothesis subset. This observation and displacement become the predicted observation and predicted displacement, and if the agent moves the predicted displacement from its current position, according to the hypothesis, the agent is expected to arrive at a position in space where it can observe the predicted observation.

[0164] It should be understood that the terms "predicted observations" and "predicted displacements" simply refer to the subset of observations and displacements associated with a determined hypothesis, each of which is predicted to lie within the space currently occupied by the agent.

[0165] In a fifth step, 305, the hypothesis is verified. To verify the hypothesis, the agent is moved sequentially along the network of stored observations and displacements that form a subset of the hypothesis, i.e., the predicted observations and predicted displacements. At each iteration of this sequential process of moving the agent along the network, the agent is configured to repeatedly make new observations, compare the new observations with predicted observations and / or compare predicted displacements with actual displacements, and decide whether to maintain the hypothesis, reject the hypothesis, or confirm the hypothesis. This hypothesis verification is described in more detail below with reference to steps 305-1 through 305-5.

[0166] In the first step 305-1 of the hypothesis testing process, the agent moves from its current location where it made a first observation to a second location in space based on a movement function. The movement function depends on the predicted displacements of the hypothesis subset. In particular, the movement function depends on the predicted displacements connected to the stored observation that is most similar to the first observation, identified from the linked network of observations and displacements that form the subset. Thus, the agent does not know the second location until it performs a movement based on the movement function.

[0167] In a second step 305-2 of the hypothesis testing process, the agent makes a second observation from a second location of the agent in space, the second observation including data describing a second portion of the space in a local vicinity of the second location. For example, if the agent is a robot or vehicle, the second observation is made in the same manner as the first observation using sensor data from a sensor such as a camera, radar, or LIDAR, and the sensor data is processed to result in an observation showing a view of the world from the robot's position at the second location.

[0168] In the third step 305-3 of the hypothesis testing process, the actual displacement of the agent from the first location to the second location is determined. In physical space, the actual displacement may be estimated using, for example, odometry. In logical space, the actual displacement may be measured according to known techniques using vector or matrix calculations. For example, if the logical space is an image, the distance may be determined as the number of pixels between the first and second locations. In physical space, the displacement is measured using an odometry system that combines visual inertial odometry and, if applicable, kinematic odometry, such as wheel odometry. This system provides an interface for input of the visual inertial odometry and kinematic odometry. The displacement indicates the physical or logical distance between pairs of observations. The odometry is stored in a coordinate system referenced to the rotation of the starting observation (the predicted observation that is most similar to the first observation). Additional data characterizing other aspects of the displacement, such as the vibrations occurring during the displacement transition and the energy expended during the displacement transition, may be stored. The similarity of the displacements is measured by the endpoint error between pairs of displacements. In this case, the displacements start at the same location and the endpoint error is the physical or logical Euclidean distance between the points identified by transitioning along the two displacements.

[0169] In the fourth step 305-4 of the hypothesis testing process, a comparison is performed. The comparison may compare observations, displacements, or both. A second observation may be compared to a predicted observation from the hypothesis subset, particularly to a predicted observation linked to the predicted displacement on which the movement function was based. Comparing the second observation to the predicted observation provides an observation comparison measure. Similarly, a displacement comparison measure may be obtained by comparing the predicted displacement on which the movement function was based when the agent moved from a first location to a second location in space with the actual displacement between the first and second locations. Thus, the comparison may generate both an observation comparison measure and a displacement comparison measure. As will be explained in more detail below, using both of these measures may be advantageous. It should be understood that the order of the above steps after the agent has moved is interchangeable.

[0170] In the fifth step 305-5 of the hypothesis testing process, the hypotheses are adjusted, maintained, or confirmed based on the observed and / or quantile comparison measures from the fourth step of the hypothesis testing process 305-4.

[0171] In this step, the observation comparison measure and / or the displacement comparison measure are evaluated to effectively determine whether the agent's perception of the space occupied in terms of the second observation and actual displacement matches the hypothesized attribute perception in terms of the predicted observation and predicted displacement. The observation comparison measure and the displacement comparison measure are described in more detail below, but generally, the stronger these measures are, the more likely the hypothesis is correct and that the agent is actually observing the hypothesized attribute in the space in which they are currently located.

[0172] The observed comparison measure and / or the variance comparison measure may be compared to one or more respective thresholds to determine whether to confirm, maintain, or reject the hypothesis.

[0173] Adjusting a hypothesis involves rejecting the hypothesis. This occurs when the space being observed by the agent is determined to be unlikely to contain or represent the attributes according to the hypothesis. This conclusion may be reached in several different ways. First, the hypothesis may be rejected if the observation comparison measure and / or the displacement comparison measure fall below a first observation threshold level and / or a first displacement threshold level. The first observation threshold level and the first displacement threshold level may be globally set as the minimum probability required to maintain the hypothesis and allow the agent to further test the hypothesis. Alternatively, these thresholds may be adaptive and modifiable based on environmental conditions of the space, such as brightness / darkness factors. In particular, the lighting of the space may affect the observation comparison measure if the agent's observations are made at a different environmental brightness level than the brightness level at which the subset of stored observations relevant to the hypothesis were captured. The thresholds may also be adjusted based on color, contrast, and other image / sensor data characteristics.

[0174] Furthermore, different observation comparison measures may be used to address additional spatial aspects, and their combined use may identify specific dimensions of difference for a hypothesis. For example, changes in lighting have a greater impact on vector comparison measures than on spatial distribution measures of filtered elements around the agent's location. Changes in object color affect color-filtered elements of the observation more than brightness or orientation-filtered elements. By considering various observation comparison measures, it becomes possible to observe the nature of the differences. When lighting conditions change, a measure of the spatial distribution of filtered elements of the observation around the agent's location can be used to adapt the threshold, and strong matching against that measure can be used to pre-update the hypothesis, lowering the acceptance criterion for the "similarity" observation comparison measure (the second observation threshold level). The various types of observation comparison measures described here each arise from different ways of comparing observations, as will be explained in more detail later.

[0175] The first observation threshold level may be set according to the initial ranking of the observation comparison measure of each stored observation relative to a first observation made by the agent. In particular, the first observation threshold level may be set equal to the next highest-ranked observation comparison measure corresponding to a stored observation in the set of stored observations and displacements that is not included in the subset associated with the current hypothesis. Thus, this particular stored observation is associated with a different hypothesis. In this way, the first observation threshold level is set according to the similarity between the first observation and stored observations associated with a hypothesis different from the currently tested hypothesis. This allows the agent to effectively consider a hypothesis different from the one being tested if the tested hypothesis generates a weaker observation comparison measure than the one initially found for the different hypothesis. Therefore, the hypothesis being tested is rejected and ultimately replaced with another hypothesis associated with the first observation threshold level.

[0176] Finally, adjusting the hypothesis involves rejecting the current hypothesis and replacing it with another hypothesis. This may involve the method returning to the second step 302, as shown in FIG. 3, and selecting a next-best observation comparison measure from the ranked observation comparison measures to determine the next-best observation and its associated hypothesis. The agent may also return to the location of the first observation in space via a movement opposite to the movement actually made before selecting a new hypothesis. Hypothesis replacement is described in more detail below.

[0177] Hypothesis retention occurs when a hypothesis is neither adjusted nor confirmed. Therefore, if the observation comparison measure and / or the displacement comparison measure match or exceed the first observation threshold level and / or the first displacement threshold level, respectively, the hypothesis is retained. This means that there is no need to adjust the hypothesis. According to the example introduced above, the first observation threshold level is set according to the "next highest observation comparison measure," which means that the current hypothesis still provides the highest observation comparison measure for the predicted observation from any subset of the set of saved observations and displacements. In other words, the attribute according to the current hypothesis is still the attribute most likely to be found in the space where the agent resides. Another condition for a hypothesis retention is the failure to meet a hypothesis confirmation condition. The hypothesis confirmation condition can be a second higher observation threshold level and / or a second higher displacement threshold level. The hypothesis confirmation condition indicates a minimum confidence that the hypothesis is correct and that the attributes of the space are those according to the hypothesis.

[0178] In other words, if the observation and / or displacement comparison measures for the predicted observation and predicted displacement are determined to meet or exceed the first observation threshold level and / or the first displacement threshold level, respectively, but not meet or exceed the second observation threshold level and / or the first displacement threshold level, respectively, the hypothesis is maintained.

[0179] Once a hypothesis is maintained, a next predicted observation and next predicted displacement, linking the current predicted observation and the next predicted observation, are obtained from the subset of stored observations and displacements associated with the hypothesis. The above hypothesis testing process is then repeated for the next predicted observation and next predicted displacement, as shown in Figure 3. Thus, method 300 involves sequential testing of hypothesis subsets of predicted observations and displacements until the hypothesis is confirmed or adjusted.

[0180] With each observation the agent makes while maintaining a hypothesis, past observation and / or variance comparison measures are updated, and then the past observation and / or variance measures are incorporated into future observation and / or variance comparisons. The past observation and / or variance measures are stored in memory and function to increase future observation and / or variance comparison measures based on the consecutive number of times the hypothesis is maintained. This effectively increases confidence that the hypothesis can be confirmed based on the hypothesis not being rejected for multiple consecutive observations and variances. It also helps prevent the agent from getting stuck in a local minimum, where the hypothesis is never rejected or confirmed. This process of utilizing past observation and / or variance data may involve sequential probability ratio testing of a single hypothesis against a null hypothesis (data inconsistent with identity) or as sequential probability ratio testing of multiple hypotheses between alternative hypotheses, including the null hypothesis (no hypothesis is consistent).

[0181] Confirming the hypothesis includes determining that a hypothesis confirmation condition is met and determining attributes of the space based on the hypothesis confirmation condition being met. As discussed above, the hypothesis confirmation condition can be a second, higher observation threshold level and / or a second, higher displacement threshold level that the observation comparison measure and / or the displacement comparison measure must meet or exceed.

[0182] Like the first observation threshold level, the second observation threshold level may be changed or adapted based on lighting conditions within the space using a lighting factor.

[0183] If the hypothesis is confirmed, the method 300 may stop as the agent identifies the attribute of the space. The identity of the attribute may be stored in memory, and the observations and displacements performed by the agent to identify the attribute may be stored in memory and linked to the space in which the agent resides.

[0184] The method 300 has the advantage that it uses observations and displacements to determine attributes of a space without having to consider the completeness of the entire space, resulting in a much more computationally efficient method of spatial analysis.

[0185] Method 300 will now be described in more detail. First, referring to FIG. 4, method 400 for making and comparing observations will be described. Making and comparing observations occurs in at least two ways in method 300. First, in a first step 301 and a second step 302, the agent makes an observation and compares it with a set of saved observations to ultimately determine a hypothesis, and then in a fifth step 105, the hypothesis is verified. Thus, this process requires at least two observations and two comparisons. However, it should be understood that more observations can be made, and that in the two specific examples above, as well as in transition observations between a first location and a second location, it may be advantageous to make the observations sequentially while the agent is moving from the first observation location to the second observation location. These transition observations will be described in more detail below. Here, the process of making and comparing observations will be described.

[0186] In a first step 401, observed sensor data is captured. In physical space, observed data is input data captured using a sensor such as a camera. In this example, the observed data is a captured image. The captured image and its content will vary depending on the orientation of the camera. In particular, the camera may be in the same position, for example attached to a robot at the same location in the world. However, if the orientation of the robot is modified, the content of the observed data may change even though the position of the robot has not changed.

[0187] For this reason, it is advantageous to perform observations in physical space using a camera system or other sensor capable of capturing a three-axis stabilized cylindrical projection of three-dimensional space, oriented so that the cylinder's major axis is perpendicular to the direction of gravity. This fixation with respect to gravity allows observations to be performed with the same roll and pitch (as these are also fixed according to the direction of gravity). The image capture resolution may be, for example, 256 columns by 64 rows of pixels, but can also be any resolution. This 256x64 pixel image represents the observation input data and is then processed in the second step 402. In non-physical or logical spaces, the observation data may be data obtained from the logical space surrounding the agent's location. The entire logical space may also be pre-stored. The observation data may be provided from a portion of the logical space within a predetermined distance from the agent's location. For example, if the logical space is an image or map, the observation data may be pixels or points within a distance of 5, 10, 50, 100, or any number of pixels / points from the agent's location.

[0188] In the second step 402, observations are generated from input data captured by sensors, virtual sensors, or retrieved from memory. The input data is encoded into a set of vector / matrix representations through a staged pipeline.

[0189] The process of generating a stored observation to form part of a set of stored observations and displacements is very similar to the process of generating a current observation from input data to compare with a stored observation. However, when comparing current observations, the relative orientation of the agent at its current location may not be the same as the orientation of the agent when it made the stored observation, and may not be easily distinguishable. To solve this problem, when generating a current observation from input data, the process involves generating an observation for each possible permutation of the agent's orientation. In practice, this means that the current observation is associated with multiple rotations, whereby each rotation represents input data that has been shifted or adjusted to represent a different orientation of the agent. How this affects the process compared to generating a "stored" observation is explained below.

[0190] The general process for generating an observation (current or saved) involves the following steps:

[0191] First, low-level features are extracted from the input data. This may involve, for example, using one or more filters and masks to preprocess the input data. Low-level features may include edges, color, shape, etc. Preprocessing may also include color transformations to provide multiple channels of the input data. The result of this first step is processed data that may potentially contain multiple channels.

[0192] Second, the resolution of the input data is reduced by using filters or the like to extract or remove portions of the input data, such as rows of pixels, and / or by summing and averaging data over large regions of the input data.

[0193] Third, further processing may be performed by convolving the input data with one or more kernels to obtain useful results, for example to identify regions of interest, edges, areas of greatest change, etc. If there are multiple channels, this process is performed for all channels.

[0194] Fourth, the resulting data forms a vector or matrix of processed and reduced input data, which is then concatenated across dimensions (columns, rows, channels) and normalized to produce the observation vector o.

[0195] As mentioned above, when storing observations, only one observation vector o is needed, e.g., for the purpose of storing a set of observations such as an identity or a map. When observing currents for comparison purposes, this general process is performed for multiple rotated permutations of the input data. If the data resolution is Xx Y, this may mean including X permutations, each representing an X+1 shift from the previous permutation. This results in X observation vectors for the current observation, each of which can then be compared to one stored observation vector.

[0196] This process of encoding input data into one or more observation vectors is performed by the central processing unit (CPU), the graphical processing unit, and the The present invention uses a computing device such as a graphics processing unit (GPU), a field-programmable gate array (FPGA), or an application-specific integrated circuit (ASIC).

[0197] A detailed example of this process is shown below for the above example of an image captured from a camera (a cylindrical projection image). In this example, the image resolution is 256x64 pixels, so there are 256 possible rotations of the projection in the current observation, and the columns are rotated (shifted) from their original capture positions. As mentioned above, these rotations represent various permutations of the original observation data. It should be understood that there can be any number of permutations based on the input resolution of the image. The detailed process is as follows:

[0198] In the first stage, a color transformation process is performed on each pixel, transforming the color channel data from the captured image. For example, red, green, and blue channel data is transformed into red, green, and luminance channels, where luminance is 1 / 3 (red + green + blue). It should be understood that these colors and transformations are exemplary, and that images may have different color channels and be transformed into different color channels.

[0199] In the second stage, the luma channel result of the first stage is convolved with a filter using the convolution kernel [1,-1;1,-1;1,-1] or a similar vertical edge filter (e.g., a Sobel filter or other filter) to create the luma vertical (luma_v) channel.

[0200] In the third stage, the luma channel result from the first stage is also convolved with a filter using the convolution kernel [1,1,1;-1,-1,-1] or a similar horizontal edge filter (e.g., a Sobel filter or other filter) to create the luma horizontal (luma_h) channel.

[0201] The result of this processing is a new image containing pixels for four channels: red, green, luma_h, and luma_v. It should be understood that other pixel filter choices can be used here. With the selected filter set, all sub-pixels of all channels (red, green, luma_h, or luma_v) are routed to the next processing pipeline.

[0202] In the fourth stage, a series of box filters are calculated and placed around a set number of equally spaced rows of the image for each channel. It should be understood that there may be more or fewer filters, and the 19x19 size is exemplary. The box filters smooth the image for each channel. Each pixel in each channel is modified by the box filter from the sum of the pixels contained within each box filter, thereby providing a smoothed (or blurred) image. Box filters may overlap each other.

[0203] In the fifth stage, the blurred image is reduced to form sliced ​​images. In particular, the resolution of the blurred image is significantly reduced. This process is illustrated in Figure 9. Specifically, four evenly spaced rows 902 are extracted from a blurred image (not shown) processed from the original image 901 (or other observational dataset) to generate sliced ​​images 904.

[0204] As shown in Figure 9, the sliced ​​image contains 256 columns and 4 rows.

[0205] In the sixth step, the columns of slice images formed in the fifth step are divided into sets of evenly spaced columns. As an example, the 256 columns are divided into 256 sets, each containing 16 columns by 4 rows of images, with the first column of each set being different from the full set of 256 columns. As can be seen in Figure 9, there are 16 sets of 16x4 images (a total of 256 permutations), and the columns are displayed in different colors to illustrate how the data from the 256-column slice image 904 is divided into 16x16x16x4 sets. These sets represent all possible rotation positions of the input data (one column rotation increment for all 256 possible rotations). Figure 9 is an illustrative example; the number of rows, columns, and permutations may be greater or less than this; i.e., the number of rotations may be greater or less than 256.

[0206] When an observation is saved to memory to form part of a set of saved observations and displacements, only one of the 16x4 images from the set formed in the fifth step needs to be saved. For example, the first permutation 906 of the first set may be selected to be saved as the observation. Rotations (i.e., the 256 permutations of the 16x4 images) need only be considered when creating an observation for comparison with a previously saved observation. Thus, permutations need only be considered for the current observation. For comparison, it is not necessary to save each permutation of the 16x4 images to memory. Rather, each permutation is iteratively compared to the saved 16x4 observation vector to identify the best match, as indicated by one or more observation comparison measures. Saving the permutations is not necessary. Comparison of observations is discussed in more detail below.

[0207] Returning to our detailed example of creating observations, in the seventh stage, a 16x4 image is periodically convolved with a center-on, perimeter-off (or vice versa) kernel. For example, the element with a value of 1 is the center of the kernel. The kernel in the first row might be, for example, [-1 / 5, 1, -1 / 5; -1 / 5, -1 / 5, -1 / 5], the kernels in the second and third rows might be [-1 / 8, -1 / 8, -1 / 8; -1 / 8, 1, -1 / 8; -1 / 8, -1 / 8, -1 / 8, -1 / 8], the kernel in the fourth row might be [-1 / 5, -1 / 5 -1 / 5; -1 / 5, 1, -1 / 5], and so on. An example of this is shown in Figure 10, which shows horizontal wrap filters 1001 through 1003 for a 16x4 image with center-on and perimeter-off. This kernel identifies the areas of greatest change in the image. As mentioned above, when saving observations, this is only necessary for one 16x4 image. When creating current observations to compare with saved observations, this is done for each separated set (i.e., the 16x16 set from stage 6).

[0208] Following this pipeline, a 16x4x4 image is generated. For comparison, the 16x4x4 image is rotated 256 times. There are 16 columns, 4 rows, and 4 channels. The channels of the 16x4 image may be divided into one data subset consisting of red and green, and two data subsets consisting of luma_h and luma_v. Each of these data subsets is normalized using vector normalization, where each element describes a value on a set of orthogonal values, resulting in a unit vector. These normalized vectors are each scaled by the square root of 2 and concatenated into a single unit vector. This provides the observation vector o. In this example, the observation vector is concatenated from 16 columns, 4 rows, and 4 channels, generating a 256-element vector that is stored. For comparison, 256 permutations of vector o are compared to the single stored vector.

[0209] In addition to the observation vector o, the final vector can store additional data related to the position of the observation (such as a user-defined label, the angular position and identity of the object, data from additional sensory modalities such as auditory or olfactory information, traversability of different directions in physical or logical space, and previously explored directions).

[0210] The observation vector o is a coarse representation of the original observation data captured by the sensors, providing a statistical summary of the agent's nearby environment in physical or logical space. The coarseness of the vector compared to the original data allows the method to run more efficiently. The observation data is also filtered at high resolution before the coarsening process, creating various channels. Coarsening also has the advantage of providing a representation of the location that is less sensitive to changes in lighting and objects in the environment than state-of-the-art techniques, since information is integrated across a large portion of the field of view.

[0211] It is understood that the observation vector o can be a vector or a matrix. The vector o may be composed of multiple sub-regions, each corresponding to a portion of the overall vector. The sub-regions may be content / color sub-regions divided based on the channels used to form the vector (in the example, red, green, luma_h, luma_v), or they may be spatial sub-regions corresponding to different directions around the agent. The sub-regions can be represented by sub-vectors, the combination of which forms the overall observation vector o. As mentioned above, a vector representing rotational permutations is necessary when comparing observations; if a current observation is made and needs to be compared to a stored observation, not all permutations need to be stored.

[0212] It should be understood that the numbers given above for dimensions, filter and kernel sizes, row selection, and rotation permutations may be more or less than the example values.

[0213] Returning to Figure 4, in a third step 403, the current observation vector o is compared to a stored observation from a set of stored observations, i.e., the current observation is compared to the target observation.

[0214] Observations can be compared using several functions. Comparing the observations results in an observation comparison measure. The observation comparison measure may vary depending on the function used to compare the observations. In some examples, multiple observation comparison measures may be generated by comparing the observations using multiple comparison functions. These comparison functions may be used alone or in combination in the hypothesis testing method 300, as well as in other related methods described below. Environmental features, such as the brightness level of a scene, may affect one observation comparison measure but not another. Therefore, using a combination of observation comparison functions and resulting observation comparison measures allows the method 300 to perform more accurately under various environmental conditions.

[0215] Functions for comparing observations include an observation similarity function 403-1, an observation rotation function 403-2, an observation action function 403-3, and an observation variance function 403-4. As shown in Figure 4, one or more of these functions can be performed on the current observation and a stored target observation. Therefore, the initial conditions and system required for each method of comparing observations are the same. This system is shown schematically in Figure 5.

[0216] FIG. 5 shows a schematic functional diagram of a system or apparatus 500 including various modules involved in comparing observations. These modules may be implemented by a computer, computing device, or computer system. It should be understood that the functionality of these modules may be further divided into other modules, separated among components, or performed by a single apparatus or component. The first module of the apparatus or system is a memory module 501. The memory module 501 stores a set of saved observations and displacements. These observations and displacements may be stored as vectors. The set of observations and displacements, including one or more hypothesis subsets, can be stored in a cyclic or acyclic graph and / or a directed graph. Alternatively, the observations and displacements may be stored with metadata identifying the displacements and observations to which they are connected, or may be organized, such as in a lookup table that maintains links between each observation and displacement.

[0217] The sensor module 504 is connected to or in communication with a current observation processing module 505. The stored observation processing module 502 is configured to select stored observations from the memory module 501 for comparison. The stored observation processing module 502 may also select one or more masks to reduce the stored observations to a subset of the stored observations for input to the comparison function. The masks may include observation masks, user / system selected masks, or a combination thereof. A mask may mask certain regions of a selected stored observation vector and retain other regions, called regions of interest (ROIs). A mask may be a binary mask, using values ​​of 0 and 1 to mask or retain, or vice versa.

[0218] The stored observation processing module 502 is connected to or in communication with a comparison engine 503 that performs comparisons according to one or more of the comparison functions 403-1, 403-2, 403-3, and 403-4.

[0219] A second side of the system or device includes a sensor module 504. The sensor module 504 includes sensors capable of inferring spatial information from the environment, and may include cameras, LIDAR sensors, tactile sensors, etc. The sensor module 504 also includes control circuitry and memory necessary to record sensor data. The sensor module 504 is configured to capture current observation data.

[0220] The sensor module 504 is connected to or in communication with a current observation processing module 505. The current observation processing module 505 processes the sensor data from the sensor module 504 to obtain a current observation, such as a current observation vector, which includes a total possible rotation for the sensor data according to the process described above with respect to Figures 9 and 10. The current observation processing module 505 may also select a mask to mask the current observation vector.

[0221] The current observation processing module 505 is connected to or in communication with a comparison engine 503 that performs comparisons according to one or more of the comparison methods 403-1, 403-2, 403-3, and 403-4.

[0222] The comparison engine 503 is configured to retrieve stored observations from the stored observations processing module 502, retrieve current observations from the current observations processing module 505, and retrieve or receive respective masks, in order to perform comparison observations according to any of the comparison methods 403-1, 403-2, 403-3, and 403-4. The comparison engine 503 may be implemented by a computing device, a server, a distributed computing system, etc.

[0223] The system 500 further includes a locomotion module (not shown) configured to move the agent in space. In a physical space, this may be a motor, engine, etc., configured to move the agent by physically moving the agent's location or changing the agent's orientation. In a logical space, the locomotion module may be implemented by a computer, etc.

[0224] Although the comparison methods are described separately below, it should be understood that they may be performed simultaneously or within the same overall process executed by comparison engine 503. These methods can be performed in the same overall process because they each rely on the same underlying data points and operations. In particular, each of the following methods compares a current observation with a stored target (predicted) observation. For example, if the observation is represented as a vector, each method does this by taking a dot product between the vector or its subvectors. The comparison methods differ by additional pre-processing / post-processing steps, as described below.

[0225] The first method 403-1 compares observations based on a similarity function to obtain a first observation comparison measure component. The first observation comparison measure component is the "similarity" observation comparison measure described above. This similarity function requires that the vectors of the observations being compared be vector-normalized. The similarity function involves calculating the vector dot product of the observation vectors being compared. The dot product is composed of the cosine of the angle between the two vectors. Thus, the similarity function returns a single value or score indicating the similarity between the two vectors corresponding to the observations being compared. If the current observation includes multiple possible rotations (256 possible rotations in the example above), the vector corresponding to each of the rotations is compared to the predicted observation. In this case, the dot product that provides the highest value is saved as the first observation comparison measure component. This single value is then used in a second step 302 of the method 300, for example, to generate a set of ranked observation comparison measures corresponding to the set of saved observations, from which the saved observation most similar to the first observation is determined and selected to determine the hypothesis. In other words, this similarity function may be used to determine the similarity between a first observation and a set of stored observations, identify which of the stored observations are most similar to the first observation, and decide which hypothesis to test.

[0226] This similarity function can be used in the fifth step 305 as well when comparing the second observation with the predicted observation.

[0227] When the current observation is associated with multiple rotational positions, each corresponding to an observation vector, the similarity function is configured to find the largest dot product between the different observation vectors and the target observation. To do this, the similarity function repeatedly calculates the dot product for each observation vector rotated with respect to the target observation, tracking the "maximum dot product value." Once all rotations have been evaluated, the maximum dot product value is taken as the first observation comparison measure component.

[0228] The advantage of using this similarity function is that it is relatively computationally efficient at run time and the resulting first observation comparison measure component decreases very smoothly as a function of distance from the original observation location (or the target observation location to which the current observation is compared).

[0229] However, this function may be affected by environmental conditions, such as the brightness or darkness of the environment. For example, if the stored observations relate to observations captured in low-light conditions and the current observation relates to observations captured in direct sunlight, the current observation may differ from the stored observations, resulting in different vectors, even if the features captured in each observation are spatially the same. To counteract these effects, the first observation comparison component may be combined with or used with a fourth observation comparison component formed from a fourth observation comparison function 403-4, as described below.

[0230] In the second observation comparison function, the observations are compared using an observation rotation function to obtain a second observation comparison measure component. This rotation function takes a rotation o of the current observation and compares it to each of the target observations, as in the similarity function. For example, during the fifth step 305 of the method 300 in which a hypothesis is tested, the observation rotation function may be used to compare the second observation (current observation) with the predicted observation (target observation). In the above example, there are 256 rotations, corresponding to 256 different column positions from the original cylindrical observation data. While the similarity function is configured to obtain a dot product indicating the best match between the rotated observation vector and the target observation, the observation rotation function is configured to identify the specific rotation responsible for this best match. The observation rotation function outputs the rotation direction of this best match as an offset value, which corresponds to the index of the observation's rotation position. For example, the best match between the observation vector o and the predicted observation occurs at 120 of the 256 rotations. 番目 This means that 120 of the 256 columns 番目The columns of correspond to vectors formed from observations rotated to the center of the data, or any other defined rotational position. Thus, the rotated observation function is 番目 The offset value may output a rotational position index of (which may be 120, for example).

[0231] In physical space, if an inertial measurement unit (IMU) is used as part of or included in the sensor, the offset value indicates how much the yaw (roll and pitch can be fixed by gravity) has drifted since the target observation was first made (assuming the stored observations that form the target observation have previously appeared in the current space). Thus, the offset value of the observation rotation function provides an output that indicates the IMU yaw drift. This IMU yaw drift can be identified from the output and used to update the set of stored observations and displacements, since they are all relative to the original yaw that was created and stored. Thus, if the first observation comparison scale component and / or second observation comparison scale component are reliable and indicate the current observation is the target observation, the offset value can be used to adjust the set of stored observations and displacements accordingly.

[0232] If the physical space in which the agent resides is not the same space in which the saved observations were captured, the offset value indicates how the space in which the agent resides is oriented compared to the space in which the saved observations were captured.

[0233] In summary, the second observation comparison measure component generated by the observation rotation function provides an offset result that indicates the strongest similarity between the target observation and multiple rotations (e.g., 256 rotations) of the current observation. Furthermore, the rotation index that indicates the strongest similarity indicates the alignment between the agent's current orientation (i.e., the agent's currently perceived x- and y-axes) and its orientation when the target observation was created and saved. Therefore, the offset allows us to recover directional information so that the target (predicted) observation and displacement can be remapped accordingly.

[0234] This rotation function may be used in the second step 302 of the method 300 when comparing the first observation to the set of stored observations, and may also be used in the fifth step 305 when comparing the second observation and predicted observations to recalculated predicted displacements and observations based on the agent's relative orientation (frame of reference).

[0235] The observation similarity function 403-1 and observation rotation function 403-2 described above can be executed in the same process or can be executed independently. An example of a single process that can execute these functions simultaneously is provided below in the form of example pseudocode for executing both the observation similarity function 403-1 and the observation rotation function 403-2. The pseudocode is: combined_function(cam_observation[16x4x4x256],cam_mask[16x4x4x256],memory_observation[16x4x4],memory_mask[16x4x4],user_mask[16x4x4]) Initialize max_similarity to -1 Initialize the offset to 0 Every 256 rotations: o Initialize cam_sum, mem_sum, and match_sum to 0 o Combine cam_mask, memory_mask, and user_mask with a logical binary OR (0 means use element, 1 means don't use element) o For each 16x4x4 (256 total) vector element: ■ If the mask is 0: Add the squares of the elements of cam_observation to cam_sum Add the squares of the elements of memory_observation to mem_sum Add the product of the elements of cam_observation and memory_observation to match_sum ■ Other: Do nothing Normalize match_sum by dividing it by the product of the square root of cam_sum and mem_sum o This is the curr_similarity score for this rotation o If the similarity score is greater than max_similiarity: ■ Set max_similarity to curr_similarity ■ Set the offset to the rotation index o Other: ■ Do nothing After considering all rotations, max_similarity is the output similarity, the first observation comparison measure component. The offset after all rotations have been taken into account is the output rotation, i.e., the second observation comparison scale component.

[0236] In the pseudocode above, "cam" refers to the sensor (e.g., camera) and therefore the current observation. Similarly, "mem" and "memory" refer to memory, i.e., the stored target observations to compare against.

[0237] The cam, memory, and user masks function to include and / or ignore specific portions of the current and target observation vectors in the comparison function. The cam and memory masks consist of information about the reliability of information from portions of the visual scene within the observation vector. In the case of a memory mask, this refers to a single observation vector, while in the case of a cam mask, it refers to a set of observation vectors for various rotations of the field of view. This can be used, for example, to filter out portions of the camera image where excessive contrast saturates the input data and reduces data quality. The user mask consists of information provided by a function internal to the system that attempts to select a subset of the complete observation vector elements, or by an external function that does the same. This can be used, for example, to determine the matching of an observation to a particular input function or various portions of the visual scene encoded in the observation.

[0238] The user mask is constructed to remove outliers / erroneous data from the observed data.

[0239] The above pseudocode iterates through each possible rotation of the current observation and identifies the maximum normalized similarity measure across all rotations (first observation comparison measure component) and the rotation offset (second observation comparison measure component) that corresponds to this maximum value.

[0240] In the first and second observation comparison functions 403-1 and 403-2 above, as well as the third comparison observation 403-3 described below, the observation vector o may be created from an image of physical space. This image has the property that the x-axis is the angle around the agent and the y-axis is some property (radial distance or vertical altitude) perpendicular to the x-axis. In the example above, the vector is made up of four channels with sub-pixels (red, green, luma_h, luma_v). There may be more or fewer channels, which may result in a larger or smaller observation vector. Furthermore, the image and each of its channels is made up of a series of rows and columns that provide spatial information about the portion of the space in which the agent resides.

[0241] Thus, the observation vector o can be relatively large, comprising spatial information in multiple directions and color / edge information from different channels. This relatively large vector can be manipulated or sliced ​​to perform various comparisons. For example, using only the edge channels luma_h and luma_v makes the comparison less sensitive to color changes. Using only the green channel can determine changes in vegetation distribution, for example. If all channels are used, but only for spatial subregions of the input image, only corresponding parts of the visual world are compared. This allows for partial comparisons, allowing us to see how well certain parts / features of the world match.

[0242] To perform slicing of the observation vector o, one or more masks are used. Each mask is a vector with the same number of elements as the observation vector o, configured to mask certain regions of the observation vector o while preserving others. For example, a binary mask of 0s and 1s may be used for this purpose. If the observation vector o has 256 elements, the mask will consist of 256 elements. Here, a value of 1 indicates that the corresponding index of the observation vector o should be included / ignored in the subset of the observation vector o being compared. On the other hand, a value of 0 in the mask indicates that the corresponding element of the vector o should not be included / ignored in the subset of the observation vector. Masks may be used to filter out regions of erroneous data, as described above.

[0243] A third method for comparing observations uses an observation action function 403-3 to obtain a third observation comparison measure component. The observation action function can be used to compare observations and can also be used as part of a movement function to move an agent to a second location while testing a hypothesis according to the fifth step 305 of method 300.

[0244] The observation action function relies on the same basic principles as the observation similarity function and the observation rotation function. The observation action function differs from the previous two functions in that multiple subregions of the observation vector o are evaluated and an offset is determined for each subregion. Thus, instead of the entire observation vector o, a subset of the observation vector corresponding to a spatial subregion is evaluated in the observation action function, from which it is possible to obtain an offset value for each subregion corresponding to that subset of vectors relative to the target observation. A mask is used to obtain the relevant subregions. Each subregion may overlap with one or more adjacent subregions.

[0245] The use of subregions for this purpose stems from the notion that not all parts of the visual field change equally as the agent moves toward or away from the predicted observation location. This differs from traditional triangulation in that it considers distortions of the entire visual scene, rather than the movement of identified objects or signal sources. This makes it robust to occlusions of parts of the visual scene and more resistant to environmental changes. Furthermore, it does not require metric calculations to function, making it more robust to measurement noise. This property allows us not only to determine how far the agent is from the target observation location, but also to calculate a further vector to move toward that location. This further vector, calculated by the observation action function, uses the same principle as the observation similarity function, except that dot products are taken for N spatial subregions and compared to the corresponding subregions of the target observation. For example, with N=8, the eight spatial subregions have half the x-axis range of the full field of view of the entire observation vector o, each centered in a different direction. In this example, the observation action function combines the observation similarity function and the observation rotation function, running eight times, once for each subregion, to find eight similarities and eight offsets. The eight similarities and eight offsets differ only with respect to specific subregions of the observation vector o and are substantially the same as the first and second observation comparison scale components, respectively.

[0246] Thus, in terms of the actual comparison performed, the stored target observation vector is divided into N subregions, and for each of the N subregions, the corresponding subregion of the current observation vector is compared against all possible rotational permutations to identify which permutation is most similar to the stored target subregion. The particular most similar permutation is associated with an index (e.g., 120) that is used to identify the offset. 番目 (rotation of). An offset is calculated for each subregion of the stored observation vector, and these offsets are used to obtain the action vector needed to reach the position of the target observation. The similarity is used to assess and weight the reliability of this motion vector when it is used in the motion function described below. Thus, the overall output of the observation action function 403-3, and the third observation comparison measure component, is this motion vector.

[0247] Figure 6 shows a schematic diagram of the observation data corresponding to the sub-regions of the observation vectors generated for the purpose of executing the observation action function. Figure 6 also shows the process by which the sub-regions are executed in the observation action function (with respect to the observation data for visualization purposes).

[0248] As shown in FIG. 6, the observation data is represented by a two-dimensional circle 601 of spatial data around the agent 602. This is a representative example, and it should be understood that the observation data may be three-dimensional, such as within a cylinder. The observation action function processes the observation vector o relative to the target observation. According to the observation action function, multiple subvectors of the observation vector are processed, with each subvector focused or centered on a different direction. In FIG. 6, these directions are shown as multiple directions 604-1 through 604-n in the corresponding observation data 603. In FIG. 6, there are eight directions and eight subregions. However, it should be noted that the process of creating the subregions may select more or fewer directions.

[0249] Each of the eight directions corresponds to eight sub-regions 605-1 through 605-n of the observation data (not all shown). These sub-regions correspond to portions of the field of view (or sensor field if not a camera), and each portion is centered on one of the eight directions. The specific direction associated with a sub-region is called the sub-region direction.

[0250] The observation action function, as described in more detail below, compares pairs of sub-regions 605-1 through 605-n to corresponding portions of stored observations, as shown in observation data 606, which compares a first sub-region 606a and a second sub-region 606b to corresponding portions of a target observation (not shown).

[0251] The observation action function calculates the offset between each sub-region 606a and 606b and the corresponding portion of the target observation, resulting in a pair of offsets 607a and 607b, as shown in observation data 607. From these offsets 607a and 607b, a resulting offset vector 609 is determined, as shown in observation data 608. The sub-regions 606a and 606b used in this process are different sub-regions, and may be, but do not have to be, opposite sub-regions.

[0252] The observation action function 403-3 will now be described in more detail with reference to the sub-regions of the observation data and the corresponding sub-vectors of the observation vector described above and illustrated in FIG.

[0253] As described above, the observation action function uses a set of masked subvectors. Each subvector is a subset of the stored observation vector o and corresponds to a contiguous subregion of the field of view. The subregions of the stored observation vector are compared to corresponding subregions of the current observation vector, which are equiangularly spaced around the agent. Following the previous example, the subregions of the stored vector might cover a set of contiguous columns in the stored 16x4 observation vector. An even number of overlapping subregions are extracted. There are N pairs of subregions, and the central column of each subregion is projected onto the real world with an opposing vector, as shown in Figure 6. Figure 6 contains eight subregions. These opposing subregions are matched with 256 rotations of the current observation, and the best matching rotation is extracted using the observation rotation function 403-2. Specifically, an offset corresponding to the best matching rotation index between the subvector corresponding to the subregion and the rotated and stored observation is determined for each opposing subregion. The difference in the direction of the offsets of the opposing pairs of observation vector subsets is compared. This provides a vector indicating the direction and magnitude of the displacement required to reach the stored observation (in Figure 6, this is along the resulting offset vector 609, perpendicular to the line connecting the centers of the two opposing subregions 606a and 606b). This direction and magnitude is the direction of where the target observation for comparison was captured. A weighted average of the offset vectors 609 from all opposing subregions provides a single action vector indicating the direction to approach the target observation for comparison from the current observation location. This vector is then rotated from the stored observation rotation to the current observation rotation using a second observation comparison scale component or another rotation measure to align the movement direction with the agent's current world frame. It should be understood that alternative implementations can also be implemented in which the subregion is evaluated in the current observation world frame. While this does not require such rotation, it introduces a higher computational burden since the subregion mask must be evaluated for each current observation rotation, and accuracy is reduced since the exact subregion cannot be evaluated with the stored observation.

[0254] Thus, the observation action function 403-3 can be thought of as a process that iteratively uses the observation similarity function 403-1 and the observation rotation function 403-2 to obtain a vector of the direction of the position of the target observation. Thus, the above observation action function 403-3, the observation similarity function 403-1 and the observation rotation function 403-2 can be performed in the same process, by the same comparison engine 503, or independently.

[0255] An exemplary process for performing observation action function 403-3 using comparison engine 503 is set forth below in the form of exemplary pseudocode. Action_function: For the current observation vector: o Generate a mask to select a spatial subregion of the observations in the following way: ■ Given an observation: observation [16x4x4], the first two dimensions are space (X, Y) and the third dimension is the different channels ■ For each subpixel spatial element: Select cyclically contiguous subsets of columns within the spatial element. By cyclically contiguous, we mean that for a 16-column dimension, columns 2, 3, 4 and columns 15, 0, 1, 2 represent examples of contiguous subsets, while columns 1, 3, 4 are not contiguous because there is no intervening column 2. These subsets must have the same width (e.g., 8 columns) and be evenly spaced between columns. For example, for four subsets of width 8 columns, indices 0, 4, 8, 12 constitute examples of even spacing when referring to the first, last, or other determined column within each subset. The determined subset is applied as a user_mask, with elements of each column within the subset set to 0 (used) and elements of outer columns set to 1 (not used). For each sub-area: o Execute combined_function (same as above) Subregion max_similarity and offset are saved A set of max_similarity and offset is given, one max_similarity and offset for each subregion o A pair of offsets where the subregions correspond to opposite directions in the agent's field of view (For example, for eight subregions, pairs consisting of indexes 0 and 4, 1 and 5, 2 and 6, and 3 and 7) are calculated by subtracting 128 from the circular distance between the first and second indexes of the offset in the pair. This corresponds to the angle between the offsets and how the two subregions moved in the field of view. The direction perpendicular to the pair of subregion directions is the action direction. The offset difference is used to calculate the magnitude of the action. The action directions and magnitudes of all pairs are vector-averaged to form the final action function output, i.e., the action vector. o The magnitude of the action is the resulting magnitude of the vector described by the offsets (the second offset increments by 128, corresponding to a PI radian rotation).

[0256] Thus, the observation-action function takes N opposing pairs of offsets (N=8), each pair representing a direction described by the perpendicularity of the opposing offset to the direction of the mask center. This direction is combined with a magnitude calculated from the difference between the two offsets; if both offsets are zero (i.e., the center of the subregion of the current observation vector is the same as the subregion of the saved observation), the magnitude is zero. If the difference between the first and second offsets is periodic, then if the first offset is 200 and the second offset is 10 (256 rotations), the difference will be -66, but if the first offset is 10 and the second offset is 200, the difference will be +190. This difference is converted to an angle and can be used to calculate an exact magnitude, or it can be scaled down, for example, by a factor of 16 to provide a rough magnitude for action vector purposes. This can then be used in the translation function to rotate using the second comparison measurement and combined with all pairs of action vectors as a vector average to relocalize the agent to the predicted observation. Vectors can be removed from the average based on a quality criterion, such as a minimum subregion similarity score, with the understanding that due to redundancy in the vectors generated by pairs, parts of the visual scene with large environmental differences will be weighted less compared to parts with small environmental differences.

[0257] A fourth method for comparing observations uses an observation variance function 403-4 to determine a fourth observation comparison measure component. The observation variance function uses similar properties as the similarity function and the action function. The observation variance function 403-4 involves calculating the circular variance or circular standard deviation of the offsets of a subregion around the agent. In effect, this observation variance function identifies a measure of the spread of the offsets on a circle around the agent.

[0258] The observation variance function 403-4 uses the same subregions of the stored vector used by the observation action function 403-3 to determine offsets in the same way with respect to the rotational permutation of the subregions of the current observation vector. For example, suppose there are eight subregions providing eight offsets. The eight offsets represent how different parts of the field of view move as the agent moves toward or away from the target observation location, so that when the agent is at the target observation location, they are all identical and equal to the offsets of a perfect vector match. As the agent moves away, they spread out. As in the observation variance function 403-4, this spread can be measured using a circular statistic, which increases with distance from the observation location. The circular statistic can be the circular variance or circular standard deviation of the eight offsets, providing a scalar value that forms the fourth observation comparison measure component. The circular statistic can be any measure of circular variance.

[0259] To perform the observation variance function 403-4, the observation action function 403-3 is first executed, taking N offsets. At this point, the periodic circular variance of the offsets is calculated after first converting them from the offset index range (0->255) to the appropriate angle range (e.g., radians (0->2*PI)). At this point, subregion offsets that exceed the specified angle from the circular mean angle (e.g., PI / 2 radians) are identified, and the circular variance is recalculated excluding these subregions, as long as more than the specified number (e.g., 4) of subregions remain.

[0260] The observation variance function 403-4 is another way to determine the similarity between the current observation and a stored observation. Unlike the observation similarity function 403-1, the observation action function is highly robust to changes in lighting conditions because it depends on the offset of a subregion rather than the entire observation vector. However, because the subvectors used have lower dimensionality than the full observation vector o, they are prone to discontinuities, and large distances can produce false matches that significantly impact the comparison. For this reason, the observation similarity function and the first observation comparison measure component provided by the observation similarity function provide a better distance measure of similarity.

[0261] Because the circular variance function 403-4 provides a scalar output indicating similarity, it can be used to measure similarity in the same way as the similarity observation function 403-1. Additionally, the circular variance function 403-4 can be used to determine an environmental adjustment factor, which can be used to weight or modify other comparison measure components, such as the first observation comparison measure component from the similarity function 403-1. In particular, if there is high confidence from another observation comparison function, such as the observation action function 403-3, that the agent is at the target observation, then the circular variance function 403-4 should provide a high measure of similarity in the complete space because each offset between the saved vector and the current observation vector should be zero. Therefore, the outputs of other functions can be weighted based on the deviation provided by the fourth observation comparison measure component of the circular variance function 403-4. For example, combining similarity function 403-1 with circular variance function 403-4 means that the first observation comparison measure component from similarity function 403-1 can be modified according to the fourth observation comparison measure component from circular variance function 403-4 to reduce and quantify the influence of environmental conditions.

[0262] Although examples of observation comparison functions 403-1, 403-2, 403-3, and 403-4 are provided above, including numerical examples, it should be understood that these functions can be performed with any selection of number of subregions, masks, channels, rotations, and array sizes.

[0263] The observation comparison measure may include one or more of first, second, third, and fourth observation comparison measure components derived from the observation similarity function, the observation rotation function, the observation action function, and the observation variance function, respectively. For example, an observation comparison measure is generated each time an observation is compared, as will be described in more detail below, when a first observation is compared to a set of stored observations in the second step 302 of method 300, when a predicted observation is compared to a second observation in the fourth step 305-4 of hypothesis testing, and when a transition observation is compared to a predicted observation.

[0264] It should be understood that the observation similarity function, observation rotation function, observation action function, and observation variance function described above are examples of functions that can be used to compare a current observation to a target observation. Other functions or adaptations of the above functions can also be used. Generally, such functions are used to compare observations to determine how similar the current observation is to a saved observation, how rotated / oriented the current observation is compared to the saved observation, and, if the agent is not at the saved observation's location, what direction the saved observation's location is from the current observation.

[0265] Once the observation comparison measures are obtained and hypotheses are determined in the second step 302 of the method 300, the first step in testing the hypotheses is to move the agent to a second location in space (first hypothesis verification step 305-1). The agent's movement is performed according to a movement function that depends on at least the predicted displacements of the hypothesis subset. In particular, the movement function depends on the predicted displacements required to reach the location of the predicted observation being compared. Therefore, it is necessary to select the correct predicted displacement from the hypothesis subset, which may contain multiple predicted displacements. To ensure that the correct predicted displacement is selected for the movement function, the predicted displacements and predicted observations stored in the hypothesis subset are linked to each other in a network of predicted observations connected by predicted displacements, as described above. The linking of observations to one or more displacements in this network can be stored and maintained in any suitable manner. For example, the set of stored observations and displacements, including the hypothesis subset, may be stored in a cyclic graph, an acyclic graph, or a directed graph. Alternatively, the observations and displacements may be stored with metadata identifying the displacements and observations to which they are connected, or may be organized in a lookup table that maintains the links between each observation and displacement.

[0266] As explained with respect to Figure 3, the hypothesis determination is based on the stored observation that is most similar to the first observation made by the agent. Therefore, the predicted displacement selected for the purposes of the transfer function is the displacement linked to the observation that is most similar to the first observation within the set of stored observations and displacements.

[0267] Once the predicted displacement from the hypothesis subset that is expected to lead from the most similar observation to the next predicted observation in the hypothesis subset is identified, the shift function is updated to be a shift based on the specified predicted displacement.

[0268] It is understood that the predicted displacement has both a magnitude and a direction.

[0269] The movement function may include one or more movement components. The first movement component of the movement function depends on the predicted displacement. In particular, the first movement component is a contribution to the movement function that decreases as the agent moves according to the predicted displacement. Thus, the first movement component is effectively weighted based on the magnitude of the predicted displacement. At the first observation location, the magnitude of the predicted displacement is relatively large, so the first movement component is weighted more strongly accordingly. When the agent begins moving according to the movement function, the magnitude of the predicted displacement is updated based on the actual displacement already traveled by the agent. Thus, as the agent moves toward the end point of the predicted displacement, the magnitude of the predicted displacement decreases, and the first movement component is weighted less and less. In one example, the first movement component is a function of the predicted displacement and is expressed as predicted displacement - traveled displacement. The traveled displacement is the actual displacement estimated using odometry, visual inertial odometry, or the like. In this example, the weighting described above may simply be a coefficient provided by subtracting the travel distance. As the distance traveled increases, the first movement component decreases from an initial value equal to the predicted displacement. In other words, this can be thought of as the predicted displacement "unraveling" as the agent moves along the predicted displacement. Thus, the value of the first movement component at any point in time is a function of the "remaining displacement" of the predicted displacement. This provides a first value that is contributed to the movement function. This first value is scaled according to a scaling factor, so that it can be understood that the contribution of the first movement component to the movement function is appropriate to account for the additional movement component.

[0270] As the agent approaches the end of the predicted displacement, the contribution of the predicted displacement to the movement function tends to zero. In one example, if the measured actual displacement equals the predicted displacement and the entire predicted displacement is moved, the first movement component based on the predicted displacement will be zero and will no longer contribute to the agent's movement function. If the first movement component is the only movement component of the movement function and the movement function depends only on the predicted displacement, the agent's position after moving the predicted displacement will be the second location, where the second observation is recorded.

[0271] Alternatively, the first movement component may stop contributing to the movement function before the entire predicted displacement has been traveled. In particular, the movement function may be configured to ignore the first movement component if its magnitude, i.e., the remaining distance of the predicted displacement, is within a displacement threshold distance from the end of the predicted displacement. This is advantageous because it reduces the impact of drift on the agent's movement. In particular, odometry-based estimation of actual displacement is susceptible to drift-related errors, which can result in an estimated actual displacement (actual distance traveled) that is smaller than the true value. Therefore, even if the estimated actual displacement is smaller than the predicted displacement, the agent may have already traveled the entire predicted displacement. To avoid overshooting the end of the predicted displacement, the first movement component may be stopped before the end of the predicted displacement based on a displacement threshold distance. The displacement threshold distance can be set based on the specific environment or scenario in which the agent is used. In physical space, this could be an appropriate value, such as 5 cm, 10 cm, 1 m, 5 m, or 10 m, depending on the scenario.

[0272] The second movement component that may be included in the movement function depends on the observation-action function described above with respect to observation comparison. When used in the movement function, this second movement component requires continuous, repeated, or sequential transition observations between a first location in space where a first observation is made and a second location in space where a second observation is made. In other words, transition observations are made as the agent moves away from the first location.

[0273] Thus, one or more transition observations are made along the agent's travel path. Each transition observation is a type of "current" observation that is compared to the predicted observation that the predicted displacement is expected to lead to according to the hypothesis. From each comparison, an action vector is generated, using the observation-action function described above, that defines both the expected magnitude and direction to where the predicted observation may be found. The action vector forms the second movement component of the movement function. Similar to the first movement component of the movement function, the contribution of the second movement component may be scaled and weighted by a scaling factor. In particular, the second movement component may be weighted such that the more the action vector of the second movement component indicates that the agent is closer to the location of the predicted observation, the stronger the contribution of the second movement component to the movement function. For example, the magnitude of the second movement component resulting from the observation-action function may be configured to increase as the agent approaches the location of the predicted observation. This increase in magnitude may directly provide a weight that makes the second movement component stronger as the agent approaches the location of the predicted observation.

[0274] The second movement component may be used alone as the sole contribution to the movement function, or in combination with the first movement component. Advantageously, when both the first and second movement components are included and both contribute to the movement function, the agent's movement depends on both the predicted displacement and the transition comparison between the predicted observation and the transition observation. This means there are two separate data sources used to move to and position the predicted observation in space. This effectively provides redundancy and is therefore more reliable and accurate than using only one set of data. Furthermore, the first and second movement components are advantageously complementary to each other in that the relative weighting of the first movement component weakens its contribution to the movement function as the agent approaches the end of the predicted movement, while the relative weighting of the second movement component strengthens its contribution to the movement function as the agent approaches the location of the predicted observation. Thus, the first and second movement components behave inversely.

[0275] If the hypothesis is correct, at least partially correct, or similar, then the endpoint of the predicted displacement should be at least close to where the predicted observation is found if the attributes associated with the hypothesis are at least similar to the attributes of the space the agent occupies.If the hypothesis is correct, at least partially correct, or similar, then the endpoint of the predicted displacement should be at least close to where the predicted observation is found if the attributes associated with the hypothesis are at least similar to the attributes of the space the agent occupies.

[0276] The relationship between the first and second movement components in terms of their contribution to the agent's movement function may take any suitable form. An example of the relationship between these two movement components is described with reference to Figure 7, which shows a schematic map of an agent moving between a first location and a second location in space.

[0277] In a first example of the relationship between the first and second movement components of a movement function, as shown in the first graph 701, the relationship is simple and depends only on the relative weighting of the first and second movement components. As the agent moves away from the first location 701-1, the relative weighting of the first and second movement components decreases and increases, respectively, so that as the agent moves closer to the first location 701-1, the first movement component naturally dominates in terms of contribution to the movement function. This is useful because, generally, when the agent is located far from the predicted observation, the observation action function responsible for the second movement component provides only a weak indication of where the predicted observation is in space. This is because the transition observations made for the purposes of the observation action function are typically very different from the predicted observations, as the distance between the capture points of these observations is likely large and the observations are therefore in different regions of space. Because the observation action function relies on comparing observations and identifying similarities between them, it is not very reliable when the observations differ significantly. Therefore, the fact that the first movement component dominates when the agent is close to the agent's first location 701-1 is advantageous because it allows the predicted displacement to be used to approximate where the predicted observation should be, which is more reliable than using only the second movement component.

[0278] In the first graph 701, as the agent moves, a series of transition observations 701-2 are made, and the second movement component, specifically the observation action function responsible for the second movement component, is periodically, continuously, or repeatedly updated. The agent follows the movement function along a path 701-3, moving to a second location 701-6 in space. As the agent approaches the end point of its displacement from the first movement component, the second movement component from the observation action function becomes dominant, relocalizing the agent to the second location 701-6. It should be understood that the second location 701-6 is not known before the agent moves; rather, the second location 701-6 represents a location in space that is identical to or best matches the predicted observation (target observation) linked to the predicted displacement. The second location 701-6 is determined by the location of the second observation from the observation action function, and the magnitude and direction of the observation action function vector indicate that the second location 701-6 is the location of the predicted observation or its best match.

[0279] In a second example of the relationship between the first and second motion components of a travel function, as shown in the second graph 702, the relationship is sequential, meaning that the travel function initially depends only on the first motion component and, once the first motion component expires, depends only on the second motion component. This relationship has the advantage that transition observations are not required for the majority of the agent's travel between the first location 702-1 and the second location 702-6, because the first motion component depends on predicted displacement values ​​rather than predicted observations. According to this relationship, the agent is configured to travel from the first location 702-1 via a first path 702-3, which corresponds to travel according to only the first motion component, based on predicted displacement. When the predicted displacement is depleted and the agent reaches the end of the predicted displacement at the intermediate location 702-5 or reaches a predetermined threshold from the end of the displacement, the travel function switches from using the first motion component to using the second motion component. At this point, the agent makes sequential or iterative transition observations 702-2 and executes the observation action function to obtain a magnitude and direction vector pointing to the location of the predicted observation. Using this information, the agent follows a second path 702-4 toward a second location 702-6. Thus, in the second relationship, the agent actually moves by a first displacement before relocalizing using the observation action function.

[0280] In a third example of the relationship between the first and second movement components of a movement function, as shown in the third graph 703, this relationship is a winner-take-all relationship, where the movement function depends only on the highest-weighted component of the first and second movement components at a particular time and location in space along the path to the second location. As noted above, the first movement component, which depends on the predicted displacement, naturally has a higher weighting than the second movement component, which is closer to the first location 703-1. Therefore, the agent moves along the first path 703-3 based on the first movement component and the predicted displacement. During this movement along the first path 703-3, the agent performs a series of transition observations 703-2 with the goal of updating the second movement component. The goal is to determine when the second movement component becomes dominant over the first movement component. The frequency of transition observations along the first path 703-3 is less important than the initial relationship shown in the first graph 701. This is because the transition observation 703-2 of the first path 703-3 is made only to compare the first movement component with the second movement component, not to guide the agent itself. Increasing the frequency simply increases the accuracy with which the second movement component is determined to be the "winner" in the winner-take-all relationship. At point 703-5 along the path, this determination occurs when the second movement component is determined to be dominant over the first movement component. At this point, depending on the observation action function, the second movement component is used in the movement function to relocate the agent to the second location 703-6. In this phase, the agent follows the second path 703-4 and makes further transition observations 703-2. This is necessary to obtain a vector from the observation action function to provide the direction and magnitude of the agent's movement to the second location 703-6.

[0281] Therefore, the transfer function uses two components to relocalize to the location of the predicted observation. The above three relationships between these two components are exemplary, and any suitable method or combination thereof may also be suitable.

[0282] The ability of the movement function to relocalize the agent to the location of the predicted observation depends on the observation-action function, which depends on whether similarities are found between the current observation and the predicted observation. Therefore, if the similarity between the observations falls below a certain level, the action function will not relocalize. In this case, the agent may move according to the first movement component of the movement function based on the predicted displacement. If it is determined that the agent cannot be relocated because there is no similarity between the current observation (transition observation) and the predicted observation, the predicted observation is not found at or near the end of the predicted displacement, and the hypothesis is rejected. In this case, the agent moves in the opposite direction along the predicted displacement, returning to the location in the space of the previous observation from which the hypothesis was determined. The hypothesis is then adjusted to another hypothesis associated with a different hypothesis subset of the set of stored observations and displacements.

[0283] Thus, the movement function provides the agent with the ability to move to a location where the predicted observation is found, a "best match" in space for the predicted observation (e.g., if the agent relocalizes to a location that is not an exact match but is still similar to the predicted observation), or a location where the predicted observation is not found. Because the movement function may be two-component, discrepancies between the predicted and actual displacements may occur. For example, if the best or perfect match between the predicted and transition observations is at the end of the predicted displacement, the agent may not need to relocalize using the second movement component. Alternatively, the agent may need to relocalize by moving slightly so that the actual movement distance is equivalent to the predicted movement distance. Finally, the second movement component may relocalize the agent to a larger area compared to the predicted displacement. In this case, the predicted displacement is not similar to the actual displacement.

[0284] These possibilities ultimately provide the method with a way to determine how similar a series of observations and displacements performed by the agent are to the hypothesized attributes in two separate dimensions. This ultimately helps determine whether to adjust, maintain, or confirm the hypothesis as described above. Specifically, the first dimension is the similarity between the current observation (or the observations for which the method 300 continues or repeats for two or more observations) and the hypothesized subset of stored observations. This similarity is determined by an observation comparison measure and is referred to as "style." The second dimension is the similarity between the current displacements (or the displacements for which the method 300 continues or repeats for two or more displacements) and the hypothesized subset of stored displacements. This similarity is determined by a displacement comparison measure and is referred to as "configuration." The displacement comparison measure is determined by comparing actual displacements with predicted displacements.

[0285] Generally, there are four possible combinations of style and composition: high style and high composition, high style and low composition, low style and high composition, and low style and low composition. We'll discuss in more detail below what each of these possibilities means for our hypothesis.

[0286] First, high style and composition indicate that the agent's observations are highly similar / identical to the predicted observations in the hypothesis subset, and the agent's actual displacements are highly similar / identical to the predicted displacements in the hypothesis subset. "Highly similar" means that both the observation comparison measure and the displacement comparison measure exceed the second higher observation threshold level and the second higher displacement threshold level, as explained above with respect to the fifth step of hypothesis testing 305-5. If the displacement and observation comparison measures match or exceed both of these levels, the space in which the agent resides is determined to contain attributes consistent with the identity defined by the hypothesis subset. As previously mentioned, identities do not need to be labeled, but labels can be associated that define the attributes associated with the identity. If the attributes of a space match the hypothesis subset of multiple consecutive predicted observations and displacements, confidence that the attributes in the space match the identity increases. Therefore, determining whether the attributes of a space match the identity may require comparing multiple consecutive predicted observations and displacements to a sufficient level of match.

[0287] Second, high style and low composition indicate that the agent's observations are very similar / identical to the predicted observations of the hypothesis subset, but the agent's actual displacements are not similar / identical to the predicted displacements of the hypothesis subset. "Very similar" means, with respect to the fifth step of hypothesis verification 305-5, that the observation comparison measure exceeds the second high observation threshold level, but the displacement comparison measure does not exceed the second high displacement threshold level. In terms of the space in which the agent resides, this means that, from the perspective of the sensor (observation) point data, the space appears the same or very similar in terms of the attributes defined by the hypothesis subset, but at least one of the attributes—size, location, and / or distance between—is not as predicted according to the hypothesis. This relationship is called "context." In the above example shown in Figures 1 and 2, the context would represent a similar-looking room with similar furniture as in Figure 1, but perhaps the room is larger and the furniture arrangement has been altered (e.g., a central element has been moved to one side). In Figure 2, context may be found if the chair is photographed from the back or if the chair's proportions are changed (e.g., the legs are shortened or the back is lengthened).

[0288] Third, low style and high composition indicate that the agent's observations are not similar / identical to the predicted observations in the hypothesis subset, but the agent's actual displacements are very similar / identical to the predicted displacements in the hypothesis subset. "Very similar" means that the displacement comparison measure exceeds the second higher displacement threshold level, but the observation comparison measure does not exceed the second higher observation threshold level, as per the fifth step of hypothesis verification 305-5. In terms of the space in which the agent resides, this means that, according to the hypothesis, the size, location, and / or distances between points in that space appear very similar, but do not appear / do not appear to share the same attributes defined by the hypothesis. This relationship is called a "class." In the examples of Figures 1 and 2 above, classes are represented by rooms of the same size and structure, but may differ in color and furniture. In Figure 2, classes may be found when chairs differ in color or when the style of details differs while maintaining the same proportions, such as several types of chairs designed for eating at a table. In terms of navigation, these may be different floors of a building, with identical floor plans but different companies providing the decoration. Multiple observation comparison measures may be used to determine class relationships between attributes in the space in which the agent resides and stored observation data. While the similarity indicated by a first observation comparison measure component may be low, a second and / or third observation comparison measure component may be used from the observation similarity function 403-1. If the observation action function does not provide convergence to a point, there is no match and the basis for comparison is invalid, in which case the hypothesis is rejected.

[0289] Finally, low style and composition indicate that the agent's observations are not similar / identical to the predicted observations in the hypothesized subset, and the agent's actual displacements are not similar / identical to the predicted displacements in the hypothesized subset. In this case, for the fifth step of Hypothesis Testing 305-5, the displacement comparison measure does not exceed the second higher displacement threshold level, and the observation comparison measure does not exceed the second higher observation threshold. In terms of the space in which the agent resides, this means that the space does not appear to be related to the structure or appearance of the attributes associated with the hypothesis. The hypothesis may be maintained for further comparison or rejected and adjusted as previously described.

[0290] By comparing the current observation and displacement with the stored observations and displacements for one or more hypothesis subsets, it is possible to identify characteristics of a space in terms of two dimensions: composition and style (structure and appearance). If a space is repeatedly or periodically evaluated by an agent, these dimensions can also be used to determine changes in the space. In this scenario, the stored hypothesis subset may include observations and displacements taken directly from the same space in previous interactions by the agent. Using this information, the agent can detect what has changed in the space over time in terms of structure and appearance. Identifying differences is useful, for example, for path planning applications and alternative route search. The agent can store the average displacement to a particular stored observation, which is updated each time the agent displaces to the observation. This average displacement provides a historical average of past displacements taken for a particular observation. Additionally, the standard deviation of the previous displacements taken by the agent for a particular observation may also be stored. Statistical analysis can be performed using the average displacement and standard deviation to determine the likelihood that the space has changed. For example, if the new displacement for an observation differs significantly from the average displacement and / or differs from the standard deviation of the previous displacements, it is determined that the space has likely changed. This may occur, for example, when spatial attributes are moved.,To perform statistical analysis, a historical record of each,observation and displacement can be kept, including the mean displacement,of the observation and the standard deviation.,This historical record may be updated with each exploration,of the agent.

[0291] A second history record can be stored and updated in a similar manner. The second history record relates to the N most recent observations and displacements made by the agent as it moves toward a particular observation. Specifically, it relates to the similarity or overall similarity score between these N most recent observations and displacements and the corresponding stored observations and displacements to which they are compared. Thus, the second history record represents a localization confidence, indicating how confident the system is that the agent was localized to the stored observations and displacements on its way to a particular observation. If the second history record indicates that the N most recent observations made by the agent closely match the N most recent stored observations, then confidence can be gained that the agent is properly localized to the subset or path it is following, in terms of observations and displacements. If the displacements made for a particular observation, or the particular observation itself, differ significantly from the corresponding stored observations and displacements, then the second history record can be used to reliably indicate and determine that the space has changed in some way with respect to that particular observation and displacement. For example, if N=5, then the last five displacements made by the agent may perfectly match the stored displacements. If the last six displacements made are significantly different from the stored ones, then the second history record can be used to reliably indicate and determine that the space has changed in some way with respect to that particular observation and displacement. For example, if N=5, then the last five displacements made by the agent may perfectly match the stored displacements. 番目 The displacement of 6 is preserved. 番目 If the displacement differs from the previous one, the fact that the second historical record shows a high level of agreement (in this case, identity) with the previous five displacements is indicative of the 6 番目 This means that it is easy to determine that the space has changed in terms of displacement. This may be due to the movement of obstacles etc.

[0292] Furthermore, as described above, while hypotheses may be confirmed, maintained, or rejected based on observations or displacements separately, it is advantageous to use both observations and displacements, i.e., composition and style, when determining the outcome of a hypothesis. Adjusting a hypothesis by combining the observation comparison measure and the displacement comparison measure to form a two-dimensional similarity measure may include adjusting the hypothesis if the two-dimensional similarity measure falls below a first two-dimensional similarity measure threshold level. Maintaining a hypothesis may include maintaining the hypothesis if the two-dimensional similarity measure exceeds a first two-dimensional similarity measure threshold level but does not exceed a higher second two-dimensional similarity measure threshold level. A step of confirming the hypothesis may occur if this second two-dimensional similarity measure threshold level (also referred to as a hypothesis-confirming two-dimensional similarity measure threshold level) is exceeded. If there is a change in the visual identity of an observation encountered at a different time, the agent may use the current observation information to update the stored observation if it is confident that the current observation and past observations were captured with the same identity. Because both observations are represented by normalized vectors, we can average the two vectors, applying optional weighting to each, so that the final observation closely matches either the past or the current observation vector. For example, multiplying the past observation by 5 and the current observation by 1 and renormalizing the sum vector will make the final observation vector more closely match the past vector.

[0293] Adjusting a hypothesis involves rejecting a hypothesis and returning to a previous observation to identify a second hypothesis. In particular, if, for any observation, the current observation does not match the predicted observation with adequate confidence and / or the actual displacement does not match the predicted displacement with adequate confidence, the hypothesis is rejected. The agent is then configured to return to the previous observation by taking a movement opposite to that taken to reach the current observation. This may include, but is not required to, returning to the first observation. The method 300 then iterates to test a different hypothesis selected based on a comparison of the previous observation with each of the multiple hypothesis subsets of the predicted observation. Optionally, the agent may store the observation and displacement even if the observation and displacement do not match the hypothesis, which is then rejected. Saving these observations and displacements can be useful in generating a map of the space for future navigation and attribute identification purposes. Thus, the agent may make and store one or more observations and displacements before returning to the previous observation, even if a hypothesis is rejected.

[0294] Verifying an observation includes, but is not limited to, identifying or locating an object, image, or destination within the space. Once an attribute is found, a reference to the attribute (including the agent's observations and displacements within the space) is stored in memory. These may form part of a map of the space in which the agent resides. If a hypothesis is associated with a labeled identity or attribute, the agent may also store a reference to the label in memory, along with the observations and displacements it made within the space. If a hypothesis is based on a previous visit to the same space, the hypothesis subset may be updated with the set of displacements and observations the agent made recently as it validates the hypothesis, since the hypothesis subset effectively defines the attributes previously observed in the space. In this way, the hypothesis subset is kept up-to-date with respect to a particular space.

[0295] It should be understood that the above method may be performed by a single agent or by one or more agents working together. When more than one agent is included in a space, the one or more agents may communicate with each other or access the same memory. For example, multiple agents may each be linked to the same server or computer system. Multiple agents may simultaneously update a map of the space or test one or more hypotheses simultaneously. Map generation is described in more detail below.

[0296] The above description describes a method 300 for analyzing a space and identifying attributes of the space. The attributes may be objects or images, in which case method 300 is used for object or image recognition purposes. Alternatively, the attributes may be destinations, in which case method 300 is used for navigation purposes. Method 300 relies on an agent making observations and displacements in the space and comparing them to stored observations and displacements. Thus, one principle of method 300 is the agent's ability to compare observations and displacements. This is illustrated in Figures 4-6 and the above description. Another principle is the agent's ability to move, which is illustrated in Figure 7 and the above description.

[0297] Further aspects will now be described that complement the method 300. These aspects may be incorporated into the method 300 or may be performed or implemented separately.

[0298] A first additional aspect is the generation and storage of linked observations and displacements that form identities and hypothesis subsets. This can be thought of as a "training process," as briefly mentioned above. The process of forming identities can be performed specifically for the space in which the agent is expected to reside, or it can be performed in another space. Both of these implementations have advantages. If identities are formed in the same space the agent is later configured to explore, it becomes possible to identify how attributes of that space change over time after further exploration of that space. If identities are formed in separate spaces, it becomes possible to identify when attributes from one space recur in another space, or to identify similarities and differences between spaces.

[0299] The process of forming and storing an identity will now be described with reference to FIG. 8. FIG. 8 shows a schematic diagram 800 of an identity. It is important to note that no observations or displacements are stored. An agent is "dropped" or activated into a previously unknown and unexplored space. An initial observation o1 is made within the space. The initial observation o1 may be based on low-level features detected within the space, but this is not required. The use of low-level features will be discussed separately below as a further complementary aspect. The selection of the initial observation can be random or based on a learned process. From the initial observation o1, an initial displacement is made from the location of o1 to make a new observation o2, the displacement between o1 and o2 is stored in memory as d12, and the inverse displacement from o2 to o1 (the negative value of d12) is stored in memory as d21.

[0300] Optionally, an additional effort value is stored that is a better measure of the characteristics being optimized in the routing (such as energy usage and / or vibrations encountered).

[0301] Next, a sequential set of observations Oi{oi,1...oi,n} and a corresponding set of displacements Di{di,1...di,n-1} are traveled by the agent and stored in the same manner. This forms an identity. The set of observations and displacements that form the identity describes a predicted path through space and the expected environment at points on the path. Traveling the path allows the agent to verify the prediction (i.e., hypothesis) by comparing the measured displacements and observations with the predictions one by one. The sequential set of observations and displacements that form the identity are stored as described above. In some instances, an identity can be stored as a subset of a larger stored set, where the larger stored set contains multiple subsets, each defining a path through the same or different spaces. Thus, an identity may be linked to other identities, where the identities share one or more common observations and / or displacements.

[0302] As shown in Figure 8, a set of observations o1 through o4 and their corresponding displacements can be taken in a closed loop to form an identity, which can then be verified against future observations and displacements in sequential order for successive subsets or the entire set using the style and composition dimensions described above. This identity can be used for purposes such as locating or identifying specific objects, extracting common repeating classes of objects or navigation terrain, including but not limited to building hallways and rooms, and extracting contexts, which are areas or objects with similar components (e.g., forests / offices / roads or wheeled vehicle characteristics / flying objects).

[0303] While or after the identities are stored as a subset of the observations and displacements, a further optional process may include using another system to determine the attributes with which the identities are associated. For example, the identities may be associated with a path for purposes of navigation to a destination in space. Alternatively, the identities may be associated with an object such as a chair, or an image such as a cartoon chair or a painting of a chair. To correctly associate the identities with these attributes, a training process may be used. The training process may involve identifying attributes via an established training algorithm, such as using training data to train a classifier. It should be understood that any machine learning process may be used, and may include the use of neural networks, etc.

[0304] A second further aspect is the formation of a map of the space the agent resides in. While the agent may be controlled to test hypotheses to identify attributes of the space it resides in according to the methods described above, it may also / alternatively be controlled to generate a map of the space it resides in. It will be appreciated that map generation can be performed independently or in combination with identity generation and attribute determination / hypothesis testing.

[0305] The agent can generate a new map if the space is completely unknown from the agent's previous explorations, and can update an existing map in memory if the space has been visited by the agent before.

[0306] To generate the map, the agent makes a first observation in space, which may be the same as the first observation made in the first step 301 of the hypothesis testing method 300 described above. The first observation is stored in memory as an indication of a corresponding "first location" on the map.

[0307] The agent is then configured to move away from the first location and make one or more further observations.

[0308] When a map is being created or updated in conjunction with an agent performing hypothesis testing, the agent moves according to the movement function described above. In this case, the agent is configured to make one or more new observations as it leaves a first location. The new observations may be made while the agent is moving according to the movement function (during a transition) or may be made afterward. In this regard, the agent may wait until it arrives at the second location of the second observation of method 300 described above before making a "new" observation, in which case the second observation becomes the new observation.

[0309] A new observation is compared to a first observation using one or more observation comparison functions described above, providing one or more observation comparison measures. The observation comparison measures are compared to a mapping threshold, which defines the maximum probability that consecutive observations must fall below to be recorded on the map. In other words, the method for generating the map requires that the observations stored in the map be dissimilar. This avoids adding unnecessary points to the map, making the map more concise and minimizing the memory required to store and access the map. If the observation comparison between the first observation and the new observation results in an observation comparison measure lower than the mapping threshold, the new observation is stored in memory at a "new location" on the map. The displacement between the first observation and the new observation is measured and / or obtained from the displacement function and is also stored in the map, along with the inverse displacement from the new observation to the first observation. When a map is generated while hypothesis testing is also being performed, the agent performs two comparisons: the first, comparing the current observation to the predicted observation (to test the hypothesis), and the second, comparing the current observation to a previous observation (e.g., the first observation) to determine whether these observations are dissimilar enough to be included in the map. Thus, even if a hypothesis is rejected and the current observation does not match the predicted observation well enough, the agent can still make an observation or use the current observation for the purpose of adding to or updating the map by comparing it with previous observations.

[0310] If the mapping threshold is exceeded, the new observation is not stored on the map, instead the agent is configured to move again at a further displacement before making further observations, until the mapping threshold is no longer exceeded.

[0311] Once a new observation is stored on the map, along with the displacement between the first and the new observation, the new observation effectively becomes the first observation (or a previous observation), and the process is repeated for further displacements and further observations. This process iteratively builds a connected network of observations that forms a map describing the space in which the agent resides.

[0312] A map of a space can also be generated independently of identifying the spatial attributes. The method for generating the map is largely the same. An agent makes a first observation and stores it in memory as a first location on the map. The agent then moves from the first location of the first observation in a direction that is unexplored and physically or logically traversable. The agent's movement is recorded as a first displacement. As described above, the agent makes one or more new observations away from the first location and compares the first and new observations to determine whether they exceed the mapping threshold. If they do not, the new observation is stored in the map, linked to the first and new observations by the first displacement and the inverse of the first displacement. The relative direction between the first and new observations is marked as "explored." This process is then repeated with further displacements and observations from the new observation location in unexplored directions. The use of a mapping threshold means that the number and density of observations made in a particular space depend on the complexity and features of that space. In feature-rich environments, more observations are made because the distance at which similarity between observations falls below the mapping threshold is shorter and decreases more quickly. In featureless environments, the space is less complex and therefore fewer observations are required. Therefore, methods for mapping and / or identifying attributes are implemented in such a way that they automatically adapt to the constraints of the space in which the agent is observing or moving. These methods are therefore sensitive to the complexity of the environment.

[0313] Multiple agents may contribute to the same map. In particular, multiple agents may each contribute to and access a shared database or memory, and may be configured to update the shared database with sets of observations and displacements to generate the map. As hypotheses are tested and attributes are identified, identities associated with the attributes are stored on the map and, if possible, labeled as such. Thus, a map may include multiple identified attributes and stored labels for those attributes. Agents may contribute to the generation of the map in real time or near real time, each communicatively coupled to the shared database via appropriate means. For example, in a physical space, multiple agents may be a fleet of drones, robots, vehicles, etc., each capable of communicating with a computer system or a distributed computer system, such as multiple servers.

[0314] This map can then be used to navigate the space: in particular, a destination on the map can be selected as the destination to which an agent should move, this destination being represented by an observation that forms part of the map.

[0315] Navigating a map works similarly to the method 300 for testing hypotheses and determining spatial attributes. When navigating a map, the attributes essentially become destinations. The navigation process is described in detail here.

[0316] A map is generated as described above and stored in memory, forming a set of stored observations and displacements, at least a subset of which are connected or form a network with the observations associated with the destination. This map is thus similar to the hypothesis subset described above.

[0317] The agent is configured to make a first observation in space and identify which of the stored observations of the stored map the agent is closest to. In particular, the first observation is compared to each observation connected to the destination in the stored map. From these comparisons, an observation closest to the first observation is identified. The agent is then moved to relocalize to the location of the closest matching observation.

[0318] From the closest matching observations on the map, a route is planned using a routing algorithm. Any suitable routing algorithm can be used, such as the Dijkstra algorithm, which determines a route that minimizes distance, effort, vibration, or other measurements that can be obtained from the displacements stored on the map.

[0319] Once a route is planned, the agent moves sequentially along the route, and the process is repeated as the agent moves from observation point to observation point on the map. The agent starts this process from the "closest matching observation," and the route may contain multiple observations between the closest matching observation and the destination, for example. The observation where the agent is currently located is the current observation, and the next observation on the map along the route to the destination is called the target observation.

[0320] The iterative process involves moving the agent toward the target observation using a stored displacement vector between the current observation and the target observation. This is similar to moving the agent according to a movement function depending on the predicted displacement, as described above with reference to the hypothesis testing method 300. Similar to hypothesis testing, this process also involves performing one or more observation comparisons using one or more of an observation similarity function, an observation rotation function, and an observation action function. For example, the stored displacement to reach the target observation can be used in combination with an observation action function to move the agent from the current observation to the target observation. Thus, the agent can make multiple transition observations while moving between the current observation and the target observation and then use the observation action function to relocalize to the location of the target observation. This is similar to the movement function including a first movement component and a second movement component described above, and can account for odometry drift and uncertainty when changing movement direction and reconstructing the displacement between the current observation and the target observation.

[0321] When the observation comparison measure (or measures) exceeds a predetermined or continuously calculated threshold, the agent is determined to have reached the target observation. This is similar to determining the agreement between the second observation and the predicted observation in hypothesis testing method 300. Once the agent is determined to have reached the target observation, the process is repeated for the next observation on the route to the destination until the destination is reached. Thus, the target observation becomes the current observation, the next observation becomes the target observation, and the displacement traveled becomes the displacement between the current observation and the target observation. Once the agent reaches the observation associated with the destination, the agent is configured to wait at that location unless instructed otherwise.

[0322] In this way, the observation, displacement, and movement functions can be used to identify attributes and features of a space, generate a map of the space, and navigate the space. It is important to note that one or more, or all, of these processes may be performed simultaneously. For example, an agent can be instructed to follow an existing route to a destination, identify attributes and features along the way, and add them to the existing map using transition observations. Thus, the agent can simultaneously learn and navigate its environment.

[0323] In the above-described methods and systems for analyzing a logical or physical space, including identifying attributes / features of the space, generating a map of the space, and navigating the map of the space, an agent is configured to make and compare observations and navigate between these observations. As noted above, the logical or physical space need not be known to the agent or have been previously visited by the agent. Thus, there are several sets of initial conditions that the agent may encounter when placed in the space.

[0324] In a first example, the agent is located in a previously visited space, and the agent has access to a saved map or saved subset containing observations and displacements made within that space. In this case, when making a first observation, the first observation may be identical to or match a saved observation (such as a predicted observation in a hypothesis subset or an observation along the route to the destination). In this case, there is no need to relocalize the agent or navigate the route to the saved destination before performing method 300 to test the hypothesis.

[0325] In a second example, an agent is placed in a previously visited space, and the agent has access to a saved map or saved subset containing observations and displacements made within that space. In this case, when making a first observation, the first observation may not be identical to any of the saved observations in the map or saved subset. Because the agent's location is unknown, the saved displacement cannot be used to relocalize the agent onto the saved observation. Rather, the agent must first relocalize to the saved observations without using a predicted displacement to understand its location on the map, or relative to the subset of saved observations and displacements. To do this, the agent compares the first observation with each saved observation and uses an observation-action function to determine a vector to one of the saved observations (referred to here as the target observation). The agent then moves along this vector, repeatedly or continuously updating further transition observations, and relocalizes to the target observation.

[0326] The selection of which stored observation becomes the target observation can be determined in any reasonable manner. In one example, the stored observations may be compared to the first rotation for each rotation using an observation similarity function. The most similar stored observation may then be selected for comparison with the first observation to determine an action vector from the observation-action function.

[0327] Alternatively, an action vector can be determined for each stored observation and the strongest one selected to relocalize the agent to the stored observation. The action vectors are determined from the observation-action function as described above and are weighted based on the similarity between a subregion of the stored observation vector and the corresponding dub region of the current observation vector, or based on an overall similarity measure, making them stronger.

[0328] In a third example, an agent may be placed in a space not previously visited, in which case the agent has access to a saved subset containing observations and displacements from a different space or multiple spaces. In this case, when making a first observation, the first observation may not be identical to any of the saved observations in the saved subset. Because the agent's location is unknown, the saved displacement cannot be used to relocalize the agent onto the saved observation. Rather, the agent must first relocalize to the saved observations without using a predicted displacement to understand its location relative to the subset of saved observations and displacements. To do this, the agent compares the first observation to each saved observation and uses an observation-action function to determine a vector to one of the saved observations (referred to here as the target observation). The agent then moves along this vector, repeatedly or continuously updating further transition observations, and relocalizes to the target observation.

[0329] The selection of which stored observation becomes the target observation can be determined in any reasonable manner. In one example, the stored observations may be compared to the first rotation for each rotation using an observation similarity function. The most similar stored observation may then be selected for comparison with the first observation to determine an action vector from the observation-action function.

[0330] Alternatively, an action vector can be determined for each stored observation and the strongest one selected to relocalize the agent to the stored observation.

[0331] However, unlike the first and second examples above, in this third example, the subset of stored observations and displacements is from a different space than the space in which the agent currently resides. Therefore, the space in which the agent resides may not contain any stored observations. In this case, the agent may be unable to relocalize to the stored observations. If the agent is testing a hypothesis, and the subset of stored observations and displacements is a hypothesis subset, the agent may reject the hypothesis associated with the hypothesis subset if it is unable to locate to the stored observations in the hypothesis subset. The agent can then use a negative displacement to return to the location in space of the first observation. From here, the agent can select another hypothesis and the corresponding hypothesis subset to test. The agent can repeat this process as necessary for all stored hypotheses and hypothesis subsets. If all hypotheses are rejected—that is, if the space does not contain any recognizable features or attributes corresponding to the subset of stored observations and displacements—the agent enters exploration mode, making observations and displacements to generate a map of the space, as described above.

[0332] In the fourth example, the agent is placed in a space that has not been visited before, and the agent does not have access to a saved subset containing observations and displacements. In this case, the agent enters exploration mode as described above, mapping the unknown space in which the agent resides, and ultimately saving a set of observations and displacements to describe the space.

[0333] The above description focuses on agents that perform sequential observation and displacement to perform one or more of the following: generating a map, navigating a route in space, or identifying spatial features. While it is possible to perform these methods using only observation and displacement, additional elements may be incorporated to gain further efficiency and accuracy benefits.

[0334] In particular, low-level features of space may be identified or detected and used to inform the process of observation and movement within the space between observations. In particular, observations by an agent contain spatial information that describes the spatial environment in the agent's local neighborhood. Therefore, features can be extracted from observations or observation data to obtain more information about the agent's neighborhood. For example, environmental cues such as high-contrast lines, colors, edges, and shapes can be extracted to determine more information about the environment and inform decision-making and actions. For example, in a map generation where an agent is configured to explore a space by sequentially making observations and moving, the selection of the direction to move the agent after each observation may be controlled by low-level features. Specifically, after each observation, the agent's selection of the next movement direction may be weighted based on low-level features and / or previously navigated directions. For example, a first weighting can be applied to directions that move the agent closer to previous observations as it moves. Because the purpose of map generation is for the agent to explore and map the space, it is beneficial to encourage the agent to move away from previous observations, and this first weighting achieves this. A second weighting may be applied to directions based on low-level features. For example, if an edge or high-contrast line appears to be going in a certain direction, that direction may be weighted higher than a direction without low-level features. In physical space, the agent may be a vehicle. Extracted low-level features may include roads (e.g., from edge detection). A direction where a road appears or leads to a road from the current observation may be weighted higher than a direction from the current observation without a road. This second weighting based on low-level features encourages the agent to follow or approach potential features in the environment. For example, following a road increases the likelihood of exploring the space and increases the likelihood that the agent will make observations that can be easily traversed and connected.While roads are man-made features added to the environment by humans, we know that similar beneficial features exist in natural environments, such as the banks of waterways, grassy forest edges, and foraging animal trails carved into vegetation. By following such features during both the exploration and navigation phases, the agent can ensure it has additional guidance to ensure reliable movement even in the face of significant environmental variations, such as thick fog. It is also important to note that such features can also help the agent recognize the identity of objects such as chairs. We guide the movement of the observation point along the edges of the chord formed by the chair's legs, seat, and back, ensuring high robustness in reconstructing identity even in the presence of 3D rotations.

[0335] It should be understood that the first and second directional weightings described herein can be used separately (without the other) or in combination.

[0336] Low-level features identified or detected in observations or observational data can also be linked to a set of actions to be performed by the agent. These actions allow the agent to approach features in a similar manner when encountering a new portion of space or object for the first time and when encountering the same portion of space or object again. For example, if a high-contrast line is identified or detected, the agent may be configured to move along the high-contrast line. The agent may be configured to move along consistent lines in the environment to a vanishing point, allowing it to move along roads, sidewalks, or paths. Such structured actions simplify the problem of robust navigation. The agent can move in different predetermined directions relative to the low-level features, such as perpendicular, parallel, clockwise, or counterclockwise relative to the low-level features.

[0337] Observations can also be made depending on the presence of low-level features in the environment. Characteristic low-level visual statistics, including but not limited to spatial frequency distribution, luminance distribution, color distribution, etc., can be used to determine when and where an agent should make a first observation.

[0338] The use of low-level features is not limited to map generation. In particular, low-level features can be used in any of the methods described above. For example, in hypothesis testing, according to the method 300 described above, low-level features or cues can be used in a movement function when determining how to move an agent from a current observation to a predicted observation. The movement function depends on the predicted travel distance required to move the agent to the predicted travel distance, but may also depend on low-level features such as high-contrast lines or edges. In physical space, an agent may be configured to follow low-level features such as roads, even if the roads do not directly align with the direction of the predicted displacement. Roads may be identified, for example, by displaying them as a single color extending upward from the bottom of the field of view where the agent is located to the top of the field of view, which primarily includes more distant locations. Alternatively, roads may be identified as areas that are extended as described above but lack strong horizontal edges and contain strong vertical edges. For example, if the road is within a threshold angle of the direction of the predicted displacement, the agent may be configured to follow the road rather than following the predicted displacement. This may improve the system if the predicted displacement is inaccurate for some reason. The agent periodically makes transition observations as it moves to ensure that its movement is properly aligned with the predicted observations. If the predicted observation is not reached, or if, for example, the observation-action function indicates that it is moving further away from the agent, the agent may leave the road. The agent may also move to the end of a low-level feature (e.g., the end of the road or the next road intersection). If the agent does not find the predicted observation at this point, it can still make and store the observation to generate a map or further populate the map with new knowledge about the space before rejecting the hypothesis and reverting to the previous observation.

[0339] Following or otherwise using low-level features is advantageous because it simplifies the process of testing hypotheses and navigating space, making it easier to backtrack from a point in space to a previous observation by simply following the low-level features in the opposite direction. Additionally, the agent can track features without any distant descriptive information, improving robustness to visually significant environmental changes such as heavy fog.

[0340] The above discussion introduces various concepts. One such concept is the use of an action function to compare a stored observation to a current or new observation, effectively determining the similarity between the stored and new observations and providing a vector from the new observation to the stored observation. The agent can then use the vector when determining how to move to reach the location corresponding to the stored observation. This process is explained in more detail in Figures 11 through 15, and with reference to Figures 6 and 9 above.

[0341] As explained above, the observation action function compares subregions of the current observation with subregions of the saved observations, where each subregion of the current observation and the saved observation is a portion of the entire current observation and the entire saved observation.

[0342] The use of subregions for this purpose stems from the notion that not all parts of the visual field change equally when an agent moves toward or away from the location where the stored observation was obtained. This differs from traditional triangulation in that it considers distortions of the entire visual scene, rather than the movement of identified objects or signal sources. This makes it robust to occlusions of parts of the visual scene and more resistant to environmental changes. Furthermore, since it does not require metric calculations to function, it is more robust to measurement noise. This property allows the agent not only to understand how far it is from the location corresponding to the stored observation, but also to calculate a further vector to move to that location. This further vector, calculated by the observation action function, uses the same principle as the observation similarity function, except that a dot product is taken between the N spatial subregions of the current observation and the N corresponding subregions of the stored observation.

[0343] To obtain the sub-regions, the process described above with respect to Figures 6, 9 and 10 is performed, which will now be described in more detail with reference to Figure 11.

[0344] FIG. 11 shows a schematic diagram of the results of several processes performed on original observation data to obtain an observation vector or matrix o from which multiple sub-regions are derived. In FIG. 11, the original observation data is acquired in a first step, and an original image 1101 is obtained. The original image 1101 is a 360-degree view of the agent's environment acquired from a first location, where the first location is the agent's location. In this case, the original image 1101 forms the first or current observation data, also referred to as first sensor data. Similarly, the original image 1101 may be historical data or stored data. The process of forming the sub-regions is the same for both stored and current observation data. However, with regard to stored observation data, also referred to as second sensor data, it is not necessary to store the original image; only a single observation vector or matrix corresponding to the original image (in a sliced ​​or reduced-size format) needs to be stored. This will be explained in more detail below. To obtain original observation data, such as the original image 1101, a sensor data acquisition process is performed by the agent. It will be understood that any sensor or virtual sensor capable of recording spatial information can be used. For the original image 1101, a camera may be used.

[0345] In a second step, the original image 1101 is split into individual feature channels, providing, for example, four feature channels. These feature channels may include, for example, red, green, luminance vertical edges, luminance horizontal edges, etc. The resolution may be the same as the original image 1101, for example, 256x64 pixels. Figure 11 shows a red feature channel 1102; other types of channels are possible. The edges can be obtained by convolution with a 1x2 kernel.

[0346] In a third step, for each feature channel of the original image 1101, such as the red channel 1102, the image is processed, for example, with a box filter of a 17x17 square (or any other size square, such as a 19x19 square), to blur the information in each channel, generating a filtered image 1103 for each channel. It should be noted that other types of image processing are possible, as will be appreciated.

[0347] In a fourth step, a reduction operation is performed on the filtered image 1103. This reduction process involves performing a slicing operation to reduce the dimensions of the filtered image 1103. In particular, the filtered image 1103 is sliced ​​to remove some of the rows of the filtered image 1103 and some of the columns of the filtered image 1103. A first reduced format image 1104 is shown in FIG. 11. This first reduced format image 1104 is the result of slicing a predetermined number of rows from the filtered image 1103. In FIG. 11, four rows are "sliced" from the filtered image 1103 (the remaining rows are discarded), resulting in a 256x4 first reduced format image 1104. The selection of rows to be cut and used is arbitrary, but it is advantageous to use rows that are not adjacent to each other, such as rows 8, 24, 40, and 56, or rows 6, 22, 38, and 54. A second reduced format image 1105 is also shown in FIG. 11. This second reduced format image 1105 is the result of slicing a predetermined number of columns from the filtered image 1103 (in addition to the slicing operation performed to obtain the first reduced format image 1104). In Figure 11, 16 columns are "sliced" from the filtered image 1103 (the remaining columns are discarded), resulting in a 16x4s reduced format image 1105. The choice of columns to use is arbitrary, but it is advantageous to use columns that are not adjacent to one another, for example, columns 1, 17, 33, 49, 65, 81, 97, 113, 129, 145, 161, 177, 193, 209, 225, and 241. It is even more advantageous to select evenly spaced columns to distribute the data evenly around the agent.

[0348] Once both slicing operations are performed (to obtain a second reduced-format image 1105 as shown in FIG. 11 ), an observation vector o is obtained. The observation vector or matrix is ​​a rough representation of the agent's 360-degree environment according to the original sensor data 1101. In the example of FIG. 11 , this observation vector is formed from the second reduced-format image 1105, which itself is a 16x4 matrix representing a reduced and simplified version of the original data 1101. Other dimensions are possible and will be understood. Selecting evenly distributed rows and columns to retain from the filtered image 1103 in the slicing operation ensures that the observation vector or matrix adequately represents the entire field of view associated with the original image 1101. In other words, it is advantageous if the columns and rows of the filtered image 1103 that are sliced ​​to form the observation vector or matrix are evenly distributed throughout the filtered image 1103.

[0349] Additional processing can be performed to obtain an observation vector in a suitable form for performing a comparison. In particular, the 16x4 s reduced-format image 1105 is periodically convolved with an off-center perimeter kernel (or vice versa). The channels of the 16x4 s reduced-format image 1105 can be divided into two data subsets: one consisting of red and green, or otherwise "color," vectors, and the other consisting of luma_h and luma_v, or otherwise "edge," vectors. Each of these data subsets is normalized by vector normalization (division by their respective Euclidean norms) so that the elements of each color and edge vector represent values ​​on an orthogonal set of values, resulting in a unit vector. These normalized vectors are each scaled by the square root of 2 and concatenated into a single unit vector. This provides the observation vector o relating the second reduced-format image 1105 to the original image 1101. The observation vector in this example is concatenated from 16 columns, 4 rows, and 4 channels, resulting in a 256-element vector. These operations are performed on the 16x4 second reduced format image 1105 to generate a one-dimensional vector suitable for comparison calculations, but it should be understood that references below to the second reduced format image 1105 (a 16x4 matrix) are synonymous with the observation vector. The observation vector is essentially the same information as the second reduced format image 1105, but in a different, normalized form. Therefore, the second reduced format image 1105 may be referred to as the observation vector.

[0350] The first through fourth steps above are applicable to either the first (current) sensor data or the second (stored) sensor data, meaning the process of forming the observation vector or matrix for each of these is exactly the same.

[0351] In some embodiments where an observation action function is performed, an additional fifth step is performed to obtain a sub-region from the observation vector.

[0352] In this fifth step, the observation vector or matrix 1105 is subsampled to obtain subregions 1106a-1106n of the sensor data. Figure 11 shows subregions 1106a-1106n for the red feature channel. The subregions 1106a-1106n are essentially portions of the field of view roughly represented by the 16x4 observation vector or matrix. In this fifth step, a mask is applied to the observation vector or matrix 1105 to obtain each subregion. The mask is overlaid on the observation vector or matrix 1105, and the output of this operation is a representation of the cells or pixels of the observation vector or matrix 1105 overlaid by the mask. Because the mask is smaller than the dimensions of the observation vector or matrix 1105, only a portion of the pixels of the observation vector or matrix 1105 are reproduced by the mask. This reproduced portion forms a subregion. In the example, the mask is 8x4 pixels and is overlaid on the 16x4 observation vector or matrix. This means that the mask discards eight columns of pixels, occupying half the field of view of the original image 1101. To obtain multiple subregions 1106a-1106n, the mask can be repeatedly permuted across different columns of the observation vector or matrix 1105, or alternatively, the observation vector or matrix 1105 can be permuted column-by-column across the mask each time. These two operations are essentially the same. It should be understood that because the observation vector or matrix 1105 is a coarse representation of the full view (e.g., a 360-degree view) corresponding to the original image 1101, the last column of the observation vector or matrix 1105 is effectively adjacent to the first column, and the mask can be cyclically permuted past the last column to the first column.

[0353] The size of the mask and the number of columns it shifts relative to the observation vector or matrix 1105 can be varied, and varying these variables can have a variety of effects. In the example above, the mask is eight columns wide, half the width of the observation vector or matrix 1105. As a result of this arrangement, the subregions 1106a-1106n generated by the mask effectively correspond to half the field of view of the observation vector or matrix 1105. Increasing the size of the mask increases the portion of the field of view retained by the subregions 1106a-1106n, while decreasing the size of the mask decreases the portion of the field of view retained by the subregions 1106a-1106n. Comparisons between larger subregions are more reliable because there are more pixels in each subregion to base the comparison on. However, smaller subregions have the advantage of being able to compare relatively limited details of the scene captured in the original image 1101. In other words, smaller subregions may be less affected by more extensive changes in the environment. Therefore, smaller subregions may be used to improve the scope over which comparisons between stored data and current data can be made. In particular, as the agent moves away from an object or other attributes, the object appears smaller and is ultimately found in a smaller region of pixels in the original image 1101 compared to when the object is closer to the agent. Similarly, as the agent moves away, the surroundings of the object may appear to change relatively dramatically in the original image 1101. Using smaller subregions (e.g., tailored to the expected size of such an object at a particular distance) allows comparisons between subregions to locate the object even when neighboring pixels in the observation vector or matrix 1105 change significantly with distance from the object. This example shows that there is a balance between selecting large or small subregions depending on the size of the observation vector or matrix 1105. Both of these options have advantages. Even when larger subregions are used, comparison methods such as the observation-action function remain robust with respect to changes in distance from the target / specific object.

[0354] As explained above, the shifting of the mask in each iteration to obtain subregions relative to the observation vector or matrix 1105 can also be varied. This is equivalent to determining the number of subregions to obtain. In one example, the mask may be shifted by one column per iteration. That is, each obtained subregion is reordered by one column compared to the previous subregion. This is the finest possible difference between subregions. In the example of a 16x4 observation matrix above, 16 subregions can be obtained in this manner (the maximum number of subregions possible for a 16x4 matrix). However, it is also possible to permute the mask by eight columns to obtain two subregions, or by permute two columns in each iteration to obtain eight subregions. The number of subregions obtained is arbitrary, but as will be discussed later, it is advantageous to obtain four or more to increase the reliability of the action vector formed from the subregion comparison. This variable again balances reliability (more subregions to compare) with computational load (more subregions to compare requires more calculations). Therefore, the choice of the number of subregions depends on the needs and requirements of the particular use case.

[0355] When applying a mask to a normalized observation vector to obtain subregions (effectively subvectors), the vector loses its normalization. Therefore, in addition to the above process, the subregions (and in particular the subvectors corresponding to these subregions) are renormalized.

[0356] Before describing the comparison of the current (first) sensor data sub-region with the stored (second) sensor data sub-region in more detail with respect to the action function, the recording and storage of the observation vector or matrix 1105 will first be described.

[0357] It should be appreciated that the sub-regions do not need to be stored and maintained by the agent or memory associated with the agent, but are only needed when a comparison is computed, for example using an observation action function, and thus the sub-regions can be generated from the observation vector or matrix 1105 as needed.

[0358] Several options are possible regarding the recording and storage of the observation vector or matrix. As explained in the fourth step above, the observation vector or matrix 1105 is a coarse representation of the original image 1101, providing information from the entire field of view represented in the original image 1101, but at a reduced size. As part of the process of forming the observation vector or matrix 1105, slicing is performed which effectively discards rows and columns of the filtered image 1103 to reduce its size. However, for purposes of performing a comparison between the current observation vector and the stored observation vector, it is advantageous that the information removed by these slicing operations is not lost. Various options for storing this data are possible for the stored (second) sensor data and the current (first) sensor data.

[0359] In the first option, when saving observations from an original image 1101 using the above process, multiple observation vectors or matrices 1105 are saved for each original image 1101. The multiple observation vectors or matrices 1105 are obtained by cutting out different columns in the fourth step above. In the above example, the original image 1101 is a 256x64 pixel image. This image is filtered and sliced ​​to provide 16x4 observation vectors. Assuming that the columns that survive the slicing operation are evenly distributed in the original image 1101, there will be 16 16x4 observation vectors for each channel along which the original image 1101 is obtained (256 / 16=16). These different observation vectors are obtained by permuting the columns of the slicing operation, one column at a time. 17 番目の In the permutation, the observation vector is 1 番目のThe resulting sequence is the same as a permutation. For example, if a first observation vector is generated from columns 1, 17, 33, 49, 65, 81, 97, 113, 129, 145, 161, 177, 193, 209, 225, and 241 of the original image 1101, then a second observation vector is generated from columns 2, 18, 34, 50, 66, 82, 98, 114, 130, 146, 162, 178, 194, 210, 226, and 242. This process can be repeated until the first column of the observation vector corresponds to column 17 of the original image 1101. In this case, this observation vector is simply a rotation of the first observation vector. Therefore, in this example, a set of 16 observation vectors is required to ensure that all columns of the original image 1101 are retained.

[0360] Additionally, each of the 16 observation vectors (each obtained by slicing from a different column) may be permuted up to 16 times to move data within the observation vector to different columns. This is effectively a rotation of the observation vector (e.g., data originally in column 1 is permuted to column 2, data in column 16 is permuted to column 1, etc.). Thus, there are 16 possible rotations of each observation vector, and 16 different observation vectors can be formed by varying the slicing operation. Thus, in total, there are 256 possible observation vectors from the 256x64 original image 1101.

[0361] In the first option above, a set of 256 observation vectors or matrices is stored for the stored sensor data. When comparing to a current observation (from the current sensor data), the corresponding set of 256 current observation vectors can be compared to the set of stored observation vectors. Using the same process of finding the set of observation vectors for the current and stored original images, for example, would result in comparing the set of 256 stored 16x4 observation vectors with the current set of 256 16x4 observation vectors. This improves comparison accuracy but is computationally less efficient due to the need to calculate many vector dot products. Storing and comparing to the complete set of stored observation vectors also means that when performing a comparison with the current observation vector, an additional step is required: selecting the best comparison result from the complete set of stored observation vectors. Therefore, comparing to the complete set of stored observation vectors introduces an additional layer or step into the comparison process, further increasing the computational load. However, obtaining the best comparison for one of the complete set of stored observation vectors may yield more accurate and useful results.

[0362] In the second option, when saving observation vectors, only one observation vector or matrix needs to be saved. This saved observation vector is sometimes called the zeroth observation vector, and in the above example, it could be the observation vector obtained from columns 1, 17, 33, 49, 65, 81, 97, 113, 129, 145, 161, 177, 193, 209, 225, and 241 of the original image. The image itself and other possible observation vectors are not saved. This means that the details of the original image 1101 are essentially lost, and the only data available to the agent about the original image 1101 is the coarse representation provided by the zeroth 16x4x4 observation vector. This is acceptable because when comparing with the current observation (which may not be saved in persistent memory but instead be recorded on the fly or in volatile memory, for example), the saved zeroth observation vector is compared with each possible permutation (e.g., a set of 256) of the current observation vector. Thus, the saved observation vector is compared with the set of 256 16x4x4 current observation vectors. That is, the 0th stored observation vector is stored for all data acquired in the current observation (all data provided by the current 256x64 original image). This ensures that if any of the current observation data is similar to the 0th stored observation vector, such similarity should be detected by comparing the complete set of the 0th stored observation vector with the current observation vector.

[0363] The second option is computationally efficient compared to the first option above, since only one observation vector needs to be stored for a given original image in order to compare it with the new, current data. Furthermore, the second option does not require the additional step of determining which stored observation vector (out of the full set) to use when performing the comparison with the current observation vector. This is particularly advantageous when the agent or a computer associated with the agent needs to store hundreds or thousands of observation vectors associated with various locations, stored objects, or other attributes.

[0364] According to the second option, when comparing observations, a saved observation vector is compared to a set of current observation vectors, which are each possible permutation of the current original image. When using an observation action function, as described in more detail below, a subregion of a saved observation vector (i.e., a saved subvector) is compared to a subregion of a set of current observation vectors (i.e., a current subvector). As shown in FIG. 11, in an example where the saved and current observation vectors are formed from a 16x4 second reduced-format image 1105 (for each of the four channels), there are 256 16x4 current observation vectors, and each current observation vector may be used to form a subregion (or current subvector). This is done by processing each current observation vector through a mask used to generate the subregions.

[0365] This means there are a total of 256 possible subregions for the current observation vector to compare with the 16 possible subregions for the saved observation vector. Figure 12 illustrates this possibility, with a spatial graph 1200 showing the number and placement of subregions 1201 around an agent 1202. It should be understood that Figure 12 is not to scale and is for illustrative purposes only. Each arm 1203 of the graph 1200 represents a subregion obtained from the same mask position relative to the current observation vector. Each subregion 1201 moving along each arm 1203 is obtained from a different one of the current set of observations (divided into different slices). Each concentric set 1204 of subregions on the graph 1200 is from the same current observation vector, but corresponds to a different orientation in space because the mask position used to generate the subregion is different (or reordered) for each subregion in the concentric set 1204, or because the current observation vector itself is rotated before extracting the subregion. While shown as rectangles in Figure 12, it should be understood that each subregion includes a portion of the 360-degree view around the agent, which portion depends on the size of the mask as explained above. In one example, each subregion corresponds to half of the field of view corresponding to the current original image to which this data pertains. The subregions overlap each other.

[0366] In the observation action function, the current sub-region (from the current first sensor data) is compared to a set of sub-regions derived from the stored observation vector. From the current set of sub-regions, a best match is found for each stored sub-region, and an offset angle and magnitude are determined for each best match. This process is described in further detail herein with respect to the observation action function.

[0367] As described above, the stored observation vector is divided into N subregions (also called "stored" subregions), and for each of the N stored subregions, a comparison is performed with the full set of subregions of the current observation vector (also called "current" subregions), comparing all possible rotation slice permutations (e.g., 256 permutations for a 16x4 observation vector) to identify the permutation that is most similar to the stored subregion (also called the best matching current subregion). The most similar identified permutation may include, for example, 120 番目の The function has an index, such as a rotation, that is used to identify an offset relative to the stored subregion. For each stored subregion, an offset is calculated, and optionally a similarity measure is calculated. This measure indicates the similarity between the stored subregion and the best-matching current subregion. The similarity measure is determined by performing the comparison itself. The comparison combines the observation similarity function and the rotation function described above. Specifically, the scalar product of the stored subregion with each of the set of current subregions is calculated to determine the best-matching current subregion. Due to subregion normalization, the scalar product is maximized (1) when the vectors are aligned (i.e., the relative difference between pixels is the same), zero (0) when the vectors are orthogonal, and minimized (-1) when the vectors are opposite. Thus, this comparison returns a value in the range [-1,1] and the index of the location of the best-matching current subregion. This associates an offset and a magnitude with the current subregion and / or the stored subregion.

[0368] FIG. 13A shows a schematic diagram of a comparison of subregions. FIG. 13A shows an agent 1301. The agent 1301 is positioned at the center of a "current observation" cylindrical projection 1302, which includes a green feature channel projection 1302a and a red feature channel projection 1302b. The current cylindrical projection 1302 is an example spatial projection of the current observation vector, illustrating how the current observation vector describes the environment around the agent's 1301's first location. In particular, the current observation vector describes how the actual scene 1303 (i.e., the world) around the agent 1301 at the first location appears in the sensor data. FIG. 13A also shows a similar depiction of a "saved observation" cylindrical projection 1302' of the agent's surroundings, which includes a green feature channel projection 1302a' and a red feature channel projection 1302b'. The cylindrical projection 1302' is an example spatial projection of the saved observation vector, illustrating how the saved observation vector describes the environment around the agent's 1301's second location. In Figure 13A, the projections of the saved observation vectors and the current observation vector are divided into sub-regions. In particular, Figure 13A shows a first (forward) sub-region 1304a for each of the "current observation" cylindrical projection 1302 and the "saved observation" cylindrical projection 1302, and a second (backward) sub-region 1304b for each of the "current observation" cylindrical projection 1302 and the "saved observation" cylindrical projection 1302. In other words, Figure 13A shows forward saved and current sub-regions 1304a and backward saved and current sub-regions 1304b. It will be understood that this is exemplary, and that in practice the current sub-regions 1304a and 1304b are a sample of the set of all possible permutations of the current sub-regions used for comparison.

[0369] The set of saved front and current sub-regions 1304a are compared to each other using an observation similarity function and a rotation function to obtain a scalar product and offset. The set of saved back and current sub-regions 1304a are also compared to each other using an observation similarity function and a rotation function to obtain a scalar product and offset. These operations provide the sub-region offset angles 1305a and 1305b, respectively. These operations result in a measure M that indicates the similarity between the saved front sub-regions 1304a and the current sub-region 1304a. F Also provided is a measure MB indicating the similarity between the previous saved sub-region 1304b and the current sub-region 1304b.

[0370] Figure 13B shows the same configuration of agent 1301. The only difference in Figure 13B is that the saved and current sub-regions being compared are not the front and back sub-regions, but rather the right current and saved sub-regions 1304c and the left current and saved sub-regions 1304d. The comparison process is otherwise the same: the complete set of right saved and current sub-regions 1304c and 1304d are compared to each other using an observation similarity function and a rotation function to obtain a scalar product and offset. The complete set of left saved and current sub-regions 1304d are also compared to each other using an observation similarity function and a rotation function to obtain a scalar product and offset. These operations provide sub-region offset angles 1305c and 1305d, respectively. These operations also provide a measure MR indicating the similarity between the right-hand conserved subregion 1304c and the current best-matching subregion, and a measure ML indicating the similarity between the left-hand conserved subregion 1304d and the current best-matching subregion.

[0371] While Figures 13A and 13B illustrate saved and current sub-regions for the front, back, right, and left sides, respectively, and the offset angle determined from their comparison, it should be understood that any number of saved sub-regions may correspond to any portion of the overall saved observation vector, and therefore direction. Figures 13A and 13B illustrate sub-regions that are substantially half the size of the observation vector and therefore contain sensor data representing half the field of view associated with the observation vector. Thus, in the example of the 16x4x4 original observation vector above, these sub-regions correspond to an 8x4x4 field of view centered in different directions (front, back, right, left). The sub-regions overlap one another. In alternative embodiments, the sub-regions may represent smaller or larger portions of the observation vector.

[0372] Sub-region offset angles 1304a, 1304b, 1304c, and 1304d are measured relative to the axis corresponding to the overall best matching current observation vector, it being understood that the determination of the overall best matching current observation vector can be determined by a similar comparison process performed before or after the sub-region comparison described above.

[0373] The overall best-matching current observation vector is determined in the same way as comparing the set of saved subregions with the current subregion. The only difference is that rather than performing a subregion comparison, the entire saved observation vector is compared against the set of permutations of the current observation vector (including rotation and slice permutations). The full saved observation vector (e.g., a 16x4x4 vector) is compared with each permutation of the current observation vector using the observation similarity function and observation rotation function to determine the best-matching current observation vector. These functions are calculated using an index (e.g., 120) corresponding to the offset angle of the best-matching observation vector, as explained above. 番目の The rotational permutation provides an associated measure of the similarity between the stored observation vector and the best matching observation vector.

[0374] FIG. 14 shows a schematic diagram of comparing a complete stored observation vector with each of a set of current observation vectors to determine the best-matching current observation vector. FIG. 14 shows an agent 1401 at a first location. The agent 1401 is positioned at the center of a cylindrical projection 1402 that includes a green feature channel projection 1402a and a red feature channel projection 1402b. The cylindrical projection 1402 is an exemplary spatial projection of the current observation vector and illustrates how the current observation vector describes the environment surrounding the agent 1401 at the first location. In particular, the current observation vector represents how the actual scene 1403 (i.e., the world) appears around the agent 1401 at the first location. While one current observation vector projection is shown, it should be understood that in practice, the comparison is performed on the entire set of current observation vectors. Also shown in FIG. 14 is a similar depiction of an agent 1401' at a second location, along with a cylindrical projection 1402' that includes a green feature channel projection 1402a' and a red feature channel projection 1402b'. Cylindrical projection 1402' is an example spatial projection of the saved observation vector, showing how the saved observation vector describes the environment around the second position of agent 1401'. In this example, both the current observation vector and the saved observation vector are shown to consist of only two channels, although it should be understood that there may be more channels, such as four channels as described above.

[0375] The stored observation vector, as indicated by the stored cylindrical projection 1402', is compared to each possible rotation and slice permutation of the current observation vector, and the best match is found using the observation similarity function and observation rotation function described above. The observation similarity function returns a measure of similarity between a particular current observation vector and a stored observation vector. The highest measure of similarity between a particular current observation vector and a stored observation vector (among the set of current observation vectors) is recorded, along with the index of the associated particular current observation vector. This index provides the best-matching direction 1405. This allows the best-matching observation vector offset angle 1404 to be determined for the current observation vector that best matches the stored observation vector. In other words, the best match of a stored observation vector in the set of current observation vectors is determined by comparing the stored observation vector with each current observation vector using a scalar product. The best-matching index is recorded, from which the best-matching observation offset angle 1404 is inferred.

[0376] This best-matching observation vector offset angle 1404 indicates the rotation that agent 1401 can perform at a first location to align itself with the best match to the stored observation vector. In the case of Figure 14, assuming the stored observation vector was recorded at coordinates (2,0) and agent 1401 has moved forward from its second (original) position when it was at 1401', the current observation vector represented by projection 1402 will rotate to the left, resulting in the current observation vector at best-matching observation vector offset angle 1404 (measured as a positive counterclockwise rotation from a forward orientation) being the closest match to the stored observation vector.

[0377] It should be appreciated that because the agent 1401 can record observations of a wide field of view (e.g., 360 degrees), it is not necessary to perform a rotation at this point. Instead, a simple readjustment of the agent's axes (frame of reference) is performed by the best-matching observation vector offset angle 1404 so that the best-matching direction 1405 is aligned with the agent's axes for purposes of comparing sub-regions. This can be performed before, during, or after the sub-region comparison operation. Because sub-regions are generated based on specific directions (and do not represent the entire field of view), aligning the agent's axes is important to ensure that the sub-regions (e.g., forward, backward) of the stored observation vectors are properly oriented (by adjusting their axes by the best-matching observation vector offset angle 1404) so ​​that when an action vector is determined, it is determined based on the offset of the sub-region relative to the adjusted axes.

[0378] Figures 15A-15D show the offset angles of each subregion relative to the axes of agent 1301. In each of Figures 15A-15B, an axis 1310 is shown for agent 1301. Axis 1310 represents the realigned axis, and axis 1310 is realigned based on the offset angle 1404 of the best-matching observation vector obtained from a comparison of a saved observation vector with the best-matching current observation vector. Figure 15A also shows subregion offset angles 1304a and 1304b associated with the forward and backward subregion comparisons, respectively. Similarly, Figure 15B shows subregion offset angles 1304c and 1304d associated with the right and left subregion comparisons, respectively. Once the offsets for each subregion have been determined in this manner, the direction of the action vector is determined. To determine the direction of the action vector, a result vector is calculated. This process is illustrated in Figures 15A and 15B. In Figure 15A, an opposite pair of front sub-region offset 1304a and back sub-region offset 1304b is used to calculate a front / back resultant vector 1306a, and in Figure 15B, an opposite pair of right sub-region offset 1304c and left sub-region offset 1304d is used to calculate a right / left resultant vector 1306b.

[0379] The front-to-back resultant vector 1306a and the right-to-left resultant vector 1306b are then aggregated to form a final action vector 1308a, as shown in Figure 15C.

[0380] While Figures 15A and 15B are intended to illustrate the process of defining front / back and right / left result vectors using opposite pairs of sub-region comparisons, it should be understood that in practice, forming result vectors is not required; instead, action vector 1308a may be calculated directly from the aggregation of all sub-region offset angles 1304a through 1304d. Furthermore, additional sub-regions centered in different directions relative to the alignment axis of agent 1301 may also be used. For example, Figure 15D shows a similar diagram to Figure 15C, but with respect to sub-regions compared at 45-degree angles relative to the front / back and left / right directions. Aggregating the sub-region offset angles of these different sub-regions provides action vector 1308b. It should be understood that these processes provide the directional components of action vectors 1308a and 1308b.

[0381] The direction of action vector 1308a determined from the front-to-back and left-to-right sub-region comparisons may be aggregated with the direction of action vector 1308b determined from additional sub-region comparisons. In the example of a 16x4x4 observation vector, there may be up to 16 sub-regions contributing to the determination of the action vector in this manner. The direction of action vector 1308a and the direction of action vector 1308b may be aggregated in any suitable manner to provide a final action vector. In one example, a vector average is calculated to determine the final action vector. Alternatively, the final action vector may be selected as either action vector 1308a or 1308b depending on one or more parameters.

[0382] The magnitude of the final action vector is determined separately from the magnitude calculated in the process of performing the observation similarity function to identify the current subregion that best matches the best-matching observation vector. The magnitude of the final action vector is based on the size and direction of the offset angles 1304a-1304d. As described above, the offset angles 1304a-1304d are determined relative to the axis 1310 that corresponds to the best-matching direction 1405. The magnitude of the action vector is determined in components. In particular, the first component is the anterior-posterior resultant vector 1306a. This resultant vector 1306a is the component of the action vector in a direction perpendicular to the anterior-posterior direction of the axis 1310, as shown in FIG. 15A. The magnitude of this resultant vector 1306a is determined based on the magnitudes of the offset angles 1304a and 1304b. The offset angles 1304a-1304b may be measured in discrete steps based on a rotation index (e.g., index 4 of 256). The magnitude can be determined by averaging the size of offset angles 1304a and 1304b (taking the average offset angle in discrete steps of 1 / 256). This magnitude is multiplied by a constant k, which can be changed based on the specific scenario requirements, where k can be any positive number such as 2, 4, 6, 8, 10, or any other number.

[0383] A further aspect of determining the magnitude may include using an additional constant to define an upper limit on the use of a particular offset angle pair. In some cases, a large offset angle may indicate a false match or an unreliable match between subregions. To prevent such matches from being used in determining , a constant j can be set such that if the average offset angle between opposing pairs is greater than j, the magnitude is set to 0. The constant j is configurable and can take any value, such as 2, 5, 10, 20, 50, etc.

[0384] The magnitude of resultant vector 1306b is similarly calculated with respect to offset angles 1304c and 1304d, providing additional orthogonal components of the action vector, as shown in Figure 15B. The resulting vectors 1306a and 1306b are added to determine the action vector.

[0385] If both the front-to-back or left-to-right offset angles are zero (i.e., the center of the best-matched subregion of the current observation vector is the same as the subregion of the stored observation), the magnitude of the associated orthogonal component is zero. The magnitude follows a relationship where the larger the offset from the stored subregion, the larger the magnitude. Such magnitudes are determined for opposing pairs of subregion offsets, such as a pair of offset angles including a front offset angle 1304a and a rear offset angle 1304b, or a pair including a right offset angle 1304c and a left offset angle 1304d. The magnitudes are then appropriately combined as part of the vector average determination when forming action vectors 1308a and 1308b and the final action vector. Thus, the magnitude of each opposing subregion offset pair provides a weighting to the final action vector based on the size of the offset angle compared to the stored subregion. This essentially means that the action vector is weighted more heavily from result vectors 1306a and 1306b arising from larger offsets (offset angles further from the aligned axis). This provides a final action vector pointing in the likely direction of where the location in space where the stored observation was made can be found, along with a magnitude indicating the distance or effort required to reach this location. The magnitude is also inversely proportional to the confidence that the location associated with the stored observation is in the direction indicated by the final action vector. Thus, a larger value for the final action vector indicates more effort required to reach the location and also indicates less certainty that the location is in the indicated direction. Conversely, a lower magnitude indicates less effort required to reach the location and more certainty that the location is in the indicated direction.

[0386] The magnitude of is determined based on the offset of the opposite pair of the best matching subregions, but also on the similarity obtained by comparing the conserved subregion with the current subregion. の Sizes: ML, MR, MF 、The MB may still be used in the process to calculate the action vector. In particular, weighting may be applied to the magnitude of the action vector based on the magnitude of the similarity between the opposing subregion pairs used to calculate the action vector. Opposing subregions with high similarity may be weighted to increase the contribution of the action vector magnitude from a particular opposing subregion, while subregions with low similarity may be weighted to decrease the contribution of the action vector magnitude.

[0387] Additionally, the similarity measures ML, MR, MF, and MB may be used in a thresholding process prior to determining the action vector to filter out or exclude subregions from determining the action vector. In particular, the similarity measure of each subregion comparison may be compared to a definable confidence threshold (e.g., 0, 0.1, 0.5, or any other numeric value). If the similarity measure does not match or exceed the confidence threshold, the corresponding subregion comparison is not used in determining the action vector. As noted above, the similarity measure M is used in determining the action vector. L , M.R., M. f It should be understood that similarities and sub-region directions other than MB may be used.

[0388] The final action vector can be used alone or in combination with the other functions and methods described above to determine how to move the agent toward the stored observation that corresponds to the stored observation vector. For example, the action vector may be used in a movement function to relocalize the agent to the predicted observation, or it may be used in addition to displacement data or a relocalization method, as shown in Figure 7.

[0389] Observation action functions are a useful tool for navigation because they provide an action vector that an agent can move along to reach a destination indicated by a stored observation vector. By storing multiple observations of a particular space, the agent can navigate the space by repeatedly using the action function on the multiple stored observations associated with that space. Action functions are relatively tolerant to small environmental changes because multiple subregions are used to determine the action vector, and the subregions themselves are preprocessed to reduce the sensor data information to its most basic components (edges, distinct features, etc.).

[0390] The observation action function may also be used in combination with existing navigation methods in a hybrid navigation process. For example, the observation action function process may form a primary navigation process used in combination with a secondary navigation process, such as a positioning system. Positioning systems include, for example, the Global Positioning System (GPS), a satellite-based radio navigation system, and other positioning systems such as the Local Position System (LPS).

[0391] In this hybrid navigation process, a specific location in the real environment forms a goal location. The goal location can be described by coordinates, etc., and goal observations. The goal observations may correspond to sensor data of environmental features that indicate the goal location. Navigation to the goal location is accomplished using a combination of a primary navigation process (observation action functions) and a secondary navigation process (positioning system). There are various ways in which these two processes or systems can be combined. In some embodiments, the relationship between the primary and secondary navigation processes can be mutually exclusive, meaning that only one process performs navigation at a given time. In alternative embodiments, the primary and secondary navigation processes work in conjunction, with each process's impact on the overall navigation adjusted based on one or more factors.

[0392] Figure 16 shows a flow diagram of a method 1600 for navigating using a hybrid navigation process, which will now be described.

[0393] In a first step 1602, a target location is determined. The target location is a destination within the world to which the agent will travel. The target location may be input by a user or other entity as part of an adjacency process. The target location may be input as any data that indicates an actual location in the world. For example, the target location may be input as text data, location (coordinate) data, etc.

[0394] In a second step 1604, target location data corresponding to the target location is obtained. The target location data includes at least primary data and secondary data for a primary navigation process and a secondary navigation process. While the primary navigation process is exemplified here as an observation action function, it should be understood that any of the navigation methods previously described (e.g., using observations and displacements) can be used as the primary navigation process. The secondary navigation process is an existing navigation process, such as using a GPS. The primary and secondary data are obtained in the format required as input to the particular primary and secondary navigation processes, respectively. In the observation action function and GPS example, the primary data are observations or observation vectors, and the secondary data are GPS coordinates.

[0395] In a third step 1606, the agent navigates to the goal location using the goal location data according to a combination scheme that determines how the primary and secondary navigation processes are used throughout the overall navigation process.

[0396] Method 1600 provides a hybrid approach that offers the benefits of both primary and secondary navigation processes. In particular, a secondary navigation process using a positioning system such as GPS is useful for navigating to a desired location because it is typically accurate to within 2 meters and available in most locations around the world. Thus, using GPS offers the advantage of being able to move an agent to within a few meters (or more conservatively, within 50 meters) of a target location. However, existing navigation systems such as GPS do not work well inside buildings, underground, or in the air. Furthermore, while GPS is reasonably accurate, it is not precise enough to guide an agent around obstacles, navigate paths, or navigate through building fixtures such as doors and windows. This is because such features are typically less accurate than GPS.

[0397] On the other hand, primary navigation processes, such as observation action functions, have the advantage of not being affected in the same way as GPS when navigating inside or under buildings, for example. Because observation action functions rely on spatial data acquired locally by the agent, satellite signals are not required, and therefore observation action functions can be used in signal dead zones where GPS is unavailable. Furthermore, by using spatial data (e.g., images), agents can navigate their environment according to environmental constraints such as obstacles and other features such as doors, open spaces, and windows when using observation action functions.

[0398] A hybrid approach combining GPS and observation action functions allows an agent to utilize each navigation process at the most effective times, realizing the benefits of both while negating their drawbacks. An agent may use GPS to navigate to a destination location and then switch to observation action functions to complete the navigation if the GPS is unreliable and cannot be used for the final, more complex navigation steps. For example, if the agent is a delivery robot, the agent may use GPS-based navigation to travel most of the distance from the origin to the destination. However, upon arriving at the intended delivery location (e.g., coordinates or a building address), the agent may switch to using observation action functions to deliver the item to the building door or navigate within the building to make the delivery (if GPS is unavailable).

[0399] The combination scheme determines how the primary and secondary navigation processes interact in the agent's overall navigation process.

[0400] In some embodiments, the combination scheme is a mutually exclusive scheme, in which an agent navigates using either the primary navigation process or the secondary navigation process at a particular time, but not both simultaneously. One or more switching conditions are examined during the overall navigation process to determine when to switch from the primary navigation process to the secondary navigation process, or vice versa.

[0401] The switch condition for switching to the primary navigation process (observation action function) may include a definable confidence threshold that is verified against the results of a comparison between the current observation and a stored observation corresponding to the target location (primary data). In particular, an observation similarity function may be periodically executed to determine the current observation vector that best matches the stored observation vector corresponding to the target location. This function returns a magnitude indicating the similarity between the best-matching current observation vector and the stored observation vector. The magnitude is then compared to a definable confidence threshold. If the magnitude meets or exceeds the definable confidence threshold, this particular switch condition may result in a switch from the secondary navigation process to the primary navigation process.

[0402] In practice, this means that if the current observation is sufficiently similar to the saved observation corresponding to the goal location, the primary navigation process, such as the observation-action function, becomes the active method of the overall navigation process. The comparison essentially involves performing an observation to obtain a current observation vector and comparing it with the saved observation vector corresponding to the goal location. In this regard, the comparison can be considered a type of “transition observation,” as described with reference to FIG. 7 , and can be performed periodically throughout the navigation process or at specific times or locations. In particular, as shown by the transition observation 701-2 in the first graph 701 of FIG. 7 , the transition observation to verify the switching condition can be performed frequently and periodically throughout the navigation process. Alternatively, the transition observation 702-2 to verify the switching condition can be performed at or near the expiration of the path 702-3 navigated according to the secondary navigation process. In other words, the transition observation can be performed in response to passing a proximity threshold, which indicates that the agent is within the vicinity of the goal location according to the secondary navigation process.

[0403] In one example, an agent is provided with goal location data that includes primary data, which is a stored observation vector indicating the goal location, and secondary data, such as coordinates, indicating the goal location. In this example, the secondary data may indicate that the goal location is 10 kilometers away from the agent's current location. In this example, the agent may navigate using the secondary navigation process until the goal location is within a proximity threshold. The proximity threshold may be set as a percentage of the overall path length (e.g., 95%, 99%, etc.) or a specific distance (e.g., within 100 meters of the goal location, within 10 meters of the goal location, within 5 meters of the goal location, etc.). Once the proximity threshold is exceeded, a transition observation is made to determine whether a definable confidence threshold is met (a switching condition). In particular, once the proximity threshold is passed, the switching condition may be directly verified by the transition observation. If the switching condition is met, the agent switches from navigation according to the secondary navigation process to navigation according to the first navigation process. If the switching condition is not met, the agent continues navigation using the secondary navigation process. In this manner, there may be multiple proximity thresholds. For example, the first proximity threshold is within 100 meters of the target location, the second proximity threshold is within 10 meters of the target location, the third proximity threshold is within 2 meters of the target location, and so on. Each proximity threshold results in a transition observation to verify the switching condition. Subsequent proximity thresholds are only required if the switching condition is not met. In this example, if the first proximity threshold is met, and the comparison with the current observation 100 meters away does not yield a magnitude of similarity that meets a definable confidence threshold (the switching condition), the agent continues to navigate according to the secondary navigation process until the second proximity threshold (10 meters) of the target location is met, at which point the process is repeated. If necessary (if the switching condition is not yet met), the agent continues to navigate using the secondary navigation process until the third proximity threshold is met.If the agent continues to navigate using the secondary navigation process until the secondary navigation process indicates that the goal location has been reached, the secondary navigation process terminates. Optionally, further transition observation is performed at this point to determine whether the primary navigation process needs to make adjustments to find the goal location. If the switch condition is met, the agent switches to the primary navigation process and continues navigation to the goal location. If the switch condition is not met, the agent can optionally be configured to search the local area using the primary navigation process or send an error message to the remote control hub to alert the user that the agent has not successfully reached the goal location. In another embodiment, the agent may be configured to return to a previously visited location using the secondary navigation process. In that case, the agent may be configured to re-approach a location where the switch condition is expected to be met. This effectively corresponds to a "try again" procedure. Alternatively, the agent may be configured to backtrack to a previously visited location and systematically attempt to navigate to one or more observation points using the primary navigation process and the hypothesis and observation / travel methods described above.

[0404] The goal location data may be associated with or linked to one or more pieces of metadata. The metadata may include an availability metric, such as a scalar value or flag, indicating the predicted or predetermined availability of a primary or secondary navigation process at the goal location and / or at a location between the agent's current location and the goal and the location. For example, the availability metric may provide a scalar value indicating the expected presence or strength of a GPS signal for the secondary navigation process. This availability metric may then be used to inform the overall navigation process in determining when to switch from the secondary navigation process to the primary navigation process. As an example, the goal location may be within a building, such as a warehouse, and the availability metric may indicate that a GPS signal is likely not present within a 10-meter radius of the goal location. Thus, the overall navigation process may determine to switch to the primary navigation process at a distance of 10 meters from the goal location. The use of metadata in extending the method is described in more detail below.

[0405] If the secondary navigation process is unavailable (e.g., the agent does not have a GPS signal), the navigation process may automatically switch to the primary navigation process.

[0406] After navigating according to the primary navigation process, observations are made as needed according to the observation action function. These observations may be subject to similar validation to identify whether the reverse-switching condition (switching from the primary navigation process to the secondary navigation process) is met. The reverse-switching condition may include the same definable confidence threshold as the switching condition. If a comparison of the current observation with a saved observation corresponding to the goal location results in a similarity below the definable confidence threshold, the reverse-switching condition is met, the agent switches from the primary navigation process to the secondary navigation process, and the process repeats.

[0407] In both of the above cases, the agent performs transition observations to determine when to switch from the secondary navigation process to the primary navigation process (or vice versa). In addition to transition observations, the agent can also perform initial observations to determine which navigation process to use from its initial position. If the initial observations do not satisfy the switching condition, i.e., the magnitude of the comparison between the initial observations and the stored observations does not meet a definable confidence threshold, then by default the secondary navigation process will be used initially.

[0408] In other embodiments, the combination schemes are not mutually exclusive, allowing input to the overall navigation process from both the primary and secondary navigation processes simultaneously.

[0409] In some embodiments, the first and second navigation processes each provide a navigation vector. A weighting is applied to each navigation vector, and the navigation vectors are combined to form an overall navigation vector. The weighting applied to the navigation vector of the secondary navigation process (e.g., GPS) may be inversely proportional to the distance from the agent's current location to the goal location. The inverse proportional relationship may be defined on a case-by-case basis to meet specific needs. For example, the relationship is exponential, such that at 100 meters away from the goal location, a stronger weighting is applied to the navigation vector of the secondary navigation process, and the agent primarily follows the secondary navigation process. However, at 10, 5, and / or 2 meters, the weighting applied to the navigation vector of the secondary navigation process decreases exponentially.

[0410] The weighting applied to the primary navigation process (e.g., observation-action function) may be proportional to the magnitude of similarity obtained by performing one or more transition observations and comparing the transition observations with a stored observation vector corresponding to the target location. Again, the proportionality may be defined on a case-by-case basis to meet specific needs. The proportionality may be linear, meaning that the weighting applied to the navigation vector (e.g., action vector) of the primary navigation process corresponds directly to the magnitude of similarity between the stored observation vector corresponding to the target location and the transition observation. Alternatively, the relationship may be different, such as an exponential relationship.

[0411] While a goal location can be the endpoint of a navigation process, navigation typically requires the agent to navigate along a path that requires turns, pauses, and so on (e.g., traveling along roads or around or through buildings). Therefore, a goal location may actually include multiple goal locations, which can be considered "checkpoints" along the path from the agent's initial location to the goal endpoint. The navigation process involves iteratively navigating each of the multiple goal locations to the endpoint. It is not necessary to use a primary and secondary navigation process to navigate each goal location. While it may be more efficient to navigate using a secondary navigation process (e.g., GPS), if this is determined to be inaccurate or unreliable, the primary navigation process is used instead. In the delivery robot example above, this means using GPS to navigate intermediate goal locations that effectively map a path from the delivery robot's original location, via roads, and so on, to the vicinity of the goal location that represents the endpoint. The observation action function then takes over to perform the final navigation to the final goal location. This may involve navigating several intermediate goal locations along the path to the final goal location, but typically occurs on a smaller scale than GPS navigation. For example, navigation using an observation action function may involve an agent using a GPS to navigate to the entrance of a building, and then navigating a route within the building to a destination such as a specific room within the building.

[0412] The ability of an agent to navigate using this hybrid method 1600 requires that primary and secondary destination location data be known to or obtainable by the agent. As will be appreciated, the secondary data is obtained by an existing system corresponding to the particular type of secondary navigation process or system being used. For example, in the case of GPS-based navigation, the agent may store or have access to a database or world map containing coordinate data.

[0413] The primary data includes stored observation vectors corresponding to various target locations, which may include, for example, several stored observations related to the interior of a particular building. This forms a stored observation dataset, which must be complete enough that the agent can use it to accurately navigate the space.

[0414] To obtain a dataset, the dataset must first be populated with saved observations, which effectively standardizes the saved observations to have the same format and quality across the dataset. Populating the dataset requires the user to record and save observations in the area of ​​interest through which the agent wants to navigate. While this can be performed by the agent itself and stored on the agent, it is more efficient to use a recording agent to record observations of the target location. The recording agent may be an agent, as described above, that explores the area of ​​interest and takes multiple observations (and displacements) as described above. The saved observations are uploaded to a remotely accessible database. The recording agent may be different from the agent performing the navigation. In another embodiment, the recording agent may be any sensor capable of recording observations of the required quality and format. For example, the recording agent may be a camera, such as the camera on the user's mobile phone. In this embodiment, the user is instructed to capture images of the area of ​​interest at different locations. The images may be panoramic. The images are then uploaded to a remotely accessible database and reformatted as necessary so that the agent can use them to navigate. Multiple remote agents may contribute to this remotely accessible database to build a global map based on their saved observations.

[0415] When an agent is instructed to navigate to a target location, the agent may retrieve target location data for both the primary and secondary navigation processes required for navigation from a remotely accessible database. Retrieval of this information occurs on the fly and is updated as needed, although retrieval preferably includes downloading the required target location data prior to navigation so that a data connection is not required during navigation.

[0416] The above-described hybrid method 1600 for navigating using a combination of primary and secondary navigation processes can be performed by any robot, vehicle, or other device and can be used for a variety of purposes. As such, the method may be computer-implemented and executed by a processor, as is well known.

[0417] In both the hybrid navigation method 1600 and the other methods described above, observation vectors are saved and used for comparison with current or new observations.

[0418] To make the comparison process more efficient and increase the likelihood of obtaining useful results from the comparison, metadata can be used in combination with the stored observation data. The metadata may be recorded by a recording agent when the observation is recorded, or it may be added in a post-processing step.

[0419] The metadata may include multiple different types of data. Some examples are listed below. First, the metadata may include metric information about the physical location where the stored observation was taken. When the recording agent records the observation, coordinate information, etc., may be recorded by an additional sensor. In this case, the additional sensor may be a transceiver for acquiring GPS data. The recorded observation is tagged or associated with metric information metadata. This metadata may be used when applying the hybrid method described above. In particular, when navigating using GPS, the agent or associated computer may select a stored observation based on the "closest" observation in terms of the metric information metadata to compare with the current observation or the observation in transition. In other words, the stored observation vector may be ranked according to a proximity determined from the metric information metadata indicating the location where the stored observation was recorded to the agent's current location. When performing a comparison, the stored observation to be compared may be the highest-ranked observation. If the comparison does not exceed a minimum threshold of similarity, lower-ranked observations may also be compared.

[0420] Linking or associating goal location data with metadata containing physical location metrics, such as linking an observation with coordinate information detailing where the observation was taken, provides the actual location of that observation. This metadata can be used to ascertain an agent's location in the world following a match (by comparison) with an observation. In effect, the metadata in this instance allows the observation match to be converted into physical location metrics, which allows the agent to perform further processing based on its physical location, such as obtaining more nearby observations to navigate to or determining the distance to a goal location.

[0421] As described above, the metadata may additionally include availability metrics, such as scalar values ​​or flags indicating the predicted or predetermined availability of signals, such as GPS signals or data connection signals, at specific locations required for navigation. The availability metrics can be used in determining a route to reach a destination location (e.g., using observations associated with locations with good signal availability in preference to locations with poor signal availability). The availability metrics can also be used to determine when, where, and how much data to download from a remotely accessible database. In particular, if an agent travels to a location indicated as having poor data connection signal availability, the agent may be instructed to download data for these locations, such as stored observation vectors, at another (previous) location where the data connection signal is stronger. As an example, a destination location may be within a warehouse or other building, and availability metrics and data associated with the destination location (neighboring locations contained within the building) may indicate that a data connection signal is unlikely to exist within the building. In this case, the agent may be instructed to download all necessary destination location data (and other data related to neighboring locations within the building) away from the building before entering the building. In this way, the agent can prepare for the possibility that a data connection signal or GPS signal may become unavailable while inside the building.

[0422] Alternatively or additionally, the metadata may include information regarding the variability of one or more portions of the sensor data acquired in recording the observation. In particular, the variability information may specify sensor data pixels, rows of pixels, columns of pixels, or groups thereof that contain environmental features that are likely to vary (and therefore not be constant parts of the environment). Identifying / detecting variable parts of the environment may involve performing classification or recognition tasks from the sensor data or may involve obtaining multiple temporally separated observations of the same location to identify constant (fixed) features of the environment. The classification or recognition tasks may be performed by known image or feature recognition algorithms, for example, relying on machine learning. These tasks may be performed by the recording agent itself, a computing device associated with the recording agent, or a remote computer or server in data communication with the recording agent. These processes identify environmental features known to change, such as roads, vehicles, t...

Claims

1. A computer implementation method for analyzing the environment of an agent in a multidimensional space, At the first location of the agent, first sensor data describing the environment surrounding the agent is acquired. To compare with the first sensor data, the system searches for saved second sensor data that describes the environment surrounding a second location exhibiting features in the multidimensional space, The method involves acquiring a plurality of first sub-regions of the first sensor data, each first sub-region describing a first portion of the environment surrounding the agent at the first location, and each first portion being associated with a first direction from the first location. The method involves obtaining a plurality of second sub-regions of the second sensor data, each second sub-region describing a second portion of the environment surrounding the second location, each second portion being associated with a second direction from the second location, and for each second sub-region, The second sub-region is compared with each first sub-region using a similarity comparison scale to determine the first sub-region that is most similar to the second sub-region. This includes determining the relative rotation between the second direction associated with the second sub-region and the first direction associated with the most similar first sub-region, The above method further, A computer implementation method comprising aggregating the relative rotations of a plurality of second sub-regions to obtain an action vector indicating the estimated direction from the first location of the agent to the second location representing the features of the multidimensional space.

2. The method according to claim 1, wherein the first sensor data and the second sensor data are first and second image data, respectively, each of the first sub-regions includes a contiguous subset of pixels of the first image data, and each of the second sub-regions includes a contiguous subset of pixels of the second image data.

3. The method according to claim 1, wherein the first and second sensor data are arranged in a first vector and a second vector concatenated from the first and second image data, respectively.

4. The method according to claim 1, wherein the similarity measure is the inner product between the first sub-region and the second sub-region.

5. Acquiring the first sensor data is Acquiring the original first sensor data of the agent's environment at the first location, The process involves processing the original first sensor data to reduce the size of the original first sensor data and acquiring the first sensor data so that the first sensor data is in a format that is reduced in size relative to the original first sensor data. Here, the saved second sensor data is also saved in a reduced-size format, so the method further... This includes obtaining original second sensor data of the agent's environment at the second location, The method according to claim 1, comprising processing the original second sensor data to reduce the size of the original second sensor data and acquiring the second sensor data so that the second sensor data is in a reduced size relative to the original second sensor data.

6. The method according to claim 5, wherein processing the original first sensor data and the original second sensor data includes applying one or more filters and / or masks to the original first sensor data and the original second sensor data to reduce the size of each dimension of the original first sensor data and the original second sensor data, respectively.

7. The method according to claim 5, wherein the original first sensor data and the original second sensor data are the original first image data and the original second image data, respectively.

8. Acquiring multiple first sub-regions of the first sensor data is: The method for obtaining a plurality of second subregions of the second sensor data is to iteratively apply a mask to the first sensor data to extract each of the plurality of first subregions, wherein in each iteration, the mask of the first sensor data is sorted by at least one data entry / cell in one dimension of the first sensor data, and the The method according to claim 1, comprising iteratively applying a mask to the second sensor data to extract each of the plurality of second subregions, wherein in each iteration, the mask or the second sensor data is sorted by at least one data entry / cell in one dimension of the second sensor data.

9. The method according to claim 8, wherein there are at least four iterations, each having at least four first subregions and at least four second subregions.

10. The mask is smaller than the first and second sensor data, and the main dimensions of the mask are 50% or less of the size of the corresponding main dimensions of the first and second sensor data, or The method according to claim 8, wherein the size of the corresponding main dimensions of the first and second sensor data is 25% or less.

11. The method according to claim 8, wherein the first and second sensor data are arranged in a first array and a second array, respectively, the first array and the second array have dimensions of X × Y cells, and the mask has dimensions of (Xm) × Y cells, where m is a positive integer.

12. The method according to claim 1, further comprising moving the agent from the first location in accordance with the action vector.

13. After moving the agent to a new position according to the action vector, The process involves acquiring third sensor data describing the environment surrounding the agent at the agent's new location, Acquiring a plurality of third sub-regions of the third sensor data, each third sub-region describing a third portion of the environment surrounding the agent at the third location, each third portion being associated with a third direction from the third location, and for each second sub-region, The second sub-region is compared with each third sub-region using the similarity comparison scale, and the third sub-region that is most similar to the second sub-region is determined. The method includes determining a relative rotation between a second direction associated with the second sub-region and a third direction associated with the most similar third sub-region, The method according to claim 12, further comprising aggregating the relative rotations of the plurality of second sub-regions to obtain an updated action vector indicating the estimated direction from the third location of the agent to the second location representing the multidimensional space features.

14. The method according to claim 1, further comprising confirming the presence of the features of the environment in the multidimensional space if the similarity scale and / or the observed comparison scale associated with the action vector matches or exceeds a confirmation threshold level.

15. The method according to claim 1, wherein determining the relative rotation includes determining the offset angle between the first direction and the second direction.

16. The magnitude of the action vector is determined for the opposing second sub-region pair, Determining the average offset angle from the offset angles associated with the pair of opposing second sub-regions, The method according to claim 15, comprising assigning a size based on the average offset angle, wherein the size of the average offset angle is proportional to the size.

17. The method according to claim 16, further comprising determining the magnitude of each of a plurality of pairs of offset angles associated with a pair of opposing second sub-regions, and aggregating / averaging the magnitudes to form the magnitude of an action vector.

18. The method according to claim 1, wherein the first sensor data represents a field of view of approximately 360 degrees around the agent, and the second sensor data represents a field of view of approximately 360 degrees around the second location, wherein each first portion described by each first sub-region is a part of the 360-degree field of view around the agent, and each second portion described by each second sub-region is a part of the 360-degree field of view around the second location.

19. The method according to claim 1, wherein each of the plurality of first subregions overlaps with at least an adjacent first subregion, and each of the plurality of second subregions overlaps with at least an adjacent second subregion.

20. The characteristics of the aforementioned environment are, A specific location in the aforementioned multidimensional space, The method according to claim 1, wherein the image is an image or a part thereof, or an object or a part thereof.

21. The method according to claim 1, wherein the agent is virtual, the multidimensional space is a two-dimensional or three-dimensional virtual space, and the first sensor data and the second sensor data are acquired using a virtual sensor.

22. The method according to claim 1, wherein the agent is a physical entity, the multidimensional space is a three-dimensional physical space, and the first sensor data is acquired using a physical sensor.

23. The second sensor data describes the environment indicating the target location in the multidimensional space, and the characteristics of the environment described by the second sensor data are associated with the target location, and the method is The method according to claim 22, further comprising moving the agent from the first location to the aforementioned location in accordance with the target location action vector.

24. The second sensor data forms part of the set of the second sensor data, the set includes multiple instances of the second sensor data, each instance describes an environment indicating its respective location in the multidimensional space, and the method is, The agent further includes iteratively navigating from the first location to the target location of each of the multiple instances of the second sensor data, in each iteration, Obtain an action vector for one of the second data instances, and according to the action vector, move from the agent's location to each location indicated by one of the second data instances. The method according to claim 23, wherein the vehicle is moved until it reaches a target location.

25. Navigating the agent from the first location to the target location according to the action vector constitutes the main navigation process, and the method further, Using a secondary navigation process, the agent is navigated from its initial position to the first location in the multidimensional space. The method according to claim 23, further comprising switching from the secondary navigation process to the primary navigation process at or near the first location.

26. The method according to claim 25, wherein the secondary navigation process is configured to use a positioning system.

27. The method according to claim 26, wherein the positioning system is a satellite-based wireless navigation system.

28. The method according to claim 1, wherein the second sensor data is stored in a remotely accessible database.

29. The recording agent records the second sensor data at the second location, The method according to claim 28, further comprising storing the second sensor data in the remotely accessible database.

30. The above method further, The method further includes recording metadata corresponding to the second sensor data and associating the metadata with the second sensor data. The aforementioned metadata is Metric information relating to the second location in the multidimensional space, information relating to the variability of one or more parts of the environment surrounding the second location captured by the second sensor data, Information relating to one or more fixed features of the environment surrounding the second location captured by the second sensor data, The time information relating to the acquisition of the second sensor data, An availability metric indicating the predicted or predetermined availability of the signal at the second location, The information includes at least one of the following: information regarding specific features captured by the second sensor data, The above method further, The method according to claim 29, comprising storing the metadata together with corresponding second sensor data in the remotely accessible database.

31. The metadata includes metric information relating to the second location in the multidimensional space, and the method further, To obtain metric information relating to the first location of the agent in the multidimensional space, Based on the metric information relating to the first location and the metric information relating to the second location, it is determined that the second location is near the first location. The method according to claim 30, further comprising obtaining the second sensor data from the remotely accessible database based on the determination that the second location is near the first location.

32. Acquiring the first sensor data is Acquiring the original first sensor data of the agent's environment at the first location, The process involves processing the original first sensor data to reduce the size of the original first sensor data and acquiring the first sensor data so that the first sensor data is in a format that is reduced in size relative to the original first sensor data. Here, the saved second sensor data is also saved in a reduced-size format, so the method further... This includes obtaining original second sensor data of the agent's environment at the second location, The process includes processing the original second sensor data to reduce the size of the original second sensor data and acquiring the second sensor data so that the second sensor data is in a reduced size format relative to the original second sensor data, The metadata includes information about the variability of one or more parts of the environment surrounding the second location captured by the original second sensor data, and processing the original second sensor data by applying one or more filters and / or masks to the original second sensor data to reduce the size of each dimension of the original second sensor data is The method according to claim 30, comprising filtering or masking one or more portions of the environment surrounding the second location acquired in the original second sensor data, based on the metadata, so that the reduced form of the second sensor data does not include one or more portions of the environment indicated by the metadata information.

33. Acquiring the first sensor data is Acquiring the original first sensor data of the agent's environment at the first location, The process involves processing the original first sensor data to reduce the size of the original first sensor data and acquiring the first sensor data so that the first sensor data is in a format that is reduced in size relative to the original first sensor data. Here, the saved second sensor data is also saved in a reduced-size format, so the method further... This includes obtaining original second sensor data of the agent's environment at the second location, The process includes processing the original second sensor data to reduce the size of the original second sensor data and acquiring the second sensor data so that the second sensor data is in a reduced size format relative to the original second sensor data, The metadata includes information about one or more fixed features of the environment surrounding the second location captured by the original second sensor data, and processing the original second sensor data by applying one or more filters and / or masks to the original second sensor data to reduce the size of each dimension of the original second sensor data is The method according to claim 30, comprising maintaining one or more fixed features of the environment surrounding the second location acquired in the original second sensor data based on the metadata, and ensuring that the reduced form of the second sensor data includes one or more fixed features of the environment indicated by the metadata information.

34. The metadata includes an availability metric indicating the predicted or predetermined availability of positioning system signals and / or data connection signals at the second location. The above method further, Navigating the agent based on the availability metric corresponding to the positioning system signal, and / or: The method according to claim 30, comprising determining when to download information from the remotely accessible database based on the availability metric corresponding to the data connection signal.

35. The recording agent further includes selecting the second location for recording the second sensor data, wherein the selection is The exploration status associated with the second location, The method according to claim 28, wherein the agent makes a determination of whether the second location is passable or impassable, based on location selection criteria including at least one of these criteria.

36. It is a system, Processor and Memory and A sensor or virtual sensor configured to acquire spatial data describing the environment of a multidimensional space in the local vicinity of the sensor or virtual sensor, A system wherein the memory stores instructions, and when executed by the processor, the system performs the method according to any one of claims 1 to 35.

37. The multidimensional space is a physical space, the system is a robot or vehicle equipped with sensors, and the robot or vehicle is an agent. The system according to claim 36, further comprising a controllable mobile module configured to move the robot or vehicle within the physical space.

38. The system according to claim 37, wherein the sensor includes one or more of the following: an ultraviolet imaging device, a camera, a LiDAR sensor, an infrared sensor, a radar, a tactile sensor, or other sensors configured to provide spatial information.

39. The system comprises a plurality of devices, the plurality of devices including the robot or vehicle and additional computing devices having a processor, memory and sensors, The system according to claim 36, wherein the additional computing device is a recording agent, and the system is configured to perform the method described in claim 23.

40. The system according to claim 36, wherein the system is a computer system including the virtual sensor, the computer system is configured to operate in a virtual space, and the agent is represented by a point in the virtual space.

41. A computer program stored on a non-temporary computer-readable medium, which, when executed by a processor, is configured to cause the processor to perform the method described in any one of claims 1 to 35.

42. A computer implementation method for analyzing environments in multidimensional space, The recording agent in the multidimensional space records sensor data describing the environment surrounding the recording agent, The process of the sensor data and identifying one or more features present in the environment captured by the sensor data, This includes associating the metadata with one or more identified features, wherein the features are The location of the recording agent when the sensor data was acquired in the aforementioned multidimensional space, The variability of one or more parts of the environment surrounding the recording agent, The recording agent includes at least one of one or more fixed features of the surrounding environment, The above method further, A computer implementation method comprising storing the sensor data and associated metadata in a remotely accessible database.

43. The method according to claim 42, comprising using a navigation agent in the multidimensional space to retrieve the sensor data and associated metadata stored from the remotely accessible database, and navigating the multidimensional space using the sensor data and associated metadata by comparing the sensor data retrieved by the navigation agent with the stored sensor data.

44. The method according to claim 43, wherein using the sensor data and associated metadata to navigate the multidimensional space further comprises reducing the stored sensor data according to the associated metadata, the reduced stored sensor data comprising sensor data corresponding to invariant or fixed features identified in the environment of the recording agent.

45. Before saving the aforementioned sensor data to the remotely accessible database, The method further includes reducing the sensor data according to the associated metadata, thereby reducing the storage of the sensor data. The method according to claim 42, comprising storing sensor data corresponding to non-variable or fixed features identified in the environment, and discarding sensor data corresponding to variable or non-fixed features identified in the environment.

46. The method according to claim 42, wherein the sensor data forms a panorama around the recording agent.

47. A system including a first computing device and a server, wherein the first computing device is Processor and Memory and A sensor or virtual sensor configured to acquire spatial data describing the environment of a multidimensional space in the local vicinity of the sensor or virtual sensor, The aforementioned server, Memory containing a remotely accessible database, A system comprising a first computing device and a server, the first computing device and the

48. A system further including a robot or vehicle or virtual robot, Processor and Memory and A sensor or virtual sensor configured to acquire spatial data describing the environment of a multidimensional space in the local vicinity of the sensor or virtual sensor, The system according to claim 47, wherein the robot or vehicle or virtual robot is a navigation agent, and the system is configured to perform the method described in claim 43.

Images