Method and system for determining the structure, connectivity and identity of a physical or logical space or its attributes

Abstract

A computer-implemented method and system are provided for analyzing a multidimensional space and exploring attributes of the space based on a series of observations and displacements performed by an agent within the space, the method including identifying hypotheses about attributes of the space and validating the hypotheses by making sequential observations and displacements from a position of the agent within the space, comparing the observations and displacements to a set of stored observations and displacements to obtain an observation comparison measure and / or a displacement comparison measure, and adjusting, maintaining, or confirming the hypotheses based on the observation comparison measure and / or the displacement comparison measure.

Description

[Technical Field]

[0001] A system and computer-implemented method for analyzing a multidimensional space and determining the structure of that space, particularly for navigation in unknown environments and visual classification of objects and images. [Background technology]

[0002] Simultaneous Map and Localization (SLAM) is the problem and process of computationally constructing or updating a map of an unknown environment while simultaneously tracking the position of an agent within that environment. SLAM algorithms are based on concepts from computational geometry and computer vision and are used in robot navigation, robotic mapping, and odometry measurements in virtual or augmented reality.

[0003] SLAM algorithms are also used in autonomous vehicle control to map the unknown environment around the vehicle, and the resulting map information can be used to perform tasks such as path planning and obstacle avoidance.

[0004] Visual SLAM uses an approach consisting of a front-end that processes images from a camera and performs "feature extraction." This feature extraction process detects small regions of an image that meet certain visually distinguishable criteria, such as high local spatial frequency or contrast. These regions can then be saved for the purpose of finding those same regions in subsequent camera images taken at short intervals with the same camera. A camera model that accounts for camera lens distortion is then applied to map the locations of these features to real-world orientations. These are provided to a back-end algorithm called a bundle adjuster. The bundle adjuster compares the predicted location in 3D space of the features based on previous camera images with their new orientation from the current camera image, simultaneously determining the location of the features relative to each other in 3D space and the camera's position relative to the features. By repeating this process, the bundle adjuster tracks the moving camera's position and the location of the feature map over time.

[0005] This approach has several drawbacks that limit its effectiveness in real-world robotics applications. First, feature extraction and bundle adjustment are both computationally expensive processes. While feature extraction can be easily parallelized for efficiency, bundle adjustment is a strictly sequential process.

[0006] Second, bundle adjustment requires that new feature orientations and predicted feature locations converge to a stable solution, which is heavily affected by sensor noise and environmental changes such as wind blowing leaves off a bush. If a feature is incorrectly detected in the wrong location on the camera image, bundle adjustment will not be able to converge to a stable solution, and the system will become "lost" and unable to locate it.

[0007] To address convergence failures in bundle adjusters, the state of the art employs two approaches: one that mitigates the failure and the other that degrades performance if convergence fails, allowing the system to continue.

[0008] The first approach is outlier removal. A set of constraints is used to assess the likelihood that when a feature is found, it is the same as one found previously. For example, if a feature moves from one side of the camera to the other, it is unlikely to be the same unless the camera is moving very fast, and is rejected as a match. If it is close to its previous location, it is more likely to be the same feature and is accepted. Accepted features are passed to the bundle adjuster, while rejected features are not. Outlier removal is another process that is very computationally expensive, and is the largest computational component in some SLMA systems.

[0009] The second approach is visual inertial odometry, a system that uses the features identified by the feature extractor in a different way. A second source of motion information, linear acceleration measurements from an inertial sensor and tracking functions, is used to predict the camera's velocity rather than its position. This velocity measurement is integrated over time and can be used in conjunction with a bundle adjuster to correct for convergence failures. However, integrating VIO causes position errors to accumulate over time, and this approach to SLMA relies on a metric measurement of position, leading to failures at long integration times. Because the same features are used by VIO and bundle adjustment, feature extraction becomes a single point of failure for the entire system.

[0010] Another weakness of the feature extraction front end is that the feature size results in a relatively low dimensionality of the data for a single feature, which increases the likelihood that multiple parts of the image will match the feature, especially in aliased environments such as a hotel hallway where identical doors are evenly spaced. In such environments, the bundle adjuster may converge to the wrong position.

[0011] Loop closure is another weakness of the current technology. When a moving camera traverses a complete loop and returns to the same location from a different direction, the system needs to recognize that the camera has returned to the same location. Because the system is metric, it also needs to ensure that the map matches the same position at the same location. Due to errors accumulated in the bundle adjustment process, this is rarely the case; there is usually a metric error between the representations of the positions at both ends of the loop. The system then needs to go back and adjust all the features stored throughout the loop to correct for and remove this error. This loop closure process is computationally expensive and prone to error.

[0012] The computational complexity of algorithms that can handle such environments and problems is very high, limiting the platforms they can be deployed on. Furthermore, the memory consumption of the generated maps is very large, ranging from hundreds of megabytes to several gigabytes.

[0013] It is therefore appreciated that there is a need for a localization and mapping method and system that allows an agent to understand the structure and identity of a space and the objects therein, that is less prone to error, and that is less computationally and memory intensive. Summary of the Invention

[0014] This summary is provided to introduce some concepts in a simplified form that are further described in the detailed description.

[0015] According to a first aspect, a computer-implemented method for analyzing a multidimensional space is provided. The method explores attributes of the space based on a series of observations and displacements performed by an agent within the space. The method includes: making a first observation from a first location of the agent within the space, where the first observation includes data describing a first portion of the space in a local neighborhood of the first location; comparing the first observation to a set of stored observations and displacements to identify a first stored observation from the set of stored observations and displacements that is most similar to the first observation; determining a hypothesis of an attribute of the space based on the first stored observation, where the first stored observation is associated with the hypothesis; obtaining a hypothesis subset of the set of stored observations and displacements, where the hypothesis subset includes at least the first stored observation, a predicted observation, and a predicted displacement required to arrive at the predicted observation, where the predicted observation and predicted displacement are also associated with the hypothesis; and testing the hypothesis. The hypothesis is verified by moving the agent to a second location in the space based on a movement function, the movement function depending on the predicted displacement of the hypothesis subset, making a second observation from the agent's second location in the space, the second observation including data describing a second portion of the space in a local neighborhood of the second location, comparing the second observation with the predicted observation to obtain an observation comparison measure, and / or comparing the actual displacement between the first location and the second location with the predicted displacement to obtain a displacement comparison measure, and adjusting, maintaining, or confirming the hypothesis based on the observation comparison measure and / or the displacement comparison measure, and confirming the hypothesis includes determining that a hypothesis confirmation condition is met and exploring attributes of the space.

[0016] The movement function includes a first movement component that depends on the predicted displacement of the hypothesis subset and a second movement component that depends on the predicted observation of the hypothesis subset, the first movement component being weighted according to the remaining distance of the predicted displacement, the first movement component weakening as the agent moves along the predicted displacement, and the second movement component being weighted according to the predicted observation, the second movement component becoming stronger as the agent approaches the predicted observation.

[0017] The method may include measuring an actual displacement traveled from a first location to a second location.

[0018] When the agent reaches the end of the predicted displacement or within a displacement threshold distance of the end of the predicted displacement, the first movement component of the movement function is weighted as zero and the first movement component no longer contributes to the movement function.

[0019] The second movement component includes an expected direction and magnitude relative to the predicted observation, the direction and magnitude being obtained by making a transition observation away from the first location after or during the agent's movement and performing a transition comparison of the transition observation with the predicted observation to obtain the direction and magnitude relative to the predicted observation, and the second movement component being weighted according to the transition comparison, the stronger the comparison with the predicted observation, the stronger the weighting of the second movement component.

[0020] The method may include repeatedly updating a second movement component including a direction and a magnitude as the agent moves away from the first location, the method including making a plurality of transition observations.

[0021] The method includes using the movement function to stop the agent at a second location, the second location being a location of one of the transition observations of a second movement component of the movement function, and a transition comparison of one of the transition observations indicating a best match with a predicted observation from the plurality of transition observations, and / or a location where the magnitude of the second movement component function is equal to or below a magnitude threshold. One of the transition observations at the second location can be a second observation, and the transition comparison indicating a best match can be the second observation.

[0022] The observation comparison measure can include one or more of a first observation comparison measure component, a second observation comparison measure component, a third observation comparison measure component, and a fourth observation comparison measure component, where the first observation comparison measure component is a similarity measure indicating the similarity between the second observation and the predicted observation, the second observation comparison measure component is a measure indicating the direction of the predicted observation relative to the second observation, the third observation comparison measure component is a vector pointing from the second observation to the predicted observation, and the fourth observation comparison measure component is a second similarity measure indicating the similarity between the second observation and the predicted observation.

[0023] The observation comparison measure may include first, second, and third observation comparison measure components, which effectively provide a method for determining the similarity between the compared observations, the directional change between the compared observations, and the direction and magnitude (e.g., in the form of a vector) from the agent's location to the location of the compared observation.

[0024] The observational comparison scale may further include a fourth observational comparison scale component, which provides redundancy and can be advantageously used to distinguish the first observational comparison scale component from environmental factors and characteristics.

[0025] The first observation comparison measure component may be formed by an observation similarity function that takes the dot product of the second observation and the predicted observation, where the second observation and the predicted observation are vectors, and in particular unit vectors.

[0026] The third observation comparison measure component can be formed by dividing the predicted observation into a first plurality of sub-regions, obtaining first and second observation comparison measure components for a comparison of the first plurality of sub-regions with a second plurality of sub-regions of the second observation, such that each of the first plurality of sub-regions is associated with a first and second observation comparison measure component, and aggregating the first and second observation comparison measure components to form a vector from the second observation to the predicted observation.

[0027] Adjusting, maintaining, or confirming the hypothesis based on the hypothesis testing may include adjusting the hypothesis when the observed comparison measure is below a first observation threshold level, maintaining the hypothesis when the observed comparison measure is above a first observation threshold level, and confirming the hypothesis when the observed comparison measure is above a hypothesis-confirming observation threshold level (wherein the hypothesis-confirming observation threshold level is a hypothesis-confirming condition), or adjusting the hypothesis when the variance comparison measure is below a first variance threshold level, maintaining the hypothesis when the variance comparison measure is above the first variance threshold level, and confirming the hypothesis when the variance comparison measure is above a hypothesis-confirming threshold level. confirming the hypothesis if the two-dimensional similarity measure exceeds a first two-dimensional similarity measure threshold level (wherein the hypothesis-confirming threshold level is a hypothesis-confirming condition); or combining the observation comparison measure and the displacement comparison measure to form a two-dimensional similarity measure, adjusting the hypothesis if the two-dimensional similarity measure is below a first two-dimensional similarity measure threshold level, maintaining the hypothesis if the two-dimensional similarity measure exceeds the first two-dimensional similarity measure threshold level, and confirming the hypothesis if the two-dimensional similarity measure exceeds a hypothesis-confirming two-dimensional similarity measure threshold level (wherein the hypothesis-confirming two-dimensional similarity measure threshold level is a hypothesis-confirming condition).

[0028] The set of stored observations is associated with a plurality of hypotheses and includes a plurality of hypothesis subsets, each including at least one predicted observation and predicted displacement associated with a particular hypothesis.

[0029] Adjusting the hypothesis may include rejecting the hypothesis, moving the agent from the second location to the first location according to a negative displacement of the actual displacement, selecting a new hypothesis from among the plurality of hypotheses based on a comparison of the first observation with the set of stored observations and displacements, and identifying a next stored observation from the set of stored observations and displacements that is next-similar to the first observation.

[0030] The hypothesis subset may include multiple predicted observations and multiple predicted displacements predicted to be required to travel between the predicted observations.

[0031] Maintaining the hypotheses may include obtaining a next predicted observation and a next predicted displacement from the hypothesis subset, and repeating the testing of the hypotheses with respect to the next predicted observation and the next predicted displacement.

[0032] The method may include repeating the testing of the hypothesis for multiple predicted observations and multiple predicted displacements of the hypothesis subset until the hypothesis is adjusted or confirmed.

[0033] The attributes of the space to be explored are objects or images in the space, the hypotheses indicate predicted objects or images, predicted observations and predicted displacements associated with the hypotheses are associated with the predicted objects or images, and exploring the attributes of the space includes identifying the objects or images in the space as the predicted objects.

[0034] The attribute of the space to be explored is a destination within the space, the hypotheses indicate routes to the destination, predicted observations and predicted displacements associated with the hypotheses are associated with expected routes to the destination, and exploring the attribute of the space includes reaching the destination within the space.

[0035] The space may be physical or logical, and may be two or three dimensional.

[0036] The method further includes extracting a set of low-level spatial features from the physical or logical space, wherein determining a hypothesis based on the set of stored predicted observations and displacements includes processing the first observations and the low-level spatial features to determine the hypothesis, wherein the set of low-level spatial features are assigned weighting coefficients in determining the hypothesis.

[0037] The method may include processing the second observation and / or the nth observation to verify a hypothesis regarding the physical or logical space; extracting a second or nth set of low-level spatial features from the physical or logical space; and processing the second or nth observation and the second or nth set of low-level spatial features to verify the hypothesis, where n is a positive real number.

[0038] According to a second aspect of the present disclosure, there is provided a system including a processor, a memory, and a sensor or virtual sensor configured to acquire spatial data describing a portion of a space in a local vicinity of the sensor or virtual sensor, the memory having instructions stored thereon that, when executed by the processor, cause the system to perform the method outlined in the first aspect above.

[0039] The system is a robot or vehicle equipped with sensors, the robot or vehicle being an agent configured to operate within a physical space, and further comprising a controllable locomotion module configured to move the robot or vehicle within the physical space, the sensors being cameras, LIDAR sensors, infrared sensors, radar, tactile sensors, or other sensors configured to provide spatial information.

[0040] The system is a computer system that includes virtual sensors, the computer system is configured to operate on a logical space, and the agents are represented by points in the logical space.

[0041] The logical space is an image, the points in the logical space are pixels of the image, and the virtual sensor is configured to capture a portion of the image data in the vicinity of the pixel.

[0042] According to a third aspect of the present disclosure, there is provided a computer program stored on a non-transitory computer readable medium, the computer program being configured, when executed by a processor, to cause the processor to perform a method according to the first aspect above.

[0043] According to a fourth aspect of the present disclosure, there is provided a computer-implemented method for analyzing a physical or logical space based on a series of observations and displacements performed by an agent in the physical or logical space, and determining attributes of the physical or logical space.The method includes selecting a hypothesis for an attribute of the physical or logical space based on a comparison of a current observation of the agent in the physical or logical space with a current predicted observation of a stored set of predicted observations and displacements associated with the hypothesis, where the current observation is recorded from the agent's current position in the physical or logical space and includes data describing a current portion of the logical or physical space in a local vicinity of the current position, obtaining a next predicted displacement and a next predicted observation from the set of predicted observations and displacements associated with the hypothesis for the current predicted observation, iteratively comparing the next predicted observation associated with the hypothesis with the current observation to obtain an observation comparison measure indicative of a similarity between the next predicted observation and the current observation, determining a direction and magnitude of movement required to reach the predicted observation based on the observation comparison measure, moving the agent according to the determined movement and next predicted displacement such that the agent is moved from its current location to a next location in the physical or logical space, and taking a next observation from a next location of the agent, where the next observation includes data describing a next portion of the logical or physical space in a local vicinity of the next location; setting the next observation as a current observation and setting the next location as the current location; repeating the method until an observation comparison condition is reached in which the agent is at a final location in the physical or logical space; measuring an actual displacement from the original current location to the final location in the physical or logical space and comparing the actual displacement with the next predicted displacement to obtain a displacement comparison measure; determining whether a hypothesis confirmation condition is met;

[0044] According to a fifth aspect of the present disclosure, there is provided a computer-implemented method for analyzing a physical or logical space based on a series of observations and displacements performed by an agent within the physical or logical space and determining attributes of the physical or logical space, the method including: recording a first observation from a first location of the agent within the logical or physical space, wherein the first observation includes data describing a first portion of the logical or physical space in a local vicinity of the first location; processing the first observation to determine hypotheses based on a set of stored predicted observations and predicted displacements, wherein each of the stored sets of predicted observations and predicted displacements is associated with one or more hypotheses; determining a direction and magnitude of a required movement based on predicted observations associated with the hypothesis; moving the agent according to the required movement such that the agent is moved from a first location in the physical or logical space to a second location; recording second observations from the agent's second location in the logical or physical space, where the second observations include data describing a second portion of the logical or physical space in a local vicinity of the second location; processing the second observations to verify the hypothesis related to the physical or logical space; adjusting or maintaining the hypothesis based on the processing of the second observations; and determining whether a hypothesis confirmation condition is met.

[0045] Processing the first observation to determine a hypothesis may include comparing the first observation with a predicted observation from the set of stored predicted observations to determine a similarity measure between the first observation and the predicted observation from the set of stored predicted observations, and obtaining a hypothesis corresponding to the predicted observation from the set of stored predicted observations having the strongest similarity measure.

[0046] According to a sixth aspect of the present disclosure, there is provided a computer-implemented method for generating a map of a physical or logical space based on a series of observations and displacements performed by an agent in the physical or logical space. The method includes recording a first observation from a first location of an agent in logical or physical space, the first observation including data describing a first portion of the logical or physical space in a local vicinity of the first location; moving the agent by a first displacement to displace the agent from the first location in the physical or logical space to a second location; generating a second observation from the agent's second location in the logical or physical space, the second observation including data describing the second portion of the logical or physical space in a local vicinity of the second location; recording the second observation, recording the first displacement from the first observation to the second observation, and recording a negative first displacement from the second observation to the first observation; and repeating this process for at least n-1 displacements, n observations, and n-1 negative displacements to generate a set of connected n-1 displacements and n observations that define a map of the physical or logical space, where n is a positive real number.

[0047] Recording observations and displacements can be seen as making observations and displacements by an agent.

[0048] Before recording the second or nth observation, the method compares the nth observation with the previous n-1th observation to determine an observation similarity measure between the nth observation and the n-1th observation; compares the observation similarity measure with a predetermined maximum similarity threshold; if the similarity measure does not exceed the maximum similarity threshold, record the nth observation; record an n-1th displacement from the n-1th observation to the nth observation; record a negative n-1th displacement from the nth observation to the n-1th observation; or if the similarity measure exceeds the maximum similarity threshold, move the agent by a further displacement to move the agent in physical or logical space. generating an n+1 observation from the agent's n+1 position in the logical or physical space (where the n+1 observation includes data describing the n+1 portion of the logical or physical space that is in a local vicinity of the n+1 position); comparing the n+1 observation with a previous n-1 observation to determine an observation similarity measure between the n+1 observation and the n-1 observation; comparing the observation similarity measure to a predefined maximum similarity threshold; and repeating this process until the predefined maximum similarity threshold is not exceeded.

[0049] The method may include forming an identity from the n observations and the n-1 displacements, where the identity includes at least a subset of the n observations and the n-1 displacements, and storing the identity.

[0050] The method may further include associating or labeling the identity and the corresponding subset of observations and displacements that form it with an attribute of the space.

[0051] Multiple identities may be formed from n observations and n-1 displacements.

[0052] A map and / or identity may include more or fewer than n-1 displacements, since an observation may have more than one displacement relative to other observations, such as imaging a set of three observations, with each observation connected to the other two to form a triangle with three observations and three displacements. It is understood that if the number of observations is large, the number of displacements may be greater than the number of observations.

[0053] According to a seventh aspect of the present disclosure, there is provided a method of navigating a map of a physical or logical space generated according to the sixth aspect above, the method comprising: obtaining a global target observation, where the target observation is an observation of a set of n observations and n-1 displacements; and making a first observation from a first location of an agent in the logical or physical space, where the first observation includes data describing a first portion of the logical or physical space in a local vicinity of the first location; comparing a first observation with a set of n observations and n-1 displacements to identify a first target observation of the set of n observations and n-1 displacements (where the first target observation is an observation connected to an overall target observation in a map of physical or logical space that is most similar to the first observation); determining a first movement from the first observation to the first target observation based on the comparison between the first observation and the first target observation; relocalizing the agent to the first target observation by moving the agent according to the first movement; iteratively moving the agent through the n-1 displacements and n observations toward the overall target observation in the map by moving the agent according to the n-1 displacements; making transition observations with the n observations; comparing the transition observations with the n observations; comparing the measured displacements with the n-1 displacements; and adjusting the movement of the agent based on the comparison until the overall target observation is reached.

[0054] According to an eighth aspect of the present disclosure, there is provided a computer-implemented method for determining similarity between a portion of a physical or logical space navigable by an agent and stored spatial data for identifying the agent's location in the physical or logical space, the method including, in order, recording a series of observations from respective positions of the agent in the physical or logical space, where each observation includes data describing a respective portion of the logical or physical space in a local neighborhood of the agent, moving the agent between the respective positions and recording a series of displacements between the respective positions, comparing each of the series of observations with a set of previously stored observations to obtain an observation similarity measure for each observation with respect to one or more of the previously stored observations, comparing each of the displacements with the set of previously stored displacements to obtain a displacement similarity measure for each displacement with respect to one or more of the previously stored displacements, and determining that the series of observations and / or displacements are similar to the previously stored series of observations and / or displacements if the observation similarity measure and / or the displacement similarity measure exceed a confidence threshold.

[0055] The method may include combining the observation similarity measure and the displacement similarity measure to generate a two-dimensional identity comparison measure, and processing the identity comparison measure to determine a type of similarity between the observation and displacement and a previously stored observation and displacement.

[0056] According to an eighth aspect of the present disclosure, there is provided a computer-implemented method for performing simultaneous localization and mapping in logical or physical space, the method including: recording a first observation from a first location in the logical or physical space, wherein the first observation includes data describing a first portion within the logical or physical space; performing a first displacement from the first location in the logical or physical space to an unknown second location; recording a second observation from the second location, wherein the second observation includes data describing the second portion within the logical or physical space; repeating the method for n observations and n displacements, where n is a positive real number, to form a series of sets of successive observations and corresponding displacements; and comparing the set of successive observations and corresponding sets of displacements with a saved set of observations and a saved set of displacements, wherein the saved sets are captured in a closed loop, and the comparing includes comparing using at least one of a displacement comparison measure and one or more observation comparison measures to determine whether a feature corresponding to the saved set is present in the space.

[0057] The method is effectively used to analyze a multidimensional space based on a series of observations and displacements performed by an agent within the space and to explore attributes of the space. The method includes identifying hypotheses about attributes of the space and verifying the hypotheses by making sequential observations and displacements from the agent's position within the space, comparing the observations and displacements with a set of stored observations and displacements to obtain an observation comparison measure and / or a displacement comparison measure, and adjusting, maintaining, or confirming the hypotheses based on the observation comparison measure and / or the displacement comparison measure.

[0058] Each of the above aspects pertains to an agent in a space configured to move and observe the space in a minimal way such that a network of connected observations and displacements describes the space in a map. Current observations and displacements made by the agent are compared to stored observations and displacements on the map and may affect the behavior of the agent in terms of movement and function. In this way, the agent can move and / or explore the space while simultaneously observing the space for purposes of image and object recognition.

[0059] An agent may be a real object such as a robot, a vehicle, an autonomous vehicle, or an unmanned aerial vehicle, but is not limited to such possibilities.

[0060] A group of agents can be used together to communicate with a computer system or the like to update a global map or resources based on the spaces in which different agents reside.

[0061] Embodiments of the present invention will now be described, by way of example only, with reference to the following drawings, in which: [Brief explanation of the drawings]

[0062] [Figure 1] FIG. 1 is a schematic diagram of a physical space analyzed by an agent. [Figure 2] 1 is a schematic diagram of the physical / logical space analyzed by the agent. [Figure 3] FIG. 1 is a flow diagram outlining a method for searching attributes of a space. [Figure 4] FIG. 1 is a detailed flow diagram outlining a portion of a method for searching attributes. [Figure 5] 1 is a schematic diagram showing an overview of a system for carrying out the method of the present invention. [Figure 6] FIG. 2 is a schematic diagram of the functions performed by the agent in various ways. [Figure 7] FIG. 1 is a schematic diagram illustrating various ways in which an agent can move according to a method. [Figure 8]FIG. 1 is a schematic diagram illustrating identities that can be used to generate a map of a space. [Figure 9] 1 shows various schematic diagrams of vectors and matrices obtained in various ways. [Figure 10] 1 shows various schematic diagrams of vectors and matrices obtained in various ways. DETAILED DESCRIPTION OF THE INVENTION

[0063] Common reference numbers are used throughout the drawings to denote like features.

[0064] Examples described herein relate to computer-implemented methods and systems for analyzing a multidimensional space and determining the structure of that space or one or more attributes of that space. The approach taken in the described embodiments treats the problem of understanding space as a closed-loop problem, where sensor data from sensors or virtual sensors that describe an agent's environment directly drives changes in the agent's behavior.

[0065] The term "agent" is used to refer to either or both physical agents that exist in the physical space of the real world, such as vehicles or robots, and non-physical agents that exist in logical space, such as points or pixels in an image or map.

[0066] Agents can move within their respective spaces. In physical spaces, agents move by activating movement modules, such as actuators and motors. In logical spaces, agents move from a first point or pixel to a second point or pixel. Agents can be real or virtual. A real agent, such as a robot, occupies an area in a space. A virtual agent is a projection onto a space and does not actually exist within the space. For example, a virtual agent may be a projection of the center of view of a camera observing a two-dimensional scene. The camera can move and / or rotate to change the center of view on the scene, changing the position of the virtual agent. This is explained in more detail below.

[0067] Physical space refers to the three-dimensional space that a real-world object, vehicle, robot, or other agent can physically reach or traverse. Three-dimensional space may consist of one or more other physical or material objects. Three-dimensional space may also include the distance (or time) interval between two points, objects, or events that an agent may interact with as it moves through physical space.

[0068] A logical space is a space with an arbitrary set of points, such as a mathematical space or a topological space. A logical space can therefore be, for example, an image, a map, or a concept map. A set of points may satisfy a set of axioms or constraints. A logical space may correspond to a physical space, as described above.

[0069] Data is captured regarding the position of an agent in space, and the captured data describes the spatial environment in the local vicinity of the agent. Data capture is performed by sensors in the physical space, such as cameras, radar, tactile sensors, LIDAR sensors, infrared sensors, etc. Data capture is performed by virtual sensors in the logical space that obtain data of the logical space in the vicinity of the agent.

[0070] In physical space, there is the issue of observability, where an agent may not be able to observe all of the physical space at any given time: the physical space may be too large for sensors to observe, or the physical space may contain objects or terrain that obstruct regions of the physical space.

[0071] In a logical space, all of the logical space may be observable at once. For example, if the logical space is an image, the entire image may be stored in memory accessible to the agent.

[0072] However, in logical space, it is more efficient for an agent not to observe the entire logical space at once. For this reason, virtual sensors only capture a portion of the logical space, simulating sensors in physical space in that only the agent's vicinity in the logical space is captured and / or acquired by the virtual sensors.

[0073] Therefore, in both scenarios with agents in logical space and agents in physical space, it is computationally advantageous for agents to observe only a portion of their neighborhood in each space, rather than requiring the agents to observe the entirety of their respective spaces.

[0074] If the agent is a non-physical agent in a logical space, the agent's neighborhood does not need to be directly observed by a sensor or virtual sensor. For example, the agent's neighborhood could be a portion or region of a pre-recorded image that defines the overall logical space.

[0075] We define observation data as data captured by sensors or virtual sensors that observe the space near an agent. This observation data is used to make an "observation." Thus, an observation is the result of processing the observation data in a way that succinctly describes the environment near the agent.

[0076] In physical space, observations may be represented by matrices or vectors that provide spatial information derived from observational data, such as images or point cloud data of the agent's surroundings.

[0077] In logical space, an observation may be represented by a matrix or vector that provides spatial information in the agent's local neighborhood. The observation data from which this observation is derived may be a set of pixels or points from the logical space surrounding the agent. For example, the observation data could be a matrix of pixels that are within a threshold observable distance from the agent.

[0078] An agent may make multiple different observations at different locations in space. The agent makes observations by moving between these locations. The vector between two observations is defined as a "displacement" that has both magnitude and direction. Thus, an agent makes an observation by moving between two specific locations and moving along the displacement between those two specific locations.

[0079] Each observation and displacement made by an agent in a space may be stored in a memory associated with the agent. When an agent makes a series or sequence of observations and displacements in a space, the series or sequence of observations and displacements may be stored together as a set of observations and displacements. A series of observations and displacements may form an "identity." An identity may describe a particular feature or attribute of the space in which it is found. The terms attribute and feature are used interchangeably hereinafter.

[0080] For example, in a physical space, an identity may describe attributes such as a particular object or a path to a particular destination in the space. The observations in the set that form the identity indicate how the agent perceives the space and therefore how a particular feature is perceived. For example, if there is a particular object, such as a chair, in the space, the identity of the chair may include multiple observations taken from various locations on parts or sides of the chair. Similarly, if there is a particular destination in the space, the identity may include multiple observations taken from locations on the path to the destination. The displacements in the set that form the identity indicate how the agent moves between observation locations. For example, a first observation of the chair in the space may be made from a position near the back legs of the chair. A second observation of the chair in the space may be made from a position near the front legs of the chair. The first displacement of the identity is the displacement required to move from a position near the back legs of the chair to a position near the front legs of the chair.

[0081] An exemplary identity is shown in Figure 1. Figure 1 shows a series of observations 101 connected by a series of displacements 102 in an exemplary indoor space containing a door and a central obstacle. This exemplary room space can be a physical space such as a real-world apartment. The observations 101 may be captured using a camera on a robot. The displacements 102 may be made by moving the robot. In this example, the identity represents a path to a door, which is considered a destination. Each observation 101 provides data describing the environment from where the observation was made.

[0082] Although the displacements are shown in Figure 1 as being unidirectional, a corresponding set of negative displacements is also recorded, with the understanding that the identity consists of an observation 101, a set of displacements 102, and the negative displacement that links the observations.

[0083] The spatial attribute of the identity shown in Figure 1 may be a destination, such as a door. Thus, this identity can be used to navigate through a room, avoid an obstacle in the middle, and navigate to the door.

[0084] A second example of identity is shown in Figure 2. Figure 2 shows a series of observations 201 connected by a series of displacements 202 on a chair. The chair may be three-dimensional in a physical space, such as the real world, or two-dimensional in a logical space, such as an image. As in Figure 1, displacements 202 may also include opposite, negative displacements.

[0085] In the physical space, for example, the camera is responsible for capturing observations 201 and performing displacements 202 to obtain identities as shown in Figure 2. In the logical space, this may be performed by a virtual agent.

[0086] In the case of a camera in a physical space, the camera may or may not be in a fixed position. If the camera is not in a fixed position relative to the space, it may move within the space. For example, the camera may be attached to a robot. In this case, the agent may be the robot (or camera), and observation data is recorded at the agent's location by simply recording images of the robot's or camera's surroundings. If in a fixed position, the camera may rotate or zoom to focus on different parts of a chair. In this case, the camera's focus and gaze point (i.e., direction of gaze) from the perspective of the camera's field of view can represent the agent. In this case, the agent may be defined, for example, as the center point or projection of another specific pixel in the camera's field of view. Although the camera does not move translationally, the agent "moves" when the camera zooms or rotates. This is because such actions change the center point or the location of a specific pixel in the camera's field of view relative to the scene / image being observed. Therefore, the agent's position does not need to match the camera's position. If fixed, the camera may not displace between observations to obtain displacement 202. Instead, displacement may be created and stored as the distance between two fixed points in the camera's field of view across two observations. Displacement can be performed by rotating the camera to "move" the agent and move the center point of the camera's field of view.

[0087] For example, the camera may first capture an image centered on the legs of a chair. This image is then processed in a first observation. The agent's position is considered to be the center point of the camera's field of view for the first image. The camera is then rotated and panned upwards to capture and process a second image of the chair back. The second observation is processed from the image of the chair back. The agent's position in this second observation is considered to be the center point of the camera's field of view for the second image, and the displacement between the first and second observations is the distance in pixels between the center point of the field of view for the first image and the center point of the field of view for the second image and / or a distance related to the size of the rotation. This can be estimated using, for example, visual inertial odometry.

[0088] If the camera is not fixed, the agent is the camera itself (or the device on which the camera is attached). In this example, the camera / device moves to a specific location, makes observations at that location, and performs movements. Displacements are recorded, for example, using odometry.

[0089] In the logical space, Figure 2 can represent an image of a chair. In this example, the entire image is stored, so there is no observability issue and the entire image can be observed at once. However, to improve computational efficiency, the logical space is treated similarly to the physical space, and instead of processing the entire image at once, each portion of the image is observed in detail. Specifically, a portion of the image plane, for example, the portion containing the legs of a chair, is observed. The size of the portion may be fixed or may vary depending on the size of the image. Similar to the example of a fixed camera in physical space, the location of the agent is considered to be a specific point in the portion of the image, for example, a specific pixel, such as the center pixel of the image portion.

[0090] The "agent" or virtual agent in this case can "move" a number of pixels on the image plane to perform a displacement before making a second observation. In practice, this simply involves retrieving a second portion of the image plane from memory, which is then centered around the agent's new position (the pixel to which the virtual agent has "moved"). Thus, in logical space, the agent is a tool for selecting different parts of the image based on a fixed point in the image. The movement of the virtual agent involves a translation from the pixel where the virtual agent was previously located to another pixel.

[0091] Therefore, when referring to an "agent," it is important to understand that an agent can be three things. First, an agent can be a mobile physical device, whereby the agent physically moves to a location in physical space and makes observations at the device's location. In this case, the agent performs the movement between observations by moving the physical device, and the observation data corresponds to the data captured by the agent's current location. Second, an agent can be a projection of a fixed physical device. In this case, observations are made at various orientations of the fixed physical device, and the agent is a projection of a point in the physical device's field of view relative to the physical device's orientation. For example, the agent is placed at the center point of the fixed physical device's field of view. Displacement in this case is measured based on the distance between projections of points in the physical device's field of view caused by changes in the physical device's orientation. Third, an agent can be a virtual agent in logical space. In this case, the agent is placed at a point or pixel in logical space. The point or pixel specifies the portion of the surrounding space that corresponds to the observation data. For example, the agent may be placed at the center pixel of the portion of space that corresponds to the observation data of the observations. Displacement is measured as the distance between pixels or points in logical space.

[0092] Therefore, when referring to an agent, and in particular to the "location of an agent" in the preceding description, it should be noted that any of the above definitions of "agent" may apply. The location of an agent need not be the location of a physical device. Similarly, when referring to an agent that performs an action, such as moving or making an observation, the above definitions apply. That is, the physical device need not move, and the observation may be made by a physical device or computing device that is not necessarily the agent itself.

[0093] Returning to FIG. 2, the attribute of space defined by the identity formed by a series of displacements 202 and observations 201 may be the identity of a feature (in this case, a chair).

[0094] Thus, Figures 1 and 2 show that identities can be formed in both two-dimensional or three-dimensional physical and logical space for the purposes of identifying an object such as a three-dimensional chair, identifying an image or feature of an image such as a two-dimensional chair, or navigating three-dimensional space to reach a destination. It should be understood that there are other uses for identities, and object / image recognition and navigation are merely examples.

[0095] Multiple identities, such as those in Figures 1 and 2, may be stored in a set of saved observations and displacements from previous experience of one or more spaces. Identities need not be labeled; that is, in the example above, the identity corresponding to a chair need not be labeled a chair or even known to correspond to a chair. Rather, once a set of observations and displacements is saved as an identity, the agent can use this information to locate and identify matches to the identity in other regions of the space or across other spaces. The system or method does not need to understand the real-world features to which the identities relate.

[0096] In some embodiments described below, identities may be labeled according to features or attributes of the space they represent. In the above example, an identity corresponding to a chair may be labeled as a chair. The process of labeling identities in a set of stored observations and displacements may be performed as part of a general training process, which may be performed according to existing techniques understood by skilled artisans. For example, various machine learning techniques may be used, such as supervised or unsupervised learning processes, the use of neural networks, or classifiers.

[0097] For example, the set of stored observations and displacements may be obtained from a validation space, such as a validation image that includes or exhibits one or more known features. For example, a validation image of a chair is presented to the agent, and an identity formed from that image and stored in the set of stored observations and displacements is found to correspond to a chair and is labeled as such. The process of obtaining the set of stored observations is described in more detail below.

[0098] Using a set of observations and displacements to effectively describe spatial features, rather than an entire map or image of the space, is computationally efficient because the observations and displacements contain only the respective portions of the feature, requiring less memory and computation to identify it in a portion of space. Furthermore, the observations and displacements can be computed using a variety of data, such as vectors encoded from the output of a series of spatial filters applied to the visual data of the observations, or, in the case of displacements, the local temporal correlations of low-level features. This eliminates common failure modes and provides versatility by allowing the two parts of the system (observations and displacements) to compensate for the weaknesses of the other. In the application domain of 3D spatial SLMA, the system inherently performs not only localization and mapping but also navigation and path planning, eliminating the need to add additional components to achieve a complete real-world implementation.

[0099] Compared to standard SLAM, the methods and systems described herein are more robust to environmental factors and occlusions. The methods and systems described herein also have the advantage of storing a minimal set of measurements necessary to navigate and observe a particular space. Observations are made within a space depending on the complexity and / or changes within that space, allowing the methods and systems to adapt to work optimally in any space. For example, in a vast field or area with little information or features, fewer observations are made and stored compared to a feature-rich environment. Because the methods do not involve tracking features or objects that occupy a small portion of the field of view but instead fully utilize the characteristics of a large portion of the full solid angle sphere, occlusions or changes to portions of the visual scene have less impact on the system's ability to complete its task compared to traditional systems and approaches.

[0100] The features or attributes to which the stored observations and displacement sets relate may be identified in other regions of the same space, or in entirely new spaces. For example, in a physical space such as a living room or dining room, a stored observation and displacement set associated with a chair can be used to identify identical or similar chairs in other parts of the living room or dining room. Furthermore, a stored observation and displacement set can be used to identify identical or similar chairs in an entirely different space, such as another building.

[0101] Similarly, a destination in space, such as a store or landmark, may be navigated from an initial known location in the world or a previously visited location based on a stored set of observations and displacements connecting the starting point and the destination. Additionally, a destination in space may be navigated from an unknown region of the world or from an unknown starting point from a set of observations and displacements.

[0102] Identifying attributes of a space in this way relies on comparing a set of stored observations and displacements with current observations and displacements made in a particular space. The next part of the description will explain this process in more detail. In the next part of the description, stored observations and displacements will be referred to as predicted observations and displacements, and target observations and displacements. It is important to understand that these terms define the same technical features and are simply different to provide context as to why and when they are used. Similarly, current observations from a space will be referred to as first observations, second observations, and transition observations. Again, these terms define the same technical features—observations made by an agent within the space in which the agent resides—but are labeled differently to provide context as to why and when they are made or used. The same terminology applies to displacements.

[0103] How this process of identifying / determining attributes of a space using existing knowledge in the form of a set of stored observations and displacements is carried out is described in detail below with reference to Figure 1. The advantage of the present method is that only a portion of the space is needed in terms of the current observations and displacements to identify the attributes.

[0104] FIG. 3 illustrates a flow diagram of a method 300 for determining attributes of a space that an agent can explore or navigate, according to various embodiments.

[0105] In the initial conditions of method 300, the agent is in a space and can observe parts of the space through sensors, virtual sensors, etc. This space may have been previously explored by the agent or may be completely unknown to the agent. The starting location may be known to the agent or may be completely unknown to the agent.

[0106] In a first step 301, the agent makes an observation in space from the agent's current location in space. This observation may be considered a current observation or a first observation from a current location or initial location, although it should be understood that this observation need not be the very first observation in the space. Rather, "first" is used here simply to distinguish between a current observation and a later observation.

[0107] The first observation includes data describing a first portion of space in the agent's local neighborhood. For example, if the agent is a robot or a vehicle, the first observation is made using sensor data from sensors such as cameras, radar, and LIDAR, where the sensor data describes a view of the world from the robot's current position. The agent's local neighborhood is the local neighborhood of the robot's current position. In physical space, the extent of the local neighborhood is determined by the sensor constraints and the robot's environment. For example, a sensor may have a limited range, Sensor data is acquired only for the environment of the space within that range.

[0108] Similarly, a robot's environment may include one or more features that limit the observable surroundings, such as walls or other obstacles that block the sensor's view. Therefore, an agent's neighborhood is not fixed; there may be a maximum distance from the agent's location based on sensor constraints. In logical space, an agent's local neighborhood is a portion of the surroundings around the agent's location, which is smaller than the entire space. Because the entire space can be stored, for example, if the space is an image, data does not need to be acquired from a sensor to make an observation. Rather, a portion of the space around the agent's location is acquired from the entire stored space. This "virtual sensor" simulates the same effect as using a sensor in physical space, but the observation considers only a portion of the space rather than the entire space. Therefore, an agent's neighborhood in logical space may be set based on the distance from the agent's location. In an example where the space is an image and the agent is located at a pixel in the image, the neighborhood may be set, for example, by the number of pixels away from the agent's pixel.

[0109] In a first step 301, a first observation is made and in a second step 302, it is compared to a stored observation from a set of stored observations and displacements.

[0110] The comparison of the first observation with the stored observations generates an observation comparison measure that indicates how similar each of the stored observations is to the first observation. The observation comparison measure for each stored observation is ranked, the highest observation comparison measure is identified, and the corresponding stored observation is obtained. In this manner, the stored observation that is most similar to the first observation is determined and selected. The process of comparing observations is described in more detail below.

[0111] In a third step 303, a hypothesis is determined regarding the stored observations that are most similar to the first observation. In particular, the set of stored observations and displacements includes one or more identities that form a subset of the set of stored observations and displacements. Each subset includes one or more observations and one or more displacements, each associated with a particular attribute or feature to which the identity is associated. In the third step 303, the attribute or feature with which the selected stored observation is associated is determined. This attribute forms the basis of a hypothesis, which in effect becomes a prediction of what the agent is observing in space. The prediction can be, for example, a two-dimensional or three-dimensional object or image, or a navigable destination.

[0112] The set of stored observations and displacements may include multiple subsets of observations and displacements, each subset associated with a particular attribute and therefore a particular hypothesis. In one example, there is a subset associated with the attribute "chair," a subset associated with the attribute "table," and a subset associated with the attribute "door." The comparison in the second step 302 is performed for all observations in the set of stored observations and displacements, and the observation comparison measure may be highest or strongest for a particular observation, and that particular observation is part of the subset associated with the attribute "door." Thus, in the third step 303, a hypothesis is made that the first observation is part of a door, i.e., the attribute of the space observed by the agent is a door in the space. The subset of stored observations and displacements associated with an attribute is called a hypothesis subset.

[0113] In a fourth step 304, a hypothesis subset of observations and displacements is obtained. In particular, the stored observation most similar to the first observation is part of the hypothesis subset of stored observations and displacements, as described above. Furthermore, the observations and displacements within the hypothesis subset are linked, forming a network of observations separated by displacements. The links between the observations and displacements within the hypothesis subset are stored in a set of stored observations and displacements, and the network arrangement is stored in memory. When obtaining the hypothesis subset of observations and displacements, the method includes obtaining the observation and displacement associated with the stored observation most similar to the first observation. In other words, the neighboring observations and displacements required to reach the first observation from the stored observation most similar to the first observation are obtained from the hypothesis subset. This observation and displacement become the predicted observation and predicted displacement, and if the agent moves the predicted displacement from its current position, according to the hypothesis, the agent is expected to arrive at a position in space where it can observe the predicted observation.

[0114] It should be understood that the terms "predicted observations" and "predicted displacements" simply refer to the subset of observations and displacements associated with a determined hypothesis, each of which is predicted to lie within the space currently occupied by the agent.

[0115] In a fifth step, 305, the hypothesis is verified. To verify the hypothesis, the agent is moved sequentially along the network of stored observations and displacements that form a subset of the hypothesis, i.e., the predicted observations and predicted displacements. At each iteration of this sequential process of moving the agent along the network, the agent is configured to repeatedly make new observations, compare the new observations with predicted observations and / or compare predicted displacements with actual displacements, and decide whether to maintain the hypothesis, reject the hypothesis, or confirm the hypothesis. This hypothesis verification is described in more detail below with reference to steps 305-1 through 305-5.

[0116] In the first step 305-1 of the hypothesis testing process, the agent moves from its current location where it made a first observation to a second location in space based on a movement function. The movement function depends on the predicted displacements of the hypothesis subset. In particular, the movement function depends on the predicted displacements connected to the stored observation that is most similar to the first observation, identified from the linked network of observations and displacements that form the subset. Thus, the agent does not know the second location until it performs a movement based on the movement function.

[0117] In a second step 305-2 of the hypothesis testing process, the agent makes a second observation from a second location of the agent in space, the second observation including data describing a second portion of the space in a local vicinity of the second location. For example, if the agent is a robot or vehicle, the second observation is made in the same manner as the first observation using sensor data from a sensor such as a camera, radar, or LIDAR, and the sensor data is processed to result in an observation showing a view of the world from the robot's position at the second location.

[0118] In the third step 305-3 of the hypothesis testing process, the actual displacement of the agent from the first location to the second location is determined. In physical space, the actual displacement may be estimated using, for example, odometry. In logical space, the actual displacement may be measured according to known techniques using vector or matrix calculations. For example, if the logical space is an image, the number of pixels between the first and second locations may be determined as the distance. In physical space, the displacement is measured using an odometry system that combines visual inertial odometry and, if applicable, kinematic odometry, such as wheel odometry. This system provides an interface for input of the visual inertial odometry and kinematic odometry. The displacement indicates the physical or logical distance between a pair of observations. The odometry is stored in a coordinate system referenced to the rotation of the starting observation (the predicted observation that is most similar to the first observation). Additional data characterizing other aspects of the displacement may be stored, such as, but not limited to, the vibrations occurring during the displacement transition and the energy expended during the displacement transition. Displacement similarity is measured by the end-point error between pairs of displacements, where the displacements start at the same location and the end-point error is the physical or logical Euclidean distance between the displacements determined by transitions along the two displacements.

[0119] In the fourth step 305-4 of the hypothesis testing process, a comparison is performed. The comparison may compare observations, displacements, or both. A second observation may be compared to a predicted observation from the hypothesis subset, particularly to a predicted observation linked to the predicted displacement on which the movement function was based. Comparing the second observation to the predicted observation provides an observation comparison measure. Similarly, a displacement comparison measure may be obtained by comparing the predicted displacement on which the movement function was based when the agent moved from a first location to a second location in space with the actual displacement between the first and second locations. Thus, the comparison may generate both an observation comparison measure and a displacement comparison measure. As will be explained in more detail below, using both of these measures may be advantageous. It should be understood that the order of the above steps after the agent has moved is interchangeable.

[0120] In the fifth step 305-5 of the hypothesis testing process, the hypotheses are adjusted, maintained, or confirmed based on the observed and / or quantile comparison measures from the fourth step of the hypothesis testing process 305-4.

[0121] In this step, the observation comparison measure and / or the displacement comparison measure are evaluated to effectively determine whether the agent's perception of the occupied space in terms of the second observation and actual displacement matches the hypothesized attribute perception in terms of the predicted observation and predicted displacement. The observation comparison measure and the displacement comparison measure are described in more detail below, but generally, the stronger these measures are, the more likely the hypothesis is correct and that the agent is actually observing the hypothesized attribute in the space in which they are currently located.

[0122] The observed comparison measure and / or the variance comparison measure may be compared to one or more respective thresholds to determine whether to confirm, maintain, or reject the hypothesis.

[0123] Adjusting a hypothesis involves rejecting the hypothesis. This occurs when the space being observed by the agent is determined to be unlikely to contain or represent the attributes according to the hypothesis. This conclusion may be reached in several different ways. First, the hypothesis may be rejected if the observation comparison measure and / or the displacement comparison measure fall below a first observation threshold level and / or a first displacement threshold level. The first observation threshold level and the first displacement threshold level may be globally set as the minimum probability required to maintain the hypothesis and allow the agent to further test the hypothesis. Alternatively, these thresholds may be adaptive and modifiable based on environmental conditions of the space, such as brightness / darkness factors. In particular, the lighting of the space may affect the observation comparison measure if the agent's observations are made at a different environmental brightness level than the brightness level at which the subset of stored observations relevant to the hypothesis were captured. The thresholds may also be adjusted based on color, contrast, and other image / sensor data characteristics.

[0124] Furthermore, different observation comparison measures may be used to address additional spatial aspects, and their combined use may identify specific dimensions of difference for a hypothesis. For example, changes in lighting have a greater impact on vector comparison measures than on spatial distribution measures of filtered elements around the agent's location. Changes in object color affect color-filtered elements of the observation more than brightness or orientation-filtered elements. By considering various observation comparison measures, it becomes possible to observe the nature of the differences. When lighting conditions change, a measure of the spatial distribution of filtered elements of the observation around the agent's location can be used to adapt the threshold, and strong matching against that measure can be used to pre-update the hypothesis, lowering the acceptance criterion for the "similarity" observation comparison measure (the second observation threshold level). The various types of observation comparison measures described here each arise from different ways of comparing observations, as will be explained in more detail later.

[0125] The first observation threshold level may be set according to the initial ranking of the observation comparison measure of each stored observation relative to a first observation made by the agent. In particular, the first observation threshold level may be set equal to the next highest-ranked observation comparison measure corresponding to a stored observation in the set of stored observations and displacements that is not included in the subset associated with the current hypothesis. Thus, this particular stored observation is associated with a different hypothesis. In this way, the first observation threshold level is set according to the similarity between the first observation and stored observations associated with a hypothesis different from the currently tested hypothesis. This allows the agent to effectively consider a hypothesis different from the one being tested if the tested hypothesis generates a weaker observation comparison measure than the one initially found for the different hypothesis. Therefore, the hypothesis being tested is rejected and ultimately replaced with another hypothesis associated with the first observation threshold level.

[0126] Finally, adjusting the hypothesis involves rejecting the current hypothesis and replacing it with another hypothesis. This may involve the method returning to the second step 302, as shown in FIG. 3, and selecting a next-best observation comparison measure from the ranked observation comparison measures to determine the next-best observation and its associated hypothesis. The agent may also return to the location of the first observation in space via a movement opposite to the movement actually made before selecting a new hypothesis. Hypothesis replacement is described in more detail below.

[0127] Hypothesis retention occurs when a hypothesis is neither adjusted nor confirmed. Therefore, if the observation comparison measure and / or the displacement comparison measure match or exceed the first observation threshold level and / or the first displacement threshold level, respectively, the hypothesis is retained. This means that there is no need to adjust the hypothesis. According to the embodiment introduced above, the first observation threshold level is set according to the "next-best observation comparison measure," which means that the current hypothesis still provides the best observation comparison measure for the predicted observation from any subset of the set of stored observations and displacements, meaning that the attribute according to the current hypothesis is still the attribute most likely to be found in the space in which the agent resides. Another condition for a hypothesis retention is that a hypothesis confirmation condition is not met. The hypothesis confirmation condition may be a second, higher observation threshold level and / or a second, higher displacement threshold level. The hypothesis confirmation condition indicates a minimum confidence that the hypothesis is correct and that the attribute of the space is the attribute according to the hypothesis.

[0128] In other words, if the observation comparison measure and / or the displacement comparison measure for the predicted observation and predicted displacement are determined to match or exceed the first observation threshold level and / or the first displacement threshold level, respectively, but not match or exceed the second observation threshold level and / or the first displacement threshold level, respectively, the hypothesis is maintained.

[0129] Once a hypothesis is maintained, a next predicted observation and next predicted displacement, linking the current predicted observation and the next predicted observation, are obtained from the subset of stored observations and displacements associated with the hypothesis. The above hypothesis testing process is then repeated for the next predicted observation and next predicted displacement, as shown in Figure 3. Thus, method 300 involves sequential testing of hypothesis subsets of predicted observations and displacements until the hypothesis is confirmed or adjusted.

[0130] With each observation the agent makes while maintaining a hypothesis, the past observation comparison scale and / or displacement comparison scale are updated, and the past observation scale and / or displacement scale are then incorporated into future observation comparisons and / or displacement comparisons. The past observation scale and / or displacement scale are stored in memory and function to increase the future observation comparison scale and / or displacement comparison scale based on the consecutive number of times the hypothesis is maintained. This allows for a hypothesis to be rejected based on multiple consecutive observations and displacements. This effectively increases confidence that the hypothesis can be confirmed. It also helps prevent the agent from getting stuck in a local minimum where the hypothesis is never rejected or confirmed. This process, which utilizes historical observational and / or quantile-scale data, may involve sequential probability ratio testing of a single hypothesis against a null hypothesis (data inconsistent with identity), or as sequential probability ratio testing of multiple hypotheses against alternative hypotheses including the null hypothesis (no hypothesis is consistent).

[0131] Confirming the hypothesis includes determining that a hypothesis confirmation condition is met and determining attributes of the space based on the hypothesis confirmation condition being met. As discussed above, the hypothesis confirmation condition can be a second, higher observation threshold level and / or a second, higher displacement threshold level that the observation comparison measure and / or the displacement comparison measure must meet or exceed.

[0132] Like the first observation threshold level, the second observation threshold level may be changed or adapted based on lighting conditions within the space using a lighting factor.

[0133] If the hypothesis is confirmed, the method 300 may stop as the agent identifies the attribute of the space. The identity of the attribute may be stored in memory, and the observations and displacements performed by the agent to identify the attribute may be stored in memory and linked to the space in which the agent resides.

[0134] The method 300 has the advantage that it uses observations and displacements to determine attributes of a space without having to consider the completeness of the entire space, resulting in a much more computationally efficient method of spatial analysis.

[0135] Method 300 will now be described in more detail. First, referring to FIG. 4, method 400 for making and comparing observations will be described. Making and comparing observations occurs in at least two ways in method 300. First, in a first step 301 and a second step 302, the agent makes an observation and compares it with a set of saved observations to ultimately determine a hypothesis, and then in a fifth step 105, the hypothesis is verified. Thus, this process requires at least two observations and two comparisons. However, it should be understood that more observations can be made, and that in the two specific examples above, as well as in transition observations between a first location and a second location, it may be advantageous to make the observations sequentially while the agent is moving from the first observation location to the second observation location. These transition observations will be described in more detail below. Here, the process of making and comparing observations will be described.

[0136] In a first step 401, observed sensor data is captured. In physical space, observed data is input data captured using a sensor such as a camera. In this example, the observed data is a captured image. The captured image and its content will vary depending on the orientation of the camera. In particular, the camera may be in the same position, for example attached to a robot at the same location in the world. However, if the orientation of the robot is modified, the content of the observed data may change even though the position of the robot has not changed.

[0137] For this reason, it is advantageous to perform observations in physical space using a camera system or other sensor capable of capturing a three-axis stabilized cylindrical projection of three-dimensional space, oriented so that the cylinder's major axis is perpendicular to the direction of gravity. This fixation with respect to gravity allows observations to be performed with the same roll and pitch (as these are also fixed according to the direction of gravity). The image capture resolution may be, for example, 256 columns by 64 rows of pixels, but can also be any resolution. This 256x64 pixel image represents the observation input data and is then processed in the second step 402. In non-physical or logical spaces, the observation data may be data obtained from the logical space surrounding the agent's location. The entire logical space may also be pre-stored. The observation data may be provided from a portion of the logical space within a predetermined distance from the agent's location. For example, if the logical space is an image or map, the observation data may be pixels or points within a distance of 5, 10, 50, 100, or any number of pixels / points from the agent's location.

[0138] In the second step 402, observations are generated from input data captured by sensors, virtual sensors, or retrieved from memory. The input data is encoded into a set of vector / matrix representations through a staged pipeline.

[0139] The process of generating a stored observation to form part of a set of stored observations and displacements is very similar to the process of generating a current observation from input data to compare with a stored observation. However, when comparing current observations, the relative orientation of the agent at its current location may not be the same as the orientation of the agent when it made the stored observation, and may not be easily distinguishable. To solve this problem, when generating a current observation from input data, the process involves generating an observation for each possible permutation of the agent's orientation. In practice, this means that the current observation is associated with multiple rotations, whereby each rotation represents input data that has been shifted or adjusted to represent a different orientation of the agent. How this affects the process compared to generating a "stored" observation is explained below.

[0140] The general process for generating an observation (current or saved) involves the following steps:

[0141] First, low-level features are extracted from the input data. This may involve, for example, using one or more filters or masks to preprocess the input data. Low-level features may include edges, color, shape, etc. Preprocessing may also include color transformations to provide multiple channels of the input data. The result of this first step is processed data that may potentially contain multiple channels.

[0142] Second, the resolution of the input data is reduced by using filters or the like to extract or remove portions of the input data, such as rows of pixels, and / or by summing and averaging data over large regions of the input data.

[0143] Third, further processing may be performed by convolving the input data with one or more kernels to obtain useful results, for example to identify regions of interest, edges, areas of greatest change, etc. If there are multiple channels, this process is performed for all channels.

[0144] Fourth, the resulting data forms a vector or matrix of processed and reduced input data, which is then concatenated across dimensions (columns, rows, channels) and normalized to produce the observation vector o.

[0145] As mentioned above, when storing observations, only one observation vector o is needed, e.g., for the purpose of storing a set of observations such as an identity or a map. When taking the current observation for comparison purposes, this general process is performed for several rotated permutations of the input data. If the data resolution is XxY, this may mean including X permutations, each representing an X+1 shift from the previous permutation. This results in X observation vectors for the current observation, each of which can then be compared to one stored observation vector.

[0146] This process of encoding input data into one or more observation vectors is performed by the central processing unit (CPU), the graphical processing unit, The present invention uses a computing device such as a graphics processing unit (GPU), a field-programmable gate array (FPGA), or an application-specific integrated circuit (ASIC).

[0147] A detailed example of this process is shown below for the above example of an image captured from a camera (a cylindrical projection image). In this example, the image resolution is 256x64 pixels, so there are 256 possible rotations of the projection in the current observation, and the columns are rotated (shifted) from their original capture positions. As mentioned above, these rotations represent various permutations of the original observation data. It should be understood that there can be any number of permutations based on the input resolution of the image. The detailed process is as follows:

[0148] In the first stage, a color transformation process is performed on each pixel, transforming the color channel data from the captured image. For example, red, green, and blue channel data is transformed into red, green, and luminance channels, where luminance is 1 / 3 (red + green + blue). It should be understood that these colors and transformations are exemplary, and that images may have different color channels and be transformed into different color channels.

[0149] In the second stage, the luma channel result of the first stage is convolved with a filter using the convolution kernel [1,-1;1,-1;1,-1] or a similar vertical edge filter (e.g., a Sobel filter or other filter) to create the luma vertical (luma_v) channel.

[0150] In the third stage, the luma channel result from the first stage is also convolved with a filter using the convolution kernel [1,1,1;-1,-1,-1] or a similar horizontal edge filter (e.g., a Sobel filter or other filter) to create the luma horizontal (luma_h) channel.

[0151] The result of this processing is a new image containing pixels for four channels: red, green, luma_h, and luma_v. It should be understood that other pixel filter choices can be used here. With the selected filter set, all sub-pixels of all channels (red, green, luma_h, or luma_v) are routed to the next processing pipeline.

[0152] In the fourth stage, a series of box filters are calculated and placed around a set number of equally spaced rows of the image for each channel. It should be understood that there may be more or fewer filters, and the 19x19 size is exemplary. The box filters smooth the image for each channel. Each pixel in each channel is modified by the box filter from the sum of the pixels contained within each box filter, thereby providing a smoothed (or blurred) image. Box filters may overlap each other.

[0153] In the fifth stage, the blurred image is reduced to form sliced ​​images. In particular, the resolution of the blurred image is significantly reduced. This process is illustrated in Figure 9. Specifically, four evenly spaced rows 902 are extracted from a blurred image (not shown) processed from the original image 901 (or other observational dataset) to generate sliced ​​images 904.

[0154] As shown in Figure 9, the sliced ​​image contains 256 columns and 4 rows.

[0155] In the sixth step, the columns of slice images formed in the fifth step are divided into sets of evenly spaced columns. As an example, the 256 columns are divided into 256 sets, each containing 16 columns by 4 rows of images, with the first column of each set being different from the full set of 256 columns. As can be seen in Figure 9, there are 16 sets of 16x4 images (a total of 256 permutations), and the columns are displayed in different colors to illustrate how the data from the 256-column slice image 904 is divided into 16x16x16x4 sets. These sets represent all possible rotation positions of the input data (one column rotation increment for all 256 possible rotations). Figure 9 is an illustrative example; the number of rows, columns, and permutations may be greater or less than this; i.e., the number of rotations may be greater or less than 256.

[0156] When an observation is saved to memory to form part of a set of saved observations and displacements, only one of the 16x4 images from the set formed in the fifth step needs to be saved. For example, the first permutation 906 of the first set may be selected to be saved as the observation. Rotations (i.e., the 256 permutations of the 16x4 images) need only be considered when creating an observation for comparison with a previously saved observation. Thus, permutations need only be considered for the current observation. For comparison, it is not necessary to save each permutation of the 16x4 images to memory. Rather, each permutation is iteratively compared to the saved 16x4 observation vector to identify the best match, as indicated by one or more observation comparison measures. Saving the permutations is not necessary. Comparison of observations is discussed in more detail below.

[0157] Returning to our detailed example of creating observations, in the seventh stage, a 16x4 image is periodically convolved with a center-on, perimeter-off (or vice versa) kernel. For example, elements with a value of 1 are the kernel centers. The kernel in the first row might be, for example, [-1 / 5, 1, -1 / 5; -1 / 5, -1 / 5, -1 / 5], the second and third rows might be 1, the fourth row might be [-1 / 5, -1 / 5 -1 / 5; -1 / 5, 1, -1 / 5], and so on. An example of this is shown in Figure 10. Figure 10 shows center-on, perimeter-off horizontal wrap filters 1001 through 1003 for a 16x4 image. This kernel identifies the regions of greatest change in the image. As mentioned above, when saving observations, this is only necessary for one 16x4 image. When creating current observations to compare with saved observations, this is performed for each separate set (i.e., the 16x16 set from the sixth stage).

[0158] Following this pipeline, a 16x4x4 image is generated. For comparison, the 16x4x4 image is rotated 256 times. There are 16 columns, 4 rows, and 4 channels. The channels of the 16x4 image may be divided into one data subset consisting of red and green, and two data subsets consisting of luma_h and luma_v. Each of these data subsets is normalized using vector normalization, where each element describes a value on a set of orthogonal values, resulting in a unit vector. These normalized vectors are each scaled by the square root of 2 and concatenated into a single unit vector. This provides the observation vector o. In this example, the observation vector is concatenated from 16 columns, 4 rows, and 4 channels to generate a 256-element vector that is stored. For comparison, 256 permutations of vector o are compared to the single stored vector.

[0159] In addition to the observation vector o, the final vector can store additional data related to the location of the observation (such as a user-defined label, the angular location and identity of the object, data from additional sensory modalities such as auditory or olfactory information, traversability of different directions in physical or logical space, and previously explored directions).

[0160] The observation vector o is a coarse representation of the original observation data captured by the sensors, providing a statistical summary of the agent's nearby environment in physical or logical space. The coarseness of the vector compared to the original data allows the method to run more efficiently. The observation data is also filtered at high resolution before the coarsening process, creating various channels. Coarsening also has the advantage of providing a representation of the location that is less sensitive to changes in lighting and objects in the environment than state-of-the-art techniques, since information is integrated across a large portion of the field of view.

[0161] It should be understood that the observation vector o can be a vector or a matrix. The vector o may be composed of multiple sub-regions, each corresponding to a portion of the overall vector. The sub-regions may be content / color sub-regions divided based on the channels used to form the vector (in the example, red, green, luma_h, luma_v), or they may be spatial sub-regions corresponding to different directions around the agent. The sub-regions can be represented by sub-vectors, the combination of which forms the overall observation vector o. As mentioned above, a vector representing rotational permutations is necessary when comparing observations; once a current observation is created and needs to be compared to a stored observation, it is not necessary to store all permutations.

[0162] It should be understood that the numbers given above for dimensions, filter and kernel sizes, row selection, and rotation permutations may be more or less than the example values.

[0163] Returning to Figure 4, in a third step 403, the current observation vector o is compared to a stored observation from a set of stored observations, i.e., the current observation is compared to the target observation.

[0164] Observations may be compared using several functions. Comparing the observations results in an observation comparison measure. The observation comparison measure may vary depending on the function used to compare the observations. In some embodiments, multiple observation comparison measures may be generated by comparing the observations using multiple comparison functions. These comparison functions may be used alone or in combination in the hypothesis testing method 300, as well as in other related methods described below. Environmental features, such as the brightness level of a scene, may affect one observation comparison measure but not another. Therefore, using a combination of observation comparison functions and resulting observation comparison measures allows the method 300 to perform more accurately in a variety of environmental conditions.

[0165] Functions for comparing observations include an observation similarity function 403-1, an observation rotation function 403-2, an observation action function 403-3, and an observation variance function 403-4. As shown in Figure 4, one or more of these functions can be performed on the current observation and a stored target observation. Therefore, the initial conditions and system required for each method of comparing observations are the same. This system is shown schematically in Figure 5.

[0166] FIG. 5 shows a schematic functional diagram of a system or apparatus 500 including various modules involved in comparing observations. These modules may be implemented by a computer, computing device, or computer system. It should be understood that the functionality of these modules may be further divided into other modules, separated among components, or performed by a single apparatus or component. The first module of the apparatus or system is a memory module 501. The memory module 501 stores a set of saved observations and displacements. These observations and displacements may be stored as vectors. The set of observations and displacements, including one or more hypothesis subsets, can be stored in a cyclic or acyclic graph and / or a directed graph. Alternatively, the observations and displacements may be stored with metadata identifying the displacements and observations to which they are connected, or may be organized, such as in a lookup table that maintains links between each observation and displacement.

[0167] The sensor module 504 is connected to or in communication with a current observation processing module 505. The stored observation processing module 502 is configured to select stored observations from the memory module 501 for comparison. The stored observation processing module 502 may also select one or more masks to reduce the stored observations to a subset of the stored observations for input to the comparison function. The masks may include observation masks, user / system selected masks, or a combination thereof. A mask may mask certain regions of a selected stored observation vector and retain other regions, called regions of interest (ROIs). A mask may be a binary mask, using values ​​of 0 and 1 to mask or retain, or vice versa.

[0168] The stored observation processing module 502 is connected to or in communication with a comparison engine 503 that performs comparisons according to one or more of the comparison functions 403-1, 403-2, 403-3, and 403-4.

[0169] A second side of the system or device includes a sensor module 504. The sensor module 504 includes sensors capable of inferring spatial information from the environment, and may include cameras, LIDAR sensors, tactile sensors, etc. The sensor module 504 also includes control circuitry and memory necessary to record sensor data. The sensor module 504 is configured to capture current observation data.

[0170] The sensor module 504 is connected to or in communication with a current observation processing module 505. The current observation processing module 505 processes the sensor data from the sensor module 504 to obtain a current observation, such as a current observation vector, which includes a total possible rotation for the sensor data according to the process described above with respect to Figures 9 and 10. The current observation processing module 505 may also select a mask to mask the current observation vector.

[0171] The current observation processing module 505 is connected to or in communication with a comparison engine 503 that performs comparisons according to one or more of the comparison methods 403-1, 403-2, 403-3, and 403-4.

[0172] The comparison engine 503 is configured to retrieve or receive stored observations from the stored observation processing module 502, current observations from the current observation processing module 505, and respective masks, for the purpose of performing comparison observations according to any of the comparison methods 403-1, 403-2, 403-3, and 403-4. The comparison engine 503 may be implemented by a computing device, a server, a distributed computing system, etc.

[0173] The system 500 further includes a locomotion module (not shown) configured to move the agent in space. In a physical space, this may be a motor, engine, etc., configured to move the agent by physically moving the agent's location or changing the agent's orientation. In a logical space, the locomotion module may be implemented by a computer, etc.

[0174] Although the comparison methods are described separately below, it should be understood that they may be performed simultaneously or within the same overall process executed by comparison engine 503. These methods can be performed in the same overall process because they each rely on the same underlying data points and operations. In particular, each of the following methods compares a current observation with a stored target (predicted) observation. For example, if the observation is represented as a vector, each method does this by taking a dot product between the vector or its subvectors. The comparison methods differ by additional pre-processing / post-processing steps, as described below.

[0175] The first method 403-1 compares observations based on a similarity function to obtain a first observation comparison measure component. The first observation comparison measure component is the "similarity" observation comparison measure described above. This similarity function requires that the vectors of the observations being compared be vector-normalized. The similarity function involves calculating the vector dot product of the observation vectors being compared. The dot product is composed of the cosine of the angle between the two vectors. Thus, the similarity function returns a single value or score indicating the similarity between the two vectors corresponding to the observations being compared. If the current observation includes multiple possible rotations (256 possible rotations in the example above), the vector corresponding to each of the rotations is compared to the predicted observation. In this case, the dot product that provides the highest value is saved as the first observation comparison measure component. This single value is then used in a second step 302 of the method 300, for example, to generate a set of ranked observation comparison measures corresponding to the set of saved observations, from which the saved observation most similar to the first observation is determined and selected to determine the hypothesis. In other words, this similarity function may be used to determine the similarity between a first observation and a set of stored observations, identify which of the stored observations are most similar to the first observation, and decide which hypothesis to test.

[0176] This similarity function may also be used in the fifth step 305 when comparing the second observation with the predicted observation.

[0177] When the current observation is associated with multiple rotational positions, each corresponding to an observation vector, the similarity function is configured to find the largest dot product between the different observation vectors and the target observation. To do this, the similarity function repeatedly calculates the dot product for each observation vector rotated with respect to the target observation, tracking the "maximum dot product value." Once all rotations have been evaluated, the maximum dot product value is taken as the first observation comparison measure component.

[0178] The advantage of using this similarity function is that it is relatively computationally efficient at run time and the resulting first observation comparison measure component decreases very smoothly as a function of distance from the original observation location (or the target observation location to which the current observation is compared).

[0179] However, this function may be affected by environmental conditions, such as the brightness or darkness of the environment. For example, if the stored observations relate to observations captured in low-light conditions and the current observation relates to observations captured in direct sunlight, the current observation may differ from the stored observations, resulting in different vectors, even if the features captured in each observation are spatially the same. To counteract these effects, the first observation comparison component may be combined with or used with a fourth observation comparison component formed from a fourth observation comparison function 403-4, as described below.

[0180] In the second observation comparison function, the observations are compared using an observation rotation function to obtain a second observation comparison measure component. This rotation function takes a rotation o of the current observation and compares it to each of the target observations, as in the similarity function. For example, during the fifth step 305 of the method 300 in which a hypothesis is tested, the observation rotation function may be used to compare the second observation (current observation) with the predicted observation (target observation). In the above example, there are 256 rotations, corresponding to 256 different column positions from the original cylindrical observation data. While the similarity function is configured to obtain a dot product indicating the best match between the rotated observation vector and the target observation, the observation rotation function is configured to identify the specific rotation responsible for this best match. The observation rotation function outputs the rotation direction of this best match as an offset value, which corresponds to the index of the observation's rotation position. For example, the best match between the observation vector o and the predicted observation occurs at 120 of the 256 rotations. 番目 This means that 120 of the 256 columns 番目The columns of correspond to vectors formed from observations rotated to the center of the data, or any other defined rotational position. Thus, the rotated observation function is 番目 The offset value may output a rotational position index of (which may be 120, for example).

[0181] In physical space, if an inertial measurement unit (IMU) is used as part of or included in the sensor, the offset value indicates how much the yaw (roll and pitch can be fixed by gravity) has drifted since the target observation was first made (assuming the stored observations that form the target observation have previously appeared in the current space). Thus, the offset value of the observation rotation function provides an output that indicates the IMU yaw drift. This IMU yaw drift can be identified from the output and used to update the set of stored observations and displacements, since they are all relative to the original yaw that was created and stored. Thus, if the first observation comparison scale component and / or second observation comparison scale component are reliable and indicate the current observation is the target observation, the offset value can be used to adjust the set of stored observations and displacements accordingly.

[0182] If the physical space in which the agent resides is not the same space in which the saved observations were captured, the offset value indicates how the space in which the agent resides is oriented compared to the space in which the saved observations were captured.

[0183] In summary, the second observation comparison measure component generated by the observation rotation function provides an offset result that indicates the strongest similarity between the target observation and multiple rotations (e.g., 256 rotations) of the current observation. Furthermore, the rotation index that indicates the strongest similarity indicates the alignment between the agent's current orientation (i.e., the agent's currently perceived x- and y-axes) and its orientation when the target observation was created and saved. Therefore, the offset allows us to recover directional information so that the target (predicted) observation and displacement can be remapped accordingly.

[0184] This rotation function may be used in the second step 302 of the method 300 when comparing the first observation to the set of stored observations, and may also be used in the fifth step 305 when comparing the second observation and predicted observations to recalculated predicted displacements and observations based on the agent's relative orientation (frame of reference).

[0185] The observation similarity function 403-1 and observation rotation function 403-2 described above can be performed in the same process or can be performed independently. An example of a single process that can perform these functions simultaneously is provided below in the form of example pseudocode for performing both the observation similarity function 403-1 and the observation rotation function 403-2. The pseudocode is as follows: combined_function(cam_observation[16x4x4x256],cam_mask[16x4x4x256],memory_observation[16x4x4],memory_mask[16x4x4],user_mask[16x4x4]) Initialize max_similarity to -1 Initialize the offset to 0 Every 256 rotations: o Initialize cam_sum, mem_sum, and match_sum to zero o Combine cam_mask, memory_mask, and user_mask with a logical binary OR (0 means use element, 1 means don't use element) o For each 16x4x4 (256 total) vector element: ■ If the mask is 0: Add the squares of the elements of cam_observation to cam_sum Add the squares of the elements of memory_observation to mem_sum Add the product of the elements of cam_observation and memory_observation to match_sum ■ Other: Do nothing Normalize match_sum by dividing it by the product of the square root of cam_sum and mem_sum o This is the curr_similarity score for this rotation o If the similarity score is greater than max_similiarity: ■ Set max_similarity to curr_similarity ■ Set the offset to the rotation index Other: ■ Do nothing After considering all rotations, max_similarity is the output similarity, the first observation comparison measure component. The offset after all rotations have been taken into account is the output rotation, i.e., the second observation comparison scale component.

[0186] In the pseudocode above, "cam" refers to the sensor (e.g., camera) and therefore the current observation. Similarly, "mem" and "memory" refer to memory, i.e., the stored target observations to compare against.

[0187] The cam, memory, and user masks function to include and / or ignore specific portions of the current and target observation vectors in the comparison function. The cam and memory masks consist of information about the reliability of information from portions of the visual scene within the observation vector. In the case of a memory mask, this refers to a single observation vector, while in the case of a cam mask, it refers to a set of observation vectors for various rotations of the field of view. This can be used, for example, to filter out portions of the camera image where excessive contrast saturates the input data and reduces data quality. The user mask consists of information provided by a function internal to the system that attempts to select a subset of the complete observation vector elements, or by an external function that does the same. This can be used, for example, to determine the matching of an observation to a particular input function or various portions of the visual scene encoded in the observation.

[0188] The user mask is constructed to remove outliers / erroneous data from the observed data.

[0189] The above pseudocode iterates through each possible rotation of the current observation and identifies the maximum normalized similarity measure across all rotations (first observation comparison measure component) and the rotation offset (second observation comparison measure component) that corresponds to this maximum value.

[0190] In the first and second observation comparison functions 403-1 and 403-2 above, as well as the third comparison observation 403-3 described below, the observation vector o may be created from an image of physical space. This image has the property that the x-axis is the angle around the agent and the y-axis is some property (radial distance or vertical altitude) perpendicular to the x-axis. In the example above, the vector is made up of four channels with sub-pixels (red, green, luma_h, luma_v). There may be more or fewer channels, which may result in a larger or smaller observation vector. Furthermore, the image and each of its channels is made up of a series of rows and columns that provide spatial information about the portion of the space in which the agent resides.

[0191] Thus, the observation vector o can be relatively large, comprising spatial information in multiple directions and color / edge information from different channels. This relatively large vector can be manipulated or sliced ​​to perform various comparisons. For example, using only the edge channels luma_h and luma_v makes the comparison less sensitive to color changes. Using only the green channel can determine changes in vegetation distribution, for example. If all channels are used, but only for spatial subregions of the input image, only corresponding parts of the visual world are compared. This allows for partial comparisons, allowing us to see how well certain parts / features of the world match.

[0192] To perform slicing of the observation vector o, one or more masks are used. Each mask is a vector with the same number of elements as the observation vector o, configured to mask certain regions of the observation vector o while preserving others. For example, a binary mask of 0s and 1s may be used for this purpose. If the observation vector o has 256 elements, the mask will consist of 256 elements. Here, a value of 1 indicates that the corresponding index of the observation vector o should be included / ignored in the subset of the observation vector o being compared. On the other hand, a value of 0 in the mask indicates that the corresponding element of the vector o should not be included / ignored in the subset of the observation vector. Masks may be used to filter out regions of erroneous data, as described above.

[0193] A third method for comparing observations uses an observation action function 403-3 to obtain a third observation comparison measure component. The observation action function can be used to compare observations and can also be used as part of a movement function to move an agent to a second location while testing a hypothesis according to the fifth step 305 of method 300.

[0194] The observation action function relies on the same basic principles as the observation similarity function and the observation rotation function. The observation action function differs from the previous two functions in that multiple subregions of the observation vector o are evaluated and an offset is determined for each subregion. Thus, instead of the entire observation vector o, a subset of the observation vector corresponding to a spatial subregion is evaluated in the observation action function, from which it is possible to obtain an offset value for each subregion corresponding to that subset of vectors relative to the target observation. A mask is used to obtain the relevant subregions. Each subregion may overlap with one or more adjacent subregions.

[0195] The use of subregions for this purpose stems from the notion that not all parts of the visual field change equally as the agent moves toward or away from the predicted observation location. This differs from traditional triangulation in that it considers distortions in the entire visual scene, rather than the movement of identified objects or signal sources. This makes it robust to occlusions in the visual scene and more resistant to environmental changes. Furthermore, it does not require metric calculations to function, making it more robust to measurement noise. This property allows us not only to determine how far the agent is from the target observation location, but also to calculate a further vector to move toward that location. This further vector, calculated by the observation action function, uses the same principle as the observation similarity function, except that dot products are taken for N spatial subregions and compared to the corresponding subregions of the target observation. For example, with N=8, eight spatial subregions would have half the x-axis extent of the entire visual field of the entire observation vector o, each centered in a different direction. In this example, the observation action function combines the observation similarity function and the observation rotation function, running eight times, once for each subregion, to find eight similarities and eight offsets. The eight similarities and eight offsets differ only with respect to specific subregions of the observation vector o and are substantially the same as the first and second observation comparison scale components, respectively.

[0196] Thus, in terms of the actual comparison performed, the stored target observation vector is divided into N subregions, and for each of the N subregions, the corresponding subregion of the current observation vector is compared against all possible rotational permutations to identify which permutation is most similar to the stored target subregion. The particular most similar permutation is associated with an index (e.g., 120) that is used to identify the offset. 番目 (rotation of). An offset is calculated for each subregion of the stored observation vector, and these offsets are used to obtain the action vector needed to reach the position of the target observation. The similarity is used to assess and weight the reliability of this motion vector when it is used in the motion function described below. Thus, the overall output of the observation action function 403-3, and the third observation comparison measure component, is this motion vector.

[0197] Figure 6 shows a schematic diagram of the observation data corresponding to the sub-regions of the observation vectors generated for the purpose of executing the observation action function. Figure 6 also shows the process by which the sub-regions are executed in the observation action function (with respect to the observation data for visualization purposes).

[0198] As shown in FIG. 6, the observation data is represented by a two-dimensional circle 601 of spatial data around the agent 602. This is a representative example, and it should be understood that the observation data may be three-dimensional, such as within a cylinder. The observation action function processes an observation vector o against a target observation. According to the observation action function, multiple subvectors of the observation vector are processed, with each subvector focused or centered on a different direction. In FIG. 6, these directions are shown as multiple directions 604-1 through 604-n in the corresponding observation data 603. In FIG. 6, there are eight directions and eight subregions. However, it should be noted that the process of creating the subregions may select more or fewer directions.

[0199] Each of the eight directions corresponds to eight sub-regions 605-1 through 605-n of the observation data (not all shown). These sub-regions correspond to portions of the field of view (or sensor field if not a camera), and each portion is centered on one of the eight directions. The specific direction associated with a sub-region is called the sub-region direction.

[0200] The observation action function, as described in more detail below, compares pairs of sub-regions 605-1 through 605-n to corresponding portions of stored observations, as shown in observation data 606, which compares a first sub-region 606a and a second sub-region 606b to corresponding portions of a target observation (not shown).

[0201] The observation action function calculates the offset between each sub-region 606a and 606b and the corresponding portion of the target observation, resulting in a pair of offsets 607a and 607b, as shown in observation data 607. From these offsets 607a and 607b, a resulting offset vector 609 is determined, as shown in observation data 608. The sub-regions 606a and 606b used in this process are different sub-regions, and may be, but do not have to be, opposite sub-regions.

[0202] The observation action function 403-3 will now be described in more detail with reference to the sub-regions of the observation data and the corresponding sub-vectors of the observation vector described above and illustrated in FIG.

[0203] As described above, the observation action function uses a set of masked subvectors. Each subvector is a subset of the stored observation vector o and corresponds to a contiguous subregion of the field of view. The subregions of the stored observation vector are compared to corresponding subregions of the current observation vector, which are equiangularly spaced around the agent. Following the previous example, the subregions of the stored vector might cover a set of contiguous columns in the stored 16x4 observation vector. An even number of overlapping subregions are extracted, resulting in N pairs of subregions. The central column of each subregion is projected onto the real world with an opposing vector, as shown in Figure 6. Figure 6 includes eight subregions. These opposing subregions are matched with 256 rotations of the current observation, and the best matching rotation is extracted using the observation rotation function 403-2. Specifically, an offset corresponding to the best matching rotation index between the subvector corresponding to the subregion and the rotated and stored observation is determined for each opposing subregion. The difference in the direction of the offsets of the opposing pairs of observation vector subsets is compared. This provides a vector indicating the direction and magnitude of the displacement required to reach the stored observation (in Figure 6, this is along the resulting offset vector 609, perpendicular to the line connecting the centers of the two opposing subregions 606a and 606b). This direction and magnitude is the direction of where the target observation for comparison was captured. A weighted average of the offset vectors 609 from all opposing subregions provides a single action vector indicating the direction to approach the target observation for comparison from the current observation location. This vector is then rotated from the stored observation rotation to the current observation rotation using a second observation comparison scale component or another rotation measure to align the movement direction with the agent's current world frame. It should be understood that alternative implementations can also be implemented in which the subregion is evaluated in the current observation world frame. While this does not require such rotation, it introduces a higher computational burden since the subregion mask must be evaluated for each current observation rotation, and accuracy is reduced since the exact subregion cannot be evaluated with the stored observation.

[0204] Thus, the observation action function 403-3 can be thought of as a process that iteratively uses the observation similarity function 403-1 and the observation rotation function 403-2 to obtain a vector of the direction of the position of the target observation. Thus, the above observation action function 403-3, the observation similarity function 403-1 and the observation rotation function 403-2 can be performed in the same process, by the same comparison engine 503, or independently.

[0205] An exemplary process for performing observation action function 403-3 using comparison engine 503 is set forth below in the form of exemplary pseudocode. Action_function: For the current observation vector: o Generate a mask to select a spatial subregion of the observations in the following way: ■ Given an observation: [16x4x4], the first two dimensions are space (X, Y) and the third is a different channel ■ For each subpixel spatial element: Select cyclically contiguous subsets of columns within the spatial element. By cyclically contiguous, we mean that for a 16-column dimension, columns 2, 3, 4 and columns 15, 0, 1, 2 represent examples of contiguous subsets, while columns 1, 3, 4 are not contiguous because there is no intervening column 2. These subsets must have the same width (e.g., 8 columns) and be evenly spaced between columns. For example, for four subsets of width 8 columns, indices 0, 4, 8, 12 constitute examples of even spacing when referring to the first, last, or other determined column within each subset. The determined subset is applied as a user_mask, with elements of each column within the subset set to 0 (used) and elements of outer columns set to 1 (not used). For each sub-area: o Execute combined_function (same as above) Subregion max_similarity and offset are saved A set of max_similarity and offset is given, one for each subregion. o For pairs of offsets where the subregions correspond to opposite directions in the agent's field of view (e.g., for 8 subregions, pairs consisting of indices 0 and 4, 1 and 5, 2 and 6, 3 and 7), the circular distance between the first and second indexes of the offsets in the pair is calculated minus 128. This corresponds to the angle between the offsets and how the two subregions moved in the field of view. The direction perpendicular to the pair of subregion directions is the action direction. The offset difference is used to calculate the action magnitude. The action directions and magnitudes of all pairs are vector-averaged to form the final action function output, i.e., the action vector. o The magnitude of the action is the resulting magnitude of the vector described by the offsets (the second offset increments by 128, corresponding to a PI radian rotation).

[0206] Thus, the observation-action function takes N opposing pairs of offsets (N=8), each pair representing a direction described by the perpendicularity of the opposing offset to the direction of the mask center. This direction is combined with a magnitude calculated from the difference between the two offsets; if both offsets are zero (i.e., the center of the subregion of the current observation vector is the same as the subregion of the saved observation), the magnitude is zero. If the difference between the first and second offsets is periodic, then if the first offset is 200 and the second offset is 10 (256 rotations), the difference will be -66, but if the first offset is 10 and the second offset is 200, the difference will be +190. This difference is converted to an angle and can be used to calculate an exact magnitude, or it can be scaled down, for example, by a factor of 16 to provide a rough magnitude for action vector purposes. This can then be used in the translation function to rotate using the second comparison measurement and combined with all pairs of action vectors as a vector average to relocalize the agent to the predicted observation. Vectors can be removed from the average based on a quality criterion, such as a minimum subregion similarity score, with the understanding that due to redundancy in the vectors generated by pairs, parts of the visual scene with large environmental differences will be weighted less compared to parts with small environmental differences.

[0207] A fourth method for comparing observations uses an observation variance function 403-4 to determine a fourth observation comparison measure component. The observation variance function uses similar properties as the similarity function and the action function. The observation variance function 403-4 involves calculating the circular variance or circular standard deviation of the offsets of a subregion around the agent. In effect, this observation variance function identifies a measure of the spread of the offsets on a circle around the agent.

[0208] The observation variance function 403-4 uses the same subregions of the stored vector used by the observation action function 403-3 to determine offsets in the same way with respect to the rotational permutation of the subregions of the current observation vector. For example, suppose there are eight subregions providing eight offsets. The eight offsets represent how different parts of the field of view move as the agent moves toward or away from the target observation location, so that when the agent is at the target observation location, they are all identical and equal to the offsets of a perfect vector match. As the agent moves away, they spread out. As in the observation variance function 403-4, this spread can be measured using a circular statistic, which increases with distance from the observation location. The circular statistic can be the circular variance or circular standard deviation of the eight offsets, providing a scalar value that forms the fourth observation comparison measure component. The circular statistic can be any measure of circular variance.

[0209] To perform the observation variance function 403-4, the observation action function 403-3 is first executed, taking N offsets. At this point, the periodic circular variance of the offsets is calculated after first converting them from the offset index range (0->255) to the appropriate angle range (e.g., radians (0->2*PI)). At this point, subregion offsets that exceed the specified angle from the circular mean angle (e.g., PI / 2 radians) are identified, and the circular variance is recalculated excluding these subregions, as long as more than the specified number (e.g., 4) of subregions remain.

[0210] The observation variance function 403-4 is another way to determine the similarity between the current observation and a stored observation. Unlike the observation similarity function 403-1, the observation action function is highly robust to changes in lighting conditions because it depends on the offset of a subregion rather than the entire observation vector. However, because the subvectors used have lower dimensionality than the full observation vector o, they are prone to discontinuities, and large distances can produce false matches that significantly impact the comparison. For this reason, the observation similarity function and the first observation comparison measure component provided by the observation similarity function provide a better distance measure of similarity.

[0211] Because the circular variance function 403-4 provides a scalar output indicating similarity, it can be used to measure similarity in the same way as the similarity observation function 403-1. Additionally, the circular variance function 403-4 can be used to determine an environmental adjustment factor, which can be used to weight or modify other comparison measure components, such as the first observation comparison measure component from the similarity function 403-1. In particular, if there is high confidence from another observation comparison function, such as the observation action function 403-3, that the agent is at the target observation, then the circular variance function 403-4 should provide a high measure of similarity in the complete space because each offset between the saved vector and the current observation vector should be zero. Therefore, the outputs of other functions can be weighted based on the deviation provided by the fourth observation comparison measure component of the circular variance function 403-4. For example, combining similarity function 403-1 with circular variance function 403-4 means that the first observation comparison measure component from similarity function 403-1 can be modified according to the fourth observation comparison measure component from circular variance function 403-4 to reduce and quantify the influence of environmental conditions.

[0212] Although examples of observation comparison functions 403-1, 403-2, 403-3, and 403-4 are provided above, including numerical examples, it should be understood that these functions can be performed with any selection of number of subregions, masks, channels, rotations, and array sizes.

[0213] The observation comparison measure may include one or more of first, second, third, and fourth observation comparison measure components derived from the observation similarity function, the observation rotation function, the observation action function, and the observation variance function, respectively. For example, an observation comparison measure is generated each time an observation is compared, as will be described in more detail below, when a first observation is compared to a set of stored observations in the second step 302 of method 300, when a predicted observation is compared to a second observation in the fourth step 305-4 of hypothesis testing, and when a transition observation is compared to a predicted observation.

[0214] It should be understood that the observation similarity function, observation rotation function, observation action function, and observation variance function described above are examples of functions that can be used to compare a current observation to a target observation. Other functions or adaptations of the above functions can also be used. Generally, such functions are used to compare observations to determine how similar the current observation is to a saved observation, how rotated / oriented the current observation is compared to the saved observation, and, if the agent is not at the saved observation's location, what direction the saved observation's location is from the current observation.

[0215] Once the observation comparison measures are obtained and hypotheses are determined in the second step 302 of the method 300, the first step in testing the hypotheses is to move the agent to a second location in space in a first hypothesis testing step 305-1. The agent's movement is performed according to a movement function that depends on at least the predicted displacements of the hypothesis subset. In particular, the movement function depends on the predicted displacements required to reach the location of the predicted observation being compared. Therefore, it is necessary to select the correct predicted displacement from the hypothesis subset, which may contain multiple predicted displacements. To ensure that the correct predicted displacement is selected for the movement function, the predicted displacements and predicted observations stored in the hypothesis subset are linked to each other in a network of predicted observations connected by predicted displacements, as described above. The linking of observations to one or more displacements in this network can be stored and maintained in any suitable manner. For example, the set of stored observations and displacements, including the hypothesis subset, may be stored in a cyclic graph, an acyclic graph, or a directed graph. Alternatively, the observations and displacements may be stored with metadata identifying the displacements and observations to which they are connected, or may be organized in a lookup table that maintains the links between each observation and displacement.

[0216] As explained with respect to Figure 3, the hypothesis determination is based on the stored observation that is most similar to the first observation made by the agent. Therefore, the predicted displacement selected for the purposes of the transfer function is the displacement linked to the observation that is most similar to the first observation within the set of stored observations and displacements.

[0217] Once the predicted displacement from the hypothesis subset that is expected to lead from the most similar observation to the next predicted observation in the hypothesis subset is identified, the shift function is updated to be a shift based on the specified predicted displacement.

[0218] It is understood that the predicted displacement has both a magnitude and a direction.

[0219] The movement function may include one or more movement components. The first movement component of the movement function depends on the predicted displacement. In particular, the first movement component is a contribution to the movement function that decreases as the agent moves according to the predicted displacement. Thus, the first movement component is effectively weighted based on the magnitude of the predicted displacement. At the first observation location, the magnitude of the predicted displacement is relatively large, so the first movement component is weighted more strongly accordingly. When the agent begins moving according to the movement function, the magnitude of the predicted displacement is updated based on the actual displacement already traveled by the agent. Thus, as the agent moves toward the end point of the predicted displacement, the magnitude of the predicted displacement decreases, and the first movement component is weighted less and less. In one example, the first movement component is a function of the predicted displacement and is expressed as predicted displacement - traveled displacement. The traveled displacement is the actual displacement estimated using odometry, visual inertial odometry, or the like. In this example, the weighting described above may simply be a coefficient provided by subtracting the travel distance. As the distance traveled increases, the first movement component decreases from an initial value equal to the predicted displacement. In other words, this can be thought of as the predicted displacement "unraveling" as the agent moves along the predicted displacement. Thus, the value of the first movement component at any point in time is a function of the "remaining displacement" of the predicted displacement. This provides a first value that is contributed to the movement function. This first value is scaled according to a scaling factor, so that it can be understood that the contribution of the first movement component to the movement function is appropriate to account for the additional movement component.

[0220] As the agent approaches the end of the predicted displacement, the contribution of the predicted displacement to the movement function tends to zero. In one example, if the measured actual displacement equals the predicted displacement and the entire predicted displacement is moved, the first movement component based on the predicted displacement will be zero and will no longer contribute to the agent's movement function. If the first movement component is the only movement component of the movement function and the movement function depends only on the predicted displacement, the agent's position after moving the predicted displacement will be the second location, where the second observation is recorded.

[0221] Alternatively, the first movement component may stop contributing to the movement function before the entire predicted displacement has been traveled. In particular, the movement function may be configured to ignore the first movement component if its magnitude—in other words, the remaining distance of the predicted displacement—is within a displacement threshold distance from the end point of the predicted displacement. This is advantageous because it reduces the impact of drift on the agent's movement. In particular, odometry-based estimation of actual displacement is susceptible to drift-related errors, which can result in an estimate of actual displacement (actual distance traveled) that is smaller than the true value. Second, high style and low configuration indicate that the observations made by the agent are very similar / identical to the predicted observations in the hypothesis subset, but the actual displacements made by the agent are not similar / identical to the predicted displacements in the hypothesis subset. "Very similar" means, with respect to the fifth step of hypothesis verification 305-5, that the observation comparison measure exceeds the second high observation threshold level, but the displacement comparison measure does not exceed the second high displacement threshold level. Therefore, even if the estimated actual displacement is smaller than the predicted displacement, the agent may have already traveled the entire predicted displacement. To avoid overshooting the predicted displacement endpoint, the first movement component may be stopped before the predicted displacement endpoint based on a displacement threshold distance. The displacement threshold distance can be set based on the particular environment or scenario in which the agent is used. In physical space, this could be 5 cm, 10 cm, 1 m, 5 m, 10 m, etc., appropriate value depending on the scenario.

[0222] The second movement component that may be included in the movement function depends on the observation-action function described above with respect to observation comparison. When used in the movement function, this second movement component requires continuous, repeated, or sequential transition observations between a first location in space where a first observation is made and a second location in space where a second observation is made. In other words, transition observations are made as the agent moves away from the first location.

[0223] Thus, one or more transition observations are made along the agent's travel path. Each transition observation is a type of "current" observation that is compared to the predicted observation that the predicted displacement is expected to lead to according to the hypothesis. From each comparison, an action vector is generated, using the observation-action function described above, that defines both the expected magnitude and direction to where the predicted observation may be found. The action vector forms the second movement component of the movement function. Similar to the first movement component of the movement function, the contribution of the second movement component may be scaled and weighted by a scaling factor. In particular, the second movement component may be weighted such that the more the action vector of the second movement component indicates that the agent is closer to the location of the predicted observation, the stronger the contribution of the second movement component to the movement function. For example, the magnitude of the second movement component resulting from the observation-action function may be configured to increase as the agent approaches the location of the predicted observation. This increase in magnitude may directly provide a weight that makes the second movement component stronger as the agent approaches the location of the predicted observation.

[0224] The second movement component may be used alone as the sole contribution to the movement function, or in combination with the first movement component. Advantageously, when both the first and second movement components are included and both contribute to the movement function, the agent's movement depends on both the predicted displacement and the transition comparison between the predicted observation and the transition observation. This means there are two separate data sources used to move to and position the predicted observation in space. This effectively provides redundancy and is therefore more reliable and accurate than using only one set of data. Furthermore, the first and second movement components are advantageously complementary to each other in that the relative weighting of the first movement component weakens its contribution to the movement function as the agent approaches the end of the predicted movement, while the relative weighting of the second movement component strengthens its contribution to the movement function as the agent approaches the location of the predicted observation. Thus, the first and second movement components behave inversely.

[0225] If the hypothesis is correct, or at least partially correct, or similar, If the attributes associated with the hypothesis are at least similar to the attributes of the space the agent occupies, then the endpoint of the predicted displacement should be at least close to where the predicted observation is found. Thus, having a first movement component that depends on the predicted displacement and a second movement component that depends on the predicted observation serves to move the agent toward the location of the predicted observation according to two mechanisms and dimensions.

[0226] The relationship between the first and second movement components in terms of their contribution to the agent's movement function may take any suitable form. An example of the relationship between these two movement components is described with reference to Figure 7, which shows a schematic map of an agent moving between a first location and a second location in space.

[0227] In a first example of the relationship between the first and second movement components of a movement function, as shown in the first graph 701, the relationship is simple and depends only on the relative weighting of the first and second movement components. As the agent moves away from the first location 701-1, the relative weighting of the first and second movement components decreases and increases, respectively, so that as the agent moves closer to the first location 701-1, the first movement component naturally dominates in terms of contribution to the movement function. This is useful because, generally, when the agent is located far from the predicted observation, the observation action function responsible for the second movement component provides only a weak indication of where the predicted observation is in space. This is because the transition observations made for the purposes of the observation action function are typically very different from the predicted observations, as the distance between the capture points of these observations is likely large and the observations are therefore in different regions of space. Because the observation action function relies on comparing observations and identifying similarities between them, it is not very reliable when the observations differ significantly. Therefore, the fact that the first movement component dominates when the agent is close to the agent's first location 701-1 is advantageous because it allows the predicted displacement to be used to approximate where the predicted observation should be, which is more reliable than using only the second movement component.

[0228] In the first graph 701, as the agent moves, a series of transition observations 701-2 are made, and the second movement component, specifically the observation action function responsible for the second movement component, is periodically, continuously, or repeatedly updated. The agent follows the movement function along a path 701-3, moving to a second location 701-6 in space. As the agent approaches the end point of its displacement from the first movement component, the second movement component from the observation action function becomes dominant, relocalizing the agent to the second location 701-6. It should be understood that the second location 701-6 is not known before the agent moves; rather, the second location 701-6 represents a location in space that is identical to or best matches the predicted observation (target observation) linked to the predicted displacement. The second location 701-6 is determined by the location of the second observation from the observation action function, and the magnitude and direction of the observation action function vector indicate that the second location 701-6 is the location of the predicted observation or its best match.

[0229] In a second example of the relationship between the first and second movement components of a movement function, the relationship is sequential, with the movement function initially depending only on the first movement component, and once the first movement component expires, depending only on the second movement component, as shown in second graph 702. This relationship is useful because it avoids the need to observe the progress of most of the agent's movement between first location 702-1 and second location 702-6.

[0230] This is because the first movement component depends on the predicted displacement, not the predicted observation. According to this relationship, the agent is configured to move from the first location 702-1 via a first path 702-3, corresponding to movement according to only the first movement component, based on the predicted displacement. When the predicted displacement is exhausted and the agent reaches the end of the predicted displacement at the intermediate location 702-5 or reaches a predetermined threshold from the end of the displacement, the movement function switches from using the first movement component to using the second movement component. At this point, the agent makes sequential or iterative transition observations 702-2 and executes an observation action function to obtain a magnitude and direction vector pointing to the location of the predicted observation. Using this information, the agent follows a second path 702-4 toward the second location 702-6. Thus, in the second relationship, the agent actually moves by the first displacement before relocating using the observation action function.

[0231] In a third example of the relationship between the first and second movement components of a movement function, as shown in the third graph 703, this relationship is a winner-take-all relationship, where the movement function depends only on the highest-weighted component of the first and second movement components at a particular time and location in space along the path to the second location. As noted above, the first movement component, which depends on the predicted displacement, naturally has a higher weighting than the second movement component, which is closer to the first location. Therefore, the agent moves along a first path 703-3 based on the first movement component and therefore the predicted displacement. During this movement along the first path 703-3, the agent performs a series of transition observations 703-2 with the goal of updating the second movement component. The goal is to determine when the second movement component becomes dominant over the first movement component. The frequency of transition observations along the first path 703-3 is less important than the initial relationship shown in the first graph 701. This is because the transition observation 703-2 of the first path 703-3 is made only to compare the first movement component with the second movement component, not to guide the agent itself. Increasing the frequency simply increases the accuracy with which the second movement component is determined to be the "winner" in the winner-take-all relationship. At point 703-5 along the path, this determination occurs when the second movement component is determined to be dominant over the first movement component. At this point, depending on the observation action function, the second movement component is used in the movement function to relocate the agent to the second location 703-6. In this phase, the agent follows the second path 703-4 and makes further transition observations 703-2. This is necessary to obtain a vector from the observation action function to provide the direction and magnitude of the agent's movement to the second location 703-6.

[0232] Therefore, the transfer function uses two components to relocalize to the location of the predicted observation. The above three relationships between these two components are exemplary, and any suitable method or combination thereof may also be suitable.

[0233] The ability of the movement function to relocalize the agent to the location of the predicted observation depends on the observation-action function, which depends on whether similarities are found between the current observation and the predicted observation. Therefore, if the similarity between the observations falls below a certain level, the action function will not relocalize. In this case, the agent may move according to the first movement component of the movement function based on the predicted displacement. If it is determined that the agent cannot be relocated because there is no similarity between the current observation (transition observation) and the predicted observation, the predicted observation is not found at or near the end of the predicted displacement, and the hypothesis is rejected. In this case, the agent moves in the opposite direction along the predicted displacement, returning to the location in the space of the previous observation from which the hypothesis was determined. The hypothesis is then adjusted to another hypothesis associated with a different hypothesis subset of the set of stored observations and displacements.

[0234] Thus, the movement function provides the agent with the ability to move to a location where the predicted observation is found, a "best match" in space for the predicted observation (e.g., if the agent relocalizes to a location that is not an exact match but is still similar to the predicted observation), or a location where the predicted observation is not found. Because the movement function may be two-component, discrepancies between the predicted and actual displacements may occur. For example, if the best or perfect match between the predicted and transition observations is at the end of the predicted displacement, the agent may not need to relocalize using the second movement component. Alternatively, the agent may need to relocalize by moving slightly so that the actual movement distance is equivalent to the predicted movement distance. Finally, the second movement component may relocalize the agent to a larger area compared to the predicted displacement. In this case, the predicted displacement is not similar to the actual displacement.

[0235] These possibilities ultimately provide the method with a way to determine how similar a series of observations and displacements performed by the agent are to the hypothesized attributes in two separate dimensions. This ultimately helps determine whether to adjust, maintain, or confirm the hypothesis as described above. Specifically, the first dimension is the similarity between the current observation (or the observations for which the method 300 continues or repeats for two or more observations) and the hypothesized subset of stored observations. This similarity is determined by an observation comparison measure and is referred to as "style." The second dimension is the similarity between the current displacements (or the displacements for which the method 300 continues or repeats for two or more displacements) and the hypothesized subset of stored displacements. This similarity is determined by a displacement comparison measure and is referred to as "configuration." The displacement comparison measure is determined by comparing actual displacements with predicted displacements.

[0236] Generally, there are four possible combinations of style and composition: high style and high composition, high style and low composition, low style and high composition, and low style and low composition. We'll discuss in more detail below what each of these possibilities means for our hypothesis.

[0237] First, high style and composition indicate that the agent's observations are highly similar / identical to the predicted observations in the hypothesis subset, and the agent's actual displacements are highly similar / identical to the predicted displacements in the hypothesis subset. "Highly similar" means that both the observation comparison measure and the displacement comparison measure exceed the second higher observation threshold level and the second higher displacement threshold level, as explained above with respect to the fifth step of hypothesis testing 305-5. If the displacement and observation comparison measures match or exceed both of these levels, the space in which the agent resides is determined to contain attributes consistent with the identity defined by the hypothesis subset. As previously mentioned, identities do not need to be labeled, but labels can be associated that define the attributes associated with the identity. If the attributes of a space match the hypothesis subset of multiple consecutive predicted observations and displacements, confidence that the attributes in the space match the identity increases. Therefore, determining whether the attributes of a space match the identity may require comparing multiple consecutive predicted observations and displacements to a sufficient level of match.

[0238] Second, high style and low composition indicate that the agent's observations are very similar / identical to the predicted observations of the hypothesis subset, but the agent's actual displacements are not similar / identical to the predicted displacements of the hypothesis subset. "Very similar" means, with respect to the fifth step of hypothesis verification 305-5, that the observation comparison measure exceeds the second high observation threshold level, but the displacement comparison measure does not exceed the second high displacement threshold level. In terms of the space in which the agent resides, this means that, from the perspective of the sensor (observation) point data, the space appears the same or very similar in terms of the attributes defined by the hypothesis subset, but at least one of the attributes—size, location, and / or distance between—is not as predicted according to the hypothesis. This relationship is called "context." In the above example shown in Figures 1 and 2, the context would represent a similar-looking room with similar furniture as in Figure 1, but perhaps the room is larger and the furniture arrangement has been altered (e.g., a central element has been moved to one side). In Figure 2, context may be found if the chair is photographed from the back or if the chair's proportions are changed (e.g., the legs are shortened or the back is lengthened).

[0239] Third, low style and high composition indicate that the agent's observations are not similar / identical to the predicted observations in the hypothesis subset, but the agent's actual displacements are very similar / identical to the predicted displacements in the hypothesis subset. "Very similar" means that the displacement comparison measure exceeds the second higher displacement threshold level, but the observation comparison measure does not exceed the second higher observation threshold level, as per the fifth step of hypothesis verification 305-5. In terms of the space in which the agent resides, this means that, according to the hypothesis, the size, location, and / or distances between points in that space appear very similar, but do not appear / do not appear to share the same attributes defined by the hypothesis. This relationship is called a "class." In the examples of Figures 1 and 2 above, classes are represented by rooms of the same size and structure, but may differ in color and furniture. In Figure 2, classes may be found when chairs differ in color or when the style of details differs while maintaining the same proportions, such as several types of chairs designed for eating at a table. In terms of navigation, these may be different floors of a building, with identical floor plans but different companies providing the decoration. Multiple observation comparison measures may be used to determine class relationships between attributes in the space in which the agent resides and stored observation data. While the similarity indicated by a first observation comparison measure component may be low, a second and / or third observation comparison measure component may be used from the observation similarity function 403-1. If the observation action function does not provide convergence to a point, there is no match and the basis for comparison is invalid, in which case the hypothesis is rejected.

[0240] Finally, low style and composition indicate that the agent's observations are not similar / identical to the predicted observations in the hypothesized subset, and the agent's actual displacements are not similar / identical to the predicted displacements in the hypothesized subset. In this case, with respect to the fifth step of Hypothesis Testing 305-5, the displacement comparison measure does not exceed the second higher displacement threshold level, and the observation comparison measure does not exceed the second higher observation threshold. In terms of the space in which the agent exists, this means that the space does not appear to be related to the structure or appearance of the attributes related to the hypotheses.

[0241] The hypothesis may be maintained for further comparison or rejected and adjusted as previously mentioned.

[0242] By comparing the current observation and displacement with the stored observations and displacements for one or more hypothesis subsets, it is possible to identify characteristics of a space in terms of two dimensions: composition and style (structure and appearance). If a space is repeatedly or periodically evaluated by an agent, these dimensions can also be used to determine changes in the space. In this scenario, the stored hypothesis subset may include observations and displacements taken directly from the same space in previous interactions by the agent. Using this information, the agent can detect what has changed in the space over time in terms of structure and appearance. Identifying differences is useful, for example, for path planning applications and alternative route searches. The agent can store the average displacement to a particular stored observation, which is updated each time the agent displaces to the observation. This average displacement provides a historical average of past displacements taken for a particular observation. Additionally, the standard deviation of the previous displacements taken by the agent for a particular observation may also be stored. Statistical analysis can be performed using the average displacement and standard deviation to determine the likelihood that the space has changed. For example, if the new displacement for an observation differs significantly from the average displacement and / or differs from the standard deviation of the previous displacements, it is determined that the space has likely changed. This may occur, for example, when spatial attributes are moved.,To perform statistical analysis, a historical record of each,observation and displacement can be kept, including the mean displacement,of the observation and the standard deviation.,This historical record may be updated with each exploration,of the agent.

[0243] A second history record can be stored and updated in a similar manner. The second history record relates to the N most recent observations and displacements made by the agent as it moves to a particular observation point. Specifically, it relates to the similarity or overall similarity score between these N most recent observations and displacements and the corresponding stored observations and displacements to which they are compared. Thus, the second history record represents a localization confidence, indicating how confident the system is that the agent was localized to the stored observations and displacements on its way to a particular observation. If the second history record indicates that the N most recent observations made by the agent closely match the N most recent stored observations, then confidence can be gained that the agent is properly localized to the subset or path it is following, in terms of observations and displacements. If the displacements made for a particular observation, or the particular observation itself, differ significantly from the corresponding stored observations and displacements, then the second history record can be used to reliably indicate and determine that the space has changed in some way with respect to that particular observation and displacement. For example, if N=5, then the last five displacements made by the agent may perfectly match the stored displacements. If the last six displacements made are significantly different from the stored ones, then the second history record can be used to reliably indicate and determine that the space has changed in some way with respect to that particular observation and displacement. For example, if N=5, then the last five displacements made by the agent may perfectly match the stored displacements. 番目 The displacement of 6 is preserved. 番目 If the displacement differs from the previous one, the fact that the second historical record shows a high level of agreement (in this case, identity) with the previous five displacements indicates that the 番目 This means that it is easy to determine that the space has changed in terms of displacement. This may be due to the movement of obstacles etc.

[0244] Furthermore, as described above, while hypotheses may be confirmed, maintained, or rejected based on observations or displacements separately, it is advantageous to use both observations and displacements, i.e., composition and style, when determining the outcome of a hypothesis. Adjusting a hypothesis by combining the observation comparison measure and the displacement comparison measure to form a two-dimensional similarity measure may include adjusting the hypothesis if the two-dimensional similarity measure falls below a first two-dimensional similarity measure threshold level. Maintaining a hypothesis may include maintaining the hypothesis if the two-dimensional similarity measure exceeds a first two-dimensional similarity measure threshold level but does not exceed a higher second two-dimensional similarity measure threshold level. A step of confirming the hypothesis may occur if this second two-dimensional similarity measure threshold level (also referred to as a hypothesis-confirming two-dimensional similarity measure threshold level) is exceeded. If there is a change in the visual identity of an observation encountered at a different time, the agent may use the current observation information to update the stored observation if it is confident that the current observation and past observations were captured with the same identity. Because both observations are represented by normalized vectors, we can average the two vectors, applying optional weighting to each, so that the final observation closely matches either the past or the current observation vector. For example, multiplying the past observation by 5 and the current observation by 1 and renormalizing the sum vector will make the final observation vector more closely match the past vector.

[0245] Adjusting a hypothesis involves rejecting a hypothesis and returning to a previous observation to identify a second hypothesis. In particular, if, for any observation, the current observation does not match the predicted observation with adequate confidence and / or the actual displacement does not match the predicted displacement with adequate confidence, the hypothesis is rejected. The agent is then configured to return to the previous observation by taking a movement opposite to that taken to reach the current observation. This may include, but is not required to, returning to the first observation. The method 300 then iterates to test a different hypothesis selected based on a comparison of the previous observation with each of the multiple hypothesis subsets of the predicted observation. Optionally, the agent may store the observation and displacement even if the observation and displacement do not match the hypothesis, which is then rejected. Saving these observations and displacements can be useful in generating a map of the space for future navigation and attribute identification purposes. Thus, the agent may make and store one or more observations and displacements before returning to the previous observation, even if a hypothesis is rejected.

[0246] Verifying an observation includes, but is not limited to, identifying or locating an object, image, or destination within the space. Once an attribute is found, a reference to the attribute (including the agent's observations and displacements within the space) is stored in memory. These may form part of a map of the space in which the agent resides. If a hypothesis is associated with a labeled identity or attribute, the agent may also store a reference to the label in memory, along with the observations and displacements it made within the space. If a hypothesis is based on a previous visit to the same space, the hypothesis subset may be updated with the set of displacements and observations the agent made recently as it validates the hypothesis, since the hypothesis subset effectively defines the attributes previously observed in the space. In this way, the hypothesis subset is kept up-to-date with respect to a particular space.

[0247] It should be understood that the above method may be performed by a single agent or by one or more agents working together. When more than one agent is included in a space, the one or more agents may communicate with each other or access the same memory. For example, multiple agents may each be linked to the same server or computer system. Multiple agents may simultaneously update a map of the space or test one or more hypotheses simultaneously. Map generation is described in more detail below.

[0248] The above description describes a method 300 for analyzing a space and identifying attributes of the space. The attributes may be objects or images, in which case method 300 is used for object or image recognition purposes. Alternatively, the attributes may be destinations, in which case method 300 is used for navigation purposes. Method 300 relies on an agent making observations and displacements in the space and comparing them to stored observations and displacements. Thus, one principle of method 300 is the agent's ability to compare observations and displacements. This is illustrated in Figures 4-6 and the above description. Another principle is the agent's ability to move, which is illustrated in Figure 7 and the above description.

[0249] Further aspects will now be described that complement the method 300. These aspects may be incorporated into the method 300 or may be performed or implemented separately.

[0250] A first additional aspect is the generation and storage of linked observations and displacements that form identities and hypothesis subsets. This can be thought of as a "training process," as briefly mentioned above. The process of forming identities can be performed specifically for the space in which the agent is expected to reside, or it can be performed in another space. Both of these implementations have advantages. If identities are formed in the same space the agent is later configured to explore, it becomes possible to identify how attributes of that space change over time after further exploration of that space. If identities are formed in separate spaces, it becomes possible to identify when attributes from one space recur in another space, or to identify similarities and differences between spaces.

[0251] The process of forming and storing an identity will now be described with reference to FIG. 8. FIG. 8 shows a schematic diagram 800 of an identity. It is important to note that no observations or displacements are stored. An agent is "dropped" or activated into a previously unknown and unexplored space. An initial observation o1 is made within the space. The initial observation o1 may be based on low-level features detected within the space, but this is not required. The use of low-level features will be discussed separately below as a further complementary aspect. The selection of the initial observation can be random or based on a learned process. From the initial observation o1, an initial displacement is made from the location of o1 to make a new observation o2, the displacement between o1 and o2 is stored in memory as d12, and the inverse displacement from o2 to o1 (the negative value of d12) is stored in memory as d21.

[0252] Optionally, an additional effort value is stored that is a better measure of the characteristics being optimized in the routing (such as energy usage and / or vibrations encountered).

[0253] Next, a sequential set of observations Oi{oi,1...oi,n} and a corresponding set of displacements Di{di,1...di,n-1} are traveled by the agent and stored in the same manner. This forms an identity. The set of observations and displacements that form the identity describes a predicted path through space and the expected environment at points on the path. Traveling the path allows the agent to verify the prediction (i.e., hypothesis) by comparing the measured displacements and observations with the predictions one by one. The sequential set of observations and displacements that form the identity are stored as described above. In some instances, an identity can be stored as a subset of a larger stored set, where the larger stored set contains multiple subsets, each defining a path through the same or different spaces. Thus, an identity may be linked to other identities, such that the identities share one or more common observations and / or displacements.

[0254] As shown in Figure 8, a set of observations o1 through o4 and their corresponding displacements can be captured in a closed loop to form an identity, which can then be validated against future observations and displacements in sequential order for successive subsets or the entire set using the style and composition dimensions described above. This identity can be used for purposes such as locating or identifying specific objects, extracting common repeating classes of objects or navigational terrain, including but not limited to building hallways and rooms, and extracting contexts, which are areas or objects with similar components (e.g., forests / offices / roads or wheeled vehicle characteristics / flying objects).

[0255] While or after the identities are stored as a subset of the observations and displacements, a further optional process may include using another system to determine the attributes with which the identities are associated. For example, the identities may be associated with a path for purposes of navigation to a destination in space. Alternatively, the identities may be associated with an object such as a chair, or an image such as a cartoon chair or a painting of a chair. To correctly associate the identities with these attributes, a training process may be used. The training process may involve identifying attributes via an established training algorithm, such as using training data to train a classifier. It should be understood that any machine learning process may be used, and may include the use of neural networks, etc.

[0256] A second further aspect is the formation of a map of the space the agent resides in. While the agent may be controlled to test hypotheses to identify attributes of the space it resides in according to the methods described above, it may also / alternatively be controlled to generate a map of the space it resides in. It will be appreciated that map generation can be performed independently or in combination with identity generation and attribute determination / hypothesis testing.

[0257] The agent can generate a new map if the space is completely unknown from the agent's previous explorations, and can update an existing map in memory if the space has been visited by the agent before.

[0258] To generate the map, the agent makes a first observation in space, which may be the same as the first observation made in the first step 301 of the hypothesis testing method 300 described above. The first observation is stored in memory as an indication of a corresponding "first location" on the map.

[0259] The agent is then configured to move away from the first location and make one or more further observations.

[0260] When a map is being created or updated in conjunction with an agent performing hypothesis testing, the agent moves according to the movement function described above. In this case, the agent is configured to make one or more new observations as it leaves a first location. The new observations may be made while the agent is moving according to the movement function (during a transition) or may be made afterward. In this regard, the agent may wait until it arrives at the second location of the second observation of method 300 described above before making a "new" observation, in which case the second observation becomes the new observation.

[0261] A new observation is compared to a first observation using one or more observation comparison functions described above, providing one or more observation comparison measures. The observation comparison measures are compared to a mapping threshold, which defines the maximum probability that consecutive observations must fall below to be recorded on the map. In other words, the method for generating the map requires that the observations stored in the map be dissimilar. This avoids adding unnecessary points to the map, making the map more concise and minimizing the memory required to store and access the map. If the observation comparison between the first observation and the new observation results in an observation comparison measure lower than the mapping threshold, the new observation is stored in memory at a "new location" on the map. The displacement between the first observation and the new observation is measured and / or obtained from the displacement function and is also stored in the map, along with the inverse displacement from the new observation to the first observation. When a map is generated while hypothesis testing is also being performed, the agent performs two comparisons: the first, comparing the current observation to the predicted observation (to test the hypothesis), and the second, comparing the current observation to a previous observation (e.g., the first observation) to determine whether these observations are dissimilar enough to be included in the map. Thus, even if a hypothesis is rejected and the current observation does not match the predicted observation well enough, the agent can still make an observation or use the current observation for the purpose of adding to or updating the map by comparing it with previous observations.

[0262] If the mapping threshold is exceeded, the new observation is not stored on the map, instead the agent is configured to move again at a further displacement before making further observations, until the mapping threshold is no longer exceeded.

[0263] Once a new observation is stored on the map, along with the displacement between the first and the new observation, the new observation effectively becomes the first observation (or a previous observation), and the process is repeated for further displacements and further observations. This process iteratively builds a connected network of observations that forms a map describing the space in which the agent resides.

[0264] A map of a space can also be generated independently of identifying the spatial attributes. The method for generating the map is largely the same. An agent makes a first observation and stores it in memory as a first location on the map. The agent then moves from the first location of the first observation in an unexplored, physically or logically traversable direction. The agent's movement is recorded as a first displacement. As described above, the agent makes one or more new observations away from the first location and compares the first and new observations to determine whether they exceed the mapping threshold. If they do not, the new observation is stored in the map, linked to the first and new observations by the first displacement and the inverse of the first displacement. The relative direction between the first and new observations is marked as "explored." The process then repeats with further displacements and observations from the new observation location in unexplored directions. The use of a mapping threshold means that the number and density of observations made in a particular space depend on the complexity and features of that space. In feature-rich environments, more observations are made because the distance at which similarity between observations falls below the mapping threshold is shorter and decreases more quickly. In featureless environments, the space is less complex and therefore fewer observations are required. Therefore, methods for mapping and / or identifying attributes are implemented in such a way that they automatically adapt to the constraints of the space in which the agent is observing or moving. These methods are therefore sensitive to the complexity of the environment.

[0265] Multiple agents may contribute to the same map. In particular, multiple agents may each contribute to and access a shared database or memory, and may be configured to update the shared database with sets of observations and displacements to generate the map. As hypotheses are tested and attributes are identified, identities associated with the attributes are stored on the map and, if possible, labeled as such. Thus, a map may include multiple identified attributes and stored labels for those attributes. Agents may contribute to the generation of the map in real time or near real time, each communicatively coupled to the shared database via appropriate means. For example, in a physical space, multiple agents may be a fleet of drones, robots, vehicles, etc., each capable of communicating with a computer system or a distributed computer system, such as multiple servers.

[0266] This map can then be used to navigate the space: in particular, a destination on the map can be selected as the destination to which an agent should move, this destination being represented by an observation that forms part of the map.

[0267] Map navigation is It functions similarly to the method 300 for testing hypotheses and determining spatial attributes. When navigating a map, the attributes are essentially destinations. The navigation process is described in more detail here.

[0268] A map is generated as described above and stored in memory, forming a set of stored observations and displacements, at least a subset of which are connected or form a network with the observations associated with the destination. This map is thus similar to the hypothesis subset described above.

[0269] The agent is configured to make a first observation in space and identify which of the stored observations of the stored map the agent is closest to. In particular, the first observation is compared to each observation connected to the destination in the stored map. From these comparisons, an observation closest to the first observation is identified. The agent is then moved to relocalize to the location of the closest matching observation.

[0270] From the closest matching observations on the map, a route is planned using a routing algorithm. Any suitable routing algorithm can be used, such as the Dijkstra algorithm, which determines a route that minimizes distance, effort, vibration, or other measurements that can be obtained from the displacements stored on the map.

[0271] Once a route is planned, the agent moves sequentially along the route, and the process is repeated as the agent moves from observation point to observation point on the map. The agent starts this process from the "closest matching observation," and the route may contain multiple observations between the closest matching observation and the destination, for example. The observation where the agent is currently located is the current observation, and the next observation on the map along the route to the destination is called the target observation.

[0272] The iterative process involves moving the agent toward the target observation using a stored displacement vector between the current observation and the target observation. This is similar to moving the agent according to a movement function depending on the predicted displacement, as described above with reference to the hypothesis testing method 300. Similar to hypothesis testing, this process also involves performing one or more observation comparisons using one or more of an observation similarity function, an observation rotation function, and an observation action function. For example, the stored displacement to reach the target observation can be used in combination with an observation action function to move the agent from the current observation to the target observation. Thus, the agent can make multiple transition observations while moving between the current observation and the target observation and then use the observation action function to relocalize to the location of the target observation. This is similar to the movement function including a first movement component and a second movement component described above, and can account for odometry drift and uncertainty when changing movement direction and reconstructing the displacement between the current observation and the target observation.

[0273] When the observation comparison measure (or measures) exceeds a predetermined or continuously calculated threshold, the agent is determined to have reached the target observation. This is similar to determining the agreement between the second observation and the predicted observation in hypothesis testing method 300. Once the agent is determined to have reached the target observation, the process is repeated for the next observation on the route to the destination until the destination is reached. Thus, the target observation becomes the current observation, the next observation becomes the target observation, and the displacement traveled becomes the displacement between the current observation and the target observation. Once the agent reaches the observation associated with the destination, the agent is configured to wait at that location unless instructed otherwise.

[0274] In this way, the observation, displacement, and movement functions can be used to identify attributes and features of a space, generate a map of the space, and navigate the space. It is important to note that one or more, or all, of these processes may be performed simultaneously. For example, an agent can be instructed to follow an existing route to a destination, identify attributes and features along the way, and add them to the existing map using transition observations. Thus, the agent can simultaneously learn and navigate its environment.

[0275] In the above-described methods and systems for analyzing a logical or physical space, including identifying attributes / features of the space, generating a map of the space, and navigating the map of the space, an agent is configured to make and compare observations and navigate between these observations. As noted above, the logical or physical space need not be known to the agent or have been previously visited by the agent. Thus, there are several sets of initial conditions that the agent may encounter when placed in the space.

[0276] In a first example, the agent is located in a previously visited space, and the agent has access to a saved map or saved subset containing observations and displacements made within that space. In this case, when making a first observation, the first observation may be identical to or match a saved observation (such as a predicted observation in a hypothesis subset or an observation along the route to the destination). In this case, there is no need to relocalize the agent or navigate the route to the saved destination before performing method 300 to test the hypothesis.

[0277] In a second example, an agent is placed in a previously visited space, and the agent has access to a saved map or saved subset containing observations and displacements made within that space. In this case, when making a first observation, the first observation may not be identical to any of the saved observations in the map or saved subset. Because the agent's location is unknown, the saved displacement cannot be used to relocalize the agent onto the saved observation. Rather, the agent must first relocalize to the saved observations without using a predicted displacement to understand its location on the map, or relative to the subset of saved observations and displacements. To do this, the agent compares the first observation with each saved observation and uses an observation-action function to determine a vector to one of the saved observations (referred to here as the target observation). The agent then moves along this vector, repeatedly or continuously updating further transition observations, and relocalizes to the target observation.

[0278] The selection of which stored observation becomes the target observation can be determined in any reasonable manner. In one example, the stored observations may be compared to the first rotation for each rotation using an observation similarity function. The most similar stored observation may then be selected for comparison with the first observation to determine an action vector from the observation-action function.

[0279] Alternatively, an action vector can be determined for each stored observation and the strongest one selected to relocalize the agent to the stored observation. The action vectors are determined from the observation-action function as described above and are weighted based on the similarity between a subregion of the stored observation vector and the corresponding dub region of the current observation vector, or based on an overall similarity measure, making them stronger.

[0280] In a third example, an agent may be placed in a space not previously visited, in which case the agent has access to a saved subset containing observations and displacements from a different space or multiple spaces. In this case, when making a first observation, the first observation may not be identical to any of the saved observations in the saved subset. Because the agent's location is unknown, the saved displacement cannot be used to relocalize the agent onto the saved observation. Rather, the agent must first relocalize to the saved observations without using a predicted displacement to understand its location relative to the subset of saved observations and displacements. To do this, the agent compares the first observation to each saved observation and uses an observation-action function to determine a vector to one of the saved observations (referred to here as the target observation). The agent then moves along this vector, repeatedly or continuously updating further transition observations, and relocalizes to the target observation.

[0281] The selection of which stored observation becomes the target observation can be determined in any reasonable manner. In one example, the stored observations may be compared to the first rotation for each rotation using an observation similarity function. The most similar stored observation may then be selected for comparison with the first observation to determine an action vector from the observation-action function.

[0282] Alternatively, an action vector can be determined for each stored observation and the strongest one selected to relocalize the agent to the stored observation.

[0283] However, unlike the first and second examples above, in this third example, the subset of stored observations and displacements is from a different space than the space in which the agent currently resides. Therefore, the space in which the agent resides may not contain any stored observations. In this case, the agent may be unable to relocalize to the stored observations. If the agent is testing a hypothesis, and the subset of stored observations and displacements is a hypothesis subset, the agent may reject the hypothesis associated with the hypothesis subset if it is unable to locate to the stored observations in the hypothesis subset. The agent can then use a negative displacement to return to the location in space of the first observation. From here, the agent can select another hypothesis and the corresponding hypothesis subset to test. The agent can repeat this process as necessary for all stored hypotheses and hypothesis subsets. If all hypotheses are rejected—that is, if the space does not contain any recognizable features or attributes corresponding to the subset of stored observations and displacements—the agent enters exploration mode, making observations and displacements to generate a map of the space, as described above.

[0284] In the fourth example, the agent is placed in a space that has not been visited before, and the agent does not have access to a saved subset containing observations and displacements. In this case, the agent enters exploration mode as described above, mapping the unknown space in which the agent resides, and ultimately saving a set of observations and displacements to describe the space.

[0285] The above description focuses on agents that perform sequential observation and displacement to perform one or more of the following: generating a map, navigating a route in space, or identifying spatial features. While it is possible to perform these methods using only observation and displacement, additional elements may be incorporated to gain further efficiency and accuracy benefits.

[0286] In particular, low-level features of space may be identified or detected and used to inform the process of observation and movement within the space between observations. In particular, observations by an agent contain spatial information that describes the spatial environment in the agent's local neighborhood. Therefore, features can be extracted from observations or observation data to obtain more information about the agent's neighborhood. For example, environmental cues such as high-contrast lines, colors, edges, and shapes can be extracted to determine more information about the environment and inform decision-making and actions. For example, in a map generation where an agent is configured to explore a space by sequentially making observations and moving, the selection of the direction to move the agent after each observation may be controlled by low-level features. Specifically, after each observation, the agent's selection of the next movement direction may be weighted based on low-level features and / or previously navigated directions. For example, a first weighting can be applied to directions that move the agent closer to previous observations as it moves. Because the purpose of map generation is for the agent to explore and map the space, it is beneficial to encourage the agent to move away from previous observations, and this first weighting achieves this. A second weighting may be applied to directions based on low-level features. For example, if an edge or high-contrast line appears to be going in a certain direction, that direction may be weighted higher than a direction without low-level features. In physical space, the agent may be a vehicle. Extracted low-level features may include roads (e.g., from edge detection). A direction where a road appears or leads to a road from the current observation may be weighted higher than a direction from the current observation without a road. This second weighting based on low-level features encourages the agent to follow or approach potential features in the environment. For example, following a road increases the likelihood of exploring the space and increases the likelihood that the agent will make observations that can be easily traversed and connected.While roads are man-made features added to the environment by humans, we know that similar beneficial features exist in natural environments, such as the banks of waterways, grassy forest edges, and foraging animal trails carved into vegetation. By following such features during both the exploration and navigation phases, the agent can ensure it has additional guidance to ensure reliable movement even in the face of significant environmental variations, such as thick fog. It is also important to note that such features can also help the agent recognize the identity of objects such as chairs. We guide the movement of the observation point along the edges of the chord formed by the chair's legs, seat, and back, ensuring high robustness in reconstructing identity even in the presence of 3D rotations.

[0287] It should be understood that the first and second directional weightings described herein can be used separately (without the other) or in combination.

[0288] Low-level features identified or detected in observations or observational data can also be linked to a set of actions to be performed by the agent. These actions allow the agent to approach features in a similar manner when encountering a new portion of space or object for the first time and when encountering the same portion of space or object again. For example, if a high-contrast line is identified or detected, the agent may be configured to move along the high-contrast line. The agent may be configured to move along consistent lines in the environment to a vanishing point, allowing it to move along roads, sidewalks, or paths. Such structured actions simplify the problem of robust navigation. The agent can move in different predetermined directions relative to the low-level features, such as perpendicular, parallel, clockwise, or counterclockwise relative to the low-level features.

[0289] Observations can also be made depending on the presence of low-level features in the environment. Characteristic low-level visual statistics, including but not limited to spatial frequency distribution, luminance distribution, color distribution, etc., can be used to determine when and where an agent should make a first observation.

[0290] The use of low-level features is not limited to map generation. In particular, low-level features can be used in any of the methods described above. For example, in hypothesis testing, according to the method 300 described above, low-level features or cues can be used in a movement function when determining how to move an agent from a current observation to a predicted observation. The movement function depends on the predicted travel distance required to move the agent to the predicted travel distance, but may also depend on low-level features such as high-contrast lines or edges. In physical space, an agent may be configured to follow low-level features such as roads, even if the roads do not directly align with the direction of the predicted displacement. Roads may be identified, for example, by displaying them as a single color extending upward from the bottom of the field of view where the agent is located to the top of the field of view, which primarily includes more distant locations. Alternatively, roads may be identified as areas that are extended as described above but lack strong horizontal edges and contain strong vertical edges. For example, if the road is within a threshold angle of the direction of the predicted displacement, the agent may be configured to follow the road rather than following the predicted displacement. This may improve the system if the predicted displacement is inaccurate for some reason. The agent periodically makes transition observations as it moves to ensure that its movement is properly aligned with the predicted observations. If the predicted observation is not reached, or if, for example, the observation-action function indicates that it is moving further away from the agent, the agent may leave the road. The agent may also move to the end of a low-level feature (e.g., the end of the road or the next road intersection). If the agent does not find the predicted observation at this point, it can still make and store the observation to generate a map or further populate the map with new knowledge about the space before rejecting the hypothesis and reverting to the previous observation.

[0291] Following or otherwise using low-level features is advantageous because it simplifies the process of testing hypotheses and navigating space, making it easier to backtrack from a point in space to a previous observation by simply following the low-level features in the opposite direction. Additionally, the agent can track features without any distant descriptive information, improving robustness to visually significant environmental changes such as heavy fog.

[0292] In the above embodiments, the system or method may be implemented using a server. The server may include a single server or a network of servers. In some examples, the functionality of the server may be provided by a server network distributed across a geographic region, e.g., a globally distributed server network, and an agent may connect to an appropriate one of the server networks based on the agent's location.

[0293] For clarity, the above description describes embodiments of the invention with reference to a single location, it being understood that in practice the system may be shared by multiple agents, and possibly by a large number of agents simultaneously.

[0294] The above-described embodiments may be fully automated and performed autonomously by an agent, although in some instances a user or operator of the system may manually direct some steps of the method to be performed.

[0295] In the described embodiments, the system may be implemented as any form of computing and / or electronic device. Such a device may include one or more processors, such as a microprocessor, controller, or other suitable type of processor, that process computer-executable instructions to control the operation of the device to collect and record routing information. In some examples, for example, when a system-on-chip architecture is used, the processor may include one or more fixed function blocks (also called accelerators) that implement portions of the method in hardware (rather than software or firmware). Platform software, including an operating system or any other suitable platform software, may be provided in the computing-based device to enable application software to run on the device.

[0296] The various functions described herein may be implemented in hardware, software, or a combination thereof. If implemented in software, the functions may be stored on or transmitted over as one or more instructions or code on a computer-readable medium. Computer-readable media include, for example, computer-readable storage media. Computer-readable storage media may include volatile or nonvolatile, removable or non-removable media implemented in any method or technology for storage of information such as computer-readable instructions, data structures, program modules, or other data. A computer-readable storage medium is any available storage medium that can be accessed by a computer. By way of non-limiting example, such computer-readable storage media may include RAM, ROM, EEPROM, flash memory or other memory devices, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or other medium that can be used to carry or store program code in the form of instructions or data structures and that can be accessed by a computer. As used herein, disk and disc include compact discs (CDs), laser discs, optical discs, digital versatile discs (DVDs), floppy disks, and Blu-ray (RTM) discs (BDs). Furthermore, propagated signals are not included within the scope of computer-readable storage media. Computer-readable media also includes communication media, which includes any medium that facilitates transfer of a computer program from one place to another. For example, a connection can be a communication medium. For example, if software is transmitted from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, or microwave, it is included in the definition of communication media.

[0297] Combinations of the above should also be included within the scope of computer-readable media.

[0298] Alternatively or additionally, the functions described herein may be performed, at least in part, by one or more hardware logic components, including, but not limited to, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on a chip (SOCs), complex programmable logic devices (CPLDs), etc.

[0299] Although illustrated as a single system, it should be understood that the computing device may be a distributed system, so that, for example, multiple devices may communicate over a network connection and may jointly perform tasks described as being performed by the computing devices.

[0300] While illustrated as a local device in some embodiments, it should be understood that the computing device may be located remotely and accessed via a network or other communications link (e.g., using a communications interface).

[0301] The term "computer" as used herein refers to any device having processing capabilities to enable the execution of instructions. Those skilled in the art will appreciate that such processing capabilities are incorporated into a variety of devices, and thus the term "computer" includes PCs, servers, mobile phones, personal digital assistants, and many other devices.

[0302] Those skilled in the art will appreciate that storage devices utilized to store program instructions can be distributed across a network. For example, a remote computer may store an example process written as software. A local or terminal computer may access a remote computer and download some or all of the software to execute the program. Alternatively, a local computer may download portions of the software as needed, execute the software instructions at a local terminal, or execute the software instructions at a remote computer (or computer network). Those skilled in the art will also appreciate that, utilizing conventional techniques known to those skilled in the art, all or a portion of the software instructions may be executed by dedicated circuitry, such as a DSP, programmable logic array, or the like.

[0303] It is understood that the benefits and advantages described above may relate to one or several embodiments, and the embodiments are not limited to those that solve any or all of the problems mentioned or that have the benefits and advantages mentioned.

[0304] A reference to "an" or "an" item means one or more of those items. As used herein, the term "comprising" means including the specified method things or elements, but such things or elements do not constitute an exclusive list and the method or apparatus may include additional things or elements.

[0305] As used herein, the terms "component" and "system" are intended to include a computer-readable data store comprised of computer-executable instructions that, when executed by a processor, cause the computer to perform a particular function. The computer-executable instructions may include routines, functions, etc. It should also be understood that a component or system may be localized on a single device or distributed across multiple devices.

[0306] Moreover, as used herein, the word "exemplary" is intended to mean "serving as an example or instance of something."

[0307] Furthermore, to the extent the term "includes" is used in the detailed description or claims, that term is intended to be inclusive in the same manner as "comprising" used as a transitional term in the claims.

[0308] Additionally, the operations described herein may include computer-executable instructions that may be implemented by one or more processors and / or stored on one or more computer-readable media. Computer-executable instructions may include routines, subroutines, programs, threads of execution, etc. Additionally, the results of the operations of the methods may be stored on a computer-readable medium or displayed on a display device.

[0309] Although the ordering of steps in the methods described herein is exemplary, these steps may be performed in any suitable order, or simultaneously where appropriate. Furthermore, steps may be added or substituted, or individual steps may be deleted, from any method without departing from the scope of the subject matter described herein. Aspects of any of the above embodiments may be combined with aspects of any of the other above embodiments to form further embodiments without losing the desired effect.

[0310] It will be understood that the above description of the preferred embodiment is given by way of example only, and that various modifications may be made by those skilled in the art. What has been described above comprises one example of one or more embodiments. Of course, it is not possible to describe every conceivable modification and variation of the above-described apparatus or method for purposes of describing the foregoing aspects, but one skilled in the art will recognize that many more modifications and combinations of the various aspects are possible. Accordingly, the described aspects are intended to include all such modifications, variations, and variations that fall within the scope of the appended claims.

Claims

1. A computer implementation method for analyzing a multidimensional space based on a series of observations and displacements performed by an agent in space, and for exploring the attributes of said space, A first observation is made from a first location of the agent in the space, wherein the first observation includes data describing a first portion of the space in the local vicinity of the first location. The first observation is compared with a set of saved observations and displacements to identify the first saved observation that is most similar to the first observation. Determining a hypothesis of the attributes of the space based on the first saved observation, wherein the first saved observation is associated with the hypothesis, Obtaining a hypothetical subset of the set of saved observations and displacements, wherein the hypothetical subset includes at least the first saved observation, the predicted observation, and the predicted displacement necessary to reach the predicted observation, and the predicted observation and the predicted displacement are also associated with the hypothesis. The purpose is to verify the aforementioned hypothesis, Based on the movement function that depends on the predicted displacement of the hypothetical subset, the agent is moved to a second location in the space. A second observation is made from the second location of the agent in the space, wherein the second observation includes data describing the second portion of the space in the local vicinity of the second location. The hypothesis is verified by obtaining an observation comparison scale by comparing the second observation with the predicted observation, and / or obtaining a displacement comparison scale by comparing the actual displacement between the first location and the second location with the predicted displacement. Adjusting, maintaining, or confirming the hypothesis based on the aforementioned observational comparison scale and / or displacement comparison scale, The confirmation of the aforementioned hypothesis includes determining that the hypothesis confirmation conditions have been met, A computer implementation method comprising exploring the attributes of the aforementioned space.

2. The aforementioned movement function is, A first movement component that depends on the predicted displacement of the hypothetical subset, A second movement component that depends on the predicted observations of the hypothetical subset, Here, the first movement component is weighted according to the remaining distance of the predicted displacement such that the first movement component weakens as the agent moves along the predicted displacement. The method according to claim 1, wherein the second movement component is weighted according to the predicted observation such that the second movement component becomes stronger as the agent approaches the predicted observation.

3. The method according to claim 2, further comprising measuring the actual displacement of the movement from the first location to the second location.

4. The method according to claim 2, wherein when the agent reaches the endpoint of the predicted displacement or moves within a displacement threshold distance from the endpoint of the predicted displacement, the first movement component of the movement function is weighted to zero such that the first movement component no longer contributes to the movement function.

5. The second movement component includes the predicted direction and magnitude for the predicted observation, wherein the direction and magnitude are After or during the movement of the agent, transition observations are performed away from the first location. By performing a transition comparison between the transition observation and the predicted observation, the direction and magnitude of the predicted observation are obtained, The method according to claim 2, wherein the second movement component is weighted according to the transition comparison such that the weighting of the second movement component increases as the comparison with the predicted observation becomes stronger.

6. The method according to claim 5, wherein the method includes performing a plurality of transition observations, wherein the second movement component, including the direction and magnitude, is repeatedly updated as the agent moves away from the first location.

7. The method further includes stopping the agent at the second location using the aforementioned movement function, where the second location is The location of the transition observation of the second movement component of the movement function, wherein one transition comparison of the transition observation shows an optimal match with a predicted observation from multiple transition observations, and / or The method according to claim 6, wherein the magnitude of the second movement component function is equal to or less than a magnitude threshold.

8. The method according to claim 7, wherein one of the transition observations at the second location is the second observation, and the transition comparison showing the optimal matching is the second observation.

9. The aforementioned observational comparison scale is, The first observational comparison scale component, The second observational comparison scale component, It includes one or more of the third observational comparison scale component and the fourth observational comparison scale component, The first observation comparison scale component is a similarity scale that shows the similarity between the second observation and the predicted observation, The second observation comparison scale component is a scale that indicates the direction of the predictive observation relative to the second observation, The third observation comparison scale component is a vector from the second observation to the predicted observation, The method according to claim 1, wherein the fourth observation comparison scale component is a second similarity scale indicating the similarity between the second observation and the predicted observation.

10. The method according to claim 9, wherein the first observation comparison scale component is formed by an observation similarity function obtained by taking the inner product of the second observation and the predicted observation, and the second observation and the predicted observation are vectors.

11. The third observational comparison scale component described above is: Dividing the aforementioned predictive observations into a first set of sub-regions, Obtain the first and second observation comparison scale components for comparing the first plurality of sub-regions with the second plurality of sub-regions of the second observation, and ensure that each of the first plurality of sub-regions is associated with the first and second observation comparison scale components, The method according to claim 9, wherein the first and second observation comparison scale components are aggregated to form a vector from the second observation to a predicted observation.

12. Adjusting, maintaining, or confirming the aforementioned hypothesis based on the verification of the hypothesis is, Adjusting the hypothesis when the aforementioned observational comparison scale falls below the first observational threshold level, The hypothesis is maintained when the observation comparison scale exceeds the first observation threshold level. This includes confirming the hypothesis when the observation comparison scale exceeds the hypothesis confirmation observation threshold level, which is the hypothesis confirmation condition, or The hypothesis is adjusted when the displacement comparison scale falls below the first displacement threshold level. The hypothesis is maintained when the displacement comparison scale exceeds the first displacement threshold level. This includes confirming the hypothesis when the displacement comparison scale exceeds the hypothesis confirmation threshold level, which is the hypothesis confirmation condition, or A two-dimensional similarity scale is formed by combining the aforementioned observation comparison scale and the aforementioned displacement comparison scale, The hypothesis is adjusted when the two-dimensional similarity measurement falls below the first two-dimensional similarity measurement threshold level. If the two-dimensional similarity measurement exceeds the first two-dimensional similarity measurement threshold level, the hypothesis is maintained. The method according to claim 1, comprising confirming the hypothesis when the two-dimensional similarity measurement exceeds the hypothesis confirmation condition, which is a hypothesis confirmation two-dimensional similarity measurement threshold level.

13. The method according to claim 1, wherein the set of saved observations is associated with a plurality of hypotheses, and each set of hypotheses comprises a plurality of hypothesis subsets, each comprising at least one predictive observation and predictive displacement associated with a particular hypothesis.

14. Adjusting the aforementioned hypothesis is equivalent to rejecting the aforementioned hypothesis, Moving the agent from the second location to the first location according to the negative displacement of the actual displacement, The method according to claim 13, comprising: selecting a new hypothesis from among a plurality of hypotheses based on a comparison of the first observation with the set of saved observations and displacements; and identifying the next saved observation in the set of saved observations and displacements that is next similar to the first observation.

15. The method according to claim 1, wherein the hypothetical subset includes a plurality of predictive observations and a plurality of predictive displacements that are predicted to be necessary for movement between the predictive observations.

16. Maintaining the aforementioned hypothesis means Obtain the next predicted observation and the next predicted displacement from the aforementioned hypothetical subset, The method according to claim 15, comprising repeatedly verifying the hypothesis with respect to the subsequent predictive observation and the subsequent predictive displacement.

17. The method according to claim 16, comprising repeatedly verifying the hypothesis for the plurality of predicted observations and the plurality of predicted displacements of the hypothesis subset until the hypothesis is adjusted or confirmed.

18. The attributes of the space being explored are objects or images within that space. The aforementioned hypothesis indicates a predicted object or image, and the predicted observation and predicted displacement associated with the aforementioned hypothesis are associated with the predicted object or image. Exploring the attributes of the aforementioned space is The method according to claim 1, comprising identifying the object or image in the space as a predicted object.

19. The attributes of the space being explored are the destination within that space, The hypothesis indicates a path to the destination, and the predicted observations and predicted displacements associated with the hypothesis are related to the expected path to the destination. Exploring the attributes of the aforementioned space is The method according to claim 1, comprising reaching a destination in space.

20. The method according to claim 1, wherein the space is a physical or logical space and is two-dimensional or three-dimensional.

21. It is a system, Processor and Memory and A sensor or virtual sensor configured to acquire spatial data describing a portion of space in the local vicinity of the sensor or virtual sensor, A system wherein the memory stores instructions, and when executed by the processor, the system performs the method according to any one of claims 1 to 20.

22. The system is a robot or vehicle equipped with the sensor, and the robot or vehicle is the agent configured to operate in physical space. The system further includes a controllable mobile module configured to move the robot or vehicle within the physical space, The system according to claim 21, wherein the sensor is a camera, a LiDAR sensor, an infrared sensor, a radar, a tactile sensor, or another sensor configured to provide spatial information.

23. The system according to claim 21, wherein the system is a computer system including the virtual sensor, the computer system is configured to operate in a logical space, and the agent is represented by a point in the logical space.

24. The system according to claim 23, wherein the logical space is an image, the points in the logical space are pixels of the image, and the virtual sensor is configured to capture a portion of the image data in the local neighborhood of the pixels.

25. A computer program stored on a non-temporary computer-readable medium, which, when executed by a processor, is configured to cause the processor to perform the method described in any one of claims 1 to 20.

Images