Method for analyzing environment of intelligent agent in multi-dimensional space

Abstract

There is provided a computer-implemented method and system for parsing an environment of an agent in a multi-dimensional space, the method comprising: obtaining first sensor data at a first location of the agent; retrieving stored second sensor data to compare with the first sensor data; obtaining a plurality of first sub-regions of the first sensor data; obtaining a plurality of second sub-regions of the second sensor data; comparing the second sub-region with each first sub-region using a similarity comparison metric to determine a first sub-region that is most similar to the second sub-region; and determining associated relative rotations and aggregating the relative rotations for the plurality of second sub-regions to obtain an action vector indicative of an estimated direction from the first position to the second position of the agent.

Description

Technical Field

[0001] A system and computer-implemented method for understanding multidimensional space based on comparison of observed data of the space with previously stored data for navigation in unfamiliar environments and for visual classification of objects and / or images. Background Art

[0002] Simultaneous Localization and Mapping (SLAM) is the problem and process of computationally building or updating a map of an unknown environment while keeping track of an agent's position within it. SLAM algorithms are based on concepts from computational geometry and computer vision and are used in robotic navigation, robotic mapping, and odometry for virtual or augmented reality.

[0003] SLAM algorithms are also used in the control of autonomous vehicles to map the unknown environment near the vehicle. The generated map information can be used to perform tasks such as path planning and obstacle avoidance.

[0004] Visual SLAM uses an approach that includes a front end that processes images from a camera to perform "feature extraction." This feature extraction process finds small areas in the image that meet certain criteria (such as high local spatial frequency and contrast) so that these areas are visually distinguishable. These areas can then be stored so that the same area can be located in subsequent camera images taken by the same camera a short time later. The positions of these features are then mapped to real-world orientations by applying a camera model that takes into account the camera lens distortion. They are then fed to the back end of the algorithm, the so-called bundle adjuster. The bundle adjuster attempts to compare the predicted positions of the features in 3D space based on previous camera images with the new orientation of the current camera image, and simultaneously determine the positions of these features in 3D space relative to each other and the position of the camera relative to these features. By repeating this process, the bundle adjuster keeps track of the position of the moving camera and the mapped positions of these features over time.

[0005] This approach suffers from several drawbacks that limit its effectiveness in real-world robotics applications. First, both feature extraction and bundle adjustment are computationally very expensive processes. Feature extraction can be easily parallelized for increased efficiency, but bundle adjustment is a strictly sequential process.

[0006] Second, bundle adjustment requires that the new feature orientations and predicted feature locations converge to a stable solution, which is heavily affected by sensor noise and environmental changes (such as wind blowing leaves on a bush). If features are falsely detected at incorrect locations in the camera image, bundle adjustment may not converge to a stable solution and the system will be "lost" and unable to localize.

[0007] To address the problem of convergence failure of the beam adjuster, the prior art adopts two approaches: one approach reduces the failure, and the other approach allows the system to continue to operate with reduced performance when convergence fails.

[0008] The first method is outlier removal. A set of constraints are used to evaluate the likelihood that when a feature is found, it is the same as a previously found feature. For example, if a feature moves from one side of the camera to the other, it is unlikely to be the same feature unless the camera is moving very quickly, so it is rejected as a match. If it is close to its previous location, then it is likely the same feature and is accepted. Accepted features are passed to the bundle adjuster, rejected features are not. Outlier removal is another very computationally expensive process, and for some SLAM systems it is the largest portion of the computation.

[0009] The second approach is visual-inertial odometry. This is a system that uses the features located by the feature extractor in a different way. It uses a second source of motion information, linear acceleration measurements from inertial sensors, and tracks features to predict the velocity of the camera instead of its location. This velocity measurement can then be integrated over time and used in conjunction with a bundle adjuster to compensate for convergence failures. However, integrating VIO accumulates location errors over time, and since this SLAM approach relies on metric measurements of location, it can fail at longer integration times. Since both VIO and bundle adjustment use the same features, feature extraction is a single point of failure for the entire system.

[0010] Another weakness of the feature extraction front end is that the size of the features means that the data dimensionality is relatively low for a single feature, and therefore there is a high probability that multiple parts of the image will match that feature, especially in aliased environments such as a hotel corridor with evenly spaced identical doors. In these environments, the bundle adjuster may converge to incorrect positions.

[0011] Loop closure is another weakness of the existing technology. When the mobile camera goes through a complete loop and returns to the same location from a different direction, the system must recognize that it has returned to the same location. Since the system is metric, it must also ensure that the map is consistent in the same location in the same place. Due to the accumulated errors in the bundle adjustment process, this rarely happens, and there is usually a metric error between the representations of the locations at both ends of the loop. The system must then re-traverse all of its stored features throughout the loop and adjust them to compensate and eliminate this error. This loop closing process is not only very computationally expensive, but also prone to error.

[0012] The computational complexity of algorithms that can cope with such environments and problems is very high, limiting the platforms they can be deployed on. In addition, the memory consumption of the generated maps is very high, on the order of hundreds of megabytes to several gigabytes.

[0013] Therefore, it is recognized that there is a need for a method and system for positioning and mapping that enables an intelligent agent to understand the structure and identity of a space and objects within it, and that is less error-prone and has low computational complexity and memory usage. Summary of the invention

[0014] This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description.

[0015] According to a first aspect of the present disclosure, there is provided a computer-implemented method for parsing an environment of an agent in a multidimensional space, the method comprising: obtaining first sensor data at a first position of the agent, wherein the first sensor data describes an environment surrounding the agent; retrieving stored second sensor data for comparison with the first sensor data, wherein the second sensor data describes an environment surrounding a second position indicating a feature in the multidimensional space; obtaining a plurality of first sub-regions of the first sensor data, each first sub-region describing a corresponding first portion of the environment surrounding the agent at the first position, each first portion being associated with a corresponding first direction from the first position; obtaining a plurality of second sub-regions of the second sensor data subregions, each second subregion describes a corresponding second part of the environment around the second position, each second part is associated with a corresponding second direction from the second position; and, for each second subregion: using a similarity comparison metric to compare the second subregion with each first subregion to determine a first subregion that is most similar to the second subregion; determining a relative rotation between the second direction associated with the second subregion and the first direction associated with the most similar first subregion; the method also includes: aggregating the relative rotation for the multiple second subregions to obtain an action vector, the action vector indicating an estimated direction from the first position of the agent to the second position, the second position indicating the feature of the multidimensional space.

[0016] The first sub-region and the second sub-region are directional portions of the first sensor data and the second sensor data and are formed by contiguous subsets of the first sensor data and the second sensor data. Comparing these sub-regions rather than comparing the entirety of the first sensor data and the second sensor data introduces directional variations, so that the sub-regions may closely match each other in a comparison in one direction, but not in another direction. This concept effectively allows the direction to be determined indicating the second position in 2D or 3D space to be determined by comparing multiple sets of sub-regions. This forms the direction of the action vector. Therefore, the method provides a method for relocating to a previously visited location (the second location) based solely on the comparison of the first sensor data and the second sensor data.

[0017] The first sensor data and the second sensor data may be first image data and second image data, respectively, wherein each first sub-region of the first sub-region includes a continuous subset of pixels of the first image data, and wherein each second sub-region of the second sub-region includes a continuous subset of pixels of the second image data. The image data may be modified to reduce its size, thereby improving computational efficiency. The first sensor data and the second sensor data may also alternatively be any data capable of geometrically representing the environment.

[0018] The first sensor data and the second sensor data may be arranged in a first vector and a second vector, respectively, the first vector and the second vector being concatenated from the first image data and the second image data, respectively. The image data may initially exist in an array with multiple channels per pixel. Converting the array to a vector is computationally efficient in performing sub-region comparisons.

[0019] The similarity measure may be the inner product between the first sub-region and the second sub-region. This occurs for each sub-region comparison.

[0020] Obtaining the first sensor data may include: obtaining original first sensor data of the environment of the agent at the first position; processing the original first sensor data to reduce the size of the original first sensor data to obtain the first sensor data, so that the first sensor data has a reduced size format relative to the original first sensor data; wherein the stored second sensor data is also stored in the reduced size format, so that the method also includes: obtaining original second sensor data of the environment of the agent at the second position; and processing the original second sensor data to reduce the size of the original second sensor data to obtain the second sensor data, so that the second sensor data has a reduced size format relative to the original second sensor data.

[0021] The original first sensor data and the original second sensor data may be original first image data and original second image data, respectively. Thus, the first sensor data and the second sensor data are reduced-size formats of the original first image data and the original second image data, respectively. The processing may include several operations, including but not limited to filtering, masking, blurring, etc.

[0022] Processing the raw first sensor data and the raw second sensor data may include applying one or more filters and / or masks to the raw first sensor data and the raw second sensor data to reduce the size of each dimension of the raw first sensor data and the raw second sensor data, respectively.

[0023] The raw first sensor data and the raw second sensor data may be raw first image data and raw second image data, respectively.

[0024] Obtaining multiple first sub-regions of the first sensor data may include: iteratively applying a mask to the first sensor data to extract each first sub-region in the multiple first sub-regions, wherein, at each iteration, the mask or the first sensor data is replaced by at least one data entry / unit in a dimension of the first sensor data; and wherein obtaining multiple second sub-regions of the second sensor data includes: iteratively applying a mask to the second sensor data to extract each second sub-region in the multiple second sub-regions, wherein, at each iteration, the mask or the second sensor data is replaced by at least one data entry / unit in a dimension of the second sensor data.

[0025] There may be at least four iterations, such that there are at least four first sub-regions and at least four second sub-regions.

[0026] The mask may be smaller than the first sensor data and the second sensor data, wherein the mask includes a principal dimension that is: equal to or smaller than 50% of the size of the corresponding principal dimension of the first sensor data and the second sensor data; or equal to or smaller than 25% of the size of the corresponding principal dimension of the first sensor data and the second sensor data.

[0027] The first sensor data and the second sensor data are arranged in a first array and a second array, respectively, wherein the first array and the second array have dimensions of X×Y units, and the mask has dimensions of (Xm)×Y units, wherein m is a positive integer.

[0028] The first array and the second array may have a plurality of channels such that the dimensions of the arrays are X×Y×Z, where Z is a positive integer.

[0029] The method may also include moving the agent from the first position according to the motion vector.

[0030] The method may also include: after moving the agent to a new position according to the action vector: obtaining third sensor data at the new position of the agent, wherein the third sensor data describes the environment around the agent; obtaining multiple third sub-regions of the third sensor data, each third sub-region describing a corresponding third portion of the environment around the agent at the third position, each third portion being associated with a corresponding third direction from the third position; and, for each second sub-region: comparing the second sub-region with each third sub-region using the similarity comparison metric to determine a third sub-region that is most similar to the second sub-region; determining a relative rotation between the second direction associated with the second sub-region and the third direction associated with the most similar third sub-region; the method also includes: aggregating the relative rotation for the multiple second sub-regions to obtain an updated action vector, the updated action vector indicating an estimated direction from the third position of the agent to the second position, the second position indicating the feature of the multi-dimensional space. The third sensor data may be of the same type as the first sensor data, and may therefore be image data, or any other data that can provide a geometric representation of the environment.

[0031] The method may further include confirming that the feature of the environment exists in the multidimensional space if an observation comparison metric matches or exceeds a confirmation threshold level, wherein the observation comparison metric is associated with the similarity metric and / or the action vector.

[0032] Determining the relative rotation may include determining an offset angle between the first direction and the second direction.

[0033] The method may also include: for a relative second sub-region pair, determining the amplitude of the motion vector by: determining an average offset angle based on the offset angles associated with the relative second sub-region pair; and allocating the amplitude based on the average offset angle, wherein the size of the average offset angle is proportional to the amplitude.

[0034] The method may further comprise determining a respective magnitude of each pair of offset angles of a plurality of pairs of offset angles associated with the opposing pairs of second sub-regions, and aggregating / averaging the respective magnitudes to form the magnitude of the motion vector.

[0035] The first sensor data may represent a substantially 360-degree view around the agent, and the second sensor data may represent a substantially 360-degree view around the second location, wherein each first portion described by each first sub-region is part of the 360-degree view around the agent, and wherein each second portion described by each second sub-region is part of the 360-degree view around the second location. Thus, the first sensor data and the second sensor data may be derived from a visual panorama around the agent.

[0036] Each first sub-region of the plurality of first sub-regions overlaps with at least one adjacent first sub-region, and each second sub-region of the plurality of second sub-regions overlaps with at least one adjacent second sub-region.

[0037] The feature of the environment may be one of: a specific location in the multi-dimensional space; an image or a portion thereof; or an object or a portion thereof.

[0038] The agent may be virtual, wherein the multi-dimensional space is a two-dimensional virtual space or a three-dimensional virtual space, and wherein the first sensor data and the second sensor data are obtained using virtual sensors.

[0039] The agent may be a physical entity, wherein the multi-dimensional space is a three-dimensional real physical space, and wherein the first sensor data is obtained using a physical sensor. For example, the physical entity may be a robot or a vehicle.

[0040] The second sensor data may describe an environment indicating a target location within the multidimensional space, such that features of the environment described by the second sensor data are associated with the target location, and the method further includes navigating the agent from the first location to the target location according to the action vector.

[0041] The second sensor data can form part of a second sensor data set, which includes multiple instances of the second sensor data, each instance describing an environment indicating a corresponding position in the multidimensional space, and the method also includes: iteratively navigating the intelligent agent from the first position to the target position of the corresponding position indicated by the multiple instances of the second sensor data, in each iteration: obtaining an action vector relative to one of the instances of the second data, and moving from the position of the intelligent agent to the corresponding position indicated by the one instance of the second data according to the action vector; until reaching the target position.

[0042] A main navigation process is formed to navigate the agent from the first position to the target position according to the action vector, and the method further includes: using an auxiliary navigation process to navigate the agent from the initial position to the first position in the multidimensional space; and switching from the auxiliary navigation process to the main navigation process at the first position or in the vicinity of the first position.

[0043] The assisted navigation process may be configured to use a positioning system.

[0044] The positioning system is a satellite-based radio navigation system.

[0045] The second sensor data can be stored in a remotely accessible database. In this way, the agent can remotely retrieve the second sensor data when a comparison is required.

[0046] The method may also include: recording, by a recording agent, the second sensor data at the second location; and storing the second sensor data in the remotely accessible database.

[0047] The method may also include: recording metadata corresponding to the second sensor data and associating the metadata with the second sensor data, the metadata including at least one of the following: metric information about the second position in the multidimensional space; information about the variability of one or more parts of the environment around the second position captured in the second sensor data; information about one or more fixed features of the environment around the second position captured in the second sensor data; time information about the capture of the second sensor data; an availability metric indicating the predicted or predetermined availability of the signal at the second position; and information about specific features captured in the second sensor data; the method also includes: storing the metadata together with the corresponding second sensor data in the remotely accessible database.

[0048] The metadata may include the measurement information about the second position in the multidimensional space, and the method also includes: obtaining measurement information about the first position of the agent in the multidimensional space; determining that the second position is located in the vicinity of the first position based on the measurement information about the first position and the measurement information about the second position, and retrieving the second sensor data from the remotely accessible database based on the determination that the second position is located in the vicinity of the first position.

[0049] The metadata may include information about the variability of one or more parts of the environment surrounding the second location captured in the raw second sensor data; wherein processing the raw second sensor data by applying the one or more filters and / or masks to the raw second sensor data to reduce the size of each dimension of the raw second sensor data includes: based on the metadata, filtering out or masking out the one or more parts of the environment surrounding the second location captured in the raw second sensor data, so that the reduced format second sensor data does not include the one or more parts of the environment indicated by the information in the metadata.

[0050] The metadata may include the information about one or more fixed features of the environment surrounding the second position captured in the raw second sensor data; wherein processing the raw second sensor data by applying the one or more filters and / or masks to the raw second sensor data to reduce the size of each dimension of the raw second sensor data includes: based on the metadata, retaining the one or more fixed features of the environment surrounding the second position captured in the raw second sensor data, so that the second sensor data in the reduced format includes the one or more fixed features of the environment indicated by the information of the metadata.

[0051] The metadata may include the availability metric indicating the predicted or predetermined availability of the positioning system signal and / or the data connection signal at the second location; the method also includes: navigating the intelligent agent based on the availability metric according to the positioning system signal, and / or: determining when to download information from the remotely accessible database based on the availability metric according to the data connection signal.

[0052] The method may also include: selecting, by the recording agent, the second location for recording the second sensor data, the selection being based on a location selection criterion including at least one of: an exploration state associated with the second location; and determining whether the second location is accessible by the agent.

[0053] According to a second aspect, a system is provided, comprising: a processor; a memory; and a sensor or a virtual sensor, wherein the sensor or the virtual sensor is configured to capture spatial data describing the environment of a multidimensional space in a local vicinity of the sensor or the virtual sensor; the memory stores instructions which, when executed by the processor, cause the system to perform the method of the first aspect described above.

[0054] The multidimensional space may be a physical space, and the system may be a robot or a vehicle including the sensor, wherein the robot or the vehicle is the agent, and further includes: a controllable movement module configured to enable the robot or the vehicle to move within the physical space.

[0055] The sensor may include one or more of an ultraviolet imaging device, a camera, a LIDAR sensor, an infrared sensor, a radar, a tactile sensor, or other sensors configured to provide spatial information.

[0056] The system may include multiple devices, including the robot or vehicle and an additional computing device, the additional computing device having a processor, a memory and a sensor; wherein the additional computing device is the recording agent, and the system is configured to execute the method described in any one of the method steps set forth above in the first aspect.

[0057] The system may be a computer system including the virtual sensor, wherein the computer system is configured to operate on a virtual space, wherein the agent is represented by a point in the virtual space.

[0058] According to a third aspect, a computer program stored on a non-transitory computer readable medium is provided. When executed by a processor, the computer program is configured to cause the processor to perform any method described in the first aspect set forth above.

[0059] According to a fourth aspect, a computer-implemented method for parsing an environment in a multidimensional space is provided, the method comprising: recording sensor data describing the environment surrounding the recording agent in the multidimensional space; processing the sensor data to identify one or more features present in the environment captured in the sensor data; associating metadata with the identified one or more features, wherein the feature comprises at least one of: the position of the recording agent when the sensor data is captured in the multidimensional space; the variability of one or more parts of the environment surrounding the recording agent; and one or more fixed features of the environment surrounding the recording agent; the method further comprising: storing the sensor data and the associated metadata in a remotely accessible database.

[0060] The method may also include: retrieving, by a navigation agent in the multidimensional space, stored sensor data and associated metadata from the remotely accessible database; and navigating the multidimensional space using the sensor data and the associated metadata by comparing the sensor data captured by the navigation agent with the stored sensor data.

[0061] Using the sensor data and the associated metadata to navigate the multidimensional space also includes reducing the stored sensor data based on the associated metadata, wherein the reduced stored sensor data includes sensor data corresponding to non-variable or fixed features identified in the environment of the recorded agent.

[0062] The method may also include reducing the sensor data based on the associated metadata before storing the sensor data in the remotely accessible database, such that storing the sensor data includes storing sensor data corresponding to non-variable or fixed features identified in the environment and discarding sensor data corresponding to variable or non-fixed features identified in the environment.

[0063] The sensor data can form a panoramic view of the surroundings of the recording agent.

[0064] According to the fifth aspect, a system is provided, which includes a first computing device and a server; the first computing device includes: a processor; a memory; and a sensor or a virtual sensor, which is configured to capture spatial data describing the environment of a multidimensional space in a local vicinity of the sensor or the virtual sensor; the server includes: a memory, on which a remotely accessible database is stored; the system is configured to execute the method of the fourth aspect.

[0065] The system may also include a robot or a vehicle or a virtual robot, which includes: a processor; a memory; and a sensor or a virtual sensor, which is configured to capture spatial data describing the environment of the multidimensional space in the local vicinity of the sensor or virtual sensor; wherein the robot or the vehicle or the virtual robot is a navigation agent, and the system is configured to perform the method of the fourth aspect.

[0066] According to a sixth aspect, a computer-implemented method for parsing a multidimensional space is provided. The method is based on a series of observations and displacements performed by an agent in the space to find a property of the space. The method includes: generating a first observation result from a first position of the agent in the space, wherein the first observation result includes data describing a first portion of the space in a local neighborhood of the first position; comparing the first observation result with a stored set of observations and displacements to identify a first stored observation result in the stored set of observations and displacements that is most similar to the first observation result; determining a hypothesis about the property of the space based on the first stored observation result, wherein the first stored observation result is associated with the hypothesis; retrieving a subset of hypotheses of the stored set of observations and displacements, wherein the subset of hypotheses includes at least the first stored observation result, a predicted observation result, and a predicted displacement required to reach the predicted observation result, wherein the predicted observation result and the predicted displacement are also associated with the hypothesis; and testing the hypothesis. The hypothesis is tested by: moving the agent to a second position in the space based on a movement function, wherein the movement function depends on the predicted displacement of the hypothesis subset; generating a second observation from the agent at the second position in the space, wherein the second observation includes data describing a second portion of the space in a local neighborhood of the second position; and comparing the second observation with the predicted observation to obtain an observation comparison metric, and / or comparing the actual displacement between the first position and the second position with the predicted displacement to obtain a displacement comparison metric; adjusting, maintaining or confirming the hypothesis based on the observation comparison metric and / or the displacement comparison metric; wherein confirming the hypothesis includes: determining that a hypothesis confirmation condition is satisfied; and finding the property of the space.

[0067] The movement function may include: a first movement component, which depends on the predicted displacement of the hypothesis subset; and a second movement component, which depends on the predicted observation result of the hypothesis subset, wherein the first movement component is weighted according to the remaining distance of the predicted displacement, so that when the intelligent agent moves along the predicted displacement, the first movement component is weakened, and wherein the second movement component is weighted according to the predicted observation result, so that when the intelligent agent approaches the predicted observation result, the second movement component is enhanced.

[0068] The method may include measuring the actual displacement travelled from the first position to the second position.

[0069] If the agent reaches the end of the predicted displacement, or is within a displacement threshold distance from the end of the predicted displacement, the first movement component of the movement function is weighted to zero so that the first movement component no longer contributes to the movement function.

[0070] The second movement component may include an expected direction and magnitude of the predicted observation result, wherein the direction and magnitude are obtained by: generating a transition observation result away from the first position after or during the movement of the agent; and performing a transition comparison between the transition observation result and the predicted observation result to obtain the direction and magnitude of the predicted observation result; and wherein the second movement component is weighted according to the transition comparison, so that the stronger the comparison with the predicted observation result, the stronger the weight of the second movement component.

[0071] The method may include: the second movement component includes the direction and magnitude that are repeatedly updated as the agent moves away from the first position, such that the method includes generating a plurality of transition observations.

[0072] The method may include stopping the agent at the second position using the movement function, wherein the second position is: the position of one of the transition observations of the second movement component of the movement function, wherein a transition comparison of the one of the transition observations indicates a best match with a predicted observation from the plurality of transition observations; and / or the position where the magnitude of the second movement component function is equal to or below a magnitude threshold. The one of the transition observations at the second position may be the second observation, and the transition comparison indicating the best match may be the second observation.

[0073] The observation comparison metric may include one or more of: a first observation comparison metric component; a second observation comparison metric component; a third observation comparison metric component; and a fourth observation comparison metric component; wherein the first observation comparison metric component is a similarity metric indicating the similarity between the second observation and the predicted observation; wherein the second observation comparison metric component is a metric indicating the orientation of the predicted observation relative to the second observation; wherein the third observation comparison metric component is a vector from the second observation toward the predicted observation; and wherein the fourth observation comparison metric component is a second similarity metric indicating the similarity between the second observation and the predicted observation.

[0074] The observation comparison metric may include the first observation comparison metric component, the second observation comparison metric component, and the third observation comparison metric component. This effectively provides a way for the method to determine the similarity between the compared observations, the change in orientation between the compared observations, and the direction and magnitude (such as in the form of a vector) from the position of the agent to the position of the compared observations.

[0075] The observation comparison metric may also include the fourth observation comparison metric component. This provides redundancy and may be advantageously used to distinguish the first observation comparison metric component from environmental factors and features.

[0076] The first observation comparison metric component may be formed by an observation similarity function that takes the inner product of the second observation and the predicted observation, wherein the second observation and the predicted observation are vectors, and specifically unit vectors.

[0077] The third observation comparison metric component can be formed in the following manner: dividing the predicted observation result into a first plurality of sub-regions; obtaining the first observation comparison metric component and the second observation comparison metric component for comparison between the first plurality of sub-regions and a second plurality of sub-regions of the second observation result, so that each sub-region in the first plurality of sub-regions is associated with the first observation comparison metric component and the second observation comparison metric component; and aggregating the first observation comparison metric component and the second observation comparison metric component to form a vector from the second observation result to the predicted observation result.

[0078] Adjusting, maintaining or confirming the hypothesis based on the test of the hypothesis may include: adjusting the hypothesis if the observation comparison metric is below a first observation threshold level; maintaining the hypothesis if the observation comparison metric exceeds the first observation threshold level; and confirming the hypothesis if the observation comparison metric exceeds a hypothesis confirmation observation threshold level, wherein the hypothesis confirmation observation threshold level is a hypothesis confirmation condition; or: adjusting the hypothesis if the displacement comparison metric is below a first displacement threshold level; maintaining the hypothesis if the displacement comparison metric exceeds the first displacement threshold level; and confirming the hypothesis if the displacement comparison metric exceeds a hypothesis confirmation threshold level, wherein the hypothesis confirmation threshold level is a hypothesis confirmation condition; or: combining the observation comparison metric and the displacement comparison metric to form a two-dimensional similarity metric; adjusting the hypothesis if the two-dimensional similarity metric is below a first two-dimensional similarity metric threshold level; maintaining the hypothesis if the two-dimensional similarity metric exceeds the first two-dimensional similarity metric threshold level; and confirming the hypothesis if the two-dimensional similarity metric exceeds a hypothesis confirmation two-dimensional similarity metric threshold level, wherein the hypothesis confirmation two-dimensional similarity metric threshold level is a hypothesis confirmation condition.

[0079] The stored set of observations is associated with a plurality of hypotheses and includes a plurality of hypothesis subsets, each hypothesis subset including at least one predicted observation and predicted displacement associated with a particular hypothesis.

[0080] Adjusting the hypothesis may include: rejecting the hypothesis; moving the agent from the second position to the first position based on a negative displacement of the actual displacement, and selecting a new hypothesis from the multiple hypotheses based on comparing the first observation result with the stored observation results and displacement set to identify the next stored observation result in the stored observation results and displacement set that is second most similar to the first observation result.

[0081] The subset of hypotheses may include a plurality of predicted observations and a plurality of predicted displacements predicted to be required to move between the predicted observations.

[0082] Maintaining the hypothesis may include: retrieving a next predicted observation and a next predicted displacement from the hypothesis subset; and repeatedly testing the hypothesis relative to the next predicted observation and the next predicted displacement.

[0083] The method may include repeatedly testing the hypothesis against the plurality of predicted observations and the plurality of predicted displacements for the hypothesis subset until the hypothesis is adjusted or confirmed.

[0084] The property of the space to be found may be an object or image in the space, wherein: the hypothesis indicates a predicted object or image, so that the predicted observation result and the predicted displacement associated with the hypothesis are associated with the predicted object or image; finding the property of the space includes: identifying the object or image in the space as the predicted object.

[0085] The attribute of the space to be found may be a destination in the space, wherein: the hypothesis indicates a path to the destination, so that the predicted observations and the predicted displacements associated with the hypothesis are associated with the expected path to the destination; finding the attribute of the space includes: arriving at the destination in the space.

[0086] The space can be physical or logical, and two-dimensional or three-dimensional.

[0087] The method may also include extracting a set of low-level spatial features from the physical or logical space; wherein determining the hypothesis based on the stored predicted observations and the set of displacements includes processing the first observations and the low-level spatial features to determine the hypothesis, wherein a weighting factor is assigned to the set of low-level spatial features when determining the hypothesis.

[0088] The method may include: processing the second observation and / or the nth observation to verify the hypothesis related to the physical or logical space, extracting a second or nth set of low-level spatial features from the physical or logical space; and processing the second or nth observation and the second or nth set of low-level spatial features to verify the hypothesis, where n is a positive real integer.

[0089] According to a seventh aspect of the present disclosure, a system is provided, comprising: a processor; a memory; and a sensor or a virtual sensor, the sensor or the virtual sensor being configured to capture spatial data describing a portion of a space in a local vicinity of the sensor or the virtual sensor, the memory having instructions stored thereon, which, when executed by the processor, cause the system to perform the method described in the sixth aspect outlined above.

[0090] The system may be a robot or a vehicle including the sensor, wherein the robot or the vehicle is an intelligent agent configured to operate in a physical space, and further comprising: a controllable movement module configured to move the robot or the vehicle within the physical space; and wherein the sensor is a camera, a LIDAR sensor, an infrared sensor, a radar, a tactile sensor, or any other sensor configured to provide spatial information.

[0091] The system may be a computer system including the virtual sensor, wherein the computer system is configured to operate on a logical space, wherein the agent is represented by a point in the logical space.

[0092] The logical space may be an image, points in the logical space being pixels of the image, and the virtual sensor being configured to capture a portion of the image data in a local neighborhood of the pixel.

[0093] According to an eighth aspect of the present disclosure, a computer program stored on a non-transitory computer-readable medium is provided. When executed by a processor, the computer program is configured to cause the processor to execute the method described in the sixth aspect above.

[0094] According to the ninth aspect of the present disclosure, there is provided a computer-implemented method for parsing a physical or logical space based on a series of observations and displacements performed by an agent in the physical or logical space to determine a property of the physical or logical space, the method comprising: selecting a hypothesis about the property of the physical or logical space, the selection being based on a comparison between a current observation of the agent in the physical or logical space and a current predicted observation in a set of stored predicted observations and displacements associated with the hypothesis; wherein the current observation is recorded from the current position of the agent in the physical or logical space and includes a description of a local neighborhood of the current position; data of the current portion of the logical or physical space; relative to the current predicted observation, retrieving a next predicted displacement and a next predicted observation from the predicted observation and displacement set associated with the hypothesis; and iteratively performing the following operations: comparing the next predicted observation associated with the hypothesis with the current observation to obtain an observation comparison metric indicating the similarity between the next predicted observation and the current observation; determining a direction and magnitude of movement required to reach the predicted observation based on the observation comparison metric; moving the agent according to the determined movement and the next predicted displacement so that the agent moves from the physical or logical space to a next position; generate a next observation from the next position of the agent in the logical or physical space, wherein the next observation includes data describing a next portion of the logical or physical space in a local neighborhood of the next position; and set the next observation to the current observation and the next position to the current position; and repeat the method until an observation comparison condition is reached, wherein the agent is located at a final position in the physical or logical space; measure the actual displacement from the original current position in the physical or logical space to the final position; compare the actual displacement with the next predicted displacement to obtain a displacement comparison metric; determine whether a hypothesis confirmation condition is met; wherein if the hypothesis confirmation condition is not met, set the next predicted observation to the current predicted observation, set the next observation to the current observation, and set the next position to the current position; retrieve a new next predicted displacement and a new next predicted observation from the set of predicted observations and displacements associated with the hypothesis relative to the current predicted observation, and repeat the method until the hypothesis confirmation condition is met; and determine the attribute of the physical or logical space based on meeting the hypothesis confirmation condition.

[0095] According to the tenth aspect of the present disclosure, there is provided: a computer-implemented method for parsing a physical or logical space based on a series of observations and displacements performed by an agent in the physical or logical space to determine properties of the physical or logical space, the method comprising: recording a first observation result based on a first position of the agent in the logical or physical space, wherein the first observation result includes data describing a first portion of the logical or physical space in a local neighborhood of the first position; processing the first observation result based on a stored set of predicted observations and predicted displacements to determine a hypothesis, wherein each stored predicted observation and predicted displacement in the stored set of predicted observations and predicted displacements is a hypothesis; Associated with one or more hypotheses; determining a direction and magnitude of a required movement based on predicted observations associated with the hypothesis; moving the agent according to the required movement so that the agent is displaced from the first position in the physical or logical space to a second position; recording a second observation based on the agent's second position in the logical or physical space, wherein the second observation includes data describing a second portion of the logical or physical space in a local vicinity of the second position; processing the second observation to verify the hypothesis associated with the physical or logical space; adjusting or maintaining the hypothesis based on processing of the second observation; and determining whether a hypothesis confirmation condition is met.

[0096] Processing the first observation to determine the hypothesis may include: comparing the first observation with the predicted observations in the stored set of predicted observations to determine a similarity measure between the first observation and the predicted observations in the stored set of predicted observations; and retrieving the hypothesis corresponding to the predicted observation in the stored set of predicted observations having the strongest similarity measure.

[0097] According to the eleventh aspect of the present disclosure, there is provided a computer-implemented method for generating a map of a physical or logical space based on a series of observations and displacements performed by an agent in the physical or logical space, the method comprising: recording a first observation result based on a first position of the agent in the logical or physical space, wherein the first observation result includes data describing a first portion of the logical or physical space in a local neighborhood of the first position; moving the agent by a first displacement so that the agent is displaced from the first position in the physical or logical space to a second position; generating a second observation result based on the second position of the agent in the logical or physical space, wherein the second observation result includes data describing a second portion of the logical or physical space in a local neighborhood of the second position; recording the second observation result; recording the first displacement from the first observation result to the second observation result; recording a first negative displacement from the second observation result to the first observation result; and repeating this process for at least n−1 displacements, n observations, and n−1 negative displacements to generate a set of connected n−1 displacements and n observations, which defines a map of the physical or logical space, wherein n is a positive real integer.

[0098] Recording observations and displacements can be thought of as producing observations and displacements by the agent.

[0099] Prior to recording a second or nth observation, the method may include: comparing the nth observation to a previous n−1th observation to determine an observation similarity measure between the nth observation and the n−1th observation; comparing the observation similarity measure to a predefined maximum similarity threshold; and if the similarity measure does not exceed the maximum similarity threshold, performing the following operations: recording the nth observation; recording the n−1th displacement from the n−1th observation to the nth observation; recording the n−1th negative displacement from the nth observation to the n−1th observation; or if the similarity measure exceeds the maximum similarity threshold, performing the following operations: recording the nth observation; recording the n−1th displacement from the n−1th observation to the n−1th observation; recording the n−1th negative displacement from the nth observation to the n−1th observation; The agent moves an additional displacement so that the agent moves from the nth position to the n+1th position in the physical or logical space; generates an n+1th observation result based on the n+1th position of the agent in the logical or physical space, wherein the n+1th observation result includes data describing the n+1th part of the logical or physical space in the local neighborhood of the n+1th position; compares the n+1th observation result with the previous n−1th observation result to determine an observation similarity measure between the n+1th observation and the n−1th observation; compares the observation similarity measure with the predefined maximum similarity threshold; and repeats this process until the predefined maximum similarity threshold is not exceeded.

[0100] The method may include: forming an identity based on the n observations and the n−1 displacements, wherein the identity includes at least a subset of the n observations and the n−1 displacements; and storing the identity.

[0101] The method may also include associating or labeling the identity and the corresponding subset of observations and displacements forming the identity with an attribute of the space.

[0102] Multiple identities can be formed from the n observations and the n−1 displacements.

[0103] The map and / or identity may include more or less than n−1 displacements, as an observation may have more than two displacements to other observations, e.g., imaging a set of three observations where each observation is connected to two other observations, forming a triangle with three observations and three displacements. It will be appreciated that for a large number of observations, the number of displacements may be higher than the number of observations.

[0104] According to the twelfth aspect of the present disclosure, there is provided a method for navigating a map of a physical or logical space generated according to the eleventh aspect described above, the method comprising: obtaining an overall target observation result, wherein the target observation result is an observation result of a set of n observation results and n-1 displacements; generating a first observation result based on a first position of an intelligent agent in the logical or physical space, wherein the first observation result includes data describing a first part of the logical or physical space in a local neighborhood of the first position; comparing the first observation result with the set of n observation results and n-1 displacements to identify a first target observation result in the set of n observation results and n-1 displacements, wherein the first target observation result is connected to the overall target observation result in the map of the physical or logical space. result and the observation that is most similar to the first observation result; based on a comparison between the first observation result and the first target observation result, determine a first movement from the first observation result to the first target observation result; reposition the agent to the first target observation result by moving the agent according to the first movement, and iteratively perform the following operations: move the agent toward the overall target observation result in the map through n-1 displacements and n observation results in the following manner: move the agent according to the n-1 displacements; and generate transition observation results at the n observations; compare the transition observation result with the n observations; compare the measured displacement with the n-1 displacements; and adjust the movement of the agent according to the comparison until the overall target observation result is reached.

[0105] According to the thirteenth aspect of the present disclosure, there is provided a computer-implemented method for determining the similarity between a portion of a physical or logical space that an agent is capable of navigating and stored spatial data for locating the agent in the physical or logical space, the method comprising: performing the following operations in sequence: recording a series of observations according to the corresponding position of the agent in the physical or logical space, wherein each observation includes data describing the respective portions of the logical or physical space in the local vicinity of the agent; moving the agent between the corresponding positions and recording a series of displacements between the corresponding positions; comparing each observation in the series of observations with a previously stored set of observations to obtain an observation similarity measure of each observation relative to one or more of the previously stored observations; comparing each displacement in the series of displacements with a previously stored set of displacements to obtain a displacement similarity measure of each displacement relative to one or more of the previously stored displacements; if the observation similarity measure and / or displacement similarity measure passes a confidence threshold, determining that the series of observations and / or displacements are similar to the previously stored observations and / or set of displacements.

[0106] The method may include combining the observation similarity metric and the displacement similarity metric to produce a two-dimensional identity comparison metric and processing the identity comparison metric to determine a similarity type between the observation and displacement and the previously stored observation and displacement.

[0107] According to the fourteenth aspect of the present disclosure, there is provided a computer-implemented method for performing instant positioning and map construction in a logical or physical space, the method comprising: recording a first observation result based on a first position in the logical or physical space, wherein the first observation result includes data describing a first part within the logical or physical space; performing a first displacement from the first position in the logical or physical space to an unknown second position; recording a second observation result based on the second position, wherein the second observation result includes data describing a second part within the logical or physical space; repeating the method for n observations and n displacements, wherein n is a positive real number, to form a sequential observation result set and a corresponding displacement result set; comparing the sequential observation result set and the corresponding displacement result set with a stored observation result set and a stored displacement result set, wherein the stored set is captured in a closed loop, wherein the comparison includes using a displacement comparison metric and at least one of one or more observation comparison metrics to compare to determine whether there is a feature corresponding to the stored set in the space.

[0108] The method can be effectively used to parse a multidimensional space based on a series of observations and displacements performed by an agent in the space in order to find a property of the space. The method includes: generating sequential observations and displacements according to the positions of the agent in the space, and comparing the observations and displacements with a stored set of observations and displacements to identify a hypothesis about the property of the space and testing the hypothesis by obtaining an observation comparison metric and / or a displacement comparison metric, and adjusting, maintaining or confirming the hypothesis based on the observation comparison metric and / or the displacement comparison metric.

[0109] Each of the above aspects involves an agent in a space that is configured to move and observe the space in a simplified manner such that a network of connected observations and displacements can describe the space in a map. Comparison of current observations and displacements made by the agent with observations and displacements stored on a map may affect the agent's behavior in terms of movement and function. In this way, the agent is able to observe the space for image and object recognition while navigating and / or exploring the space.

[0110] The agent may be a real object such as a robot, a vehicle, a self-driving car or a drone, etc., but is not limited to such possibilities.

[0111] A group of agents can work together and can communicate with a computer system or the like to update a global map or resource based on the spaces in which the different agents reside. BRIEF DESCRIPTION OF THE DRAWINGS

[0112] Embodiments of the present invention are described by way of example with reference to the following drawings, in which:

[0113] Figure 1 is a schematic diagram of the physical space to be parsed by the agent;

[0114] Figure 2 is a schematic diagram of the physical / logical space to be parsed by the agent;

[0115] Figure 3 is a flow chart outlining a method for finding properties of a space;

[0116] Figure 4 is a detailed flow chart outlining part of the method for finding properties;

[0117] Figure 5 is a schematic diagram outlining a system for performing the method;

[0118] Figure 6 is a schematic diagram of the functions performed by the agent in various methods;

[0119] Figure 7is a schematic diagram showing various ways in which an agent can move according to a method;

[0120] Figure 8 is a schematic diagram showing an identity that can be used to generate a map of the space; and

[0121] Fig. 9 and Fig.10 Various schematic representations of vectors and matrices obtained by various methods are shown;

[0122] Fig.11 shows a flow chart illustrating a process of forming sub-regions according to observation vectors according to various embodiments;

[0123] Fig.12 A schematic diagram showing a set of spatial sub-regions corresponding to a current observation vector according to various embodiments;

[0124] FIG13A is a schematic diagram showing a comparison between a sub-region of a stored observation vector and a sub-region of a current observation vector according to various embodiments;

[0125] FIG13B is a schematic diagram showing a comparison between a sub-region of a stored observation vector and a sub-region of a current observation vector according to various embodiments;

[0126] Fig.14 A schematic diagram showing a comparison of a stored observation vector and a current observation vector by an agent according to various embodiments;

[0127] 15A is a schematic diagram showing an axis of an agent superimposed with an offset determined based on a comparison of a current sub-region with a stored sub-region according to various embodiments;

[0128] 15B is a schematic diagram showing an overlap of an axis of an agent and an offset determined based on a comparison of a current sub-region with a stored sub-region according to various embodiments;

[0129] 15C is a schematic diagram showing an overlap of an axis of an agent and a motion vector determined based on a comparison of a current sub-region with stored sub-regions according to various embodiments;

[0130] 15D is a schematic diagram showing an overlap of an axis of an agent and a motion vector determined based on a comparison of a current sub-region with stored sub-regions according to various embodiments;

[0131] Fig.16 A flow chart showing a method using two navigation processes according to various embodiments;

[0132] Fig.17 A schematic diagram showing the results of a method of associating sensor data using metadata according to various embodiments; and

[0133] Fig.18 A schematic diagram showing results of a method of associating sensor data using metadata according to various embodiments.

[0134] Common reference numerals are used throughout the drawings to identify similar features. DETAILED DESCRIPTION

[0135] The examples described herein relate to a computer-implemented method and system for parsing a multidimensional space and determining the structure of the space or one or more properties thereof. The methods employed by the examples described herein treat the problem of understanding the space as a closed-loop problem, in which sensory data from sensors or virtual sensors describing the environment of an agent directly drives changes in the agent's behavior.

[0136] The term "agent" is used to refer to either or both physical agents that exist in real-world physical space (such as vehicles or robots) and non-physical agents that exist in logical space (such as points or pixels in an image or map).

[0137] An agent is able to move within its corresponding space. In physical space, an agent moves by activating a movement module, which can be, for example, an actuator or a motor. In logical space, an agent moves from a first point or pixel to a second point or pixel. An agent can be real or virtual. A real agent (such as a robot) occupies an area in space. A virtual agent is a projection on space, but does not actually exist in space. For example, a virtual agent can be a projection of the center point of the field of view of a camera observing a two-dimensional scene. The camera can move and / or rotate to change the center point of the field of view on the scene and thereby change the position of the virtual agent. This will be explained in more detail below.

[0138] Physical space refers to any three-dimensional space that is physically reachable or traversable by real-world objects, vehicles, robots, or any other agent. A three-dimensional space may include one or more other physical or material objects. A three-dimensional space may also encompass an interval of distance (or time) between two points, two objects, or two events, which an agent may interact with while traversing the physical space.

[0139] Logical space refers to a space with any set of points, such as a mathematical space or a topological space. Thus, a logical space can be, for example, an image, a map, or a conceptual map. A set of points can satisfy a set of postulates or a set of constraints. A logical space can correspond to a physical space as described above.

[0140] Data is captured relative to the location of the agent in space, and the captured data describes the environment of the space in the vicinity of the agent. Data capture is performed in physical space by sensors such as cameras, radars, tactile sensors, LIDAR sensors, or infrared sensors. Data capture is performed in logical space by virtual sensors that retrieve data of the logical space in the vicinity of the agent.

[0141] In physical space, there is an observability problem because not all of the physical space may be observable by the agent at any given time. The physical space may be too large for sensors to observe, and / or the physical space may include objects or terrain that occlude areas of the physical space.

[0142] In a logical space, all of the logical space can be observed simultaneously. For example, when the logical space is an image, the entire image can be stored in a memory that the agent can access.

[0143] However, in logical space, this is unnecessary and more efficient if the agent does not observe the entire logical space at once. Therefore, a virtual sensor may only obtain a portion of the logical space, thereby emulating a sensor in the physical space, because the virtual sensor only captures and / or retrieves the vicinity of the agent in the logical space.

[0144] Therefore, in both scenarios of agents in logical space and agents in physical space, it is not necessary for the agent to observe the entire corresponding space, and it is computationally advantageous if the agent only observes a portion of the corresponding space in the agent's vicinity.

[0145] When the agent is a non-physical agent in the logical space, it is not necessary to directly observe the vicinity of the agent through sensors or virtual sensors. For example, the vicinity of the agent may be a part or an area of ​​a pre-recorded image that defines the overall logical space.

[0146] The data captured by observing the space in the vicinity of the agent through a sensor or virtual sensor is defined as observation data. The observation data is used to generate an "observation result". Therefore, the observation result is the result of processing the observation data in a way that succinctly describes the environment in the vicinity of the agent.

[0147] In physical space, observations can be represented by matrices or vectors that provide spatial information derived from the observation data. For example, the observation data can be an image or point cloud data of the agent's surroundings.

[0148] In the logical space, the observation result can be represented by a matrix or a vector, which provides spatial information in the vicinity of the agent. The observation data from which the observation result is derived can be a set of pixels or points from the logical space around the agent. For example, the observation data can be a matrix of pixels located within a threshold distance that the agent can observe.

[0149] An agent can generate multiple different observations at different locations in space. The agent moves between these locations to generate observations. The vector between two observations is defined as a "displacement", which has both magnitude and direction. Thus, an agent moves between two specific locations, generating observations by traveling along the displacement between the two specific locations.

[0150] Each of the observations and displacements made by an agent in a space may be stored in a memory associated with the agent. If an agent makes a series or sequence of observations and displacements in a space, the series or sequence of observations and displacements may be stored together as a set of observations and displacements. A sequence of observations and displacements may form an "identity". An identity may describe a particular feature or property of the space in which it resides. The terms property and feature are used interchangeably below.

[0151] For example, in a physical space, an identity may describe properties such as a particular object or a path to a particular destination in the space. The observations in the set that form the identity indicate how the agent perceives the space and therefore how it perceives particular features. For example, if there is a particular object in the space, such as a chair, then the identity of the chair may include several observations taken at different locations of parts or aspects of the chair. Similarly, if there is a particular destination in the space, then the identity may include several observations taken at locations on the path to the destination. The displacements in the set that form the identity indicate how the agent moves between the locations of the observations. For example, a first observation of the chair in the space may be made at a location near the back legs of the chair. A second observation of the chair in the space may be made at a location near the front legs of the chair. The first displacement of the identity is the displacement required to move from a location near the back legs of the chair to a location near the front legs of the chair.

[0152] Figure 1 Example identities are illustrated. Figure 1A series of observations 101 connected by a series of displacements 102 in an exemplary room space including a door and a central obstacle are shown. The exemplary room space may be a physical space, such as an apartment in the real world. Observations 101 may be captured using a camera on a robot. Displacements 102 may be generated by moving the robot. In this example, an identity indicates a path to a door, which may be considered a destination. Each observation in observations 101 provides data describing the environment at the location where the observation was generated.

[0153] although Figure 1 The displacements illustrated are unidirectional, but it will be appreciated that a corresponding set of negative displacements may also be recorded, such that the identity comprises an observation 101 and a set of displacements 102 and negative displacements connecting the observations.

[0154] like Figure 1 The property of the space of the illustrated identity may be a destination, such as a door. Thus, the identity may be used for the following purposes: navigating within the room, navigating around a central obstacle, and navigating to a door.

[0155] Figure 2 A second example of identity identification is illustrated. Figure 2 A series of observations 201 are shown connected together by a series of displacements 202 in a chair. The chair may be three-dimensional in a physical space (such as the real world) or two-dimensional in a logical space (such as an image). Figure 1 , displacement 202 may also include an opposite negative displacement.

[0156] The camera may be responsible for capturing observations 201 and performing displacements 202 in physical space, for example, to obtain Figure 2 The identification of what is seen. In logical space, this can be performed by a virtual agent.

[0157] In the case of a camera in a physical space, the camera may or may not be in a fixed location. If in a non-fixed location relative to the space, the camera may move within the space. For example, the camera may be attached to a robot. In this case, the agent may be the robot (or camera), and observation data may be recorded at the location of the agent simply by recording one or more images of the robot or camera's surroundings. If in a fixed location, the camera may be rotated or zoomed, etc., so as to focus on different parts of the chair. In this case, the camera's target and focus (i.e., gaze direction) may represent the agent in terms of the camera's field of view. In this case, for example, the agent may thus be defined as the center point or projection of another specific pixel in the camera's field of view. The camera does not move in a translational manner, but the agent "moves" when the camera zooms or rotates, because such actions modify the center point or location of a pixel in the camera's field of view relative to the observed scene / image. Therefore, the location of the agent does not necessarily correspond to the location of the camera. When fixed, the camera may not be displaced between observations to obtain the displacement 202. Instead, the displacement may be generated and stored as the distance between two fixed points in the camera's field of view in two observations. The agent can be "moved" by rotating the camera, and translation can be performed by moving the center point of the camera's field of view.

[0158] For example, a camera may first capture an image centered on a chair leg. This image is then processed as a first observation. The position of the agent is considered to be the center point of the camera's field of view relative to the first image. The camera may then pan upward by rotation and capture and process a second image of the back of the chair. Thus, the second observation is processed based on the image of the back. For this second observation, the position of the agent is considered to be the center point of the camera's field of view relative to the second image, and the displacement between the first observation and the second observation is the distance (in pixels and / or related to the size of the rotation) between the center point of the field of view relative to the first image and the center point of the field of view relative to the second image. For example, visual inertial odometry may be used to estimate this distance.

[0159] When the camera is not fixed, the agent is the camera itself (or the device to which it is attached). In this example, the camera / device moves to a certain location, makes observations at that location, and then performs displacements. As will be understood, the displacements are recorded, for example, using an odometer.

[0160] In the case of logical space, Figure 2An image of a chair can be represented. In this example, the entire image is stored, so there is no observability problem and the entire image can be observed at the same time. However, for computational efficiency, the logical space is treated in the same way as the physical space, in that the entire image is not processed at the same time, but rather smaller observations are made on portions of the image. Specifically, a portion of the image plane is observed, such as the portion that includes the legs of the chair. The size of the portion can be fixed or can depend on the size of the image. As with the fixed camera example in physical space, the position of the agent is considered to be a specific point in a portion of the image, such as a specific pixel (such as the center pixel of the image portion).

[0161] In this case, the "agent" or virtual agent can then perform a displacement by "moving" a number of pixels in the image plane before producing a second observation. In practice, this simply involves fetching a second portion of the image plane from memory, whereby the second portion is centered on the agent's new position (the pixel to which the virtual agent has "moved"). Thus, in logical space, the agent is a tool for selecting different parts of the image based on a fixed point in the image. Moving the virtual agent consists of translating from the pixel where the virtual agent was previously located to a different pixel.

[0162] Therefore, when referring to an "agent", it should be understood that an agent can be three things. First, an agent can be a physical device that can move, so that the agent physically moves to a certain position in the physical space to produce observations at the position of the device. In this case, the agent performs displacement between observations by moving the physical device, and the observation data corresponds to the data captured by the agent at its current position. Secondly, the agent can be a projection of a fixed physical device. In this case, observations are produced at different orientations of the fixed physical device, and the agent is a projection of a point in the field of view of the physical device that is related to the orientation of the physical device. For example, the agent is located at the center point of the field of view of the fixed physical device. In this case, the displacement is measured based on the distance between the projections of the points in the field of view of the physical device, which is caused by the change in the orientation of the physical device. Third, the agent can be a virtual agent in a logical space. In this case, the agent is located at a certain point or pixel in the logical space. The point or pixel specifies the portion of the space around it that corresponds to the observation data. For example, the agent may be located at the center pixel of the portion of the observation data corresponding to the observation in the space. The displacement is measured as the distance between pixels or points in the logical space.

[0163] Thus, when referring to an agent and specifically to the "location of the agent" in the foregoing description, it should be noted that any of the above definitions of "agent" may apply. The location of the agent need not be the location of the physical device. Similarly, when referring to an agent performing an action (such as moving) or making an observation, the above definitions also apply, such that the physical device need not be moved, and the observations may be made by the physical device or a computing device (not necessarily the agent itself).

[0164] return Figure 2 , the attribute of the space defined by the identity formed by the series of displacements 202 and the observations 201 may be the identity of the feature, in this case, the chair.

[0165] therefore, Figure 1 and Figure 2 It is shown that the identity can be formed in two or three dimensions in both physical and logical space and for the following purposes: identifying an object such as a three-dimensional chair, identifying an image or image feature such as a two-dimensional chair, or navigating in three-dimensional space to reach a destination. It should be understood that the identity has other uses, and object / image recognition and navigation are just examples.

[0166] Such as Figure 1 and Figure 2 Several identities, such as the identity in, may be stored in a stored set of observations and displacements from prior experience in one or more spaces. There is no need to label the identities, meaning that in the above example, there is no need to label the identity corresponding to a chair as a chair, or even to know that it corresponds to a chair. Instead, once the observations and displacement sets are stored as an identity, the agent is able to use that information to locate and identify matches to that identity in other areas of the space or in other spaces entirely. The system or method does not require understanding of real-world features associated with the identity.

[0167] In some examples described below, the identifiers may be labeled according to features or attributes of the space they represent. In the above example, therefore, the identifier corresponding to a chair may be labeled as a chair. The process of labeling the identifiers in the stored set of observations and displacements may be performed as part of a general training process, wherein the training process is performed according to prior art as will be understood by those skilled in the art. For example, various machine learning techniques may be used, including the use of supervised and unsupervised learning processes, neural networks, classifiers, etc.

[0168] For example, a stored set of observations and displacements may be obtained from a test space (such as a test image) that includes or illustrates one or more known features. For example, one or more test images of a chair may be presented to the agent such that an identity formed by the one or more images and stored in the stored set of observations and displacements is known to correspond to the chair and is labeled as such. The process of obtaining a stored set of observations is described in more detail later.

[0169] It is computationally efficient to use a set of observations and displacements to effectively describe the features of a space rather than an entire map or image of the space, because the observations and displacements only include the corresponding portion of the feature and therefore require less storage space and computation to identify the feature in a portion of the space. In addition, observations and displacements can be calculated using different data (such as vectors encoded according to the output of a set of spatial filters applied to the observed visual data) and local temporal correlations of low-level features in the case of displacements. This eliminates common failure modes and thus enables the two parts of the system (observations and displacements) to make up for each other's weaknesses, thereby providing diversity. In the application field of 3D spatial SLAM, the system not only performs positioning and map construction, but also essentially performs navigation and path planning, which can be fully implemented in the real world without adding additional components.

[0170] Compared to standard SLAM, the method and system described herein are more robust to environmental factors and occlusion. The method and system described herein also have the advantage of storing the minimum set of measurements required for navigation and observation of a specific space. Observations are generated in space according to the complexity and / or changes in the space, making the method and the system suitable for optimal execution in any space. For example, compared to a feature-rich environment, in an open field or area with little information and features, fewer observations will be generated and stored. Since the method does not involve tracking features or objects facing a small part of the field of view, but fully uses the characteristics of most of the entire spherical solid angle, compared with traditional systems and methods, the occlusion of a portion of the visual scene or the change of a portion of the visual scene has a smaller impact on the ability of the system to complete the task.

[0171] Features or attributes associated with stored observations and displacement sets may be identified in different areas of the same space or in entirely new spaces. For example, in a physical space, such as a living room or dining room, a stored observation and displacement set associated with a chair may be used to identify the same or similar chair in another part of the living room or dining room. Additionally, the stored observations and displacement sets may be used to identify the same or similar chair in an entirely different space, such as a separate building.

[0172] Similarly, a destination in space (such as a store or landmark) can be navigated from a first known or previously visited location in the world based on a stored set of observations and displacements linking the origin to the destination. Additionally, a destination in space can be navigated from an unknown area of ​​the world or an unknown origin based on a set of observations and displacements.

[0173] Identifying the properties of a space in this way relies on comparing a set of stored observations and displacements with the current observations and displacements generated in a particular space. The following description section will explain the process in more detail. In the following description section, the stored observations and displacements are referred to as predicted observations and displacements and target observations and displacements. It should be understood that these terms define the same technical features, and the only difference is that the context about why or when to use them is given. Similarly, the current observations from the space are referred to as first observations, second observations, and transition observations. Similarly, these terms define the same technical features (the technical features are the observations generated by the agent in the space where it resides), but are labeled in different ways to provide context about why or when to generate or use them. The same terminology also applies to displacements.

[0174] References below Figure 1 A method is described in detail in which the process of identifying / determining the properties of a space is performed using existing knowledge in the form of stored sets of observations and displacements. The benefit of this method is that only portions of the space are needed to identify the properties, given the current observations and displacements.

[0175] Figure 3 A flow diagram of a method 300 of determining properties of a space in which an agent may explore or move is shown, according to various examples.

[0176] In the initial conditions of method 300, the agent is in a space and is able to observe a portion of the space through sensors, virtual sensors, etc. The space may have been previously explored by the agent or may be completely unknown to the agent. The starting location may be known to the agent or may be completely unknown to the agent.

[0177] In a first step 301, an agent in the space generates an observation based on its current position in the agent's space. The observation may be considered a current observation or a first observation from a current position or a first position, although it should be understood that the observation is not required to be the first observation in the space. Instead, "first" is used here only to distinguish between a current observation and subsequent observations.

[0178] The first observation includes data describing a first portion of the space in the local vicinity of the agent. For example, in the case where the agent is a robot or a vehicle, the first observation may be generated using sensor data from a sensor (such as a camera, radar, or LIDAR, etc.), wherein the sensor data indicates a world view from the current position of the robot. The local vicinity of the agent is the local vicinity of the current position of the robot. In physical space, the range of the local vicinity is determined by the constraints on the sensor and the environment of the robot. For example, the sensor may have a limited range, so that data for observing sensor data is only obtained for the environment of the space within the range. Similarly, the environment of the robot may include one or more features that limit the observable surrounding environment, such as walls or other obstacles that block the field of view of the sensor. Therefore, the vicinity of the agent is not fixed, but it may have a maximum distance from the position of the agent according to the constraints of the sensor. In logical space, the local vicinity of the agent is a portion around the position of the agent and is smaller than the entire space. Since the entire space can be stored (for example, if the space is an image), it is not necessary to obtain data from the sensor for observation. Instead, the portion of the space around the position of the agent is retrieved from the stored entire space. This "virtual sensor" simulates the same effect as using a sensor in a physical space, where the observation only considers a portion of the entire space instead of the entirety. Therefore, the vicinity of the agent in the logical space can be set based on the distance from the agent's location. In an example where the space is an image and the agent is located at a pixel in the image, the vicinity can be set, for example, by a number of pixels away from the pixel of the agent.

[0179] Once a first observation is generated in a first step 301 , it is compared in a second step 302 with stored observations and stored observations in a set of displacements.

[0180] The comparison between the first observation and the stored observations produces an observation comparison metric that indicates how similar each of the stored observations is to the first observation. The observation comparison metrics for each stored observation are sorted; the highest observation comparison metrics are identified and the stored observations corresponding thereto are retrieved. In this manner, the stored observations that are most similar to the first observation are determined and selected. The process of comparing observations is described in more detail later.

[0181] In a third step 303, a hypothesis is determined regarding the stored observation that is most similar to the first observation. Specifically, the stored observations and displacement sets include one or more identities that form subsets of the stored observations and displacement sets. Each subset includes one or more observations and one or more displacements, each of which is associated with a specific attribute or feature to which the identity relates. In a third step 303, an attribute or feature associated with the selected stored observation is determined. The attribute forms the basis of a hypothesis, which is actually a prediction of what the agent is observing in space. For example, the prediction can be a two-dimensional object or image or a three-dimensional object or image, or a destination that can be navigated to.

[0182] The stored set of observations and displacements may include several subsets of observations and displacements, each of which is associated with a particular attribute and, therefore, with a particular hypothesis. In one example, there is a subset associated with the attribute "chair", a subset associated with the attribute "table", and a subset associated with the attribute "door". The comparison in the second step 302 is performed relative to all observations in the stored set of observations and displacements, and may result in the observation comparison metric being the highest or strongest for a particular observation that is part of the subset associated with the attribute "door". In the third step 303, the hypothesis thus becomes: the first observation is part of a door, or in other words, the attribute of the space that the agent has observed is a door in the space. The subset of stored observations and displacements associated with that attribute is called the hypothesis subset.

[0183] In a fourth step 304, observations and displacements of a hypothesis subset are retrieved. Specifically, the stored observation that is most similar to the first observation is part of the hypothesis subset of stored observations and displacements as described above. In addition, the observations and displacements in the hypothesis subset are connected so that they form an observation network separated by displacements. The links between the observations and displacements in the hypothesis subset are stored in the stored observations and displacements set so that the network arrangement is saved in the memory. When retrieving the hypothesis subset of observations and displacements, the method includes retrieving the observations and displacements connected to the stored observation that is most similar to the first observation. In other words, the adjacent observations and the displacements required to reach the adjacent observations from the stored observation that is most similar to the first observation are retrieved from the hypothesis subset. The observations and displacements become predicted observations and predicted displacements, whereby, if the agent moves the predicted displacement from its current position, then according to the hypothesis, it is expected that the agent will reach a position in space where it can observe the predicted observations.

[0184] It should be understood that the terms "predicted observations" and "predicted displacements" refer only to the subset of observations and displacements associated with the determined hypothesis. Each of these observations and displacements is predicted to occur in the space currently occupied by the agent according to the hypothesis.

[0185] In a fifth step 305, the hypothesis is tested. To test the hypothesis, the agent moves sequentially along a network of stored observations and displacements (or in other words, predicted observations and predicted displacements) that form a subset of the hypothesis. In each iteration of this sequential process of the agent moving along the network, the agent is configured to repeatedly perform new observations, compare the new observations with the predicted observations, and / or compare the predicted displacements with the actual displacements, and determine whether to maintain the hypothesis, reject the hypothesis, or confirm the hypothesis. This hypothesis testing will be described in more detail below with reference to steps 305-1 to 305-5.

[0186] In the first step 305-1 of the hypothesis testing process, the agent moves from its current position (the position that produced the first observation) to a second position in space based on a movement function. The movement function depends on the predicted displacement of the hypothesis subset. Specifically, the movement function depends on the predicted displacement connected to the stored observation that is most similar to the first observation, which is identified from the linked network of observations and displacements that form the subset. Therefore, the second position is unknown to the agent until it has performed a movement based on the movement function.

[0187] In a second step 305-2 of the hypothesis testing process, the agent generates a second observation based on a second position of the agent in the space, wherein the second observation includes data describing a second portion of the space in a local vicinity of the second position. For example, where the agent is a robot or a vehicle, the second observation may be generated in the same manner as the first observation using sensor data from a sensor such as a camera, radar, or LIDAR, wherein the sensor data is processed into an observation indicating a world view from the robot's location at the second position.

[0188] In the third step 305-3 of the hypothesis testing process, the actual displacement of the agent from the first position to the second position is determined. In physical space, the actual displacement can be estimated, for example, using an odometer. In logical space, the actual displacement can be measured using vector or matrix calculations according to known techniques. For example, if the logical space is an image, the number of pixels between the first position and the second position can be determined as the distance. In physical space, the displacement is measured using an odometer system that includes a visual inertial odometer combined with a motion odometer such as a wheel odometer (if applicable). The system provides input interfaces for the visual inertial odometer and the motion odometer. The displacement indicates the physical or logical distance between the pair of observations. The odometer is stored in a coordinate system that is referenced to the rotation of the starting observation (the predicted observation that is most similar to the first observation). Additional data characterizing other aspects of the displacements may be stored, such as, but not limited to, vibrations encountered when converting the displacements or energy consumed when converting the displacements, displacement similarity being measured by endpoint errors between pairs of displacements, where each displacement starts at the same location and the endpoint error is the physical or logical Euclidean distance between points located by converting along the two displacements.

[0189] In the fourth step 305-4 of the hypothesis testing process, a comparison is performed. The comparison compares the observations, the displacements, or both. The second observation can be compared with the predicted observations from the hypothesis subset, and specifically with the predicted observations linked to the predicted displacements on which the movement function is based. The comparison between the second observation and the predicted observation provides an observation comparison metric. Similarly, the predicted displacement on which the movement function is based when the agent moves from a first position to a second position in space can be compared with the actual displacement between the first position and the second position to obtain a displacement comparison metric. Therefore, a comparison can be made to generate both an observation comparison metric and a displacement comparison metric. As will be described in detail later, it is advantageous to use both metrics. It should be understood that the order of the above steps is interchangeable after the agent moves.

[0190] In a fifth step 305 - 5 of the hypothesis testing process, the hypothesis is adjusted, maintained, or confirmed based on the observation comparison metric and / or the displacement comparison metric from the fourth step 305 - 4 of the hypothesis testing process.

[0191] In this step, the observation comparison metric and / or the displacement comparison metric are evaluated to effectively determine whether the agent's perception of the space it occupies (in terms of the second observation and the actual displacement) matches its perception of the properties according to the hypothesis (in terms of the predicted observation and the predicted displacement). The observation comparison metric and the displacement comparison metric are described in more detail later, but in general, the stronger these metrics, the more likely the hypothesis is correct and the agent is actually observing the properties according to the hypothesis in the space it currently resides.

[0192] The observation comparison metric and / or the displacement comparison metric may be compared to one or more corresponding thresholds to determine whether to confirm, maintain, or reject the hypothesis.

[0193] Adjusting the hypothesis includes rejecting the hypothesis. This occurs when it is determined based on the hypothesis that the space the agent is observing is unlikely to include or represent the attribute. This conclusion can be reached in several different ways. First, the hypothesis can be rejected when the observation comparison metric and / or the displacement comparison metric is below a first observation threshold level and / or a first displacement threshold level. The first observation threshold level and the first displacement threshold level can be globally set to the minimum likelihood required to maintain the hypothesis and for the agent to further test the hypothesis. Alternatively, these thresholds can be adaptive and may be able to change based on environmental conditions of the space (such as brightness / darkness factors). Specifically, if the agent is generating observations at an ambient brightness level that is different from the brightness level at which the subset of stored observations associated with the hypothesis was captured and stored, the lighting in the space may affect the observation comparison metric. The thresholds can also be adjusted based on color, contrast, and other image / sensor data characteristics.

[0194] In addition, different observation comparison metrics can be used to address different additional aspects of the space, and when used in combination, allow for the identification of specific dimensions that differ from the assumptions. For example, lighting changes have a greater impact on vector comparison metrics than they do on metrics of the spatial distribution of filtered elements around the agent's location. Changes in the color of an object will have a greater impact on the color-filtered elements of the observations than on the brightness- and orientation-filtered elements. Considering multiple observation comparison metrics allows the properties of dissimilarity to be observed. In the case of changing lighting conditions, a metric of the spatial distribution of filtered elements of the observations around the agent's location can be used to adjust the threshold so that a strong match of this metric can be used to update the prior assumptions and thereby lower the acceptance criteria (second observation threshold level) of the "similarity" observation comparison metric. The different types of observation comparison metrics discussed here each come from a different method of comparing observations. This will be explained in more detail later.

[0195] The first observation threshold level can be set based on an initial ranking of the observation comparison metric for each stored observation relative to the first observation generated by the agent. Specifically, the first observation threshold level can be set equal to the second highest ranked observation comparison metric, which corresponds to a stored observation in the stored observation and displacement set that is not in the subset associated with the current hypothesis. Therefore, this particular stored observation is associated with a different hypothesis. In this way, the first observation threshold level is set based on the similarity of the first observation to the stored observations, which are associated with a hypothesis different from the hypothesis currently being tested. If the observation comparison metric generated by the tested hypothesis is weaker than the observation comparison metric originally found for the different hypothesis, then this effectively allows the agent to consider a hypothesis different from the hypothesis being tested. Therefore, the hypothesis being tested can be rejected and ultimately replaced with a different hypothesis associated with the first observation threshold level.

[0196] Ultimately, adjusting the hypothesis involves rejecting the current hypothesis and replacing it with a different hypothesis. This may involve the method returning to Figure 3 The illustrated second step 302 is to select the next best observation comparison metric among the ordered observation comparison metrics to determine the next best observation and the hypothesis associated therewith. Before selecting a new hypothesis, the agent may also return to the position of the first observation in space via a displacement opposite to the actual displacement made. The replacement of hypotheses is described in more detail later.

[0197] When the hypothesis is neither adjusted nor confirmed, the hypothesis is maintained. Therefore, when the observation comparison metric and / or the displacement comparison metric matches or exceeds the first observation threshold level and / or the first displacement threshold level, respectively, the hypothesis is maintained. This means that the hypothesis does not need to be adjusted. According to the example introduced above, in which the first observation threshold level is set according to the "second highest observation comparison metric", this means that the current hypothesis still provides the highest observation comparison metric relative to the predicted observations in any subset of the stored observations and displacement sets, meaning that the attributes according to the current hypothesis are still the attributes most likely to be found in the space where the agent resides. Another condition that leads to the maintenance of the hypothesis is that the hypothesis confirmation condition is not met. The hypothesis confirmation condition can be a second higher observation threshold level and / or a second higher displacement threshold level. The hypothesis confirmation condition indicates that the hypothesis is correct and the attributes of the space are according to the minimum confidence level of the attributes of the hypothesis.

[0198] In other words, the assumption is maintained when it is determined that the observation comparison metric and / or the displacement comparison metric relative to the predicted observation and predicted displacement matches or exceeds the first observation threshold level and / or the first displacement threshold level, respectively, but does not match or exceed the second observation threshold level and / or the first displacement threshold level, respectively.

[0199] When a hypothesis is maintained, the next predicted observation and the next predicted displacement linking the current predicted observation to the next predicted observation are retrieved from the subset of stored observations and displacements associated with the hypothesis. Then, with respect to the next predicted observation and the next predicted displacement, as Figure 3 The process of testing hypotheses set forth above is repeated as shown. In this manner, method 300 involves sequentially testing subsets of hypotheses predicting observations and displacements until the hypothesis is confirmed or adjusted.

[0200] At each observation made by the agent (for which the hypothesis is maintained), the historical observation and / or displacement comparison metric is updated and then incorporated into future observations and / or displacement comparisons. The historical observation and / or displacement metrics are stored in a memory and serve to increase future observations and / or displacement comparison metrics based on the number of consecutive times the hypothesis has been maintained. This effectively increases the confidence that the hypothesis can be confirmed because the hypothesis has not been rejected due to multiple consecutive observations and displacements. It also helps prevent the agent from falling into a local minimum, whereby the hypothesis is never rejected or confirmed. This process using historical observation and / or displacement metric data can include a single hypothesis sequential probability ratio test for the null hypothesis (i.e., the data does not match the identity), or as a multiple hypothesis sequential probability ratio test between alternative hypotheses including the null hypothesis (i.e., all hypotheses do not match).

[0201] Confirming the hypothesis includes: determining that a hypothesis confirmation condition is satisfied, and determining the property of the space based on satisfying the hypothesis confirmation condition. As described above, the hypothesis confirmation condition can be a second higher observation threshold level and / or a second higher displacement threshold level that the observation comparison metric and / or the displacement comparison metric must match or exceed.

[0202] Similar to the first observation threshold level, the second observation threshold level may be modified or adjusted using a brightness / darkness factor based on lighting conditions in the space.

[0203] When the hypothesis is confirmed, method 300 can stop because the agent has identified a property of the space. The identity of the property can be stored in memory, and the observations and displacements made by the agent to identify the property can be stored in memory and linked to the space where the agent resides.

[0204] Method 300 advantageously uses observations and displacements to determine properties of a space without having to consider the entire space as a whole. This provides a more computationally efficient method of spatial resolution.

[0205] The method 300 will now be described in more detail. First, reference will be made to Figure 4 A method 400 for generating and comparing observations is described. In method 300, observations are generated and compared at least at two points. First, in a first step 301 and a second step 302, where the agent generates observations and compares them to a stored set of observations to ultimately determine a hypothesis; and secondly, during a fifth step 105, the hypothesis is tested. Thus, the process requires a minimum of two observations and two comparisons. However, it should be understood that more observations may be performed, and that it is advantageous to generate observations in sequence, not only in the two cases described above, but also in the transition between a first position and a second position, when the agent is moving from the position of a first observation to the position of a second observation. These transition observations are discussed in more detail later. The process of generating and comparing observations is described herein.

[0206] In a first step 401, observation sensor data is captured. In physical space, observation data is input data captured using a sensor, such as a camera. In this example, the observation data is a captured image. The captured image and its content depend on the orientation of the camera. In particular, the camera may be in the same location, for example, attached to a robot at the same location in the world. However, if the orientation of the robot is modified, the content of the observation data may change even if the location of the robot has not changed.

[0207] Therefore, it is advantageous to produce observations in physical space using a camera system or other sensor capable of capturing a three-axis stabilized cylindrical projection of a three-dimensional space, which is oriented with the main axis of the cylinder perpendicular to the direction of gravity. Fixing relative to gravity in this way allows observations to be made at the same roll and pitch angles (because these observations can also be fixed according to the direction of gravity). The capture resolution of the image can be, for example, 256 columns × 64 rows of pixels, but it can also be any resolution. The 256×64 pixel image represents the input data of the observation, which is then manipulated in the second step 402. In a non-physical or logical space, the observation data may be data obtained from the logical space around the position of the agent. The entire logical space can be pre-stored. The observation data may come from a portion within a predetermined distance of the position of the agent in the logical space. For example, in the case where the logical space is an image or a map, the observation data may be pixels or points contained within a pixel / point distance of 5, 10, 50, 100 or any number of pixels from the position of the agent.

[0208] In a second step 402, observations are generated from input data captured by sensors, virtual sensors or retrieved from memory. The input data is encoded into a set of vector / matrix representations through pipeline stages.

[0209] The process of generating observations to be stored to form part of a set of stored observations and displacements is very similar to the process of generating current observations based on input data for comparison with stored observations. However, when generating current observations for comparison, the relative orientation of the agent at its current location and the orientation of the agent when the stored observations were generated may not be the same, and may not be easily distinguishable. To address this issue, when generating current observations based on the input data, the process includes generating observations for each possible permutation of the agent's orientation. In practice, this means that the current observations are associated with multiple rotations, whereby each rotation represents that the input data has been shifted or adjusted to represent a different orientation of the agent. See below for how this affects the process compared to generating "stored" observations.

[0210] The general process for generating observations (current or stored) consists of the following steps.

[0211] First, low-level features are extracted from the input data. For example, this may include preprocessing the input data using one or more filters and / or masks. The low-level features may be edges, colors, or shapes, etc. The preprocessing may also include color conversion, etc., to provide multiple channels of the input data. The result of this first step is processed data, which may include several channels.

[0212] Second, the resolution of the input data is reduced using a filter or the like by extracting or deleting portions of the input data (such as rows of pixels), and / or summing and averaging large areas of the input data.

[0213] Third, further processing may be performed using one or more kernels to convolve the input data to obtain useful results (e.g., to identify regions of interest, edges, areas of greatest variation, etc.) If there are several channels, this process is performed on all of them.

[0214] Fourth, the resulting data forms a vector or matrix of the processed, simplified input data, which is then concatenated across dimensions (columns, rows, and channels) and normalized to produce the observation vector o.

[0215] As described above, when storing observations (e.g., for the purpose of storing a set of observations for identification or mapping), only one observation vector o is needed. When generating a current observation for comparison purposes, the general process is performed for multiple rotational permutations of the input data. In the case of data with a resolution of X×Y, this may mean including X permutations, each representing an X+1 shift from the previous one. This results in X observation vectors for the current observation, each of which can then be compared to a stored observation.

[0216] For example, encoding the input data to form one or more observation vectors o using a computer device such as a central processing unit (CPU), a graphics processing unit (GPU), or a processor.

[0217] unit (GPU), field programmable gate array (FPGA), or application-specific integrated circuit (ASIC).

[0218] A detailed example of this process is provided below for the above example of an image captured from a camera, where the image is a cylindrical projection. In this example, the resolution of the image is 256×64 pixels, so for the current observation, there are 256 possible rotations of the projection, whereby the columns are rotated (shifted) from where they were originally captured. As described above, these rotations represent different permutations of the original observation data. It should be understood that there may be any number of permutations depending on the input resolution of the image. The detailed process is as follows.

[0219] In the first stage, for each pixel, a color conversion process may be performed, thereby converting the color channel data from the captured image. For example, red, green, and blue channel data are converted to red, green, and lightness channels, where lightness is 1 / 3 (red + green + blue). It should be understood that these colors and conversions are exemplary, and the image may have different color channels and may be converted to different color channels.

[0220] In the second stage, the luma channel result of the first stage is convolved with a filter using the convolution kernel [1, -1; 1, -1; 1, -1] or a similar vertical edge filter (e.g., Sobel filter or other filter) to create a luma vertical (luma_v) channel.

[0221] In the third stage, the luma channel result of the first stage is convolved with a filter using a convolution kernel of [1,1,1; -1, -1, -1] or a similar horizontal edge filter (e.g., a Sobel filter or other filter) to create a luma horizontal (luma_h) channel.

[0222] This process produces a new image containing pixels of the following four channels: red, green, luma_h, and luma_v. It should be understood that other choices of pixel filters can be used here, and for the set of selected filters, all sub-pixels from all channels (red, green, luma_h or luma_v) are passed to the next processing pipeline.

[0223] In the fourth stage, a series of box filters are calculated and, for each channel, centered around a fixed number of evenly spaced rows of the image, wrapping around those rows. It should be understood that there may be more or fewer filters, and that the size of 19×19 is exemplary. For each channel, a box filter smoothes the image. Each pixel of each channel is modified by the box filter according to the sum of the pixels contained within each box filter, thereby providing a smoothed (or blurred) image. The box filters may overlap each other.

[0224] In the fifth stage, the blurred image is reduced to form a slice image. Specifically, the resolution of the blurred image is significantly reduced. Fig. 9 Specifically, four evenly spaced rows 902 are extracted from a blurred image (not shown) processed according to the original image 901 (or other observed data sets) to generate a slice image 904 .

[0225] Therefore, in Fig. 9 In , the slice image consists of 256 columns and 4 rows.

[0226] In the sixth stage, the columns of the slice image formed according to the fifth stage are divided into sets of evenly spaced columns. In one example, 256 columns are divided into 256 sets, each set includes an image of 16 columns x 4 rows, where the first column of each image is different from the complete set of 256 columns. Fig. 9 As can be seen in , there are 16×16 sets of 16×4 images (a total of 256 permutations), so the columns are displayed in different colors to illustrate how the data from the 256 columns of slice image 904 can be divided into 16×16×16×4 sets. These sets represent each possible rotation location of the input data (for all possible 256 rotations, in increments of one column rotation). It should be understood that Fig. 9 are illustrative examples, and there may be more or fewer rows, columns, and permutations, meaning there may be more or fewer than 256 possible rotations.

[0227] When observations are to be stored in memory to form part of a stored set of observations and displacements, only one of the 16×4 images in the set formed in the fifth stage need be stored. For example, the first arrangement 906 of the first set may be selected to be stored as an observation. When generating observations for comparison with previously stored observations, only the rotations (and therefore the 256 possible arrangements of the 16×4 image) need be considered. Therefore, arrangements only need to be considered for the current observation. When making comparisons, it is still not necessary to store every arrangement of the 16×4 image in memory. Instead, each arrangement is iteratively compared to the stored 16×4 observation vectors to identify the best match indicated by one or more observation comparison metrics. There is no need to store these arrangements. Comparing observations is described in more detail later.

[0228] Returning to the detailed example of producing the observed results, in the seventh stage, the 16x4 image is circularly convolved with a central excitation-peripheral inhibition (or negation) kernel. For example, the element with value 1 is the kernel center. For example, for the first row, the kernel can be: [-1 / 5, 1, -1 / 5; -1 / 5, -1 / 5, -1 / 5]; for the second and third rows, the kernel can be [-1 / 8,-1 / 8,-1 / 8; -1 / 8,1,-1 / 8;-1 / 8,-1 / 8,-1 / 8]; and for the fourth row, the kernel can be [-1 / 5,-1 / 5-1 / 5;-1 / 5,1,-1 / 5]. Fig.10 An example of this is shown in the figure showing a central excitation-peripheral inhibition horizontal surround filter 1001 to 1003 for a 16×4 image. The kernel identifies the areas of greatest variation in the image. As described above, when observations are stored, this need only be done for one 16×4 image. When a current observation is generated for comparison with a stored observation, a comparison is made to each of the separate sets (i.e., the 16×16 sets from the sixth stage).

[0229] According to the pipeline, a 16×4×4 image is generated, or in the case of comparison, 256 rotations of a 16×4×4 image are generated. There are 16 columns, 4 rows, and 4 channels. The channels of the 16×4 image can be subdivided into two data subsets, one consisting of red and green, and the other consisting of luma_h and luma_v. Each of these data subsets is normalized via vector normalization so that the elements of each data subset describe the values ​​on the orthogonal value set, and the result is a unit vector. These normalized vectors are then concatenated into a single unit vector by scaling each normalized vector with the square root of two. This provides an observation vector o. The observation vector in this example is concatenated by 16 columns, 4 rows, and 4 channels to give a vector with 256 elements, which is then stored. If a comparison is made, the 256 permutations of vector o are compared with a stored vector.

[0230] In addition to the observation vector o, the final vector may also store additional data related to the observed location, such as but not limited to user-defined labels, the angular position and identity of the object, data from additional sensory modalities (such as but not limited to auditory or odor information), the traversability of different directions in physical or logical space, and directions that have been previously explored.

[0231] The observation vector o is a coarse representation of the raw observation data captured by the sensor that provides a statistical summary of the environment in the vicinity of the agent in physical or logical space. The coarseness of the vector allows the method to operate more efficiently compared to the raw data. In addition, the observation data is filtered at high resolution to create different channels before the coarsening process. Coarsening is also advantageous because it provides a representation of the location that is less sensitive to changes in lighting or objects in the environment than prior art techniques (because the information is integrated over a larger portion of the field of view).

[0232] It should be understood that the observation vector o can be a vector or a matrix. The vector o may include multiple sub-regions, each sub-region corresponding to a portion of the entire vector. The sub-regions may be content / color sub-regions divided based on the channels used to form the vector (in the example, red, green, luma_h, and luma_v), or may be spatial sub-regions corresponding to different directions around the agent. The sub-regions can be represented by sub-vectors, thereby combining the sub-vectors together to form the entire observation vector o. As described above, when comparing observations, a vector representing a rotational arrangement is needed; when a current observation is generated and needs to be compared with a stored observation, it is not necessary to store all arrangements.

[0233] It should be understood that the numbers given above for dimensions, filter and kernel sizes, row selections, and rotational arrangements may be larger or smaller than the example values.

[0234] Return to Figure 4 In the third step 403, the current observation vector o is compared with the stored observations in the stored observation set. In other words, the current observation is compared with the target observation.

[0235] Several functions may be used to compare observations. Comparison of observations provides an observation comparison metric. The observation comparison metric may vary, depending on the function used to compare the observations. In some examples, more than one observation comparison metric may be generated by using more than one comparison function to compare observations. These comparison functions may be used alone or in combination in method 300 to test hypotheses, and may be used in other related methods described later. Environmental features (such as scene brightness levels) may affect one observation comparison metric but not another. Therefore, using a combination of observation comparison functions and resulting observation comparison metrics allows method 300 to be performed more accurately under a range of environmental conditions.

[0236] The functions used to compare the observation results include an observation similarity function 403-1, an observation rotation function 403-2, an observation motion function 403-3, and an observation variance function 403-4. Figure 4 As illustrated, any one or more of these functions may be performed relative to the current observation and the stored target observation. Thus, the initial conditions and systems required for each method of comparing observations are the same. Figure 5 The system is shown schematically.

[0237] Figure 5 A schematic functional diagram of a system or device 500 is shown, which includes various modules involved in the observation comparison. These modules can be implemented by a computer, a computing device or a computer system. It should be understood that the functions of these modules can be divided into other modules and can be separated across components, or their functions can be performed by a device or component. The first module of this device or system is a memory module 501. The memory module 501 stores the stored observations and displacement sets. These observations and displacements can be stored as vectors. The observations and displacement sets with one or more hypothesis subsets can be stored in a cyclic graph or acyclic graph and / or a directed graph. Alternatively, the observations and displacements can be stored together with metadata identifying the displacements and observations to which they are connected respectively, or can be organized in a lookup table, etc., which retains the link between the corresponding observations and the displacements.

[0238] The memory module 501 is connected to or otherwise capable of communicating with the stored observation processing module 502. The stored observation processing module 502 is configured to select stored observations to be compared from the memory module 501. The stored observation processing module 502 may also select one or more masks for the purpose of reducing the stored observations to a portion of the stored observations for input to the comparison function. The mask may include an observation mask, a user / system selected mask, or a combination thereof. The mask masks certain areas of the selected stored observation vectors while retaining other areas (referred to as regions of interest (ROIs)). The mask may be a binary mask that masks / retains values ​​0 and 1, or vice versa.

[0239] The stored observations processing module 502 is connected to or otherwise capable of communicating with a comparison engine 503 that performs comparisons according to one or more of the comparison functions 403-1, 403-2, 403-3, and 403-4.

[0240] On the second side of the system or device, a sensor module 504 is provided. The sensor module 504 includes sensors that can infer spatial information from its environment, and thus may include a camera, a LIDAR sensor, or a tactile sensor, etc. The sensor module 504 also includes any necessary control circuitry and a memory for recording sensor data. The sensor module 504 is configured to capture current observation data.

[0241] The memory module 504 is connected to or otherwise capable of communicating with the current observation processing module 505. The current observation processing module 505 processes the sensor data from the sensor module 504 to obtain the current observation (such as the current observation vector). This includes the above relative to Fig. 9 and Fig.10 All possible rotations of the sensor data for the described process. The current observation processing module 505 may also select a mask to mask the current observation vector.

[0242] The current observation processing module 505 is connected to or otherwise capable of communicating with the comparison engine 503, which performs comparisons according to one or more of the comparison methods 403-1, 403-2, 403-3, and 403-4.

[0243] For the purpose of performing comparison observations according to any of the comparison methods 403-1, 403-2, 403-3, and 403-4, the comparison engine 503 is configured to obtain or receive stored observations from the stored observation processing module 502, current observations from the current observation processing module 505, and corresponding masks. The comparison engine 503 may be implemented by a computing device, a server, a distributed computing system, or the like.

[0244] The system 500 also includes a movement module (not shown) configured to move the agent within the space. In the physical space, the movement module can be a motor or an engine, etc., which is configured to translate the agent by physically moving the location of the agent and / or by changing the orientation of the agent. In the logical space, the movement module can be implemented by a computer, etc.

[0245] Although the comparison methods are described separately below, it should be understood that they can be performed simultaneously or in the same overall process performed by the comparison engine 503. These methods can be performed in the same overall process because they each rely on the same basic data points and operations. Specifically, each of the following methods compares the current observation with the stored target (predicted) observation. When the observation is represented by a vector, for example, each method implements the process by taking the inner product between the vector or its subvectors. The comparison methods differ in the additional pre-processing / post-processing steps described below.

[0246] In the first method 403-1, observations are compared based on a similarity function to obtain a first observation comparison metric component. The first observation comparison metric component is the "similarity" observation comparison metric discussed above. The similarity function requires vector normalization of the vectors of the observations to be compared. The similarity function includes calculating the vector inner product of the observation vectors to be compared. The inner product includes the cosine of the angle between the two vectors. Therefore, the similarity function returns a single value or score indicating the similarity between the two vectors corresponding to the observations being compared. When the current observation includes multiple possible rotations, such as the case where there are 256 possible rotations in the above example, the vector corresponding to each of these rotations is compared with the predicted observation. In this case, the inner product that provides the highest value is stored as the first observation comparison metric component. This single value can be used in the second step 302 of the method 300, for example, to generate a sorted observation comparison metric set corresponding to the stored observation set, and the stored observation that is most similar to the first observation is determined and selected according to the sorted observation comparison metric set to determine the hypothesis. In other words, the similarity function may be used to determine the similarity between a first observation and a set of stored observations to identify which of the stored observations is most similar to the first observation, and therefore determine which hypothesis to test.

[0247] The similarity function may also be used in a similar manner when comparing the second observation and the predicted observation in the fifth step 305 .

[0248] When the current observation is associated with multiple rotation locations (each rotation location corresponds to an observation vector), the similarity function is configured to find the highest inner product between the different observation vectors and the target observation. To do this, the similarity function keeps track of the "maximum inner product value" as it iteratively calculates the inner product of each of the rotated observation vectors relative to the target observation. Once all rotations have been evaluated, the maximum inner product value is retrieved as the first observation comparison metric component.

[0249] The advantage of using this similarity function is that it is relatively computationally efficient at run time, and the resulting first observation comparison metric component decays very smoothly with the distance from the original observation location (or from the location of the target observation being compared with the current observation).

[0250] However, the function may be affected by environmental conditions, such as the brightness or darkness of the environment. For example, if the stored observations relate to observations captured under low-level light conditions and the current observations relate to observations captured under direct sunlight, the current observations may be different from the stored observations, resulting in different vectors, even if the features captured in each observation are spatially identical. To eliminate these effects, the first observation comparison component may be combined or used with a fourth observation comparison component formed by a fourth observation comparison function 403-4 described below.

[0251] In the second observation comparison function, the observations are compared using an observation rotation function to obtain a second observation comparison metric component. The rotation function uses the rotation of the current observation o, and compares each of these rotations with the target observation, as in the similarity function above. For example, during the fifth step 305 of method 300 (wherein the hypothesis is tested), the observation rotation function can be used to compare the second observation (the current observation) with the predicted observation (the target observation). In the above example, there are 256 rotations, corresponding to 256 different possible column locations from the original cylindrical observation data. The similarity function is configured to retrieve the inner product indicating the best match between the rotated observation vector and the target observation, but the observation rotation function is configured to identify which specific rotation caused the best match. The observation rotation function outputs the rotation direction of the best match as an offset value corresponding to the index of the observed rotation location. For example, the best match between the observation vector o and the predicted observation may be the 120th rotation out of the 256 possible rotations. This corresponds to a vector formed by the observation data when the observation data is rotated so that the 120th column of the 256 columns is located at the center of the data, or at any other predefined rotation location. Therefore, the rotation observation function can output an offset value corresponding to the index of the 120th rotation location, which can be 120, for example.

[0252] In physical space, for example, when an inertial measurement unit (IMU) is used as part of or included in a sensor, the offset value indicates how much the yaw angle has drifted (due to gravity, the roll and pitch angles may be fixed) since the target observation was originally generated (assuming that the stored observations that formed the target observation have been previously accessed in the current space). Therefore, the offset value of the observation rotation function provides an output that indicates the IMU yaw angle drift. The IMU yaw angle drift can be discerned from the output and used to update the stored observations and displacements, because each of these stored observations and displacements is relative to the original yaw angle at which it was generated and stored. Therefore, if there is a confidence level of the first observation comparison metric component and / or the second observation metric component indicating that the current observation is the target observation, the offset value can be used to adjust the stored observations and displacements accordingly.

[0253] If the physical space in which the agent resides is different from the space from which the stored observations were captured, the offset value indicates how the space in which the agent resides is oriented compared to the space in which the stored observations were captured.

[0254] Thus, in summary, the second observation comparison metric component generated by the observation rotation function provides an offset result that indicates the strongest similarity between the target observation and multiple rotations (e.g., 256 rotations) of the current observation. Furthermore, the rotation index indicating the strongest similarity indicates the alignment between the agent's current orientation (i.e., the x-axis and y-axis currently understood by the agent) and the orientation of the target observation when the target observation was generated and stored. Thus, the offset can restore the orientation information so that the target (predicted) observation and displacement can be remapped accordingly.

[0255] This rotation function may be used in the second step 302 of method 300 when comparing the first observation to a set of stored observations, and may be used in the fifth step 305 when comparing the second observation to the predicted observations to recalculate the predicted displacement and observations based on the relative orientation of the agent (reference coordinate system).

[0256] The observation similarity function 403-1 and the observation rotation function 403-2 described above may be executed in the same process or may be executed independently. An example single process in which these functions may be executed together is described below in the form of example pseudo code for running both the observation similarity function 403-1 and the observation rotation function 403-2. The pseudo code is as follows: combined_function(cam_observation[16x4x4x256], cam_mask [16x4x4x256], memory_observation [16x4x4], memory_mask [16x4x4], user_mask [16x4x4])

[0257] • Initialize max_similarity to -1

[0258] • Initialize offset to 0

[0259] • For each of the 256 spins:

[0260] o Initialize cam_sum, mem_sum, match_sum to zero

[0261] o Combine cam_mask, memory_mask and user_mask using a logical binary OR (if 0, the element is used, if 1, the element is not used)

[0262] o For each of the 16x4x4 (256 total) vector elements:

[0263] • If the mask is 0:

[0264] • Add the square of the elements of cam_observation to cam_sum

[0265] • Add the square of the elements of memory_observation to mem_sum

[0266] • Add the product of the elements of cam_observation and memory_observation to match_sum

[0267] Otherwise:

[0268] • Do nothing

[0269] o Renormalize by dividing match_sum by the product of cam_sum and the square root of mem_sum

[0270] oThis is the curr_similarity score for this spin

[0271] o If the similarity score is greater than max_similiarity:

[0272] ■ Set max_similarity to curr_similarity

[0273] ■Set offset to the rotation index

[0274] Otherwise:

[0275] ■Do not perform any operation

[0276] • max_similarity is the output similarity after all rotations are considered, i.e. the first observation comparison metric component

[0277] •After considering all rotations, offset is the output rotation, i.e. the second observed comparison metric component.

[0278] In the pseudo code above, "cam" refers to a sensor (eg, a camera), and thus refers to the current observation. Similarly, "mem" and "memory" refer to memory, and thus refer to the stored target observation to be compared.

[0279] The camera mask, memory mask, and user mask are used to include and / or ignore certain parts of the current observation vector and the target observation vector in the comparison function. The camera mask and the memory mask include information about the reliability of the information from the part of the visual scene in the observation vector. For the memory mask, the information refers to a single observation vector; and for the camera mask, the information refers to a set of observation vectors for different rotations of the field of view. For example, this information can be used to exclude parts of the camera image where the contrast is too high, resulting in saturation of the input data and low data quality. The user mask includes information provided by an internal function of the system (which attempts to select a subset of the full observation vector elements) or an external function to perform the same operation. For example, this information can be used to determine whether the observation results of certain input features match or the observation results of different parts of the visual scene encoded into the observation results match.

[0280] The user mask is configured to remove anomalous / erroneous data from the observed data.

[0281] The above pseudo code iterates for each of the possible rotations of the current observation to identify the maximum normalized similarity measure over all rotations (the first observation comparison measure component) and the rotation offset (the second observation comparison measure component) corresponding to the maximum normalized similarity measure.

[0282] In the above-mentioned first observation comparison function 403-1 and second observation comparison function 403-2 and in the third comparison observation 403-3 described below, an observation vector o can be created based on an image of the physical space. The image has the following characteristics: the x-axis is the angle around the agent, and the y-axis is a certain characteristic perpendicular to the x-axis (which can be a radial distance or a vertical height). In the example provided above, the vector includes 4 channels with sub-pixels (which are red, green, luma_h, and luma_v). There may be more or fewer channels, which make the observation vector larger or smaller. In addition, the image and each of its channels include a series of rows and columns, which provide spatial information of a portion of the space where the agent resides.

[0283] Therefore, the observation vector o may be relatively large, with spatial information in multiple directions and color / edge information from different channels. This relatively large vector can be manipulated or sliced ​​to make a series of different comparisons. For example, if only the edge channels luma_h and luma_v are used, the comparison is less affected by color changes. If only the green channel is used, changes in the distribution of plants, etc., can be determined. If all channels are used (but only for spatial sub-regions of the input image), only corresponding parts of the visual world will be compared. This enables partial comparisons to understand how well certain parts / features of the world match.

[0284] In order to perform slicing of the observation vector o, one or more masks are used. Each mask is a vector having the same number of elements as the observation vector o, which is configured to mask certain regions of the observation vector o while retaining other regions. For example, binary masks 0 and 1 may be used for such purposes. When there are 256 elements in the observation vector o, the mask comprises 256 elements, where a value of 1 indicates that the corresponding index from the observation vector o should be included / ignored in the subset of the observation vector o to be compared, and a value of 0 in the mask indicates that the corresponding element of the vector o should not be included / ignored in the subset of the observation vector. The mask may additionally be used to filter regions of erroneous data as described above.

[0285] In the third method of comparing observations, the observation action function 403-3 is used to obtain the third observation comparison metric component. The observation action function can be used to compare observations and can also be used as part of a movement function to move the agent to a second location while testing the hypothesis according to the fifth step 305 of the method 300.

[0286] The observation action function relies on the same basic principles as the observation similarity function and the observation rotation function. The observation action function differs from the previous two functions in that multiple sub-regions of the observation vector o are evaluated and an offset is determined for each sub-region. Thus, instead of the entire observation vector o, a subset of the observation vectors corresponding to sub-regions of space is evaluated using the observation action function, whereby an offset value relative to the target observation can be obtained for each sub-region corresponding to the subset of vectors. In order to obtain the relevant sub-regions, a mask is used. Each sub-region may overlap with one or more adjacent sub-regions.

[0287] The use of subregions for this purpose stems from the concept that not all parts of the field of view will change the same when the agent moves away from or toward the location of the predicted observation. This is different from classical triangulation because the use considers not the movement of the identified object or signal source, but the distortion of the entire visual scene. Therefore, the use is robust to occluding parts of the visual scene, making it more tolerant to environmental changes. Additionally, the use does not require measurement calculations, which makes it more robust to measurement noise. This feature can not only be used to understand how far the agent is from the location of the target observation, but also allows another vector to be calculated for the purpose of moving to that location. This other vector calculated by the observation action function uses the same principle as the observation similarity function (except that the inner product is obtained for N spatial subregions), and compares these spatial subregions with the corresponding subregions of the target observation. For example, when N=8, the 8 spatial subregions have half of the x-axis range of the complete field of view of the entire observation vector o, and each is centered in a different direction. In this example, the observed action function is therefore actually a combination of the observed similarity function and the observed rotation function, performed 8 times (once for each sub-region) to find 8 similarities and 8 offsets. The 8 similarities and 8 offsets are actually the same as the first and second observed comparison metric components, respectively, except only with respect to a specific sub-region of the observation vector o.

[0288] Thus, in terms of the actual comparison performed, the stored target observation vector is segmented into N sub-regions, and for each of the N sub-regions, the corresponding sub-region of the current observation vector is compared for all possible rotational permutations to identify which permutation is most similar to the stored target sub-region. The particular permutation that is most similar has an index (e.g., the 120th rotation) that is used to identify the offset. Offsets are calculated for each sub-region of the stored observation vectors, and these offsets are used to obtain the motion vector required to reach the location of the target observation, and the similarity is used to evaluate the confidence in the motion vector and weight that confidence when the motion vector is used in the motion function as explained later. Thus, the overall output of the observation action function 403-3 and the third observation comparison metric component is the motion vector.

[0289] Figure 6 A schematic diagram showing observation data corresponding to a sub-region of an observation vector, which is generated for the purpose of executing an observation-action function. Figure 6 Also shown is the process that the subregion goes through in the observed action function (relative to the observed data for visualization purposes).

[0290] like Figure 6 As shown, the observation data is represented by a two-dimensional circle 601 of spatial data around the agent 602. It should be understood that this is a representative example and that the observation data can be three-dimensional, such as in the form of a cylinder. The observation action function processes the observation vector o in comparison with the target observation. According to the observation action function, multiple sub-vectors of the observation vector are processed, where each sub-vector is focused in a different direction or centered in a different direction. Figure 6 In , these directions are illustrated as a plurality of directions 604-1 to 604-n in the corresponding observation data 603. Figure 6 In FIG. 8 , there are 8 directions and 8 sub-regions. However, it should be understood that more or fewer directions may be selected for the process of creating sub-regions.

[0291] Each of the eight directions corresponds to eight sub-regions 605-1 to 605-n (not all shown) of observed data. These sub-regions correspond to portions of the field of view (or sensor field in the case of non-cameras), each centered on one of the eight directions. The particular direction associated with a sub-region is referred to as a sub-region direction.

[0292] In the observation action function, as described in more detail below, each pair of sub-regions 605-1 to 605-n is compared with a corresponding portion of a stored observation result. The comparison is illustrated in the observation data 606, including a comparison of the first sub-region 606a and the second sub-region 606b with a corresponding portion of the target observation result (not shown).

[0293] The observation action function calculates the offset between each sub-region 606a and 606b and the corresponding portion of the target observation. This produces a pair of offsets 607a and 607b, as shown in observation data 607. From these offsets 607a and 607b, a resultant offset vector 609 is determined, as shown in observation data 608. The sub-regions 606a and 606b used in this process are different sub-regions and can be relative sub-regions (although this is not required).

[0294] Reference will now be made to the above described and Figure 6 The illustrated sub-regions of the observation data and corresponding sub-vectors of the observation vector describe the observation-action function 403 - 3 in more detail.

[0295] As described above, the observation-action function uses a set of masked sub-vectors, each of which is a subset of the stored observation vector o and corresponds to a continuous sub-region of the field of view. The sub-regions of the stored observation vectors are compared with corresponding sub-regions of the current observation vector arranged at equal angles around the agent. Following the previous example, the sub-regions of the stored vectors may cover a set of continuous columns in the stored 16×4 observation vectors. An even number of overlapping sub-regions are extracted, where there are N pairs of sub-regions and where the center column of each sub-region is projected into the real world with relative vectors, such as Figure 6 exemplified. Figure 6 403-2. The relative subregions are matched to 256 rotations of the current observation, and the rotation for which the best match occurs is extracted using the observation rotation function 403-2. Specifically, an offset is determined for each relative subregion, the offset corresponding to the rotation index of the best match between the subvector corresponding to the subregion and the rotated stored observation. The differences in the offset directions of the relative subset pairs of observation vectors are compared. This provides an indication of the displacement required to reach the stored observation (in Figure 6 , the displacement is a vector of direction and magnitude of a composite offset vector 609) along a line perpendicular to the center of the two opposing sub-regions 606a and 606b. The direction and magnitude point to the location where the target observation being compared was captured. The weighted average of the composite offset vectors 609 from all opposing sub-regions provides a single action vector that indicates the direction of movement from the location of the current observation to a location closer to the target observation being compared. The vector is rotated from the stored observation rotation to the current observation rotation using a second observation comparison metric component or another rotation metric to align the direction of movement with the current world coordinate system of the agent. It should be understood that an alternative implementation may be performed in which the sub-regions are evaluated in the current observation world coordinate system and therefore such a rotation is not required; however, this process is more computationally intensive due to the need to evaluate the sub-region mask for each current observation rotation in the current observation rotation and has lower accuracy due to the inability to evaluate the exact sub-region in the stored observations.

[0296] Therefore, the observation action function 403-3 can be considered as a process including repeatedly using the observation similarity function 403-1 and the observation rotation function 403-2 to obtain a vector in the direction of the position of the target observation result. Therefore, the observation action function 403-3, the observation rotation function 403-2 and the observation similarity function 403-1 described above can be executed in the same process, by the same comparison engine 503, or independently.

[0297] An example process for executing the observation action function 403-3 using the comparison engine 503 is provided below in the form of example pseudo code:

[0298] Action_function:

[0299] • For the current observation vector:

[0300] o Generate a mask to select a spatial sub-region of observations by:

[0301] ■Given observations: observations [16x4x4] where the first two dimensions are space (X, Y) and the third dimension is the different channels

[0302] ■For each sub-pixel spatial element:

[0303] • Select cyclically contiguous subsets of the columns in the spatial elements; note that cyclically contiguous means that for a 16-column dimension, columns 2, 3, 4 and columns 15, 0, 1, 2 will represent contiguous subset examples, while columns 1, 3, 4 do not represent contiguous subset examples because they are missing the middle column 2. The subsets should have the same width (e.g., 8 columns) and should be evenly spaced across the columns: for example, for 4 subsets of width 8 columns, indices 0, 4, 8, 12 will include evenly spaced examples when referring to the first, last, or otherwise determined column within each subset;

[0304] • Apply the determined subset as user_mask, where the elements in each column in the subset are set to 0 (these elements are used), and the elements of the columns outside the subset are set to 1 (these elements are not used);

[0305] •For each sub-region:

[0306] oExecute combined_function (same as above)

[0307] • Stores the max_similarity and offset of the sub-regions

[0308] • Given a set of max_similarity and offset, where for each sub-region in the sub-region, there is a single max_similarity and offset:

[0309] o for which the subregion corresponds to

[0310] For pairs of offsets that are directions that are opposite in the agent's field of view (e.g., for 8 sub-regions, this would be pairs of offsets that include indices 0 and 4, 1 and 5, 2 and 6, 3 and 7), the cyclic distance between the offsets in that pair (the first index and the second index) is calculated, minus 128. This corresponds to the angle between the offsets, and how the two sub-regions moved in the field of view. The direction perpendicular to the pair of sub-region directions is the action direction. The difference in offsets is used to calculate the action magnitude. The action directions and magnitudes for all pairs are vector-averaged to form the final action function output (the action vector).

[0311] oThe action magnitude is the sum of the action magnitudes of the vectors described by the offsets (where the second offset is increased by 128, thus corresponding to a rotation of π radians).

[0312] Thus, in the observation-action function, N pairs of relative offsets are taken (N=8), and each pair represents a direction described by a direction perpendicular to the center of the mask of the relative offset. This direction is combined with a magnitude calculated from the difference between the two offsets, so that if both offsets are zero (i.e., the center of the subregion of the current observation vector is the same as the center of the subregion of the stored observation), then the magnitude is zero. If the difference between the first and second of these offsets is cyclic, for example, if the first offset is 200 and the second offset is 10 (256 rotations), the difference is -66, but if the first offset is 10 and the second offset is 200, the difference is +190. This difference can then be converted to an angle and used to calculate the exact magnitude, or simply scaled down by a certain factor (e.g., 16) to provide a rough magnitude for the action vector. This difference can then be used in the movement function to reposition the agent to the predicted observation by rotating using the second comparison metric and combining with the action vectors of all pairs as a vector average. Vectors may be excluded from the averaging operation based on a quality metric, such as a minimum similarity score for the sub-regions. It will be appreciated that portions of the visual scene with high ambient variance are allowed to be weighted less than portions of the visual scene with low ambient variance due to the redundancy in the vectors generated by these pairs.

[0313] In a fourth method of comparing observations, an observation variance function 403-4 is used to determine a fourth observation comparison metric component. The observation variance function uses similar properties to the similarity function and the action function. The observation variance function 403-4 includes calculating the circular variance or circular standard deviation of the offsets of the sub-region around the agent. In effect, the observation variance function identifies some measure of the spread of the offsets on a circle around the agent.

[0314] The observation variance function 403-4 uses the same sub-region of the stored vector as the sub-region used for the observation action function 403-3 to determine the offset in the same manner relative to the rotational arrangement of the sub-region of the current observation vector. For example, consider that there are 8 sub-regions and 8 offsets are provided. Because these 8 offsets describe how different parts of the field of view move in different ways when the agent moves toward / away from the location of the target observation result, these offsets should all be the same and equal to the offset of the complete vector match when the agent is at the target observation location. As the agent moves away, these offsets diverge and spread. By using circular statistics, as in the observation variance function 403-4, the spread can be measured, and the spread increases with the distance from the observation location. The circular statistic (which can be the circular variance or circular standard deviation of the eight offsets) provides a scalar value that constitutes the fourth observation comparison metric component. The circular statistic can be any measure of circular dispersion.

[0315] To perform the observed variance function 403-4, the observed action function 403-3 is first performed, but with N offsets. From this point, the cyclic circular variance of the offsets is calculated after first converting from the offset index range (0->255) to the appropriate angle range (e.g., radians (0->2*π)). From this point, any sub-region offsets that deviate from the circular mean angle by more than a given angle (e.g., π / 2 radians) are identified, and as long as more than a given number (e.g., 4) of sub-regions remain, these sub-regions are excluded to recalculate the circular variance.

[0316] The observation variance function 403-4 is another way to determine the similarity between the current observation result and the stored observation result. Unlike the observation similarity function 403-1, the observation action function is extremely robust to changes in the lighting environment because it relies on the offset of the sub-region rather than the entire observation vector. However, the observation action function often has discontinuities because the dimension of the sub-vector used is lower than the complete observation vector o, and false matches may occur at larger distances, which greatly affects the comparison. Therefore, the observation similarity function and the first observation comparison metric component provided by the observation similarity function are better long-distance similarity metrics.

[0317] The circular variance function 403-4 can be used to measure similarity in the same manner as the similarity observation function 403-1, because it provides a scalar output indicating similarity. In addition, the circular variance function 403-4 can be used to determine an environmental adjustment factor, which can then be used to weight or modify other comparison metric components (such as the first observation comparison metric component from the similarity function 403-1). Specifically, if another observation comparison function (such as the observation action function 403-3) has a high confidence that the agent is at the target observation result, the circular variance function 403-4 should provide a high similarity metric in the ideal space because each of the offsets between the stored vector and the current observation vector should be 0. Therefore, the outputs of other functions can be weighted based on the deviations provided by the fourth observation comparison metric component of the circular variance function 403-4. For example, when similarity function 403-1 is combined with circular variance function 403-4, this means that the first observation comparison metric component from similarity function 403-1 may be modified according to the fourth observation comparison metric component from circular variance function 403-4 to reduce and quantify the impact of environmental conditions.

[0318] It should be appreciated that although examples (including numerical examples) of observation comparison functions 403-1, 403-2, 403-3, and 403-4 are provided above, these functions may be performed with any selection of number of sub-regions, masks, channels, rotations, and array size.

[0319] The observation comparison metric may include any one or more of a first observation comparison metric component, a second observation comparison metric component, a third observation comparison metric component, and a fourth observation comparison metric component obtained from the observation similarity function, the observation rotation function, the observation action function, and the observation variance function, respectively. The observation comparison metric is generated each time the observations are compared: such as in the second step 302 of the method 300, when the first observation is compared with the stored set of observations; in the fourth step 305-4 of the hypothesis test, when the predicted observation is compared with the second observation; and when the transition observation is compared with the predicted observation, which will be described in detail later.

[0320] It should be understood that the observation similarity function, observation rotation function, observation action function, and observation variance function described above are examples of functions that can be used to compare the current observation to the target observation. Other functions or adapted versions of the above functions may also be used. Typically, such functions are used to compare observations to determine: how similar the current observation looks to the stored observation, how the current observation is rotated / oriented compared to the stored observation; and how the direction from the current observation to the location of the stored observation is when the agent is not at the location of the stored observation.

[0321] Once the observation comparison metric of the second step 302 of the method 300 is obtained and the hypothesis is determined, the first step of testing the hypothesis is to move the agent to a second position in the space in the first hypothesis testing step 305-1. The movement of the agent is performed according to a movement function, which depends at least on the predicted displacement of the hypothesis subset. Specifically, the movement function depends on the predicted displacement required to reach the position of the predicted observation to be compared. Therefore, the correct predicted displacement must be selected from the hypothesis subset, which may include multiple predicted displacements. In order to ensure that the correct predicted displacement is selected for the movement function, the predicted displacements and predicted observations stored in the hypothesis subset are linked to each other in a network formed by connecting the predicted observations with the predicted displacements, as previously described. The links of the observations to one or more displacements in the network can be stored and retained in any suitable manner. For example, the stored observations and displacement sets (which include the hypothesis subset) can be stored in a cyclic graph or an acyclic graph or a directed graph. Alternatively, the observations and displacements can be stored together with metadata identifying the displacements and observations to which they are respectively connected, or can be organized in a lookup table that retains the links between the corresponding observations and displacements.

[0322] As compared to Figure 3 As discussed, a hypothesis is determined based on the stored observation that is most similar to the first observation produced by the agent. Thus, the predicted displacement selected for the purpose of the shift function is the displacement in the set of stored observations and displacements that is linked to the observation that is most similar to the first observation.

[0323] Once a predicted shift from a subset of hypotheses is identified that is expected to lead from the most similar observation to the next predicted observation of the subset of hypotheses, the move function is updated so that the move is based on the particular predicted shift.

[0324] It should be understood that the predicted displacement has both magnitude and direction.

[0325] The movement function may include one or more movement components. The first movement component of the movement function depends on the predicted displacement. Specifically, the first movement component is a contribution to the movement function that decreases as the agent moves according to the predicted displacement. Therefore, the first movement component is effectively weighted based on the magnitude of the predicted displacement. At the location of the first observation, the magnitude of the predicted displacement is relatively large, so the first movement component is correspondingly weighted more strongly. When the agent begins to move according to the movement function, the magnitude of the predicted displacement is updated based on the actual displacement that the agent has traveled. Therefore, as the agent moves toward the end of the predicted displacement, the magnitude of the predicted displacement decreases, and the first movement component is weighted so that it gradually becomes weaker. In one example, the first movement component is a function of the predicted displacement given by: predicted displacement-travel displacement. The travel displacement is the actual displacement estimated using an odometer, a visual inertial odometer, or the like. In this example, the weight discussed above can be simply a factor provided by subtracting the travel distance. When traveling a greater distance, the first movement component decreases from an initial value equal to the predicted displacement. In other words, as the agent travels along the predicted displacement, this can be viewed as a "disassembly" of the predicted displacement. Thus, the value of the first motion component at any given time is a function of the "remaining displacement" of the predicted displacement. This provides a first value that contributes to the motion function. It will be appreciated that this first value can be scaled according to a scaling factor so that the contribution of the first motion component to the motion function is appropriate taking into account any additional motion components.

[0326] As the agent approaches the end of the predicted displacement, the contribution of the predicted displacement to the movement function approaches zero. In one example, when the measured actual displacement equals the predicted displacement so that the entire predicted displacement has been traveled, the first movement component based on the predicted displacement becomes zero so that it ceases to contribute to the movement function of the agent. When the first movement component is the only movement component of the movement function so that the movement function depends only on the predicted displacement, then once the predicted displacement has been traveled, the location of the agent becomes the second location, i.e., where the second observation is recorded.

[0327] Alternatively, the first movement component may stop contributing to the movement function before the entire predicted displacement has been traveled. Specifically, the movement function may be configured to ignore the first movement component when the amplitude of the first movement component (or in other words, the remaining distance of the predicted displacement) is within a displacement threshold distance from the end point of the predicted displacement. This is advantageous because it reduces the impact of drift on the movement of the agent. Specifically, the use of an odometer to estimate the actual displacement is susceptible to drift-related errors, so that the estimated value of the actual displacement (actual travel distance) may be less than the true value. Therefore, even if the estimated actual displacement is less than the predicted displacement, the agent may have traveled the entire predicted displacement. In order to avoid exceeding the end point of the predicted displacement, the first movement component may be stopped before the end point of the predicted displacement based on the displacement threshold distance. The displacement threshold distance may be set based on the specific environment or scenario in which the agent is used. In physical space, this may be 5cm, 10cm, 1m, 5m, 10m before the end point of the predicted displacement, or any suitable value depending on the scenario.

[0328] The second movement component that may be included in the movement function depends on the observation action function described above with respect to the observation comparison. When using this second movement component on the movement function, it is necessary to continuously, iteratively or sequentially generate transition observations between the first position in the space that produced the first observation and the second position in the space that produced the second observation. In other words, when the agent moves away from the first position, the transition observation is generated.

[0329] Thus, one or more transition observations are generated along the path of movement of the agent. Each transition observation is a "current" observation that is compared with the predicted observation to which the expected predicted displacement will lead, based on the assumption. Based on each comparison, an action vector is generated using the observation action function described above, which defines both the expected amplitude and the expected direction toward the location where the predicted observation can be found. The action vector constitutes the second movement component of the movement function. As with the first movement component of the movement function, the contribution of the second movement component can be scaled by a scaling factor and can be weighted. Specifically, the second movement component can be weighted so that the closer the action vector of the second movement component indicates the location of the predicted observation, the stronger the contribution of the second movement component to the movement function becomes. For example, the amplitude of the vector of the second movement component obtained from the observation action function is configured to increase as the agent moves closer to the predicted observation. This increase in amplitude can directly provide a weight that makes the second movement component stronger as the agent approaches the location of the predicted observation.

[0330] The second moving component can be used alone as the sole contribution to the moving function, or can be used in combination with the first moving component. Advantageously, if both the first moving component and the second moving component are included so that both components contribute to the moving function, the movement of the agent depends on both the predicted displacement and the transition comparison between the predicted observations and the transition observations. This means that there are two separate data sources, which are used to move to the predicted observations in the space and find the predicted observations in the space. This is more reliable and more accurate than using only one data set because it effectively provides redundancy. In addition, the first moving component and the second moving component advantageously complement each other because the relative weight of the first moving component is such that when the agent approaches the end point of the predicted displacement, the first moving component weakens its contribution to the moving function, while the relative weight of the second moving component is such that when the agent approaches the position of the predicted observations, the second moving component enhances its contribution to the moving function. Therefore, the first moving component and the second moving component behave in opposite ways.

[0331] If the hypothesis is correct or at least partially correct or similar, such that the properties associated with the hypothesis are at least similar to the properties of the space occupied by the agent, then the end point of the predicted displacement should be at least very close to the location where the predicted observation was found. Therefore, having a first movement component that depends on the predicted displacement and a second movement component that depends on the predicted observation helps to move the agent towards the location of the predicted observation according to two mechanisms and dimensions.

[0332] The relationship between the first movement component and the second movement component with respect to their contribution to the movement function of the agent may take any suitable form. Figure 7 To illustrate an example of the relationship between these two movement components, the figure shows a schematic map of an agent moving between a first position and a second position in space.

[0333] In a first example of the relationship between the first and second moving components of the moving function, as illustrated in the first diagram 701, the relationship is simple and depends only on the relative weights of the first and second moving components. Since the relative weights of the first and second moving components decrease and increase, respectively, as the agent moves away from the first position 701-1, the first moving component naturally dominates in terms of contribution to the moving function when the agent is closer to the first position 701-1. This is useful because, in general, when the agent is in a location far from the predicted observation, the observation action function responsible for the second moving component only provides a weak indication of where the predicted observation may be in space. This is because the transition observations generated for the purpose of the observation action function will generally be very different from the predicted observations because the distance between the capture points of these observations may be large and these observations therefore belong to different regions of space. The observation action function depends on the comparison of observations and the identification of similarities between observations, and therefore when the observations appear to be very different, the observation action function is less reliable. Therefore, the fact that the first movement component dominates when the agent is closer to the agent's first location 701-1 is advantageous because it can use the predicted displacement to travel closer to where the predicted observation should be. This is more reliable than using the second movement component alone.

[0334] In the first figure 701, a series of transition observations 701-2 are generated as the agent moves to periodically, continuously or iteratively update the second movement component, specifically the observation action function responsible for the second movement component. The agent moves along the path 701-3 according to the movement function to reach the second position 701-6 in space. When the agent approaches the end of the displacement starting from the first movement component, the second movement component from the observation action function becomes dominant and repositions the agent to the second position 701-6. It should be understood that the second position 701-6 is unknown to the agent before the movement, instead, the second position 701-6 represents the same or best match in space with the predicted observation (target observation) linked to the predicted displacement. The second position 701-6 is determined by the position of the second movement component from the observation action function staying at a certain point in space, so that the magnitude and direction of the observation action function vector indicate that the second position 701-6 is the position of the predicted observation or its best match.

[0335] In a second example of the relationship between the first moving component and the second moving component of the moving function, as illustrated in the second figure 702, the relationship is sequential, so that the moving function initially depends only on the first moving component, and at the end of the first moving component, depends only on the second moving component. Since the first moving component depends on the predicted displacement rather than the predicted observation, this relationship effectively does not require transition observations for most of the agent's journey between the first position 1 and the second position 702-6. According to this relationship, the agent is configured to move from the first position 702-1 via a first path 702-3, which corresponds to movement based only on the first moving component based on the predicted displacement. When the predicted displacement is exhausted at the intermediate position 702-5, so that the agent reaches the end of the predicted displacement or reaches a predetermined threshold from the end of the displacement, the moving function switches from using the first moving component to using the second moving component. At this time, the agent generates sequential or iterative transition observations 702-2, and executes the observation action function to obtain a vector of magnitude and direction pointing to the location of the predicted observation. Using this information, the agent arrives at the second position 702-6 along the second path 702-4. Thus, in the second relation, the agent effectively travels through a first displacement before relocalizing using the observation-action function.

[0336] In the third example of the relationship between the first moving component and the second moving component of the moving function, as illustrated in the third figure 703, the relationship is a winner-takes-all relationship, so that the moving function depends only on which of the first moving component and the second moving component has the highest weight at any specific time and place along the space of the path to the second position. As described above, the first moving component that depends on the predicted displacement naturally has a higher weight than the second moving component that is closer to the first position 703-1. Therefore, the agent travels according to the first path 703-3 based on the first moving component and therefore based on the predicted displacement. In the process of this movement along the first path 703-3, the agent generates a series of transition observations 703-2 for the purpose of updating the second moving component. The purpose of doing so is to determine when the second moving component becomes more dominant than the first moving component. When compared with the first relationship listed in the first figure 701, the frequency of the transition observations generated along the first path 703-3 is not equally important, because only the transition observations 703-2 for the first path 703-3 are generated to compare the first moving component with the second moving component, rather than guiding the agent itself. Increasing the frequency simply increases the accuracy of determining that the "winner" in a winner-take-all relationship is the second moving component. At point 703-5 along the path, this determination occurs when the second moving component is determined to be more dominant than the first moving component. At this point, the second moving component is used in the moving function to reposition the agent to the second position 703-6 based on the observed action function. At this stage, the agent follows the second path 703-4 and generates more transition observations 703-2, which are required to obtain vectors from the observed action function to provide the agent with the direction and magnitude of travel to the second position 703-6.

[0337] Therefore, the shift function uses two components to relocate to the location of the predicted observation. The above three relationships between the two components are exemplary, and any suitable method or combination thereof may also be appropriate.

[0338] The ability of the movement function to relocate the agent to the location of the predicted observation depends on the observation action function, which relies on finding similarity between the current observation and the predicted observation. Therefore, if the similarity between the observations is below a certain level, the action function will not relocate. In this case, the agent can move according to the first movement component of the movement function based on the predicted displacement. When it is determined that the agent cannot be relocated due to the lack of similarity between the current observation (the transition observation) and the predicted observation, the hypothesis is rejected because the predicted observation cannot be found at or near the end point of the predicted displacement. In this case, the agent will move backward along the predicted displacement in the opposite way to return to the position in the space of the previous observation that determined the hypothesis. The hypothesis is then adjusted to a different hypothesis associated with a different subset of the stored observations and displacements.

[0339] Thus, the movement function provides the agent with the ability to move to: the location where a predicted observation is found; the location in space that is the "best match" to the predicted observation (for example, if the agent is relocated to a location that does not exactly match the predicted observation but is still similar); or the location where no predicted observation is found. Because the movement function can include two components, there may also be differences between the predicted displacement and the actual travel displacement. For example, if the best match or identical match of the predicted observation to the transition observation is at the end point of the predicted displacement, it may not be necessary to use the second movement component to reposition the agent. Alternatively, it may be necessary to reposition the agent through a smaller further displacement so that the actual travel displacement is similar to the predicted displacement. Finally, the second movement component may reposition the agent to a greater extent than the predicted displacement, in which case the predicted displacement is not similar to the actual displacement.

[0340] These possibilities ultimately provide the method with a way to determine how similar a set of observations and displacements made by the agent are to the properties of a hypothesis in two separate dimensions. This ultimately informs the decision whether to adjust, maintain, or confirm the hypothesis as described above. Specifically, the first dimension is the similarity between the current observation (or the observation in which method 300 is continued or repeated to make more than two observations) and the hypothesized subset of stored observations. This similarity is determined by an observation comparison metric and is referred to as "style". The second dimension is the similarity between the current displacement (or the displacement in which method 300 is continued or repeated to make more than two displacements) and the hypothesized subset of stored displacements. This similarity is determined by a displacement comparison metric and is referred to as "composition". The displacement comparison metric is determined by comparing the actual displacement with the predicted displacement.

[0341] Generally speaking, there are four combinations of style and composition. These combinations are high style and high composition, high style and low composition, low style and high composition, and low style and low composition. Each of these possibilities will be explained in more detail below with respect to their significance to the hypothesis.

[0342] First, high style and high composition indicate that the observations produced by the agent are very similar / identical to the predicted observations of the hypothesis subset, and the actual displacements performed by the agent are very similar / identical to the predicted displacements of the hypothesis subset. "Very similar" means that both the observation comparison metric and the displacement comparison metric are higher than the second higher observation threshold level and the second higher displacement threshold level, as described above in the fifth step 305-5 of hypothesis testing. When the displacement comparison metric and the observation comparison metric match or exceed these two levels, it is determined that the space in which the agent resides includes attributes that match the identity as defined by the hypothesis subset. As previously described, it is not necessary to mark the identity, but it can be associated with a label that defines the attributes related to the identity. If the attributes of the space match multiple consecutive predicted observations and hypothesis subsets of displacements, the confidence that the attributes in the space match the identity will increase. Therefore, determining that the attributes of the space match the identity may require comparing multiple consecutive predicted observations and displacements and matching them to a sufficient level.

[0343] Secondly, high style and low composition indicate that the observations produced by the agent are very similar / identical to the predicted observations of the hypothesis subset, but the actual displacements made by the agent are not similar / identical to the predicted displacements of the hypothesis subset. "Very similar" means that the observation comparison metric is above the second higher observation threshold level, but the displacement comparison metric is not above the second higher displacement threshold in the fifth step 305-5 relative to the hypothesis testing. In terms of the space in which the agent resides, this means that in terms of the sensor (observation) data, the space "seems" to be the same or very similar to the attributes defined by the hypothesis subset, but at least one of the size, position and / or distance between points of the attribute is not the same as predicted according to the hypothesis. This relationship is called "background". In Figure 1 and Figure 2 In the above example, the Figure 1 The background is represented by a room with similar furniture in the room (but it may be larger or the furniture may have been rearranged (for example, the central feature may have been moved to one side)). Figure 2 In , the background can be found if the chair is imaged from a back viewpoint or if the proportions of the chair are modified (e.g., shorter legs, longer backrest).

[0344] Third, low style and high composition indicate that the observations produced by the agent are not similar / identical to the predicted observations of the hypothesis subset, but the actual displacements made by the agent are very similar / identical to the predicted displacements of the hypothesis subset. "Very similar" means that the displacement comparison metric is above the second higher displacement threshold level, but the observation comparison metric is not above the second higher observation threshold in the fifth step 305-5 of the hypothesis test. In terms of the space where the agent resides, this means that the space appears to have very similar size, position and / or distance between points according to the hypothesis, but does not seem to be / does not look like the same properties defined by the hypothesis. This relationship is called a "class". In a Figure 1 and Figure 2 In the example shown above, the classes can be represented by rooms of the same size and structure (but perhaps with different colors and different furnishings). Figure 2 , classes can be found if the chairs are different colors, or if the details are different in style while maintaining the same proportions (such as certain types of chairs are designed for dining tables). For navigation, this can be different floors of a building where the floor plan is the same but the decorations may have been provided by different companies. In order to determine the class relationship between the attributes in the space where the agent resides and the stored observation data, multiple observation comparison metrics can be used. Although there may be a lower similarity indicated by the first observation comparison metric component, a second observation comparison metric component and / or a third observation comparison metric component may be used depending on the observation similarity function 403-1. If the observation-action function does not converge to a certain point, then there is no match and the basis for the comparison is invalid, in which case the hypothesis is rejected.

[0345] Finally, low style and low composition indicate that the observations produced by the agent are not similar / identical to the predicted observations for the hypothesis subset, and the actual displacements performed by the agent are not similar / identical to the predicted displacements for the hypothesis subset. In this case, relative to the fifth step 305-5 of hypothesis testing, the displacement comparison metric is not above the second higher displacement threshold level, and the observation comparison metric is not above the second higher observation threshold. In terms of the space in which the agent resides, this means that the space does not appear to be related to the structure or appearance of the attributes to which the hypothesis is related. The hypothesis can be maintained for further comparison or rejected and adjusted, as previously explained.

[0346] By comparing current observations and displacements with stored observations and displacements, for one or more subsets of hypotheses, features of the space can be identified along the dimensions of composition and style (structure and appearance). These dimensions can also be used to determine changes to a space when the agent evaluates the space repeatedly or periodically. In this scenario, the stored subset of hypotheses can include observations and displacements that the agent obtained directly from the same space in previous operations. Using this information, the agent can detect how the space has changed over time in terms of structure and appearance. For example, identifying differences can be helpful in path planning applications and finding alternative routes. The agent can store an average displacement to a particular stored observation, which is updated each time the agent makes a displacement to that observation. The average displacement provides a historical average of previous displacements made to a particular observation. In addition, the standard deviation of previous displacements made by the agent to a particular observation can also be stored. Using the average displacement and standard deviation, a statistical analysis can be performed to determine the likelihood that the space has changed. For example, if a new displacement to the observation is significantly different from the average displacement and / or different from the standard deviation of previous displacements, it is determined that there is a high likelihood that the space has changed. This can occur, for example, if an attribute of the space has been moved. A history of each observation and displacement can be stored, including the mean displacement, observation, and standard deviation for statistical analysis. This history can be updated with each exploration by the agent.

[0347] A second history record may be stored and updated in a similar manner. The second history record relates to the N most recent observations and displacements made by the agent as it traveled to a particular observation, specifically the similarity score or total similarity score between these N most recent observations and displacements and the corresponding stored observations and displacements with which they have been compared. Thus, the second history record represents a positioning confidence that indicates the system's confidence that the agent has located the stored observations and displacements on its way to the particular observation. If the second history record indicates that the N most recent observations made by the agent closely match the N most recent stored observations, then a confidence is given that the agent is well located to the subset or path it is following in terms of observations and displacements. If the displacement made to a particular observation or the particular observation itself is significantly different from the corresponding stored observation and displacement, then the second history record may be used to display and confidently determine that the space has changed in some way for that particular observation and displacement. For example, if N=5, then the five most recent displacements made by the agent may completely match the stored displacements. If the sixth displacement made 个 The displacement is different from the stored 6displacements, the second history indicates that the first five displacements are highly matched (in this case identical), meaning that the spatial relative displacement to the sixth displacement can be easily determined. 个 The displacement has changed. This could be due to moving obstacles, etc.

[0348] Furthermore, although as described above, a hypothesis may be confirmed, maintained, or rejected based on observations or displacements, respectively, it is advantageous to use both observations and displacements, and therefore composition and style, when determining the outcome of a hypothesis. By combining the observation comparison metric and the displacement comparison metric to form a two-dimensional similarity metric, adjusting the hypothesis may include: if the two-dimensional similarity metric is below a first two-dimensional similarity metric threshold level, then adjusting the hypothesis. Maintaining the hypothesis may include: if the two-dimensional similarity metric exceeds the first two-dimensional similarity metric threshold level, but does not exceed a higher second two-dimensional similarity metric threshold level, then maintaining the hypothesis. If the second two-dimensional similarity metric threshold level (also referred to as the hypothesis confirmation two-dimensional similarity metric threshold level) is exceeded, then the hypothesis may be confirmed. In the case where the visual identity of the observations encountered at different times changes, if it is believed that the current and past observations are captured with the same identity, the agent can use the current observation information to update the stored observations. Since both observations are represented by normalized vectors, this may include averaging the two vectors, wherein an optional weight is applied to each vector so that the final observation will be more or less aligned with the past or current observation vector. For example, if the past observations are multiplied by 5 and the current observations are multiplied by 1, and the summed vector is then renormalized, the final observation vector will be more consistent with the past vectors.

[0349] Adjusting the hypothesis includes: rejecting the hypothesis and returning to the previous observation to determine the second hypothesis. Specifically, if in any observation, the match of the current observation and the predicted observation does not reach a suitable confidence level, and / or the match of the actual displacement and the predicted displacement does not reach a suitable confidence level, the hypothesis is rejected. The agent is then configured to return to the previous observation by generating a displacement opposite to the displacement performed to reach the current observation. This may include returning to the initial observation, although this is not necessary. The method 300 then repeats testing different hypotheses, which are selected based on the comparison between each hypothesis subset in several hypothesis subsets of the previous observation and the predicted observation. Optionally, the agent can store observations and displacements, even if such observations and displacements do not match the hypothesis, and then the hypothesis is rejected. Storing these observations and displacements can help generate a map of the space for future navigation and attribute identification purposes. Therefore, before returning to the previous observation, even if the hypothesis is rejected, the agent can also generate and store one or more observations and displacements.

[0350] Confirming observations includes, but is not limited to, identifying or finding an object, image, or destination in a space. Once an attribute has been found, a reference to it (including the observations and displacements made by the agent in the space) is stored in memory. These may form part of a map of the space in which the agent resides. If the hypothesis is associated with a tagged identity or attribute, the agent may also store a reference to the tag in memory along with the observations and displacements it has made in the space. In the case where the hypothesis is based on a previous visit to the same space (because the hypothesis subset effectively defines the attributes previously observed in the space), the hypothesis subset may be updated with the set of displacements and observations recently made by the agent when testing the hypothesis. In this way, the hypothesis subset may be kept up to date with respect to a particular space.

[0351] It should be understood that the above method can be performed by one agent or one or more agents working in collaboration. When one or more agents are included in the space, the one or more agents can communicate with each other and / or can access the same memory. For example, multiple agents can each be linked to the same server or computer system. Multiple agents can simultaneously update the map of the space, or test one or more hypotheses at the same time. The generation of the map will be discussed in more detail below.

[0352] In the above description, a method 300 for parsing a space and identifying an attribute of the space is described. The attribute may be an object or an image, such that the method 300 is used for object or image recognition purposes, or may be a destination, such that the method 300 is used for navigation purposes. The method 300 relies on the agent making observations and displacements in the space and comparing them to stored observations and displacements. Thus, one principle of the method 300 is the ability of the agent to compare observations and displacements. This is useful in Figures 4 to 6 and the accompanying description above. Another principle is the ability of the agent to move, which is Figure 7 and the accompanying description above.

[0353] Additional aspects complementary to method 300 will now be described, which aspects may be incorporated into method 300 or may alternatively be performed or implemented separately.

[0354] The first additional aspect is to generate and store the linked observations and displacements that form the identity and hypothesis subsets. This can be considered the "training process" briefly mentioned previously. The process of forming the identity can be performed specifically for the space in which the agent is expected to reside, or it can be performed in a separate space. Both of these specific implementations have benefits. If the identity is formed in the same space that the agent is subsequently configured to investigate, it is possible to identify how the properties of the space change over time after further exploration of the space. If the identity is formed in a separate space, it is possible to identify the recurrence of properties of one space in another space, and to identify similarities and differences between spaces.

[0355] Reference here Figure 8 The process of forming and storing identities is described. Figure 8 A schematic diagram 800 of identity identification is shown. Consider that no observations or displacements are stored. The agent is "dropped" or otherwise activated in a previously unknown and unexplored space. An initial observation o1 is generated in the space. The initial observation o1 can be generated based on low-level features detected in the space, although this is not mandatory. The use of low-level features will be discussed separately later as a further supplementary aspect. The selection of the initial observations can be further randomly selected or selected based on the learned process. Starting with the initial observation o1, an initial displacement is made from the position of o1 to produce a new observation o2, and the displacement between o1 and o2 (denoted as d12) and the inverse displacement from o2 to o1 (the negative of d12, denoted as d21) are stored to the memory.

[0356] Optionally, an additional effort value is stored which is a better measure of the characteristics to be optimized in the route, such as but not limited to energy usage and / or vibrations encountered.

[0357] The agent then travels through a continuous set of observations Oi{oi,1...oi,n} and a corresponding set of displacements Di{di,1..,di,n-1} and stores them in the same manner. This forms an identity. The set of observations and displacements that form the identity describes a predicted path through space, as well as the expected surroundings at each point on the path. Traveling through the path allows the agent to test predictions (i.e., hypotheses) by comparing the measured displacements and observations with the predictions one by one. The continuous set of observations and displacements that form the identity is stored as described above. In some examples, the identity may be stored as a subset of a larger stored set, wherein the larger stored set includes multiple subsets, and wherein each subset defines a path within the same or different space. Therefore, it is possible for an identity to be linked to other identities, and it is possible for identities to share one or more common observations and / or displacements.

[0358] like Figure 8 As shown, a series of observations o1 to o4 and corresponding displacements can be captured in a closed loop to form an identity signature, and then for a continuous subset or the entire set, the style and composition dimensions as described above can be used to sequentially check against further future observations and displacements. The identity signature can be used for purposes including but not limited to: localization or recognition of specific objects; extraction of generally repetitive classes, such as objects or navigation terrain, such as but not limited to building corridors and rooms; and extraction of contexts, which are areas or objects with similar parts or characteristics (e.g., forests / offices / streets or wheeled vehicles / flying objects).

[0359] During or after the identity is stored as a subset of observations and displacements, further optional processes may include using a separate system to determine the attributes to which the identity relates. For example, the identity may be associated with a path for the purpose of navigating to a destination in space. Alternatively, the identity may be associated with an object (such as a chair) or an image (such as a cartoon chair or a drawing of a chair). In order to correctly associate the identity with these attributes, a training process may be used. The training process may involve identifying the attributes via an established training algorithm, for example including training a classifier using training data. It will be appreciated that any machine learning process may be used, and may include the use of neural networks and the like.

[0360] A second additional aspect is to form a map of the space in which the agent resides. While the agent may be controlled to test hypotheses to identify properties of the space in which it resides according to the methods set out above, the agent may additionally / alternatively be controlled to generate a map of the space in which it resides. It will be appreciated that the generation of the map may thus be performed independently or in conjunction with identity generation and property determination / hypothesis testing.

[0361] If the space is completely unknown to the agent's previous exploration, the agent can generate a new map, or if the agent has previously visited the space, the agent can update an existing map in memory.

[0362] To generate a map, the agent generates a first observation in space. This can be the same observation as the first observation generated in step 301 of method 300 for testing the above hypothesis. The first observation is stored in memory, indicating a corresponding "first position" in the map.

[0363] The agent is then configured to move away from the first location to generate one or more further observations.

[0364] When performing hypothesis testing in conjunction with the agent to make or update a map, the agent moves according to the movement function as described above. In this case, the agent is configured to generate one or more new observations as it moves away from a first position. New observations may be generated while the agent is moving according to the movement function (transition), or may be generated after the movement. In this regard, the agent may wait until it reaches the second position of the second observation of the above-described method 300 before generating a "new" observation, in which case the second observation is a new observation.

[0365] The new observation is compared to the first observation using one or more observation comparison functions as described above. This gives one or more observation comparison metrics. The observation comparison metric is compared to a mapping threshold that defines the maximum likelihood that consecutive observations must be below in order to be recorded on the map. In other words, the method of generating a map requires that the observations stored on the map are not too similar. This avoids unnecessary points being added to the map, which keeps the map concise and keeps the memory required to store and access the map at a minimum. When the observation comparison between the first observation and the new observation results in an observation comparison metric below the mapping threshold, the new observation is stored in memory at a "new location" on the map. The displacement between the first observation and the new observation is measured and / or retrieved from the movement function and stored in the map together with the inverse displacement from the new observation to the first observation. When the map is generated while hypothesis testing is also in progress, this means that the agent performs two comparisons, the first one is a comparison between the current observation and the predicted observation (to test the hypothesis), and the second one is a comparison between the current observation and the previous observation (such as the first observation) to determine whether these observations are sufficiently dissimilar so that both are included on the map. Therefore, even if the hypothesis is rejected and the current observation does not closely match the predicted observation, the agent can still effectively generate observations or use the current observation to add to or update the map by comparing it to previous observations.

[0366] If the mapping threshold is exceeded, new observations are not stored on the map, and instead the agent is configured to move further displacements again before generating further observations until the mapping threshold is not exceeded.

[0367] Once a new observation is stored on the map along with the displacement between the first and the new observation, the new observation effectively becomes the first observation (or previous observation), and the process repeats for further displacements and further observations. This process iteratively builds a network of connected observations that form a map that describes the space the agent resides in.

[0368] A map of a space can also be generated separately from identifying the properties of the space. The method for generating a map is roughly the same. The agent generates a first observation and stores it in a memory as the first position in the map. Then, the agent moves away from the first position of the first observation in an unexplored and physically or logically navigable direction. The movement of the agent is recorded as a first displacement. As described above, the agent then generates one or more new observations away from the first position, and compares the first observation with the new observation to determine whether the mapping threshold is exceeded. If the mapping threshold is not exceeded, the new observation will be stored on the map, where the first displacement and the inverse first displacement link the first observation with the new observation. The relative direction between the first observation and the new observation is marked as "explored". Then, the process is repeated for further displacements and observations from the position of the new observation in an unexplored direction. The use of a mapping threshold means that the number and density of observations generated in a particular space depends on the complexity and characteristics of the space. For a feature-rich environment, more observations will be generated because the similarity between observations will drop below the mapping threshold faster and over a shorter distance. In feature-poor environments where the space is less complex, fewer observations are needed. Therefore, methods for mapping and / or identifying attributes are performed in such a way that it automatically adapts to the constraints of the space being observed or traveled by the agent. Therefore, these methods are sensitive to the complexity of the environment.

[0369] Multiple agents may contribute to the same map. Specifically, multiple agents may each contribute to and access a shared database or memory, and may be configured to update the shared database using observations and sets of displacements to generate a map. Where a hypothesis is tested and an attribute is identified, an identity associated with the attribute may be stored on the map and, where possible, labeled as such. Thus, the map may include multiple identified attributes and stored labels for these attributes. Agents may contribute to the generation of a map in real time or near real time, and may each be communicatively coupled to the shared database by any suitable means. For example, in a physical space, multiple agents may be a fleet of drones, robots, or vehicles, etc., each of which is capable of communicating with a computer system or a distributed computer system (such as, multiple servers).

[0370] Thereafter, the map can be used for navigation in space. Specifically, a destination on the map can be selected for the mobile agent. The destination is represented by an observation that forms part of the map.

[0371] Map navigation works similarly to method 300, testing hypotheses and determining properties of a space. When navigating the map, the property is actually the destination. The process of navigation is described in more detail here.

[0372] The map is generated and stored in memory as described above and forms a stored set of observations and displacements, wherein at least a subset of the stored observations and displacements are connected to or networked with observations associated with the destination. Thus, the map is similar to the hypothetical subset as described above.

[0373] The agent is configured to generate a first observation in space to identify which of the stored observations of the stored map the agent is closest to. Specifically, the first observation is compared to each of the observations in the stored map associated with the destination. Through these comparisons, the closest matching observation to the first observation is identified. The agent is then moved so as to relocate to the location of the closest matching observation.

[0374] A route is planned using a routing algorithm based on the location on the map that most closely matches the observation. Any suitable routing algorithm may be used, such as Dijkstra's algorithm, which determines the route that minimizes a metric, which may be distance, effort, vibration, or some other metric that can be retrieved from the displacement stored in the map.

[0375] Once a route is planned, the agent moves along the route in sequence, where the process is repeated as the agent moves from one observation to another on the map. The agent starts the process with the "closest matching observation", and the route may, for example, include multiple observations between the closest matching observation and the destination. The observation that the agent is currently at is the current observation, and the next observation on the map in the route to the destination is called the target observation.

[0376] The repeated process includes moving the agent in the direction of the target observation using the stored displacement vector between the current observation and the target observation. This is similar to moving the agent according to the movement function based on the predicted displacement, as described above with reference to method 300 for hypothesis testing. As with hypothesis testing, the process also includes using one or more observation comparisons using one or more of an observation similarity function, an observation rotation function, and an observation action function. For example, the stored displacement to the target observation can be used in conjunction with the observation action function to move the agent from the current observation to the target observation. Thus, the agent can generate multiple transition observations while traveling between the current observation and the target observation to relocate to the location of the target observation using the observation action function. This is similar to the movement function including a first movement component and a second movement component as described above, and allows modification of the direction of travel and consideration of mileage drift and uncertainty when reproducing the displacement between the current observation and the target observation.

[0377] Once the observation comparison metric (or metrics) is above a predetermined threshold or continuously calculated thresholds, it is determined that the agent has reached the target observation. This is similar to determining a match between the second observation and the predicted observation in hypothesis testing method 300. Once it is determined that the agent has reached the target observation, the process is repeated for the next observation in the route to the destination until the destination is reached. Thus, the target observation becomes the current observation, the next observation becomes the target observation, and the displacement to be traveled is the displacement between the current observation and the target observation. Once the agent reaches the observation associated with the destination, the agent is configured to wait at that location unless otherwise instructed.

[0378] Thus, observations, displacements, and movement functions may be used to identify properties and features of a space, generate a map of the space, and navigate the space. It should be understood that one or more or all of these processes may occur simultaneously. For example, an agent may be instructed to follow an existing route to a destination, identify properties or features on the way to the destination, and use transition observations to add to the existing map. Thus, the agent is able to simultaneously learn and navigate its environment.

[0379] In the above-described method and system for parsing a logical or physical space (including identifying properties / features of the space, generating a map of the space, and navigating a map of the space), an agent is configured to generate observations and compare observations and move between these observations. As previously described, the agent does not need to know the logical or physical space, nor does it need to have previously visited the space. Therefore, there are several sets of initial conditions that the agent may encounter when placed in the space.

[0380] In a first example, an agent may be placed in a previously visited space and wherein the agent has access to a stored map or a stored subset including observations and displacements made in the space. In this example, when a first observation is made, the first observation may be the same as or otherwise match a stored observation (such as a predicted observation of a hypothesis subset and / or an observation in a path to a destination). In this case, there is no need to reposition the agent prior to executing method 300 to test a hypothesis or navigate a route to a stored destination.

[0381] In a second example, an agent may be placed in a previously visited space, and wherein the agent has access to a stored map or a stored subset (including observations and displacements generated in the space). In this example, when a first observation is generated, the first observation may be different from any stored observation in the stored observations of the map or the stored subset. Since the position of the agent is therefore unknown, it is not possible to reposition the agent to the stored observations using the stored displacements. Instead, the agent must first reposition to the stored observations without using predicted displacements in order to understand its location on the map or relative to the stored observations and displacement subsets. To this end, the first observation is compared with each of the stored observations, and an observation-action function is used to determine a vector pointing to one of the stored observations (referred to herein as the target observation). The agent then moves along the vector, iteratively or continuously updating it with further transition observations to reposition to the target observation.

[0382] The selection of which of the stored observations becomes the target observation can be determined in any reasonable manner. In one example, for all rotations, each of the stored observations can be compared to the first rotation using an observation similarity function. The most similar stored observation can then be selected for comparison with the first observation to determine the action vector according to the observation action function.

[0383] Alternatively, an action vector may be determined for each stored observation, and the strongest of these is then selected for use in relocating the agent to the stored observation. The action vector is determined according to the observation-action function as described above, and is weighted based on the similarity between a sub-region of the stored observation vector and the corresponding dubbing region of the current observation vector, or based on an overall similarity measure, and is therefore stronger.

[0384] In a third example, an agent may be placed in a space that has not been previously visited, but in which the agent has access to a stored subset (including observations and displacements from different spaces or spaces). In this example, when a first observation is made, the first observation may be different from any of the stored observations of the stored subset. Since the position of the agent is therefore unknown, it is not possible to reposition the agent to the stored observations using the stored displacements. Instead, the agent must first reposition to the stored observations without using predicted displacements in order to understand its location in the space relative to the stored observations and displacement subsets. To this end, the first observation is compared to each of the stored observations, and an observation-action function is used to determine a vector pointing to one of the stored observations (referred to herein as the target observation). The agent then moves along the vector, iteratively or continuously updating it with further transition observations to reposition to the target observation.

[0385] The selection of which of the stored observations becomes the target observation can be determined in any reasonable manner. In one example, for all rotations, each of the stored observations can be compared to the first rotation using an observation similarity function. The most similar stored observation can then be selected for comparison with the first observation to determine the action vector according to the observation action function.

[0386] Alternatively, an action vector may be determined for each stored observation, and the strongest of these action vectors selected for use in relocating the agent to the stored observation.

[0387] However, unlike the first and second examples described above, in this third example, the stored observations and displacement subsets are from a space different from the space in which the agent currently resides. Therefore, there may not be any stored observations in the space in which the agent resides. In this example, the agent may not be able to relocate to the stored observations. If the agent is testing a hypothesis such that the stored observations and displacement subsets are a hypothesis subset, the agent may reject the hypothesis associated with the hypothesis subset if the agent cannot locate the stored observations of the hypothesis subset. The agent can then return to the position in the space of the first observation by a negative displacement. From here, the agent can select different hypotheses and corresponding hypothesis subsets to test. The agent can repeat this process for all stored hypotheses and hypothesis subsets as needed. If all of the hypotheses are rejected, meaning that the space does not include any identifiable features or attributes corresponding to the stored observations and displacement subsets, the agent can enter an exploration mode, through which the agent generates observations and displacements to generate a map of the space, as described above.

[0388] In a fourth example, an agent may be placed in a space that has not been previously visited and in which the agent does not have access to a stored subset (including observations and displacements). In this example, the agent enters the above-described exploration mode to map out the unknown space in which it resides and ultimately stores a set of observations and displacements to describe the space.

[0389] The above description focuses on the generation of continuous observations and displacements by an agent to perform one or more of the following: generate a map, navigate a route through a space, or identify features of a space. While it is possible to perform these methods using only observations and displacements, additional factors can be incorporated to provide further efficiency and accuracy advantages.

[0390] Specifically, low-level features of a space can be identified or detected and used to inform the process of generating observations within the space and moving between observations. Specifically, the observations generated by the agent include spatial information of the environment of the space indicating the vicinity of the agent. Therefore, it is possible to extract features from the observations or observation data to obtain more information about the vicinity of the agent. For example, environmental clues such as high-contrast lines, colors, edges, and shapes can be extracted to determine further information about the environment, which can be used to inform decisions and behaviors. For example, in map generation, where the agent is configured to explore the space by generating observations and displacements in sequence, the choice of the direction to move the agent after each observation can be driven by low-level features. Specifically, after each observation is generated, the choice of the next direction of movement of the agent can be weighted based on the low-level features and / or the direction of previous navigation. For example, a first weighting can be applied to a direction that would move the agent closer to a previous observation if traversed by the agent. Since the goal of map generation is for the agent to explore and map the space, it is beneficial to encourage the agent to move away from previous observations, and the first weighting achieves this. The second weighting can be applied to the direction based on the low-level features. For example, if an edge or high-contrast line appears to point in a specific direction, the direction can be given a weight higher than the direction in which the low-level features are not present. In the physical space, the agent can be a vehicle. The extracted low-level features can include roads (from edge detection or the like). The direction in which the road appears and / or leads to according to the current observations can be given a weight higher than the direction in which the current observations do not have roads. Therefore, the second weighting based on the low-level features encourages the agent to follow or get closer to the potential features of the environment. For example, by following the road, the space is more likely to be explored, and the agent is more likely to produce observations that are easy to traverse and connect. Although roads are artifacts added to the environment by humans, it is understood that there are features in the natural environment that can provide similar benefits, such as the banks of waterways, the edges of woodlands with grass, and the traces left by foraging animals in the vegetation. By following such features as part of both the exploration and navigation phases of movement, the agent can ensure that there are additional guides that ensure reliable traversal, even in situations where the environment varies greatly (such as thick fog). It should also be noted that these features can be helpful when the agent is trying to identify the identity of an object, such as a chair. Following the line edges formed by the legs, seat, and back of the chair guides the movement of the observation points to ensure high robustness in reproducing the identity, even in the presence of 3D rotations.

[0391] It should be understood that the first directional weighting and the second directional weighting discussed herein may be used alone (without the other weighting) or in combination.

[0392] Low-level features identified or detected in observations or observation data can also be linked to a set of behaviors to be performed by the agent. These behaviors allow the agent to approach features in a similar manner when encountering a new part of a space or object for the first time as when encountering the same part of the space or object again. For example, when a high-contrast line is identified or detected, the agent can be configured to travel along the high-contrast line. The agent can be configured to reach its vanishing point along a consistent line in the environment, so that the agent can move along a road, sidewalk, or trail. These types of structured actions simplify the problem of robust navigation. The agent can move in another predetermined direction relative to the low-level feature, such as perpendicular, parallel, clockwise, or counterclockwise relative to the low-level feature.

[0393] Observations can also be generated based on the presence of low-level features in the environment. Unique low-level visual statistics (which can include but are not limited to spatial frequency distribution, brightness distribution, color distribution) can be used to determine when and where the agent should generate initial observations.

[0394] The use of low-level features is not limited to map generation. Specifically, low-level features can be used in any of the above methods. For example, in hypothesis testing, according to the above method 300, low-level features or clues can be used in the movement function when determining how to move the agent from the current observation to the predicted observation. The movement function depends on the predicted displacement required to move the agent to the predicted displacement, but may also depend on low-level features, such as high-contrast lines or edges. In physical space, the agent can be configured to follow low-level features such as roads, even if the road is not directly aligned with the direction of the predicted displacement. For example, a road can be identified by presenting a single color extending upward from the bottom of the field of view where the agent is located toward the top of the field of view, where the top of the field of view mainly contains more distant locations. Alternatively, a road can be identified as an area extending as described above, but including strong vertical edges in the absence of strong horizontal edges. For example, if a road is within a threshold angle of the predicted displacement direction, the agent can be configured to follow the road instead of following the predicted displacement. This can improve the system if the predicted displacement is inaccurate for some reason. The agent can then periodically generate transition observations on the road to ensure that the agent's movement is acceptably toward the predicted observation. If the predicted observation is not reached, or if the predicted observation appears to be moving further relative to the agent, for example according to the observation-action function, the agent may leave the road. The agent can also move to the conclusion of a low-level feature (e.g., the end of the road or the next intersection of the road). If the agent does not find the predicted observation at this time, the agent can still generate observations and store them to generate a map or otherwise exploit new knowledge of the space to further populate the map, then reject the hypothesis and return to the previous observation.

[0395] Following low-level features or otherwise using them advantageously simplifies the process of testing hypotheses and navigating space, since it is easier to backtrack from a point in space to previous observations and only requires following the low-level features in reverse. Additionally, robustness to visually significant environmental changes (such as thick fog) is improved, since the agent can follow the feature even without more distant descriptive information.

[0396] The above description introduces various concepts. One such concept is the use of an action function to compare stored observations to current or new observations to effectively determine the similarity between the stored observations and the new observations and provide a vector from the new observation to the stored observations that the agent can use when determining how to move to reach the location corresponding to the stored observations. Reference will now be made to Figure 11 to Figure 1 5 and as described above Figure 6 and Fig. 9 Describe the process in more detail.

[0397] As discussed above, the observation action function compares a sub-region of the current observation to a sub-region of a stored observation. The sub-region of each of the current observation and the stored observation is a portion of the entire current observation and the entire stored observation.

[0398] The use of subregions for this purpose stems from the concept that not all parts of the field of view change the same when the agent moves away from or towards the location where the stored observations were captured. This is different from classical triangulation because the use does not consider the movement of identified objects or signal sources, but rather the distortion of the entire visual scene. Therefore, the use is robust to occluding parts of the visual scene, making it more tolerant to environmental changes. Additionally, it does not require metric calculations to run, which makes it more robust to measurement noise. This property can be used to understand how far the agent is from the location corresponding to the stored observations, while also allowing the calculation of an additional vector to move to that location. The additional vector calculated by the observation action function uses the same principle as the observation similarity function, except that the inner product is obtained for the N spatial subregions of the current observation and the N corresponding subregions of the stored observations.

[0399] To obtain the sub-regions, execute the above Figure 6 , Fig. 9 and Fig.10 The process described here. Fig.11 The process is explained in detail.

[0400] Fig.11A schematic diagram showing the results of multiple processes performed on the original observation data to obtain an observation vector or matrix o, from which multiple sub-regions can be obtained. Fig.11 In the first step, raw observation data is captured to capture raw image 1101. Raw image 1101 may be a 360-degree view of the environment of an agent captured from a first position, wherein the first position is the position of the agent. In this case, raw image 1101 forms first observation data or current observation data (also referred to as first sensor data). Similarly, raw image 1101 may be historical data or stored data. The process of forming sub-regions is the same for stored observation data as for current observation data. However, for stored observation data (also referred to as second sensor data), it is not necessary to store the raw image, but only a single observation vector or matrix corresponding to the raw image (slice image or image in a reduced size format) needs to be stored. This is described in more detail below. In order to obtain raw observation data (such as raw image 1101), it is necessary to perform a process of capturing sensor data by an agent. It should be understood that any sensor or virtual sensor capable of recording spatial information may be used. In the case of raw image 1101, a camera may be used.

[0401] In a second step, the original image 1101 is divided into separate feature channels to provide, for example, four feature channels. These feature channels may include, for example, red, green, luminance vertical edge, and luminance horizontal edge. The resolution may be the same as that of the original image 1101, for example, 256×64 pixels. Fig.11 In FIG. 1 , a red feature channel 1102 is shown. Other types of channels are also possible. The edges can be obtained by convolution with a 1×2 kernel.

[0402] In a third step, for each feature channel of the original image 1101, such as the red channel 1102, the image is processed, such as by a box filter having a 17×17 square (or any other size of square, such as a 19×19 square) to blur the information of each channel, thereby producing a filtered image 1103 for each channel. It should be noted that other types of image processing are also possible, which is well understood.

[0403] In the fourth step, a reduction process is performed on the filtered image 1103. The reduction process includes reducing the size of the filtered image 1103 by performing a slicing operation. Specifically, the filtered image 1103 is sliced ​​to remove a portion of the rows of the filtered image 1103 and a portion of the columns of the filtered image 1103. Fig.11 A first reduced format image 1104 is shown. The first reduced format image 1104 is the result of slicing a predetermined number of rows in the filtered image 1103. Fig.11, 4 rows are "sliced" from the filtered image 1103 (discarding the remaining rows) to obtain a 256×4 first reduced format image 1104. The choice of rows to slice out and use is arbitrary, but it is advantageous to use rows that are not adjacent to each other (e.g., rows 8, 24, 40, and 56, or rows 6, 22, 38, and 54). Fig.11 Also shown is a second reduced format image 1105. The second reduced format image 1105 is the result of slicing a predetermined number of columns in the filtered image 1103 in addition to performing the slicing operation to obtain the first reduced format image 1104. Fig.11 , 16 columns are "sliced" from the filtered image 1103 (the remaining columns are discarded) to obtain a 16×4 second reduced format image 1105. The choice of columns to slice out and use is arbitrary, but it is advantageous to use columns that are not adjacent to each other (e.g., columns 1, 17, 33, 49, 65, 81, 97, 113, 129, 145, 161, 177, 193, 209, 225, 241). It is further advantageous to choose evenly spaced columns to obtain an even distribution of data from around the agent.

[0404] When performing two slicing operations (to obtain Fig.11 When the second reduced format image 1105 is shown, the observation vector o can then be obtained. Based on the original sensor data 1101, the observation vector or matrix is ​​actually a rough representation of the 360-degree environment of the intelligent body. Fig.11 In the example of , the observation vector is formed from the second reduced format image 1105, which itself is a 16×4 matrix representing a reduced and simplified version of the original data 1101. Other sizes are possible and will be appreciated. Selecting the rows and columns to be retained from the filtered image 1103 in a uniformly distributed manner during the slicing operation ensures that the observation vector or matrix well represents the entire field of view associated with the original image 1101. In other words, it is advantageous if the columns and rows in the filtered image 1103 that can still form the observation vector or matrix after slicing are uniformly distributed throughout the filtered image 1103.

[0405] In order to obtain the observation vector so that it is in a format suitable for comparison, additional processing can be performed. Specifically, the 16×4 second reduced format image 1105 is circularly convolved with a central excitation-peripheral inhibition (or negation) kernel. The channels of the 16×4 second reduced format image 1105 can be subdivided into two data subsets, one consisting of red and green or other "color" vectors, and the other consisting of luma_h and luma_v or other "edge" vectors. Each of these data subsets is normalized via vector normalization (by dividing by the corresponding Euclidean norm) so that the elements of each of the color vector and the edge vector describe the value on a set of orthogonal values, and the result is a unit vector. Then, by scaling each normalized vector with the square root of two, these normalized vectors are cascaded into a single unit vector. This provides an observation vector o associated with the second reduced format image 1105 and the original image 1101. The observation vector in this example is cascaded by 16 columns, 4 rows, and 4 channels to produce a vector of 256 elements. Although these processes are performed on the 16×4 second reduced format image 1105 to produce a one-dimensional vector suitable for comparison calculations, it should be understood that reference to the second reduced format image 1105 (16×4 matrix) below is synonymous with the observation vector. The observation vector is actually the same information as the second reduced format image 1105, but in a different format and in a normalized form. Therefore, the second reduced format image 1105 can be referred to as an observation vector.

[0406] The first to fourth steps described above are applicable to either the first (current) sensor data or the second (stored) sensor data, which means that the process of forming an observation vector or matrix for each of these sensor data is exactly the same.

[0407] In some embodiments implementing an observation-action function, an additional fifth step is performed to obtain a sub-region based on the observation vector.

[0408] In this fifth step, the observation vector or matrix 1105 is sub-sampled to obtain sub-regions 1106a to 1106n of sensor data. Fig.11, sub-regions 1106a to 1106n of the red feature channel are shown. Sub-regions 1106a to 1106n are actually part of the field of view, which is roughly represented by a 16×4 observation vector or matrix. The fifth step involves applying a mask to the observation vector or matrix 1105 to obtain each sub-region. The mask is superimposed on the observation vector or matrix 1105, and the output of the operation is a reproduction of the cells or pixels of the observation vector or matrix 1105 superimposed by the mask. The size of the mask is smaller than the size of the observation vector or matrix 1105, so only a small portion of the pixels of the observation vector or matrix 1105 are reproduced by the mask. This small portion that is reproduced forms a sub-region. In one example, the mask is 8×4 pixels and is superimposed on the 16×4 observation vector or matrix. This means that the mask discards 8 columns of pixels, which accounts for half of the field of view of the original image 1101. In order to obtain a plurality of sub-regions 1106a to 1106n, the mask is iteratively arranged on different columns of the observation vector or matrix 1105, or alternatively, the observation vector or matrix 1105 is arranged on the mask by column each time. These two operations are actually the same. It should be understood that since the observation vector or matrix 1105 is a rough representation of the complete view (e.g., a 360-degree view) corresponding to the original image 1101, the last column of the observation vector or matrix 1105 is actually next to the first column, which means that the mask can be cyclically arranged through the last column and arrive at the first column.

[0409] The size of the mask and the number of columns that the mask moves relative to the observation vector or matrix 1105 are changeable, and changing these variables may produce different effects. In the above example, the width of the mask is 8 columns, which means that its width is half the width of the observation vector or matrix 1105. The result of this arrangement is that the sub-regions 1106a to 1106n produced by the mask are actually equal to half of the field of view of the observation vector or matrix 1105. Increasing the size of the mask will increase the field of view portion retained in the sub-regions 1106a to 1106n, while reducing the size of the mask will reduce the field of view portion retained in the sub-regions 1106a to 1106n. The comparison between larger sub-regions is more reliable, because there are more pixels in each sub-region that can be used as a basis for comparison. However, the benefit of having smaller sub-regions is to allow the relatively more limited details of the scene captured in the original image 1101 to be compared. In other words, smaller sub-regions may be less affected by more extensive changes in the environment. Therefore, smaller sub-regions can be used to improve the effective range for comparison between stored data and current data. Specifically, when the agent moves away from an object or other attribute, the object appears smaller and is ultimately found to be located in a smaller pixel region in the original image 1101 than when the object is close to the agent. Similarly, when the agent moves away, the surroundings of the object may change relatively dramatically in the original image 1101. Using smaller sub-regions (e.g., tuned to the expected size of such an object at a certain distance) will allow comparisons to be made between sub-regions to find the object, even when neighboring pixels of the observation vector or matrix 1105 vary greatly due to the distance to the object. This example illustrates that there is a balance between selecting a sub-region that is larger than the size of the observation vector or matrix 1105 and selecting a smaller sub-region. Both options have benefits. Even with larger sub-regions, comparison methods such as observation action functions are still robust to changes in distance from the target / specific object.

[0410] As explained above, in each iteration of obtaining subregions, the shift of the mask relative to the observation vector or matrix 1105 is also variable. This is synonymous with determining the number of subregions to be obtained. In one example, the mask can be shifted by one column per iteration, which means that each subregion obtained is arranged in only one column compared to the previous subregion. This is the slightest difference that can exist between subregions. In the example of the 16×4 observation matrix above, 16 subregions can be obtained in this way (the maximum possible number of subregions for a 16×4 matrix). Then, it can be decided to arrange the mask in 8 columns to obtain two subregions, or in 2 columns to obtain 8 subregions in each iteration. The number of subregions to be obtained is arbitrary, but it is advantageous to obtain four or more subregions to increase the reliability of the motion vector formed by the comparison of the subregions, as explained later. Again, with this variable, there is a balance between reliability (more subregions in comparison) and computational burden (comparing more subregions requires more computation). Therefore, the choice of the number of subregions depends on the needs and requirements of the specific use case.

[0411] When a normalized observation vector is masked to obtain subregions (actually subvectors), the normalization of this vector is lost. Therefore, in addition to the above process, the subregions (and especially the subvectors corresponding to these subregions) are also renormalized.

[0412] Before the comparison of a subregion of current (first) sensor data with a subregion of stored (second) sensor data is explained in more detail with respect to the action function, firstly the recording and storage of observation vectors or matrices 1105 is explained here.

[0413] It should be understood that the sub-region does not need to be stored and saved by the agent or a memory associated with the agent, but is only needed when, for example, the observation action function is used to calculate the comparison. Therefore, the sub-region can be generated from the observation vector or matrix 1105 when needed.

[0414] Several options are possible with respect to recording and storing the observation vector or matrix. As described in the fourth step above, the observation vector or matrix 1105 is a coarse representation of the original image 1101, providing information of the entire field of view represented in the original image 1101, but at a reduced size. As part of the process of forming the observation vector or matrix 1105, slicing is performed to effectively discard rows and columns of the filtered image 1103 to reduce its size. However, for the purpose of comparing the current observation vector and the stored observation vector, it is advantageous if the information deleted by these slicing operations is not lost. Different options for storing this data are possible for both the stored (second) sensor data and the current (first) sensor data.

[0415] In the first option, when the observation results from the original image 1101 are stored using the above process, multiple observation vectors or matrices 1105 are stored for each original image 1101. The multiple observation vectors or matrices 1105 are obtained by cutting out different columns in the fourth step described above. In the example described above, the original image 1101 is an image of 256×64 pixels. The image is filtered and sliced ​​to provide 16×4 observation vectors. Assuming that the columns retained in the slicing operation are evenly distributed in the original image 1101, this means that for each channel in which the original image 1101 is captured, there are 16 possible 16×4 observation vectors (256 / 16=16). These different observation vectors are obtained by arranging the columns in the slicing operation one column at a time. At the 17th arrangement, the observation vector is the same as the 1st arrangement. For example, if a first observation vector is generated from columns 1, 17, 33, 49, 65, 81, 97, 113, 129, 145, 161, 177, 193, 209, 225, and 241 in the original image 1101, a second observation vector may be generated from columns 2, 18, 34, 50, 66, 82, 98, 114, 130, 146, 162, 178, 194, 210, 226, and 242. This process may be repeated until the first column in the observation vector corresponds to column 17 of the original image 1101, in which case the observation vector is simply a rotation of the first observation vector. Thus, in this example, a set of 16 observation vectors is required to ensure that each column of the original image 1101 is preserved.

[0416] Additionally, each of the 16 observation vectors (obtained from slices of different corresponding columns) can itself be permuted up to 16 times to move the data in that observation vector to different columns. This is effectively a rotation of the observation vector (e.g., whereby data originally in column 1 is permuted to column 2, and so on, data in column 16 is permuted to column 1). Thus, there are 16 possible rotations for each observation vector, and 16 different observation vectors can be formed by modifying the slicing operation. Thus, there are a total of 256 possible observation vectors from the 256×64 original image 1101.

[0417] In the first option as described above, a set of 256 observation vectors or matrices is stored for the stored sensor data. When comparing with the current observation result (from the current sensor data), the corresponding set of 256 current observation vectors can be compared with the set of stored observation vectors. Using the same process as finding the set of observation vectors for the current original image and the stored original image, this means, for example, comparing a set of 256 16×4 stored observation vectors with a set of 256 16×4 current observation vectors. Although this can provide higher comparison accuracy, it is computationally inefficient because a large number of vector dot products need to be calculated. Storing and comparing with the complete set of stored observation vectors also means that when comparing with the current observation vector, an additional step is required, namely selecting the best comparison result from the complete set of stored observation vectors. Therefore, comparing with the complete set of stored observation vectors introduces an additional layer or step in the comparison process, which further increases the computational burden. However, obtaining the best comparison based on one stored observation vector in the complete set of stored observation vectors may provide more accurate and useful results.

[0418] In the second option, when storing observation vectors, only one observation vector or matrix need be stored. The observation vector stored may be referred to as the zeroth observation vector, and in the example above may be, for example, the observation vectors obtained from columns 1, 17, 33, 49, 65, 81, 97, 113, 129, 145, 161, 177, 193, 209, 225, and 241 in the original image. The image itself and other possible observation vectors are not stored. This means that the details in the original image 1101 are effectively lost, and the only data available to the agent about the original image 1101 is a rough representation of the original image provided by the zeroth 16×4×4 observation vector. The reason this is acceptable is that when comparing with the current observation (which may not be stored in permanent storage, but may be recorded dynamically or recorded in, for example, volatile storage), the stored zeroth observation vector is still compared with every possible permutation (e.g., a set of 256) of the current observation vector. Therefore, the stored observation vector is compared to a set of 256 16×4×4 current observation vectors. In other words, the zeroth observation vector stored is stored for all data obtained in the current observation (all data in the data provided by the 256×64 current original image). This ensures that if any data in the data of the current observation is similar to the zeroth stored observation vector, such similarity should be found by comparing the zeroth stored observation vector to the complete set of current observation vectors.

[0419] The second option is computationally more efficient than the first option described above because, for a given original image, only one observation vector needs to be stored to compare with the new current data. Furthermore, in the second option, there is no additional step of determining which stored observation vector (of the complete set) to use when performing the comparison with the current observation vector. This is particularly advantageous when the agent or a computer associated with the agent may have to store hundreds / thousands of observation vectors associated with different locations, stored objects, or other attributes.

[0420] According to a second option, when comparing observations, the stored observation vectors are compared with a set of current observation vectors, each of which is a possible arrangement of the current original image. When an observation action function is used, as described in more detail below, this involves comparing a sub-region of the stored observation vectors (or in other words, the stored sub-vectors) with a sub-region of the set of current observation vectors (or in other words, the current sub-vectors). When the stored observation vectors and the current observation vector are compared by Fig.11 In the example of the illustrated 16×4 second reduced format image 1105 formation (for each of every 4 channels), there are 256 possible 16×4 current observation vectors, each of which can be used to form a sub-region (or current sub-vector) by processing each current observation vector with the aid of a mask for generating a sub-region.

[0421] This means that there are a total of 256 possible sub-regions of the current observation vector to compare with the 16 possible sub-regions of the stored observation vector. Fig.12 This possibility is illustrated in FIG. 12 where a spatial diagram 1200 indicates the number and arrangement of sub-areas 1201 around an agent 1202. It should be understood that Fig.12 It is not to scale and is for illustration purposes only. Each arm 1203 of the diagram 1200 represents a sub-region obtained from the same mask location relative to the current observation vector. Each sub-region 1201 moving down each arm 1203 is from a different current observation in the set of current observations (of different slices). Each concentric set 1204 of sub-regions on the diagram 1200 is from the same current observation vector, but corresponds to a different orientation in space, either because the location of the mask used to generate the sub-region is different (or arranged) for each sub-region in the concentric set 1204, or because the current observation vector itself has been rotated before extracting the sub-region. Although in Fig.12 , but it should be understood that each sub-region includes a portion of the 360-degree view around the agent, which portion depends on the size of the mask as described above. In one example, each sub-region is equivalent to half of the field of view corresponding to the current original image associated with the data. The sub-regions overlap each other.

[0422] In the observation action function, the current sub-region (from the current first sensor data) is compared to a set of sub-regions derived from the stored observation vectors. The best match for each "stored" sub-region in the set of "stored" sub-regions is found from the current sub-region, and the offset angle and magnitude are determined for each best match. This process is described in more detail here for the observation action function.

[0423] As indicated above, the stored observation vector is divided into N sub-regions (also referred to as "stored" sub-regions), and for each of the N stored sub-regions, a comparison is made with the complete set of sub-regions of the current observation vector (also referred to as the "current" sub-region) to obtain all possible rotation and slicing arrangements (e.g., 256 arrangements of a 16×4 observation vector) to determine which arrangement is most similar to the stored sub-region (also referred to as the current best matching sub-region). The most similar particular arrangement has an index, such as the 120th rotation, which is used to identify the offset relative to the stored sub-region. The offset is calculated for each stored sub-region, and a similarity measure is optionally calculated, where the measure indicates the similarity between the stored sub-region and the current best matching sub-region. The similarity measure is determined by performing the comparison itself, where the comparison is a combined observation similarity function and a rotation function as described above. Specifically, the scalar product of each current sub-region in the set of stored sub-regions and the current sub-region is calculated to determine the current best matching sub-region. Due to the normalization of the sub-regions, the scalar product is maximum (1) when the vectors are aligned (i.e., the relative differences between the pixels are the same); zero (0) when the vectors are orthogonal; and minimum (-1) when the vectors are completely opposite. Therefore, the comparison returns a value in the range [-1,1] and the index of the location of the current sub-region that provides the best match. Thus, the offset and magnitude are associated with the current sub-region and / or the stored sub-region.

[0424] FIG13A shows a schematic diagram of sub-region comparison. FIG13A shows agent 1301. Agent 1301 is located at the center of a “current observation” cylindrical projection 1302, which includes a green feature channel projection 1302a and a red feature channel projection 1302b. Current cylindrical projection 1302 is an exemplary spatial projection of a current observation vector, and shows how the current observation vector describes the environment around a first position of agent 1301. Specifically, the current observation vector describes in the sensor data how a real scene 1303 (i.e., the world) appears around agent 1301 at the first position. FIG13A also shows a similar depiction of a “stored observation” cylindrical projection 1302′ around the agent, which includes a green feature channel projection 1302a′ and a red feature channel projection 1302b′. Cylindrical projection 1302' is an illustrative spatial projection of the stored observation vectors, and shows how the stored observation vectors describe the environment around the second position of the agent 1301. In FIG. 13A, the projections of the stored observation vectors and the current observation vector are divided into sub-areas. Specifically, FIG. 13A shows the first (front) sub-area 1304a of each of the "current observation" cylindrical projection 1302 and the "stored observation" cylindrical projection 1302, and the second (rear) sub-area 1304b of each of the "current observation" cylindrical projection 1302 and the "stored observation" cylindrical projection 1302. In other words, FIG. 13A shows the front stored sub-area and the current sub-area 1304a and the rear stored sub-area and the current sub-area 1304b. It should be understood that this is merely illustrative, and in fact, the current sub-areas 1304a and 1304b are samples of a set of all possible arrangements of the current sub-areas for comparison.

[0425] The set of sub-regions stored in the front and the current sub-region 1304a are compared to each other using the observed similarity function and the rotation function to obtain a scalar product and an offset. In addition, the set of sub-regions stored in the back and the current sub-region 1304a are compared to each other using the observed similarity function and the rotation function to obtain a scalar product and an offset. These operations provide sub-region offset angles 1305a and 1305b, respectively. These operations also provide a magnitude M indicating the similarity between the sub-regions stored in the front and the current sub-region 1304a. F and a magnitude MB indicating the similarity between the sub-region stored at the rear and the current sub-region 1304 b .

[0426] FIG. 13B shows the same arrangement of agent 1301. The only difference in FIG. 13B is that the stored sub-region and the current sub-region that are compared are not the front sub-region and the rear sub-region, but the current sub-region and the stored sub-region 1304c on the right side and the current sub-region and the stored sub-region 1304d on the left side. The comparison process is otherwise the same, i.e., the complete set of the sub-region stored on the right side and the current sub-region 1304c is compared with each other using the observed similarity function and the rotation function to obtain the scalar product and the offset. In addition, the complete set of the sub-region stored on the left side and the current sub-region 1304d is also compared with each other using the observed similarity function and the rotation function to obtain the scalar product and the offset. These operations provide sub-region offset angles 1305c and 1305d, respectively. These operations also provide the magnitude Mr indicating the similarity between the sub-region 1304c stored on the right side and the current best matching sub-region and the magnitude ML indicating the similarity between the sub-region 1304d stored on the left side and the current best matching sub-region.

[0427] It should be understood that while Figures 13A and 13B show corresponding front, rear, right, and left stored sub-regions and the current sub-region and the offset angle determined by comparing them, there may be any number of stored sub-regions corresponding to any portion of the entire stored observation vector and therefore to the direction of the entire stored observation vector. Figures 13A and 13B show sub-regions that are actually half the size of the observation vector and therefore include sensor data indicating half of the field of view associated with the observation vector. Therefore, in the example of the 16×4×4 raw observation vector described above, these sub-regions would be equivalent to an 8×4×4 field of view centered in different directions (front, rear, right, left). The sub-regions overlap each other. In alternative embodiments, the sub-regions may represent smaller or larger portions of the observation vector.

[0428] The sub-region offset angles 1304a, 1304b, 1304c and 1304d are measured relative to the axis corresponding to the overall best matching current observation vector. It should be understood that the overall best matching current observation vector can be determined by a similar comparison process that is performed simultaneously, before or after the sub-region comparison as described above.

[0429] The overall best matching current observation vector is determined in the same manner as the comparison between a set of stored sub-regions and the current sub-region. The only difference is that instead of performing a sub-region comparison, the entire stored observation vector is compared to a set of permutations of the current observation vector (including rotations and slice permutations). The entire stored observation vector (e.g., a 16×4×4 vector) is compared to each permutation of the current observation vector using an observation similarity function and an observation rotation function to determine the best matching current observation vector. These functions, as described above, provide an index (e.g., the 120th rotation permutation) corresponding to the best matching observation vector offset angle and an associated magnitude that indicates the similarity between the stored observation vector and the best matching observation vector.

[0430] Fig.14 A schematic diagram is shown of comparing a complete stored observation vector with each current observation vector in a set of current observation vectors to determine the best matching current observation vector. Fig.14 Agent 1401 at a first position is shown. Agent 1401 is located at the center of a cylindrical projection 1402, which includes a green feature channel projection 1402a and a red feature channel projection 1402b. Cylindrical projection 1402 is an illustrative spatial projection of a current observation vector, and shows how the current observation vector describes the environment around the first position of agent 1401. Specifically, the current observation vector describes how the real scene 1403 (i.e., the world) around agent 1401 at the first position appears. It should be understood that although one current observation vector projection is shown, in practice the comparison is performed for the entire set of current observation vectors. Fig.14 A similar depiction of agent 1401' at a second position is also shown, along with a cylindrical projection 1402' that includes a green feature channel projection 1402a' and a red feature channel projection 1402b'. Cylindrical projection 1402' is an exemplary spatial projection of the stored observation vector, and shows how the stored observation vector describes the environment around the second position of agent 1401'. Although both the current observation vector and the stored observation vector in this example are shown as only containing two channels, it should be understood that there may be more channels, such as, as illustrated above, four channels.

[0431] The stored observation vector indicated by the stored cylindrical projection 1402' is compared with each possible rotation and slice arrangement of the current observation vector to find the best match using the observation similarity function and observation rotation function discussed above. The observation similarity function returns a magnitude indicating the similarity between a particular current observation vector and the stored observation vector. The magnitude indicating the highest similarity between a particular current observation vector and a stored observation vector (in the set of current observation vectors) is recorded, and the index of the associated particular current observation vector is also recorded. The index provides the best match direction 1405. Thus, the best matching observation vector offset angle 1404 can be determined relative to the stored observation vector and the best matching current observation vector. In other words, the best match of the stored observation vector in the set of current observation vectors is determined by comparing the stored observation vector with each current observation vector using a scalar product. The best match index is recorded, and the best matching observation offset angle 1404 is inferred therefrom.

[0432] The best match observation vector offset angle 1404 indicates the rotation that the agent 1401 can perform in the first position to align itself with the best match relative to the stored observation vector. Fig.14 In the case where the stored observation vector is recorded at coordinates (0,0), and agent 1401 has moved forward from its second (original) location at 1401', the current observation vector represented by projection 1402 has been rotated to the left, with the result that its current observation vector at best matching observation vector offset angle 1404 (measured as a positive rotation counterclockwise from forward) is the best match to the stored observation vector.

[0433] It should be understood that since the agent 1401 is capable of recording observations over a wide field of view (e.g., 360 degrees), it is not necessary to perform a rotation at this time. Instead, a simple realignment can be performed based on the agent's axes (reference frame) via the best matching observation vector offset angle 1404 so that the best matching direction 1405 is aligned with the agent's axes for comparing sub-regions. This can be done before, during, or after the sub-region comparison operation. Since the sub-regions are generated based on specific directions (they do not represent the entire field of view), alignment of the agent's axes is very important to ensure that the (e.g., forward, backward) sub-regions of the stored observation vectors are appropriately oriented (by adjusting the axes by the best matching observation vector offset angle 1404) so ​​that when determining the action vector, this is done based on the sub-region offset relative to the adjusted axis.

[0434] Figures 15A to 15D show each of the sub-region offset angles relative to the axis of the agent 1301. In Figures 15A to 15B, an axis 1310 relative to the agent 1301 is illustrated. Axis 1310 represents a realigned axis, wherein axis 1310 is realigned based on the best matching observation vector offset angle 1404 obtained by comparing the stored observation vector with the best matching current observation vector. Figure 15A also shows sub-region offset angles 1304a and 1304b that are relatively relevant to the front sub-region and the rear sub-region, respectively. Similarly, Figure 15B shows sub-region offset angles 1304c and 1304d that are relatively relevant to the right sub-region and the left sub-region, respectively. Once the offset of each sub-region is determined in this way, the direction of the action vector is determined. In order to determine the direction of the action vector, a synthetic vector is calculated. This process is shown in Figures 15A and 15B. In Figure 15A, an opposing pair of front and rear sub-region offsets 1304a and 1304b are used to calculate a front / rear resultant vector 1306a. In Figure 15B, an opposing pair of right and left sub-region offsets 1304c and 1304d are used to calculate a right / left resultant vector 1306b.

[0435] The front / back resultant vector 1306a and the right / left resultant vector 1306b are then aggregated to form a final motion vector 1308a as shown in FIG. 15C.

[0436] It should be understood that Figures 15A and 15B are intended to illustrate the process of defining front / back and right / left composite vectors using relative sub-region comparison pairs, but in fact, it is not necessary to form a composite vector, and the action vector 1308a can be calculated directly based on the aggregation of all sub-region offset angles 1304a to 1304d. In addition, additional sub-regions centered at different directions relative to the alignment axis of the agent 1301 can be used. For example, Figure 15D shows a similar diagram to Figure 15C, but involves sub-region comparisons made at 45 degree angles to the front / back and left / right directions. The aggregation of the sub-region offset angles of these different sub-regions provides the action vector 1308b. It should be understood that these processes provide the directional components of the action vectors 1308a and 1308b.

[0437] The direction of motion vector 1308a determined from the front / back and left / right directional sub-region comparisons may be aggregated with the direction of motion vector 1308b determined from the additional sub-region comparisons. In the example of a 16×4×4 observation vector, there may be up to 16 sub-regions that contribute to determining the motion vector in this manner. The direction of motion vector 1308a and the direction of motion vector 1308b may be aggregated in any suitable manner to provide a final motion vector. In one example, a vector average is calculated to determine the final motion vector. Alternatively, the final motion vector may be selected as one of motion vectors 1308a or 1308b based on one or more parameters.

[0438] The magnitude of the final motion vector is determined separately from the magnitude calculated during the execution of the observation similarity function to identify the best matching observation vector and the current best matching sub-region. The final motion vector has a magnitude based on the magnitude and direction of the offset angles 1304a to 1304d. As described above, the offset angles 1304a to 1304d are determined relative to the axis 1310 aligned with the best matching direction 1405. The magnitude of the motion vector is determined by the components. Specifically, the first component is the front / back composite vector 1306a. The composite vector 1306a is the component of the motion vector in a direction orthogonal to the front / back direction of the axis 1310, as shown in Figure 15A. The magnitude of the composite vector 1306a is determined based on the magnitude of the offset angles 1304a and 3014b. The offset angles 1304a and 1304b can be measured in discrete steps based on a rotation index (e.g., index 4 in 256). The magnitude may be determined by averaging the magnitudes of the offset angles 1304a and 1304b (averaging the offset angles in discrete 1 / 256 steps). The magnitude may be multiplied by a constant k, which may be modified based on specific scene requirements, and k may be a positive number such as 2, 4, 6, 8, 10, or any other number.

[0439] Additional aspects of the amplitude determination may include the use of additional constants to define an upper limit for the use of specific offset angle pairs. In some cases, larger offset angles may indicate a false match, or that the match between sub-regions is unreliable. To prevent such matches from being used in determining motion vectors, a constant j may be set such that the amplitude is set to 0 if the average offset angle between opposing pairs is greater than j. The constant j may be modified and take any value, such as 2, 5, 10, 20, or 50, etc.

[0440] The magnitude of the resultant vector 1306b is calculated in the same manner relative to the offset angles 1304c and 1304d. This provides another orthogonal component of the motion vector, such as Fig.15b The resultant vectors 1306a and 1306b may then be added together to determine the motion vector.

[0441] If the offset angles in either the front / back or left / right directions are both zero (i.e., the center of the best matching subregion of the current observation vector is the same as the subregion of the stored observation), then the magnitude of the associated orthogonal component is zero. The magnitude follows a relationship where the larger the offset from the stored subregion, the larger the magnitude. Such magnitudes exist for each relative subregion offset pair (such as an offset angle pair including the front offset angle 1304a and the rear offset angle 1304b, or an offset angle pair including the right offset angle 1304c and the left offset angle 1304d). These magnitudes are then combined accordingly as part of the vector average determination when forming the action vectors 1308a and 1308b and the final action vector. Thus, the magnitude of each relative subregion offset pair provides a weight to the final action vector based on the magnitude of the offset angle compared to the stored subregion. This essentially means that the action vector is weighted more based on the resultant vectors 1306a and 1306b that originate from larger offsets (offset angles that are away from the alignment axis). This in turn provides a final action vector that points in the direction of where the location in space that produced the stored observation is likely to be found, with the magnitude of the final action vector indicating the distance or effort required to reach that location. The magnitude is also inversely proportional to the confidence level, which indicates the confidence that the location associated with the stored observation is in the direction indicated by the final action vector. Thus, a larger magnitude of the final action vector indicates a greater effort required to reach that location, and likewise indicates a lower certainty that the location is in the indicated direction. Conversely, a lower magnitude indicates less effort required to reach that location, and a higher certainty that the location is in the indicated direction.

[0442] Although the magnitude of the motion vector is determined based on the relative offset pair of the best matching sub-region, the similarity measure M obtained in the comparison between the stored sub-region and the current sub-region can still be used in this process L Mr. M f , Mb. Specifically, weights may be applied to motion vector magnitudes based on similarity values ​​of pairs of relative sub-regions used to calculate motion vectors. Relative sub-regions with high similarity values ​​may be weighted to increase the motion vector magnitude contribution from a particular relative sub-region, while sub-regions with low similarity values ​​may be weighted to reduce their motion vector magnitude contribution.

[0443] In addition, the similarity values ​​Ml, Mr, Mf, Mb can also be used in a threshold operation before determining the motion vector to remove or ignore any sub-regions from the determination of the motion vector. Specifically, the similarity value of each sub-region comparison can be compared with a definable confidence threshold, which can be, for example, 0, 0.1, 0.5, or any other number. If the similarity value does not match the confidence threshold or exceeds the confidence threshold, the sub-region comparison corresponding to it is not used to determine the motion vector. As described above, it should be understood that the similarity values ​​and sub-region directions used to determine the motion vector may be more than the similarity value M. L Mr. M f , Mb should be larger.

[0444] The final action vector can be used alone, or in conjunction with any of the other functions or methods described above, to determine how to move the agent toward the stored observations corresponding to the stored observation vectors. For example, the action vector can be used in a move function to reposition the agent toward the predicted observations, and can be used as a proxy for Figure 7 The displacement data shown may be used in addition to the repositioning technique.

[0445] The observation-action function is a useful navigation tool because it provides an action vector along which the agent can move to reach a destination indicated by a stored observation vector. By storing multiple observations of a particular space, the agent is able to iteratively use the action function to navigate that space for multiple stored observations associated with that space. Since multiple sub-regions are used in determining the action vector, and these sub-regions themselves are pre-processed to reduce the sensor data to its most basic components (edges, salient features, etc.), the action function is relatively resilient to small environmental changes.

[0446] The observation action function can also be used in combination with existing navigation methods to form a hybrid navigation process. For example, the observation action function process can form the main navigation process to be used in combination with an auxiliary navigation process (such as a positioning system). The positioning system can be, for example, the Global Positioning System (GPS), a satellite-based radio navigation system, or any other positioning system, such as a Local Positioning System (LPS).

[0447] In this hybrid navigation process, a specific location in the real environment forms a target location. The target location can be described by coordinates, etc. and target observations. The target observations can correspond to sensor data indicating environmental features of the target location. A combination of a primary navigation process (observation action function) and an auxiliary navigation process (positioning system) is used to navigate to the target location. The combination of these two processes or systems may be different. In some embodiments, the relationship between the primary navigation process and the auxiliary navigation process can be mutually exclusive, which means that only one process performs navigation at any given time. In an alternative embodiment, the primary navigation process and the auxiliary navigation process can work together to adjust the impact of each process on the overall navigation based on one or more factors.

[0448] Fig.16 A flow chart of a method 1600 of navigating using a hybrid navigation process is shown. The method will now be described.

[0449] In a first step 1602, a target location is determined. The target location is a destination in the world to which the agent will navigate. The target location may be input by a user or other entity as part of an adjacent process. The target location may be input as any data indicating an actual location in the world. For example, the target location may be input as text data or location (coordinate) data, etc.

[0450] In a second step 1604, target position data corresponding to the target position is retrieved. The target position data includes at least primary data and auxiliary data, which are used for a primary navigation process and an auxiliary navigation process, respectively. Although the primary navigation process is exemplified as an observation action function here, it should be understood that any navigation method in the above-mentioned navigation methods (e.g., using observations and displacements) can be used as a primary navigation process. The auxiliary navigation process is an existing navigation process, such as using GPS. The primary data and auxiliary data are retrieved in the required format as inputs to a specific primary navigation process and an auxiliary navigation process, respectively. In the example of the observation action function and GPS, the primary data are observations or observation vectors, and the auxiliary data are GPS coordinates.

[0451] In the third step 1606, the agent uses the target location data to navigate to the target location according to the combined scheme, which specifies how to use the primary navigation process and the auxiliary navigation process in the entire navigation process.

[0452] Method 1600 provides a hybrid method that combines the benefits of a primary navigation process and an auxiliary navigation process. Specifically, an auxiliary navigation process (such as using a positioning system such as GPS) is useful for navigating to a target location because it is usually accurate to within 2m and can be used in most places in the world. Therefore, the benefit of using GPS is that the agent can be navigated to within a few meters (or more conservatively, within 50m) of the target location. However, existing navigation systems such as GPS cannot fully function in buildings, underground or under aerial cover. In addition, although GPS is quite accurate, it is not enough to guide the agent around obstacles, along a path, or through building fixtures (such as doors and windows) because such features are usually less than the accuracy of GPS.

[0453] At the same time, the benefit of the primary navigation process (such as the observation-action function) is that it is not affected like GPS when, for example, navigating through or under a building. Since the observation-action function relies on spatial data captured locally by the agent, no satellite signals are required, so the observation-action function can be used in signal-blind areas where GPS cannot be used. In addition, the use of spatial data (such as images) enables the agent to navigate in the environment based on the constraints of the environment (such as obstacles) and other features (such as doors, open spaces, and windows) when using the observation-action function.

[0454] A hybrid approach that combines the use of GPS and an observed-action function enables the agent to leverage each navigation process when it is most advantageous, realizing the benefits of both while eliminating their drawbacks. The agent can use GPS to navigate to a target location, then switch to the observed-action function to complete navigation when GPS is not reliable enough for the more complex final navigation steps. For example, if the agent is a delivery robot, the agent can use GPS-based navigation to complete the majority of the distance from the original location to the target location. However, when arriving at a predetermined delivery point (e.g., coordinates or a building address), the agent can switch to using the observed-action function to deliver the item to the building door or even navigate itself inside a building (where GPS is not available) to make a delivery.

[0455] The combined scheme specifies how the primary navigation process and the auxiliary navigation process interact during the entire navigation process of the intelligent agent.

[0456] In some embodiments, the combination scheme is a mutually exclusive scheme. In this scheme, the agent navigates using either the primary navigation process or the secondary navigation process at any given time, but never both at the same time. In order to determine when to switch from the primary navigation process to the secondary navigation process (or vice versa), one or more switching conditions need to be tested throughout the navigation process.

[0457] The switching condition for switching to the main navigation process (observation action function) may include a definable confidence threshold that is tested based on a comparison between the current observation and the stored observations corresponding to the target location (primary data). Specifically, an observation similarity function may be periodically executed to determine the best matching current observation vector of the stored observation vectors corresponding to the target location. The function returns a magnitude representing the similarity between the best matching current observation vector and the stored observation vectors. This magnitude may then be compared to a definable confidence threshold. If the magnitude matches or exceeds the definable confidence threshold, this particular switching condition may switch from the secondary navigation process to the primary navigation process.

[0458] In practice, this means that when the current observations are sufficiently similar to the stored observations corresponding to the target location, the primary navigation process (such as the observation action function) becomes the primary method used for the entire navigation process. The comparison essentially involves generating observations to obtain the current observation vector for comparison with the stored observation vector corresponding to the target location. In this regard, the comparison can be viewed as a reference. Figure 7 The "transition observations" described above can be made periodically during navigation or at a specific time or location. Specifically, transition observations can be frequently and periodically generated throughout the navigation process to test switching conditions, such as Figure 7 As shown in the transition observation result 701-2 in the first figure 701 of FIG. Alternatively, the transition observation result 702-2 can be generated when the path 702-3 navigated according to the auxiliary navigation process expires or is close to expiration to test the switching condition. In other words, the transition observation result can be generated according to meeting the proximity threshold, which represents the proximity area where the agent is located at the target location according to the auxiliary navigation process.

[0459] In one example, the agent is provided with target location data, which includes primary data as a stored observation vector indicating the target location, and auxiliary data as coordinates indicating the target location, etc. In this example, the auxiliary data may indicate that the target location is 10 km away from the current location of the agent. In this example, the agent can navigate using the auxiliary navigation process until the target location is within a proximity threshold. The proximity threshold can be set as a percentage of the entire path length (e.g., 95%, 99%, etc.) or a specific distance (e.g., within 100m of the target location, within 10m of the target location, within 5m of the target location, etc.). Once the proximity threshold is met, a transition observation result is generated to determine whether a definable confidence threshold (switching condition) is met. Specifically, meeting the proximity threshold can directly lead to testing the switching condition via the transition observation result. If the switching condition is met, the agent will switch from navigating according to the auxiliary navigation process to navigating according to the main navigation process. If the switching condition is not met, the agent will continue to navigate using the auxiliary navigation process. In this way, there can be multiple proximity thresholds. For example, the first proximity threshold may be within 100m of the target location, the second proximity threshold may be within 10m of the target location, and the third proximity threshold may be within 2m of the target location. Each proximity threshold causes a transition observation to test the switching condition. Subsequent proximity thresholds are only required if the switching condition is not met. In this example, if the similarity measure does not meet a definable confidence threshold (switching condition) when the first proximity threshold is met, compared to the current observation 100m from the target location, the agent will continue to navigate according to the auxiliary navigation process until the second proximity threshold (10m) is met, at which point the process is repeated. If necessary (if the switching condition is still not met), the agent will continue to navigate using the auxiliary navigation process until the third proximity threshold is met, and so on. If the agent continues to navigate using the auxiliary navigation process until the auxiliary navigation process indicates that the target location has been reached, the auxiliary navigation process terminates. At this point, further transition observations may be optionally generated to determine whether any adjustments to the primary navigation process are required to find the target location. If the switching condition is met, the agent will switch to the primary navigation process and continue to navigate to the target location. If the switching condition is not met, then the agent can be optionally configured to explore the local area using the primary navigation process, or alternatively, an error message can be transmitted to the remote control center to remind the user that the agent did not successfully reach the target location. In an alternative embodiment, the agent can be configured to use the auxiliary navigation process to trace back to a previously visited location. The agent can then be configured to re-approach the location where the switching condition is expected to be met. This is actually equivalent to a "retry" process.Alternatively, the agent can be configured to backtrack to previously visited locations and, using the primary navigation process, systematically attempt to move to one or more observations using hypotheses and the observation / displacement method described above.

[0460] The target location data may be associated with or otherwise linked to one or more metadata segments. The metadata may include an availability metric, such as a scalar value or flag, indicating the predicted or predetermined availability of a primary navigation process or an auxiliary navigation process at a target location and / or a location between the current location of the agent and the target location. For example, the availability metric may provide a scalar value to indicate the expected presence or strength of a GPS signal for an auxiliary navigation process. Thus, the availability metric may be used to inform the entire navigation process to determine when to switch from the auxiliary navigation process to the primary navigation process. In one example, the target location may be located in a building, such as a warehouse, and the availability metric may indicate that it is unlikely that there will be any GPS signals within a 10m radius of the target location. Therefore, the entire navigation process may accordingly determine to switch to the primary navigation process at a distance of 10m from the target location. How to use metadata to enhance the method will be described in more detail later.

[0461] If the auxiliary navigation process is not available (eg, due to the absence of a GPS signal at the agent), the navigation process may automatically switch to the primary navigation process.

[0462] Once navigation is performed according to the primary navigation process, observations are generated as needed according to the observation action function. These observations may be subject to similar tests to determine whether a reverse switching condition (switching from the primary navigation process back to the auxiliary navigation process) is met. The reverse switching condition may include a definable confidence threshold that is the same as the switching condition. If the similarity value generated by the comparison of the current observation with the stored observation corresponding to the target location is lower than the definable confidence threshold, the reverse switching condition is met and the agent can switch from the primary navigation process back to the auxiliary navigation process and repeat the process.

[0463] In any of the above examples, the agent may generate transition observations to determine when to switch from the secondary navigation process to the primary navigation process (or vice versa). In addition to the transition observations, the agent may also make initial observations to determine which navigation process to use starting from an initial location. Unless the initial observations satisfy a switching condition such tha...

Claims

1. A computer-implemented method for parsing an agent's environment in a multi-dimensional space, the method comprising: obtaining first sensor data at a first location of the agent, wherein the first sensor data describes an environment surrounding the agent; retrieving stored second sensor data for comparison with the first sensor data, wherein the second sensor data describes an environment surrounding a second location indicative of a feature in the multi-dimensional space; obtaining a plurality of first sub-regions of the first sensor data, each first sub-region describing a respective first portion of the environment around the agent at the first position, each first portion being associated with a respective first direction from the first position; obtaining a plurality of second sub-regions of the second sensor data, each second sub-region describing a respective second portion of the environment around the second location, each second portion being associated with a respective second direction from the second location; and for each second sub-region: comparing the second sub-region to each first sub-region using a similarity comparison metric to determine a first sub-region that is most similar to the second sub-region; determining a relative rotation between the second direction associated with the second sub-region and the first direction associated with the most similar first sub-region; The method further comprises: The relative rotations for the plurality of second sub-regions are aggregated to obtain a motion vector indicating an estimated direction from the first position of the agent to the second position indicating the feature of the multi-dimensional space.

2. The method of claim 1 , wherein the first sensor data and the second sensor data are first image data and second image data, respectively, wherein each first sub-region in the first sub-regions comprises a continuous subset of pixels of the first image data, and wherein each second sub-region in the second sub-regions comprises a continuous subset of pixels of the second image data.

3. A method according to any preceding claim, wherein the first sensor data and the second sensor data are arranged in a first vector and a second vector, the first vector and the second vector being concatenated from the first image data and the second image data respectively.

4. A method according to any preceding claim, wherein the similarity measure is an inner product between the first sub-region and the second sub-region.

5. The method of any preceding claim, wherein obtaining the first sensor data comprises: obtaining raw first sensor data of the environment of the agent at the first location; processing the raw first sensor data to reduce a size of the raw first sensor data to obtain the first sensor data, such that the first sensor data has a reduced-size format relative to the raw first sensor data; wherein the stored second sensor data is also stored in the reduced size format, so that the method further comprises: obtaining raw second sensor data of the environment of the agent at the second location; as well as The raw second sensor data is processed to reduce a size of the raw second sensor data to obtain the second sensor data, such that the second sensor data has a reduced-size format relative to the raw second sensor data.

6. The method of claim 5, wherein processing the raw first sensor data and the raw second sensor data comprises applying one or more filters and / or masks to the raw first sensor data and the raw second sensor data to reduce the size of each dimension of the raw first sensor data and the raw second sensor data, respectively. 7 . The method according to claim 5 , wherein the raw first sensor data and the raw second sensor data are raw first image data and raw second image data, respectively.

8. The method of any preceding claim, wherein obtaining a plurality of first sub-regions of the first sensor data comprises: iteratively applying a mask to the first sensor data to extract each first sub-region of the plurality of first sub-regions, wherein at each iteration, the mask or the first sensor data is permuted by at least one data entry / cell in one dimension of the first sensor data; and wherein obtaining a plurality of second sub-regions of the second sensor data comprises: A mask is iteratively applied to the second sensor data to extract each second sub-region of the plurality of second sub-regions, wherein at each iteration the mask or the second sensor data is permuted by at least one data entry / cell in one dimension of the second sensor data.

9. The method of claim 8, wherein there are at least four iterations, such that there are at least four first sub-regions and at least four second sub-regions.

10. The method of claim 8 or 9, wherein the mask is smaller than the first sensor data and the second sensor data, wherein the mask comprises a major dimension that is: equal to or less than 50% of the size of the corresponding major dimension of the first sensor data and the second sensor data; or Equal to or less than 25% of the size of the corresponding major dimensions of the first sensor data and the second sensor data.

11. The method according to any one of claims 8 to 10, wherein the first sensor data and the second sensor data are arranged in a first array and a second array, respectively, wherein the first array and the second array have dimensions of X×Y units, and the mask has dimensions of (Xm)×Y units, wherein m is a positive integer.

12. The method according to any preceding claim, further comprising: The agent is moved from the first position according to the motion vector.

13. The method of claim 12, further comprising, after moving the agent to a new position according to the motion vector: obtaining third sensor data at the new position of the agent, wherein the third sensor data describes an environment surrounding the agent; obtaining a plurality of third sub-regions of the third sensor data, each third sub-region describing a respective third portion of the environment around the agent at the third position, each third portion being associated with a respective third direction from the third position; and for each second sub-region: comparing the second sub-region with each third sub-region using the similarity comparison metric to determine a third sub-region that is most similar to the second sub-region; determining a relative rotation between the second direction associated with the second sub-region and the third direction associated with the most similar third sub-region; the method further comprising: Aggregating the relative rotations for the plurality of second sub-regions to obtain an updated motion vector, wherein the updated motion vector indicates an estimated direction from the third position of the agent to the second position, wherein the second position indicates the feature of the multi-dimensional space.

14. The method according to any preceding claim, further comprising: The presence of the feature of the environment in the multi-dimensional space is confirmed if an observation comparison metric matches or exceeds a confirmation threshold level, wherein the observation comparison metric is associated with the similarity metric and / or the action vector.

15. A method according to any preceding claim, wherein determining the relative rotation comprises determining an offset angle between the first direction and the second direction.

16. The method according to claim 15, wherein the method further comprises: For the second relative sub-region pair, the magnitude of the motion vector is determined by: determining an average offset angle based on the offset angles associated with the opposing pair of second sub-regions; and The magnitude is allocated based on the average deviation angle, wherein the magnitude of the average deviation angle is proportional to the magnitude.

17. The method of claim 16, further comprising determining a respective magnitude of each pair of offset angles in a plurality of pairs of offset angles associated with opposing pairs of second sub-regions, and aggregating / averaging the respective magnitudes to form the magnitude of the motion vector.

18. A method according to any preceding claim, wherein the first sensor data represents a substantially 360 degree view around the agent, and wherein the second sensor data represents a substantially 360 degree view around the second location, wherein each first portion described by each first sub-region is part of the 360 ​​degree view around the agent, and wherein each second portion described by each second sub-region is part of the 360 ​​degree view around the second location.

19. The method according to any preceding claim, wherein each first sub-region of the plurality of first sub-regions overlaps with at least one adjacent first sub-region, and each second sub-region of the plurality of second sub-regions overlaps with at least one adjacent second sub-region.

20. A method according to any preceding claim, wherein the characteristic of the environment is one of: a specific location in the multidimensional space; An image or a portion thereof; or an object or a portion thereof.

21. The method according to any preceding claim, wherein the agent is virtual, wherein the multi-dimensional space is a two-dimensional virtual space or a three-dimensional virtual space, and wherein the first sensor data and the second sensor data are obtained using virtual sensors.

22. The method according to any one of claims 1 to 20, wherein the agent is a physical entity, wherein the multi-dimensional space is a three-dimensional real physical space, and wherein the first sensor data is obtained using a physical sensor.

23. The method of claim 22, wherein the second sensor data describes an environment indicative of a target location within the multi-dimensional space, such that the feature of the environment described by the second sensor data is associated with the target location, the method further comprising: The agent is navigated from the first position to the target position according to the action vector.

24. The method of claim 23, wherein the second sensor data forms part of a second sensor data set, the set comprising a plurality of instances of the second sensor data, each instance describing an environment indicative of a corresponding location in the multidimensional space, the method further comprising: Iteratively navigating the agent from the first location to the target location of the corresponding locations indicated by the plurality of instances of second sensor data, wherein in each iteration: obtaining a motion vector for one of the instances of the second data, and moving from the position of the agent to the corresponding position indicated by the one instance of the second data according to the motion vector; until reaching the target position.

25. The method according to any one of claims 23 or 24, wherein navigating the agent from the first position to the target position according to the action vector forms a main navigation process, and wherein the method further comprises: navigating the agent from an initial position to the first position in the multidimensional space using an assisted navigation process; as well as Switching from the secondary navigation process to the primary navigation process at or in a neighborhood of the first location.

26. The method of claim 25, wherein the assisted navigation process is configured to use a positioning system.

27. The method of claim 26, wherein the positioning system is a satellite-based radio navigation system.

28. A method according to any preceding claim, wherein the second sensor data is stored in a remotely accessible database.

29. The method according to claim 28, further comprising: recording, by a recording agent, the second sensor data at the second location; as well as The second sensor data is stored in the remotely accessible database.

30. The method according to claim 29, further comprising: recording metadata corresponding to the second sensor data and associating the metadata with the second sensor data, the metadata comprising at least one of: metric information about the second position in the multidimensional space; information regarding variability of one or more portions of the environment surrounding the second location captured in the second sensor data; information about one or more fixed features of the environment surrounding the second location captured in the second sensor data; temporal information regarding said capture of said second sensor data; an availability metric indicative of a predicted or predetermined availability of the signal at the second location; and information about a specific feature captured in the second sensor data; The method further comprises: The metadata is stored in the remotely accessible database along with the corresponding second sensor data.

31. The method of claim 30, wherein the metadata includes the metric information about the second position in the multidimensional space, the method further comprising: Obtaining metric information about the first position of the agent in the multidimensional space; determining, based on the metric information about the first position and the metric information about the second position, that the second position is located in the neighborhood of the first position, and Based on the determining that the second location is within the proximity of the first location, the second sensor data is retrieved from the remotely accessible database.

32. A method according to claim 30 or 31, when dependent on claim 5, wherein the metadata comprises the information about the variability of one or more parts of the environment around the second location captured in the raw second sensor data; wherein processing the raw second sensor data to reduce the size of each dimension of the raw second sensor data by applying the one or more filters and / or masks to the raw second sensor data comprises: Based on the metadata, one or more portions of the environment surrounding the second location captured in the raw second sensor data are filtered out or masked out so that the second sensor data in the reduced format does not include the one or more portions of the environment indicated by the information of the metadata.

33. A method according to any one of claims 30 to 32, when dependent on claim 5, wherein the metadata comprises the information about one or more fixed features of the environment around the second location captured in the raw second sensor data; wherein processing the raw second sensor data to reduce the size of each dimension of the raw second sensor data by applying the one or more filters and / or masks to the raw second sensor data comprises: Based on the metadata, the one or more fixed features of the environment around the second location captured in the original second sensor data are maintained so that the second sensor data in the reduced format includes the one or more fixed features of the environment indicated by the information of the metadata.

34. A method according to any one of claims 30 to 33, wherein the metadata comprises the availability metric indicative of a predicted or predetermined availability of a positioning system signal and / or a data connection signal at the second location; The method further comprises: According to the positioning system signal, navigating the agent based on the availability metric, and / or: Based on the data connection signal, determining when to download information from the remotely accessible database is performed based on the availability metric.

35. The method according to any one of claims 28 to 34, further comprising: The second location for recording the second sensor data is selected by the recording agent, the selection being based on a location selection criterion, the location selection criterion comprising at least one of: an exploration state associated with the second location; and A determination is made as to whether the second location is accessible by the agent.

36. A system, comprising: processor; Memory; and a sensor or virtual sensor configured to capture spatial data describing an environment of a multi-dimensional space in a local vicinity of the sensor or virtual sensor; The memory has instructions stored thereon, which when executed by the processor cause the system to perform the method according to any one of claims 1 to 35.

37. The system of claim 36, wherein the multidimensional space is a physical space, the system is a robot or a vehicle comprising the sensor, wherein the robot or the vehicle is the agent, and further comprising: A controllable movement module is configured to move the robot or vehicle within the physical space.

38. The system of claim 37, wherein the sensor comprises one or more of: an ultraviolet imaging device, a camera, a LIDAR sensor, an infrared sensor, a radar, a tactile sensor, or other sensor configured to provide spatial information.

39. The system of any one of claims 36 to 38, wherein the system comprises a plurality of devices, the plurality of devices comprising the robot or vehicle and an additional computing device having a processor, a memory, and a sensor; Wherein the additional computing device is the recording agent, the system is configured to perform the method according to any one of claims 23 to 30.

40. The system of claim 36, wherein the system is a computer system including the virtual sensor, wherein the computer system is configured to operate on a virtual space, wherein the agent is represented by a point in the virtual space.

41. A computer program stored on a non-transitory computer readable medium, which, when executed by a processor, is configured to cause the processor to perform the method according to any one of claims 1 to 35.

42. A computer-implemented method of parsing an environment in a multi-dimensional space, the method comprising: recording, by a recording agent in the multidimensional space, sensor data describing an environment surrounding the recording agent; processing the sensor data to identify one or more features present in the environment captured in the sensor data; Associating metadata with the identified one or more characteristics, wherein the characteristics include at least one of: the position of the recording agent when the sensor data is captured in the multi-dimensional space; said recording variability of one or more portions of said environment surrounding said agent; as well as The recording of one or more fixed features of the environment surrounding the agent; The method further comprises: The sensor data and the associated metadata are stored in a remotely accessible database.

43. The method of claim 42, further comprising: retrieving, by a navigation agent in the multi-dimensional space, the stored sensor data and the associated metadata from the remotely accessible database; and navigating the multi-dimensional space using the sensor data and the associated metadata by comparing sensor data captured by the navigation agent with stored sensor data.

44. A method according to claim 43, wherein using the sensor data and the associated metadata to navigate the multidimensional space also includes reducing the stored sensor data based on the associated metadata, wherein the reduced stored sensor data includes sensor data corresponding to non-variable or fixed features identified in the environment of the recorded intelligent agent.

45. The method according to any one of claims 42 to 44, wherein the method further comprises: Prior to storing the sensor data in the remotely accessible database, reducing the sensor data based on the associated metadata such that storing the sensor data comprises: Sensor data corresponding to non-variable or fixed features identified in the environment is stored, and sensor data corresponding to variable or non-fixed features identified in the environment is discarded.

46. ​​A method according to any one of claims 42 to 45, wherein the sensor data forms a panoramic view of the entire surrounding of the recording.

47. A system, the system comprising a first computing device and a server; the first computing device comprising: processor; Memory; and a sensor or virtual sensor configured to capture spatial data describing an environment of a multi-dimensional space in a local vicinity of the sensor or virtual sensor; The server comprises: A memory having a remotely accessible database stored therein; The system is configured to perform the method according to claim 42.

48. The system of claim 47, further comprising a robot or a vehicle or a virtual robot, wherein the robot or the vehicle or the virtual robot comprises: processor; Memory; and a sensor or virtual sensor configured to capture spatial data describing an environment of a multi-dimensional space in a local vicinity of the sensor or virtual sensor; Wherein the robot or vehicle or virtual robot is a navigation agent, when dependent on claim 43, the system is configured to perform a method according to any one of claims 43 to 46.

Images