Semantic mapping for localization and navigation in off-road environment

Abstract

A mapping method is configured to perform the following: extracting, through analysis of a global semantic map, pathways traversable by a vehicle; processing data acquired during motion of said vehicle and estimating a trajectory thereof; selecting pathways to which the estimated trajectory is fitting; processing imagery data acquired during motion of said vehicle and generating a local semantic map indicative of scenery components and properties determined for each of the identified scenery components; mapping said local semantic map onto said global semantic map; generating hypotheses concerning possible locations of the vehicle on the semantic map along one of the pathways based on approximated matches between the estimated possible path and said plurality of traversable pathways extracted from the semantic map; and determining from said plurality of hypotheses a most likely location of the vehicle on the semantic map along one of the pathways based on one or more likelihood functions.

Description

[0001] SEMANTIC MAPPING FOR LOCALIZATION AND NAVIGATION IN OFF-ROAD

[0002] ENVIRONMENT

[0003] RELATED APPLICATIONS

[0004] This application claims the benefit of priority to Israeli Patent Application No. 311,373, filed March 10, 2024, entitled “Semantic Mapping for Localization and Navigation,” the contents of which are hereby incorporated by reference as if fully set forth herein.

[0005] TECHNOLOGICAL FIELD

[0006] The present invention is generally in the field of sensor-based localization and navigation.

[0007] BACKGROUND

[0008] Most localization techniques employed nowadays require external infrastructures and / or signalling e.g., global navigation satellite systems (GNSS, such as global positioning systems- GPS), cellular networks triangulation, time-difference of arrival (TDOA) navigation, WiFi, etc. Standalone (z.e., independent) sensor-based localization techniques primarily rely on onboard sensing equipment, such as cameras, accelerometers, gyroscopes, magnetometers, or suchlike, that can be used to estimate the user / vehicle's relative position with respect to its initial starting point and / or reference frames / objects in the user / vehicle's surroundings. Such sensor-based localization techniques are particularly useful in situations of unreliable and / or unavailable perception of external signals e.g., indoor environments, dense urban areas, remote rural or uninhabited locations and technical terrains.

[0009] Visual simultaneous localization and mapping (vSLAM) is often used in sensor-based localization techniques to calculate the position and orientation of the user / vehicle's camera with respect to its surroundings, while simultaneously mapping the environment. vSLAM implementations can be used to generate simultaneous localization and maps (SLAM maps) that are local relative relationship maps that usually do not contain world coordinates information. Since SLAM maps are designed for relative positioning information, they are not suitable for accurate determination of user / vehicle's real-world position / coordinates.

[0010] Some sensor-based solutions known from patent literature are briefly described hereinbelow. International patent publication No. W02020 / 042349 [1] discloses a positioning initialization method applied to vehicle positioning and a vehicle-mounted terminal. The method comprises constructing a surrounding environment of a vehicle by means of a target image photographed by an image collection device, so as to obtain a local map; matching the local map with a pre-constructed global map to obtain a position of the local map in the global map; and on the basis of the position of the local map in the global map, mapping the position of the vehicle in the local map into the global map to obtain an initial position of the vehicle in the global map. According to the technical solutions, the initial position of the vehicle in the global map can be determined by utilizing image data photographed by the image collection device under the condition that position prior information such as a GPS signal misses, so that initialization of vehicle positioning is completed.

[0011] Chinese patent publication No. CN114111817 [2] discloses a vehicle positioning method and system based on matching of a SLAM map and a high-precision map, and the method comprises the steps: constructing a visual SLAM map in real time, carrying out the semantic segmentation through a deep learning technology, extracting element information in a scene, carrying out the registration of a point set composed of SLAM map elements with a same-type map element point set in the high-precision map, and carrying out the positioning of a vehicle.

[0012] In addition, systems are known for correlating images obtained by on-board sensors with known semantic maps. US patent publication No. 2019 / 0271554 [3] discloses a navigation system for autonomous vehicles. The vehicles are pre-loaded with high-definition maps including semantic data describing objects on the route on which the vehicle is driving. The vehicles also include capabilities for localizing the vehicle along the route. When the vehicle drives along the route, the vehicle captures sensor data and creates semantic labels for objects that are visible on the route. The vehicle computing system then generates a combined image by overlaying current sensor data with a rendered image of the scene. When the localization is sufficiently accurate, this technique may be used to create a rendered image that “filters” certain objects from view, thus enabling the on-board computer to make more informed navigational choices.

[0013] Also, systems are known for utilizing probabilities to determine a route on which a vehicle is traveling, among known routes in a database. U.S. Patent Publication No. 2020 / 0088526 [4] describes a method of determining a path of travel of a vehicle that is traveling in a region having mapped roads. During travel of the vehicle, the vehicle receives data from on-board sensors. The vehicle computer also has access to a database having on- board map data. Starting with the assumption that the vehicle is traveling on the road network, the system determines a plurality of different paths of travel on which the vehicle may be traveling. The system then compares data received from one or more on-board sensors on the vehicle (e.g., GNSS receiver or camera) with predetermined map data (e.g. digital map, street level and aerial imagery, road coordinates, speed limits, road restrictions, etc.). The system applies a Bayesian filter to determine the likelihood of the vehicle being located on a respective path of travel and a probability distribution of locations along this path of travel.

[0014] The systems and methods described in the last two examples are enabled by the presence of detailed databases having preexisting marked routes. In the case of publication [3], the specific route of the vehicle is also known in advance, so that the semantic matching is used only to filter out objects from view in order to enable autonomous driving. In the case of publication [4], the route of the vehicle is selected from among known paths of travel. These examples do not provide any guidance for determining a route of a vehicle on unmapped terrain.

[0015] PUBLICATIONS

[0016] [1] International Publication No. WO 2020 / 042349

[0017] [2] Chinese Patent Application No. CN114111817

[0018] [3] US patent publication No. 2019 / 0271554

[0019] [4] US Patent Publication No. 2020 / 0088526.

[0020] GENERAL DESCRIPTION

[0021] The present disclosure introduces systems and methods for determining a route of a vehicle traveling in an off-road environment, in which roads have not previously been mapped. The system utilizes preexisting digital semantic maps, in conjunction with semantic processing of images captured by an on-board 3D camera, to both determine potential routes within the off-road environment and to correlate the movement of the vehicle with one of the potential routes.

[0022] In specific examples, sensor-based localization and navigation techniques are disclosed wherein a three-dimensional (3D) camera e.g., stereo camera, is used together with a digital compass device and inertial measurement unit (IMU), to continuously determine a user / vehicle's location on a digital semantic map. The localization techniques disclosed herein are particularly useful for technical terrains, and most effectively exploited when a well-defined starting point is available e.g., by an onboard / offboard GPS system or entered as a waypoint manually extracted from a map. Embodiments hereof thus rely on an initialization step in which the initial location of the user / vehicle is acquired or estimated once e.g., by a standard positioning system (e.g., GPS, TDOA), or manually inputted (e.g., waypoint extracted from a map) from the user via a user interface (UI) device.

[0023] The initialization step can further include acquiring a global semantic map from an onboard (or offboard) repository of maps e.g., based on the acquired / estimated starting point. The initialization step can be further configured to acquire and process initial compass and IMU data to determine initial orientation and / or direction of movement of the user / vehicle. Optionally, but in some embodiments preferably, the global semantic map is processed and analyzed e.g., by image processing and / or artificial intelligence (Al) / deep learning (DL) tools, to identify therein and extract therefrom a plurality of traversable pathways, such as trails and roads, that are likely to be traversed by the user / vehicle.

[0024] The acquired starting point and the determined initial orientation and / or direction of movement are then used to commence an iterative localization process of determining location of the user / vehicle with respect to the pathways extracted from the global semantic map based on compass and IMU data, and imagery data from the 3D camera, newly acquired during motion of the user / vehicle. This iterative localization process can be continuously performed throughout a navigation session, or until the user / vehicle's motion is stopped and / or a desired target location is reached.

[0025] In each iteration of the localization process, new imagery data acquired by the 3D camera is processed and analyzed to predict possible directions and / or locations headed by the user / vehicle based on matches between properties of scenery components detected in the new imagery data and of attributes of semantic objects of the global semantic map. New compass and IMU data is also acquired to estimate possible trajectories being traversed by the user / vehicle during the respective iteration. A likelihood measure can be computed to each of the predicted possible directions and / or locations of the user / vehicle based on a level of correlation between properties of the scenery components that were captured in the new imagery data, that are associated with the predicted possible directions and / or locations, and attributes of semantic objects associated with pathways identified in the global semantic map that match with the estimated possible trajectories. A current location / direction / pathway of the user / vehicle can be then determined based on the computed likelihood measures.

[0026] A semantic segmentation process can be utilized to bound and partition scenery components (e.g., road / trail segments, trees, bushes, rocks, ground contour lines, buildings, and suchlike) captured in the imagery data from the 3D camera, and generate respective segmented imagery data therefrom. Image (e.g., AI / DL) recognition can be further used to detect and classify the scenery components in the segmented imagery data and generate sematic imagery data including classifications of the segmented scenery components captured therein. The semantic imagery data, and / or the imagery data from the 3D camera, can be processed and analyzed to generate therefrom vSLAM maps indicative of the scenery components acquired by the 3D camera and of position and orientation of the 3D camera of the user / vehicle's with respect to the acquired scenery components.

[0027] A distance of each of the scenery components detected in the vSLAM maps can be determined utilizing a depth estimation procedure based on properties of the 3D camera. For example, in some embodiments, the 3D camera used is a type of stereo camera and a depth estimation procedure (e.g., cascaded recurrent network with adaptive correlation) is used to determine the distance between the user / vehicle and each of the scenery components detected in the imagery data. The azimuth of each of the acquired scenery components with respect to the 3D camera can be determined from the vSLAM maps. The information determined for each scenery component from the vSLAM maps and the depth estimation procedure can be combined into semantic imagery data.

[0028] The possible trajectories being traversed can be estimated based on orientation and / or direction of movement of the user / vehicle as reflected by the acquired IMU and compass data. Correlation of the estimated possible trajectories with the traversable pathways extracted from the global semantic map can be examined to narrow down the plurality of traversable pathways extracted from the global semantic map into a small number of suitable pathways along which the user / vehicle is likely to be located, and thereby enable removal of predicted possible locations and / or directions that are associated with semantic objects not associated with the pathways found as suitable for the localization. The likelihood measure can be thus assigned to each of the remaining predicted locations and / or directions based on level of correlation between the properties of the scenery components associated therewith, as indicated in the semantic imagery data, and attributes of semantic objects associated with the pathways identified in the global semantic map and found suitable for the localization.

[0029] The prediction includes in some embodiments generating a plurality of hypotheses, each mapping the user / vehicle onto the global map based on a match found between properties (e.g., landmark and / or azimuth and / or distance with respect to the 3D camera) of at least one of the scenery components captured in the scenery data and attributes (e.g., landmark and / or azimuth and / or distance with respect to a location along one of the pathways) of semantic objects identified in the global map. The matching between the scenery components and the semantic objects is at least partially based in some embodiments on a measure of correlation between azimuth and / or distance associated with them. Particularly, the global semantic map (or some portion thereof) can be searched for semantic objects therein having attributes that match properties of scenery components identified in the imagery data, and which azimuth and / or distance with respect to locations on the pathways identified in the global semantic map is in agreeable correlation with the azimuth and / or distance of the scenery component with respect to the 3D camera of the user / vehicle.

[0030] The possible trajectories being traversed can be matched with corresponding segments of the pathways identified in the global semantic map, to thereby narrow down the number of possible pathways of the global semantic map that are likely to be traversed by the user / vehicle. A probability can be computed to each hypothesis indicating the likelihood of the hypothesis being indicative of the user / vehicle position e.g., based on the level of correlation between the properties of the scenery components and the attributes of its matching semantic object. The hypothesis of higher probabilities can be examined against the matching pathways of the global sematic map to determine a most likely location of the user / vehicle along one of the pathways based thereon.

[0031] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. Though suitable methods and equipment are described hereinbelow for implementing the embodiments disclosed, similar methods and / or equipment can be used in practice. In case of conflict, the specification, including definitions, will control. The equipment, methods, and examples disclosed herein are thus illustrative only and not intended to be limiting.

[0032] In one aspect there is provided a computer-implemented mapping method utilizing one or more processors configured to execute instructions stored in a memory to perform the following: identifying in a global semantic map or an extracted portion thereof one or more pathways traversable by a user / vehicle; processing compass and / or inertial measurement unit (IMU) data / signal acquired during motion of the user / vehicle and estimating based thereon a trajectory thereof; selecting from the determined traversable pathways one or more pathways to which the estimated trajectory is at least partially fitting; processing imagery data acquired by a 3D camera during motion of the user / vehicle and generating a local semantic map indicative of one or more scenery components identified in said imagery data and one or more properties determined for each one of the identified scenery components; mapping the local semantic map onto the global semantic map by matching one or more properties of at least one scenery component of the local semantic map to one or more attributes of at least one semantic object of the global semantic map, when the at least one semantic object is associated with at least some portion of at least one of the traversable pathways to which the estimated trajectory is at least partially fitting; generating a plurality of hypotheses concerning possible locations of the user / vehicle on the semantic map along one of the pathways based on approximated matches between the estimated possible path and said plurality of traversable pathways extracted from the semantic map; and determining from said plurality of hypotheses a most likely location of the user / vehicle on the semantic map along one of the pathways based on one or more likelihood functions.

[0033] The processing of the imagery data can comprise one or more of the following: applying a semantic segmentation process to the imagery data to bound / partition scenery components thereby captured and generating a semantic segmented image indicative thereof; applying a depth estimation process to the imagery data and / or to the semantic segmented image and determining a depth property for at least one of the scenery components; applying a vSLAM process to the imagery data and / or to the semantic segmented image to determine one or more properties of at least one of the scenery components. When the depth estimation process is used, the process of matching one or more properties of at least one scenery component of the local semantic map to one or more attributes of at least one semantic object of the global semantic map may include, optionally, verifying that azimuth and / or distance of semantic objects with respect to locations on the pathways identified in the global semantic map is in agreeable correlation with the azimuth and / or distance of the scenery component with respect to the 3D camera. In this manner, the matching process ensures that the object that is identified in the local semantic map is properly correlated to its positioning in the global semantic map.The method may comprise in some embodiments identifying the traversable pathways in the global semantic map, and / or the scenery components in the imagery data, by artificial intelligence and / or deep learning processes.

[0034] The method can comprise receiving position data indicative of a navigation starting point of the user / vehicle and fetching a first portion of the global semantic map based on said position data. The selected portion of the semantic map that is chosen is that which is near the starting point of the user vehicle. The method can further comprise extracting an additional portion of the global semantic map based on one or more of the received position data or a position of the user / vehicle in a previously extracted portion of said global semantic map. The method further includes performing the step of determining one or more traversable pathways with respect to the additional portion. Optionally, the method further includes iteratively repeating each of the steps during movement of the user / vehicle, to thereby iteratively update a determination of a most likely location of the user / vehicle along one of the pathways, until the user / vehicle’ s motion is stopped and / or a desired target location is reached. The iterative nature of the analysis helps ensure that a minimal amount of computing power is exerted during each iteration, and further helps improve accuracy of the calculation, because each comparison is performed with the benefit of the analyses performed in earlier stages of the comparison.

[0035] The method may comprise in possible embodiments predicting one or more possible locations of the user / vehicle based on the matching of the one or more properties of at least one scenery component of the local semantic map to the one or more attributes of at least one semantic object of said global semantic map. Additionally, the method can comprise computing a probability for one or more of the scenery components indicative of a measure of correlation of its properties to attributes of one or more of the semantic objects, and mapping the local semantic map to the global semantic map at least partially based on computed probability. Optionally, the mapping of the local semantic map to the global semantic map is carried out utilizing an adaptive particle filter.

[0036] In especially advantageous embodiments, prior to the initial determining step, the traversable pathways are not identified on the global semantic map or extracted portion thereof. The semantic map includes objects (e.g., trees, rocks, buildings, etc.) but not paths. Thus, the vehicle is not traveling on a previously known or identified path. One of the particularly advantageous benefits of the method disclosed herein is that it enables determination of a path on which a vehicle is traveling without any prior information about the paths available to the vehicle.

[0037] The traversable pathways may include one or more of surface roads, dirt paths, tracks, trails, and bridges. The definition of traversable pathway is thus quite expansive, and includes not only paved roads, but any sort of pathway that may be utilized in an off-road environment.

[0038] In another aspect there is provided a localization system comprising: IMU and digital compass units configured to generate IMU and compass data / signals of a moving user / vehicle, a 3D camera configured generate imagery data indicative of scenery components in the user / vehicle' s environment, and a processor configured to determine from analysis of a global semantic map or an extracted portion thereof a plurality of pathways traversable by the user / vehicle, estimate from the IMU and compass data / signals a possible path of the user / vehicle, process imagery data acquired by the 3D camera during motion of said user / vehicle and generate a local semantic map indicative of one or more scenery components identified in said imagery data and one or more properties determined for each one of the identified scenery components; map said local semantic map onto said global semantic map by matching one or more properties of at least one scenery component of the local semantic map to one or more attributes of at least one semantic object of said global semantic map, said at least one semantic object is associated with at least some portion of at least one of the traversable pathways to which said estimated trajectory is at least partially fitting; generate a plurality of hypotheses concerning possible locations of the user / vehicle on the semantic map based on approximated matches between the estimated possible path and the plurality of traversable pathways extracted from the semantic map, and determine from the plurality of hypotheses a most likely location of the user / vehicle on the semantic map based one or more likelihood functions.

[0039] The system can be configured to acquire an initial position of the user / vehicle from a positioning system or from a map.

[0040] The system comprises in some embodiments a segmentation module configured to detect the scenery components in the imagery data from the 3D camera. The system can also comprise a depth estimation module configured to estimate from the imagery data a distance of each of the scenery components from the user / vehicle. In possible embodiments the system comprises a vSlam module configured to estimate orientation of the scenery components with respect to the user / vehicle. In embodiments utilizing depth estimation, the processor may be is configured match one or more properties by verifying that azimuth and / or distance of semantic objects with respect to locations on the pathways identified in the global semantic map is in agreeable correlation with the azimuth and / or distance of the scenery component with respect to the 3D camera.

[0041] The system can comprise one or more Al units configured to determine properties of the scenery components. In a possible application the system comprises at least one processor configured to acquire and at least partially process data / signals from the IMU, compass and 3D camera, and generate the IMU and compass data and the imagery data, and at least another processor configured carry out a mapping process utilizing the IMU and compass data and the imagery data from said at least one processor.

[0042] Functionality of features and component disclosed herein can be carried out utilizing circuitry or processing circuitry including all kinds of processors, integrated circuits (ICs), application specific integrated circuits (ASICs), or field-programmable gated arrays (FPGAs), and / or combinations thereof which are configured or programmed to perform the disclosed functionality. The processors and / or Al machine disclosed herein can be hardware units utilizing processing circuitries including transistors and other circuitry programmed to perform various functionalities disclosed herein.

[0043] The foregoing has outlined rather broadly the features and technical advantages of the present invention in order that the detailed description of the invention that follows may be better understood. Additional features and advantages of the invention will be described hereinafter which form the subject of the claims of the invention. It should be appreciated that the conception and specific embodiments disclosed may be readily utilized as a basis for modifying or designing other structures for carrying out the same purposes of the present invention. It should also be realized that such equivalent constructions do not depart from the invention as set forth in the appended claims.

[0044] The invention further provides a computer-implemented mapping method utilizing one or more processors configured to execute instructions stored in a memory to perform the following: determining, through analysis of a global semantic map or an extracted portion thereof, one or more pathways traversable by a user / vehicle; processing compass and / or inertial measurement unit (IMU) data / signal acquired during motion of said user / vehicle and estimating based thereon a trajectory thereof; selecting from the determined traversable pathways one or more pathways to which the estimated trajectory is at least partially fitting; processing imagery data acquired by a 3D camera during motion of said user / vehicle and generating a local semantic map indicative of one or more scenery components identified in said imagery data and one or more properties determined for each one of the identified sceneries components; mapping said local semantic map onto said global semantic map by matching one or more properties of at least one scenery component of the local semantic map to one or more attributes of at least one semantic object of said global semantic map, said at least one semantic object is associated with at least some portion of at least one of the traversable pathways to which said estimated trajectory is at least partially fitting; generating a plurality of hypotheses concerning possible locations of the user / vehicle on the semantic map along one of the pathways based on approximated matches between the estimated possible path and said plurality of traversable pathways extracted from the semantic map; and determining from said plurality of hypotheses a most likely location of the user / vehicle on the semantic map along one of the pathways based on one or more likelihood functions. In some embodiments of methods of the invention, the processing of the imagery data comprises applying a semantic segmentation process to the imagery data to bound / partition scenery components thereby captured and generating a semantic segmented image indicative thereof.

[0045] In some embodiments of methods of the invention, the processing of the imagery data comprises applying a depth estimation process to the imagery data and / or to the semantic segmented image to determine a depth property for at least one of the scenery components, and wherein the step of matching one or more properties comprises verifying that azimuth and / or distance of semantic objects with respect to locations on the pathways identified in the global semantic map is in agreeable correlation with the azimuth and / or distance of the scenery component with respect to the 3D camera.

[0046] In some embodiments of methods of the invention, the processing of the imagery data comprises applying a vSLAM process to the imagery data and / or to the semantic segmented image to determine one or more properties of at least one of the scenery components.

[0047] In some embodiments of methods of the invention, the method comprising identifying the traversable pathways in the global semantic map, and / or the scenery components in the imagery data, by artificial intelligence and / or deep learning processes.

[0048] In some embodiments of methods of the invention, the method comprising receiving position data indicative of a navigation starting point of the user / vehicle, fetching a first portion of the global semantic map based on said position data, and determining the one or more traversable pathways with respect to said first portion.

[0049] In some embodiments of methods of the invention, the method comprising extracting an additional portion of the global semantic map based on one or more of the received position data or a position of the user / vehicle in a previously extracted portion of said global semantic map, and further performing the step of determining one or more traversable pathways with respect to said additional portion.

[0050] In some embodiments of methods of the invention, the method comprising iteratively repeating each of the steps during movement of the user / vehicle, to thereby iteratively update a determination of a most likely location of the user / vehicle along one of the pathways, until the user / vehicle’ s motion is stopped and / or a desired target location is reached.

[0051] In some embodiments of methods of the invention, the method comprising predicting one or more possible locations of the user / vehicle based on the matching of the one or more properties of at least one scenery component of the local semantic map to the one or more attributes of at least one semantic object of said global semantic map. In some embodiments of methods of the invention, the method comprising computing a probability to one or more of the scenery components indicative of a measure of correlation of its properties to attributes of one or more of the semantic objects, and mapping the local semantic map to the global semantic map at least partially based on computed probability.

[0052] In some embodiments of methods of the invention, the mapping of the local semantic map to the global semantic map is carried out utilizing an adaptive particle filter.

[0053] In some embodiments of methods of the invention, prior to the initial determining step, the traversable pathways are not identified on the global semantic map or extracted portion thereof.

[0054] In some embodiments of methods of the invention, the traversable pathways comprise one or more of surface roads, dirt paths, tracks, trails, and bridges.

[0055] Also provided is a localization system comprising: IMU and digital compass units configured to generate IMU and compass data / signals of a moving user / vehicle, a 3D camera configured generate imagery data indicative of scenery components in the user / vehicle' s environment, and a processor configured to determine from analysis of a global semantic map or an extracted portion thereof a plurality of pathways traversable by the user / vehicle, estimate from the IMU and compass data / signals a possible path of the user / vehicle, process imagery data acquired by the 3D camera during motion of said user / vehicle and generate a local semantic map indicative of one or more scenery components identified in said imagery data and one or more properties determined for each one of the identified scenery components; map said local semantic map onto said global semantic map by matching one or more properties of at least one scenery component of the local semantic map to one or more attributes of at least one semantic object of said global semantic map, said at least one semantic object is associated with at least some portion of at least one of the traversable pathways to which said estimated trajectory is at least partially fitting; generate a plurality of hypotheses concerning possible locations of the user / vehicle on the semantic map based on approximated matches between the estimated possible path and said plurality of traversable pathways extracted from the semantic map, and determine from said plurality of hypotheses a most likely location of the user / vehicle on the semantic map based one or more likelihood functions.

[0056] In some embodiments of systems of the invention, the system is configured to acquire an initial position of the user / vehicle from a positioning system or from a map.

[0057] In some embodiments of systems of the invention, the system comprising a segmentation module configured to detect the scenery components in the imagery data from the 3D camera. In some embodiments of systems of the invention, the system comprising a depth estimation module configured to estimate from the imagery data a distance of each of the scenery components from the user / vehicle.

[0058] In some embodiments of systems of the invention, the processor is configured match one or more properties by verifying that azimuth and / or distance of semantic objects with respect to locations on the pathways identified in the global semantic map is in agreeable correlation with the azimuth and / or distance of the scenery component with respect to the 3D camera.

[0059] In some embodiments of systems of the invention, the system comprising a vSlam module configured to estimate orientation of the scenery components with respect to the user / vehicle.

[0060] In some embodiments of systems of the invention, the system comprising one or more Al units configured to determine properties of the scenery components.

[0061] In some embodiments of systems of the invention, the system comprising at least one processor configured to acquire and at least partially process data / signals from the IMU, compass and 3D camera, and generate the IMU and compass data and the imagery data, and at least another processor configured carry out a mapping process utilizing the IMU and compass data and the imagery data from said at least one processor.

[0062] BRIEF DESCRIPTION OF THE DRAWINGS

[0063] In order to understand the invention and to see how it may be carried out in practice, embodiments will now be described, by way of non-limiting example only, with reference to the accompanying drawings. Features shown in the drawings are meant to be illustrative of only some embodiments of the invention, unless otherwise implicitly indicated. In the drawings same reference signs are used to indicate members (configural elements) having identical or corresponding functions and / or structures, and in which:

[0064] Figs. 1A and IB are block diagrams schematically illustrating a localization / navigation system according to possible embodiments, wherein Fig. 1A demonstrates a general implementation of the localization / navigation system, and Fig. IB demonstrates an implementation of the localization / navigation system specifically designed for a vehicle;

[0065] Fig. 2A and Fig. 2B are block diagrams schematically illustrating localization processes according to possible embodiments, wherein Fig. 2A demonstrates use of local mapping in a global mapping process and Fig. 2B demonstrates an iterative mapping process; Figs. 3A to 31 schematically illustrate initialization and prediction steps according to possible embodiments, wherein Fig. 3A is a functional flow chart, Fig. 3B exemplifies the semantic segmentation, Fig. 3C shows a color image (depicted in grayscale) of the segmented scenery, Fig. 3D exemplifies trajectory estimation, Fig. 3E depicts a global map, Fig. 3F demonstrates identification and extraction of pathways from the global map, Fig. 3G demonstrates hypothesis sampling, Fig. 3H demonstrates evaluation of sampled hypothesis, and Fig. 31 demonstrates mapping an estimated trajectory onto a pathway of a global map;

[0066] Figs. 4A to 4D exemplify an initial stage of the mapping process according to possible embodiments, wherein Fig. 4A shows an image acquired by a thermal 3D camera, Fig. 4B shows segmentation of the acquired image, Fig. 4C demonstrates trajectory estimation, and Fig. 4D shows a color image (depicted in grayscale) of imaged scenery;

[0067] Figs. 5A to 5E exemplify an advanced stage of the mapping process, wherein Fig. 5A shows an image acquired by the thermal 3D camera, Fig. 5B shows segmentation of the acquired image, Fig. 5C shows a color image (depicted in grayscale) of imaged scenery, Fig. 5D demonstrates trajectory estimation, and Fig. 5E shows the total estimated traversed path with the newly estimated trajectory; and

[0068] Figs. 6A to 6D pictorially exemplify the estimation, update and recovery stages of the mapping process according to possible embodiments, wherein Figs. 6A and 6B respectively demonstrate prediction without and with an anchoring step at beginning stages of the mapping process, Figs. 6C and 6D respectively demonstrate the prediction without and with the anchoring step at advanced stages of the mapping process.

[0069] DETAILED DESCRIPTION OF EMBODIMENTS

[0070] One or more specific and / or alternative embodiments of the present disclosure will be described below with reference to the drawings, which are to be considered in all aspects as illustrative only and not restrictive in any manner. It shall be apparent to one skilled in the art that these embodiments may be practiced without such specific details. In an effort to provide a concise description of these embodiments, not all features or details of an actual implementation are described at length in the specification. Elements illustrated in the drawings are not necessarily to scale, or in correct proportional relationships, which are not critical. Emphasis instead being placed upon clearly illustrating the principles of the invention such that persons skilled in the art will be able to make and use the mapping systems hereof, once they understand the principles of the subject matter disclosed herein. This invention may be provided in other specific forms and embodiments without departing from the essential characteristics described herein.

[0071] A semantic mapper is disclosed that is usable for localization and navigation (generally referred to herein as mapping), particularly in off-road terrain with no previously mapped roads or pathways. The term semantic generally means the provisioning of context to data, and in embodiments hereof to properties associated with scenery components captured in imagery data acquired by a 3D camera. The imagery data can be associated with one or more categories, names, and / or shapes. Semantic objects generally refer to elements of a semantic map, which in some embodiments represent a technical terrain, each semantic object having characterizing attributes (e.g., location, name, landmark, function configuration, lifecycle stage, etc). Semantic ontologies allow more than one category to be assigned to a shape or an object giving it context.

[0072] The term global map as used herein refers to an existing map (as opposed to a local map generated employing a vSLAM process) representing elements of a certain portion of the planet. The term sparse object map as used herein refers to a local map produced by a semantic mapper and that represents scenery components captured in acquired imagery data.

[0073] The semantic mapper of embodiments hereof is configured to receive localization data from a SLAM process together with perceptual data (also referred to herein as scenery components e.g., depth image / map and / or semantic segmentation), and generate therefrom a dense semantic map and / or sparse vector objects map. The SLAM process is applied in embodiments hereof to acquired imagery data of a scenery in a field-of-view (FOV) of a user / vehicle traversing a terrain e.g., a front view observed by driver / passenger of a vehicle.

[0074] The perceptual data is associated in embodiments with scenery components of the acquired imagery data. The scale of the map generated by the semantic mapper can be relatively small (e.g., about 50 to 500 meters). The semantic mapper is configured in some embodiments to produce a probabilistic binary map not including data associations / attributes. In possible embodiments the semantic mapper is configured to produce a large scale map (e.g., in a scale of about 1-10 km, or more) being a vector graph having a plurality of nodes representing junction elements in a global map, and a plurality of (typically curved) edges representing pathway elements (e.g., dirt roads, surface roads, trails, tracks, bridges, and suchlike) substantially extending from the junction elements and / or located / connecting therebetween. The large scale map is readily designed for anchoring an estimated path / location determined from a local map to a junction and / or edge of the large scale map. A semantic matcher in embodiments hereof is configured to receive a sparse object map, such as generated by the semantic mapper, and a digital global map from a digital storage / map repository (e.g., a mapping server) representing preexisting global map data, and match the sparse object map to the global map, or to some portion thereof. This way, localization of the sparse object map on the global preexisting map can be achieved. Landmarks from the global map can be then used for localization and navigation.

[0075] The semantic mapper is configured in some embodiments to generate probabilistic scores to the perceptual data thereby received (z.e., hypotheses) indicative of matching likelihood between the elements in the sparse object map and the global map. A statistical examination process (e.g., utilizing a Bayesian approach) can be applied to the perceptual data to update the probabilistic scores of the perceptual data's hypotheses and based thereon extract features and / or label landmarks. Optionally, the statistical examination process utilizes a “prior” estimate (e.g., an initial distribution / gauss, such a uniform distribution over some portion of the global map) that can be recursively refined as new data is acquired by sensors (z.e., 3D camera, digital compass, and / or IMU) of the system.

[0076] In some embodiments the semantic matcher utilizes an adaptive particle filter that is statistically consistent model based, such as visual inertial navigation system, that uses multiple likelihood functions for anchoring one or more of the hypotheses associated with the perceptual data to at least one of the possible pathways identified in the global map. For example, the adaptive particle filter can be configured to adaptively change the amount of samples acquired by the sensors over time, and thereby alleviate the computation overhead burden of the system. In possible embodiments the amount of samples acquired over time is affected / controlled in accordance with a determined uncertainty measure of the filter e.g., probability density function (PDF).

[0077] The mapping techniques disclosed herein, utilizing semantic mapping and matching, are computationally less burdensome and require less computing resources compared to convolutional neural networks techniques often used nowadays. The disclosed mapping techniques can be advantageously used for localization and navigation of a robot / autonomous vehicle, or an unmanned aerial vehicle (UAV) e.g., a micro aerial vehicle (MAV), during its motion. In addition, accuracy is also improved, and also the rate of converging to a reliable solution to the localization challenge.

[0078] For an overview of several exemplary features, process stages, and principles of the invention, the mapping examples illustrated schematically and diagrammatically in the figures are intended for vehicular navigation. These vehicular navigation systems are shown in one exemplary implementation that demonstrates a number of features, processes, and principles used to realize the localization / mapping, but they are also useful for other applications and can be made in different variations. Therefore, this description will proceed with reference to the shown examples, but with the understanding that the invention recited in the claims below can also be implemented in myriad other ways, once the principles are understood from the descriptions, explanations, and drawings herein. All such variations, as well as any other modifications apparent to one of ordinary skill in the art and useful in localization / mapping applications may be suitably employed, and are intended to fall within the scope of this disclosure.

[0079] Fig. 1A shows a block diagram schematically illustrating a localization and navigation system (also referred to herein mapping system) 10 according to possible embodiments. The mapping system 10 comprises an IMU unit 12, a 3D camera (e.g., a stereo thermal camera) 13, and a digital compass 14, configured for data communication with a computer system (e.g., general-purpose computer) 11. The computer system 11 comprises one or more processors 11c and memories 11m configured to receive measurement data from the compass and IMU units 14,12 and imagery data acquired by the 3D camera 13, and determine location of a user / vehicle based thereon and on an initial navigation starting point (INSP) lln. In some embodiments the 3D camera 13 is implemented by stereo uncooled thermal cameras, such the Boson uncooled infrared camera manufactured by Teledyne FLIR LLC.

[0080] The mapping system 10 optionally comprises a navigation system (e.g., GNSS / GPS and / or TDOA) 16 configured to determine the INSP lln and provide the same to the computer system 11. The mapping system 10 optionally also comprises a user interface device (UI, e.g., keypad / keyboard, pointing device / stick, and / or a display such as a touchscreen) 15, which may be used for manually entering the INSP lln by a user e.g., as a waypoint extracted from a map or simply by pointing it on a map presented to user in the display device. Optionally, but in some embodiments preferably, the UI 15 is part of a vehicle computer system (e.g., powertrain control module - PCM) coupled to the computer system 11 e.g., via OBD2 (on-board diagnostics) port.

[0081] The computer system 11 comprises a communication interface (I / F) Hi configured to communicate data / signals with the IMU 12, 3D camera 13, compass 14, and the optional UI 15 and / or navigation system 16. Optionally, the I / F Hi is further configured to wirelessly (e.g., using radiofrequency satellite and / or cellular communication) communicate with one or more data networks 18 e.g., to fetch one or more digital global maps 11g from a remote maps repository 17. The digital global map 11g is preferably a global semantic map, which may be pre-stored in the memory 11m of the computer system 11, or downloaded from the remote maps repository 17.

[0082] A segmentation module Ils is used in some embodiments in the computer system 11 configured to perform semantic segmentation to the imagery data acquired by the 3D camera 13 and generate respective one or more semantic segmentation images (e.g. , Fig. 3B) indicative of the captured scenery components (e.g., road / trail segments, trees, bushes, rocks, ground contour lines, buildings, and suchlike) and their attributes. One or more Al modules Ila can be used in the computer system 11 to identify the scenery components presented in the semantic segmentation images and update their properties therein accordingly. A depth module lid can be used in the computer system 11 to estimate distances of the scenery components presented in the semantic segmentation images, and update their properties accordingly.

[0083] The computer system 11 can be configured to extract a certain portion of the global semantic map 11g found to be relevant for the mapping process thereby performed e.g., based on the INSP lln. The computer system 11 can be further configured (e.g., using the one or more of the Al modules Ila) to detect in the global semantic map 11g, or in an extracted portion thereof, traversable pathways (e.g., surface roads, dirt roads, tracks, trails, bridges, and suchlike) that may be accessed (traveled) by the user / vehicle during the process.

[0084] The computer system 11 further comprises a vSLAM module llu configured to process the 3D imagery data acquired by the 3D camera 13 and / or respective segmentation image thereof, and generate therefrom respective SLAM maps indicative of the scenery components captured in the 3D imagery, their orientations with respect to the 3D camera, and possible other properties inherent to SLAM maps. The properties of the scenery components can be accordingly updated to include azimuth thereof as determined from the SLAM maps. A prediction module lip can be utilized in the computer system 11 to process the semantic segmentation images and the global map (or an extracted portion thereof), and determine based thereon possible directions and / or locations of the user / vehicle based on matches between properties of one or more of the scenery components captured in the imagery data and attributes of one or more semantic objects from the global map.

[0085] A mapper module llv can be used in the computer system 11 to generate a local map Ho (also referred to herein as a sparse object map) by combining properties determined by the depth module lid, the vSLAM module llu (vSlam odometry), into the semantic segmentation image to create a semantic map of the local environment. A path estimation module Hr can be used in the computer system 11 to process the compass and IMU data from the IMU 12 and compass 13 units, and estimate a possible trajectory of the user / vehicle based thereon. A matcher module lie of the computer system 11 can be configured to match the local map Ila generated by the mapper module llv to the global map 11g, at least partially based on a level of correlation between the estimated possible trajectory and at least some portion of one of the traversable pathways (e.g., 33 in Fig. 3F) identified in the global 11g.

[0086] Fig. IB schematically illustrates a mapping system 10' configured according to possible embodiments for installation in a land vehicle lOv. The mapping system 10' of Fig. IB is generally similar to the mapping system 10 of Fig. 1A. A main difference between the mapping systems 10,10' is in dividing the functionality and operation of the computer system 11 of system 10 between the internal computer system (e.g., mini personal computer - PC) 11" and the external computer system (e.g., mini PC) of system 10', and optional reliance on internal vehicle resources and hardware. For example, the UI unit 15, and / or the optional navigation system (NS e.g., coupled to an antenna 19a of the vehicle lOv) 19 of system 10' can be implemented utilizing internal units of the vehicle lOv.

[0087] As seen, the external computer 11' of the mapping system 10' is coupled to, and configured to operate, the external 3D camera 13, IMU unit 12, and compass unit 14, and can be accordingly mounted external to / on the vehicle lOv. The internal computer 11" is coupled to, and configured to operate, internal units / functionalities of the system e.g., optional UI 15 and / or NS 19.

[0088] The internal computer 11" (and / or external computer 11') can be accordingly coupled for data communication (e.g., using OBD connectivity) with the vehicle's computer system 23 for accessing its internal units. Regardless of whether the mapping system is implemented as exemplified in Fig. 1A or IB, in possible embodiments one or more external Al machines (e.g., such as the Jetson family of products manufacture by Nvidia) Ila' are used instead (or in addition) to the Al modules Ila. It is noted that the mapping system exemplified in Figs. 1A or IB can be similarly installed in aerial vehicle (UAV or MAV).

[0089] Fig. 2A is a functional block diagram of the mapping system 10 / 10' according to possible embodiments, exemplified to include a local mapping portion 21 and a global mapping portion 22. The local mapping portion 21 is configured to generate a local (dense) map 21m indicative of the scenery components captured in the imagery data and properties determined by the system for at least some of scenery components. The global mapping portion 22 is configured to match properties of one or more of the scenery components of the local map 21m to attributes of one or more of the semantic objects of the global map, and generate based thereon a global positioning map 22m indicative of an estimated location of the user / vehicle on the global map (11g). The local mapping portion 21 utilizes a vSLAM unit 27 (and / or vSLAM module 17 of Fig. 1A) configured to receive and process the IMU and compass data signals from the compass unit 14 and the IMU unit 12, and the imagery data from the 3D camera 13i, and generate based thereon local SLAM (sparse) maps 21p indicative of the scenery components captured in the imagery data and / or the estimated trajectory.

[0090] As exemplified in Fig. 2A, the data / signals acquisition rate from the compass and IMU units 14,12 can be substantially (e.g., more than three times) greater than the imagery data / signals acquisition rate from the 3D camera, such that for each local SLAM map 21p and / or local map 21m an estimated trajectory of a certain length can be generated, depending on the velocity of the user / vehicle. Optionally, but in some embodiments preferably, the rate of generation of the local maps 21m is substantially (e.g., about hundred times) smaller than the rate of generation of the local SLAM maps 21p, such that the local map 21m generated by the local mapping portion 21 is significantly denser than the local SLAM (sparse) maps 21p generated by the vSLAM unit 27, with respect to the amount of captured scenery components and their respective properties, dependent on the velocity of the user / vehicle.

[0091] The global mapping portion 22 utilizes in some embodiments a map server 22s configured to (e.g., offline) receive and process a global semantic map 11g, identify traversable pathways thereof, and extract therefrom a bounded global map 22b including one or more of the identified traversable pathways likely to be accessed / traveled by the user / vehicle. For example, the bounded global map 22b can be initially extracted based on the INSP (lln). Optionally, but in some embodiments preferably, the map server 22s is configured to receive and process after each cycle the global positioning map 22m generated by the global mapping portion 22 and determine based thereon which part of the global map 11g to extract for the next cycle e.g., in accordance with the estimated location of the user / vehicle.

[0092] The feature matcher unit 28 (and / or the matcher nodule lie) of the global mapping portion 22 can be configured to receive and process the bounded global map 22b extracted by the map server 22s, and the local (sparse) map 21m generated by the local mapping portion 21, to map as closely as possible the estimated trajectories (Hr) onto traversable pathways identified in the global map 11g and included in the bounded global map 22b. In possible embodiments this is achieved by (e.g., utilizing an adaptive particle filter 28f) matching properties of one or more the scenery components included in the local (sparse) map 21m with attributes of one or more semantic objects included in the bounded global map 22b.

[0093] Fig. 2B is a functional flowchart schematically illustrating stages of an iterative mapping process 29 according to possible embodiments. The process 29 includes an initialization stage 22i, in which the global semantic map 11g and the INSP lln are obtained, and the initial IMU and compass data / signals 12,14 are acquired, and an initial bounded global map 22b is generated. An iterative mapping sequence is thereafter commenced, which includes a prediction stage lip, an update stage llu, and a recovery stage Hr. When the iterative mapping sequence is commenced it utilizes the initial bounded global map 22b generated in the initialization stage 22i, and thereafter a new bounded global map 22b (e.g., generated by the map server 22s) is used in the update stage llu, and thereafter a new bounded global map 22b (e.g., generated by the map server 22s) is used in the recovery stage Hr, of each consecutive iteration of the mapping process.

[0094] In the prediction stage lip the local (dense) map 21m obtained by application of the vSLAM llv process to the acquired imagery and IMU and compass data / signals is processed and analyzed to generate a plurality of hypotheses concerning a respective plurality of possible locations of the user / vehicle determined by matching properties of one or more scenery components of the acquired imagery data to attributes of one or more semantic objects of the bounded global map 22b. The prediction stage lip can be configured to generate probabilistic scores to the scenery components in the imagery data indicative of matching likelihood between the elements in the sparse and global maps.

[0095] In the update stage llu a new bounded global map 22b is received, new compass and IMU data / signals are acquired and a possible trajectory 22e of the user / vehicle is estimated based thereon for the specific iteration, and the local (dense) map 21m obtained in the update stage lip with the plurality of hypotheses is mapped onto the new bounded global map 22b by matching the estimated trajectory 22e to at least one of the traversable pathways identified in the new bounded global map 22b. A statistical examination process (e.g., utilizing a Bayesian approach) can be used to update the probabilistic scores determined in prediction stage lip for the imagery components of the the local (dense) map 21m, which can be used to extract features and / or to label landmarks.

[0096] In the recovery stage Hr, a new bounded global map 22b is received and processed with the updated local (dense) map 21m' comprising the updated probabilistic scores determined in the the update stage llu to determine therefrom, utilizing one or more likelihood functions, the most likely hypothesis indicative of the user’ s / vehicle’s position on the currently obtained bounded global map 22b. Optionally, but in some embodiments preferably, an adaptive particle filter utilizing multiple likelihood functions is used for anchoring one or more of the hypotheses associated with the imagery data to at least one of the possible pathways identified on the currently obtained bounded global map 22b. EXAMPLES

[0097] The present invention will be described below by way of specific examples. The following examples are offered for illustrative purposes only and are not intended to limit the invention in any manner. Those of skill in the art will readily recognize a variety of non-critical parameters that can be changed or modified to yield essentially the same results.

[0098] Figs. 3A to 31 schematically illustrate a mapping process 30 according to possible embodiments. The process 30 can start with receiving and processing IMU (si) and compass (s2) data to estimate (s3) a traversed path / roadmap (trajectory e.g., Fig. 3D) of the user / vehicle. A semantic map (s4 e.g., Fig. 3E) can be simultaneously processed to identify (s5) traversable pathways (e.g., 33 in Fig. 3F) available to the user / vehicle. The estimated traversed path (s3) and the traversable pathways (s5) can be then processed to generate a plurality of hypothesis (e.g., 34 in Fig. 3G) of the user / vehicle location on the global map, by approximating matches (e.g., Fig. 3H) between the estimated traversed path (s3) and the traversable pathways (s5).

[0099] Semantic imagery data (s7 e.g., Fig. 3B) of the user / vehicle's environment can be then used with the semantic map (s4) to determine (s9) for each hypothesis (s6) probabilistic score(s) based on a level of match between scenery components in the semantic imagery data (s7) and semantic objects of the sematic map s4. A color image (s8) corresponding to the semantic imagery data (s7) of the user / vehicle's environment is optionally acquired and displayed. Optionally, the color image (s8) is used, with or without the (e.g., thermal) imagery data (13i) from the 3D camera, by the system for the semantic segmentation (s7). Finally, the probabilistic score(s) (s9) of the hypotheses (s6) are examined (slO) using one or more likelihood functions to determine a most likely hypothesis (e.g., 34 in Fig. 31) indicative of the user’s / vehicle’s position on the sematic global map.

[0100] Figs. 4A to 4D exemplify an initial stage of the mapping process according to possible embodiments, wherein Fig. 4A shows an image acquired by a 3D thermal camera, Fig. 4B shows a segmented image of the image from the 3D thermal camera, Fig. 4C shows an initial path estimation (41) obtained from the IMU and compass data with respect to a starting point (lln) of the process, and Fig. 4D shows an image corresponding to the image from the 3D thermal camera.

[0101] Figs. 5A to 5E exemplify an advanced stage of the mapping process, wherein Fig. 5A shows an image acquired by a 3D thermal camera, Fig. 5B shows a segmented image of the image from the 3D thermal camera, Fig. 5C shows a color image (depicted in grayscale) corresponding to the image from the 3D thermal camera, Fig. 5D shows the path estimation (51) obtained from the IMU and compass data and the last match (52) achieved by the system to a traversable pathway identified in the semantic map, and Fig. 5E shows the mapping results achieved by the system with respect to the starting point (lln) of the process. As seen, in this stage of the process the user / vehicle is not traveling on the pathway / road (53 in Fig. 5C), causing discontinuation between the estimated path (51 in Fig. 5D) and last match (52) with the traversable pathways.

[0102] Figs. 6A to 6D pictorially exemplify the estimation, update and recovery stages of the mapping process according to possible embodiments, wherein Fig. 6A demonstrates an initial stage of the mapping process wherein the actual location (66) of the user / vehicle is substantially in congruence with the plurality of hypotheses 62 generated by the system, Fig. 6B demonstrates the same initial stage of the mapping process carried out without the anchoring step, such that the plurality of hypotheses 62 generated by the system increasingly diverge from the actual location (66) of the user / vehicle. Figs. 6C and 6D demonstrate a more advanced stage of the mapping process without and with the anchoring step, respectively. As seen in Fig. 6C, over time as the location of the user / vehicle becomes farther away from the starting point, the hypotheses (62) generated by the system can significantly depart / diverge from the actual location (66) of the user / vehicle if the anchoring step is skipped.

[0103] The application also provides a computer program and a computer program product for carrying out any of the methods described herein, and a computer readable medium having stored thereon a program for carrying out any of the methods described herein. The application also provides a method substantially as described herein with reference to the accompanying drawings, and apparatus substantially as described herein with reference to and as illustrated in the accompanying drawings.

[0104] It should also be understood that throughout this disclosure, where a process or method is shown or described, the steps / acts of the method may be performed in any order and / or simultaneously, and / or with other steps / acts not illustrated / described herein, unless it is clear from the context that one step depends on another being performed first. In possible embodiments not all of the illustrated / described steps / acts are required to carry out the method.

[0105] Those skilled in the art will understand and appreciate that the depicted methods may alternatively, or additionally, be illustrated as a series of interrelated states via a state diagram and / or events that can be implemented by a state machine. Additionally, or alternatively, the methods disclosed herein can be stored on an article of manufacture e.g., program instructions and / or data stored on storage media and executable by a computer device, to facilitate implement the method by computing devices. As described hereinabove and shown in the associated figures, the present invention provides mapping techniques usable for localization and navigation, and related methods. While particular embodiments of the invention have been described, it will be understood, however, that the invention is not limited thereto, since modifications may be made by those skilled in the art, particularly in light of the foregoing teachings. As will be appreciated by the skilled person, the invention can be carried out in a great variety of ways, employing more than one technique from those described above, all without exceeding the scope of the claims.

Claims

CLAIMS1. A computer-implemented mapping method utilizing one or more processors configured to execute instructions stored in a memory to perform the following: determining, through analysis of a global semantic map or an extracted portion thereof, one or more pathways traversable by a user / vehicle; processing compass and / or inertial measurement unit (IMU) data / signal acquired during motion of said user / vehicle and estimating based thereon a trajectory thereof; selecting from the determined traversable pathways one or more pathways to which the estimated trajectory is at least partially fitting; processing imagery data acquired by a 3D camera during motion of said user / vehicle and generating a local semantic map indicative of one or more scenery components identified in said imagery data and one or more properties determined for each one of the identified scenery components; mapping said local semantic map onto said global semantic map by matching one or more properties of at least one scenery component of the local semantic map to one or more attributes of at least one semantic object of said global semantic map, said at least one semantic object is associated with at least some portion of at least one of the traversable pathways to which said estimated trajectory is at least partially fitting; generating a plurality of hypotheses concerning possible locations of the user / vehicle on the semantic map along one of the pathways based on approximated matches between the estimated possible path and said plurality of traversable pathways extracted from the semantic map; and determining from said plurality of hypotheses a most likely location of the user / vehicle on the semantic map along one of the pathways based on one or more likelihood functions.

2. The method of claim 1 wherein the processing of the imagery data comprises applying a semantic segmentation process to the imagery data to bound / partition scenery components thereby captured and generating a semantic segmented image indicative thereof.

3. The method of claim 1 or 2 wherein the processing of the imagery data comprises applying a depth estimation process to the imagery data and / or to the semantic segmented image to determine a depth property for at least one of the scenery components, and wherein the step of matching one or more properties comprises verifying that azimuth and / or distance of semantic objects with respect to locations on the pathways identified in the global semantic map is in agreeable correlation with the azimuth and / or distance of the scenery component with respect to the 3D camera.

4. The method of any one of the preceding claims wherein the processing of the imagery data comprises applying a vSLAM process to the imagery data and / or to the semantic segmented image to determine one or more properties of at least one of the scenery components.

5. The method of any one of the preceding claims comprising identifying the traversable pathways in the global semantic map, and / or the scenery components in the imagery data, by artificial intelligence and / or deep learning processes.

6. The method of any one of the preceding claims comprising receiving position data indicative of a navigation starting point of the user / vehicle, fetching a first portion of the global semantic map based on said position data, and determining the one or more traversable pathways with respect to said first portion.

7. The method of claim 6 comprising extracting an additional portion of the global semantic map based on one or more of the received position data or a position of the user / vehicle in a previously extracted portion of said global semantic map, and further performing the step of determining one or more traversable pathways with respect to said additional portion.

8. The method of claim 7, further comprising iteratively repeating each of the steps during movement of the user / vehicle, to thereby iteratively update a determination of a most likely location of the user / vehicle along one of the pathways, until the user / vehicle’ s motion is stopped and / or a desired target location is reached.

9. The method of any one of the preceding claims comprising predicting one or more possible locations of the user / vehicle based on the matching of the one or more properties of at least one scenery component of the local semantic map to the one or more attributes of at least one semantic object of said global semantic map.

10. The method of any one of the preceding claims comprising computing a probability to one or more of the scenery components indicative of a measure of correlation of its properties to attributes of one or more of the semantic objects, and mapping the local semantic map to the global semantic map at least partially based on computed probability.

11. The method of claim 10 wherein the mapping of the local semantic map to the global semantic map is carried out utilizing an adaptive particle filter.

12. The method of any of the preceding claims, wherein, prior to the initial determining step, the traversable pathways are not identified on the global semantic map or extracted portion thereof.

13. The method of any of the preceding claims, wherein the traversable pathways comprise one or more of surface roads, dirt paths, tracks, trails, and bridges.

14. A localization system comprising: IMU and digital compass units configured to generate IMU and compass data / signals of a moving user / vehicle, a 3D camera configured generate imagery data indicative of scenery components in the user / vehicle's environment, and a processor configured to determine from analysis of a global semantic map or an extracted portion thereof a plurality of pathways traversable by the user / vehicle, estimate from the IMU and compass data / signals a possible path of the user / vehicle, process imagery data acquired by the 3D camera during motion of said user / vehicle and generate a local semantic map indicative of one or more scenery components identified in said imagery data and one or more properties determined for each one of the identified scenery components; map said local semantic map onto said global semantic map by matching one or more properties of at least one scenery component of the local semantic map to one or more attributes of at least one semantic object of said global semantic map, said at least one semantic object is associated with at least some portion of at least one of the traversable pathways to which said estimated trajectory is at least partially fitting; generate a plurality of hypotheses concerning possible locations of the user / vehicle on the semantic map based on approximated matches between the estimated possible path and said plurality of traversable pathways extracted from the semantic map, and determine from said plurality of hypotheses a most likely location of the user / vehicle on the semantic map based one or more likelihood functions.

15. The system of claim 14 configured to acquire an initial position of the user / vehicle from a positioning system or from a map.

16. The system of any of claims 14 to 15 comprising a segmentation module configured to detect the scenery components in the imagery data from the 3D camera.

17. The system of any one of claims 14 to 16 comprising a depth estimation module configured to estimate from the imagery data a distance of each of the scenery components from the user / vehicle.

18. The system of claim 17, wherein the processor is configured match one or more properties by verifying that azimuth and / or distance of semantic objects with respect to locations on the pathways identified in the global semantic map is in agreeable correlation with the azimuth and / or distance of the scenery component with respect to the 3D camera.

19. The system of any one of claims 14 to 18 comprising a vSlam module configured to estimate orientation of the scenery components with respect to the user / vehicle.

20. The system of any one of claims 14 to 19 comprising one or more Al units configured to determine properties of the scenery components.

21. The system of any one of claims 14 to 20 comprising at least one processor configured to acquire and at least partially process data / signals from the IMU, compass and 3D camera, and generate the IMU and compass data and the imagery data, and at least another processor configured carry out a mapping process utilizing the IMU and compass data and the imagery data from said at least one processor.

Images