Autonomous navigation based on local sensing and associated systems and methods

Abstract

A navigation map based on local sensing can form the basis of autonomous navigation of a mobile platform (110). A method for controlling an autonomous vehicle includes: obtaining real-time environmental information indicating an environment within a certain distance of the mobile platform (110) based on a first sensor carried by the mobile platform (110); detecting navigation features based on sensor data obtained from the first sensor or a second sensor carried by the mobile platform (110); integrating information corresponding to the navigation features with environmental information to generate a local navigation map; and generating navigation commands for controlling the movement of the mobile platform (110) based on the local navigation map.

Description

Technical Field

[0001] The present disclosure relates to autonomous navigation, and more particularly, to local sensing-based autonomous motion of a mobile platform. Background Art

[0002] Autonomous vehicles typically rely on access to externally provided high-precision maps to perform autonomous driving functions. External high-precision maps are typically pre-generated based on data collected from other sources (e.g., data collection vehicles). For example, a data collection vehicle may include stereo cameras that scan roads along pre-planned routes and collect image data, which can be combined with high-precision GPS overlays (with location data) to create a three-dimensional (3D) model of a region, country, or world. Based on the 3D model, basic navigation features such as lane markings, navigable areas, etc. can be automatically or semi-automatically marked, thereby providing basic "real value" for autonomous driving. Relying on external high-precision maps, the autonomous driving system of a vehicle typically focuses its computing resources on dynamic obstacle detection and tracking, enabling the vehicle to "drive" or otherwise navigate on its own. However, relying on access to externally provided maps may have various disadvantages and inflexibility. Therefore, there is a need for an improved autonomous navigation system for a mobile platform. BRIEF DESCRIPTION OF THE DRAWINGS

[0003] Figure 1 is a schematic diagram of a representative system having elements configured according to some embodiments of the disclosed technology.

[0004] Figure 2 is a flowchart illustrating a method of autonomous navigation based on a local navigation map according to some embodiments of the disclosed technology.

[0005] Figure 3 is a flow chart illustrating a method for generating a local navigation map based on local sensor data and a predetermined navigation map according to some embodiments of the disclosed technology.

[0006] Figure 4 is a flow chart illustrating a method for determining actions of a mobile platform based on a local navigation map and a predetermined navigation map according to some embodiments of the disclosed technology.

[0007] Figure 5 An example representation of 3D environment information in the form of a point cloud according to some embodiments of the disclosed technology is shown.

[0008] Figure 6 An example of sensor data obtained from sensors carried by a mobile platform according to some embodiments of the disclosed technology is shown.

[0009] Figure 7According to some embodiments of the present disclosure, Figure 6 Examples of navigation features detected in sensor data.

[0010] Figure 8 An example of a mobile platform configured in accordance with various embodiments of the disclosed technology is shown.

[0011] Fig. 9 is a block diagram illustrating an example of the architecture of a computer system or other control device that can be utilized to implement various portions of the disclosed technology.

[0012] Fig.10 Some embodiments of the present disclosure are shown in Figure 1 An example architecture implemented on a system with

[0013] The technology presented herein may be better understood by referring to the following detailed description in conjunction with the drawings, wherein like reference numbers indicate identical or functionally similar elements. DETAILED DESCRIPTION

[0014] Although external high-precision maps and high-precision GPS can provide autonomous vehicles with data for autonomous driving or other navigation, there are situations where the GPS signal is weak or not the desired signal, or where high-precision maps are not available (for example, in indoor environments). These situations may cause autonomous navigation to malfunction or fail. In addition, relying on external high-precision maps has at least the following disadvantages:

[0015] 1. Producing high-precision maps offline can be expensive, for example, because high-precision data (e.g., errors are limited to a few centimeters or less) must usually be collected from numerous sources by relatively expensive sensors;

[0016] 2. Updating and maintaining high-precision maps can be expensive, for example, because maps that are considered “true” by autonomous vehicles typically need to be frequently updated and maintained to provide an accurate reflection of the current environment;

[0017] 3. If the high-precision map fails (for example, the map is not updated in a timely manner, is not downloaded to the autonomous vehicle, or is otherwise inaccessible to the autonomous vehicle), it will seriously affect the safety of the autonomous driving system; and

[0018] 4. Producing high-precision maps may involve the qualification or certification of map providers, which may be complicated considering the local laws and regulations implemented in different countries and regions.

[0019] In general, some aspects of the disclosed technology involve integrating navigation features detected locally by a mobile platform to form a local high-precision navigation map. The local navigation map can be generated in real time based in part on high-precision attitude information obtained locally when the mobile platform is stationary, moving, or otherwise performing various autonomous navigation functions. Some aspects of the disclosed technology involve using features obtained from external maps as supplemental or reference data (as opposed to "real" data) to enhance the locally generated navigation map and / or assist other autonomous navigation functions of the mobile platform.

[0020] As an illustration, the disclosed technology provides a method for controlling a mobile platform. According to a representative method, a controller associated with the mobile platform obtains real-time environmental information, which indicates at least a portion of an environment within a certain distance of the mobile platform. For example, the real-time environmental information may include depth information corresponding to a 3D representation of at least a portion of the environment. The real-time environmental information may be based on sensor data generated by one or more sensors (e.g., a stereo camera or a laser radar (LiDAR) sensor) carried by the mobile platform.

[0021] According to a representative method, the controller may detect navigation features based on sensor data obtained from one or more sensors carried by the mobile platform. These sensors may or may not overlap with sensors used to generate real-time environmental information. For example, sensors for detecting navigation features may include cameras, LiDAR sensors, ultrasonic sensors, or radars. As an illustration, the detected navigation features may include lane lines or markings, edge lines or markings, guide lines or markings, zebra crossings, traffic lights, traffic signs, guide signs, navigable surfaces or spaces, stationary obstacles, or moving obstacles. In order to detect navigation features, the controller may apply one or more artificial neural networks (ANNs) and / or other suitable machine learning techniques to the sensor data.

[0022] According to a representative method, the controller can integrate information corresponding to the navigation features with real-time environmental information to generate a local navigation map. The information integration process can include associating a subset of the navigation features with one or more parts of the environmental information in real time. As an illustration, the integration process can be based on spatial and / or temporal transformations between or among reference systems associated with the following sensors: (a) sensors for generating environmental information and (b) sensors for detecting navigation features. Alternatively or additionally, the information integration process can be based on an estimated position and / or orientation of a mobile platform that meets one or more accuracy criteria. For example, for an estimated position, the positioning error of the mobile platform can be limited to a few centimeters, a few millimeters, or less, and / or for an estimated orientation angle, the positioning error of the mobile platform can be limited to one tenth of a degree, one hundredth of a degree, or less. With the help of a local navigation map, the controller can generate one or more navigation commands to at least partially control the movement of the mobile platform.

[0023] In some examples, the controller may also obtain additional (e.g., second) navigation features based on a predetermined navigation map (e.g., an external high-precision map). The controller may determine the position of the mobile platform relative to the predetermined navigation map, and may obtain the second navigation feature from the predetermined navigation map within a certain distance of the determined position of the mobile platform. In these examples, the controller may additionally integrate the second navigation feature with the environmental information to generate a local navigation map.

[0024] In some examples, the controller may also obtain action predictions for controlling the mobile platform based on a predetermined navigation map (e.g., an external low-precision map). Action predictions may include predictions of lane changes, stops, or turns within a threshold distance along the direction of travel of the mobile platform. In these examples, in addition to the local navigation map, the controller may also generate one or more navigation commands for the mobile platform based on action predictions.

[0025] More specifically, in order to navigate in an unknown environment, a representative mobile platform can use various sensors to sense its surroundings and generate high-precision 3D information (e.g., 3D depth map, 3D point cloud, etc.) to model the environment. The measurement error of the high-precision 3D environmental information can be limited to a few centimeters (e.g., 10 centimeters or less), a few millimeters (e.g., 10 millimeters or less), or smaller (e.g., less than 1 millimeter). At the same time, the mobile platform can use the posture information obtained from the local sensor to locate itself (e.g., its relative position and / or orientation) within the 3D environmental information. The positioning error (e.g., the error used to locate the mobile platform within the 3D environmental information) can also be limited to a few centimeters, a few millimeters, or smaller.

[0026] While the mobile platform is navigating, the mobile platform may continuously update the 3D environment information based on newly collected sensor data (e.g., new stereo images, new point clouds, etc.). In the case where the computing resources associated with the mobile platform are limited, the mobile platform may ignore or marginalize sensor data that reflects environmental features that are a certain distance beyond the current position of the mobile platform. When the 3D environment information is updated, the 3D environment information may be represented in a local reference system (e.g., a 3D coordinate system) centered at the current position of the mobile platform (or one of its sensors) or otherwise associated with the local reference system.

[0027] As an illustration, in order to locate the mobile platform within the 3D environment information, a controller associated with the mobile platform can determine the attitude of an applicable sensor carried by the mobile platform. For example, the controller can access attitude information provided by an inertial measurement unit (IMU) or a wheel odometer carried by the mobile platform.

[0028] As another example, the controller can compare a point cloud based on newly collected sensor data with the 3D environment information to be updated to determine the pose of the sensor. Two representative methods for implementing this comparison are landmark-based methods and raw data methods. Landmarks are uniquely identifiable objects in the environment that have positions that can be estimated by sensors. They can be physically different objects, such as wi-fi access points, or spatial points with other different features (e.g., color). Under this method, the controller can identify landmarks in the newly collected sensor data and 3D environment information, and then match these landmarks to determine the pose. On the other hand, the raw data method does not assume that landmarks can be identified, but is modeled directly based on the position. Optionally, data provided by the IMU and / or odometer can be used to further reduce the pose error. After the controller obtains the pose of the sensor, the controller can update the 3D environment information by fusing the newly collected sensor data (e.g., depth image) into the controller according to the pose.

[0029] Therefore, without being based on any external high-precision map, embodiments of the disclosed technology can enable a mobile platform controller to estimate changes in attitude (e.g., with 6 degrees of freedom and high precision) in real time using, for example, visual sensors (e.g., stereo cameras), IMUs, and / or wheel odometers carried by the mobile platform. Based on high-precision attitude estimation, the disclosed technology can provide a smaller attitude error within a certain distance of the mobile platform (e.g., within 200 to 500 meters), thereby creating a local navigation map that can integrate or fuse detected navigation features with high precision. The local high-precision navigation map can include all the information required for autonomous navigation, thereby realizing a robust and efficient autonomous navigation system without relying on external high-precision maps.

[0030] According to some embodiments, a controller associated with a mobile platform may include multiple functional layers. For example, the controller may include a sensor layer, in which many types of sensors can provide depth information for generating 3D environmental information. For example, a laser sensor (e.g., a LiDAR sensor) and / or a visual sensor (e.g., a stereo vision camera or an RGB-D camera) can be used with the sensor layer. Although the operating principles of these sensors may be different, they are able to collect depth information and / or other 3D information related to some parts of the surrounding environment, or at least can provide raw data that can be used to generate such depth or 3D information. When the mobile platform moves, the onboard sensor can continuously collect depth or 3D information detected from different angles. With the help of the sensor layer, the mobile platform controller can generate 3D environmental information (e.g., in the form of a point cloud) that models at least a portion of the surrounding environment. The sensor layer may also include other sensors (e.g., IMU, GPS sensor, etc.) that can be used according to various embodiments of the disclosed technology.

[0031] The controller may include a recognition layer, wherein the controller may detect or extract navigation features from sensor data obtained from various sensors carried by the mobile platform. As described above, navigation features may include elements related to road conditions, traffic regulations, directional information, obstacles, etc. The controller may implement suitable machine learning techniques, such as convolutional neural networks (CNNs), in the recognition layer. In some embodiments, certain navigation feature detections may be performed on separate single frames of sensor data. In some embodiments, certain navigation feature detections may be performed on a series of sensor data frames.

[0032] The controller may also include an integration layer, wherein the controller may integrate detected navigation features (e.g., lane lines or markings, edge lines or markings, guide lines or markings, zebra crossings, traffic lights, traffic signs, guide boards, navigable surfaces or spaces, stationary obstacles or moving obstacles) with 3D environmental information in time and / or space, and generate a real-time local navigation map. With the help of the integration layer, the controller can provide a local environment within a certain distance relative to the mobile platform with a high-precision representation of the detected navigation features. In some embodiments, all detected navigation features are expressed in a local coordinate system associated with the mobile platform (e.g., in the form of updating a rolling buffer based on the position and / or orientation of the mobile platform). In some other embodiments, the detected navigation features may be expressed in other suitable coordinate systems. As described above, using the local navigation map, the controller may generate one or more navigation commands to at least partially control the movement of the mobile platform.

[0033] For the sake of clarity, several details describing structures and / or processes that are well known and commonly associated with mobile platforms (e.g., autonomous vehicles, UAVs, and / or other types of mobile platforms) and corresponding systems and subsystems are not set forth in the following description, but may unnecessarily obscure some important aspects of the technology disclosed herein. In addition, although the following disclosure sets forth several embodiments of different aspects of the technology disclosed herein, several other embodiments may have different configurations or different components than the embodiments described herein. Therefore, the technology disclosed herein may have other embodiments that have additional elements and / or do not have the following references. Figures 1 to 10 Several of the elements described.

[0034] supply Figures 1 to 10 The accompanying drawings are not intended to limit the scope of the claims in this application unless otherwise specified.

[0035] Many embodiments of the present technology described below may take the form of computer or controller executable instructions, including routines executed by a programmable computer or controller. A programmable computer or controller may or may not reside on a corresponding mobile platform. For example, a programmable computer or controller may be an onboard computer of a mobile platform, or a separate but dedicated computer associated with a mobile platform, or a part of a network-based or cloud-based computing service. Those skilled in the relevant art will appreciate that, in addition to those shown and described below, the technology may also be implemented on a computer or controller system. The technology may be embodied in a dedicated computer or data processor that is specially programmed, configured or constructed to execute one or more computer executable instructions described below. Therefore, the terms "computer" and "controller" generally used herein refer to any data processor, and may include Internet home appliances and handheld devices (including palmtop computers, wearable computers, cellular or mobile phones, multiprocessor systems, processor-based or programmable consumer electronics, network computers, mini computers, etc.). The information processed by these computers and controllers may be presented on any suitable display medium including an LCD (liquid crystal display). Instructions for performing computer or controller executable tasks may be stored in or on any suitable computer readable medium, including hardware, firmware, or a combination of hardware and firmware. The instructions may be contained in any suitable memory device, including, for example, a flash drive, a universal serial bus (USB) device, and / or other suitable media. In certain embodiments, the instructions are therefore non-transitory.

[0036] Figure 11 is a schematic diagram of a representative system 100 having elements configured according to some embodiments of the disclosed technology. System 100 includes a mobile platform 110 (e.g., an autonomous vehicle) and a control system 120. Mobile platform 110 can be any suitable type of movable object, such as an autonomous vehicle, an unmanned aerial vehicle, or a robot.

[0037] The mobile platform 110 may include a body 112, which may carry a payload (e.g., passengers and / or cargo). The body 112 may also carry a sensor group 114, which includes one or more sensors, such as an imaging device and / or an optoelectronic scanning device. For example, the sensor group 114 may include a stereo camera, a monocular camera, a video camera, a static camera, and / or a LiDAR sensor. In some embodiments, the sensor group 114 may be used to perform or support functions such as navigation. In other embodiments, the sensor group 114 may be used to capture images for other purposes, such as detection, real estate photography, forest fire monitoring, etc.

[0038] In some embodiments, the sensor group 114 is supported relative to the body 112 using a mounting mechanism 116 (eg, a pan, rack, and / or beam). The mounting mechanism 116 can allow the sensor group 114 to be independently positioned relative to the body 112.

[0039] The mobile platform 110 may be configured to receive control commands from the control system 120 and / or send data to the control system 120. Figure 1In the embodiment shown in , the control system 120 includes some components carried on the mobile platform 110 and some components located outside the mobile platform 110. For example, the control system 120 may include a first controller 122 carried on the mobile platform 110 and a second controller 124 (e.g., a manually operated remote control such as a remote car key or a smartphone running a controller application) located away from the mobile platform 110 and connected via a communication link 128 (e.g., a wireless link such as a radio frequency (RF)-based link, a Wi-Fi network, a cellular network, a Bluetooth-based link, and / or other near-field communication links). The first controller 122 may include a computer-readable medium 126 that executes instructions to direct the actions of the mobile platform 110, including but not limited to the movement of the mobile platform and the operation of the payload 114. The second controller 124 may include one or more input / output devices, such as a display and control buttons. In some embodiments, the operator at least partially manipulates the second controller 124 to remotely control the mobile platform 110 and receives feedback from the mobile platform 110 via a display and / or other interface on the second controller 124. In some embodiments, mobile platform 110 operates autonomously, in which case second controller 124 may be eliminated or may be used only for operator override functions.

[0040] In order to provide safe and efficient operation, it is beneficial for a mobile platform to be able to autonomously or semi-autonomously detect obstacles and / or perform avoidance maneuvers to avoid obstacles. In addition, sensing environmental objects can be useful for mobile platform functions such as navigation, target tracking, and map building, especially when the mobile platform operates in a semi-autonomous or fully autonomous manner.

[0041] The mobile platforms described herein may include one or more sensors (e.g., separate and independent from sensor group 114) configured to detect objects in the environment surrounding the mobile platform. In some embodiments, the mobile platform includes one or more sensors (e.g., Figure 1The distance measuring device 140 is configured to measure the distance between an object and a mobile platform. The distance measuring device can be carried by the mobile platform in various ways, such as above, below, on the side or inside the body of the mobile platform. Optionally, the distance measuring device can be coupled to the mobile platform via a gimbal or other mounting mechanism that allows the device to translate and / or rotate relative to the mobile platform. In some embodiments, the distance measuring device may include an optical distance measuring device that uses light to measure the distance to an object. In some embodiments, the distance measuring device is a stereoscopic vision system that can provide stereoscopic vision data, and depth information can be determined based on the stereoscopic vision data. The stereoscopic vision system can be a stereo camera. In some embodiments, the distance measuring device may include a LiDAR system, a radar system, or a laser rangefinder.

[0042] Fig.10 Some embodiments of the present disclosure are shown in Figure 1 1. As an illustration, the architecture 1000 may include multiple functional layers, some of which are implemented via a controller (e.g., an onboard computer of the mobile platform, associated computing devices, and / or associated computing services).

[0043] refer to Fig.10 , the architecture 1000 may include a sensor layer 1010, which may include access to various sensors (e.g., LiDAR, stereo camera, radar, odometer, IMU, GPS, etc.) carried by the mobile platform. In some embodiments, the sensor layer may also include access to an external map (e.g., an external high-precision map). The architecture 1000 may include a plurality of functional modules that perform various types of estimation, detection, positioning, fusion, or other analysis based on direct or indirect data access to the sensor layer. For example, different modules may perform multi-sensor state estimation and / or positioning, multi-sensor depth estimation, map-based positioning, LiDAR-based 3D detection, stereo-based 3D detection, radar-based detection, real-time 3D reconstruction, lane and sign detection, map-based traversability, lane and sign extraction, multi-sensor fusion for moving object tracking, traversability analysis, fusion of lane observation, etc. As an illustration, an example of a local navigation map may be generated by performing a module for multi-sensor fusion for moving object tracking, traversability analysis, and fusion of lane observation. The architecture 100 may also include a semantic layer 1020 that may include a semantic mapping module for mapping results generated by the module into semantic values ​​or representations that may be fed into a higher-level navigation module (eg, for mobile platform operational decision making and / or route planning).

[0044] Figure 22 is a flow chart showing a method 200 for autonomous navigation based on a local navigation map according to some embodiments of the disclosed technology. The method 200 may be implemented by a controller (eg, an onboard computer of a mobile platform, an associated computing device, and / or an associated computing service).

[0045] refer to Figure 2 At block 210, method 200 includes determining (a) three-dimensional (3D) environmental information indicating at least a portion of an environment within a certain distance of a mobile platform, and (b) a position and / or orientation of an autonomous vehicle relative to the 3D environmental information. In some embodiments, the determination is performed in real time.

[0046] As an illustration, 3D environmental information can be determined based on a visual sensor (e.g., a stereo camera), LiDAR, or radar carried by a mobile platform. For example, a stereo image produced by a stereo camera can be used to generate high-precision 3D environmental information, which models at least a portion of the environment around the mobile platform (e.g., the direction of the mobile platform). 3D environmental information can be expressed and / or formatted in various ways. For example, 3D information can be in the form of a 3D point cloud, a 3D grid, a symbolic distance field, etc. The measurement error of high-precision 3D environmental information can be limited to a few centimeters, a few millimeters, or less. For example, this can be achieved by using LiDAR, a stereo camera, and / or other sensors that meet specific accuracy requirements and only incorporating sensor data corresponding to measurements within a certain distance of the mobile platform into the 3D environmental information. When the mobile platform is navigating, the mobile platform can continuously update the 3D environmental information based on newly collected sensor data (i.e., new stereo images, new point clouds, etc.). In the case where the computing resources associated with the mobile platform are limited, the controller can skip or marginalize sensor data that reflects environmental features beyond a certain distance from the current position of the mobile platform. Figure 5 An example representation of at least a portion of 3D environment information in the form of a point cloud in accordance with some embodiments of the disclosed technology is shown.

[0047] Return to reference Figure 2, substantially simultaneously with creating and / or updating the 3D environment information, the controller may locate the mobile platform (e.g., its relative position and / or orientation) within the 3D environment information using posture information obtained from one or more sensors carried by the mobile platform. In other words, the operation of creating and / or updating the 3D environment information may overlap in time with the positioning operation of the mobile platform. This process may generally be referred to as localization. As an illustration, the controller may locate the mobile platform by accessing data provided by an IMU, a wheel odometer, a visual sensor, and / or other sensors carried by the mobile platform. For example, the controller may compare a point cloud generated based on newly collected sensor data with existing, yet-to-be-updated 3D environment information (e.g., a point cloud based on previously collected sensor data) to determine the posture of the sensor / mobile platform. Two representative methods for achieving this purpose include a landmark-based method and a raw data method. Landmarks are uniquely identifiable objects in the environment whose positions can be estimated by sensors. They can be physically distinct objects, such as wi-fi access points, or feature points in the environment with other distinct features (e.g., color). Under this approach, the controller can (e.g., using applicable pattern recognition techniques) identify landmarks in both the newly collected sensor data and the existing 3D environment information, and then match them to determine the current pose (e.g., relative rotation and / or translation between the coordinate systems associated with the existing 3D environment information and the newly collected sensor data, so as to achieve matching of the landmarks). On the other hand, the raw data approach does not assume that the landmarks can be identified, but directly models the newly collected sensor data according to the new orientation and / or position. Optionally, the data provided by the IMU, odometer and / or other sensors can be used to further reduce the pose error. The positioning error (e.g., the error used to locate the mobile platform within the 3D environment information) can also be limited to a few centimeters, a few millimeters, or less. These accuracies can be achieved at least in part because the 3D environment information is obtained from the mobile platform with high accuracy (e.g., the error is limited to a few tens of centimeters, a few tens of centimeters, or less) and is limited to a certain distance of the mobile platform (e.g., hundreds of meters, tens of meters, a few meters, or less). After the controller obtains the pose of the sensor, the controller can update the 3D environment information by fusing the newly collected sensor data (e.g., depth data) into the controller according to the pose. For example, after a rotation and / or translation transformation based on the pose, a new point cloud generated based on the newly collected sensor data can be merged with the existing 3D environment information (e.g., in the form of a point cloud). As part of the update process, the 3D environment information can be expressed in a local reference system (e.g., a 3D coordinate system) centered on the current position of the mobile platform or associated with the reference system.

[0048] Continue to refer Figure 2At box 220, method 200 includes detecting navigation features (e.g., lane lines, edge lines, guide lines, zebra crossings, traffic lights, traffic signs, guide boards, navigable surfaces or spaces, stationary obstacles, or moving obstacles) based at least in part on sensor data obtained from one or more sensors carried by the mobile platform. As an illustration, the controller can access sensor data obtained from various sensors carried by the mobile device (e.g., LiDAR sensors, cameras, radars, ultrasonic sensors, etc.) to detect navigation features. The sensors used for navigation feature detection may or may not overlap with the sensors used to determine 3D environmental information. For example, in a cost-sensitive system, the sensors used for navigation feature detection may be the same as the sensors (or a subset thereof) used to determine 3D environmental information. As another example, sensors that are not used to determine 3D environmental information can provide additional data for navigation feature detection, thereby enhancing the performance and robustness of autonomous navigation.

[0049] As described above, navigation features may include elements related to road conditions, traffic regulations, directional information, obstacles, etc. The controller may implement suitable machine learning or other object detection techniques to analyze the sensor data and extract navigation features (e.g., their type, content, size, boundaries, location, etc.). For example, selective search object detectors, region-based convolutional neural networks (R-CNN), fast R-CNN, etc. may be used to implement navigation feature detection. In some embodiments, certain navigation feature detections may be performed on separate single frames of sensor data. In some embodiments, certain navigation feature detections may be performed on a series of sensor data frames.

[0050] Taking lane markings as an example, for each frame of image data (or, point cloud data), the pixels in the image (or points in the point cloud) corresponding to the lane markings are detected. Optionally, the controller can then appropriately perform 2D to 3D (or, 3D to 3D) position conversion on the detected markings. In order to track multiple parallel lanes, the controller can also implement a suitable lane fitting algorithm to link lane markings and represent them in lane segments.

[0051] Figure 6 An example of sensor data (e.g., a frame of image) obtained from a sensor (e.g., a camera) carried by a mobile platform according to some embodiments of the disclosed technology is shown. The sensor data obtained from the sensor can be Figure 5 The 3D environment information shown in at least partially overlap. That is, they may include representations of the same portion of the environment surrounding the mobile platform. In various embodiments, the format of the sensor data may or may not be different from the format of the 3D environment information.

[0052] As an illustration, Figure 6The image shown in may be a single frame image captured by a front camera carried by the mobile platform. The image shows the environment in which the mobile platform is moving and / or needs to navigate through. The image may or may not have depth information (for example, if generated by a stereo camera).

[0053] Figure 7 According to some embodiments of the present disclosure, Figure 6 Examples of navigation features (e.g., lane markings, directional signs, speed limit signs, moving vehicles) detected in sensor data. As an illustration, by analyzing Figure 6 In the single frame image shown, the controller may use applicable pattern recognition and / or machine learning techniques to detect lane markings 702, speed limits 704, moving obstacles 706, and direction signs 708. In some embodiments, the controller may not be able to detect lane markings 702, speed limits 704, moving obstacles 706, and direction signs 708 using only Figure 6 The controller may detect each possible navigation feature using sensor data collected from different sensors. For example, if the confidence associated with the detection is above a threshold, the controller may decide to confirm the detection of the navigation feature. In some embodiments, the controller may use a combination of sensor data collected from different sensors to detect more navigation features.

[0054] Return to reference Figure 2 At block 230, method 200 includes generating a local navigation map based at least in part on the 3D environment information and the detected navigation features. As an illustration, the controller may integrate the detected navigation features in time and / or space with the 3D environment information in real time. The controller may provide the detected navigation features with a high-precision representation of the local environment within a certain distance relative to the mobile platform. In some embodiments, all detected navigation features are expressed in a local reference system associated with the mobile platform (e.g., in the form of a rolling buffer that is updated based on the position and / or orientation of the mobile platform).

[0055] As an illustration, for each frame of sensor data, the pose (e.g., position and / or orientation) of the mobile platform (or, sensor) may be estimated as discussed above with reference to block 210. Based on the pose information, the controller may determine a transformation (e.g., translation and / or rotation) between a reference system associated with the detected navigation feature and the current 3D environment information. The controller may then project the detected navigation feature onto the 3D environment information to generate a local navigation map.

[0056] As a more specific example, for each frame of an image generated from a camera carried by a mobile platform, the controller can identify the pixel position of the lane marking in the image (e.g., based on a region-based convolutional neural network (R-CNN), a fast R-CNN, a faster R-CNN, and / or other suitable algorithms or methods). Using a suitable depth estimation algorithm (e.g., a semi-global matching and mutual information method, a method based on a multi-scale deep network, and / or other suitable algorithms or methods), the controller can calculate the depth information of the lane marking pixels. The controller can also calculate the three-dimensional (3D) coordinates of the identified lane marking pixels relative to the camera.

[0057] Given the 3D coordinates (x, y, z) of a lane marking, the controller can project the lane marking into the two-dimensional (2D) bird's-eye view space (x', y') using the following projection formula:

[0058] x′=s x *x+c x

[0059] y′=s y *y+c y

[0060] where the vector s x s x and y s y is the scale factor, and the scalar c x c x and c y c y is the offset. These constants are optional and can be predetermined. For example, s x 、s y 、c x s x s y c x and c y c y The value of can be set in a way that appropriately aligns the viewpoint of the bird's-eye view space.

[0061] The controller can then group the lane markings and perform curve fitting on the lane markings belonging to different groups to obtain structured lane lines, which can be projected onto the 3D environment information as part of the local navigation map.

[0062] As another specific example, for each frame of image generated from a camera carried by a mobile platform, the controller can identify the pixel location of a traffic sign (e.g., specifying a speed limit, traffic regulations, directions, etc.) from the image (e.g., based on a region-based convolutional neural network (R-CNN), a fast R-CNN, a faster R-CNN, and / or other suitable algorithms or methods). Using a suitable depth estimation algorithm (e.g., a semi-global matching and mutual information method, a method based on a multi-scale deep network, and / or other suitable algorithms or methods), the controller can calculate the depth information of the traffic sign pixels. The controller can also calculate the three-dimensional (3D) coordinates of the identified traffic pixels relative to the camera. Given the 3D coordinates of the traffic sign pixels, the controller can project the traffic sign into a two-dimensional (2D) top view space in a manner similar to the lane line projection described above.

[0063] To eliminate or mitigate misidentification, the controller may use the position and / or orientation information of the mobile platform associated with each frame image to perform a weighted time accumulation of the identified traffic sign. As an illustration, a Kalman filter, an octree, a probabilistic multiple hypothesis tracking method, and / or other suitable algorithms or methods may be used to perform the time accumulation and the confidence of whether the sign is detected at the location. As an illustration, let the confidence before and after the update be z and z respectively. n-1 z n-1 and z n z n , the update weight is ww and the current observation value is zz, then the update formula for the confidence of detecting the sign at the current position can be:

[0064] z n =w*z+(1-w)*z n-1

[0065] The controller may project traffic signs onto the 3D environment information and associate them with their respective confidence levels that are updated as the mobile platform moves, as part of a local navigation map.

[0066] Continue to refer Figure 2 At block 240, method 200 includes determining a next action of the mobile platform based at least in part on the local navigation map and the mobile platform position and / or orientation. As described above, the local high-precision navigation map may include all information required for autonomous navigation of the mobile platform, thereby enabling the controller to generate one or more commands (e.g., stop, accelerate, decelerate, turn left, turn right, etc.) that at least partially control the movement of the mobile platform without relying on an external map.

[0067] For example, given lane lines identified in the local navigation map, the mobile platform can perform lane keeping operations. As another example, given speed limits identified in the local navigation map, the mobile platform can perform speed control operations. As yet another example, given static and / or moving objects identified in the local navigation map, the mobile platform can perform obstacle avoidance operations.

[0068] If it will Figure 3 and Figure 4 As discussed in more detail in , in some embodiments, the controller can obtain features from external maps as supplemental or reference data (as opposed to "real" data) to enhance locally generated navigation maps and / or assist other autonomous navigation functions of the mobile platform.

[0069] At block 250, the method 200 includes determining whether to continue updating the local navigation map and performing autonomous navigation. If the controller determines to continue the process, the method 200 returns to block 210. Otherwise, the method 200 ends.

[0070] Figure 3 1 is a flow chart showing a method 300 for generating a local navigation map based on local sensor data and a predetermined navigation map according to some embodiments of the disclosed technology. The method 300 may be implemented by a controller (e.g., an onboard computer of a mobile platform, an associated computing device, and / or an associated computing service).

[0071] refer to Figure 3 At block 310, method 300 includes detecting a first set of navigation features based at least in part on data obtained from sensors carried by the mobile platform. As an illustration, this can be similar to reference Figure 2 This is accomplished in the manner described in blocks 210 and 220.

[0072] At box 320, the method includes positioning the mobile platform relative to a predetermined navigation map. As an illustration, the predetermined navigation map can be an external high-precision map accessible to the controller (e.g., a high-definition (HD) map with an accuracy error of 10 cm or less). In order to position the mobile platform relative to the predetermined navigation map, the controller can apply a global positioning method or other suitable positioning technology. As an illustration, the positioning process can be based on an iterative neighbor point (ICP) algorithm that aims to find a transformation between two reference systems (e.g., one associated with the mobile platform and the other associated with the predetermined navigation map) by minimizing the mean square error.

[0073] At block 330, the method includes obtaining a second set of navigation features from a predetermined navigation map based on the position of the mobile platform relative to the map. As an illustration, given the relative position of the mobile platform, the controller may search and retrieve navigation features included in the predetermined navigation map and within a certain distance (e.g., 200 to 500 meters) of the position of the mobile platform on the map.

[0074] At box 340, the method includes generating a local navigation map using at least a first group of navigation features and a second group of navigation features. As an illustration, both the first group of navigation features and the second group of navigation features can be projected onto 3D environment information generated / updated based on sensor data obtained from local sensors of the mobile platform. In some embodiments, at least some of the navigation features in the second group of navigation features do not overlap with the first group of navigation features. The controller may decide to assign different weights to the two groups and combine their projections. In some embodiments, conflicting information may occur between the first group of navigation features and the second group of navigation features, and the controller may decide to use only the first group of features or at least skip some of the features in the second group of features because they may be outdated or inaccurate for other reasons. In some embodiments, the controller may project all navigation features from the first group, but only a subset of the navigation features from the second group that are complementary to the first group (e.g., in terms of feature type) is projected.

[0075] Therefore, the local navigation map can act as a safety redundancy mechanism for autonomous navigation. When an external or other predetermined HD map is available, the HD map can be merged into the local navigation map in the form of sensor data. When the information provided by the external or other predetermined HD map conflicts with the information of the local navigation map or is inconsistent for other reasons, the controller can use the local navigation map for autonomous navigation, thereby improving navigation safety.

[0076] By utilizing features from external maps as supplemental or reference data (as opposed to “real” data), the local navigation map can be more comprehensive and reliable for autonomous navigation.

[0077] Figure 4 4 is a flow chart showing a method 400 for determining an action of a mobile platform based on a local navigation map and a predetermined navigation map according to some embodiments of the disclosed technology. The method 400 may be implemented by a controller (e.g., an onboard computer of a mobile platform, an associated computing device, and / or an associated computing service).

[0078] refer to Figure 4 At block 410, method 400 includes obtaining (1) a local navigation map generated based on sensor data obtained from sensors carried by a mobile platform, and (2) a position and / or orientation of the mobile platform. As an illustration, this may be performed in a manner generally similar to reference Figure 2 This is accomplished in the manner described in blocks 210, 220, and 230.

[0079] At block 420, the method includes positioning the mobile platform relative to a predetermined navigation map. As an illustration, the predetermined navigation map may be an external low-precision map (e.g., a standard definition (SD) map having an accuracy error of 5 m or greater) accessible to the controller. The controller may locate the mobile platform in a manner substantially similar to the reference map. Figure 3 The controller may position the mobile platform relative to the predetermined navigation map in the manner described in block 320 of FIG. In some embodiments, the controller may not need to accurately position the mobile platform relative to the predetermined navigation map, as long as certain navigation instructions (e.g., “change to the left lane in 300 meters”) can be retrieved from the predetermined navigation map.

[0080] At box 430, the method includes obtaining action predictions based at least in part on a predetermined navigation map. As an illustration, given the relative position and / or orientation of the mobile platform and the target destination or direction, the controller can search for navigation instructions in the predetermined navigation map. The navigation instructions may include predictions of lane changes, stops, or turns within a threshold distance of the mobile platform along its forward direction. Because the predetermined navigation map may not have high accuracy and / or the relative position and / or orientation of the mobile platform may not be accurate, not all navigation instructions are useful predictions. Some of the navigation instructions that predict actions beyond a threshold distance or time can be obtained by the controller and taken into account.

[0081] At box 440, the method includes determining the next action of the mobile platform based on the local navigation map and the action prediction retrieved from the predetermined navigation map. For example, the action prediction retrieved from the predetermined navigation map is "turn right in 400 meters", and the predetermined navigation map has an error of approximately + / -50 meters. In this case, the controller can locate the relevant road topology (e.g., the intersection for the predicted right turn) in the local high-precision navigation map, and ensure that the mobile platform is traveling in the right lane of the road so that the intersection is detected when it appears. When the right turn detection is successful, the controller can issue a "turn right" command to control the mobile platform to complete the turn. Therefore, the fidelity of the mobile platform operation can be improved to a few centimeters, millimeters, or less, instead of tens of meters.

[0082] Thus, utilizing features obtained from external maps as supplemental or reference data (as opposed to "real" data), the controller can enhance locally generated navigation maps and / or assist other autonomous navigation functions of the mobile platform.

[0083] The following is a non-exhaustive list of additional examples of the disclosed technology.

[0084] 1. A computer-implemented method for controlling an autonomous vehicle, comprising:

[0085] Determine, in real time based at least in part on data obtained from visual sensors and inertial sensors onboard the autonomous vehicle, (a) three-dimensional (3D) environmental information indicative of at least a portion of an environment within a certain distance of the autonomous vehicle and (b) a position and / or orientation of the autonomous vehicle relative to the 3D environmental information;

[0086] Detecting navigation features including at least one of the following based at least in part on sensor data obtained from one or more sensors carried by the autonomous vehicle: lane lines, edge lines, guide lines, zebra crossings, traffic lights, traffic signs, guide boards, navigable surfaces or spaces, stationary obstacles, or moving obstacles;

[0087] integrating information corresponding to the navigation features with at least 3D environment information to generate a local navigation map;

[0088] Determining the actions of the autonomous vehicle based at least in part on (a) the generated local navigation map and (b) the determined position and / or orientation of the autonomous vehicle; and

[0089] Make the autonomous vehicle move according to the determined actions.

[0090] 2. The method of Example 1, wherein the one or more sensors include at least one of a visual sensor or an inertial sensor.

[0091] 3. A computer-implemented method for controlling a mobile platform, comprising:

[0092] obtaining real-time environmental information indicative of at least a portion of an environment within a distance of the mobile platform based at least in part on data received from a first sensor carried by the mobile platform;

[0093] detecting a navigation feature based at least in part on sensor data obtained from at least one of a first sensor or a second sensor carried by the mobile platform;

[0094] integrating information corresponding to the navigation features with at least environmental information to generate a local navigation map; and

[0095] One or more navigation commands for at least partially controlling movement of the mobile platform are generated based at least in part on the generated local navigation map.

[0096] 4. The method according to Example 3, wherein the environmental information includes depth information.

[0097] 5. A method according to Example 3, wherein the environmental information corresponds to a 3D representation of at least a portion of the environment within a certain distance of the mobile platform.

[0098] 6. The method of Example 3, wherein the first sensor comprises at least one of a LiDAR sensor or a stereo camera.

[0099] 7. The method of Example 3, wherein the second sensor comprises at least one of a camera, a LiDAR sensor, an ultrasonic sensor, or a radar.

[0100] 8. The method according to Example 3, wherein the navigation feature includes at least one of the following: lane markings, edge lines, guide lines, zebra crossings, traffic lights, traffic signs, guide boards, navigable surfaces or spaces, stationary obstacles, or moving obstacles.

[0101] 9. The method of example 3, wherein detecting the navigation feature is based at least in part on applying one or more artificial neural networks to the sensor data.

[0102] 10. The method of example 3, wherein integrating information corresponding to the navigation features with at least environmental information comprises associating at least a subset of the navigation features with one or more portions of the environmental information.

[0103] 11. The method of example 3, wherein integrating the information corresponding to the navigation feature with at least the environmental information is performed in real time.

[0104] 12. A method according to Example 3, wherein the integration of information corresponding to the navigation feature with at least environmental information is based at least in part on a spatial and / or temporal transformation between reference systems associated with at least two sensors carried by the mobile platform.

[0105] 13. A method according to Example 3, wherein integrating the information corresponding to the navigation feature with at least environmental information is based at least in part on an estimated position and / or orientation of the mobile platform.

[0106] 14. The method of example 13, wherein the estimated position and / or orientation satisfies one or more accuracy criteria.

[0107] 15. The method of Example 3, wherein the navigation feature is a first navigation feature, and wherein the method further comprises: obtaining a second navigation feature based at least in part on at least one predetermined navigation map.

[0108] 16. The method according to Example 15 also includes determining the position of the mobile platform relative to a predetermined navigation map.

[0109] 17. The method according to Example 16 also includes obtaining a second navigation feature from a predetermined navigation map within a certain distance of the determined position of the mobile platform.

[0110] 18. The method according to Example 15 further includes integrating the second navigation feature with at least environmental information to generate a local navigation map.

[0111] 19. The method according to Example 3 further includes obtaining action predictions for controlling the mobile platform based on a predetermined navigation map.

[0112] 20. The method of Example 19, wherein the action prediction includes a prediction of at least one of a lane change, a stop, or a turn within a threshold distance of the mobile platform.

[0113] 21. The method of example 19, wherein generating one or more navigation commands is also based on motion prediction.

[0114] 22. The method of Example 3, wherein the mobile platform comprises at least one of: an unmanned aerial vehicle (UAV), a manned aerial vehicle, an autonomous vehicle, a self-balancing vehicle, or a robot.

[0115] 23. A non-transitory computer readable medium storing computer executable instructions that, when executed, cause one or more processors associated with a mobile platform to perform any of example methods 1 to 22.

[0116] 24. A vehicle comprising a controller programmed to at least partially control one or more movements of the vehicle, wherein the programmed controller comprises one or more processors configured to perform any one of example methods 1 to 22.

[0117] Figure 8 Examples of mobile platforms configured according to various embodiments of the disclosed technology are shown. As shown, the representative mobile platforms disclosed herein may include at least one of the following: an unmanned aerial vehicle (UAV) 802, a manned aerial vehicle 804, an autonomous vehicle 806, a self-balancing vehicle 808, a ground robot 810, a smart wearable device 812, a virtual reality (VR) head-mounted display 814, or an augmented reality (AR) head-mounted display 816.

[0118] Fig. 9 9 is a block diagram showing an example of the architecture of a computer system or other control device 900 that can be used to implement various parts of the disclosed technology. Fig. 9In the embodiment, the computer system 900 includes one or more processors 905 and memory 910 connected via an interconnect 925. The interconnect 925 can represent one or more separate physical buses, point-to-point connections, or both connected through appropriate bridges, adapters, and / or controllers. Thus, the interconnect 925 can include, for example, a system bus, a peripheral component interconnect (PCI) bus, a HyperTransport or Industry Standard Architecture (ISA) bus, a Small Computer System Interface (SCSI) bus, a Universal Serial Bus (USB), an IIC (I2C) bus, or an Institute of Electrical and Electronics Engineers (IEEE) Standard 674 bus (sometimes referred to as "Firewire").

[0119] The processor 905 may include a central processing unit (CPU) that controls the overall operation of, for example, a host computer. In some embodiments, the processor 905 implements this by executing software or firmware stored in the memory 910. The processor 905 may be or may include one or more programmable general-purpose or special-purpose microprocessors, digital signal processors (DSPs), programmable controllers, application-specific integrated circuits (ASICs), programmable logic devices (PLDs), etc., or a combination of these devices.

[0120] The memory 910 may be or include the main memory of the computer system. The memory 910 represents any suitable form of random access memory (RAM), read-only memory (ROM), flash memory, etc., or a combination of these devices. In use, the memory 910 may contain a machine instruction set, which, when executed by the processor 905, causes the processor 905 to perform operations to implement embodiments of the disclosed technology.

[0121] Also connected to processor 905 via interconnect 925 is an (optional) network adapter 915. Network adapter 915 provides computer system 900 with the ability to communicate with remote devices such as storage clients and / or other storage servers, and may be, for example, an Ethernet adapter or a Fibre Channel adapter.

[0122] The techniques described herein may be implemented, for example, by programmable circuits (e.g., one or more microprocessors) programmed with software and / or firmware, or entirely in dedicated hardwired circuits, or in a combination of these forms. The dedicated hardwired circuits may be in the form of, for example, one or more application specific integrated circuits (ASICs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), etc.

[0123] Software or firmware for implementing the techniques described herein may be stored on a machine-readable storage medium and may be executed by one or more general or special purpose programmable microprocessors. As used herein, the term "machine-readable storage medium" or "computer-readable storage medium" includes any mechanism that can store information in a form accessible to a machine (a machine may be, for example, a computer, a network device, a cellular phone, a personal digital assistant (PDA), a manufacturing tool, any device with one or more processors, etc.). For example, a machine-accessible storage medium includes recordable / non-recordable media (e.g., read-only memory (ROM); random access memory (RAM); magnetic disk storage media; optical storage media; flash memory devices, etc.), etc.

[0124] The term "logic" as used herein may include, for example, programmable circuits programmed with specific software and / or firmware, dedicated hardwired circuits, or combinations thereof. It will be appreciated by those skilled in the art that the components and steps or processing elements shown in the above figures may be changed in various ways without departing from the disclosed technology. For example, the order of the logic may be rearranged, sub-steps may be performed in parallel, illustrated logic may be omitted, other logic may be included, etc. In some embodiments, one or more of the described components may perform one or more of the described steps.

[0125] References in this specification to "embodiments" (e.g., "some embodiments," "various embodiments," "one embodiment," "an embodiment," etc.) mean that a particular feature, structure, or characteristic described in conjunction with the embodiment is included in at least one embodiment or implementation of the present disclosure. The appearances of these phrases in various places in the specification do not necessarily all refer to the same embodiment, nor are separate or alternative embodiments mutually exclusive of other embodiments. In addition, various features are described that may be exhibited by some embodiments but not by other embodiments. Similarly, various requirements are described that may be requirements of some embodiments but not others.

[0126] As used herein, the phrase "and / or" as in "A and / or B" refers to A alone, B alone, and both A and B.

[0127] To the extent that any material incorporated herein by reference conflicts with the present disclosure, the present disclosure shall prevail. Although the subject matter has been described in specific language of structural features and / or method actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the above-mentioned specific features or actions. For the purpose of illustration, specific embodiments and implementations have been described herein, but various modifications may be made without departing from the scope of the embodiments and implementations. The specific features and actions described above are disclosed as example forms of implementing the appended claims. Therefore, the embodiments and implementations are not limited except in the appended claims.

[0128] It will be appreciated from the foregoing that, for illustrative purposes, a specific embodiment of the disclosed technology has been described herein, but various modifications may be made without departing from the present technology. For example, the present invention may be implemented with respect to any suitable mobile platform. Figure 2 The method 200 includes but is not limited to Figure 8 Certain aspects of the technology described in the context of a particular embodiment may be combined or excluded in other embodiments. For example, Figure 3 Method 300 is implemented Figure 4 Method 400. In addition, when advantages associated with certain embodiments of the disclosed technology have been described in the context of those embodiments, other embodiments may also exhibit such advantages, and not all embodiments must exhibit such advantages to fall within the scope of the technology. Therefore, the present disclosure and associated technology may encompass other embodiments not explicitly shown or described herein.

Claims

1. A computer-implemented method for controlling an autonomous vehicle, comprising: Determine in real time (a) 3D environmental information indicating at least a portion of an environment within a certain distance of the autonomous vehicle and (b) a position and / or orientation of the autonomous vehicle relative to the 3D environmental information based at least in part on data obtained from visual sensors and inertial sensors carried by the autonomous vehicle; Detecting navigation features including at least one of the following based at least in part on sensor data obtained from one or more sensors carried by the autonomous vehicle: lane lines, edge lines, guide lines, zebra crossings, traffic lights, traffic signs, guide boards, navigable surfaces or spaces, stationary obstacles, or moving obstacles; integrating information corresponding to the navigation features with at least the 3D environment information to generate a local navigation map; Determining the movement of the autonomous vehicle based at least in part on (a) a generated local navigation map and (b) a determined position and / or orientation of the autonomous vehicle; as well as causing the autonomous vehicle to move according to the determined action; The step of integrating the information corresponding to the navigation feature with at least the 3D environment information to generate a local navigation map comprises: Determining a transformation relationship between a reference system associated with the navigation feature and the 3D environment information based on the posture information of one or more of the sensors; Projecting the navigation features onto the 3D environment information to generate a local navigation map; The method further comprises: Obtaining a motion prediction for controlling the autonomous vehicle based on a predetermined navigation map, wherein the predetermined navigation map is a low-precision map, and the motion prediction includes a prediction of at least one of a lane change, a stop, or a turn within a threshold distance of the autonomous vehicle; Based on the local navigation map, the action prediction of the autonomous vehicle, and the position and / or orientation of the autonomous vehicle, the next action of the autonomous vehicle is determined.

2. The method according to claim 1, wherein: The one or more sensors include at least one of a visual sensor or an inertial sensor.

3. A computer-implemented method for controlling a mobile platform, comprising: obtaining real-time environmental information indicative of at least a portion of an environment within a distance of the mobile platform based at least in part on data received from a first sensor carried by the mobile platform; detecting a navigation feature based at least in part on sensor data obtained from at least one of a first sensor or a second sensor carried by the mobile platform; integrating information corresponding to the navigation feature with at least the environmental information to generate a local navigation map; as well as generating one or more navigation commands for at least partially controlling movement of the mobile platform based at least in part on the generated local navigation map; The step of integrating the information corresponding to the navigation feature with at least the environmental information to generate a local navigation map comprises: Determining a transformation relationship between a reference system associated with the navigation feature and the environmental information based on the posture information of one or more of the sensors; Projecting the navigation feature onto the environmental information to generate a local navigation map; The method further comprises: Obtaining a maneuver prediction for controlling the mobile platform based on a predetermined navigation map, wherein the predetermined navigation map is a low-precision map, and the maneuver prediction includes a prediction of at least one of a lane change, a stop, or a turn within a threshold distance of the mobile platform; A next action of the mobile platform is determined based on the local navigation map, the prediction of the action of the mobile platform, and the location and / or orientation of the mobile platform.

4. The method according to claim 3, wherein: The environmental information includes depth information.

5. The method according to claim 3, wherein: The environment information corresponds to a 3D representation of at least a portion of an environment within a distance of the mobile platform.

6. The method according to claim 3, wherein: The first sensor includes at least one of a lidar sensor or a stereo camera.

7. The method according to claim 3, wherein: The second sensor includes at least one of a camera, a lidar sensor, an ultrasonic sensor, or a radar.

8. The method according to claim 3, wherein: The navigation features include at least one of the following: Lane markings, edge lines, guide lines, zebra crossings, traffic lights, traffic signs, guide boards, navigable surfaces or spaces, stationary obstacles or moving obstacles.

9. The method according to claim 3, wherein: Detecting navigational features is based at least in part on applying one or more artificial neural networks to the sensor data.

10. The method according to claim 3, wherein: Integrating information corresponding to the navigation features with at least the environmental information includes associating at least a subset of the navigation features with one or more portions of the environmental information.

11. The method according to claim 3, wherein: Integrating the information corresponding to the navigation feature with at least the environmental information is performed in real time.

12. The method according to claim 3, wherein: Integrating the information corresponding to the navigation feature with at least the environmental information is based at least in part on a spatial and / or temporal transformation between reference systems associated with at least two sensors carried by the mobile platform.

13. The method according to claim 3, wherein: Integrating information corresponding to the navigation feature with at least the environmental information is based at least in part on an estimated position and / or orientation of the mobile platform.

14. The method according to claim 13, wherein: The estimated position and / or orientation satisfies one or more accuracy criteria.

15. The method according to claim 3, wherein: The navigation feature is a first navigation feature, and wherein the method further comprises obtaining a second navigation feature based at least in part on at least one predetermined navigation map.

16. The method according to claim 15, further comprising: A position of the mobile platform relative to the predetermined navigation map is determined.

17. The method according to claim 16, further comprising: The second navigation feature is obtained from the predetermined navigation map within a certain distance of the determined position of the mobile platform.

18. The method according to claim 15, further comprising: The second navigation feature is integrated with at least the environmental information to generate the local navigation map.

19. The method according to claim 3, wherein: Generating one or more navigation commands is also based on the motion prediction.

20. The method according to claim 3, wherein: The mobile platform includes at least one of the following: an unmanned aerial vehicle (UAV), a manned aerial vehicle, an autonomous car, a self-balancing vehicle, or a robot.

21. A non-transitory computer-readable medium storing computer-executable instructions that, when executed, cause one or more processors associated with a mobile platform to perform actions comprising: obtaining real-time environmental information indicative of at least a portion of an environment within a distance of the mobile platform based at least in part on data received from a first sensor carried by the mobile platform; detecting a navigation feature based at least in part on sensor data obtained from at least one of a first sensor or a second sensor carried by the mobile platform; integrating information corresponding to the navigation feature with at least the environmental information to generate a local navigation map; as well as generating one or more navigation commands for at least partially controlling movement of the mobile platform based at least in part on the generated local navigation map; The step of integrating the information corresponding to the navigation feature with at least the environmental information to generate a local navigation map comprises: Determining a transformation relationship between a reference system associated with the navigation feature and the environmental information based on the posture information of one or more of the sensors; Projecting the navigation feature onto the environmental information to generate a local navigation map; The actions also include: Obtaining a maneuver prediction for controlling the mobile platform based on a predetermined navigation map, wherein the predetermined navigation map is a low-precision map, and the maneuver prediction includes a prediction of at least one of a lane change, a stop, or a turn within a threshold distance of the mobile platform; A next action of the mobile platform is determined based on the local navigation map, the prediction of the action of the mobile platform, and the location and / or orientation of the mobile platform.

22. The computer-readable medium of claim 21, wherein: The environmental information includes depth information.

23. The computer-readable medium of claim 21, wherein: The environment information corresponds to a 3D representation of at least a portion of an environment within a distance of the mobile platform.

24. The computer-readable medium of claim 21, wherein: The first sensor includes at least one of a lidar sensor or a stereo camera.

25. The computer-readable medium of claim 21, wherein: The second sensor includes at least one of a camera, a lidar sensor, an ultrasonic sensor, or a radar.

26. The computer-readable medium of claim 21, wherein: The navigation feature includes at least one of the following: lane markings, edge lines, guide lines, zebra crossings, traffic lights, traffic signs, guide boards, navigable surfaces or spaces, stationary obstacles, or moving obstacles.

27. The computer-readable medium of claim 21, wherein: Detecting navigational features is based at least in part on applying one or more artificial neural networks to the sensor data.

28. The computer-readable medium of claim 21, wherein: Integrating information corresponding to the navigation features with at least the environmental information includes associating at least a subset of the navigation features with one or more portions of the environmental information.

29. The computer-readable medium of claim 21, wherein: Integrating the information corresponding to the navigation feature with at least the environmental information is performed in real time.

30. The computer-readable medium of claim 21, wherein: Integrating the information corresponding to the navigation feature with at least the environmental information is based at least in part on a spatial and / or temporal transformation between reference systems associated with at least two sensors carried by the mobile platform.

31. The computer readable medium of claim 21, wherein: Integrating information corresponding to the navigation feature with at least the environmental information is based at least in part on an estimated position and / or orientation of the mobile platform.

32. The computer readable medium of claim 31, wherein: The estimated position and / or orientation satisfies one or more accuracy criteria.

33. The computer-readable medium of claim 21, wherein: The navigation feature is a first navigation feature, and wherein the actions further include obtaining a second navigation feature based at least in part on at least one predetermined navigation map.

34. The computer readable medium of claim 33, wherein: The actions also include determining a position of the mobile platform relative to the predetermined navigation map.

35. The computer readable medium of claim 34, wherein: The actions also include obtaining the second navigation feature from the predetermined navigation map within a certain distance of the determined location of the mobile platform.

36. The computer readable medium of claim 33, wherein: The actions also include integrating the second navigation feature with at least the environmental information to generate the local navigation map.

37. The computer-readable medium of claim 21, wherein: Generating one or more navigation commands is also based on the motion prediction.

38. The computer-readable medium of claim 21, wherein: The mobile platform includes at least one of the following: an unmanned aerial vehicle (UAV), a manned aerial vehicle, an autonomous car, a self-balancing vehicle, or a robot.

39. A vehicle comprising a controller programmed to at least partially control one or more movements of the vehicle, wherein: The programmed controller includes one or more processors configured to: Obtaining real-time environmental information indicative of at least a portion of an environment within a certain distance of the vehicle based at least in part on data received from a first sensor carried by the vehicle; Detecting a navigation feature based at least in part on sensor data obtained from at least one of a first sensor or a second sensor carried by the vehicle; integrating information corresponding to the navigation feature with at least the environmental information to generate a local navigation map; as well as generating one or more navigation commands for at least partially controlling movement of the vehicle based at least in part on the generated local navigation map; The step of integrating the information corresponding to the navigation feature with at least the environmental information to generate a local navigation map comprises: Determining a transformation relationship between a reference system associated with the navigation feature and the environmental information based on the posture information of one or more of the sensors; Projecting the navigation feature onto the environmental information to generate a local navigation map; The one or more processors are further configured to: Obtaining a motion prediction for controlling the vehicle based on a predetermined navigation map, wherein the predetermined navigation map is a low-precision map, and the motion prediction includes a prediction of at least one of a lane change, a stop, or a turn within a threshold distance of the vehicle; Based on the local navigation map, the predicted movement of the vehicle, the position and / or orientation of the vehicle, the next movement of the vehicle is determined.

40. The vehicle according to claim 39, wherein: The environmental information includes depth information.

41. The vehicle according to claim 39, wherein: The environmental information corresponds to a 3D representation of at least a portion of the environment within a certain distance of the vehicle.

42. The vehicle according to claim 39, wherein: The first sensor includes at least one of a lidar sensor or a stereo camera.

43. The vehicle according to claim 39, wherein: The second sensor includes at least one of a camera, a lidar sensor, an ultrasonic sensor, or a radar.

44. The vehicle according to claim 39, wherein: The navigation feature includes at least one of the following: lane markings, edge lines, guide lines, zebra crossings, traffic lights, traffic signs, guide boards, navigable surfaces or spaces, stationary obstacles, or moving obstacles.

45. The vehicle of claim 39, wherein: Detecting navigational features is based at least in part on applying one or more artificial neural networks to the sensor data.

46. ​​The vehicle of claim 39, wherein: Integrating information corresponding to the navigation features with at least the environmental information includes associating at least a subset of the navigation features with one or more portions of the environmental information.

47. The vehicle according to claim 39, wherein: Integrating the information corresponding to the navigation feature with at least the environmental information is performed in real time.

48. A vehicle according to claim 39, wherein: Integrating the information corresponding to the navigation feature with at least the environmental information is based at least in part on a spatial and / or temporal transformation between reference systems associated with at least two sensors carried by the vehicle.

49. The vehicle according to claim 39, wherein: Integrating information corresponding to the navigation feature with at least the environmental information is based at least in part on an estimated position and / or orientation of the vehicle.

50. A vehicle according to claim 49, wherein: The estimated position and / or orientation satisfies one or more accuracy criteria.

51. The vehicle according to claim 39, wherein: The navigation feature is a first navigation feature, and wherein the one or more processors are further configured to obtain a second navigation feature based at least in part on at least one predetermined navigation map.

52. A vehicle according to claim 51, wherein: The one or more processors are also configured to determine the position of the vehicle relative to the predetermined navigation map.

53. A vehicle according to claim 52, wherein: The one or more processors are also configured to obtain the second navigation feature from the predetermined navigation map within a certain distance of the determined position of the vehicle.

54. A vehicle according to claim 51, wherein: The one or more processors are further configured to integrate the second navigation feature with at least the environmental information to generate the local navigation map.

55. The vehicle according to claim 39, wherein: Generating one or more navigation commands is also based on the motion prediction.

56. The vehicle of claim 39, wherein: The vehicle includes at least one of the following: an unmanned aerial vehicle (UAV), a manned aerial vehicle, an autonomous car, a self-balancing vehicle or a robot.

Images