To generate digital map of place from data by which pixel color data is excluded, though being associated with image of place

Abstract

To provide a system by which a digital map is generated from data associated with an image of a place.SOLUTION: A system of generating a digital map from data associated with an image of a place may include a processor and a memory. The memory may store a generation module and a communication module. The generation module may include an instruction for generating the digital map from the data associated with the image of the place, with respect to the processor. The data concerning the image may include information about (1) a posture of a camera by which the image is generated, and (2) at least one of (a) a lane boundary of a lane of a road in the image, (b) a road boundary of the road, and (c) a position of a point concerning a mark in the image, though pixel color data may be excluded. The communication module may include an instruction for transmitting the digital map to a vehicle so as to be used for controlling movement of the vehicle, with respect to the processor.SELECTED DRAWING: Figure 15

Description

[Technical field]

[0001] The techniques of this disclosure are directed to generating digital maps of a place from data that relates to an image of the place but excludes pixel color data. [Background technology]

[0002] A digital map may be an electronic representation of a traditional paper road map. For example, a navigation system in an automobile may generate a turn-by-turn navigation service using information received from the digital map and information received from a Global Navigation Satellite System (GNSS). The turn-by-turn navigation service may provide a route between a start point and a destination point. The position of the vehicle determined by such a turn-by-turn navigation service may be within one meter of the actual position.

[0003] More recently, technologies have been developed to automate one or more operations of one or more vehicle systems to control vehicle movement. Such technologies may control such movement using information received from digital maps. However, such digital maps may be required to indicate the location of objects with an accuracy within a decimeter range. Thus, technological developments for automating the control of vehicle movement have been accompanied by efforts to improve the accuracy of digital maps. This has led to the creation of High Definition (HD) maps.

[0004] An HD map may be a digital map that includes additional information to improve the accuracy required to automate the control of vehicle movement. An HD map may be characterized by having layers of additional information. Each layer of additional information may relate to a particular category of additional information. The layers may include, for example, a layer of a base map, a layer of a geometric map, and a layer of a semantic map. The base map, the geometric map, and the semantic map may include information about static aspects of a place.

[0005] The geometric map may be generated, for example, using simultaneous localization and mapping (SLAM) techniques. SLAM techniques may use proprioceptive information to estimate the vehicle's pose (i.e., position and orientation) and sensory information to refine the pose estimate. Typically, the proprioceptive information may be one or more of GNSS information, inertial measurement unit (IMU) information, odometry information, or the like. For example, the odometry information may be values ​​contained in signals sent to a vehicle system (e.g., an accelerometer). The sensory information often may be one or more of point cloud information from a ranging sensor (e.g., a light detection and ranging (LiDAR) system), image data from one or more images by one or more image sensors or cameras, or the like. The geometric map may include, for example, a ground map of a maintained surface (e.g., a drivable surface (e.g., a road)) used by vehicles and pedestrians, as well as a voxelized geometric representation of three-dimensional objects at the location.

[0006] A semantic map may include semantic information about objects contained in the place. Objects may include, for example, landmarks. Landmarks may be, for example, features that can be easily re-identified and differentiated from other features in the place. The term landmark may differ from the traditional use of the term landmark in the context of indicating the location of an object with an accuracy within one decimeter. For example, landmarks may include lane boundaries, road boundaries, intersections, crosswalks, bus lanes, parking spots, signs, markings painted on roads, traffic lights, or the like.

[0007] Because HD maps may be used to control vehicle movement, not only do object positions need to be represented on the HD map with high accuracy, but the HD map may also be required to be updated at a high rate to take into account changes in the positions of objects or objects that are required to be represented on the HD map. Summary of the Invention

[0008] In an embodiment, a system for generating a digital map from data associated with an image of a place may include a processor and a memory. The memory may store a generation module and a communication module. The generation module may include instructions that, when executed by the processor, cause the processor to generate a digital map from data associated with an image of the place. The data about the image may exclude pixel color data, but may include information regarding (1) the pose of a camera that generated the image, and (2) one or more of: (a) lane boundaries of lanes of a road in the image, (b) road boundaries of a road, or (c) locations of points relative to landmarks in the image. The communication module may include instructions that, when executed by the processor, cause the processor to transmit the digital map to a vehicle for use in controlling movement of the vehicle.

[0009] In another embodiment, a method of generating a digital map from data associated with an image of a place may include generating, by a processor, a digital map from data associated with an image of a place. The data for the image may exclude pixel color data, but may include information regarding (1) a pose of a camera that generated the image, and (2) one or more of (a) lane boundaries of lanes of a road in the image, (b) road boundaries of a road, or (c) locations of points relative to landmarks in the image. The method may include transmitting, by the processor, the digital map to a vehicle for use in controlling movement of the vehicle.

[0010] In another embodiment, a non-transitory computer readable medium for generating a digital map from data associated with an image of a place may include instructions that, when executed by one or more processors, cause the one or more processors to generate a digital map from data associated with an image of a place. The data for the image may exclude pixel color data, but may include information regarding (1) a pose of a camera that generated the image, and (2) one or more of: (a) lane boundaries of lanes of a road in the image, (b) road boundaries of a road, or (c) locations of points relative to landmarks in the image. The non-transitory computer readable medium may include instructions that, when executed by the one or more processors, transmit the digital map to a vehicle for use in controlling movement of the vehicle. [Brief description of the drawings]

[0011] The accompanying drawings, which are incorporated herein by reference and form a part of this specification, illustrate various systems, methods, and other embodiments of the present disclosure. It will be understood that the boundaries of elements shown in the figures (e.g., boxes, groups of boxes, or other shapes) represent one embodiment of the boundaries. In some embodiments, one element may be designed as multiple elements, or multiple elements may be designed as one element. In some embodiments, an element shown as an internal component of another element may be implemented as an external component, and vice versa. Additionally, elements may not be drawn to scale.

[0012] [Figure 1] FIG. 1 includes diagrams illustrating example environments for generating digital maps of places from data related to images of the places but excluding pixel color data, in accordance with techniques of this disclosure. [Diagram 2] FIG. 2 includes diagrams showing example images generated at a first time (t1) by a forward-facing camera mounted on a vehicle in accordance with the techniques of the present disclosure. [Diagram 3] 10 includes diagrams showing example images generated at a second time (t2) by a forward-facing camera mounted on a vehicle in accordance with the techniques of the present disclosure. [Figure 4] 3 includes a diagram showing examples of landmark keypoints in the image included in FIG. 2 in accordance with the techniques of the present disclosure. [Diagram 5] 4 includes a diagram showing example landmark keypoints in the image included in FIG. 3 in accordance with the techniques of this disclosure. [Figure 6A] FIG. 2 includes an example table showing data related to images of places, in accordance with the techniques of this disclosure. [Figure 6B] FIG. 2 includes an example table showing data related to images of places, in accordance with the techniques of this disclosure. [Figure 7] FIG. 1 is a block diagram illustrating an example system for generating a digital map from data associated with images of a place, in accordance with techniques of this disclosure. [Figure 8] 7A and 7B include diagrams showing examples of locations of landmark keypoints associated with items of data contained in the tables included in FIGS. 6A and 6B in accordance with the techniques of the present disclosure. [Figure 9] FIG. 1 includes an example of a digital map in accordance with the techniques of this disclosure. [Figure 10] 2 includes a diagram showing an example of a hexagonal grid superimposed on the environment shown in FIG. 1 in accordance with the techniques of this disclosure. [Figure 11] 2 is a diagram including example aerial images of the environment shown in FIG. 1 in accordance with the techniques of this disclosure. [Figure 12] 2 is a diagram including an example of a first temporary rectified aerial image of the environment shown in FIG. 1 in accordance with the techniques of this disclosure. [Figure 13] 2 is a diagram including an example of a second temporarily rectified aerial image of the environment shown in FIG. 1 in accordance with the techniques of this disclosure. [Figure 14] 1 includes diagrams illustrating example operations for moving the positions of points associated with generating a version of a digital map to positions of pixels on a segmentation map in accordance with techniques of this disclosure. [Figure 15] FIG. 1 includes a flow diagram illustrating an example method associated with generating a digital map of a place from data related to an image of the place but excluding pixel color data, in accordance with techniques of this disclosure. [Figure 16A] FIG. 1 includes a flow diagram illustrating a first example of a method associated with aligning positions of points associated with the generation of a version of a digital map of a place to positions of points in an aerial image of the place to generate an aerial image-aligned digital map of the place, in accordance with the techniques of the present disclosure. [Figure 16B] FIG. 1 includes a flow diagram illustrating a first example of a method associated with aligning positions of points associated with the generation of a version of a digital map of a place to positions of points in an aerial image of the place to generate an aerial image-aligned digital map of the place, in accordance with the techniques of the present disclosure. [Figure 16C] FIG. 1 includes a flow diagram illustrating a first example of a method associated with aligning positions of points associated with the generation of a version of a digital map of a place to positions of points in an aerial image of the place to generate an aerial image-aligned digital map of the place, in accordance with the techniques of the present disclosure. [Figure 16D] FIG. 1 includes a flow diagram illustrating a first example of a method associated with aligning positions of points associated with the generation of a version of a digital map of a place to positions of points in an aerial image of the place to generate an aerial image-aligned digital map of the place, in accordance with the techniques of the present disclosure. [Figure 17] FIG. 1 includes a flow diagram illustrating an example of a method associated with detecting, in a copy of an aerial image, a representation of road boundaries in a smooth surface used by vehicles and pedestrians, in accordance with the techniques of the present disclosure. [Figure 18] A figure including a flow diagram illustrating a second example of a method associated with aligning positions of points associated with the generation of a version of a digital map of a place to positions of points in an aerial image of the place to generate an aerial image-aligned digital map of the place, in accordance with the techniques of the present disclosure. [Figure 19] FIG. 1 includes a block diagram illustrating an example of elements located in a vehicle in accordance with the techniques of this disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] Simultaneous localization and mapping (SLAM) is a phrase that may refer to a technique that allows a mobile robot (e.g., an automatic or autonomous vehicle) to navigate through an unknown location while simultaneously determining (i.e., localization) the vehicle's pose (i.e., position and orientation) in the location and mapping the location. Typically, SLAM techniques may work at discrete time units and use proprioceptive information to estimate the vehicle's pose and sensory information to refine the pose estimate. Typically, the proprioceptive information may be one or more of Global Navigation Satellite System (GNSS) information, Inertial Measurement Unit (IMU) information, odometry information, or the like. For example, the odometry information may be values ​​contained in signals transmitted to a vehicle system (e.g., an accelerometer). The sensory information often may be one or more of point cloud information from a ranging sensor (e.g., a light detection and ranging (LiDAR) system), image data from one or more images by one or more image sensors or cameras, or the like.

[0014] For example, for SLAM techniques that use point cloud information from a ranging sensor, the ranging sensor may provide the vehicle with distance and orientation to objects at the location, and the SLAM technique may function to identify salient objects as landmarks. For example, for SLAM techniques that use image data from one or more images from one or more image sensors or cameras, which may be referred to as visual SLAM, the distance and orientation to objects may be determined using photogrammetric range imaging techniques (e.g., Structure from Motion (SfM) techniques) applied to a series of two-dimensional images. Because cameras may be less expensive than LiDAR devices, and more vehicles are equipped with cameras than LiDAR devices, significant effort has been expended to develop visual SLAM for use in generating geometric maps as layers of high-definition (HD) maps used to control vehicle movement.

[0015] Furthermore, while initially SLAM techniques were developed to function in real time (i.e., simultaneous localization and mapping), the use of SLAM techniques to generate geometric maps has led to the development of SLAM techniques that can function in environments other than moving vehicles. In such SLAM techniques, records of proprioceptive and sensory information may be used. Such SLAM techniques may be referred to as offline SLAM. By using the records of proprioceptive and sensory information, corrections to the estimate of the vehicle's pose may be made simultaneously in one or more finite, discrete series of time units in which the SLAM technique functioned. Such corrections may be achieved by various procedures, which may include, for example, one or more techniques of optimization. The optimization may result in more accurate corrections to the estimate of the vehicle's pose if one or more objects included in the record of sensory information are included in multiple instances of the record. (Such a situation may be referred to as closing the loop.) That is, corrections to the estimate of the vehicle's pose may be more accurate for optimizations in which the same object is included in the record of sensory information in multiple instances than for optimizations in which the same object is not included in the record of sensory information in multiple instances.

[0016] The proprioceptive and sensory information records may be obtained, for example, by one or more probe vehicles. A probe vehicle may be a vehicle that purposefully makes one or more passes through a location to obtain the proprioceptive and sensory information records. Furthermore, multiple instances of proprioceptive and sensory information records may be obtained during each of the one or more passes. When (1) a probe vehicle obtains multiple instances of proprioceptive and sensory information records during a pass through a location, (2) multiple probe vehicles pass through a location, or (3) both, the likelihood that one or more objects included in the sensory information records are included in the multiple instances of the record may increase, such that the optimization result includes a loop closure situation.

[0017] Because HD maps may be used to control the movement of a vehicle, the inclusion of representations of certain objects (e.g., landmarks) in HD maps may be more important than other representations. Such important landmarks may include, for example, lane boundaries, road boundaries, intersections, crosswalks, bus lanes, parking spots, signs, markings painted on roads, traffic lights, or the like. The techniques of this disclosure are directed to generating digital (e.g., HD) maps of a place from data associated with images of the place. The digital map may be generated from data associated with the images. The data for one of the images may exclude pixel color data, but may include information regarding (1) the pose of the camera that generated the image, and (2) one or more of (a) lane boundaries of lanes of a road in the image, (b) road boundaries of a road, or (c) the location of a point relative to another landmark in the image. The digital map may be transmitted to a first vehicle for use in controlling the movement of the first vehicle.

[0018] Further, for example, data related to images may be received from one or more second vehicles (e.g., probe vehicles). One or more cameras may be attached to one or more second vehicles. For example, a camera of the one or more cameras may generate images. For example, the images may be generated at a specific generation rate. For example, the specific generation rate may be 10 Hertz. For example, the camera may be a component in a lane-keeping assist (LKA) system. For example, (1) data related to images may be received from one or more second vehicles (e.g., probe vehicles) at a first time by a system implementing the technology of the present disclosure, (2) a digital map generated from the data by the system implementing the technology of the present disclosure may be transmitted to the first vehicle at a second time, and (3) a difference between the first time and the second time may be less than a specific duration. For example, the specific duration may be 30 minutes.

[0019] The techniques of the present disclosure may generate data related to an image of a location, for example, using visual SLAM techniques. For example, a camera mounted on a second vehicle (e.g., a probe vehicle) may generate an image. For example, the image may be generated at a particular generation pace. For example, the particular generation pace may be 10 Hertz. Objects in the image may be detected, for example, using object detection techniques. Objects in the image may be recognized, for example, using object recognition techniques. Semantic information may be associated with the object. For example, objects that qualify as landmarks may be determined. For example, landmarks may include lane boundaries, road boundaries, intersections, crosswalks, bus lanes, parking spots, signs, markings painted on roads, traffic lights, or the like.

[0020] A lane boundary may separate one lane of a road from another lane of the road. A lane boundary may be indicated, for example, by one or more of road surface markings, observations of pavement differences on the road, observations of vehicle trajectories, or the like. A road surface marking for a lane boundary may be, for example, a lane marking. A lane marking may be, for example, a series of dashed line segments along the lane boundary.

[0021] A road boundary may separate a maintained surface used by vehicles and pedestrians (e.g., a drivable surface (e.g., a road)) from other surfaces. A road boundary may be indicated by one or more of a road surface marking, a curb, an observation regarding differences in maintenance between adjacent surfaces, or the like. A road surface marking for a road boundary may be, for example, a continuous line along the road boundary.

[0022] (1) The location, non-depiction, of landmarks in HD maps used to control vehicle movement needs to be represented with high accuracy, and (2) images of a place may be generated at a certain generation pace, so that a depiction of a landmark may be included in some of the images of the place. However, for a certain image of the place, the location of either a lane boundary of a road lane, a road boundary of a road, or another landmark in the image may be represented by a location of a point relative to the lane boundary, road boundary, or other landmark. For example, the location of a point relative to the lane boundary, road boundary, or landmark may be related to the location of a key point of an object in the image that represents the lane boundary, road boundary, or landmark. A key point may be a point in an object that may be repeatedly detected under different imaging conditions. A point in an object may be extracted, for example, using a key point extraction technique.

[0023] The second vehicle (e.g., a probe vehicle) may, for example, use proprioceptive information (e.g., one or more of GNSS information, IMU information, odometry information, or the like) to estimate the pose (i.e., position and orientation) of a camera (e.g., mounted on the second vehicle). The second vehicle (e.g., a probe vehicle) may, for example, use the results of photogrammetric range imaging techniques (e.g., SfM techniques) as sensory information to determine distances and orientations to landmarks (e.g., keypoints) in the image. The location of a point (e.g., keypoint) relative to the landmark may be determined using, for example, (1) the pose of a camera (e.g., mounted on the second vehicle) and (2) the distances and orientations to the landmarks (e.g., keypoints) in the image.

[0024] Thus, the data associated with the images of the location may exclude pixel color data for certain of the images, but may include information regarding (1) the pose of the camera that generated the image, and (2) one or more positions of points in the image relative to the landmark. For example, the amount of data associated with the images may be less than a threshold amount. For example, the threshold amount may be 300 bytes. For example, the landmark may be a sign. For example, the data associated with the images may include information regarding the sign. For example, the information regarding the sign may include (1) a latitude location, a longitude location, and an altitude relative to a center of the sign, (2) a height of the sign, and (3) a width of the sign. Additionally or alternatively, for example, the information regarding the sign may include information regarding a message conveyed by the sign. For example, the data associated with the images may be generated by an autonomous driving system of active safety technology and advanced driver assistance systems (ADAS). For example, the autonomous driving system may be a third generation Toyota Safety Sense™ system (TSS3).

[0025] For example, for a vehicle of the one or more second vehicles (e.g., probe vehicles), a batch transmission of data related to images generated by a camera of the vehicle of the one or more second vehicles (e.g., probe vehicles) may be received at a particular duration. For example, the particular duration may be 30 seconds. For example, the batch transmission may be received at a particular communication pace. For example, the particular communication pace may be once every 30 seconds.

[0026] Techniques of this disclosure may generate a digital (e.g., HD) map of a place from data associated with images of the place, e.g., using offline SLAM techniques. For example, the digital map may be generated by processing the data associated with the images using one or more data association techniques to determine correspondences between locations of points (e.g., keypoints) associated with particular objects (e.g., landmarks) included in a first one of the images and locations of points (e.g., keypoints) associated with particular objects (e.g., landmarks) included in a second one of the images.

[0027] For example, a digital (e.g., HD) map may be generated by processing data associated with an image of a location using one or more SLAM techniques. For example, the location may be a particular region. Advantageously, processing data for the particular region may limit the number of data association operations performed to generate a digital (e.g., HD) map of the location. For example, the shape of the particular region may be defined by seven regular hexagons. Seven regular hexagons may be arranged with one of the seven regular hexagons at a center location and another of the seven regular hexagons arranged on the sides of the one hexagon. For example, each side of the one hexagon at the center location may be directly adjacent to the sides of another hexagon. In this manner, the particular region may be a region in a hexagonal grid. Advantageously, the use of a hexagonal grid may simplify calculations when a digital (e.g., HD) map is constructed over the particular region. The distance between the center of the particular hexagon and the center of any adjacent hexagon may be the same as the distance between the center of the particular hexagon and the center of any other adjacent hexagon. In contrast, for a square grid, calculations regarding the distance between the center of a particular square and the center of any adjacent squares may require consideration of whether the adjacent square centers are perpendicular or diagonal to the center of the particular square.Furthermore, advantageously, when digital (e.g., HD) maps are constructed over a particular region, a hexagonal grid fits better than a square grid to the surface of a sphere (e.g., a sphere).

[0028] For example, a digital (e.g., HD) map may be generated by categorizing images into keyframes and processing the keyframes using at least one SLAM optimization technique. For example, (1) a first one of the keyframes may feature a first criterion, (2) a second one of the keyframes may feature a second criterion, and (3) a difference between the first criterion and the second criterion may be greater than a threshold. The first criterion may relate to a value of data included in the first keyframe. The second criterion may relate to a value of data included in the second keyframe. For example, the count of images included in a keyframe may be a function of a distance traveled by a second vehicle (e.g., a probe vehicle).

[0029] FIG. 1 includes a diagram illustrating an example of an environment 100 for generating a digital map of a place from data related to an image of the place but excluding pixel color data, according to the techniques of this disclosure. For example, the environment 100 may include a first road 102 (disposed along a meridian), a second road 104 (disposed along a latitude), and a parking lot 106. For example, an intersection 108 may be formed by the first road 102 and the second road 104. For example, the intersection 108 may be a T-junction. For example, the second road 104 may connect the first road 102 to the parking lot 106. For example, a building 110 may be located within the parking lot 106. For example, the environment 100 may include a first road sign 112 and a second road sign 114. For example, the first road sign 112 may be located at the southeast corner of the intersection 108. For example, the first road sign 112 may be a "No Parking" road sign. For example, the second road sign 114 may be located 4 meters south of the first road sign 112. For example, the second road sign 114 may be a "Speed ​​Limit 25" road sign.

[0030] For example, the first road 102 may include lane #1 116 for southbound traffic, lane #1 118 for northbound traffic, and lane #2 120 for northbound traffic. For example, lane #1 116 may be bounded on the west side by road boundary 122. For example, lane #2 120 may be bounded on the east side by road boundary 124 south of the intersection 108 and by road boundary 126 north of the intersection 108. For example, lane #1 116 may be bounded on the east side and lane #1 118 may be bounded on the west side by lane boundary 128. For example, lane boundary 128 may be a lane marking 130 indicating a separation between lanes flowing in opposite directions of traffic. For example, lane marking 130 may be two solid yellow lines. For example, lane #1 118 may be bounded on the east side and lane #2 120 may be bounded on the west side by lane boundary 132. For example, lane boundary 132 may be lane markings 134 indicating a separation between lanes flowing in the same direction of traffic. For example, lane markings 134 may be dashed white lines. For example, from north to south, lane markings 134 may include a first portion 136, a second portion 138, a third portion 140, a fourth portion 142, a fifth portion 144, a sixth portion 146, a seventh portion 148, and an eighth portion 150.

[0031] For example, the second road 104 may include lane #1 152 for westbound traffic and lane #1 154 for eastbound traffic. For example, lane #1 152 may be bounded on the north side by road boundary 156. For example, lane #1 154 may be bounded on the south side by road boundary 158. For example, lane #1 152 may be bounded on the south side and lane #1 154 may be bounded on the north side by lane boundary 160. For example, lane boundary 160 may be a lane marking 162 indicating a separation between lanes flowing in opposite directions of traffic. For example, lane marking 162 may be two solid yellow lines.

[0032] For example, environment 100 may include a first vehicle 164, a second vehicle 166, and a third vehicle 168. For example, forward facing camera 170 may be mounted on first vehicle 164. For example, forward facing camera 172 may be mounted on second vehicle 166. For example, communication device 174 may be located on first vehicle 164. For example, communication device 176 may be located on second vehicle 166. For example, communication device 178 may be located on third vehicle 168.

[0033] For example, the environment 100 may include a system 180 that generates a digital map from data related to images of a place. For example, the system 180 may include a communication device 182.

[0034] For example, the environment 100 may include an area 184. For example, the area 184 may include a portion of the road boundary 124, a portion of the road boundary 158, and the first road mark 112.

[0035] For example, at a first time (t1), a first vehicle 164 may be located immediately behind the third portion 140 in lane #2 120, a second vehicle 166 may be located immediately behind the eighth portion 150 in lane #2 120, and a third vehicle 168 may be located approximately 15 miles (approximately 24.14 km) behind the second vehicle 166 in lane #2 120.

[0036] For example, at a second time (t2), a first vehicle 164 may be located alongside the third portion 140 in lane #2 120, a second vehicle 166 may be located alongside the eighth portion 150 in lane #2 120, and a third vehicle 168 may be located approximately 15 miles (approximately 24.14 km) behind the second vehicle 166 in lane #2 120.

[0037] For example, at a third time (t3), the second vehicle 166 may be located immediately behind the third portion 140 in lane #2 120. That is, at the third time (t3), the position of the second vehicle 166 may be at the position of the first vehicle 164 at the first time (t1).

[0038] For example, at a fourth time (t4), the second vehicle 166 may be located side-by-side with the third portion 140 in lane #2 120. That is, at the fourth time (t4), the position of the second vehicle 166 may be at the position of the first vehicle 164 at the second time (t2).

[0039] As described above, objects in an image may be detected, for example, using object detection techniques, and recognized, for example, using object recognition techniques. Semantic information may be associated with the objects, and objects that qualify as landmarks may be determined. For example, landmarks may include lane boundaries, road boundaries, signs, or the like.

[0040] 2 includes a diagram illustrating an example of an image 200 generated at a first time (t1) by a forward-facing camera 170 mounted on a first vehicle 164 in accordance with the techniques of this disclosure. For example, image 200 may include the following landmarks: first road sign 112, second road sign 114, road boundary 122, road boundary 124, lane boundary 128, second portion 138, third portion 140, and road boundary 158. For example, image 200 may also be generated at a third time (t3) by a forward-facing camera 172 mounted on a second vehicle 166.

[0041] 3 includes a diagram illustrating an example of an image 300 generated at a second time (t2) by forward-facing camera 170 mounted on first vehicle 164 in accordance with the techniques of this disclosure. For example, image 300 may include the following landmarks: first road sign 112, road boundary 122, road boundary 124, lane boundary 128, second portion 138, road boundary 158, and lane boundary 160. For example, image 300 may also be generated at a fourth time (t4) by forward-facing camera 172 mounted on second vehicle 166.

[0042] For example, the images (i.e., image 200 and image 300) generated by front-facing camera 170 (or front-facing camera 172) may be images in a series of images generated by front-facing camera 170 (or front-facing camera 172). For example, the images (i.e., image 200 and image 300) generated by front-facing camera 170 (or front-facing camera 172) may be generated at a particular generation pace. For example, the particular generation pace may be 10 Hertz.

[0043] As mentioned above, the location of the landmark may be represented by the location of a point relative to the landmark. For example, the location of a point relative to the landmark may relate to the location of a keypoint of an object in the image that represents the landmark. A keypoint may be a point in the object that may be repeatedly detected under different imaging conditions. The point in the object may be extracted, for example, using a keypoint extraction technique.

[0044] 4 includes a diagram illustrating example keypoints 400 of landmarks in an image 200 in accordance with the techniques of this disclosure. For example, the keypoints 400 may include a first keypoint 402 of the first road sign 112, a second keypoint 404 of the second road sign 114, a third keypoint 406 of the road boundary 122, a fourth keypoint 408 of the road boundary 124, a fifth keypoint 410 of the lane boundary 128, a sixth keypoint 412 of the second portion 138, a seventh keypoint 414 of the third portion 140, and an eighth keypoint 416 of the road boundary 158. For example, because only a portion of the road boundary 122, road boundary 124, and lane boundary 128 captured by the forward-facing camera 170 (or forward-facing camera 172) is included in the image 200, the third keypoint 406, the fourth keypoint 408, and the fifth keypoint 410 may relate to a portion of the road boundary 122, road boundary 124, and lane boundary 128 captured by the forward-facing camera 170 (or forward-facing camera 172) at the first time (t1) (or the third time (t3)).

[0045] 5 includes a diagram illustrating example keypoints 500 of landmarks in an image 300 in accordance with the techniques of this disclosure. For example, the keypoints 500 may include the first keypoint 402 of the first road sign 112, the ninth keypoint 502 of the road boundary 122, the tenth keypoint 504 of the road boundary 124, the eleventh keypoint 506 of the lane boundary 128, the sixth keypoint 412 of the second portion 138, the eighth keypoint 416 of the road boundary 158, and the twelfth keypoint 508 of the lane boundary 160. For example, the ninth key point 502, the tenth key point 504, and the eleventh key point 506 may relate to the portions of the road boundary 122, the road boundary 124, and the lane boundary 128 captured by the forward facing camera 170 (or the forward facing camera 172) at the second time (t2) (or the fourth time (t4)) because only the portions of the road boundary 122, the road boundary 124, and the lane boundary 128 captured by the forward facing camera 170 (or the forward facing camera 172) are included in the image 300. Furthermore, the portions of the portions of the road boundary 122, the road boundary 124, and the lane boundary 128 captured by the forward facing camera 170 (or the forward facing camera 172) included in the image 300 may differ from the portions of the road boundary 122, the road boundary 124, and the lane boundary 128 captured by the forward facing camera 170 (or the forward facing camera 172) included in the image 200.

[0046] As described above, the location of a point (e.g., a key point) relative to a landmark may be determined using, for example, (1) the pose (i.e., position and orientation) of a camera (e.g., forward-facing camera 170 mounted on first vehicle 164 or forward-facing camera 172 mounted on second vehicle 166) (e.g., mounted on the second vehicle) and (2) the distance and orientation to the landmark (e.g., key point) in the image. The second vehicle may, for example, use proprioceptive information (e.g., one or more of GNSS information, IMU information, odometry information, or the like) to estimate the pose of the camera (e.g., mounted on the second vehicle). The second vehicle may, for example, use results of photogrammetric range imaging techniques (e.g., SfM techniques) as perceptual information to determine the distance and orientation to the landmark (e.g., key point) in the image.

[0047] Thus, as described above, data associated with the images of a location may, for certain ones of the images, exclude pixel color data, but may include information regarding (1) the pose of the camera that generated the image, and (2) one or more locations of points (e.g., key points) relative to the landmark in the image. For example, if the landmark is a sign, the data associated with the image may include information regarding the sign. For example, the information regarding the sign may include (1) the latitude position, longitude position, and altitude relative to the center of the sign, (2) the height of the sign, and (3) the width of the sign. Additionally or alternatively, for example, the information regarding the sign may include information regarding a message conveyed by the sign.

[0048] 6A and 6B include an example of a table 600 showing data associated with images of a location in accordance with the techniques of this disclosure. The table 600 may include (1) a first table 602 showing an item of data associated with an image 200 generated at a first time (t1) by a forward-facing camera 170 mounted on a first vehicle 164, (2) a second table 604 showing an item of data associated with an image 300 generated at a second time (t2) by a forward-facing camera 170 mounted on a first vehicle 164, (3) a third table 606 showing an item of data associated with an image 200 generated at a third time (t3) by a forward-facing camera 172 mounted on a second vehicle 166, and (4) a fourth table 608 showing an item of data associated with an image 300 generated at a fourth time (t4) by a forward-facing camera 172 mounted on a second vehicle 166.

[0049] For example, (1) a first table 602 may include a pose 610 of a forward-facing camera 170 mounted on a first vehicle 164 at a first time (t1), (2) a second table 604 may include a pose 612 of a forward-facing camera 170 mounted on a first vehicle 164 at a second time (t2), (3) a third table 606 may include a pose 614 of a forward-facing camera 172 mounted on a second vehicle 166 at a third time (t3), and (4) a fourth table 608 may include a pose 616 of a forward-facing camera 172 mounted on a second vehicle 166 at a fourth time (t4).

[0050] Each of the first table 602 and the third table 606 may include data related to, for example, a first keypoint 402, a second keypoint 404, a third keypoint 406, a fourth keypoint 408, a fifth keypoint 410, a sixth keypoint 412, a seventh keypoint 414, and an eighth keypoint 416.

[0051] The second table 604 and the fourth table 608 may each include data related to, for example, the first keypoint 402, the ninth keypoint 502, the tenth keypoint 504, the eleventh keypoint 506, the sixth keypoint 412, the eighth keypoint 416, and the twelfth keypoint 508.

[0052] One or more conditions associated with generating data related to an image of a location may cause information related to, for example, (1) the pose of the camera, (2) one or more positions of the points relative to the landmarks, or (3) both to include one or more errors. For example, errors in proprioceptive information (e.g., one or more of GNSS information, IMU information, odometry information, or the like) may cause information related to the pose of the camera to include one or more errors. For example, changes in illumination related to one or more of the landmarks at one or more of a first time (t1), a second time (t2), a third time (t3), or a fourth time (t4) may cause results of a photogrammetric range imaging technique (e.g., SfM technique) to include one or more errors, such that distances and orientations to landmarks (e.g., key points) in an image determined from the photogrammetric range imaging technique (e.g., SfM technique) include one or more errors. In light of the description herein, it will be appreciated by those skilled in the art that one or more other circumstances may cause information relating to (1) the pose of the camera, (2) one or more locations of the points relative to the landmarks, or (3) both to contain one or more other errors. Individually or cumulatively, these errors may cause the information contained in an item of data associated with an image generated at a given time by a particular source (e.g., forward-facing camera 170 mounted on first vehicle 164 or forward-facing camera 172 mounted on second vehicle 166) to differ from a corresponding item of data associated with an image generated (1) at a different time, (2) by a different particular source, or (3) both. This situation is illustrated by the values ​​of the items of data contained in table 600 included in Figures 6A and 6B.

[0053] As described above, the first vehicle 164, the second vehicle 166, or both may transmit data related to images to a system 180 that generates a digital map from data related to images of a location. For example, a communication device 174 located in the first vehicle 164 may transmit data generated at a first time (t1) and a second time (t2) (e.g., the first table 602 and the second table 604) to a communication device 182 included in the system 180. Similarly, for example, a communication device 176 located in the second vehicle 166 may transmit data generated at a third time (t3) and a fourth time (t4) (e.g., the third table 606 and the fourth table 608) to a communication device 182 included in the system 180.

[0054] 7 is a block diagram illustrating an example of a system 700 for generating a digital map from data associated with images of a place, in accordance with techniques of this disclosure. For example, the system 700 may be the system 180 shown in FIG. 1. The system 700 may include, for example, a processor 702 and a memory 704. The memory 704 may be communicatively coupled to the processor 702. For example, the memory 704 may store a generating module 706 and a communicating module 708.

[0055] For example, generation module 706 may include instructions that function to control processor 702 to generate a digital map from the data. The data for a given one of the images may exclude pixel color data, but may include information regarding (1) the pose of the camera that generated the image, and (2) one or more of: (a) lane boundaries of lanes of a road in the image, (b) road boundaries of a road, or (c) the location of a point relative to another landmark in the image.

[0056] For example, one or more of the locations of the points relative to the lane boundaries, road boundaries, or landmarks may be related to the locations of key points of an object in the image that represents the lane boundaries, road boundaries, or landmarks. A key point may be a point in the object that may be repeatedly detected under different imaging conditions. The point in the object may be extracted, for example, using a key point extraction technique.

[0057] For example, the landmark may be a sign. For example, the data associated with the image may include information about the sign. For example, the information about the sign may include (1) a latitudinal position, a longitudinal position, and an altitude relative to a center of the sign, (2) a height of the sign, and (3) a width of the sign. Additionally or alternatively, for example, the information about the sign may include information about a message conveyed by the sign.

[0058] For example, the communications module 708 may function to control the processor 702 to include instructions causing the processor 702 to transmit a digital map to the first vehicle for use in controlling the movement of the first vehicle. With reference to FIG. 1 , for example, the instructions causing the processor 702 to transmit the digital map may cause the communications device 182 included in the system 180 to transmit the digital map to the communications device 178 located in the third vehicle 168.

[0059] Further, for example, communications module 708 may include instructions that function to control processor 702 to receive data related to images from one or more second vehicles (e.g., the camera may include one or more cameras, and the one or more cameras may be mounted on one or more second vehicles).

[0060] For example, a camera of the one or more cameras may be a component in a lane-keeping assist (LKA) system. For example, the images may be generated at a particular generation rate. For example, the particular generation rate may be 10 Hertz. For example, the amount of data for the images may be less than a threshold amount. For example, the threshold amount may be 300 bytes. For example, the data related to the images may be generated by an active safety technology and an advanced driver assistance system (ADAS) autonomous driving system. For example, the autonomous driving system may be a third generation Toyota Safety Sense™ system (TSS3).

[0061] With reference to Figures 1, 6A, and 6B, for example, instructions to cause processor 702 to receive data may cause communication device 182 included in system 180 to receive data (e.g., items of data included in table 600) from one or more of communication device 174 located in first vehicle 164 or communication device 176 located in second vehicle 166.

[0062] For example, (1) an operation related to an instruction to receive data by the system 700 (e.g., the system 180 shown in FIG. 1 ) may be configured to occur at a first time, (2) an operation related to an instruction to transmit a digital map by the system 700 (e.g., the system 180 shown in FIG. 1 ) may be configured to occur at a second time, and (3) a difference between the first time and the second time may be less than a specific duration. For example, the specific duration may be 30 minutes. For example, (1) data from one or more of the first vehicle 164 or the second vehicle 166 (e.g., an item of data included in the table 600 included in FIG. 6A and FIG. 6B ) may be received by the communication module 708 at a first time, (2) the digital map may be transmitted by the communication module 708 to the third vehicle 168 at a second time, and (3) a difference between the first time and the second time may be less than a specific duration (e.g., 30 minutes).

[0063] For example, the instructions to receive may include instructions to receive, at a particular communication pace, from a vehicle of the one or more second vehicles, a transmission of a batch of data associated with images generated by a corresponding camera for a duration associated with the particular communication pace, e.g., the particular communication pace may be once every 30 seconds. 1-3, 6A, and 6B, for example, if (1) forward-facing camera 170 mounted on first vehicle 164 generates images at a particular generation rate (e.g., 10 Hertz), (2) image 200 is generated by forward-facing camera 170 at a first time (t1), (3) image 300 is generated by forward-facing camera 170 at a second time (t2), and (4) communication device 174 located on first vehicle 164 transmits batches of data related to the images generated by forward-facing camera 170 at a particular communication rate (e.g., once every 30 seconds), image 200 and image 300 may be a subset of images related to the batch (e.g., 300 images), and the items of data included in first table 602 (related to image 200 generated by forward-facing camera 170) and second table 604 (related to image 300 generated by forward-facing camera 170) may be a subset of the data included in the batch. That is, operations performed by the first vehicle 164 may include: (1) generating images at a particular generation rate (e.g., 10 Hertz); (2) generating data associated with the images at a particular generation rate; (3) for each image, storing the data associated with the image; and (4) transmitting the data associated with the images at a particular communication rate (e.g., once every 30 seconds) that is generated for a duration associated with a particular communication rate (e.g., 30 seconds).

[0064] 8 includes a diagram 800 illustrating example locations of landmark points (e.g., keypoints) associated with items of data included in the table 600 included in FIGS. 6A and 6B in accordance with the techniques of this disclosure. For example, diagram 800 may include (1) a location 802 of a first keypoint 402 determined at a first time (t1), (2) a location 804 of a second keypoint 404 determined at a first time (t1), (3) a location 806 of a third keypoint 406 determined at a first time (t1), (4) a location 808 of a fourth keypoint 408 determined at a first time (t1), (5) a location 810 of a fifth keypoint 410 determined at a first time (t1), and (6) a location 812 of a sixth keypoint 410 determined at a first time (t1). (7) a position 812 of the seventh keypoint 414 determined at the first time (t1); (8) a position 816 of the eighth keypoint 416 determined at the first time (t1); (9) a position 818 of the first keypoint 402 determined at the second time (t2); (10) a position 820 of the ninth keypoint 502 determined at the second time (t2); (11) a position 822 of the tenth keypoint 504 determined at the second time (t2); (13) a position 826 of the sixth keypoint 412 determined at the second time (t2); (14) a position 828 of the eighth keypoint 416 determined at the second time (t2); (15) a position 830 of the twelfth keypoint 508 determined at the second time (t2); (16) a position 832 of the first keypoint 402 determined at the third time (t3); (17) a position 834 of the second keypoint 404 determined at the third time (t3); and (18) a position 835 of the second keypoint 404 determined at the third time (t4). (19) a position 836 of a third keypoint 406 determined at the third time (t3); (20) a position 840 of a fifth keypoint 410 determined at the third time (t3); (21) a position 842 of a sixth keypoint 412 determined at the third time (t3); (22) a position 844 of a seventh keypoint 414 determined at the third time (t3); and (23) a position 846 of an eighth keypoint 416 determined at the third time (t3).(24) a position 848 of the first key point 402 determined at the fourth time (t4); (25) a position 850 of the ninth key point 502 determined at the fourth time (t4); (26) a position 852 of the tenth key point 504 determined at the fourth time (t4); (27) a position 854 of the eleventh key point 506 determined at the fourth time (t4); (28) a position 856 of the sixth key point 412 determined at the fourth time (t4); (29) a position 858 of the eighth key point 416 determined at the fourth time (t4); and (30) a position 860 of the twelfth key point 508 determined at the fourth time (t4).

[0065] For example, the instructions for generating a digital map may include instructions for processing data associated with the images using one or more data association operations to determine a correspondence between locations of points associated with particular objects included in a first one of the images and locations of points associated with particular objects included in a second one of the images. For example, one or more data association operations may determine the correspondence of: (1) position 802 to position 818, position 832, and position 848; (2) position 804 to position 834; (3) position 806 to position 836; (4) position 808 to position 838; (5) position 810 to position 840; (6) position 812 to position 826, position 842, and position 856; (7) position 814 to position 844; (8) position 816 to position 828, position 846, and position 858; (8) position 820 to position 850; (9) position 822 to position 852; (10) position 824 to position 854; and (11) position 830 to position 860.

[0066] 9 includes an example of a digital map 900 in accordance with the techniques of this disclosure. For example, the digital map 900 includes (1) a first road sign 112 (based on positions 802, 818, 832, and 848), (2) a second road sign 114 (based on positions 804 and 834), (3) a road boundary 122 (based on positions 806, 820, 836, and 850), (4) a road boundary 124 (based on positions 808, 822, 838, and 852), (5) a road boundary 125 (based on positions 809, 810, 822, 836, and 852), (6) a road boundary 126 (based on positions 814, 816, 822, 838, and 852), (7) a road boundary 128 (based on positions 818, 819, 823, 836, and 852), (8) a road boundary 129 (based on positions 820, 822, 838, and 852), (9) a road boundary 130 (based on positions 822, 823, 836, and 852), (10) a road boundary 131 (based on positions 824, 825, 838, and 852), (11) a road boundary 132 (based on positions 826, 827, 838, and 852), (12) a road boundary 133 (based on positions 828, 829, 830, and 840), (13) a road boundary 134 (based on positions 828, 829, 830, and 840), (14) a road boundary 135 (based This may include representations of positions for lane boundary 128 (based on position 810, position 824, position 840, and position 854), (6) lane boundary 132 (based on position 812, position 814, position 826, position 842, position 844, and position 856), (7) road boundary 158 (based on position 816, position 828, position 846, and position 858), and (8) lane boundary 160 (based on position 830 and position 860).

[0067] For example, instructions for generating a digital map may include (1) instructions for classifying images into key frames, and (2) instructions for processing the key frames using one or more simultaneous localization and mapping (SLAM) optimization techniques. For example, (1) a first one of the key frames may feature a first criterion, (2) a second one of the key frames may feature a second criterion, and (3) a difference between the first criterion and the second criterion may be greater than a threshold. For example, the first criterion may relate to a value of data included in the first key frame, and the second criterion may relate to a value of data included in the second key frame. As an illustrative example, image 200 includes a second road sign 114 and a third portion 140 that are not included in image 300. Similarly, for example, image 300 includes a lane boundary 160 that is not included in image 200. If the threshold value is set such that the difference between the reference value of the data included in the image 200 and the reference value of the data included in the image 300 is greater than the threshold value, the image 200 may be associated with the first key frame and the image 300 may be associated with the second key frame. More generally, for example, the count of images included in a key frame of the key frames may be a function of the distance traveled by the vehicle that generated the image. For example, the distance traveled by the first vehicle 164 between the first time (t1) and the second time (t2) may be about 1 meter, since the portion of the dashed white lane marking indicating the separation between lanes flowing in the same direction of traffic may have a length of about 1 meter. Thus, both the image 200 and the image 300 include the first road sign 112, the road boundary 122, the road boundary 124, the lane boundary 128, the second portion 138, and the road boundary 158. Depending on the value to which the threshold is set, both images 200 and 300 may be included in the same key frame because the distance traveled by the first vehicle 164 (e.g., about one meter) may not be long enough for the difference between the reference value of the data contained in image 200 and the reference value of the data contained in image 300 to be greater than the threshold.

[0068] For example, the instructions for generating a digital map may include instructions for processing data associated with an image of a place using one or more SLAM techniques. For example, the place may be a particular region. Advantageously, processing data for the particular region may limit the number of data association operations performed to generate a digital map of the place. For example, one or more SLAM techniques may be performed for each of a first particular region and a second particular region that may be adjacent to the first particular region. For example, after such SLAM techniques are performed for each of the first particular region and the second particular region, one or more SLAM techniques may be performed in a third particular region that may overlap with each of the first particular region and the second particular region. For example, the shape of the particular region may be defined by seven regular hexagons, with one of the seven regular hexagons being located at a center position and another of the seven regular hexagons being located at a side of the one hexagon. For example, each side of the one hexagon at the center position may be directly adjacent to a side of another hexagon. In this manner, the particular region may be a region in a hexagonal grid. Advantageously, the use of a hexagonal grid may simplify calculations when a digital map is constructed over a particular region. The distance between the center of a particular hexagon and the center of any adjacent hexagon may be the same as the distance between the center of the particular hexagon and the center of any other adjacent hexagon. In contrast, for a square grid, calculations regarding the distance between the center of a particular square and the center of any adjacent square may require consideration of whether the center of the adjacent square is perpendicular or diagonal to the center of the particular square. Furthermore, advantageously, when a digital map is constructed over a particular region, a hexagonal grid fits better than a square grid to the surface of a sphere (e.g., a sphere).

[0069] Fig. 10 includes a diagram illustrating an example of a hexagonal grid 1000 superimposed on the environment 100 shown in Fig. 1 in accordance with the techniques of this disclosure. For example, the hexagonal grid 1000 may include (1) a first regular hexagon 1002, (2) a second regular hexagon 1004, (3) a third regular hexagon 1006, (4) a fourth regular hexagon 1008, (5) a fifth regular hexagon 1010, (6) a sixth regular hexagon 1012, and (7) a seventh regular hexagon 1014. , (8) an eighth regular hexagon 1016, (9) a ninth regular hexagon 1018, (10) a tenth regular hexagon 1020, (11) an eleventh regular hexagon 1022, (12) a twelfth regular hexagon 1024, (13) a thirteenth regular hexagon 1026, and (14) a fourteenth regular hexagon 1028.

[0070] For example, the shape of the particular region may be defined by a third regular hexagon 1006, a fourth regular hexagon 1008, a fifth regular hexagon 1010, a sixth regular hexagon 1012, a seventh regular hexagon 1014, an eighth regular hexagon 1016, and a ninth regular hexagon 1018. For example, instructions for processing data related to images of the particular region using one or more SLAM techniques may process not only data related to images generated by the first vehicle 164 and the second vehicle 166 within the third regular hexagon 1006, but also data related to images generated by one or more vehicles within any of the fourth regular hexagon 1008, the fifth regular hexagon 1010, the sixth regular hexagon 1012, the seventh regular hexagon 1014, the eighth regular hexagon 1016, or the ninth regular hexagon 1018. For example, the specific region may be the first specific region. For example, the shape of the second specific region may be defined by the twelfth regular hexagon 1024, the thirteenth regular hexagon 1026, the fourteenth regular hexagon 1028, and four other regular hexagons (not shown) south of the twelfth regular hexagon 1024, the thirteenth regular hexagon 1026, and the fourteenth regular hexagon 1028. For example, one or more SLAM techniques may be performed on each of the first specific region and the second specific region. For example, the shape of the third specific region may be defined by the sixth regular hexagon 1012, the seventh regular hexagon 1014, the eighth regular hexagon 1016, the ninth regular hexagon 1018, the tenth regular hexagon 1020, the eleventh regular hexagon 1022, and the twelfth regular hexagon 1024. For example, after such SLAM techniques have been performed for each of a first particular region and a second particular region, one or more SLAM techniques may be performed in a third particular region.

[0071] 7, for example, the generation module 706 of the system 700 (e.g., the system 180 shown in FIG. 1) may include an alignment submodule 710. For example, the alignment submodule 710 may include instructions functioning to control the processor 702 to align positions of points associated with the generation of a version of a digital map to positions of points in a two-dimensional image of a location to generate a two-dimensional aligned digital map. For example, the two-dimensional image may be an aerial image and the two-dimensional aligned digital map may be an aerial image aligned digital map. For example, the aerial image may be a satellite image. Alternatively, for example, the aerial image may be generated by a camera associated with an aircraft.

[0072] FIG. 11 includes an example aerial image 1100 of the environment 100 shown in FIG. 1 in accordance with the techniques of this disclosure. For example, the aerial image 1100 may include a first road 102, a second road 104, a parking lot 106, an intersection 108, a building 110, a first road sign 112, a second road sign 114, a road boundary 122, a road boundary 124, a road boundary 126, a lane boundary 128 (i.e., a lane marking 130 having two solid yellow lines), a lane boundary 132 (i.e., a lane marking 134 having a dashed white line, including a first portion 136, a second portion 138, a third portion 140, a fourth portion 142, a fifth portion 144, a sixth portion 146, a seventh portion 148, and an eighth portion 150), a road boundary 156, a road boundary 158, and a lane boundary 160 (i.e., a lane marking 162 having two solid yellow lines).

[0073] For example, the instructions for aligning may include instructions for aligning positions of points associated with the generation of a version of the digital map to positions of points in an aerial image to correct for errors in one or more of the proprioceptive or sensory information used by one or more SLAM techniques to generate an aerial-image aligned digital map. For example, the instructions for aligning may include at least one SLAM optimization technique.

[0074] As described above, one or more circumstances associated with the generation of data related to an image of a location may cause information related to, for example, (1) the pose of the camera, (2) one or more positions of points relative to landmarks, or (3) both to include one or more errors. For example, errors in proprioceptive information (e.g., one or more of GNSS information, IMU information, odometry information, or the like) may cause information related to the pose of the camera to include one or more errors. For example, changes in illumination relative to one or more of the landmarks at one or more of a first time (t1), a second time (t2), a third time (t3), or a fourth time (t4) may cause results of a photogrammetric range imaging technique (e.g., SfM technique) to include one or more errors, such that distances and orientations to landmarks (e.g., key points) in an image determined from the photogrammetric range imaging technique (e.g., SfM technique) include one or more errors. In light of the description herein, it will be understood by those skilled in the art that one or more other circumstances may cause information relating to (1) the pose of the camera, (2) one or more locations of the points relative to the landmarks, or (3) both to contain one or more other errors. Individually or cumulatively, these errors may cause the information contained in an item of data associated with an image generated at a given time by a particular source (e.g., forward-facing camera 170 mounted on first vehicle 164 or forward-facing camera 172 mounted on second vehicle 166) to differ from a corresponding item of data associated with an image generated (1) at a different time, (2) by a different particular source, or (3) both.

[0075] For example, GNSS-generated information regarding position may contain errors that result in inaccuracies that distort position to approximately the same distance in approximately the same direction. Thus, in situations where the proprioceptive information for SLAM techniques is generated by GNSS in the presence of such errors, the digital maps generated by the SLAM techniques may be misaligned.

[0076] For example, the instructions to align may be to road boundaries. For example, the instructions to align may include instructions to detect, within the first copy of the aerial image, indications of road boundaries in a maintained surface (e.g., a drivable surface (e.g., a road)) used by vehicles and pedestrians.

[0077] For example, instructions for detecting indications of road boundaries may include: (1) instructions for generating a temporary rectified aerial image by giving each pixel in a first copy of the aerial image associated with the groomed surface its highest value; (2) instructions for giving all other pixels in the temporary rectified aerial image its lowest value; and (3) instructions for detecting road boundaries in the temporary rectified aerial image.

[0078] 12 includes an example of a first temporary rectified aerial image 1200 of the environment 100 shown in FIG 1 in accordance with the techniques of this disclosure. For example, the first temporary rectified aerial image 1200 may include the first road 102, the second road 104, the parking lot 106, and the intersection 108.

[0079] Further, for example, the instructions for detecting indications of road boundaries may further include instructions for causing each pixel in the first copy of the aerial image associated with a maintained surface used by the vehicle for parking to have a lowest value.

[0080] 13 includes an example of a second temporary rectified aerial image 1300 of the environment 100 shown in FIG 1 in accordance with the techniques of this disclosure. For example, the second temporary rectified aerial image 1300 may include the first road 102, the second road 104, and the intersection 108.

[0081] For example, the instructions for aligning may further include: (1) instructions for generating a first rectified aerial image with each pixel in the first copy of the aerial image associated with the road boundary having a lowest value, (2) instructions for generating all other pixels in the first rectified aerial image having a highest value, (3) instructions for determining a first distance for a first pixel of all other pixels, and (4) instructions for changing the value for the first pixel from the highest value to a first value representing the first distance as a first operation for generating the first segmentation mask. For example, the first distance may be between a position at a location represented by a center of the first pixel and a position at a location represented by a nearest first pixel having a lowest value.

[0082] For example, the location of the point associated with the generation of the version of the digital map may be a location of the point at a road boundary and may match a location of a first pixel in the first segmentation mask. For example, the instructions for aligning may further include instructions for moving the location of the point associated with the generation of the version of the digital map by a first distance to a location that matches a location of a nearest first pixel in the first segmentation mask having a lowest value as an operation for generating the aerial image aligned digital map. For example, the instructions for moving the location of the point may include at least one SLAM optimization technique.

[0083] 14 includes a diagram 1400 illustrating an example of operations for moving a location of a point associated with generating a version of a digital map to a location of a pixel on a first segmentation map according to techniques of this disclosure. For example, diagram 1400 may include a portion 1402 of a first segmentation map associated with region 184 in aerial image 1100 of environment 100. For example, portion 1402 may include an arrangement of 36 pixels arranged in 6 rows and 6 columns. For example, for each pixel, the height of the pixel may represent a distance of 7.5 centimeters and the width of the pixel may represent a distance of 7.5 centimeters.

[0084] For example, the configuration may include: (1) pixel (1,1) having a value of 10.6; (2) pixel (1,2) having a value of 7.5; (3) pixel (1,3) having a value of 7.5; (4) pixel (1,4) having a value of 7.5; (5) pixel (1,5) having a value of 7.5; (6) pixel (1,6) having a value of 7.5; (7) pixel (2,1) having a value of 7.5; (8) pixel (2,2) having a value of 0.0; (9) pixel (2,3) having a value of 0.0; and (10) pixel (1,4) having a value of 7.5. (0) pixel (2,4) having a value of 0.0; (11) pixel (2,5) having a value of 0.0; (12) pixel (2,6) having a value of 0.0; (13) pixel (3,1) having a value of 7.5; (14) pixel (3,2) having a value of 0.0; (15) pixel (3,3) having a value of 7.5; (16) pixel (3,4) having a value of 7.5; (17) pixel (3,5) having a value of 7.5; (18) pixel (3,6) having a value of 7.5; (19) pixel (3,1) having a value of 7.5; (20) pixel (4, 2) having a value of 0.0; (21) pixel (4, 3) having a value of 7.5; (22) pixel (4, 4) having a value of 15.0; (23) pixel (4, 5) having a value of 15.0; (24) pixel (4, 6) having a value of 15.0; (25) pixel (5, 1) having a value of 7.5; (26) pixel (5, 2) having a value of 0.0; (27) pixel (5, 3) having a value of 7.5; and (28) pixel (5, 4) having a value of 15.0. (29) pixel (5, 4) having a value of 22.5, (30) pixel (5, 6) having a value of 22.5, (31) pixel (6, 1) having a value of 7.5, (32) pixel (6, 2) having a value of 0.0, (33) pixel (6, 3) having a value of 7.5, (34) pixel (6, 4) having a value of 15.0, (35) pixel (6, 5) having a value of 22.5, and (36) pixel (6, 6) having a value of 30.0.

[0085] For example, (1) pixel (2, 2), pixel (3, 2), pixel (4, 2), pixel (5, 2), and pixel (6, 2) may be associated with a partial location of road boundary 126, and (2) pixel (2, 2), pixel (2, 3), pixel (2, 4), pixel (2, 5), and pixel (2, 6) may be associated with a partial location of road boundary 158.

[0086] For example, the positions of the points relevant to generating a version of the digital map may be positions of points at road boundaries that coincide with the following center positions: (1) pixel (1,1) should be moved 10.6 centimeters toward the position of pixel (2,2); (2) pixel (1,2) should be moved 7.5 centimeters toward the position of pixel (2,2); (3) pixel (1,3) should be moved 7.5 centimeters toward the position of pixel (2,3); and (4) pixel (1,4) should be moved 7.5 centimeters toward the position of pixel (2,4). (5) pixel (1, 5) should be moved 7.5 centimeters towards the location of pixel (2, 5); (6) pixel (1, 6) should be moved 7.5 centimeters towards the location of pixel (2, 6); (7) pixel (2, 1) should be moved 7.5 centimeters towards the location of pixel (2, 2); (8) pixel (3, 1) should be moved 7.5 centimeters towards the location of pixel (3, 2); (9) pixel (3, 3) should be moved 7.5 centimeters towards the location of pixel (2, 3) or should be moved 7.5 centimeters towards the location of pixel (3, 2), (10) pixel (3, 4) should be moved 7.5 centimeters towards the location of pixel (2, 4), (11) pixel (3, 5) should be moved 7.5 centimeters towards the location of pixel (2, 5), (12) pixel (3, 6) should be moved 7.5 centimeters towards the location of pixel (2, 6), (13) pixel (4, 1) should be moved 7.5 centimeters towards the location of pixel (4, 2), (14) pixel (3, 5) should be moved 7.5 centimeters towards the location of pixel (3, 6), (15) pixel (3, 6) should be moved 7.5 centimeters towards the location of pixel (3, 6), (16) pixel (3, 1) should be moved 7.5 centimeters towards the location of pixel (3, 2), (17) pixel (3, 1) should be moved 7.5 centimeters towards the location of pixel (3, 2), (18) pixel (3, 2) should be moved 7.5 centimeters towards the location of pixel (3, 2), (19) pixel (3, 3) should be moved 7.5 centimeters towards the location of pixel (3, 4), (20) pixel (3, 4) should be moved 7.5 centimeters towards the location of pixel (3, 4), (21) pixel (3, 5) should be moved 7.5 centimeters towards the location of pixel (2, 5), (22) pixel (3, 5) should be moved 7.5 centimeters towards the location of pixel (2, 5), (23) pixel (3, 6) should be moved 7.5 centimeters towards the location of pixel (2, 6), (24) pixel (3, 6) should be moved 7.5 centimeters towards the location of pixel (14) pixel (4, 3) should be moved 7.5 centimeters toward the location of pixel (4, 2), (15) pixel (4, 4) should be moved 15.0 centimeters toward the location of pixel (2, 4) or the location of pixel (4, 2), (16) pixel (4, 5) should be moved 15.0 centimeters toward the location of pixel (2, 5), (17) pixel (4, 6) should be moved 15.0 centimeters toward the location of pixel (2, 6), and (18) pixel (5, 1) should be moved 7.5 centimeters toward the location of pixel (5, 2).(19) pixel (5, 3) should be moved 7.5 centimeters toward the location of pixel (5, 2); (20) pixel (5, 4) should be moved 15.0 centimeters toward the location of pixel (5, 2); (21) pixel (5, 5) should be moved 22.5 centimeters toward the location of pixel (2, 5) or pixel (5, 2); (22) pixel (5, 6) should be moved 22.5 centimeters toward the location of pixel (2, 6); (23) pixel (6, 1) should be moved 22.5 centimeters toward the location of pixel (6, 2); (23) pixel (6, 2) should be moved 7.5 centimeters toward the location of pixel (6, 2), (24) pixel (6, 3) should be moved 7.5 centimeters toward the location of pixel (6, 2), (25) pixel (6, 4) should be moved 15.0 centimeters toward the location of pixel (6, 2), (26) pixel (6, 5) should be moved 22.5 centimeters toward the location of pixel (6, 2), and (27) pixel (6, 6) should be moved 30.0 centimeters toward the location of pixel (2, 6) or pixel (6, 2).

[0087] For example, diagram 1400 may also include (1) a point 1404 at position 816 of the eighth key point 416 determined at a first time (t1), (2) a point 1406 at position 828 of the eighth key point 416 determined at a second time (t2), (3) a point 1408 at position 846 of the eighth key point 416 determined at a third time (t3), and (4) a point 1410 at position 858 of the eighth key point 416 determined at a fourth time (t4).

[0088] For example, (1) the location of point 1404 may coincide with the location of the center of pixel (1, 5), (2) the location of point 1406 may coincide with the location of the center of pixel (2, 5), (3) the location of point 1408 may coincide with the location of the center of pixel (1, 1), and (4) the location of point 1410 may be within pixel (3, 5) near pixel (2, 5), pixel (2, 6), and pixel (3, 6).

[0089] Therefore, (1) the position of point 1404 should be moved 7.5 centimeters toward the position of pixel (2, 5), (2) the position of point 1406 should not be moved, and (3) the position of point 1408 should be moved 10.6 centimeters toward the position of pixel (2, 2).

[0090] In this way, the positions of the points associated with generating a version of the digital map may be moved a certain distance to positions that correspond to the positions of the nearest pixels in the first segmentation mask that have the lowest values, to generate a digital map of aerial image alignment.

[0091] Further, for example, the instructions for aligning may include: (1) instructions for determining a second distance for a second pixel of all other pixels; (2) instructions for changing the value for the second pixel from the highest value to a second value representing the second distance as a second operation for generating the first segmentation mask; (3) instructions for determining a third distance for a third pixel of all other pixels; (4) instructions for changing the value for the third pixel from the highest value to a third value representing the third distance as a third operation for generating the first segmentation mask; and (5) instructions for determining a fourth distance for a fourth pixel of all other pixels. (5) as a fourth operation for generating the first segmentation mask, instructions for changing a value for the fourth pixel from the highest value to a fourth value representing a fourth distance; and (6) as a fourth operation for generating the first segmentation mask, instructions for determining, for a particular point within the first pixel, the second pixel, the third pixel, or the fourth pixel, a corresponding value representing a corresponding distance between a position at a location represented by the particular point and a position at a location represented by a nearest corresponding pixel having the lowest value based on the first value, the second value, the third value, and the fourth value. For example, (1) the second distance may be between a position at a location represented by the center of the second pixel and a position at a location represented by the nearest second pixel having the lowest value, (2) the third distance may be between a position at a location represented by the center of the third pixel and a position at a location represented by the nearest third pixel having the lowest value, and (3) the fourth distance may be between a position at a location represented by the center of the fourth pixel and a position at a location represented by the nearest fourth pixel having the lowest value. For example, the first pixel, the second pixel, the third pixel, and the fourth pixel may be arranged in a configuration having two rows and two columns. For example, the instructions for determining the corresponding values ​​may include instructions for determining the corresponding values ​​using bilinear interpolation.For example, the corresponding value determined using bilinear interpolation may be a weighted average of: (1) a first value and the distance between the particular point and the center of the first pixel; (2) a second value and the distance between the particular point and the center of the second pixel; (3) a third value and the distance between the particular point and the center of the third pixel; and (4) a fourth value and the distance between the particular point and the center of the fourth pixel. In this manner, the function that determines the corresponding value (i.e., the bilinear interpolation) may be a differentiable function. By using a differentiable function, the instructions for moving the location of the point may be able to include at least one SLAM optimization technique.

[0092] For example, a position of a point associated with generating a version of the digital map may match a position of a particular point in the first segmentation mask. For example, the instructions for aligning may further include instructions for moving a position of a point associated with generating a version of the digital map a corresponding distance to a position that matches a position of a nearest corresponding pixel having a lowest value in the first segmentation mask as an operation for generating a digital map of the aerial image alignment.

[0093] 14, for example, the particular point may be point 1410. For example, the first pixel may be pixel (2, 5), the second pixel may be pixel (2, 6), the third pixel may be pixel (3, 5), and the fourth pixel may be pixel (3, 6). For example, the corresponding values ​​may be based on values ​​of 0.0, 0.0, 7.5, and 7.5. For example, the corresponding values ​​may be determined using bilinear interpolation. For example, the corresponding value determined using bilinear interpolation may be a weighted average of: (1) the first value (0.0) and the distance between point 1410 and the center of pixel (2, 5), (2) the second value (0.0) and the distance between point 1410 and the center of pixel (2, 6), (3) the third value (7.5) and the distance between point 1410 and the center of pixel (3, 5), and (4) the fourth value (7.5) and the distance between point 1410 and the center of pixel (3, 6). The location of point 1410 should be moved the corresponding distance (represented by the corresponding value) toward the location of pixel (2, 5).

[0094] Further, for example, the instructions for aligning may further be for lane boundaries. For example, the instructions for aligning may further include: (1) instructions for detecting in the second copy of the aerial image an indication of a lane boundary in a maintained surface (e.g., a drivable surface (e.g., a road)) used by vehicles and pedestrians; (2) instructions for generating a second modified aerial image with each pixel in the second copy of the aerial image associated with a lane boundary having a lowest value; (3) instructions for all other pixels in the second modified aerial image having a highest value; (4) instructions for determining a specific distance for a particular pixel of all other pixels in the second modified aerial image; and (5) instructions for changing the value for the particular pixel from the highest value to a specific value representing the specific distance as an operation for generating a second segmentation mask. For example, the specific distance may be between a location represented by the center of the particular pixel and a location represented by the nearest pixel having the lowest value.

[0095] For example, the location of the point associated with generating the version of the digital map may be a location of the point at a lane boundary and may match a location of a particular pixel in the second segmentation mask. For example, the instructions for aligning may further include instructions for moving the location of the point associated with generating the version of the digital map a particular distance to a location that matches a location of a nearest pixel in the second segmentation mask having a lowest value as an operation for generating the aerial image aligned digital map.

[0096] Alternatively, for example, the instructions to align may be relative to lane boundaries (e.g., without regard to road boundaries). For example, the instructions to align may include: (1) instructions to detect in the aerial image an indication of a lane boundary on a maintained surface (e.g., a drivable surface (e.g., a road)) used by vehicles and pedestrians; (2) instructions to generate a rectified aerial image with each pixel in the aerial image associated with a lane boundary having a lowest value; (3) instructions to generate a rectified aerial image with all other pixels having a highest value; (4) instructions to determine a distance for a pixel among all other pixels; and (5) instructions to change the value for the pixel from the highest value to a value representing the distance as an operation to generate a segmentation mask. For example, the distance may be between a location represented by the center of the pixel and a location represented by the nearest pixel having the lowest value.

[0097] For example, the location of the point associated with generating the version of the digital map may be the location of the point at a lane boundary and may match the location of the pixel in the segmentation mask. For example, the instructions for aligning may further include instructions for moving the location of the point associated with generating the version of the digital map by the distance to a location that matches the location of the nearest pixel in the segmentation mask having the lowest value as an operation for generating the aerial image aligned digital map.

[0098] 15 includes a flow diagram illustrating an example method 1500 associated with generating a digital map of a place from data related to an image of the place but excluding pixel color data, in accordance with the techniques of this disclosure. Although the method 1500 is described in conjunction with the system 700 shown in FIG. 7, it will be understood by those skilled in the art in light of the description herein that the method 1500 is not limited to implementation by the system 700 shown in FIG. 7. Rather, the system 700 shown in FIG. 7 is one example of a system that may be used to implement the method 1500. Additionally, while the method 1500 is generally illustrated as a sequential process, various aspects of the method 1500 may be capable of being performed in parallel.

[0099] In method 1500, at operation 1502, for example, generation module 706 may generate a digital map from the data. The data for a certain one of the images may exclude pixel color data, but may include information regarding (1) the pose of the camera that generated the image, and (2) one or more of: (a) lane boundaries of lanes of a road in the image, (b) road boundaries of a road, or (c) the location of a point relative to another landmark in the image.

[0100] For example, one or more of the locations of the points relative to the lane boundaries, road boundaries, or landmarks may relate to the locations of key points of an object in the image that represents the lane boundaries, road boundaries, or landmarks.

[0101] For example, the landmark may be a sign. For example, the data associated with the image may include information about the sign. For example, the information about the sign may include (1) a latitudinal position, a longitudinal position, and an altitude relative to a center of the sign, (2) a height of the sign, and (3) a width of the sign. Additionally or alternatively, for example, the information about the sign may include information about a message conveyed by the sign.

[0102] At operation 1504, for example, the communications module 708 may transmit a digital map to the first vehicle for use in controlling the movement of the first vehicle.

[0103] Further, at operation 1506, for example, the communications module 708 may receive data related to the images from one or more second vehicles (e.g., the camera may include one or more cameras, and the one or more cameras may be mounted on the one or more second vehicles).

[0104] For example, a camera of the one or more cameras may be a component in a lane-keeping assist (LKA) system. For example, the images may be generated at a particular generation rate. For example, the particular generation rate may be 10 Hertz. For example, the amount of data for the images may be less than a threshold amount. For example, the threshold amount may be 300 bytes. For example, the data related to the images may be generated by an active safety technology and an advanced driver assistance system (ADAS) autonomous driving system. For example, the autonomous driving system may be a third generation Toyota Safety Sense™ system (TSS3).

[0105] For example, (1) operation 1506 by system 700 (e.g., system 180 shown in FIG. 1 ) may occur at a first time, (2) operation 1502 by system 700 (e.g., system 180 shown in FIG. 1 ) may occur at a second time, and (3) the difference between the first time and the second time may be less than a particular duration. For example, the particular duration may be 30 minutes.

[0106] For example, at operation 1506, the communications module 708 may receive a transmission of a batch of data associated with images generated by a corresponding camera at a particular communications pace from a vehicle of the one or more second vehicles at a duration associated with the particular communications pace. For example, the particular communications pace may be once every 30 seconds.

[0107] For example, at operation 1502, the generation module 706 may use one or more data association operations to process data related to the images and determine a correspondence between the positions of points related to a particular object included in a first one of the images and the positions of points related to a particular object included in a second one of the images.

[0108] For example, in operation 1502, the generating module 706 may (1) classify images into keyframes and (2) process the keyframes using one or more simultaneous localization and mapping (SLAM) optimization techniques. For example, (1) a first one of the keyframes may feature a first criterion, (2) a second one of the keyframes may feature a second criterion, and (3) a difference between the first criterion and the second criterion may be greater than a threshold. For example, the first criterion may relate to a value of data included in the first keyframe, and the second criterion may relate to a value of data included in the second keyframe. More generally, for example, the count of images included in a keyframe of the keyframes may be a function of a distance traveled by the vehicle that generated the image.

[0109] For example, in operation 1502, the generating module 706 may process data related to an image of a place using one or more SLAM techniques. For example, the place may be a particular region. Advantageously, processing data for the particular region may limit the number of data association operations performed to generate a digital map of the place. For example, the shape of the particular region may be defined by seven regular hexagons, with one of the seven regular hexagons located at a center position and another of the seven regular hexagons located at the sides of the one hexagon. For example, each side of the one hexagon located at the center position may be directly adjacent to the sides of another hexagon. In this manner, the particular region may be a region in a hexagonal grid. Advantageously, the use of a hexagonal grid may simplify calculations when a digital map is constructed over the particular region. The distance between the center of the particular hexagon and the center of any adjacent hexagon may be the same as the distance between the center of the particular hexagon and the center of any other adjacent hexagon. In contrast, for a square grid, calculations regarding the distance between the center of a particular square and the center of any adjacent squares may require consideration of whether the adjacent square centers are perpendicular or diagonal to the center of the particular square. Moreover, advantageously, when digital maps are constructed over a particular region, a hexagonal grid fits better than a square grid to the surface of a sphere (e.g., a sphere).

[0110] Further, in operation 1508, for example, the alignment submodule 710 may align positions of points associated with the generation of a version of the digital map to positions of points in an aerial image of the location to generate an aerial image aligned digital map. For example, the aerial image may be a satellite image. Alternatively, for example, the aerial image may be generated by a camera associated with an aircraft.

[0111] For example, in operation 1508, the alignment submodule 710 may align positions of points associated with generating a version of the digital map to positions of points in an aerial image to generate an aerial-image aligned digital map in order to correct for errors in one or more proprioceptive or sensory information used by one or more SLAM techniques.

[0112] 16A-16D include a flow diagram illustrating a first example of a method 1600 associated with aligning positions of points associated with generating a version of a digital map of a place to positions of points in an aerial image of the place to generate an aerial image-aligned digital map of the place, in accordance with the techniques of this disclosure. Although the method 1600 is described in conjunction with the system 700 shown in FIG. 7, it will be understood by those skilled in the art in light of the description herein that the method 1600 is not limited to implementation by the system 700 shown in FIG. 7. Rather, the system 700 shown in FIG. 7 is one example of a system that may be used to implement the method 1600. Additionally, while the method 1600 is generally illustrated as a sequential process, various aspects of the method 1600 may be capable of being performed in parallel.

[0113] In method 1600 in FIG. 16A, at operation 1602, for example, the alignment submodule 710 may detect, within a first copy of the aerial image, a representation of a road boundary in a maintained surface (e.g., a drivable surface (e.g., a road)) used by vehicles and pedestrians.

[0114] 17 includes a flow diagram illustrating an example method 1700 associated with detecting in a copy of an aerial image an indication of road boundaries in a cleared surface used by vehicles and pedestrians, in accordance with the techniques of this disclosure. Although the method 1700 is described in conjunction with the system 700 illustrated in FIG. 7, it will be understood by those skilled in the art in light of the description herein that the method 1700 is not limited to implementation by the system 700 illustrated in FIG. 7. Rather, the system 700 illustrated in FIG. 7 is one example of a system that may be used to implement the method 1700. Additionally, while the method 1700 is generally illustrated as a sequential process, various aspects of the method 1700 may be capable of being performed in parallel.

[0115] In method 1700, at operation 1702, for example, the alignment sub-module 710 may generate a temporary rectified aerial image by aligning each pixel in a copy of the aerial image associated with the groomed surface with the highest value.

[0116] At operation 1704, for example, the alignment sub-module 710 may align all other pixels in the temporary rectified aerial image to have the lowest value.

[0117] At operation 1706, for example, the alignment sub-module 710 may detect road boundaries in the temporarily rectified aerial image.

[0118] Further, in operation 1708, for example, the alignment sub-module 710 may cause each pixel in the copy of the aerial image that is associated with a landscaped surface used by the vehicle for parking to have the lowest value.

[0119] Referring to FIG. 16A, in method 1600, at operation 1604, for example, the alignment sub-module 710 may generate a first corrected aerial image by causing each pixel in the first copy of the aerial image associated with the road boundary to have the lowest value.

[0120] At operation 1606, for example, the alignment sub-module 710 may align all other pixels in the first rectified aerial image to have the highest value.

[0121] At operation 1608, for example, the alignment sub-module 710 may determine a first distance for a first pixel among all other pixels.

[0122] In operation 1610, for example, the alignment submodule 710 may change a value for a first pixel from the highest value to a first value representing a first distance as a first operation to generate a first segmentation mask. For example, the first distance may be between a position at a location represented by a center of the first pixel and a position at a location represented by a nearest first pixel having a lowest value.

[0123] For example, the location of the point associated with the generation of the version of the digital map may be the location of the point at the road boundary and may match the location of the first pixel in the first segmentation mask. Further, in operation 1612, for example, the alignment submodule 710 may move the location of the point associated with the generation of the version of the digital map by a first distance to a location that matches the location of the nearest first pixel in the first segmentation mask that has the lowest value as an operation for generating the aerial image aligned digital map.

[0124] Further in the method 1600 in FIG. 16B, at operation 1614, for example, the alignment sub-module 710 may determine a second distance for a second pixel among all other pixels.

[0125] Further, at operation 1616, for example, the alignment submodule 710 may change the value for the second pixel from the highest value to a second value representing a second distance as a second operation to generate the first segmentation mask.

[0126] Further, at operation 1618, for example, the alignment sub-module 710 may determine a third distance for a third pixel among all other pixels.

[0127] Further, at operation 1620, for example, the alignment submodule 710 may change the value for a third pixel from the highest value to a third value representing a third distance as a third operation to generate a first segmentation mask.

[0128] Further, at operation 1622, for example, the alignment sub-module 710 may determine a fourth distance for a fourth pixel among all other pixels.

[0129] Further, in the method 1600 in FIG. 16C, at operation 1624, for example, the alignment submodule 710 may change the value for a fourth pixel from the highest value to a fourth value representing a fourth distance as a fourth operation to generate a first segmentation mask.

[0130] Further, in operation 1626, for example, the alignment submodule 710 may determine, for a particular point in the first pixel, the second pixel, the third pixel, or the fourth pixel, a corresponding value representing a corresponding distance between a position at a location represented by the particular point and a position at a location represented by the nearest corresponding pixel having the lowest value based on the first value, the second value, the third value, and the fourth value as a fifth operation for generating a first segmentation mask. For example, (1) the second distance may be between a position at a location represented by the center of the second pixel and a position at a location represented by the nearest corresponding pixel having the lowest value, (2) the third distance may be between a position at a location represented by the center of the third pixel and a position at a location represented by the nearest corresponding third pixel having the lowest value, and (3) the fourth distance may be between a position at a location represented by the center of the fourth pixel and a position at a location represented by the nearest corresponding fourth pixel having the lowest value. For example, the first pixel, the second pixel, the third pixel, and the fourth pixel may be arranged in a configuration having 2 rows and 2 columns. For example, the instructions for determining the corresponding values ​​may include instructions for determining the corresponding values ​​using bilinear interpolation.

[0131] For example, the location of the point associated with the generation of the version of the digital map may match the location of the particular point in the first segmentation mask. Further, in operation 1628, for example, the alignment submodule 710 may move the location of the point associated with the generation of the version of the digital map by a corresponding distance to a location that matches the location of the nearest corresponding pixel having the lowest value in the first segmentation mask as an operation to generate the aerial image aligned digital map.

[0132] Further, in method 1600 in FIG. 16D, at operation 1630, for example, the alignment submodule 710 may detect, within the second copy of the aerial image, indications of lane boundaries on a maintained surface used by vehicles and pedestrians (e.g., a drivable surface (e.g., a road)).

[0133] Further, in operation 1632, for example, the alignment sub-module 710 may generate a second corrected aerial image by causing each pixel in the second copy of the aerial image that is associated with a lane boundary to have the lowest value.

[0134] Further, in operation 1634, for example, the alignment sub-module 710 may cause all other pixels in the second rectified aerial image to have the highest value.

[0135] Further, for example, the alignment sub-module 710 may determine a particular distance for a particular pixel among all other pixels in the second corrected aerial image in operation 1636. For example, the particular distance may be between a position at a location represented by the center of the particular pixel and a position at a location represented by the nearest pixel having the lowest value.

[0136] Further, in operation 1638, for example, the alignment sub-module 710 may change the value for a particular pixel from the highest value to a particular value representing a particular distance as an operation to generate a second segmentation mask.

[0137] For example, the location of the point relevant to the generation of the version of the digital map may be the location of the point at the lane boundary and may match the location of a particular pixel in the second segmentation mask. Further, in operation 1640, for example, the alignment submodule 710 may move the location of the point relevant to the generation of the version of the digital map by a particular distance to a location that matches the location of the nearest pixel in the second segmentation mask that has the lowest value as an operation to generate the aerial image aligned digital map.

[0138] 18 includes a flow diagram illustrating a second example method 1800 associated with aligning positions of points associated with generating a version of a digital map of a place to positions of points in an aerial image of the place to generate an aerial image-aligned digital map of the place, in accordance with the techniques of this disclosure. Although the method 1800 is described in conjunction with the system 700 shown in FIG. 7, it will be understood by those skilled in the art in light of the description herein that the method 1800 is not limited to implementation by the system 700 shown in FIG. 7. Rather, the system 700 shown in FIG. 7 is one example of a system that may be used to implement the method 1800. Additionally, while the method 1800 is generally illustrated as a sequential process, various aspects of the method 1800 may be capable of being performed in parallel.

[0139] In method 1800, at operation 1802, for example, the alignment submodule 710 may detect within an aerial image the indication of lane boundaries on a maintained surface (e.g., a drivable surface (e.g., a road)) used by vehicles and pedestrians.

[0140] At operation 1804, for example, the alignment sub-module 710 may generate a modified aerial image by causing each pixel in the aerial image associated with a lane boundary to have the lowest value.

[0141] At operation 1806, for example, the alignment sub-module 710 may align all other pixels in the rectified aerial image to have the highest value.

[0142] At operation 1808, for example, the alignment sub-module 710 may determine a distance for a pixel among all other pixels.

[0143] At operation 1810, for example, the alignment submodule 710 may change the value for the pixel from the highest value to a value that represents the distance, for example, between the location represented by the center of the pixel and the location represented by the nearest pixel with the lowest value, as an operation to generate a segmentation mask.

[0144] For example, the location of the point relevant to the generation of the version of the digital map may be the location of the point at the lane boundary and may match the location of the pixel in the segmentation mask. Further, in operation 1812, for example, the alignment submodule 710 may move the location of the point relevant to the generation of the version of the digital map by the distance to a location that matches the location of the nearest pixel in the segmentation mask that has the lowest value as an operation to generate the aerial image aligned digital map.

[0145] FIG. 19 includes a block diagram illustrating an example of elements disposed in a vehicle 1900 in accordance with the techniques of this disclosure. As used herein, a "vehicle" may be any form of motorized transportation. In one or more implementations, the vehicle 1900 may be an automobile. Although the mechanisms described herein relate to automobiles, in light of the description herein, it will be understood by those skilled in the art that the embodiments are not limited to automobiles. For example, the functions and / or operations of one or more of the first vehicle 164 (shown in FIG. 1), the second vehicle 166 (shown in FIG. 1), or the third vehicle 168 (shown in FIG. 1) may be implemented by the vehicle 1900.

[0146] In some embodiments, vehicle 1900 may be configured to selectively switch between an automatic mode, one or more semi-automatic operating modes, and / or a manual mode. Such switching may be implemented in any suitable manner now known or later developed. As used herein, "manual mode" may refer to all or most of the navigation and / or operation of vehicle 1900 being performed according to input received from a user (e.g., a human driver). In one or more arrangements, vehicle 1900 may be a conventional vehicle configured to operate only in a manual mode.

[0147] In one or more embodiments, the vehicle 1900 may be an automated vehicle. As used herein, an "automated vehicle" may refer to a vehicle operating in an automated mode. As used herein, an "automated mode" may refer to using one or more computing systems to control the vehicle 1900 with minimal or no input from a human driver to navigate and / or operate the vehicle 1900 along a travel route. In one or more embodiments, the vehicle 1900 may be highly automated or fully automated. In an embodiment, the vehicle 1900 may be configured with one or more semi-automated modes of operation in which one or more computing systems perform a portion of the navigation and / or operation of the vehicle along a travel route, and a vehicle operator (i.e., the driver) provides input to the vehicle 1900 to perform a portion of the navigation and / or operation of the vehicle 1900 along a travel route.

[0148] For example, the taxonomy and definitions of terms related to driving automation systems for road vehicles in standard J3016 202104, published by the Society of Automotive Engineers (SAE) International on January 16, 2014, and last revised on April 30, 2021, defines six levels of driving automation: (1) Level 0, no automation, where all aspects of dynamic driving tasks are performed by a human driver; (2) Level 1, driver assistance, where a driver assistance system, if selected, may perform either steering or acceleration / deceleration tasks using information about the driving environment, but all remaining dynamic driving tasks are performed by a human driver; and (3) Level 2, partial automation, where one or more driver assistance systems, if selected, may perform both steering and acceleration / deceleration tasks using information about the driving environment, but all remaining dynamic driving tasks are performed by a human driver. (4) Level 3, conditional automation, where the automated driving system may, if selected, perform all aspects of the dynamic driving task with the possibility that a human driver may respond appropriately to a request to intervene; (5) Level 4, high automation, where the automated driving system may, if selected, perform all aspects of the dynamic driving task even if the human driver does not respond appropriately to a request to intervene; and (6) Level 5, full automation, where the automated driving system may perform all aspects of the dynamic driving task under all roadway and environmental conditions that may be managed by a human driver.

[0149] Vehicle 1900 may include various elements. Vehicle 1900 may have any combination of the various elements shown in FIG. 19. In various embodiments, vehicle 1900 may not need to include all of the elements shown in FIG. 19. Additionally, vehicle 1900 may have elements other than those shown in FIG. 19. Although various elements are shown in FIG. 19 as being located within vehicle 1900, one or more of the elements may be located outside vehicle 1900. Additionally, the elements shown may be physically separated by large distances. For example, as described, one or more components of the system of the present disclosure may be implemented within vehicle 1900, while other components of the system may be implemented within a cloud computing environment, as described below. For example, the elements may include one or more processors 1910, one or more data stores 1915, a sensor system 1920, an input system 1930, an output system 1935, a vehicle system 1940, one or more actuators 1950, one or more autonomous driving modules 1960, and a communication system 1970.

[0150] In one or more arrangements, the one or more processors 1910 may be a main processor of the vehicle 1900. For example, the one or more processors 1910 may be an electronic control unit (ECU).

[0151] The one or more data stores 1915 may store, for example, one or more types of data. The one or more data stores 1915 may include volatile memory and / or non-volatile memory. Examples of suitable memory for the one or more data stores 1915 may include random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, magnetic disks, optical disks, hard drives, any other suitable storage medium, or any combination thereof. The one or more data stores 1915 may be components of the one or more processors 1910. Additionally or alternatively, the one or more data stores 1915 may be operably connected to the one or more processors 1910 for use by the one or more processors 1910. As used herein, "operably connected" includes direct or indirect connections and may include connections without direct physical contact. As used herein, the statement that a component may be "configured to" perform an operation may be understood to mean that the component does not require structural modification, but merely needs to be placed in an operational state (e.g., provided with power, having basic operating systems running, etc.) to perform the operation.

[0152] In one or more arrangements, one or more data stores 1915 may store map data 1916. The map data 1916 may include maps of one or more geographic areas. In some examples, the map data 1916 may include information or data about roads, traffic control devices, road marks, structures, features, and / or landmarks in one or more geographic areas. The map data 1916 may be in any suitable form. In some examples, the map data 1916 may include aerial photographs of the area. In some examples, the map data 1916 may include ground photographs of the area, including 360 degree ground photographs. The map data 1916 may include measurements, dimensions, distances, and / or information for one or more items included in the map data 1916 and / or for other items included in the map data 1916. The map data 1916 may include a digital map with information about road geometry. The map data 1916 may be of high quality and / or high definition. For example, one or more functions and / or operations of digital map 900 (shown in FIG. 9) may be implemented by map data 1916.

[0153] In one or more arrangements, the map data 1916 may include one or more terrain maps 1917. The one or more terrain maps 1917 may include information about the ground, terrain, roads, terrain surfaces, and / or other features of one or more geographic areas. The one or more terrain maps 1917 may include elevation data for one or more geographic areas. The map data 1916 may be of high quality and / or high definition. The one or more terrain maps 1917 may define one or more ground surfaces, which may include paved roads, unpaved roads, land, and others that define a ground surface.

[0154] In one or more arrangements, the map data 1916 may include one or more stationary obstacle maps 1918. The one or more stationary obstacle maps 1918 may include information about one or more stationary obstacles located within one or more geographic areas. A "stationary obstacle" may be a physical object whose location does not change (or does not change substantially) over a period of time and / or whose size does not change (or does not change substantially) over a period of time. Examples of stationary obstacles may include trees, buildings, curbs, fences, railings, centerlines, utility poles, statues, monuments, signs, benches, furniture, mailboxes, large rocks, and hills. A stationary obstacle may be an object that extends above ground level. One or more stationary obstacles included in the one or more stationary obstacle maps 1918 may have location data, size data, dimension data, material data, and / or other data associated with them. The one or more stationary obstacle maps 1918 may include measurements, dimensions, distances, and / or information about one or more stationary obstacles. The one or more static obstacle maps 1918 may be of high quality and / or high definition. The one or more static obstacle maps 1918 may be updated to reflect changes in the map area.

[0155] In one or more arrangements, one or more data stores 1915 may store sensor data 1919. As used herein, "sensor data" may refer to any information related to a sensor that the vehicle 1900 may include, including capabilities and other information related to that sensor. The sensor data 1919 may relate to one or more sensors of a sensor system 1920. For example, in one or more arrangements, the sensor data 1919 may include information related to one or more LiDAR sensors 1924 of the sensor system 1920.

[0156] In some arrangements, at least a portion of the map data 1916 and / or the sensor data 1919 may be located in one or more data stores 1915 located on-board the vehicle 1900. Additionally or alternatively, at least a portion of the map data 1916 and / or the sensor data 1919 may be located in one or more data stores 1915 located remotely from the vehicle 1900.

[0157] The sensor system 1920 may include one or more sensors. As used herein, a "sensor" may refer to any device, component, and / or system that can detect and / or sense something. The one or more sensors may be configured to detect and / or sense in real-time. As used herein, the term "real-time" may refer to a level of processing responsiveness that is quickly enough recognized by a user or system for a particular process or decision to be made or that allows a processor to follow some external process.

[0158] In arrangements where the sensor system 1920 includes multiple sensors, the sensors may function independently of one another. Alternatively, two or more of the sensors may function in combination with one another. In such cases, the two or more sensors may form a sensor network. The sensor system 1920 and / or one or more sensors may be operatively connected to one or more processors 1910, one or more data stores 1915, and / or another element (including any of the elements shown in FIG. 19) of the vehicle 1900. The sensor system 1920 may obtain data regarding at least a portion of the external environment of the vehicle 1900 (e.g., nearby vehicles). The sensor system 1920 may include any suitable type of sensor. Various examples of various types of sensors are described herein. However, it will be understood by those skilled in the art that the embodiments are not limited to the specific sensors described herein.

[0159] The sensor system 1920 may include one or more vehicle sensors 1921. The one or more vehicle sensors 1921 may detect, determine, and / or sense information about the vehicle 1900 itself. In one or more arrangements, the one or more vehicle sensors 1921 may be configured to detect and / or sense changes in position and orientation of the vehicle 1900, for example, based on inertial acceleration or the like. In one or more arrangements, the one or more vehicle sensors 1921 may include one or more accelerometers, one or more gyroscopes, an inertial measurement unit (IMU), a dead reckoning system, a global navigation satellite system (GNSS), a global positioning system (GPS), a navigation system 1947, and / or other suitable sensors. The one or more vehicle sensors 1921 may be configured to detect and / or sense one or more characteristics of the vehicle 1900. In one or more arrangements, the one or more vehicle sensors 1921 may include a speedometer for determining a current speed of the vehicle 1900.

[0160] Additionally or alternatively, the sensor system 1920 may include one or more environmental sensors 1922 configured to obtain and / or detect driving environment data. As used herein, "driving environment data" may include data or information regarding the external environment in which the vehicle is located or one or more portions thereof. For example, the one or more environmental sensors 1922 may be configured to detect, quantify, and / or detect obstacles in at least a portion of the external environment of the vehicle 1900 and / or information / data regarding the obstacles. The obstacles may be stationary objects and / or dynamic objects. The one or more environmental sensors 1922 may be configured to detect, measure, quantify, and / or detect other things in the external environment of the vehicle 1900, such as, for example, lane markers, signs, traffic lights, traffic signs, lanes, crosswalks, curbs near the vehicle 1900, off-road objects, etc.

[0161] Described herein are various examples of sensors for sensor system 1920. The example sensors may be part of one or more vehicle sensors 1921 and / or one or more environmental sensors 1922. However, it will be understood by those skilled in the art that the embodiments are not limited to the particular sensors described.

[0162] In one or more arrangements, the one or more environmental sensors 1922 may include one or more radar sensors 1923, one or more LiDAR sensors 1924, one or more sonar sensors 1925, and / or one or more cameras 1926. In one or more arrangements, the one or more cameras 1926 may be one or more high dynamic range (HDR) cameras or one or more infrared (IR) cameras. For example, the one or more cameras 1926 may be used to record real-world conditions related to items of information that may appear on a digital map. For example, the functionality and / or operation of forward-facing camera 170 (shown in FIG. 1 ) or forward-facing camera 172 (shown in FIG. 1 ) may be realized by one or more cameras 1926.

[0163] Input system 1930 may include any device, component, system, element, mechanism, or group thereof that allows for information / data to be input into the machine. Input system 1930 may receive input from a vehicle occupant (e.g., the driver or passenger). Output system 1935 may include any device, component, system, element, mechanism, or group thereof that allows for information / data to be presented to a vehicle occupant (e.g., the driver or passenger).

[0164] Various examples of one or more vehicle systems 1940 are shown in FIG. 19 . However, it will be understood by one of ordinary skill in the art that the vehicle 1900 may include more, fewer, or different vehicle systems. Although certain vehicle systems may be defined separately, each or any of the systems or portions thereof may be otherwise combined or separated within the vehicle 1900 via hardware and / or software. For example, the one or more vehicle systems 1940 may include a propulsion system 1941, a braking system 1942, a steering system 1943, a throttle system 1944, a transmission system 1945, a signaling system 1946, and / or a navigation system 1947. Each of these systems may include one or more devices, components, and / or combinations thereof, now known or later developed.

[0165] Navigation system 1947 may include one or more devices, applications, and / or combinations thereof, now known or later developed, configured to determine a geographic location of vehicle 1900 and / or determine driving routes for vehicle 1900. Navigation system 1947 may include one or more map applications for determining driving routes for vehicle 1900. Navigation system 1947 may include a global positioning system, a local positioning system, a geolocation system, and / or combinations thereof.

[0166] The one or more actuators 1950 may be any element or combination of elements operable to modify, adjust, and / or alter one or more of the vehicle systems 1940 or its components in response to receiving signals or other inputs from the one or more processors 1910 and / or the one or more autonomous driving modules 1960. Any suitable actuator may be used. For example, the one or more actuators 1950 may include motors, pneumatic actuators, hydraulic pistons, relays, solenoids, and / or piezoelectric actuators.

[0167] The one or more processors 1910 and / or the one or more autonomous driving modules 1960 may be operatively connected to communicate with the various vehicle systems 1940 and / or their individual components. For example, the one or more processors 1910 and / or the one or more autonomous driving modules 1960 may communicate to send and / or receive information from the various vehicle systems 1940 to control the movement, speed, operation, heading, direction, etc. of the vehicle 1900. The one or more processors 1910 and / or the one or more autonomous driving modules 1960 may control some or all of these vehicle systems 1940 and thus may be partially or fully autonomous.

[0168] The one or more processors 1910 and / or the one or more autonomous driving modules 1960 may be operable to control the navigation and / or operation of the vehicle 1900 by controlling the vehicle systems 1940 and / or one or more of its components. For example, when operating in an autonomous mode, the one or more processors 1910 and / or the one or more autonomous driving modules 1960 may control the direction and / or speed of the vehicle 1900. The one or more processors 1910 and / or the one or more autonomous driving modules 1960 may accelerate the vehicle 1900 (e.g., by increasing the supply of fuel provided to the engine), decelerate the vehicle 1900 (e.g., by decreasing the supply of fuel to the engine and / or by applying the brakes), and / or change direction of the vehicle 1900 (e.g., by turning the front two wheels). As used herein, "cause" or "causing" may mean, either directly or indirectly, to make, force, compel, direct, command, indicate, and / or enable an event or action to occur, or to make, force, compel, direct, command, indicate, and / or enable the event or action to at least be in a state in which such event or action may occur.

[0169] The communication system 1970 may include one or more receivers 1971 and / or one or more transmitters 1972. The communication system 1970 may receive and transmit one or more messages over one or more wireless communication channels. For example, the one or more wireless communication channels may conform to the Institute of Electrical and Electronics Engineers (IEEE) 802.11p standard for adding wireless access in vehicular environments (WAVE) (based on Dedicated Short Range Communications (DSRC)), the 3rd Generation Partnership Project (3GPP®) Long Term Evolution (LTE) Vehicle-to-Everything (V2X) (LTE-V2X) standard (including the LTE Uu interface between mobile communication devices and Evolved Node Bs of the Universal Mobile Telecommunications System), the 3GPP® Fifth Generation (5G) New Radio (NR) Vehicle-to-Everything (V2X) standard (including the 5G NR Uu interface), or the like. For example, the communication system 1970 may include "connected vehicle" technology. “Connected vehicle” technologies may include, for example, devices for exchanging communications between a vehicle and other devices in a packet-switched network. The other devices may include, for example, another vehicle (e.g., “vehicle-to-vehicle” (V2V) technology), roadside infrastructure (e.g., “vehicle-to-infrastructure” (V2I) technology), cloud platforms (e.g., “vehicle-to-cloud” (V2C) technology), pedestrians (e.g., “vehicle-to-pedestrian” (V2P) technology, or networks (e.g., “vehicle-to-network” (V2N) technology). “Vehicle-to-everything” (V2X) technology may integrate aspects of these individual communication technologies. For example, the functionality and / or operation of communication device 174 (shown in FIG. 1 ), communication device 176 (shown in FIG. 1 ), or communication device 178 (shown in FIG. 1 ) may be implemented by communication system 1970.

[0170] Additionally, the one or more processors 1910, the one or more data stores 1915, and the communication system 1970 may be configured to one or more of form a micro-cloud, participate as a member of a micro-cloud, or act as a mobile micro-cloud leader. A micro-cloud may be characterized by the distribution of one or more of one or more computational resources or one or more data storage resources among the members of the micro-cloud to collaborate in performing operations. The members may include at least connected vehicles.

[0171] Vehicle 1900 may include one or more modules, at least some of which are described herein. The modules may be implemented as computer readable program code that, when executed by one or more processors 1910, implements one or more of the various processes described herein. One or more of the modules may be components of one or more processors 1910. Additionally or alternatively, one or more of the modules may be executed on and / or distributed among other processing systems to which one or more processors 1910 may be operatively connected. The modules may include instructions (e.g., program logic) executable by one or more processors 1910. Additionally or alternatively, one or more data stores 1915 may include the instructions.

[0172] In one or more arrangements, one or more of the modules described herein may include artificial or computational intelligence elements, such as neural networks, fuzzy logic, or other machine learning algorithms. Further, in one or more arrangements, one or more of the modules may be distributed among multiple modules described herein. In one or more arrangements, two or more of the modules described herein may be combined into a single module.

[0173] The vehicle 1900 may include one or more autonomous driving modules 1960. The one or more autonomous driving modules 1960 may be configured to receive data from the sensor system 1920 and / or any other type of system capable of capturing information about the vehicle 1900 and / or the vehicle's 1900's external environment. In one or more mechanisms, the one or more autonomous driving modules 1960 may use the data to generate one or more driving scene models. The one or more autonomous driving modules 1960 may determine the position and speed of the vehicle 1900. The one or more autonomous driving modules 1960 may determine the location of obstacles, obstacles, or other environmental features including traffic signs, trees, shrubs, nearby vehicles, pedestrians, etc.

[0174] One or more autonomous driving modules 1960 may be configured to receive and / or determine location information about obstacles in the external environment of vehicle 1900 for use by one or more processors 1910 and / or one or more of the modules described herein, and to estimate the position and orientation of vehicle 1900, the vehicle's position in global coordinates, based on signals from multiple satellites or any other data and / or signals that may be used to determine the current state of vehicle 1900 or to determine the position of vehicle 1900 relative to its environment used in creating a map or determining the position of vehicle 1900 relative to map data.

[0175] The one or more autonomous driving modules 1960 may be configured to determine one or more driving paths, a current autonomous driving maneuver for the vehicle 1900, a future autonomous driving maneuver, and / or a modification to the current autonomous driving maneuver based on data from any other suitable sources, such as data acquired by the sensor system 1920, a driving scene model, and / or a determination from the sensor data 1919. As used herein, a "driving maneuver" may refer to one or more actions that affect the movement of the vehicle. Examples of driving maneuvers include accelerating, decelerating, braking, turning, moving the vehicle 1900 laterally, changing lanes of travel, merging into lanes of travel, and / or backing up, just to name a few possibilities. The one or more autonomous driving modules 1960 may be configured to implement the determined driving maneuvers. The one or more autonomous driving modules 1960 may directly or indirectly cause such autonomous driving maneuvers to be implemented. As used herein, "cause" or "causing" means to cause, command, direct, and / or enable an event or action to occur, either directly or indirectly, or to cause, command, direct, and / or enable an event or action to at least become in a state in which such event or action may occur. One or more autonomous driving modules 1960 may be configured to perform various vehicle functions and / or transmit data to, receive data from, interact with, and / or control vehicle 1900 or one or more systems thereof (e.g., one or more of vehicle systems 1940). For example, the functions and / or operations of an automobile navigation system may be implemented by one or more autonomous driving modules 1960.

[0176] Detailed embodiments are disclosed herein. However, in view of the description herein, it will be understood by those skilled in the art that the embodiments of the present disclosure are intended to be merely examples. Therefore, the specific structural and functional details disclosed herein should not be construed as limiting, but merely as a basis for the claims and as a representative basis to teach those skilled in the art to variously employ the aspects of the present specification in substantially any suitable detailed structure. Furthermore, the terms and phrases used herein are not intended to be limiting, but rather to provide an understandable description of possible implementations. Although various embodiments are shown in Figures 1-5, 6A, 6B, 7-15, 16A-16D, and 17-19, the embodiments are not limited to the illustrated structures or applications.

[0177] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments. In this regard, each block in the flowchart or block diagram may represent a module, segment, or portion of code, comprising one or more executable instructions for implementing a specified logical function. In view of the description herein, it will be understood by those skilled in the art that in some alternative implementations, the functions described in the blocks may occur out of the order depicted by the figures. For example, two blocks depicted in succession may in fact be executed substantially simultaneously, or the blocks may be executed in reverse order, depending on the functionality involved.

[0178] The above-mentioned systems, components, and / or processes can be realized in hardware or a combination of hardware and software, in a centralized manner in one processing system, or in a distributed manner with various elements spread across several interconnected processing systems. Any kind of processing system or other device configured to perform the methods described herein is suitable. A typical combination of hardware and software can be a processing system having computer readable program code that, when loaded and executed, controls the processing system such that the processing system performs the methods described herein. The systems, components, and / or processes can also be incorporated in a computer readable storage, such as a computer program product or other data program storage device, readable by a machine, tangibly embodying a program of instructions executable by the machine to perform the methods and processes described herein. These elements can also be incorporated in an application product that comprises all the features that enable the implementation of the methods described herein and that, when loaded in a processing system, can execute the methods.

[0179] Furthermore, the mechanisms described herein may take the form of a computer program product in which computer readable program code is embodied, e.g., stored, in one or more computer readable media. Any combination of one or more computer readable media may be utilized. The computer readable medium may be a computer readable signal medium or a computer readable storage medium. The phrase "computer readable storage medium" as used herein means a non-transitory storage medium. The computer readable storage medium may be, for example, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination thereof. More specific examples of computer readable storage media include, in a non-exhaustive list, the following: portable computer diskette, hard disk drive (HDD), solid state drive (SSD), read only memory (ROM), erasable programmable read only memory (EPROM or flash memory), portable compact disc read only memory (CD-ROM), digital versatile disk (DVD), optical storage device, magnetic storage device, or any suitable combination thereof. As used herein, a computer-readable storage medium may be any tangible medium that can contain or store a program for use by or in connection with an instruction execution system, apparatus, or device.

[0180] Generally, a module as used herein includes a routine, program, object, component, data structure, etc., that performs a particular task or implements a particular data type. In a further aspect, a memory generally stores such a module. The memory associated with a module may be a buffer or may be a cache, a random access memory (RAM), a ROM, a flash memory, or another suitable electronic storage medium incorporated within a processor. In yet a further aspect, a module as used herein may be implemented as an application specific integrated circuit (ASIC), as a hardware component of a system on a chip (SoC), as a programmable logic array (PLA), or as another suitable hardware component that incorporates a configuration set (e.g., instructions) defined to perform the functions of the present disclosure.

[0181] The program code embodied on the computer-readable medium may be transmitted using any suitable medium, including, but not limited to, wireless, wired, optical fiber, cable, radio frequency (RF), etc., or any suitable combination of the above. The computer program code performing operations for aspects of the technology of this disclosure may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, Smalltalk, C++, or the like, and traditional procedural programming languages ​​such as the "C" programming language or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or wide area network (WAN), or a connection to an external computer may be made (e.g., through the Internet using an Internet Service Provider).

[0182] The terms "a" and "an," as used herein, are defined as one or more than one. The term "multiple," as used herein, are defined as two or more than two. The term "another," as used herein, are defined as at least a second or more. The terms "including" and / or "having," as used herein, are defined as comprising (i.e., open language). The phrase "at least one of ... or ..." as used herein refers to and includes any and all possible combinations of one or more of the associated listed items. For example, the phrase "at least one of A, B, or C" includes A only, B only, C only, or any combination thereof (e.g., AB, AC, BC, or ABC).

[0183] The aspects herein may be embodied in other forms without departing from the spirit or essential attributes thereof, and reference should accordingly be made to the following claims, rather than the foregoing specification, as indicating the scope herein.

Claims

1. A processor; Memory, The system includes: a generation module including instructions that, when executed by the processor, cause the processor to generate a digital map from data relating to an image of a place, the data for the image excluding pixel color data; the pose of the camera that generated the image; and lane boundaries of lanes of a road in said image; the road boundary of said road, or Landmarks in the image and at least one of the positions of the points with respect to the a communications module comprising instructions that, when executed by the processor, cause the processor to transmit the digital map to the first vehicle for use in controlling movement of the first vehicle; A system that remembers the following:

2. The system of claim 1 , wherein the indicia comprises a sign.

3. The information regarding the label includes: a latitude position, a longitude position, and an altitude relative to the center of said landmark; The height of the sign; The width of the sign; The system of claim 2 , comprising:

4. The system of claim 2 , wherein the information about the sign includes information about a message conveyed by the sign.

5. the instructions for generating the digital map include instructions for processing the data associated with the image of the location using at least one concurrent localization and mapping technique; The system of claim 1 , wherein the location is a particular area.

6. 6. The system of claim 5, wherein the shape of the particular area is defined by seven regular hexagons, one of the seven regular hexagons being at a central position and another of the seven regular hexagons being disposed on a side of the one hexagon.

7. 2. The system of claim 1, wherein the communication module further includes instructions for receiving the data associated with the image from at least one second vehicle, the camera comprising at least one camera, the at least one camera being mounted on the at least one second vehicle.

8. a camera of the at least one camera is mounted on a vehicle of the at least one second vehicle; the instructions for receiving include instructions for receiving from the at least one second vehicle once every 30 seconds a transmission of the batches of data relating to the images generated by the camera of the at least one camera for a duration of 30 seconds; The instructions for generating the digital map include: instructions for classifying said images into key frames; instructions for processing the keyframes using at least one concurrent localization and mapping optimization technique; The system of claim 7 , comprising:

9. A first one of the key frames is characterized by a first criterion, the first criterion being a value of the data contained in the first key frame, A second one of the key frames is characterized by a second criterion, the second criterion being a value of the data contained in the second key frame, The system of claim 8 , wherein the difference between the first criterion and the second criterion is greater than a threshold value.

10. The system of claim 8 , wherein the count of images included in a keyframe of the keyframes is a function of a distance traveled by the vehicle of the at least one second vehicle.

11. 2. The system of claim 1, wherein the generation module includes an alignment submodule that, when executed by the processor, causes the processor to align the positions of the points associated with generating a version of the digital map to positions of points in an aerial image of the location to generate an aerial-image aligned digital map.

12. The instructions for aligning include: instructions for detecting within the first copy of the aerial imagery a representation of a road boundary in a maintained surface used by vehicles and pedestrians; instructions for generating a first modified aerial image by causing each pixel in the first copy of the aerial image associated with the road boundary to have a lowest value; instructions to cause every other pixel in the first rectified aerial image to have the highest value; instructions for determining a first distance for a first pixel of all other pixels, the first distance being between a position at a location represented by a center of the first pixel and a position at a location represented by a nearest first pixel having the lowest value; instructions for changing a value for the first pixel from the highest value to a first value representing the first distance as a first operation for generating a first segmentation mask; The system of claim 11 , comprising:

13. the location of the point associated with the generation of the version of the digital map is a location of a point on the road boundary that coincides with a location of the first pixel in the first segmentation mask; 13. The system of claim 12, wherein the instructions for aligning further include instructions for moving the location of the point associated with the generation of the version of the digital map by the first distance to a location that coincides with the location of the nearest first pixel in the first segmentation mask having the lowest value as an operation for generating a digital map of the aerial image alignment.

14. the first pixel, a second pixel of every other pixel, a third pixel of every other pixel, and a fourth pixel of every other pixel are arranged in a configuration having two rows and two columns; The instructions for aligning may further include: instructions for determining a second distance for the second pixel, the second distance being between a position at a location represented by a center of the second pixel and a position at a location represented by a nearest second pixel having the lowest value; as a second operation for generating the first segmentation mask, instructions for changing a value for the second pixel from the highest value to a second value representing the second distance; instructions for determining a third distance for the third pixel, the third distance being between a position at a location represented by a center of the third pixel and a position at a location represented by a nearest third pixel having the lowest value; as a third operation for generating the first segmentation mask, instructions for changing a value for the third pixel from the highest value to a third value representing the third distance; instructions for determining a fourth distance for the fourth pixel, the fourth distance being between a position at a location represented by a center of the fourth pixel and a position at a location represented by a nearest fourth pixel having the lowest value; as a fourth operation for generating the first segmentation mask, instructions for changing a value for the fourth pixel from the highest value to a fourth value representing the fourth distance; as a fifth operation for generating the first segmentation mask, instructions for determining, for a particular point within the first pixel, the second pixel, the third pixel, or the fourth pixel, a corresponding value representing a corresponding distance between a position at a location represented by the particular point and a position at a location represented by a nearest corresponding pixel having the lowest value based on the first value, the second value, the third value, and the fourth value; The system of claim 12 , comprising:

15. The instructions for aligning include: instructions for detecting within the aerial imagery indications of lane boundaries on a maintained surface used by vehicles and pedestrians; instructions for generating a modified aerial image by causing each pixel in the aerial image associated with the lane boundary to have a lowest value; instructions for causing every other pixel in said modified aerial image to have the highest value; instructions for determining a distance for a pixel of all other pixels, the distance being between a position at a location represented by a center of the pixel and a position at a location represented by a nearest pixel having the lowest value; instructions for changing a value for the pixel from the highest value to a value representing the distance as an operation to generate a segmentation mask; The system of claim 11 , comprising:

16. the locations of the points relative to the generation of the version of the digital map are locations of points at the lane boundaries that correspond to the locations of the pixels in the segmentation mask; 16. The system of claim 15, wherein the instructions for aligning further include instructions for moving the location of the point associated with the generation of the version of the digital map by the distance to a location that coincides with the location of the nearest pixel in the segmentation mask having the lowest value as an operation for generating a digital map of the aerial image alignment.

17. generating, by a processor, a digital map from data relating to an image of the location, said data for the image excluding pixel color data; the pose of the camera that generated the image; and lane boundaries of lanes of a road in said image; the road boundary of said road, or Landmarks in the image and at least one of a position of the point with respect to transmitting, by the processor, the digital map to the vehicle for use in controlling movement of the vehicle; A method comprising:

18. 20. The method of claim 17, wherein the at least one of the positions of the points relative to the lane boundary, road boundary, or landmark is related to a position of a key point of an object in the image that represents the lane boundary, road boundary, or landmark.

19. 18. The method of claim 17, wherein the generating of the digital map includes processing the data associated with the images using at least one data association technique to determine a correspondence between the locations of the points associated with particular objects included in a first one of the images and the locations of the points associated with the particular objects included in a second one of the images.

20. 1. A non-transitory computer-readable medium for generating a digital map from data associated with images of places, the non-transitory computer-readable medium comprising instructions that, when executed by one or more processors, cause the one or more processors to: generating said digital map from said data, said data for an image excluding pixel color data, the pose of the camera that generated the image; and lane boundaries of lanes of a road in said image; the road boundary of said road, or Landmarks in the image and at least one of a position of the point with respect to transmitting said digital map to said vehicle for use in controlling movement of said vehicle; Non-transitory computer-readable medium.

Images