Pedestrian crossing detection

Abstract

Systems and methods for detecting a crosswalk location are disclosed. A crosswalk location can be detected by determining at least two colored lines on a road surface and then combining the at least two colored lines on the road surface into a grouping of related elements. The grouping of related elements can be classified as a crosswalk (or in some embodiments, a type of crosswalk) based on attributes of the grouping of related elements, where the attributes of the grouping of elements include a distance between the at least two colored lines and an orientation of the grouping of related elements on the road surface.

Description

[0001] Cross-reference to related applications

[0002] This application claims priority to U.S. Application No. 16 / 589,020, filed on September 30, 2019, entitled “Pedestrian Crossing Detection,” the entire contents of which are hereby expressly incorporated by reference. Technical Field

[0003] This technology relates to using data captured by autonomous vehicles to update a portion of a map database, and more specifically to using data obtained from autonomous vehicles to update the location of pedestrian crossings in a portion of a map database with high-resolution data. Background Technology

[0004] An autonomous vehicle is a motor vehicle that can navigate without a human driver. An exemplary autonomous vehicle includes multiple sensor systems, such as, but not limited to, camera sensor systems, lidar sensor systems, radar sensor systems, etc., wherein the autonomous vehicle operates based on sensor signals output by these sensor systems. Specifically, the sensor signals are provided to an internal computing system that communicates with the multiple sensor systems, where a processor executes instructions based on the sensor signals to control the autonomous vehicle's mechanical systems, such as its propulsion, braking, or steering systems.

[0005] Autonomous vehicles navigate using a combination of data captured by at least one of their sensors and maps stored on the vehicle. These maps are typically created using dedicated mapping vehicles that capture data at a much higher resolution than at least one sensor on the autonomous vehicle. However, in some locations, maps may be incomplete, missing, or pedestrian crossing locations may have changed due to construction, lane changes, or other factors. When this occurs, autonomous vehicles may encounter navigation problems based on incomplete or outdated maps, and inconsistent road sections may become restricted areas for the vehicle until the map is updated. Attached Figure Description

[0006] The above and other advantages and features of this technology will become apparent from the specific implementations shown in the accompanying drawings. Those skilled in the art will understand that these drawings illustrate only some examples of this technology and are not intended to limit the scope of the technology to these examples. Furthermore, those skilled in the art will understand the principles of this technology as described and explained with additional features and details using the accompanying drawings, wherein:

[0007] Figure 1 An example system for driving and managing autonomous vehicles according to some aspects of this technology is shown;

[0008] Figure 2 An example system for updating pedestrian crossing sections in a map, based on some aspects of this technology, is shown;

[0009] Figure 3 An example visualization of data according to some aspects of this technology is shown, which illustrates sensor data for detecting pedestrian crossings;

[0010] Figure 4 An example visualization of data filtering and pedestrian crossing type classification based on some aspects of this technology is shown;

[0011] Figure 5 An example visualization of data filtering and pedestrian crossing type classification based on some aspects of this technology is shown;

[0012] Figure 6 An example method for detecting pedestrian crossings according to some aspects of this technology is shown;

[0013] Figure 7 An example of a system used to implement some aspects of this technology is shown. Detailed Implementation

[0014] Various examples of this technology are discussed in detail below. While specific implementations are discussed, it should be understood that this is done for illustrative purposes only. Those skilled in the art will recognize that other components and configurations can be used without departing from the spirit and scope of this technology. In some cases, well-known structures and devices are shown in block diagram form for ease of description of one or more aspects. Furthermore, it should be understood that functions described as being performed by certain system components can be performed by more or fewer components than those shown.

[0015] The disclosed technology addresses the need in the art for a technology that can rapidly update maps used for route selection for autonomous vehicles, reflecting changing pedestrian crossing locations, without requiring dedicated map-making vehicles or human intervention.

[0016] Autonomous vehicles navigate using a combination of data captured by at least one of their sensors and maps stored on the vehicle. These maps are typically created using dedicated mapping vehicles that capture data at a much higher resolution than at least one of the sensors on the autonomous vehicle. In this way, a map is generated before navigation to inform the autonomous vehicle's navigation, allowing it to determine the route to the specific destination once it receives the information. However, pedestrian crossing locations may appear or change since the dedicated mapping vehicle last mapped the route. For example, if pedestrian crossings have been mapped on the route since the last time the dedicated mapping vehicle mapped the route, and the autonomous vehicle relies on the map's semantic boundaries indicating pedestrian crossings to drive, then the autonomous vehicle will not behave correctly on the road because the traffic signs have been modified. Since updating high-resolution maps requires dedicated mapping vehicles, a considerable amount of time may pass before the high-resolution maps are updated.

[0017] The disclosed technology addresses the aforementioned problems by enabling various embodiments to locate pedestrian crossings relative to areas where the map has changed. The changed extent can trigger various responses, such as determining which parts of the map can be trusted (e.g., using stored semantic boundaries in certain parts) vs. determining which parts of the map need updating (e.g., not using stored semantic boundaries), real-time detection of pedestrian crossings, and redrawing semantic boundaries based on detected lines, etc.

[0018] This technology provides a system that can update the semantic labels of maps stored on autonomous vehicles in real time using data from sensors, thus avoiding the need to schedule dedicated mapping vehicles for these updates. This makes route planning and scheduling for autonomous vehicles more efficient and significantly reduces downtime before map updates are released. This technology also reduces the workload on dedicated mapping vehicles, as they are scheduled less frequently. Therefore, even when maps require updates from dedicated mapping vehicles, the time spent updating the maps is shortened due to the reduced number of jobs queued by dedicated mapping vehicles. Consequently, semantic label updates can be released faster, regardless of whether dedicated mapping vehicles are needed, and the cycle of autonomous vehicle access to areas restricted by outdated maps is shortened.

[0019] In the following systems and methods, the location of a pedestrian crossing can be detected by identifying at least two colored lines on a road surface and then grouping the at least two colored lines on the road surface into a group of related elements. The group of related elements can be classified as a pedestrian crossing (or, in some embodiments, a type of pedestrian crossing) based on attributes of the grouping of elements, wherein the attributes of the grouping of elements include the distance between at least two colored lines and the orientation of the grouping of related elements on the road surface.

[0020] Figure 1 An environment 100 is shown, including an unmanned vehicle 102 communicating with a remote computing system 150.

[0021] The autonomous vehicle 102 can navigate on roads without a human driver using sensor signals output from its sensor systems 104-106 and maps stored in map database 123. The autonomous vehicle 102 includes multiple sensor systems 104-106 (first sensor system 104 to Nth sensor system 106). The sensor systems 104-106 are of different types and are arranged around the autonomous vehicle 102. For example, the first sensor system 104 may be a camera sensor system, and the Nth sensor system 106 may be a lidar sensor system. Other exemplary sensor systems include radar sensor systems, global positioning system (GPS) sensor systems, inertial measurement units (IMUs), infrared sensor systems, laser sensor systems, sonar sensor systems, etc.

[0022] The autonomous vehicle 102 also includes several mechanical systems for achieving proper movement of the autonomous vehicle 102. For example, the mechanical systems may include, but are not limited to, a vehicle propulsion system 130, a braking system 132, and a steering system 134. The vehicle propulsion system 130 may include an electric motor, an internal combustion engine, or both. The braking system 132 may include engine brakes, brake pads, actuators, and / or any other suitable components configured to assist in decelerating the autonomous vehicle 102. The steering system 134 includes suitable components configured to control the direction of movement of the autonomous vehicle 102 during navigation.

[0023] The driverless vehicle 102 also includes a safety system 136, which may include various lights and signal indicators, parking brakes, airbags, etc. The driverless vehicle 102 also includes a passenger compartment system 138, which may include a passenger compartment temperature control system, an in-passenger compartment entertainment system, etc.

[0024] The autonomous vehicle 102 further includes an internal computing system 110 that communicates with sensor systems 104-106 and systems 130, 132, 134, 136, and 138. The internal computing system includes at least one processor and at least one memory having computer-executable instructions that are executed by the processor. The computer-executable instructions may constitute one or more services responsible for controlling the autonomous vehicle 102, communicating with the remote computing system 150, receiving input from passengers or a human co-pilot, recording measurements of data collected by sensor systems 104-106 and the human co-pilot, etc.

[0025] The internal computing system 110 may include a control service 112 configured to control the operation of the vehicle propulsion system 130, braking system 132, steering system 134, safety system 136, and cabin system 138. The control service 112 receives sensor signals from sensor systems 104-106 and communicates with other services of the internal computing system 110 to enable the operation of the autonomous vehicle 102. In some embodiments, the control service 112 may coordinate with one or more other systems of the autonomous vehicle 102 to perform operations.

[0026] The internal computing system 110 may also include a constraint service 114 to facilitate the safe movement of the autonomous vehicle 102. The constraint service 114 includes instructions for activating constraints based on rule-based restrictions on the operation of the autonomous vehicle 102. For example, a constraint may be a navigation restriction activated according to a protocol configured to avoid occupying space shared with other objects, comply with traffic laws, avoid avoidance zones, etc. In some embodiments, the constraint service may be part of a control service 112.

[0027] The internal computing system 110 may also include a communication service 116. The communication service may include software and hardware components for transmitting and receiving signals from / to the remote computing system 150. The communication service 116 is configured to wirelessly transmit information over a network, for example, via an antenna array providing personal cellular (LTE, 3G, 5G, etc.) communication.

[0028] In some embodiments, one or more services of the internal computing system 110 are configured to send and receive communications to the remote computing system 150 for reasons such as reporting data for training and evaluating machine learning algorithms, requesting assistance from the remote computing system or via the remote computing system 150 to a human operator, software service updates, map updates, carpooling instructions, etc.

[0029] The internal computing system 110 may also include a delayed service 118. The delayed service 118 may utilize timestamps of communications to and from the remote computing system 150 to determine whether useful communication has been received from the remote computing system 150 in a timely manner. For example, when the internal computing system 110 requests feedback from the remote computing system 150 regarding a time-sensitive process, the delayed service 118 may determine whether a response has been received from the remote computing system 150 in a timely manner, as the information may quickly become too outdated to be actionable. When the delayed service 118 determines that no response has been received within a threshold, it may enable other systems of the autonomous vehicle 102 or passengers to make necessary decisions or provide required feedback.

[0030] The internal computing system 110 may also include a user interface service 120 that can communicate with the cabin system 138 to provide or receive information to or from a human co-driver or human passenger. In some embodiments, the human co-driver or human passenger may be required to assess and disregard constraints from the constraint service 114, or the human co-driver or human passenger may wish to provide instructions to the driverless vehicle 102 regarding the destination, the requested route, or other requested actions.

[0031] Map Inconsistency Service 122 can compare data collected by sensors 104-106 with a map stored in map database 123. For example, a map may initially be created using pre-drawn data. The configuration of pedestrian crossings and roads often changes due to repainting, construction, or other factors. When this happens, Map Inconsistency Service 122 determines that the map stored in map database 123 reflects inconsistencies compared to the current conditions and can mark the inconsistent road sections for autonomous vehicles until the map is updated. Map Inconsistency Service 122 can communicate with Map Update Service 160 via Communication Service 116 to receive updated portions of the map.

[0032] As described above, the remote computing system 150 is configured to send / receive signals from the autonomous vehicle 102, including report data for training and evaluating machine learning algorithms, requests for assistance to or via the remote computing system 150 to a human operator, software service updates, map updates, ride-sharing instructions, etc.

[0033] The remote computing system 150 includes an analytics service 152 configured to receive data from the autonomous vehicle 102 and analyze the data to train or evaluate machine learning algorithms for operating the autonomous vehicle 102. The analytics service 152 can also perform analysis on data associated with one or more errors or constraints reported by the autonomous vehicle 102.

[0034] The remote computing system 150 may also include a user interface service 154 configured to present measurements, videos, images, and sounds reported from the autonomous vehicle 102 to the operator of the remote computing system 150. The user interface service 154 may also receive input instructions from the operator, which may be sent to the autonomous vehicle 102.

[0035] The remote computing system 150 may also include an instruction service 156 for sending instructions regarding the operation of the autonomous vehicle 102. For example, in response to the output of the analysis service 152 or the user interface service 154, the instruction service 156 may prepare instructions for one or more services of the autonomous vehicle 102 or for the co-driver or passenger of the autonomous vehicle 102.

[0036] The remote computing system 150 may also include a ride-sharing service 158 configured to interact with a ride-sharing application 170 operating on a (potential) passenger computing device. The ride-sharing service 158 can receive pick-up / drop-off requests from the passenger ride-sharing application 170 and can dispatch an autonomous vehicle 102 to perform the trip. The ride-sharing service 158 can also act as an intermediary between the ride-sharing application 170 and the autonomous vehicle, where passengers can provide instructions to the autonomous vehicle 102 to bypass obstacles, change routes, honk the horn, etc.

[0037] As described above, this technology provides a system that can use low-resolution data from at least one sensor 104-106 of an autonomous vehicle 102 to identify changed aspects of a map stored in a map database 123 of the autonomous vehicle 102, such as the current location of a pedestrian crossing. In some embodiments, the initial map may include pre-mapped data and semantic labels, wherein the pre-mapped data includes high-density points obtained from a high-resolution LiDAR system on a dedicated mapping vehicle, also known as a high-resolution point map, and the semantic labels identify features represented in the high-density points obtained from the high-resolution LiDAR system. The semantic labels may identify features such as lane lines, line colors, driving lanes, the location of stop signs and stop lights, pedestrian crossings, etc. In one or more embodiments, the map may also include low-resolution point map data indicating updates to the pre-mapped features and semantic labels as described herein.

[0038] In some embodiments, this technology can use low-resolution data from at least one sensor 104-106 of the autonomous vehicle 102 to detect new features, and an administrator can manually relabel semantic tags on top of existing high-density points obtained from a high-resolution LiDAR system already represented in a stored pre-drawn map. In some embodiments, semantic tags can be relabeled automatically.

[0039] Figure 2 An example system embodiment illustrating map update service 160 and map inconsistency service 122 in more detail is shown. While regarding... Figure 6 The method shown discusses Figure 2 The system is illustrated, but it should be understood that, except as defined in the claims, each of the figures represents its own separate embodiment and should not be limited by such cross-reference between the figures.

[0040] Map inconsistency service 122 is used to determine when pedestrian crossing features shown in a pre-drawn high-resolution map stored in map database 123 are inconsistent with pedestrian crossing features in the current data detected by sensors 104-106 (such as absence, addition, or other modification). While in some embodiments, sensors 104-106 may capture data at a lower resolution than reflected in the high-resolution map, the current data captured by sensors 104-106 is sufficient to determine such inconsistencies. Detecting these inconsistencies is important because the high-resolution map may become outdated and no longer reflect the road configuration. The current data from sensors 104-106 reflects the current configuration of pedestrian crossing locations.

[0041] When the autonomous vehicle 102 navigates a route, sensors 104-106 capture current data reflecting the environment surrounding the autonomous vehicle 102. As the autonomous vehicle 102 traverses a geographic area, such as an area including pedestrian crossings, the data aggregator 204 can accumulate current data from at least one sensor 104-106. Even over short distances, the data aggregator 204 can accumulate current data from the same sensor as the sensors continuously collect data. For example, in the case of a LiDAR sensor, the LiDAR continuously creates a point map based on current data collected from the environment surrounding the autonomous vehicle 102, and this data is aggregated by the data aggregator 204.

[0042] While at least one sensor 104-106 of the autonomous vehicle 102 is capturing current data, in some embodiments, the pedestrian crossing detector 202 can detect new features represented in the captured current data by clustering the data into feature types (such as pedestrian crossings, or, in some embodiments, the type of pedestrian crossing), and can compare the features detected in the captured current data with features represented in pre-drawn data (e.g., a map stored in the map database 123).

[0043] In some cases, the pedestrian crossing detector 202 can determine that features in the current data differ from features represented in pre-drawn data. For example, pre-drawn data may reflect pedestrian crossings at specific locations on the road, while the current data may reflect pedestrian crossings at different locations on the road. In some embodiments, the pedestrian crossing detector 202 can determine the type of feature based on characteristics of the current data. For example, the feature detector may determine that the pedestrian crossing has changed from a zebra crossing to a parallel crossing. Other examples of inconsistencies may include the addition or removal of pedestrian crossings, the presence or absence of pedestrian crossings, changes in pedestrian crossing orientation (e.g., closer to the lane at a 30-degree angle), etc.

[0044] If the pedestrian crossing detector 202 determines that the pre-drawn data does not include semantic labels reflecting the features represented in the current data, the pedestrian crossing detector 202 can publish the location and type of the detected inconsistencies. For example, if the pedestrian crossing detector 202 determines that the location of a pedestrian crossing differs from the location of a pedestrian crossing reflected in the pre-drawn data, the pedestrian crossing detector 202 can identify the location of the detected pedestrian crossing and recognize the change in location.

[0045] In some embodiments, the pedestrian crossing detector 202 can also classify the type of change. For example, when the change concerns a change in the pedestrian crossing type from a zebra crossing to a parallel crossing, the pedestrian crossing detector 202 can flag the inconsistency as a pedestrian crossing type added to a new location, a pedestrian crossing type removed from a intended location, or a pedestrian crossing type that has altered its current representation in the map data portion. For pedestrian crossing changes, the pedestrian crossing detector 202 can also include pedestrian crossing color (e.g., white, yellow, etc.). Furthermore, the pedestrian crossing detector 202 can identify the type of sensor that detected the change, such as a LiDAR sensor or a camera.

[0046] The data aggregator 204 can label accumulated current data from at least one sensor 104-106 as reflecting detected changes, and can send accumulated current data reflecting inconsistencies to the map update service 160. For example, the accumulated current data may include low-resolution point maps reflecting inconsistencies. The accumulated current data may also include any other sensor information from the autonomous vehicle 102, such as camera data for determining pedestrian crossing colors or bounding boxes, which can assist in and be incorporated into the map update.

[0047] In some embodiments, pre-drawn data may be stored as a collection of map portions. In such embodiments, when the current data being received by the data aggregator 204 is no longer applicable to that map portion, the data aggregator 204 may stop sending accumulated current data reflecting inconsistencies to the map update service 160. The data aggregator 204 may then collect current data reflecting inconsistencies regarding different map portions. In some embodiments, map portions may be defined by geographic boundaries with location coordinates, such as tiles that may be reflected on a physical map.

[0048] In some embodiments, map service 214 may indicate the current data as low-resolution data, which prevents the current data from being directly included in any updates to pre-drawn data (however, in some embodiments, low-resolution data may be used for label revisions included in pre-drawn data). Map service 214 may manage versions of map portions and access to map portions. Once a version of a map portion is under the management of map service 214, the map service may store the version of the map portion and make it accessible for viewing, and, where appropriate, include it in the latest version of the map data for use by the autonomous vehicle 102.

[0049] Once the received data has been stored as a version of the map portion and placed under the management of map service 214, marker service 222 can retrieve and display versions of map portions that are inconsistent with the pre-drawn data map portion, and check the low-resolution current data against the pre-drawn data map portion to confirm the existence of inconsistencies.

[0050] In some embodiments, if the labeling service 222 confirms an inconsistency, it may mark the relevant map portion of the pre-drawn data as restricted in the map service 214. When a map portion is marked as restricted by the map service 214, this information can be published to the autonomous vehicle 102, and the autonomous vehicle 102 may be prohibited from autonomously driving within the area represented on the restricted map portion. The autonomous vehicle 102 will be brought to a slow stop. In some embodiments, if the labeling service 222 confirms an inconsistency, the autonomous vehicle 102 may continue navigating, as long as the changed range does not make driving unsafe. In some embodiments, the autonomous vehicle 102 is able to dynamically analyze changes and modify the semantic labels on the map, and navigate accordingly. In some embodiments, the administrator is able to navigate the autonomous vehicle 102 through changed areas in real time or near real time.

[0051] In some embodiments, the labeling service 222 may further determine whether the inconsistency can be remedied with the current low-resolution data. If the inconsistency has the nature of requiring new high-resolution data, the scheduling service 218 may schedule a dedicated mapping vehicle to remap the locations represented in the map portion. New high-resolution data may be required when there is a large map portion inconsistency, the details of the inconsistency are not clear enough for relabeling the map portion, or for data deemed critical to the driving of the autonomous vehicle 102. Examples of serious inconsistencies that may require new high-resolution data would include extensive repainting of drivable areas associated with new intersections, adding new light rail lines, etc.

[0052] In response to annotation service 222 determining that inconsistencies can be remedied using low-resolution current data, annotation service 222 can analyze the received low-resolution current data and pre-drawn data to re-annotate the map data, resulting in updated portions of the map data. Annotation service 222 can use a heuristic approach to identify patterns requiring re-annotation, or in some embodiments, it can utilize machine learning algorithms to perform re-annotation. In some embodiments, all or part of the re-annotation can be performed manually, and the machine learning algorithm provides human annotators with clues to revise and update the map. Then, in some embodiments, the autonomous vehicle 102 can stop, and in some embodiments, it can be remotely driven manually in real-time or near real-time.

[0053] The tagging service 222 can associate a revised map data portion with the data source used to create it, which includes low-resolution current data from sensors 104-106 of the autonomous vehicle 102 and high-resolution pre-drawn data from a previous version of the map portion providing the aforementioned high-resolution map points, and store this information in the map metadata database 224. If a new revision is made to the revised map data portion, the tagging service 222 can save the updated low-resolution current data from sensors 104-106 of the autonomous vehicle 102 to the map metadata database 224, where the new revised map data portion is based on the updated low-resolution current data. The low-resolution current data can be versioned and appropriately associated with the corresponding revised map data portion.

[0054] Figure 3An example visualization of sensor data for detecting the location of a pedestrian crossing according to some aspects of this technology is shown. Road 310, graphically represented as 330, includes one or more lane boundaries, such as dashed lane boundaries 318 and 314, and road boundary 312. When an autonomous vehicle navigates a route including road 310, it can receive current data from one or more sensors that may indicate inconsistencies between pre-drawn data describing the location on a map and current data describing the characteristics of that location. For example, the pre-drawn data includes lane boundaries 314, 318 and road boundary 312 from the last time a dedicated drawing vehicle passed through, but since then road 310 has been redrawn to include pedestrian crossings 316a, 316b (e.g., pedestrian crossings added to the corresponding intersections).

[0055] To detect road 310 and features on it, such as pedestrian crossings, lane boundaries, turning lanes, etc., the autonomous vehicle can perceive its surrounding environment through at least one sensor mounted to or otherwise arranged on the vehicle. For example, the sensor may be part of a sensor system including a lidar sensor and / or a camera sensor. Other exemplary sensor systems may include radar sensor systems, global positioning system (GPS) sensor systems, inertial measurement units (IMUs), infrared sensor systems, laser sensor systems, sonar sensor systems, etc. The lidar sensor can, for example, detect and / or capture intensity variations between the tinted portions of road 310 (e.g., road features such as lane boundaries 314, 318; road boundaries such as 312; and pedestrian crossings such as 316a, 316b) and the untinted portions of road 310. Lighter tinted portions of road 310 will appear as areas with higher brightness than the darker, untinted portions of road 310.

[0056] An intensity map can be constructed using sensors that detect changes in intensity along road 310, and then compared with features on a pre-mapped map. For example, graph 320 shows the intensity of a portion of the intensity map along road 310 as a function of position (x), and graph 322 shows the intensity of a portion of the intensity map along road 310 as a function of position (x) after filtering 324 into a binary map. In some embodiments, graph 320 represents a horizontal cross-section 326h of road 310, showing peaks 328, 340, 342, and 344 with higher intensity corresponding to the parallel line pattern of pedestrian crossing 316a. Lower intensity areas correspond to uncolored portions of road 310. In some embodiments, graph 320 may represent a vertical cross-section 326v of road 310, which would similarly show peaks with higher intensity corresponding to the parallel line pattern of pedestrian crossing 316b. In some embodiments, the horizontal cross-section 326h and the vertical cross-section 326v of road 310 may be combined to form a two-dimensional intensity map with both sets of information.

[0057] In some embodiments, noise signals from the sensors can be removed. For example, in some embodiments, intensity measurements from the lidar can indicate lane boundaries when intensity peaks span at least a threshold distance 346 but not greater than a maximum threshold distance. This is because pedestrian crossing signs should produce higher intensity signals over a specific width between some minimum and some maximum dimensions. Any intensity peaks outside the threshold distance 346 can be filtered out as noise during filtering 324 (because it may indicate lane lines or other features of road 310 that are not pedestrian crossings).

[0058] Additionally and / or alternatively, in some embodiments, the variation in intensity, rather than its width, can be used to filter out noise in the current data. For example, any peaks in the variation less than a threshold of the average non-peak intensity can be removed as noise during filtering 324. Although Figure 3 The described embodiment measures the tinted and untinted areas of road 310 by intensity variation, as this eliminates the need to calibrate sensors to maintain consistency across all autonomous vehicles. However, in some alternative embodiments, the intensity may be measured to certain values. In this way, peaks at or above a certain intensity value at a threshold distance can be detected features, and other data points can be discarded as noise during filtering 324.

[0059] Filter 324 removes colored regions that do not indicate the detection of a pedestrian crossing (e.g., noise). Once filter 324 is performed, the data can be binary-coded so that colored regions have one value (e.g., intensity = 1, or "white" pixels) and uncolored regions have another value (e.g., intensity = 0, or "black" pixels). Graph 322 shows the binary intensity as a function of the positions of peaks 328, 340, 342, and 344.

[0060] Figure 4 A graphical representation of the current point cloud of road segment 404 is shown at 402. The current data describes, for example,... Figure 3 The described region is one where a high-intensity region (e.g., a colored lane boundary) spans a spatial proximity narrower than a threshold maximum width and wider than a threshold minimum width, and the high-intensity region has a higher threshold intensity than a low-intensity region (e.g., bare road). Graphical representation 406 illustrates, according to some aspects of this technique, the current point cloud after data filtering and pedestrian crossing type classification into zebra crossing type. Pixels from grayscale LiDAR data can be filtered into black pixels (e.g., intensity = 0 for uncolored areas) or white pixels (e.g., intensity = 1 for colored areas) (e.g., binary map).

[0061] Based on the spatial proximity of high-intensity regions within a range between minimum and maximum lengths, the current data can be clustered into one or more features. For example, after evaluating white pixels in both horizontal and vertical scans, pixels can be connected when they fall within a specific range. This range could be large enough to distinguish a dashed lane sign, but small enough to distinguish a solid lane boundary. Any group of white pixels with a pixel thickness within a pixel range can be grouped into a line. Any pixels not belonging to a cluster, such as pixel 412, are removed during filtering and correspondingly removed from the graphical representation 406.

[0062] At least two lines (representing the colored lines on the surface of road 404) can be combined to form a group of relevant elements that indicate the characteristics of road 404. These lines can be clustered into features 408 and features 410.

[0063] In some embodiments, a trial-and-error approach may be applied to generate clustering lines for certain structures and / or shapes of features. For example, the shape and structure of high-intensity regions of feature 408 and feature 410 (in different orientations) may indicate a zebra crossing. In some embodiments, connectors may be clustered based on the determination that the shape of a feature is not in the same direction as road 404 (which may indicate a bike lane rather than a pedestrian crossing).

[0064] In some embodiments, when lines are combined into features (e.g., 408 or 410), a secondary axis 412 and a primary axis 414 can be determined. The primary axis 414 can be the maximum diameter of features 408 or 410. The secondary axis 414 can be the shortest diameter of features 408 or 410. In some embodiments, the primary axis 414 and the secondary axis 414 are orthogonal to each other, and the secondary axis can be within 20 degrees parallel to the direction of travel.

[0065] In some embodiments, the grouping of related elements can be categorized based on the attributes of the grouping of related elements. The attributes of the grouping of elements can include the distance between at least two colored lines, and the orientation of the grouping of related elements on the road surface, indicating zebra crossings (near distance) and parallel line crossings (far distance).

[0066] Figure 5 The image shows a graphical representation of the current point cloud of a road segment 504 (image 502). Similar to... Figure 4 The current data describes regions where high-intensity regions (e.g., colored lane boundaries) span spatial proximity that is narrower than a threshold maximum width and wider than a threshold minimum width, and where the high-intensity regions have a higher threshold intensity than low-intensity regions (e.g., bare road). Graphical representation 506 illustrates the current point cloud after data filtering and pedestrian crossing type classification into parallel line type pedestrian crossings, according to some aspects of this technique. Pixels from grayscale LiDAR data can be filtered into black pixels (e.g., intensity = 0 for uncolored areas) or white pixels (e.g., intensity = 1 for colored areas) (e.g., binary map).

[0067] Based on the spatial proximity of high-intensity regions within the range between the minimum and maximum lengths, the current data can be clustered into one or more features. After evaluating white pixels in both horizontal and vertical scans, pixels can be connected when they fall within a specific range. Any group of white pixels with pixel thickness within a pixel range can be grouped into a line. Any pixels that do not belong to a cluster, such as pixel 512, are removed during filtering and accordingly removed in the graphical representation 506.

[0068] At least two lines (representing colored lines on the surface of road 504) can be combined to form a group of relevant elements that indicate the characteristics of road 504. These lines can be clustered into features 508 and features 510, each feature having a primary axis 514 and a secondary axis 512.

[0069] In some embodiments, stop lines may be present in front of and / or on both sides of a pedestrian crossing. In some cases, when a stop line precedes a pedestrian crossing (especially a parallel pedestrian crossing), an autonomous vehicle may have difficulty distinguishing the stop line from the pedestrian crossing feature and may attempt to define the pedestrian crossing feature as also encompassing the stop line. For example, the pedestrian crossing feature might be defined as starting from the stop line, or if the stop line is included in the feature, the pedestrian crossing feature, depending on line density (such as a zebra crossing), might appear to have a lower density. This can lead not only to inaccurate determination of the pedestrian crossing location but also to the potential erroneous conclusion that there is no pedestrian crossing there. In other cases, the stop line can be an indicator that a potential pedestrian crossing is located near the stop line. Therefore, the stop line may be marked as a feature of interest in pre-mapped data or, in some embodiments, detected as a line spanning a minimum and maximum length comparable to the lane width. In some embodiments, lidar and / or camera data can detect the colored word "stop" preceding the stop line.

[0070] In the case of stop lines, only stop lines at approximately a 20-degree angle to the direction orthogonal to the autonomous vehicle's direction of movement are considered. This is because stop lines within this range are road features that the autonomous vehicle will encounter head-on (e.g., stop lines in other directions of travel or parallel to the autonomous vehicle's direction of travel are irrelevant to the autonomous vehicle's navigation and are therefore skipped to save computational resources). Once a stop line is detected or marked, it will not be merged with pedestrian crossing features (such as feature 508 or feature 510), allowing the autonomous vehicle to accurately determine the correct location of the pedestrian crossing. Thus, pedestrian crossing features (e.g., features 508 and 510) are processed separately from stop line features.

[0071] In some embodiments, since the stop line will be parallel to the main axis 514, the detection of the stop line can determine the main axis 514 of the pedestrian crossings 508, 510.

[0072] Figure 6 An example method for detecting pedestrian crossings according to some aspects of the present technology is shown. The method may begin by receiving (602) grayscale pixels from lidar data. Pixels from the grayscale lidar data may be filtered (604) into black or white pixels, such that the data is binary-coded into information indicating colored portions (white pixels) or uncolored portions (black pixels) of the road. White pixels may be evaluated (606) in both horizontal and vertical scans to identify any group of white pixels having a pixel thickness or pixel size within a defined pixel range. For example, the defined pixel range may be larger than dashed lane lines but smaller than stop lines or solid lane lines.

[0073] Based on the above, at least two colored lines on the road surface can be identified (608). Based on the orientation of the first of the at least two lines relative to the second of the identified lines, these lines can be grouped into features that can indicate a pedestrian crossing (e.g., each feature is a grouping of related lines / grouping of related elements). For example, these lines can be grouped into rectangular features. Each diameter of the rectangular feature can then be labeled (610) as a primary axis (maximum diameter) or a secondary axis (minimum diameter). In some embodiments, the primary and secondary axes are orthogonal to each other and can be determined based on statistical variations representing how each feature is shaped, which can approximate a rectangular shape with a minimum diameter (secondary axis) and a maximum diameter (primary axis). The feature can then be classified (612) into a candidate type of pedestrian crossing (e.g., a potential zebra crossing or parallel line type pedestrian crossing). In some embodiments, stop line features are marked and removed to avoid being grouped into features.

[0074] The travel angle relative to the pedestrian crossing feature can be determined (614). If the main axis orientation of the pedestrian crossing feature is parallel to the road or the direction of travel of the autonomous vehicle (AV), then the feature is a bicycle lane and can therefore be ignored or discarded. If the main axis orientation is perpendicular to the road or the direction of travel, then the feature is a pedestrian crossing. Based on the travel angle being perpendicular to the main axis of the pedestrian crossing feature, the pedestrian crossing feature can be classified (616) as a candidate zebra crossing or parallel line pedestrian crossing.

[0075] When a sufficient number of attributes of the grouping of relevant elements constituting candidate zebra crossings in the pedestrian crossing features match the attributes of zebra crossings, the candidate zebra crossing can be identified / confirmed (618) as a pedestrian crossing. When a sufficient number of attributes of the grouping of relevant elements constituting candidate parallel line pedestrian crossings in the pedestrian crossing features match the attributes of zebra crossings, the candidate parallel line pedestrian crossing can be identified / confirmed (620) as a pedestrian crossing.

[0076] As described herein, one aspect of this technology is the collection and use of data available from various sources to improve quality and experience. This disclosure takes into account that, in some cases, such collected data may include personal information. This disclosure anticipates that entities dealing with such personal information will respect and value privacy policies and practices.

[0077] Figure 7An example of a computing system 700 is shown. The computing system 700 can be, for example, any computing device constituting an internal computing system 110, a remote computing system 150, a (potential) passenger device executing a ride-sharing application 170, or any component of the system that communicates with each other using connection 705. Connection 705 can be a physical connection via a bus, or a direct connection to processor 710, such as in a chipset architecture. Connection 705 can also be a virtual connection, a network connection, or a logical connection.

[0078] In some embodiments, the computing system 700 is a distributed system, wherein the functions described herein may be distributed across a data center, multiple data centers, a peer-to-peer network, etc. In some embodiments, one or more of the described system components represent a plurality of such components, each performing some or all of the functions described for that component. In some embodiments, a component may be a physical device or a virtual device.

[0079] Example system 700 includes at least one processing unit (CPU or processor) 710 and a connection 705 that couples various system components, including system memory 715 (such as read-only memory (ROM) 720) and random access memory (RAM) 725, to processor 710. Computing system 700 may include a cache of high-speed memory 712, which is directly connected to, adjacent to, or integrated into processor 710.

[0080] Processor 710 may include any general-purpose processor and hardware or software services, such as services 732, 734, and 736 stored in storage device 730, configured to control processor 710 and dedicated processors, where software instructions are incorporated into the actual processor design. Processor 710 can essentially be a completely independent computing system, containing multiple cores or processors, buses, memory controllers, caches, etc. Multi-core processors can be symmetric or asymmetric.

[0081] To enable user interaction, the computing system 700 includes an input device 745, which can represent any number of input mechanisms, such as a microphone for voice, a touch-sensitive screen for gesture or graphical input, a keyboard, a mouse, motion input, voice input, etc. The computing system 700 may also include an output device 735, which can be one or more of a variety of output mechanisms known to those skilled in the art. In some cases, a multi-mode system allows the user to provide multiple types of input / output to communicate with the computing system 700. The computing system 700 may include a communication interface 740, which typically controls and manages user input and system output. There are no limitations on operation on any particular hardware configuration; therefore, the basic features described herein can be readily replaced by developed, improved hardware or firmware configurations.

[0082] Storage device 730 may be a non-volatile storage device and may be a hard disk or other type of computer-readable medium that can store computer-accessible data, such as magnetic tape cassettes, flash memory cards, solid-state storage devices, digital multifunction disks, magnetic tape cartridges, random access memory (RAM), read-only memory (ROM), and / or some combination of these devices.

[0083] Storage device 730 may include software services, servers, services, etc. When processor 710 executes code defining such software, it enables the system to perform functions. In some embodiments, hardware services performing a particular function may include software components stored in a computer-readable medium and necessary hardware components, such as processor 710, connection 705, output device 735, etc., to perform that function.

[0084] For clarity, in some cases, this technology may be presented as comprising individual functional blocks, including devices, device components, steps or routines in a method embodied in software or a combination of hardware and software.

[0085] Any steps, operations, functions, or processes described herein may be performed or implemented by a combination of hardware and software services, alone or in combination with other devices. In some embodiments, a service may be software residing in the memory of one or more servers of a client device and / or a content management system, and performing one or more functions when a processor executes the software associated with the service. In some embodiments, a service is a program or collection of programs that performs a specific function. In some embodiments, a service may be considered as a server. The memory may be a non-transitory computer-readable medium.

[0086] In some embodiments, computer-readable storage devices, media, and memories may include wired or wireless signals containing bit streams, etc. However, when referred to, non-transitory computer-readable storage media explicitly exclude media such as energy, carrier signals, electromagnetic waves, and the signals themselves.

[0087] The methods described in the examples above can be implemented using computer-executable instructions stored in or obtained from computer-readable media. Such instructions may include, for example, instructions and data that cause or otherwise configure a general-purpose computer, special-purpose computer, or special-purpose processing device to perform a specific function or group of functions. Some of the computer resources used may be available via a network. The executable computer instructions may be, for example, binary, intermediate-format instructions, such as assembly language, firmware, or source code. Examples of computer-readable media that may be used to store instructions, information, and / or information created during the methods according to the examples include hard disks or optical disks, solid-state storage devices, flash memory, USB devices equipped with non-volatile memory, network storage devices, etc.

[0088] Devices implementing the methods according to these disclosures may include hardware, firmware, and / or software, and may take any of a variety of forms. Typical examples of such forms include servers, laptops, smartphones, minicomputers, personal digital assistants, etc. The functionality described herein may also be implemented in peripheral devices or add-in cards. As a further example, such functionality may also be implemented on a circuit board between different chips or different processes executing in a single device.

[0089] The instructions, the medium for transmitting such instructions, the computing resources for executing them, and other structures for supporting such computing resources are means for providing the functions described in these disclosures.

[0090] Although various examples and other information have been used to interpret aspects within the scope of the appended claims, no limitation on the claims should be implied based on specific features or arrangements in these examples, as those skilled in the art will be able to derive a wide variety of implementations from these examples. Furthermore, although some subjects may have been described in example-specific language regarding structural features and / or method steps, it should be understood that the subjects defined in the appended claims are not necessarily limited to these described features or actions. For example, such functionality may be distributed differently or performed in components other than those identified herein. Rather, the described features and steps are disclosed as examples of components of systems and methods within the scope of the appended claims.

Claims

1. A method for detecting pedestrian crossings, the method comprising: Based on sensor data acquired from one or more sensors of the autonomous vehicle (AV), at least two colored lines are determined on the road surface, wherein one or more clusters of the sensor data represent multiple features of the environment along the navigation route. The sensor data is processed through one or more clusters to identify at least one of a plurality of features in the environment, wherein the at least one feature is identified based on the spatial proximity of the one or more clusters; At least two colored lines forming the road surface are generated by grouping the one or more clusters of the sensor data. The at least two colored lines on the road surface are grouped into groups of related elements, the groups including the orientation of the groups of related elements on the road surface and associated with one or more clusters of the sensor data, each of the at least two colored lines containing a minor axis and a major axis; When the spatial proximity of one or more clusters reaches a predetermined threshold, the grouping of the relevant elements is identified as a candidate zebra crossing or a parallel-line crossing; and The candidate zebra crossings or parallel-line crossings are classified as zebra crossings or parallel-line crossings based on the attributes of the grouping of the relevant elements. The attributes of the grouping of the relevant elements include the distance between the at least two colored lines and the orientation of the grouping of the relevant elements relative to the travel angle of the autonomous vehicle on the road surface, the travel angle being perpendicular to the major axis of one of the at least two colored lines. The direction of movement of the autonomous vehicle is controlled during navigation based on the classification of the crossings as candidate zebra crossings or parallel-line crossings.

2. The method of claim 1, wherein determining the at least two colored lines on the road surface further comprises: Filter pixels from grayscale LiDAR data into black or white pixels; as well as The white pixels are evaluated in both horizontal and vertical scans to identify groups of white pixels with a pixel thickness within a defined pixel range, wherein any group of white pixels with the pixel thickness within the defined pixel range is combined into a line to produce the at least two colored lines.

3. The method of claim 1, wherein combining the at least two colored lines on the road surface into the grouping of the related elements further comprises: Determine the main axis of the grouping of the related elements, wherein the main axis includes the grouping of the related elements with a length greater than a threshold number of pixels; as well as Determine the secondary axis of the grouping of the related elements, wherein the secondary axis includes the grouping of the related elements whose length is less than a threshold number of pixels.

4. The method of claim 1, wherein the zebra crossing comprises a closer distance between the groups of related elements compared to the parallel line crossing.

5. The method according to claim 4, further comprising: Before classifying the grouping of the related elements, the travel angle relative to the grouping of the related elements is determined; as well as Based on the travel angle, it is determined that the candidate zebra crossing or the parallel line crossing is still a candidate crossing.

6. The method of claim 4, wherein classifying the grouping of the related elements based on the attributes of the grouping of the related elements further comprises: When a sufficient number of the attributes of the group of the relevant elements constituting the candidate zebra crossing match the attributes of the zebra crossing, the candidate zebra crossing is determined to be a pedestrian crossing.

7. The method of claim 4, wherein classifying the grouping of the related elements based on the attributes of the grouping of the related elements further comprises: When a sufficient number of the attributes of the group of the relevant elements constituting the parallel-line pedestrian crossing match the attributes of the parallel-line pedestrian crossing, the parallel-line pedestrian crossing is determined to be a pedestrian crossing.

8. A system for detecting pedestrian crossings, comprising: One or more processors; and At least one non-transitory computer-readable medium, the at least one non-transitory computer-readable medium comprising instructions stored thereon, wherein the instructions effectively cause the one or more processors to: Based on sensor data acquired from one or more sensors of the autonomous vehicle (AV), at least two colored lines are determined on the road surface, wherein one or more clusters of the sensor data represent multiple features of the environment along the navigation route. The sensor data is processed through one or more clusters to identify at least one of a plurality of features in the environment, wherein the at least one feature is identified based on the spatial proximity of the one or more clusters; At least two colored lines forming the road surface are generated by grouping the one or more clusters of the sensor data. The at least two colored lines on the road surface are grouped into groups of related elements, the groups including the orientation of the groups of related elements on the road surface and associated with one or more clusters of the sensor data, each of the at least two colored lines containing a minor axis and a major axis; When the spatial proximity of one or more clusters reaches a predetermined threshold, the grouping of the relevant elements is identified as a candidate zebra crossing or a parallel-line crossing; and The candidate zebra crossings or parallel-line crossings are classified as zebra crossings or parallel-line crossings based on the attributes of the grouping of the relevant elements. The attributes of the grouping of the relevant elements include the distance between the at least two colored lines and the orientation of the grouping of the relevant elements relative to the travel angle of the autonomous vehicle on the road surface, the travel angle being perpendicular to the major axis of one of the at least two colored lines. The direction of movement of the autonomous vehicle is controlled during navigation based on the classification of the crossings as candidate zebra crossings or parallel-line crossings.

9. The system of claim 8, wherein the instructions effectively cause the one or more processors to: Filter pixels from grayscale LiDAR data into black or white pixels; and The white pixels are evaluated in both horizontal and vertical scans to identify groups of white pixels with a pixel thickness within a defined pixel range, wherein any group of white pixels with the pixel thickness within the defined pixel range is combined into a line to produce the at least two colored lines.

10. The system of claim 8, wherein the instructions effectively cause the one or more processors to: Determine the main axis of the grouping of the related elements, wherein the main axis includes the grouping of the related elements with a length greater than a threshold number of pixels; and Determine the secondary axis of the grouping of the related elements, wherein the secondary axis includes the grouping of the related elements whose length is less than a threshold number of pixels.

11. The system of claim 8, wherein the instructions effectively cause the one or more processors to: Before classifying the grouping of the relevant elements, the grouping of the relevant elements is identified as either the candidate zebra crossing or the parallel line crossing.

12. The system of claim 8, wherein the instructions effectively cause the one or more processors to: Before classifying the grouping of the related elements, the travel angle relative to the grouping of the related elements is determined; and Based on the travel angle, it is determined that the candidate zebra crossing or the parallel line crossing is still a candidate crossing.

13. The system of claim 8, wherein the instructions effectively cause the one or more processors to: When a sufficient number of the attributes of the group of the relevant elements constituting the candidate zebra crossing match the attributes of the zebra crossing, the candidate zebra crossing is determined to be a pedestrian crossing.

14. A non-transitory computer-readable medium comprising instructions stored thereon, the instructions effectively causing one or more processors to perform the method according to any one of claims 1-7.

Images