Method and device for positioning on a road using road surface characteristics

Abstract

The disclosed embodiments provide a technical improvement for providing positioning for a transport vehicle by detecting a road wear reference line in a road on which the transport vehicle is traveling, and controlling, guiding or otherwise facilitating alignment of the transport vehicle wheel center with the detected road wear center.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] none. Technical Field

[0003] The present disclosure relates to methods, operations and apparatus for on-road localization of a transportation vehicle. Summary of the invention

[0004] According to the present disclosure, systems, assemblies, and methods are provided for improving positioning of a transport vehicle by detecting a road wear reference line in a road on which the transport vehicle is traveling.

[0005] In accordance with the disclosed embodiments, systems, components and methods enable control, guidance, suggestion or otherwise facilitate alignment of a transport vehicle wheel center with a detected road wear center to facilitate improved navigation of the transport vehicle with respect to a road, and more specifically with respect to traffic lanes and transitions within those traffic lanes due to road characteristics, including but not limited to merge areas, on / off ramps, roundabouts and other road junction areas.

[0006] According to at least one disclosed embodiment, road wear reference line data is used in conjunction with data detected or accessed by on-vehicle equipment to identify lane markings to control, guide, advise, or otherwise facilitate alignment of a transport vehicle wheel center with a detected road wear center to facilitate improved navigation of the transport vehicle with respect to a road. Such reference and analysis of multiple data sets generated by multiple on-vehicle data sensors and / or accessed in one or more databases provides additional redundancy with respect to the monitored data and, therefore, provides increased safety with respect to an on-vehicle system that enables control, guidance, advice for navigating a transport vehicle on a road.

[0007] According to at least one disclosed embodiment, the road wear reference line data can also be used to assist in clarifying or analyzing data detected or accessed by an on-vehicle device to identify lane markings, for example when lane markings are not visible or confusing. Such reference and analysis of the road wear reference line data improves safety by augmenting lane marking data generated by one or more on-vehicle data sensors or accessed in one or more databases.

[0008] Additional features of the present invention will become apparent to those skilled in the art upon consideration of the illustrative embodiments illustrating what is presently perceived as the best mode of carrying out the invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] The detailed description makes particular reference to the accompanying drawings, in which:

[0010] Figures 1A-1D Examples of characteristics of the road surface on the road, such as road wear or wheel rutting, are illustrated.

[0011] Figure 2 An example of a system and components for high-resolution measurement of road surface texture at a high frame rate using a frequency modulated continuous wave (FMCW) laser radar (LiDAR) sensor in accordance with the disclosed embodiments is illustrated.

[0012] Figure 3 Operations according to the disclosed embodiments are illustrated for performing a machine learning approach for data analysis to convert high-dimensional sensor data generated by an FMCW lidar sensor into characteristic low-dimensional features that enable discrimination between smooth and rough surface textures for the purpose of performing on-road positioning based on simple and rapidly processable discrimination criteria.

[0013] Figure 4 The operations of performing deep neural network operations to perform feature extraction and classification based on low-dimensional characteristic features are illustrated in additional detail.

[0014] Figure 5 An example of a surface profile in a road area including road wear / wheel rutting is illustrated.

[0015] Figure 6 An example of an exit scenario is illustrated where the distance between lane markings is wider than the typical lane width, which creates confusion.

[0016] Figure 7 Illustrated is an example of a line of treatment chemicals applied to a road in preparation for a freeze or frozen precipitation.

[0017] Figure 8 Another example of a line of treatment chemicals being applied to a portion of a road including a turning lane in preparation for a freeze or freezing precipitation is illustrated.

[0018] Fig. 9Illustrated is an example of a portion of a road that includes a deposit of material that was initially applied to the road in preparation for a freeze or freezing precipitation, but the material has worn away the road and is deposited in intersection areas of the road where transport vehicles are not driving on it. DETAILED DESCRIPTION

[0019] The accompanying drawings and descriptions provided herein may have been simplified to illustrate the aspects related to the equipment, system and method described herein being clearly understood, while for the purpose of clarity, eliminating other aspects that may be found in typical equipment, system and method. It will be appreciated by those of ordinary skill in the art that in order to realize the equipment, system and method described herein, other elements and / or operations may be desirable and / or necessary. Because such elements and operations are well known in the art and because they are not convenient for a better understanding of the present disclosure, the discussion of such elements and operations may not be provided herein. However, the present disclosure is considered to inherently include all such elements, variations and all modifications to the described aspects that those of ordinary skill in the art will know.

[0020] The disclosed embodiments provide a technical solution for improving positioning of a transport vehicle by detecting a road wear reference line in a road on which the transport vehicle is traveling, and controlling, guiding or otherwise facilitating alignment of the transport vehicle wheel center with the detected road wear center.

[0021] For purposes of this disclosure, the term "road" includes any road, thoroughfare, route, or land route between two places that has been paved or otherwise improved to allow travel by a vehicle, including, but not limited to, a motor vehicle or other vehicle including one or more wheels. It should be understood, therefore, that such a road may include one or more lanes and junctions with other roads, including on / off ramps, merging areas, etc., which may be included in a parkway, boulevard, freeway, toll road, interstate, highway, or primary, secondary, or tertiary local road.

[0022] For purposes of this disclosure, the term "on-road positioning" is used to refer to the ability to determine the position of a transport vehicle relative to a road or portion of a road (such as a lane) on which the transport vehicle is traveling.

[0023] As autonomous and driver assistance related transportation vehicle technologies are further incorporated, it is contemplated that, in implementation, autonomous and / or assistance functions will rely at least partially and possibly entirely on-road positioning, which is performed in an automated or semi-automated manner based on global positioning service (GPS) data and data generated by multiple sensors located on the vehicle. As explained below, various conventional methods are known for performing on-road positioning using various different types of sensors; however, each of these methods has deficiencies in the quality / quantity of data to be processed and / or the time and computational load for processing, which is an obstacle to real-world real-time use to safely control the movement of a transportation vehicle.

[0024] For the purposes of this disclosure, the phrase "autonomous and / or auxiliary functions" refers to functions that enable partial, full or complete automation of vehicle control ranging, and covers what are currently known as the five levels of driving automation. Therefore, it should be understood that autonomous and / or auxiliary functions refer to operations performed in an automated manner by the vehicle through on-vehicle equipment, or to outputs of alerts, prompts, recommendations or guidance to the user, where these outputs are generated in an automated manner by on-vehicle equipment. In addition, autonomous and / or auxiliary functions may include driver assistance functions (level one), in which on-vehicle equipment assists but does not control steering, braking and / or acceleration, but the driver ultimately controls acceleration, braking, and monitoring of the vehicle's surroundings.

[0025] It should therefore be understood that such autonomous and / or assistive functionality may also include a lane departure warning system that provides a mechanism for alerting the driver when the transport vehicle begins to move out of its lane on highways and major roads (unless the turn signal is on in that direction). Such systems may include systems (visual, audible and / or vibration warnings) that alert the driver if the vehicle leaves its lane (lane departure warning), and systems that alert the driver and automatically take steps to ensure that the vehicle remains in its lane if no action is taken (lane keeping system).

[0026] Likewise, autonomous and / or assistive functions may include partial automation (Level 2), where the transport vehicle assists with steering or acceleration functions and accordingly monitors the vehicle's surroundings to enable the driver to be relieved of some of the tasks of driving the transport vehicle. As understood in the automotive industry, partial automation still requires the driver to be ready to assume all tasks of transport vehicle operation and also requires continuous monitoring of the vehicle's surroundings at all times.

[0027] Autonomous and / or assistive functions may include conditional automation (Level Three), in which the transport vehicle equipment is responsible for monitoring the vehicle's surroundings and controlling the vehicle's handling, braking, and acceleration without driver intervention. It should be understood that at this level and above, the on-vehicle equipment used to perform autonomous and / or assistive functions will be interfaced with or include navigation functions so that the components have data for determining where the vehicle will travel. At Levels Three and above, the driver is theoretically allowed to disengage from monitoring the vehicle's surroundings, but may be prompted to take over control of the transport vehicle's operation in certain circumstances that may impede safe operation in conditional automation mode.

[0028] Thus, it should be understood that autonomous and / or assisted functionality may include systems that take over steering and keep the transport vehicle centered in a lane of traffic.

[0029] Likewise, autonomous and / or assistive functionality may include high automation (Level Four) and full automation (Level Five), where on-vehicle equipment enables automated steering, braking, and acceleration in an automated manner in response to monitoring of the vehicle's surroundings without driver intervention.

[0030] It should be understood, therefore, that autonomous and / or assisted functionality may require monitoring of the vehicle's surroundings - which includes the vehicle's roadway and identification of objects in the surroundings - in order to enable safe operation of the vehicle in response to traffic events and navigation directions, where such safe operation requires determining when to change lanes, when to change direction, when to change roads (exit / entry roads), when and in what order to merge or traverse road junctions, and when to use turn signals and other navigation indicators to ensure that other vehicles / vehicle drivers are aware of upcoming vehicle maneuvers.

[0031] Furthermore, it should be understood that high and full automation may include analyzing and considering data provided from sources outside the vehicle in order to determine whether such level of automation is safe. For example, autonomous and / or assistive functions at such levels may involve determining the likelihood that pedestrians are in the environment surrounding the transportation vehicle, which may involve referencing data indicating whether the current road is a highway or a parkway. Additionally, autonomous and / or assistive functions at such levels may involve accessing data indicating whether there is a traffic jam on the current road.

[0032] With this understanding of potential implementations of on-vehicle sensors for autonomous and / or assistive functionality in mind, presently disclosed systems, components, and methods may be provided to implement such functionality by improving positioning of a transport vehicle by detecting road wear reference lines in the road on which the transport vehicle is traveling.

[0033] Conventional transportation vehicle navigation systems and conventional autonomous vehicles use GPS technology to perform their on-road positioning. However, the drawback of conventional use of GPS for global positioning is that positioning is limited to a certain level of accuracy, more specifically, 5-10 meters in the best case (this generally requires a geographic area that provides an unobstructed view of the sky). In addition, lower accuracy is more likely to exist in geographic areas that include relatively large buildings, trees, or geographic contours such as canyons. This is because GPS-based positioning services require signals from GPS satellites. Dense materials (e.g., rocks, steel, etc.), tall buildings, and large geographic terrain features may block or degrade GPS signals.

[0034] Therefore, traditionally, GPS has been combined with local landmarks for positioning on the road (e.g., lane markings) to improve the ability of vehicles with autonomous and / or auxiliary vehicle functions to accurately perform positioning on the road. Traditionally, these local landmarks have been detected and identified from camera images obtained from one or more cameras / sensors located on the vehicle or sensor data from other sensors. For example, it has been discussed traditionally to combine GPS data with data collected from forward-looking cameras and lidars, and even with data generated by ground-penetrating radars. In addition, it has been discussed that for such cameras, there is the following utility: namely, extracting a simplified feature presentation of road characteristics from the camera on the vehicle to generate data indicating a roadside pattern that can be analyzed to perform positioning on the road. Additionally, traditionally, high-speed longitudinal road profiling using discrete point measurements of lasers has been combined with CMOS line scan camera technology for autonomous and / or auxiliary functions of transport vehicles.

[0035] However, on-road positioning based on lane markings, with or without GPS, is still insufficient in many situations, because, for example, conventionally available techniques are unable to distinguish between the actual driving lane and, for example, an upcoming concrete lane divider, since both the driving lane and the concrete lane divider are bounded by left and right lane markings.

[0036] The disclosed embodiments are based on the recognition that recent autonomous vehicle traffic accidents provide evidence of the existence of a technical and real-world need to increase the robustness and / or reliability of data used for autonomous and / or auxiliary vehicle equipment and systems by using a combination of data generated by multiple different sensors and different types of sensors to ensure that if data from one sensor or one type of sensor becomes unavailable or unreliable, data from another sensor or another sensor type can be utilized.

[0037] Thus, in accordance with the disclosed embodiments, systems, components and methods enable control, guidance, suggestion or otherwise facilitate alignment of a transport vehicle wheel center with a detected road wear center to facilitate improved navigation of the transport vehicle with respect to a road, and more specifically with respect to traffic lanes and transitions within those traffic lanes due to road characteristics, including but not limited to merge areas, on / off ramps, roundabouts and other road junction areas.

[0038] It is noteworthy that road wear such as wheel rutting is less visually noticeable than lane markings traditionally used for positioning on roads. Road rutting is the depression, groove or other removal of material on the road surface that is worn away from the road due to the travel of the wheels of the transport vehicle. Rutting may be formed due to wear (e.g., from the studded snow tires common in cold climate regions), or they may be formed by the deformation of asphalt concrete pavement or subbase materials. Regardless of the physical mechanism that forms the rutting, the rutting itself indicates the past, repeated (and theoretically safe) traversal of the transport vehicle on the road on the path that includes the rutting. Therefore, by aligning the current operation of the transport vehicle with these rutting, the possibility of increasing the safety of such operation is at least theoretically increased.

[0039] like Figures 1A-1D As shown in FIG. 1 , characteristics of the road surface on road 110, such as road wear or wheel ruts 120, indicate repeated traversal of the road by transport vehicles in the past. Thus, it should be appreciated that wheel ruts 120 extend generally parallel to lane markings representing traffic lanes 130. Except when there are multiple paths for traversing traffic lanes 130 or areas of road 110 (as referred to herein), the vehicle may be traversed by a vehicle in the past. Figure 6 ), this is usually correct.

[0040] Therefore, sensing and analyzing the location of such wheel-road surface features can improve the ability to locate on the road while performing autonomous and / or assisted driving functions provided by the on-vehicle device. It is worth noting that when precipitation causes water (which can reflect light) to collect in the ruts 120 (e.g. Figure 1C), and / or when precipitation causes blurred lane markings on road 110 (such as Figure 1D This may be especially true for

[0041] Implementation of the above-described conventionally known on-road positioning techniques has been limited due to both the quality and quantity of the sensory data analyzed. The quality of the data can affect the accuracy of determinations made based on the analysis of the data. The quantity of the data can affect the time required to analyze the data, thereby making implementation of autonomous and / or assistive functions difficult or impossible, depending on which conventional techniques are implemented.

[0042] In contrast, the disclosed embodiments take advantage of recently developed LiDAR sensors with advanced frequency modulated continuous wave (FMCW) capabilities that have recently become available in the automotive industry.

[0043] Traditional lidar measures the distance to one or more targets by illuminating the target with a pulsed laser and measuring the reflected pulses with a sensor. The difference in the laser's return time and wavelength can then be used to make a digital 3D representation of the target.

[0044] Recently developed LiDAR components, such as those now commercially available from Blackmore Systems (Bozeman, Montana), combine coherent optical communications hardware with radar signal processing techniques to provide FMCW LiDARs that provide improved sensor sensitivity, robustness, and speed measurements. Such FMCW LiDAR devices are therefore able to achieve increased functionality in sensing or "seeing" both bright and dark objects included in the transportation vehicle environment. In particular, for the purposes of the presently disclosed embodiments, dark objects, such as road surfaces, and ruts in those road surfaces, may be detectable. For example, intensity data (dark / bright) may be an indicator of rut location, but the actual number of returns and how the LiDAR returns are distributed may also be analyzed to determine rut location. In theory, speckle information, as described below, may provide insight into the characteristics of where the roughness of the road surface varies, which in turn may be an additional indicator for determining lane center. The key factor for this principle to work is the integration time of the light pulses. If the time it takes for a given ray to reach the target is short enough, the speckle pattern may affect the point cloud in such a way that it provides the information needed to detect road markings or characteristic road wear.

[0045] This capability is not available using traditional direct time-of-flight (dToF) because those sensors do not show any speckle artifacts. Instead, lidars that implement dToF determine distance based on the known speed of light, measuring the time of flight of discrete light signals between the lidar and the subject at each scan point.

[0046] In contrast, FMCW lidar provides coherent lidar using a continuously emitting laser that changes its frequency through a range, and compares the frequency of the return signal with a corresponding local reference signal to determine the distance. In addition, FMCW lidar and Blackmore's lidar components can determine the relative speed of objects, which helps in terms of speed and ability to identify the shape of environmental objects based on the moving transport vehicle.

[0047] like Figure 2 As shown in , the disclosed embodiments provide a system 200 and components for high-resolution measurement of road surface texture 210 that can be performed at a high frame rate using a frequency modulated continuous wave (FMCW) lidar sensor 220 mounted on a transport vehicle 230. It should be appreciated that software for performing functions that implement data measurement and analysis can be executed at least in part using software code stored on one or more non-transitory computer-readable media that runs on one or more processors (not shown) in the transport vehicle 230.

[0048] The system 200 can operate in the invisible spectrum (1550 nm), for example, with a field of view of 30 (vertical) x 40 (horizontal) and an accuracy in the range of <5 cm, with a throughput in the range of, for example, 300,000 samples / second to 1 million samples / second. Thus, each sample from the FMCW lidar sensor 220 can provide information about the Euclidean metric coordinates (e.g., x, y, z in sensor coordinates) of a given point, the intensity of the surface, and the radial velocity. In addition, within one sample, multiple returns can be obtained for each discrete azimuth and elevation angle.

[0049] In operation, the FMCW laser radar sensor 220 generates a continuous laser, compared to a conventional pulsed laser radar that sends a train of laser pulses. The continuously emitted light in the FMCW laser radar is split into two beams, one beam is sent out to hit the target, and its reflection is combined with the second beam and then evaluated for actual distance and radial velocity measurements. The target light pulse is not completely collimated, and there is some beam divergence. The backscattered light may interfere constructively or destructively, depending on the roughness of the surface. This effect is called speckle in the field of optics. It should be appreciated that no return is provided in the case where the FMCW laser radar 220 performs an invalid measurement (e.g., due to a surface that does not have a sensor range, or an invalid measurement due to speckle).

[0050] It should be understood that the FMCW laser radar sensor 220 Figure 2 2 is shown merely as an example of the positioning of the sensor. Thus, the sensor 220 may be located elsewhere on or in the vehicle, including, for example, in the front grill assembly of the vehicle or otherwise facing forward, in order to implement functionality for sensing and analyzing optical data generated by the sensor 220. However, it should be understood that for at least some implementations, the sensor 220 should be located higher on the vehicle (e.g., on the roof) to provide the most useful data for tracking road position.

[0051] According to the disclosed embodiments, this data (including information about the number of returns per sample combined with the full point cloud data) can be used to generate a characteristic road surface profile. Such a characteristic profile can provide sufficient data (quantity and quality) to obtain accurate lateral position data in order to determine on-road positioning and / or to assist other active and passive measurement sensors (e.g., odometers, cameras) located on the vehicle 230. Therefore, analysis of the extracted lidar signal can optionally be used alone or in combination with other sensor data for more robust absolute on-road positioning.

[0052] Figure 3-4 1 illustrates an example data analysis performed in accordance with the disclosed embodiments. Figure 3 As shown in , operation starts at 305, and control proceeds to 310, where generated data from a FMCW lidar sensor is received. Control then proceeds to 315, where lidar point cloud data is generated, including x, y, and z coordinates, intensity data, and radial velocity data, as well as speckle characteristics for each sample of the generated data received at 310. It should be understood that speckle characteristics are implicit information and can only be obtained directly; therefore, depending on the road surface roughness, the sensor may receive more or less actual returns.

[0053] Control then proceeds to 320 where a deep neural network operation is performed to perform feature extraction and classification based on the low-dimensional feature features 320. This will be referred to herein as Figure 4 Further discussion.

[0054] Control then proceeds to 325 where the data is analyzed to identify data indicative of the presence of road rut(s) in a format that can be used by vehicle components to determine on-road positioning (e.g., for providing autonomous and / or auxiliary functions). Control then proceeds to 330 where the data is output to the on-vehicle autonomous and / or auxiliary function components. Control then proceeds to 335 where the operation ends.

[0055] like Figure 4 As shown in FIG. 3 , a deep neural network operation is performed to perform feature extraction and classification based on low-dimensional feature features 320 (see FIG. Figure 3 ) may begin at 405, where control then proceeds to 410, where lidar point cloud data is received. This may include x, y, z coordinates, intensity data, radial velocity data, and implicit speckle characteristics for each sample included in the received lidar data. Control may then proceed to 415, where low-dimensional characteristic features may be extracted. Control may then proceed to 420, where a classification process is performed on those extracted features, and control may proceed to 425, where the deep neural network operation for feature extraction and classification ends.

[0056] It should be understood that if Figure 4 As shown in , feature extraction and classification can also be performed in a single component (e.g., a single convolutional neural network), as shown in Figure 3 Therefore, without departing from the concept of the present invention, the disclosed embodiments can be implemented in a single component or in multiple components.

[0057] Figure 3-4 The operations performed as depicted in make it possible to distinguish surface profiles indicative of road wear / wheel rutting from other parts of the road. Figure 5 As shown in , the surface profile in the road wear / wheel rutting 120 area (between the dashed lines) corresponds to the area when the transport vehicle has worn the road surface significantly compared to the remaining road surface 130 (e.g., between the road wear and in the middle of the road between the road wear and lane markings).

[0058] Based on a combination of the full point cloud data provided by the sensor (x, y, z coordinates, reflectivity, and radial velocity for each return) and the absolute number of returns generated in that data, when a lidar scan is performed at a point along the horizontal line, an operation can be performed to identify a road wear reference line 140. More specifically, the transition between the rougher surface 130 and the smoother surface 120 is indicated by the dashed lines, from which the corresponding center of the rut 140 can be determined, as indicated by the solid line 140 located in the middle of the dashed lines.

[0059] Thus, road wear can be identified and used to assist or automate the alignment of the transport vehicle wheel center on the road wear center. In this regard, it should be noted that road wear / rutting may typically be wider than the tire width of a typical car because tire width varies from vehicle to vehicle and due to different types of vehicles (e.g., trucks vs. cars), and there is some degree of variation from driver to driver in driving preferences regarding where to position the vehicle in the lane of travel. However, even taking into account such variation, road wear is an indicator of past behavior of most drivers. Therefore, performing on-road positioning based at least in part on such data increases the ability to perform on-road positioning for the purpose of performing autonomous and / or auxiliary functions in a transport vehicle.

[0060] It should be understood that by utilizing a deep neural network (e.g., Figure 3 and 4 ) to distinguish these different surface types, the deep neural network is used to perform machine learning methods for data analysis, thereby converting high-dimensional sensor data into characteristic low-dimensional features, which are used as simple and quickly processable differentiation criteria. For example, computer vision and machine learning with speckle pattern analysis can be performed, for example, using one or more of a variety of deep neural network operations, including those implemented using convolutional neural networks (CNNs), recurrent neural networks (RNNs), convolutional autoencoders, generative adversarial networks (GANs), graph neural networks (GNNs), Siamese / Triplet networks, etc.

[0061] For example, analysis of lidar point cloud data may be performed using various types of machine learning operations, including supervised learning, unsupervised learning, deep learning, and what may be referred to as supervised deep learning.

[0062] In accordance with at least one embodiment, unsupervised learning may be performed to automatically identify thresholds on feature values ​​that distinguish smooth from rough regions, for example using an unsupervised clustering approach, such as with the convolutional autoencoders described herein.

[0063] In accordance with at least one embodiment, unsupervised classification (e.g., one-class support vector machines, k-means clustering, density-based clustering such as DBSCAN, distribution-based clustering such as expectation-maximization, subspace clustering) can be performed as a combination of clustering and dimensionality reduction (e.g., principal component analysis / singular value decomposition).

[0064] Optionally, in addition, supervised learning, which is the most common form of machine learning, involves enabling learning during a training phase based on a training data set so as to enable the ability to learn to identify how to label input data for classification. Deep learning improves on supervised learning methods by considering several levels of representation, each of which uses information from previous levels to learn more deeply. Deeper architectures with many stacked layers are one aspect, and CNNs also consider 2D / 3D local neighborhood relationships in pixel / voxel space through convolutions on spatial filters.

[0065] Thus, supervised deep learning involves applying multiple layers or stages of functional operations to improve the understanding of the resulting data, which is then fed into further functional operations. For example, supervised deep learning for classifying data into one or more categories can be performed, for example, by performing feature learning (involving one or more stages of convolution, rectified linear units (ReLU), and pooling) to enable subsequent classification of sample data, thereby identifying learned features by applying a softmax function to enable differentiation between background and objects in the input image data. These operations can be performed to generate image class labels for classification purposes.

[0066] Likewise, supervised deep learning operations can be performed for regression by operating in parallel on the RGB image, and the distance to ground / disparity map data by performing multiple convolutions and concatenating the results together for subsequent processing. These operations can be performed to generate image regression labels for subsequent use in analysis.

[0067] In addition, supervised deep learning operations can be performed for semantic segmentation by inputting the RGB image data into a convolutional encoder / decoder, which can include multiple stages: convolution, batch normalization (which is not only applicable to segmentation, but also to other networks), ReLUS, and pooling, followed by multiple stages: convolution, batch normalization, and ReLUS with upsampling. The resulting data can then be processed by applying a softmax function to provide output data with segmentation labels for each pixel.

[0068] As mentioned above, analysis of LiDAR point cloud data can also be performed by a convolutional autoencoder using unsupervised learning, where the training data set is not associated with class labels, so that data is generated and organized, and the system develops and organizes the data to search for common characteristics, such as in clusters. According to the disclosed embodiments, unsupervised deep learning operations can be performed, such as for feature extraction in image data. Thus, an input data frame can be input into multiple convolution and pooling layers in the encoder, followed by processing by multiple deconvolution and pooling layers. The result of this feature extraction operation enables the ability to effectively detect anomalies in image data or identify multiple categories in the data (such as rough and smooth surfaces).

[0069] Additionally, GANs can be used to perform so-called "semi-supervised learning" for data augmentation and domain adaptation. In semi-supervised learning performed by GANs, random noise can be input into an image generator to generate a pseudo image, which is then compared to a data training set to enable a discriminator to distinguish between training data (i.e., real data) and pseudo data. Therefore, it should be understood that this operation also generates image data with some labels.

[0070] The twin / triplet networks can similarly be used for similarity learning, enabling the machine learning component to identify similarities between datasets, as well as different categories within a dataset.

[0071] Furthermore, RNNs can be used to analyze time-dependent behavior in input / output relationship data by, for example, modeling or describing the behavior, and potentially utilizing trained RNNs to predict time series.

[0072] According to at least one disclosed embodiment, the road wear reference line data may also be used to assist in clarifying or analyzing data detected or accessed by an on-vehicle device to identify lane markings, for example when lane markings are not visible or confusing. Such reference and analysis of the road wear reference line data improves safety by augmenting the lane marking data generated by one or more on-vehicle data sensors or accessed in one or more databases. Therefore, it should be understood that while the disclosed embodiments may be generally used for the purpose of facilitating robust autonomous and / or auxiliary transport vehicle functions, the disclosed embodiments may have particular utility in providing such functions when the lane width at a road junction (such as an on-ramp or off-ramp of a highway) is temporarily increased. Such road areas may cause temporary confusion for human drivers because they are unable to determine when and where to change direction to commit to a path in the junction. Therefore, it can be assumed that autonomous and / or auxiliary functions may similarly find such determination difficult.

[0073] For example, Figure 6 An exit scenario is shown where the distance between lane markings is wider than a typical lane width. It should therefore be appreciated that, in accordance with the disclosed embodiments, a driving path prediction algorithm based on rutting as well as lane markings may be used to perform autonomous and / or assistive functions, as considering only lane markings may result in a transport vehicle being positioned in the middle of an increased lane width, which may be unsafe. Therefore, detecting and analyzing rutting in roads in such areas as part of autonomous and / or assistive functions may provide improved safety. This may be particularly true if the road includes changes in elevation that may result in an increased risk of loss of control of the vehicle (due to, for example, loss of gravel, ice, water, etc.).

[0074] Furthermore, it should be appreciated that while the disclosed embodiments may be generally useful for the purpose of facilitating robust autonomous and / or assisted transport vehicle functionality, the disclosed embodiments may have particular utility in providing such functionality when lane markings are obscured by weather conditions such as snow. Under such conditions, lane markings and roadside patterns that are traditionally used to assist with on-road positioning may become obscured. For example, any amount of snow may obscure lane markings and road signs; additionally, large amounts of snow may change the appearance of the surroundings along the road such that the roadside patterns cannot be analyzed to provide additional data to be combined with GPS analysis for performing on-road positioning.

[0075] According to at least one disclosed embodiment, the road wear reference line data can be used in conjunction with data detected or accessed by on-vehicle equipment to identify lane markings to control, guide, advise, or otherwise facilitate alignment of a transport vehicle wheel center (e.g., the lateral center of the wheel surface in contact with the road) with a detected road wear center to facilitate improved navigation of the transport vehicle with respect to the road. Such reference and analysis of multiple data sets generated by multiple on-vehicle data sensors or accessed in one or more databases provides additional redundancy with respect to the monitored data and, therefore, provides increased safety with respect to an on-vehicle system that enables control, guidance, advice for navigating a transport vehicle on a road.

[0076] Therefore, it should also be understood that according to the disclosed embodiments, the robustness of on-road positioning operations can be increased by combining data generated by different sensors, including FMCW lidar, to improve reliability, thereby accounting for the possibility that data generated by one or more sensors becomes unavailable or unreliable during autonomous and / or auxiliary functions. For example, this may occur if lane markings are not visible or confusing (e.g., lane splits / merges are not continuous), or GPS-based positioning becomes impossible due to tunnels or tall buildings blocking access and reference to satellites.

[0077] In this regard, it should be understood that at least one embodiment may include a feedback mechanism that determines the quantity and / or quality of data generated and / or analyzed in the disclosed operations. Such a feedback mechanism may be used to selectively increase or decrease reliance on rut-based road positioning data when providing autonomous and / or auxiliary functions. This may be achieved, for example, by dynamically weighting the data generated by different positioning technologies / components based on a determination of the relative accuracy or availability of the data generated by these positioning technologies / components. It should be understood that data weighting may be based on weather, the amount of ambient light, known road conditions stored in a database for cross-reference with GPS data determined for a transport vehicle. It should also be understood that such a feedback mechanism may include comparison with a threshold value for maintaining at least a minimum parameter to ensure the safety of autonomous and / or auxiliary function operation.

[0078] Furthermore, it should be understood that mechanisms for dynamically weighting such data may be implemented in one or more of a variety of conventionally known techniques that enable sensor data fusion using, for example, Kalman filters, processing based on the central limit theorem, Bayesian networks, the Dempster-Shafer theorem, CNNs, or any other mathematical operation disclosed herein.

[0079] While the above description has made reference to rut-based positioning data, it will be appreciated that road conditions may be such that other data may be similarly used to improve autonomous and / or assisted transport vehicle functionality. Figure 7-9 Various examples of road conditions that occur as a result of, or in preparation for, freezing or freezing precipitation are illustrated. For example, Figure 7 An example of a line of treatment chemicals (e.g., magnesium chloride or calcium chloride) applied to a road in preparation for a freeze or freezing precipitation is shown. Figure 7 As shown in , the application of these chemicals typically forms lines that extend parallel to the road lane markings because the vehicle applying the chemicals must obey traffic rules, including staying within the lane of travel. Therefore, it should be appreciated that the use of FMCW lidar to identify such lines can enable further analysis to determine how best to stay within the lane of traffic. Therefore, it should be appreciated that the sensing of multiple lines of road treatment materials can enable improved ability to center a transport vehicle within a lane of traffic.

[0080] Similarly, Figure 8 Another example of a line of treatment chemicals being applied to a portion of a road including a turning lane in preparation for a freeze or freezing precipitation is illustrated. Figure 8 As shown in , multiple lines of road treatment material further distinguish the appropriate path that branches from the main road through the exit area of ​​the road into the turning lane, which is represented by a smaller dashed line in the lane marking than the dashed line present on the main road. Therefore, sensing and analyzing such lines can assist in providing autonomous and / or auxiliary transport vehicle functions.

[0081] Additionally, it should be appreciated that FMCW LiDAR can be used to identify areas of a roadway that include material deposits that were initially applied to the roadway in preparation for a freeze or freezing precipitation. Fig. 9 An example of a portion of a road is illustrated that includes a material deposit (e.g., magnesium chloride or calcium chloride) that was initially applied to the road in preparation for a freeze or freezing precipitation, but the material has worn away the road and is deposited in intersection areas of the road where a transport vehicle is not driving on it. Thus, sensing and analyzing such deposits can assist in providing autonomous and / or assistive transport vehicle functionality.

[0082] Furthermore, it should be appreciated that sensing, monitoring and analyzing such additional road characteristics may improve the accuracy, robustness and efficiency of autonomous and / or auxiliary functional devices with respect to providing such functionality in environmental conditions that reduce the ability to provide such functionality due to precipitation and / or lighting.

[0083] As explained above, the disclosed embodiments can be implemented in conjunction with components of autonomous and / or assisted driving systems included in a transportation vehicle. Thus, the utility of the disclosed embodiments within these technical contexts has been described in detail. However, the scope of the innovative concepts disclosed herein is not limited to these technical contexts.

[0084] Additionally, it should be understood that the presently disclosed apparatus for analyzing image data depicting a road on which a transport vehicle is traveling may include any combination of sensors and functionality disclosed herein implemented in hardware and / or software to provide the disclosed functionality.

[0085] Likewise, it should be understood that the presently disclosed apparatus for facilitating alignment of the center of at least one wheel surface of a wheel of a transport vehicle with a detected center of a road wear reference line will be implemented at least in part using autonomous and / or assisted driving functionality (and corresponding technical systems) to provide that functionality at different levels (described herein), depending on the degree of control to be provided for the transport vehicle.

[0086] Furthermore, it should be understood that such assistive technologies may include, but are not limited to, technologies that have traditionally been referred to as driver assistance systems (DAS) or advanced driver assistance systems (ADAS) implemented using hardware and software included in the transportation vehicle. These conventionally known systems assist the driver in making decisions and controlling, but decisions and controlling are inevitably the responsibility of the driver. Furthermore, these systems may be "active" or "passive" in terms of how they are implemented. Active DAS means that the vehicle itself controls various longitudinal and / or lateral aspects of the vehicle's driving behavior through its sensors, algorithms, processing systems, and actuators, or more precisely, controls very specific driving tasks. Passive DAS means that the vehicle will simply assist the driver in controlling various longitudinal and / or lateral aspects of vehicle control through its sensors, algorithms, processing systems, and human-machine interfaces (HMIs). For example, in a collision avoidance situation, an active system will stop the vehicle, or route the vehicle around obstacles in a direct path. A passive system will provide the driver with some type of visual, auditory, and tactile prompts that are used to stop the vehicle or route the vehicle around obstacles.

[0087] Thus, DAS systems assist drivers with many tasks that are ingrained into the driving process and are implemented specifically for the purpose of increasing automobile and road safety and driver convenience. Such DAS systems include, but are not limited to, cruise control, adaptive cruise control (ACC), active steering for lane keeping, lane change assistance, highway merge assistance, collision mitigation and avoidance systems, pedestrian protection systems, automated and / or assisted parking, sign recognition, blind spot detection for collision mitigation, and stop and go traffic assistance. Thus, the disclosed embodiments provide such DAS systems with additional and potentially more accurate data to provide such assistive functionality.

[0088] It should be further appreciated that the disclosed embodiments leverage functionality from a number of different technical fields to provide additional mechanisms and methods for performing on-road positioning to facilitate autonomous and / or assisted driving functions by combining analysis performed in computer vision and machine learning with innovations in speckle pattern analysis in materials science.

[0089] Furthermore, it should be understood that if speckle is present, the point cloud will be missing points in various areas. Taking the entire information of the point cloud into account, speckle may introduce characteristic patterns that give insights into the roughness of the road. It is expected that these cues caused by speckle can only be correctly interpreted by machine learning methods. Furthermore, the wavelength used by the laser system and the angle of incidence between a given lidar ray and the surface will affect the speckle and measurement results that can be expected. Furthermore, the integration time of the laser is of interest. All of these factors need to be taken into account in the technical implementation of the method.

[0090] Geometric and statistical parameters have traditionally been shown to enable characterization of road pavement surface texture at both macro and micro scales. These properties can be measured using a profile meter, where a triangulation laser is combined with a charge coupled device (CCD) / complementary metal oxide semiconductor (CMOS) sensor to perform asphalt texture characterization. However, the conventionally known measurement and analysis of the generated data as a three-dimensional point cloud is slower than real-time measurement and analysis. Additionally, post-processing 3D displacement measurements have traditionally been performed only in fixed mounted experimental rigs with electronic speckle pattern interferometry (ESPI) sensors.

[0091] Therefore, traditionally, such operations have not been considered, or perhaps not implemented in the real world for the purpose of on-road positioning to support autonomous and / or assisted transportation vehicle functions.

[0092] Although the functionality of the disclosed embodiments and system components for providing that functionality have been discussed with reference to specific terminology representing the functionality to be provided, it should be understood that in an implementation, the component functionality may be provided at least in part by components that are present in and known to be included in conventional transportation vehicles.

[0093] For example, as discussed above, the disclosed embodiments use software for performing functions that implement data measurement and analysis, at least in part, using software code stored on one or more non-transitory computer-readable media that runs on one or more processors in a transportation vehicle. Such software and processors can be combined to form at least one controller that is coupled to other components of the transportation vehicle to support and provide autonomous and / or auxiliary transportation vehicle functions in conjunction with the vehicle navigation system and multiple sensors. Such components can be coupled to the at least one controller for communication and control via a CAN bus of the transportation vehicle.

[0094] It should be further understood that the embodiments disclosed herein can be implemented using dedicated or shared hardware included in a transport vehicle. Therefore, without departing from the scope of the present invention, the components of the module can be used by other components of the transport vehicle to provide vehicle functions.

[0095] Exemplary embodiments are provided so that the present disclosure will be thorough and will fully convey the scope to those skilled in the art. Numerous specific details are set forth, such as examples of specific components, devices, and methods, to provide a thorough understanding of the embodiments of the present disclosure. In some illustrative embodiments, well-known processes, well-known device structures, and well-known technologies are not described in detail.

[0096] The terms used herein are only for the purpose of describing specific illustrative embodiments, and are not intended to be limiting. Unless the context indicates otherwise, the singular form of the elements involved above may be intended to include the plural form. The method steps, processes and operations described herein should not be interpreted as necessarily requiring them to be performed in the specific order discussed or illustrated, unless specifically identified as an execution order or a specific order is inherently necessary for the embodiment to be operable. It is also to be understood that additional or alternative steps may be adopted.

[0097] The disclosed embodiments include the methods described herein and their equivalents, non-transitory computer-readable media programmed to perform these methods, and computer systems configured to perform these methods. In addition, included is a carrier comprising: a component containing any of these methods, a non-transitory computer-readable medium programmed to implement instructions or perform these methods, and a system for performing these methods. The computer system and any sub-computer system will generally include: a machine-readable storage medium containing executable code; one or more processors; a memory coupled to one or more processors; an output device connected to one or more processors for executing the code, and an input device. The machine-readable medium may include any mechanism for storing or transmitting information in a machine-readable form, such as a computer processor. The information may be stored in, for example, volatile or non-volatile memory. Additionally, the embodiment functions may be implemented using embedded devices and an online connection to a cloud computing infrastructure that is available through a radio connection (e.g., wireless communication) to the cloud computing infrastructure.

[0098] Although certain embodiments have been described and illustrated in an exemplary form with a certain degree of particularity, it should be noted that the description and illustration are by way of example only. Many changes may be made in the details of the construction, combination and arrangement of components and operations. Therefore, such changes are intended to be included within the scope of the present disclosure, and the scope of protection of the present disclosure is defined by the claims.

[0099] Various embodiments of the present invention have been described, each with a different combination of elements. The present invention is not limited to the specific embodiments disclosed, and may include different combinations of disclosed elements, omissions of some elements, or replacement of elements with equivalents of such structures.

Claims

1. A transport vehicle device for performing on-road positioning, the device comprising: at least one frequency modulated continuous wave lidar sensor; means for generating lidar point cloud data based on data received from the at least one frequency modulated continuous wave lidar sensor, wherein the lidar point cloud data includes speckle characteristics of samples of the generated data; means for analyzing the lidar point cloud data to detect at least one road wear reference line in a road on which a transport vehicle is traveling, wherein speckle pattern analysis of the lidar point cloud data is utilized to provide information for detecting road wear based on changes in road surface roughness; as well as Means for facilitating alignment of a center of at least one wheel surface of a transport vehicle wheel with a detected center of the road wear reference line are coupled to the means for analyzing the lidar point cloud data.

2. The transport vehicle device according to claim 1 further includes a device for providing autonomous and / or auxiliary functions for operating the transport vehicle on a road, and the device for operating the transport vehicle on a road includes the device for facilitating the alignment of the center of at least one wheel surface of the transport vehicle wheel with the detected center of the road wear reference line.

3. The transport vehicle apparatus of claim 1 , wherein the frequency modulated continuous wave lidar sensor provides a coherent lidar using a continuously emitting laser that varies its frequency over a range, and compares the frequency of a return signal with a corresponding local reference signal to determine the distance to a target.

4. A transport vehicle device according to claim 2, wherein the transport vehicle includes a plurality of sensors, the plurality of sensors including the at least one frequency modulated continuous wave lidar sensor, and wherein the device for providing autonomous and / or auxiliary functions analyzes data generated by the plurality of sensors to provide autonomous and / or auxiliary functions in the transport vehicle.

5. A transport vehicle device according to claim 4, wherein the device for providing autonomous and / or auxiliary functions changes the weighting of sensor data from at least some of the multiple sensors based on an analysis of data indicating the reliability and / or availability of the sensor data from the multiple sensors.

6. A transport vehicle device according to claim 1, wherein the device for analyzing the lidar point cloud data analyzes the lidar point cloud data including x, y and z coordinates, intensity data and radial velocity data.

7. A transport vehicle device according to claim 1, wherein the device for analyzing the lidar point cloud data uses a deep neural network operation to perform feature extraction and classification based on low-dimensional characteristic features.

8. The transport vehicle apparatus of claim 7, wherein the deep neural network operation performs a machine learning method for data analysis to convert high-dimensional sensor data generated by the at least one frequency modulated continuous wave lidar sensor into low-dimensional characteristic features.

9. The transport vehicle apparatus of claim 7, wherein the deep neural network operation utilizes a deep neural network to perform a machine learning method for data analysis, the data analysis comprising speckle pattern analysis.

10. A transport vehicle device according to claim 7, wherein the deep neural network operation performs a machine learning method for data analysis, the machine learning method including at least one of a convolutional neural network (CNN), an application of a recurrent neural network (RNN), the use of a convolutional autoencoder, a generative adversarial network (GAN), a twin / triplet network, and / or an application of a graph neural network (GNN).

11. The transport vehicle apparatus of claim 1 , wherein the means for facilitating alignment comprises a lane departure warning system configured to warn a driver of the transport vehicle when the transport vehicle begins to move away from its lane of traffic on the road.

12. The transport vehicle apparatus of claim 1, wherein the means for facilitating alignment comprises a lane departure warning system configured to control maneuvering of the transport vehicle to maintain its lane of traffic on a road.

13. A transport vehicle device according to claim 1, wherein the device for analyzing the lidar point cloud data is further configured to: detect the presence or absence of road treatment materials on the road based on the at least one frequency modulated continuous wave lidar sensor mounted to the transport vehicle, wherein the device for facilitating alignment takes into account data indicating the presence or absence of the road treatment material so as to provide a lane departure warning function.

14. A method for performing on-road positioning using a device included in a transport vehicle, the method comprising: generating lidar point cloud data based on data received from at least one frequency modulated continuous wave lidar sensor mounted to the transport vehicle, wherein the lidar point cloud data includes speckle characteristics of samples of the generated data; analyzing the lidar point cloud data to detect at least one road wear reference line in a road on which the transport vehicle is traveling, wherein speckle pattern analysis of the lidar point cloud data is utilized to provide information for detecting road wear based on changes in road surface roughness; as well as Alignment of a center of at least one wheel surface of a transport vehicle wheel with a detected center of the road wear reference line is facilitated based on analysis of the lidar point cloud data.

15. The method of claim 14, providing autonomous and / or auxiliary functionality for operating a transport vehicle on a road by facilitating alignment of a center of at least one wheel surface of a transport vehicle wheel with a detected center of the road wear reference line.

16. The method of claim 14, wherein the FMCW LiDAR sensor provides a coherent LiDAR using a continuously transmitting laser that varies its frequency through a range, and compares the frequency of a return signal with a corresponding local reference signal to determine the distance to a target.

17. The method of claim 15, wherein the transport vehicle comprises a plurality of sensors, the plurality of sensors comprising the at least one FMCW LiDAR sensor, and The provision of autonomous and / or auxiliary functions involves analyzing data generated by the plurality of sensors to provide autonomous and / or auxiliary functions in the transport vehicle.

18. The method of claim 17, wherein autonomous and / or assistive functionality is provided by altering the weighting of sensor data from at least some of the plurality of sensors based on an analysis of data indicative of reliability and / or availability of sensor data from the plurality of sensors.

19. The method of claim 14, wherein the analysis of the lidar point cloud data comprises: Analyze LiDAR point cloud data including x-, y-, and z-coordinates, intensity data, and radial velocity data.

20. The method of claim 14, wherein the analysis of the lidar point cloud data comprises: Use deep neural network operations to perform feature extraction and classification based on low-dimensional feature characteristics.

21. The method of claim 20, wherein the deep neural network operation utilizes a deep neural network to perform a machine learning method for data analysis to convert high-dimensional sensor data generated by the at least one FMCW lidar sensor into low-dimensional characteristic features.

22. The method of claim 20, wherein the deep neural network operation utilizes a deep neural network to perform a machine learning method for data analysis, the data analysis comprising speckle pattern analysis.

23. The method of claim 20, wherein the deep neural network operation utilizes a deep neural network to perform a machine learning method for data analysis, the machine learning method comprising at least one of a convolutional neural network (CNN), application of a recurrent neural network (RNN), use of a convolutional autoencoder, a generative adversarial network (GAN), a twin / triplet network, and / or an application of a graph neural network (GNN).

24. The method of claim 14, wherein facilitating alignment comprises: When the transport vehicle begins to move out of its traffic lane on the road, an operation for warning of lane departure is performed to warn the driver of the transport vehicle.

25. The method of claim 14, wherein facilitating alignment comprises: Lane maneuvers for warning of lane departure are performed to control the maneuvering of the transport vehicle to maintain its lane of traffic on the road.

26. The method of claim 14, wherein the analysis of the lidar point cloud data comprises: The presence or absence of road treatment material on the road is detected based on the at least one frequency modulated continuous wave lidar sensor mounted to the transport vehicle, wherein, for the purpose of facilitating alignment, data indicating the presence or absence of the road treatment material is taken into account in order to provide a lane departure warning function.

27. A non-transitory machine-readable medium comprising machine-readable software code which, when executed on a processor, controls a method for performing on-road positioning using a device included in a transportation vehicle, the method comprising: generating lidar point cloud data based on data received from at least one frequency modulated continuous wave lidar sensor mounted to the transport vehicle, wherein the lidar point cloud data includes speckle characteristics of samples of the generated data; analyzing the lidar point cloud data to detect at least one road wear reference line in a road on which the transport vehicle is traveling, wherein speckle pattern analysis of the lidar point cloud data is utilized to provide information for detecting road wear based on changes in road surface roughness; as well as Alignment of a center of at least one wheel surface of a transport vehicle wheel with a detected center of the road wear reference line is facilitated based on analysis of the lidar point cloud data.

Images