Road curvature generation from real-world images as a data augmentation method

Abstract

The disclosed embodiments provide systems and methods for transforming real-world images to augment image data for use in improving training datasets used by machine learning algorithms to provide autonomous and / or assisted driving functionality.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] none. Technical Field

[0003] The disclosed embodiments relate to method operations and apparatus for use in transforming real-world images to enhance image data. Summary of the Invention

[0004] According to the disclosed embodiments, systems, components, and methods are provided for improving training data sets utilized by machine learning algorithms to provide autonomous and / or assisted driving functionality.

[0005] According to at least some disclosed embodiments, systems, components, and methods, real-world data is augmented with synthetic data generated from real-world data. The disclosed embodiments provide systems and methods for transforming real-world images to augment image data for use in improving training datasets utilized by machine learning algorithms to provide autonomous and / or assisted driving functionality.

[0006] Additional features of the disclosed embodiments will become apparent to those skilled in the art in view of the disclosure provided herein. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] The detailed description makes particular reference to the accompanying drawings, in which:

[0008] Figure 1A and Figure 1B Illustrated are examples of highway images used to train machine learning algorithms that facilitate autonomous and / or assisted driving functionality, according to disclosed embodiments.

[0009] Figure 2A and Figure 2B An example of a highway image with vehicle paths superimposed on the highway image is illustrated for training a machine learning algorithm according to the disclosed embodiments.

[0010] Figure 3 An example of a system and constituent components for performing raw image data acquisition and transformation of raw data according to the present disclosure is illustrated.

[0011] Figure 4 Illustrated are operations for performing image data acquisition and transformation to develop an enhanced training dataset for training a machine learning algorithm in accordance with the present disclosure. DETAILED DESCRIPTION

[0012] Each figure and description provided herein may have been simplified, to illustrate the various aspects relevant to clearly understanding the equipment, system and method described herein, while for the sake of clarity, eliminating other aspects that may be found in typical equipment, system and method. Those of ordinary skill will recognize that other elements and / or operations may be desirable and / or necessary for realizing the equipment, system and method described herein. Because such elements and operations are well known in the art, and because they do not promote a better understanding of the present disclosure, the discussion of such elements and operations may not be provided herein. However, the present disclosure is considered to inherently include all such elements, variations and modifications to described aspects that should be known to those of ordinary skill in the art.

[0013] Machine learning algorithms, often used in conjunction with autonomous delivery vehicles, require large amounts of data when developing the models upon which they perform autonomous and / or assisted driving functions. The large amounts of data used during the development of these models should be diverse and balanced, but acquiring this type of data from real-world data collection is often expensive.

[0014] The disclosed embodiments provide a technical solution for improving the data imbalance problem of training machine learning models using data collected on roads, particularly access-controlled highways, etc. For example, highways and similar roads typically include a proportionally larger number of straight roads.

[0015] For purposes of this disclosure, the term "road" includes any road, thoroughfare, route, or land-based path between two locations that has been paved or otherwise improved to permit travel by a transportation vehicle, including, but not limited to, a motor vehicle or other transportation vehicle including one or more wheels. It should be understood, therefore, that such a road may include one or more lanes and intersections with other roads, including on / off ramps, merging areas, etc., and may be included in a parkway, avenue, boulevard, expressway, toll road, interstate, highway, or primary, secondary, or tertiary local road.

[0016] Thus, as used herein, the term "highway" refers to a specific subset of roads, which typically include a greater proportion of long straight sections of road as compared to curved or arcuate roads. As contemplated by the present disclosure, exemplary highways are access-controlled highways or arterial roads. Thus, in the present disclosure, the term "road" refers inclusively to "highways."

[0017] For purposes of this disclosure, the term "on-road positioning" is used to refer to the ability to determine the position of a transport vehicle relative to a road, or a portion of a road (such as a lane) on which the transport vehicle is traveling.

[0018] As autonomous and driver-assisted transportation vehicle technologies further merge, it is envisioned that, in implementation, autonomous and / or assisted functionality will rely at least partially, and potentially entirely, on-road positioning performed in an automated or semi-automated manner based on Global Positioning Service (GPS) data, data generated by a plurality of sensors located on the vehicle, and machine learning algorithms and / or neural networks operatively coupled to the plurality of sensors and / or GPS for processing and interpreting such data to facilitate on-road positioning. As explained below, various conventional methods are known for developing and training machine learning models to perform on-road positioning using a variety of different types of sensors and a variety of different types of training data sets. However, each of these methods has drawbacks in the quality / quantity of data to be collected and / or processed during the construction of (one or more) machine learning models in real time for real-world use to safely control the travel of a transportation vehicle.

[0019] For the purposes of this disclosure, the term "autonomous and / or assisted functions" refers to functions that enable partial, full, or complete automation of vehicle control, encompassing and encompassing the five currently known levels of driving automation. Therefore, it should be understood that autonomous and / or assisted functions refer to operations performed by the vehicle in an automated manner via onboard equipment, or outputting alerts, prompts, recommendations, and / or guidance to the user, where these outputs are generated in an automated manner by the onboard equipment. Furthermore, autonomous and / or assisted functions may include driver assistance functions (Level 1), where the onboard equipment assists but does not control steering, braking, and / or acceleration, but the driver ultimately controls acceleration, braking, and monitoring of the vehicle's surroundings.

[0020] It should therefore be understood that such autonomous and / or assisted functionality may also include lane departure warning systems that provide a mechanism to warn the driver when a transport vehicle begins to move out of its lane on highways and major roads (unless the turn signal is on in that direction). Such systems may include those that warn the driver (visual, audible, and / or vibration warnings) in response to the vehicle leaving its lane (lane departure warning) and those that automatically take action to ensure the vehicle stays in its lane if no action is taken (lane keeping systems).

[0021] Likewise, autonomous and / or assisted functions may include partial automation (Level 2), in which the transport vehicle assists with steering or acceleration functions and monitors the vehicle's surroundings accordingly, allowing the driver to be relieved of some of the tasks associated with operating the transport vehicle. As understood in the automotive industry, partial automation still requires the driver to be prepared to assume all tasks of transport vehicle operation and to continuously monitor the vehicle's surroundings at all times.

[0022] Autonomous and / or assisted functions may include conditional automation (Level 3), in which the transport vehicle's equipment is responsible for monitoring the vehicle's surroundings and controlling the vehicle's steering, braking, and acceleration without driver intervention. It should be understood that at this level and above, the onboard equipment used to perform autonomous and / or assisted functions will be integrated with or include navigation functionality, providing the components with data to determine where the vehicle will travel. At Level 3 and above, the driver is theoretically permitted to disengage from monitoring the vehicle's surroundings, but may be prompted to take control of the transport vehicle's operations in certain situations that could hinder safe operation in conditional automation mode.

[0023] Thus, it should be understood that autonomous and / or assisted functionality may include systems that take over steering and / or keep the transport vehicle relatively centered in the traffic lane.

[0024] Likewise, autonomous and / or assisted functions may include high automation (Level 4) and full automation (Level 5), where onboard equipment enables automated steering, braking, and acceleration in an automated manner in response to monitoring of the vehicle's surroundings without driver intervention.

[0025] Therefore, it should be understood that autonomous and / or assisted functions may require monitoring the vehicle's surroundings, including the vehicle's roadway, and identifying objects in the surroundings to enable safe operation of the vehicle in response to traffic events and navigation directions, where such safe operation requires determining when to change lanes, when to change direction, when to change roads (exit / entry roads), when and in what order to merge or cross road intersections, and when to use turn signals and other navigation indicators to ensure that other vehicles / vehicle drivers are aware of upcoming vehicle maneuvers.

[0026] Furthermore, it should be understood that high and full automation may include analyzing and considering data provided by sources outside the vehicle to determine whether such level of automation is safe. For example, autonomous and / or assisted functionality at such levels may involve determining the likelihood of pedestrians in the environment surrounding the transport vehicle, which may involve referencing data indicating whether the current road is a highway or a parkway. Furthermore, autonomous and / or assisted functionality at such levels may involve accessing data indicating whether there is a traffic jam on the current road.

[0027] Conventional transportation vehicle navigation systems and conventional autonomous vehicles use GPS technology to locate their positions on the road. However, the conventional use of global positioning using GPS has the drawback that positioning is limited to a certain level of accuracy, more specifically, 5-10 meters in the best case (this typically requires providing an unobstructed, open view of the sky in a geographical area). In addition, in geographical areas that include relatively large buildings, trees, or geographical contours (such as canyons), the possibility of lower accuracy is much greater. This is because GPS-based positioning services require signals from GPS satellites. Dense materials (such as rock, steel, etc.), tall buildings, and large geographical terrain features may block or degrade GPS signals.

[0028] Thus, GPS is routinely used in conjunction with local landmarks, such as lane markings, for on-road positioning to improve the ability of vehicles with autonomous and / or assisted vehicle functionality to accurately perform on-road positioning. Conventionally, these local landmarks have been detected and identified from camera images or sensor data from other sensors obtained by one or more cameras / sensors located on the vehicle. For example, combining GPS data with data collected from forward-looking cameras and LiDAR, and even with data generated by ground-penetrating radar, has been conventionally discussed. Furthermore, the utility of such cameras for extracting simplified feature representations of road characteristics from onboard cameras to generate data indicative of roadside patterns that can be analyzed to perform on-road positioning has been discussed. Machine learning algorithms and the models developed therefrom facilitate the combination and operational utilization of various data inputs, including camera images, to provide autonomous and / or assisted functionality.

[0029] The disclosed embodiments are based on the recognition that recent autonomous vehicle traffic accidents provide evidence that there is a technical and real-world need to increase the robustness and / or reliability of machine learning models that govern autonomous and / or assistance functions; particularly in scenarios that occur less frequently and / or require the system to take more significant countermeasures to enable autonomous and / or assistance functions and control the movement of transportation vehicles.

[0030] Furthermore, autonomous and / or assisted functions often leverage machine learning models. Generally, the development of autonomous and / or assisted functions can take one of two different approaches. The first emphasizes modular system inputs (e.g., sensor fusion modules, scene understanding modules, dynamics prediction modules, path planning modules, and control modules). The second approach is an end-to-end approach, where sensor data is directly mapped to control signals under the guidance of computer algorithms.

[0031] Machine learning can be used extensively in both the first and second approaches. Machine learning models are trained using data sets that can be referred to as "training data." The capabilities and operational value of machine learning algorithms are indirectly affected by the quality and quantity of the training data. Still further, the collection or generation of training data must depend on a number of considerations and comparisons associated with both the availability and expense of generating and collecting specific subsets of training data. In an exemplary machine learning model, real-world data is collected for training. Furthermore, according to this example of a machine learning system that supports autonomous and / or assisted functions of a transportation vehicle, the training data can be collected by sensors installed in a test vehicle that is driven for extended periods by a test driver.

[0032] Machine learning models can be built and trained with the goal of facilitating their operation, in this case autonomous and / or assistance functions, with all types of input spanning the model's input domain (e.g., all road environments and conditions). To use a simplified but analogous example, if a machine learning algorithm is designed to detect dogs in images, the algorithm is designed to be able to detect all dog breeds. However, dog breeds can be visually very different from each other. Therefore, in order to fully develop the machine learning algorithm so that all dogs are recognized, the training data should be well-balanced and diverse. Still referring to the hypothetical dog detection algorithm, the meaning of the training data is that, in order to build a robust machine learning model, images of all dog breeds should appear in roughly equal proportions.

[0033] Therefore, the problem of data imbalance within training datasets presents several challenges for autonomous and / or assisted functions during highway driving. First, most sections of highway are straight or very gently curved, with only a small portion including significant curvature. Consequently, a machine learning algorithm trained using unfiltered and / or unmodified data collected from extended highway driving will be overtrained on straight roads and undertrained on curved sections. As a result, a delivery vehicle utilizing autonomous and / or assisted functions relying on this exemplary machine learning model will likely perform poorly on curved highway sections while still failing to achieve significant performance improvements over the baseline on straight highway sections.

[0034] The systems and methods of training data transformation described with reference to Figures 1-4 address the natural imbalance in training data collected for training machine learning algorithms used in the performance of autonomous and / or assisted functions.

[0035] Conventionally, two strategies are used to address the data imbalance between straight highway / road data 104 and curved highway / road data within a training dataset. First, a machine learning algorithm can be trained through simulation. During simulation, the ratio of straight and curved sections experienced along the simulated highway can be balanced as desired. In other words, the entire body of training data can be customized. This approach can be costly. For example, a team of engineers and artists is required to build a simulator that simulates real-world highway driving, and the mileage and scenery diversity specifications of the training data will increase the cost of any developed simulator.

[0036] Instead, generating diverse road data without developing a simulator might be useful. Furthermore, machine learning models trained using simulated data may not necessarily perform well in the real world. Simulated data, no matter how realistic it appears, can be completely different from real-world data. There is no a priori guarantee that this simulation approach will produce a robust machine learning model.

[0037] In contrast, according to the disclosed embodiments, systems and methods can generate synthetic data that is more realistic and more adaptable to real-world problems because the disclosed embodiments draw from real-world data.

[0038] However, if it is desired to train using real-world data collected by driving along highways, a portion of the straight road data can be removed from the training dataset to develop a more diverse training dataset, as per conventional methods. Similarly, the curved road data can be manually replicated, perhaps multiple times, until the training dataset includes the desired ratio of straight road data to curved road data. This approach also has drawbacks, as removing data is wasteful. Data collection is expensive. Removing a portion of the straight road driving data removes a relatively large portion of the total collected data. Alternatively, the curved road data can be replicated multiple times to match the amount of straight road data, resulting in reduced within-class diversity in the curved road data. Machine learning algorithms thrive on developing models that are more diverse and balanced in the training data. Artificially replicating one type of data to achieve data type balance can address data imbalance, but such an action creates another type of data imbalance (for example, the training data for one class is much more diverse than the training data for another class). This can cause the algorithm to perform better with one type of input than with another.

[0039] According to the disclosed embodiments, and as Figures 1A-1B, in addition to other transformed image data, the straight road data 104 may include data describing one or more straight road segments 112, and the curved road data includes data describing one or more curved road segments 114. Thus, systems and methods configured in accordance with the disclosed embodiments may generate new data in the curved road segment category and may bring this category to the same level of diversity as the straight road segment category.

[0040] The systems and / or methods of the presently disclosed embodiments may transform images of one or more straight highway segments 112 into images of one or more synthetic curved highway segments 114 using computer vision algorithms and / or processes.

[0041] Figure 1A The apparatus is depicted as being mounted on a transport vehicle 120 (e.g., a car, again see Figure 3 ) of the camera 118 (see Figure 3 ) captures an example raw image 116 of one of the straight highway segment(s) 112 . Figure 1B A transformed arcuate image 122 of an example of one or more synthetic curved highway segments 114 is depicted. Figure 1B In the transformed image 122, the original image 116 ( Figure 1A ) is modified so that the road shown therein is given a curvature of 1 / 300 meter. In the following, the term "curvature" refers to the inverse of the radius, so that a smaller curvature implies a relatively larger radius.

[0042] exist Figure 1B In the illustrated example, a curvature of 1 / 300 meter indicates that one of the synthetic curved highway segment(s) 114 includes a radius of 300 meters if the road segment were extended to complete a circle.

[0043] According to the disclosed embodiments and as Figure 4 As shown in , a method 400 for transforming a straight highway image 116 into a transformed arcuate image 122 may include: (i) identifying a horizontal line 402; (ii) identifying directional information 404; (iii) processing pixels 406 based on an assumption that the highway is relatively flat; and (iv) generating a synthetic curved highway segment 408 from information derived during previous operations and external knowledge of the camera used during a data collection operation 410.

[0044] like Figure 3 As shown in FIG, and as part of identifying a horizontal line, first, an original straight highway image 116 may be acquired by a camera 118 deployed on or within a transport vehicle 120. A horizontal line 124 may be identified within the image(s). Many horizontal detection algorithms may be utilized to assist in the identification. Figures 1A-2B. A horizontal line 124 is shown in FIG. Thereafter, a plurality of pixels 126 disposed below the horizontal line 124 may be identified as corresponding to the straight highway segment 112. Thus, the disclosed embodiments contemplate that approximately all pixels below the horizontal line 124 represent the straight highway segment 112. Furthermore, according to the disclosed embodiments, additional techniques, such as identifying road lines, may be used to increase the accuracy of pixel regions belonging to roads.

[0045] Subsequently, the camera model and camera parameters attached during image collection can be used to inform the operation of transforming each road pixel 126 into a ray vector described by camera coordinates. This information is described in image space, where each ray vector indicates the direction associated with the portion of straight road segment 112 imaged by a single pixel. The direction of each ray vector can be indicated relative to camera 118, but distance information need not be provided. If the example camera has a 60-degree field of view, it is possible to infer that the direction of the object is, for example, approximately 30 degrees to the left of the forward direction. However, a 2D image does not provide the viewer with information about how far away the imaginary object is located. To expand on this example, the object can be small and nearby, or it can be relatively large and very far away. Therefore, it is impossible to discern which of these possibilities represents reality based solely on a 2D image.

[0046] As noted above, raw straight highway image 116 can be analyzed to reveal the orientation of each highway pixel 126, and highways tend to be relatively flat. Furthermore, the vertical distance 130 of camera 118 from highway surface 132 is a known parameter among the extrinsic parameters of the camera model used to collect raw image 116. Therefore, this information informs the assumption that all colors in highway pixels 126 originate from a shared flat plane orthogonal to the vertical line connecting camera 118 and road surface 132. Consequently, the orientation information for each highway pixel 126 can be combined with the presence of each pixel 126 on the flat plane at the known vertical distance 130 from camera 118. This combination of information can be processed to determine the location of each highway pixel 126 on the flat plane. Thus, as distance increases, highway pixels 126 can be mathematically described using spatial coordinates in physical space.

[0047] Once the position information for each road pixel 126 is determined, a curvature transformation can be applied to each point on the flat plane to develop a curved surface. The same camera model and extrinsic parameters previously used to determine the ray vector(s) for each road pixel 126 can then inform the transformation of the curved surface back into image space (e.g., camera coordinates). Thus, a transformed arcuate image 122 of one or more synthetic curved road segments 114 can be generated.

[0048] Systems and methods provided according to the disclosed embodiments can be used to perform training data transformations that produce blocks of black (or empty) pixels. The color and image data originally present at image-space corners of road pixels 126 may be shifted left or right, depending on the curvature applied during the transformation. As a result, the corners of the image of one or more synthetic curved road segments 114 may appear devoid of color information. The presence of black or colorless corners is undesirable because machine learning algorithms can be trained to identify empty pixels as an indication of curvature, thereby reducing the effectiveness of synthetic curved road segments 114 within the training dataset. The presence of colorless pixels can be addressed by preserving the original color at image-space locations that would otherwise be black pixels.

[0049] The systems and methods provided according to the disclosed embodiments can leverage techniques from both the fields of machine learning and computer vision. Furthermore, the systems and methods provided according to the disclosed embodiments can exploit assumptions unique to the data collection method; specifically, the disclosed embodiments can operate based on the assumption that colors present at image pixels belonging to roads can be considered to be reflected from the same flat surface. Therefore, the curvature transformation process can then be performed in a spatial coordinate system, and the mapping between pixel coordinates and spatial coordinates can produce usable images for supplementing the training dataset. Furthermore, overwriting black / colorless pixels resulting from the curvature transformation process with their original colors (e.g., preserving the original color information even after it has been spatially transformed away from its original location) can optionally further improve the quality of the training dataset.

[0050] Systems and methods provided according to the disclosed embodiments can be useful for developing autonomous and / or assisted functionality, such as when a neural network predicts a vehicle path spanning approximately the next 30 meters for each image. In particular, systems and methods provided according to the disclosed embodiments can generate images of one or more synthetic curved highway segments 114 to enhance the number of curved highway segments present in a collected training dataset. Notably, transforming the training images to increase specific curvature can be desirably combined with transforming the data associated with the training images. For example, when a vehicle path is associated with a transformed training image, the corresponding vehicle path can be transformed to reflect the same curvature. Thus, the vehicle path information can still be used to follow the highway and / or lanes therein over the next 30 meters for autonomous and / or assisted functionality. When curvature is added to the images without transforming the corresponding vehicle path, the original path may cross lanes or otherwise incongruently indicate the vehicle path along the synthetic curved highway segment 114. This data augmentation technique can improve the accuracy and stability of neural network models used to implement autonomous and / or assisted functionality.

[0051] Now refer to Figure 2A and Figure 2B , depicting the original image 116b and the transformed bow image 122b. Figure 2A is a raw image 116b of an exemplary section of relatively straight highway segment 112 captured by camera 118 mounted on transport vehicle 120. A first set of points 138 indicates the actual path taken by the driver 30 meters into the future from the time the raw image 116b was captured. Figure 2B , shows the transformed bow-shaped image 122b after the transformation imparts a high negative curvature (e.g., curving to the right) to the highway segment. Figure 2B , a second set of points 140 indicates a new path transformed to correspond to the curvature imparted to the highway segment that the driver should take to maintain lane positioning on the synthetic curved highway segment 114 of the transformed arcuate image 122b. The second set of points 140 of the curved path is obtained by applying the same curvature transformation as that applied to the highway segment. Figure 2B A comparison is further made between the first set of points 138 and the second set of points 140, which represent the actual travel path and the path reflecting the highway segment appropriately adjusted to fit the transformed arcuate image 122b, respectively.

[0052] Alternatively, supervised learning, the most common form of machine learning, involves enabling learning based on a training dataset during a training phase, enabling the ability to learn how to label input data for classification. Deep learning improves on supervised learning methods by considering multiple layers of representation, where each layer uses information from the previous layer to learn more deeply. Deeper architectures with many stacked layers are one aspect, and convolutional neural networks (CNNs) also consider 2D / 3D local neighborhood relationships in pixel / voxel space through convolutions on spatial filters.

[0053] Supervised deep learning involves applying multiple layers or stages of functional operations to improve understanding of the resulting data, which is then fed into further functional operations. For example, supervised deep learning for classifying data into one or more categories can be performed, for example, by performing feature learning (involving one or more stages of convolution, rectifier linear units (ReLU), and pooling) to enable subsequent classification of sample data, thereby identifying learned features by applying a softmax function to enable differentiation between objects and background in input image data. These operations can be performed to generate image class labels for classification purposes.

[0054] Similarly, supervised deep learning operations can be performed for regression by operating in parallel on the red, green, and blue (RGB) image and the distance / disparity map data to the ground, performing multiple convolutions and concatenating the results for subsequent processing. These operations can be performed to generate image regression labels for subsequent analysis.

[0055] Furthermore, supervised deep learning operations for semantic segmentation can be performed by feeding RGB image data into a convolutional encoder / decoder, which can include multiple stages of convolution, batch normalization (which is not only applicable to segmentation but also to other networks), ReLU, and pooling, followed by multiple stages of convolution, batch normalization, and ReLU with upsampling. The resulting data can then be processed by applying a softmax function to provide output data with segmentation labels for each pixel. Thus, the disclosed systems and methods for transforming image data can preserve real-world information to successfully apply the aforementioned techniques.

[0056] Furthermore, it should be understood that while the disclosed embodiments may be used generally for the purpose of facilitating robust autonomous and / or assisted transportation vehicle functionality, the disclosed embodiments may have particular utility in providing such functionality when lane markings are obscured by weather conditions such as snowfall. In such conditions, lane markings and road markings conventionally used to assist with on-road positioning become obscured. For example, any amount of snow can obscure lane markings and road markings; furthermore, heavy snow can alter the appearance of the surrounding environment along the road to the point where analyzing roadside patterns to provide additional data to combine with GPS analysis for performing on-road positioning is impossible.

[0057] In this regard, it should be understood that at least one embodiment may include a feedback mechanism that determines the quantity and / or quality of data generated and / or analyzed during the disclosed operation. Such a feedback mechanism may be used to selectively increase or decrease reliance on the transformed image(s) 122 in providing autonomous and / or auxiliary functions. This may be achieved, for example, by dynamically weighting data that has or has not undergone a transformation. It should also be understood that such a feedback mechanism may include comparisons to thresholds to maintain at least minimum parameters to ensure safe operation of autonomous and / or auxiliary functions.

[0058] Furthermore, it should be understood that the mechanisms for dynamically weighting such data may be implemented in one or more of a variety of conventionally known techniques for enabling sensor data fusion, such as using a Kalman filter, processing based on the central limit theorem, Bayesian networks, Dempster-Shafer theorem, CNNs, or any other mathematical operation disclosed herein.

[0059] As explained above, the disclosed embodiments can be implemented in conjunction with components of autonomous and / or assisted driving systems included in transportation vehicles. Thus, the utility of the disclosed embodiments has been described in detail within those technical contexts. However, the scope of the innovative concepts disclosed herein is not limited to those technical contexts.

[0060] Furthermore, it should be understood that the presently disclosed components for analyzing image data depicting a road on which a vehicle is transported may include any combination of sensors and functionality disclosed herein implemented in hardware and / or software to provide the disclosed functionality.

[0061] Furthermore, it should be understood that such assistance technologies may include, but are not limited to, technologies commonly referred to as driver assistance systems (DAS) or advanced driver assistance systems (ADAS), implemented using hardware and software included in transportation vehicles. These conventionally known systems assist the driver in decision-making and control, but decision-making and control are inevitably the driver's responsibility. Furthermore, these systems can be "active" or "passive" in terms of how they are implemented. Active DAS means that the vehicle itself controls various longitudinal and / or lateral aspects of the vehicle's driving behavior, or more specifically, controls very specific driving tasks, through its sensors, algorithms, processing systems, and actuators. Passive DAS means that the vehicle simply assists the driver in controlling various longitudinal and / or lateral aspects of vehicle control through its sensors, algorithms, processing systems, and human-machine interface (HMI). For example, in a collision avoidance situation, an active system would stop the vehicle or steer it around an obstacle in its direct path. Passive systems would provide the driver with some type of visual, auditory, and tactile cues to stop the vehicle or steer it around an obstacle.

[0062] Thus, DAS systems assist drivers with many tasks deeply embedded in the driving process and are specifically implemented to enhance vehicle and road safety and driver convenience. Such DAS systems include, but are not limited to, cruise control, adaptive cruise control (ACC), active steering for lane keeping, lane change assist, highway merge assist, collision mitigation and avoidance systems, pedestrian protection systems, automated and / or assisted parking, sign recognition, blind spot detection for collision mitigation, and stop-and-go traffic assist. Thus, the disclosed embodiments provide such DAS systems with additional and potentially more accurate data to provide this assistance functionality.

[0063] It should be further appreciated that the disclosed embodiments leverage functionality from a number of different technical fields to provide additional mechanisms and methods for developing training datasets to facilitate autonomous and / or assisted driving functionality by combining analyses performed in computer vision and machine learning.

[0064] While the functionality of the disclosed embodiments and system components for providing that functionality have been discussed with reference to specific terminology indicating the functionality to be provided, it should be understood that in implementation, the component functionality may be provided at least in part by components currently and known to be included in conventional transportation vehicles.

[0065] For example, as discussed above, the disclosed embodiments use software to perform functions to enable measurement and analysis of data at least in part using software code stored on one or more non-transitory computer-readable media running on one or more processors in a transport vehicle. Such software and processors can be combined to form at least one controller that is coupled to other components of the transport vehicle to support and provide autonomous and / or auxiliary transport vehicle functions in conjunction with the vehicle navigation system and multiple sensors. Such components can be coupled to the at least one controller for communication and control via the CAN bus of the transport vehicle. It should be understood that such a controller can be configured to perform the functions disclosed herein.

[0066] It should be further understood that the embodiments disclosed herein can be implemented using dedicated or shared hardware included in a transport vehicle. Thus, without departing from the scope of the present invention, the components of the module can be used by other components of the transport vehicle to provide vehicle functionality.

[0067] The exemplary embodiments are provided so that this disclosure will be thorough and will fully convey the scope to those skilled in the art. Numerous specific details are set forth, such as examples of specific components, devices, and methods, to provide a thorough understanding of the embodiments of the present disclosure. In some illustrative embodiments, well-known processes, well-known device structures, and well-known technologies are not described in detail.

[0068] The terms used herein are for the purpose of describing specific illustrative embodiments only and are not intended to be limiting. Unless the context indicates otherwise, the singular form of the elements mentioned above may be intended to include the plural form. The method processes and operations described herein should not be interpreted as necessarily requiring them to be performed in the particular order discussed or illustrated, unless specifically identified as an execution order, or a particular order is inherently necessary for the operation of the embodiment. It should also be understood that additional or alternative operations may be employed.

[0069] Disclosed embodiments include the methods described herein and their equivalents, non-transitory computer-readable media programmed to perform the methods, and computer systems configured to perform the methods. Furthermore, a vehicle is included that includes components containing any of the methods, non-transitory computer-readable media programmed to implement instructions or perform the methods, and a system for performing the methods. The computer system and any sub-computer systems will typically include: a machine-readable storage medium containing executable code; one or more processors; memory coupled to the one or more processors; input devices; and output devices coupled to the one or more processors for executing the code. A machine-readable medium may include any mechanism for storing or transmitting information in a machine-readable form, such as a computer processor. The information may be stored, for example, in volatile or non-volatile memory. Furthermore, embodiment functionality may be implemented using an embedded device and an online connection to a cloud computing infrastructure, available via a radio connection (e.g., wireless communication) to the cloud computing infrastructure. The training dataset, image data, and / or transformed image data may be stored in one or more memory modules coupled to the memory of the one or more processors.

[0070] Although certain embodiments have been described and illustrated in an exemplary manner with a certain degree of particularity, it should be noted that the description and illustration are provided by way of example only. Many changes may be made to the details of the construction, combination, and arrangement of components and operations. Therefore, such changes are intended to be included within the scope of this disclosure, the scope of which is defined by the claims.

[0071] The embodiment(s) detailed above may be combined in whole or in part with any of the alternative embodiment(s) described.

Claims

1. A transportation vehicle apparatus for developing transformed training data, the apparatus comprising: transport vehicles; at least one processor; at least one memory module, wherein the at least one processor analyzes image data depicting a road stored on the at least one memory module and collected by a transport vehicle to detect horizontal lines and directional information of pixels representing the road, and wherein the image data is generated using at least one sensor mounted to the transport vehicle; and a component for analyzing the image data coupled to the at least one sensor, wherein the component for analyzing the image data transforms the image data depicting one or more straight road segments of a road into transformed image data depicting a synthetic curved road segment of an arcuate road, and utilizes the transformed image data to develop a training data set for training a machine learning algorithm.

2. The transport vehicle apparatus according to claim 1, wherein The image data depicts a corresponding driving path in addition to the road, and The means for analyzing the image data transforms the image data into transformed image data depicting a synthetic curved road segment and a corresponding curved path, and applies the same curvature transform to pixels representing the road and the corresponding driving path.

3. The transport vehicle apparatus according to claim 1 further comprises an autonomous and / or assisted driving system for operating the transport vehicle on a road, wherein the autonomous and / or assisted driving system comprises a component for training one or more machine learning algorithms using the transformed image data. 4 . The transport vehicle apparatus according to claim 3 , wherein an autonomous and / or assisted driving system that operates the transport vehicle on a road replaces a portion of the image data with the transformed image data.

5. The transport vehicle apparatus according to claim 1, wherein: The transport vehicle equipment further includes the at least one sensor.

6. The transport vehicle apparatus according to claim 5, wherein: The at least one sensor is a camera.

7. The transport vehicle apparatus according to claim 6, wherein: The camera includes predetermined extrinsic parameters.

8. The transport vehicle apparatus according to claim 7, wherein: The camera is deployed at a predetermined distance above the road.

9. The transport vehicle apparatus according to claim 1, wherein: The means for analyzing the image data transforms direction information of pixels representing the road from image coordinates into space coordinates.

10. The transport vehicle apparatus according to claim 9, wherein: The component for analyzing the image data determines a flat plane based on a perpendicular distance of the camera from the road.

11. The transport vehicle apparatus according to claim 10, wherein: The component for analyzing the image data determines that pixels disposed on a flat plane are indicative of a surface of a road.

12. The transport vehicle apparatus according to claim 11, wherein: The means for analyzing the image data determines distance information for pixels indicative of a road surface.

13. An image transformation method for training a machine learning algorithm, the method comprising: analyzing, with a processor, image data stored in a memory depicting one or more straight road segments of a road and collected by one or more data collection transport vehicles to detect horizontal lines and directional information of pixels representing the road, wherein the image data was generated using at least one sensor mounted to the one or more data collection transport vehicles; transforming the image data into transformed image data depicting a synthetic curved road segment of an arcuate road; and The transformed image data is used to develop a training dataset for training a machine learning algorithm.

14. The image conversion method according to claim 13, wherein the image data depicts a corresponding driving path in addition to a road, and the method further comprises: The image data is transformed into transformed image data depicting a synthetic curved road segment and a corresponding curved path, and the same curvature transform is applied to pixels representing the road and the corresponding driving path.

15. The image transformation method of claim 13, further comprising providing autonomous and / or assisted functionality for operating a transport vehicle on a road, which includes transforming a vehicle path corresponding to the transformed image data.

16. The image transformation method according to claim 15, wherein: The operation of providing autonomous and / or assisted functionality for operating a transport vehicle on a road further includes replacing a portion of the image data with the transformed image data.

17. The image conversion method according to claim 13, wherein: The at least one sensor is a camera.

18. The image transformation method according to claim 17, wherein: The camera includes predetermined extrinsic parameters.

19. The image conversion method according to claim 18, wherein: The camera is deployed at a predetermined distance above the road.

20. The image conversion method according to claim 13, further comprising converting direction information of pixels representing the road from image coordinates to space coordinates. The image transformation method of claim 20 , further comprising determining the flat plane based on a vertical distance between the camera and the road.

22. The image transformation method of claim 21, further comprising determining that pixels disposed on a flat plane indicate a road surface.

23. The image conversion method of claim 22, further comprising determining distance information for pixels indicative of a road surface.

24. The image transformation method of claim 20, further comprising preserving original color information in the image coordinated for pixels moved in spatial coordinates during the transformation operation.

25. A non-transitory machine-readable medium comprising machine-readable software code, which, when executed on a processor, controls an image transformation method for training a machine learning algorithm, the method comprising: analyzing image data depicting one or more straight road segments of a road and collected by a transport vehicle to detect horizontal lines and directional information of pixels representing the road, wherein the image data is generated using at least one sensor mounted to the transport vehicle; transforming the image data into transformed image data depicting a synthetic curved road segment of an arcuate road; and The transformed image data is used to develop a training dataset for training a machine learning algorithm.

26. The non-transitory machine-readable medium of claim 25 comprising machine-readable software code, which, when executed on a processor, controls an image transformation method for training a machine learning algorithm, wherein the image data depicts a corresponding driving path in addition to a road, and the method further comprises: The image data is transformed into transformed image data depicting a synthetic curved road segment and a corresponding curved path, and the same curvature transform is applied to pixels representing the road and the corresponding driving path.

27. The non-transitory machine-readable medium of claim 26 comprising machine-readable software code, which, when executed on a processor, controls an image transformation method for training a machine learning algorithm, the method comprising: A machine learning algorithm is trained using a portion of the image data and the transformed image data.

28. The non-transitory machine-readable medium of claim 27 comprising machine-readable software code, which, when executed on a processor, controls an image transformation method for training a machine learning algorithm, the method comprising: Machine learning algorithms are executed to provide autonomous and / or assisted functionality for operating a transportation vehicle on a road.

Images