System and method for estimating future paths

Abstract

A system and method estimates a future path ahead of a current position of a vehicle. The system includes at least one processor programmed to: obtain images of an environment ahead of a current arbitrary position of a vehicle navigating a road; obtain a trained system trained to estimate a future path based on a first plurality of images of the environment ahead of the vehicle navigating the road; apply the trained system to the images of the environment ahead of the current arbitrary position of the vehicle; and provide an estimated future path ahead of the current arbitrary position of the vehicle based on applying the trained system to the images.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of priority to U.S. Provisional Patent Application No. 62 / 275,046, filed January 5, 2016, and U.S. Provisional Patent Application No. 62 / 373,153, filed August 10, 2016. Both of the foregoing applications are incorporated herein by reference in their entireties. Technical Field

[0003] The present disclosure generally relates to advanced driver assistance systems (ADAS) and autonomous vehicle (AV) systems. Additionally, the present disclosure relates to systems and methods for processing images and systems and methods for estimating a future path of a vehicle. Background Art

[0004] Advanced driver assistance systems (ADAS) and autonomous vehicle (AV) systems use cameras and other sensors along with object classifiers that are designed to detect specific objects in the environment of the vehicle navigating the road. Object classifiers are designed to detect predefined objects and are used within ADAS and AV systems to control the vehicle or alert the driver based on, for example, the type of object whose location is detected. However, as standalone solutions, pre-configured classifiers are limited in their ability to handle the infinitesimal variations and details of the road environment and its surroundings, as well as its generally dynamic nature (moving vehicles, shadows, etc.). As ADAS and AV systems evolve towards fully autonomous operation, it would be beneficial to enhance the capabilities of these systems. Summary of the Invention

[0005] The following detailed description refers to the accompanying drawings. Whenever possible, the same reference numerals are used in the drawings and the following description to refer to the same or similar parts. Although several illustrative embodiments are described herein, modifications, adaptations, and other implementations are possible. For example, components shown in the drawings may be replaced, added, or modified, and the illustrative methods described herein may be modified by replacing, rearranging, removing, or adding steps to the disclosed methods. Accordingly, the following detailed description is not limited to the disclosed embodiments and examples.

[0006] The disclosed embodiments provide systems and methods that can be used as part of, or in combination with, autonomous navigation / driving and / or driver assistance technology features. Driver assistance technology refers to any suitable technology used to assist a driver with navigation and / or control of their vehicle (e.g., FCW, LDW, and TSR), as opposed to fully autonomous driving. In various embodiments, the system may include: one, two, or more cameras that can be installed in the vehicle; and an associated processor that monitors the vehicle's environment. In other embodiments, additional types of sensors can be installed in the vehicle and can be used in autonomous navigation and / or driver assistance systems. In some examples of the presently disclosed subject matter, the system may provide technology for processing images of the environment in front of a vehicle navigating a road for use in training a system (e.g., a neural network, a deep learning system applying, for example, a deep learning algorithm, etc.) to estimate the vehicle's future path based on the images. In other examples of the presently disclosed subject matter, the system may provide technology for using a trained system to process images of the environment in front of a vehicle navigating a road to estimate the vehicle's future path.

[0007] According to an example of the presently disclosed subject matter, a system for estimating a future path ahead of a current position of a vehicle is provided. The system may include at least one processor programmed to: obtain images of an environment ahead of a current arbitrary position of a vehicle navigating a road; obtain a trained system trained to estimate a future path based on a first plurality of images of the environment ahead of the vehicle navigating the road; apply the trained system to the images of the environment ahead of the current arbitrary position of the vehicle; and provide an estimated future path ahead of the current arbitrary position of the vehicle based on applying the trained system to the images.

[0008] In some embodiments, the trained system comprises a piecewise affine function of a global function. In some embodiments, the global function may comprise: convolution, max pooling, and / or rectified linear unit (ReLU).

[0009] In some embodiments, the method may further include: utilizing the estimated future path ahead of the current position of the vehicle to control at least one electronic or mechanical unit of the vehicle to change at least one motion parameter of the vehicle. In some embodiments, the method may further include: utilizing the estimated future path ahead of the current position of the vehicle to provide sensory feedback to a driver of the vehicle.

[0010] In some embodiments, the estimated future path of the vehicle ahead of the current position may be further based on identifying one or more predefined objects appearing in the image of the environment using at least one classifier.

[0011] The method may further comprise utilizing the estimated future path ahead of the current position of the vehicle to provide a control point for a steering control function of the vehicle.

[0012] In some embodiments, applying the trained system to images of the environment in front of the vehicle's current position provides two or more estimated future paths of the vehicle in front of the current position.

[0013] In some embodiments, the at least one processor may be further programmed to utilize the estimated future path ahead of the vehicle's current position in estimating a road profile ahead of the vehicle's current position.

[0014] In some embodiments, applying the trained system to an image of the environment in front of the vehicle's current position provides two or more estimated future paths of the vehicle in front of the current position, and may also include: estimating a road profile along each of the two or more estimated future paths of the vehicle in front of the current position.

[0015] In some embodiments, the at least one processor may be further programmed to utilize an estimated future path ahead of the current position of the vehicle in detecting one or more vehicles located in or near the future path of the vehicle.

[0016] In some embodiments, the at least one processor can be further programmed to cause at least one electronic or mechanical unit of the vehicle to change at least one motion parameter of the vehicle based on the position of one or more vehicles determined to be in or near the future path of the vehicle.

[0017] In some embodiments, the at least one processor may be further programmed to trigger a perception alert to indicate to a user that the one or more vehicles are determined to be in or near the future path of the vehicle.

[0018] The method for processing images may include: obtaining a first plurality of training images, each of the first plurality of training images being an image of an environment in front of a vehicle navigating a road; for each of the first plurality of training images, obtaining a pre-stored path of the vehicle in front of a corresponding current position of the vehicle; given an image, training a system to provide a future path of the vehicle for navigating a road in front of a corresponding current position of the vehicle, wherein training the system includes: providing the first plurality of training images as input to the system; calculating a loss function at each iteration of training based on the corresponding temporary future path estimated by the current state of the weights and the corresponding pre-stored path; and updating the weights of the neural network according to the result of the loss function.

[0019] In some embodiments, obtaining the first plurality of training images may further comprise: for each image from the first plurality of training images, obtaining data indicating the position of the vehicle on the road at the time the image was captured. In some embodiments, obtaining the first plurality of training images comprises: obtaining the position of at least one lane marking in at least one image from the first plurality of training images, and wherein, for each image from the first plurality of training images, obtaining data indicating the position of the vehicle on the road at the time the image was captured comprises: for the at least one image from the first plurality of training images, determining the position of the vehicle on the road at the time the at least one image was captured based on the position of the at least one lane marking in the at least one image.

[0020] In some embodiments, determining the position of the vehicle on the road at the moment of capturing the at least one image from the first plurality of training images based on the position of the at least one lane marking in the at least one image may include: determining the position of the vehicle on the road at a predefined offset from the position of the at least one lane marking.

[0021] In some embodiments, the pre-stored path of the vehicle ahead of the corresponding current position of the vehicle can be determined based on the position of the vehicle on the road at the corresponding moment when the corresponding second plurality of training images were captured, and wherein the second plurality of training images can be images from the first plurality of training images captured after the images associated with the current position.

[0022] In some embodiments, training the system comprises multiple iterations and may be performed until a stopping condition is met.

[0023] In some embodiments, the method may further comprise providing as output a trained system configured to provide an estimate of a future path for a vehicle navigating a road, given any input image of an environment in front of the vehicle.

[0024] In some embodiments, the first plurality of training images may include a relatively high number of images of environments that appear relatively sparsely on a road. In some embodiments, the first plurality of training images may include a relatively high number of images of environments that include curved roads. In some embodiments, the first plurality of training images may include a relatively high number of images of environments that include lane splits, lane merges, highway exits, highway entrances, and / or merging points. In some embodiments, the first plurality of training images may include a relatively high number of images of environments that include poor or no lane markings, Botts' points, and / or shadows on the road ahead of the vehicle.

[0025] In some embodiments, the stopping condition may be a predefined number of iterations.

[0026] In some embodiments, the method may further comprise providing as output a trained system having a configuration of the trained system reached at a last iteration of training the system.

[0027] According to another aspect of the presently disclosed subject matter, a method for estimating a future path ahead of a current position of a vehicle is provided. According to an example of the presently disclosed subject matter, the method for estimating a future path ahead of a current position of a vehicle may include: obtaining images of an environment ahead of a current arbitrary position of a vehicle navigating a road; obtaining a trained system trained to estimate a future path based on a first plurality of images of the environment ahead of the vehicle navigating a road; and applying the trained system to the images of the environment ahead of the current arbitrary position of the vehicle to thereby provide an estimated future path ahead of the current arbitrary position of the vehicle.

[0028] In some embodiments, the trained system comprises a piecewise affine function of a global function. In some embodiments, the global function may comprise: convolution, max pooling, and / or rectified linear unit (ReLU).

[0029] In some embodiments, the method may further include: utilizing the estimated future path ahead of the current position of the vehicle to control at least one electronic or mechanical unit of the vehicle to change at least one motion parameter of the vehicle. In some embodiments, the method may further include: utilizing the estimated future path ahead of the current position of the vehicle to provide sensory feedback to a driver of the vehicle.

[0030] In some embodiments, the estimated future path of the vehicle ahead of the current position may be further based on identifying one or more predefined objects appearing in the image of the environment using at least one classifier.

[0031] The method may further comprise utilizing the estimated future path ahead of the current position of the vehicle to provide a control point for a steering control function of the vehicle.

[0032] In some embodiments, applying the future path estimation trained system to an image of the environment ahead of the vehicle's current position provides two or more estimated future paths for the vehicle ahead of the current position.

[0033] In some embodiments, the method may further comprise utilizing the estimated future path ahead of the current position of the vehicle in estimating a road profile ahead of the current position of the vehicle.

[0034] In some embodiments, applying the future path estimation trained system to an image of the environment in front of the vehicle's current position provides two or more estimated future paths of the vehicle in front of the current position, and may also include: estimating a road profile along each of the two or more estimated future paths of the vehicle in front of the current position.

[0035] In some embodiments, the method may further comprise utilizing an estimated future path ahead of the current position of the vehicle in detecting one or more vehicles located in or near the future path of the vehicle.

[0036] In some embodiments, the method may further include causing at least one electronic or mechanical unit of the vehicle to change at least one motion parameter of the vehicle based on the position of one or more vehicles determined to be in or near the future path of the vehicle.

[0037] In some embodiments, the method may further include triggering a perception alert to indicate to a user that the one or more vehicles are determined to be in or near the future path of the vehicle. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] The accompanying drawings, which are incorporated in and constitute a part of this disclosure, illustrate various disclosed embodiments. In the drawings:

[0039] Figure 1 is a block diagram representation of a system consistent with the disclosed embodiments.

[0040] Figure 2A is a diagrammatic side view representation of an exemplary vehicle including a system consistent with the disclosed embodiments.

[0041] Figure 2B is consistent with the disclosed embodiments Figure 2A Diagrammatic top view representation of the vehicle and systems shown.

[0042] Figure 2C is a diagrammatic top view representation of another embodiment of a vehicle including a system consistent with the disclosed embodiments.

[0043] Figure 2D is a diagrammatic top view representation of yet another embodiment of a vehicle including a system consistent with the disclosed embodiments.

[0044] Figure 2E is a diagrammatic top view representation of yet another embodiment of a vehicle including a system consistent with the disclosed embodiments.

[0045] Figure 2F is a pictorial representation of an exemplary vehicle control system consistent with the disclosed embodiments.

[0046] Figure 3A is a pictorial representation of the interior of a vehicle including a rearview mirror and a user interface for a vehicle imaging system, consistent with the disclosed embodiments.

[0047] Figure 3B is an illustration of an example of a camera mount configured to be positioned forward of a rearview mirror and against a vehicle windshield, consistent with the disclosed embodiments.

[0048] Figure 3C is consistent with the disclosed embodiments from different perspectives Figure 3B Instructions for the camera mount shown.

[0049] Figure 3D is an illustration of an example of a camera mount configured to be positioned forward of a rearview mirror and against a vehicle windshield, consistent with the disclosed embodiments.

[0050] Figure 4 is a flowchart illustration of a method of processing images to provide a trained system consistent with the disclosed embodiments.

[0051] Figures 5A-5C is a graphical illustration of features of a method of processing images to provide a trained system consistent with the disclosed embodiments.

[0052] Figure 6 is a graphical illustration of certain aspects of a method of estimating a future path ahead of a current position of a vehicle consistent with the disclosed embodiments.

[0053] Figure 7 is a flowchart illustration of a method of estimating a future path ahead of a current position of a vehicle according to an example of the presently disclosed subject matter.

[0054] Figure 8A An image showing the environment in front of a vehicle navigating a road consistent with some disclosed embodiments.

[0055] Figure 8B An image showing the environment in front of a vehicle navigating a road consistent with some disclosed embodiments.

[0056] Figure 9 Shown are images provided to the system during a training phase consistent with some disclosed embodiments.

[0057] Figure 10 Shown are images provided to the system during a training phase consistent with some disclosed embodiments.

[0058] Figure 11A and Figure 11B are pictorial illustrations of certain aspects of a training system consistent with some disclosed embodiments.

[0059] Figures 12A-12D An image including an estimated future path is shown consistent with disclosed embodiments.

[0060] Figure 13 An input image is shown with virtual lane markings added to mark a highway exit, consistent with the disclosed embodiments. DETAILED DESCRIPTION

[0061] Before discussing in detail examples of processing images of the environment ahead of a vehicle navigating a road for use in training a system (e.g., a neural network or deep learning system) to estimate features of the vehicle's future path based on the images, or using a trained system to process images of the environment ahead of a vehicle navigating a road to estimate features of the vehicle's future path, a description is provided of various possible implementations and configurations of vehicle-mountable systems that can be used to perform and implement methods according to examples of the presently disclosed subject matter. In some embodiments, various examples of vehicle-mountable systems can be installed in a vehicle and can be operated while the vehicle is in motion. In some embodiments, the vehicle-mountable systems can implement methods according to examples of the presently disclosed subject matter.

[0062] Now for reference Figure 11 is a block diagram representation of a system consistent with the disclosed embodiments. Depending on the requirements of a particular implementation, system 100 may include various components. In some examples, system 100 may include a processing unit 110, an image acquisition unit 120, and one or more memory units 140, 150. Processing unit 110 may include one or more processing devices. In some embodiments, processing unit 110 may include an application processor 180, an image processor 190, or any other suitable processing device. Similarly, depending on the requirements of a particular application, image acquisition unit 120 may include any number of image acquisition devices and components. In some embodiments, image acquisition unit 120 may include one or more image capture devices (e.g., cameras) (e.g., image capture device 122, image capture device 124, and image capture device 126). In some embodiments, system 100 may also include a data interface 128 that communicatively connects processing unit 110 to image acquisition device 120. For example, data interface 128 may include any one or more wired and / or wireless links for transmitting image data captured by image acquisition device 120 to processing unit 110.

[0063] Both the application processor 180 and the image processor 190 may include various types of processing devices. For example, one or both of the application processor 180 and the image processor 190 may include one or more microprocessors, preprocessors (e.g., image preprocessors), graphics processors, central processing units (CPUs), support circuits, digital signal processors, integrated circuits, memories, or any other type of device suitable for running applications and image processing and analysis. In some embodiments, the application processor 180 and / or the image processor 190 may include any type of single-core or multi-core processor, mobile device microcontroller, central processing unit, etc. Various processing devices may be used (including, for example, those available from, for example, processors from various manufacturers), and can include various architectures (e.g., x86 processors, wait).

[0064] In some embodiments, the application processor 180 and / or the image processor 190 may include a processor available from Any of the EyeQ series processor chips. These processor designs include multiple processing units with local memory and instruction sets. These processors may include video inputs for receiving image data from multiple image sensors and may also include video output capabilities. In one example, Using 90nm-micron technology operating at 332Mhz. Architecture features two floating point hyperthreaded 32-bit RTSC CPUs cores), five Visual Compute Engines (VCEs), and three vector microcode processors Denali 64-bit mobile DDR controller, 128-bit internal sound energy interconnect, dual 16-bit video input and 18-bit video output controllers, 16-channel DMA and several peripherals. MIPS34K CPU manages five VCEs, three VMP.TM. and DMA, second MIPS34K CPU and multi-channel DMA and other peripherals. Five VCEs, three And MIPS34K CPU can perform the large amount of visual calculations required for multi-function batch applications. In another example, as a third generation processor and more Six times more powerful Can be used in the disclosed examples. In yet another example, (fourth generation processors) may be used in the disclosed examples.

[0065] Although Figure 1 While two separate processing devices are described as being included in processing unit 110, more or fewer processing devices may be used. For example, in some examples, a single processing device may be used to perform the tasks of application processor 180 and image processor 190. In other embodiments, these tasks may be performed by more than two processing devices.

[0066] The processing unit 110 may include various types of devices. For example, the processing unit 110 may include various devices (e.g., a controller, an image preprocessor, a central processing unit (CPU), support circuits, a digital signal processor, an integrated circuit, memory, or any other type of device for image processing and analysis). The image preprocessor may include a video processor for capturing, digitizing, and processing images from an image sensor. The CPU may include any number of microcontrollers or microprocessors. The support circuits may be any number of circuits known in the art, including cache circuits, power supply circuits, clock circuits, and input-output circuits. The memory may store software that, when executed by the processor, controls the operation of the system. The memory may include a database and image processing software. The memory may include any number of random access memories, read-only memories, flash memory, disk drives, optical storage, removable storage, and other types of storage. In one embodiment, the memory may be separate from the processing unit 110. In another embodiment, the memory may be integrated into the processing unit 110.

[0067] Each memory 140, 150 may include software instructions that, when executed by a processor (e.g., application processor 180 and / or image processor 190), may control the operation of various aspects of system 100. These memory units may include various databases and image processing software. The memory units may include random access memory, read-only memory, flash memory, disk drives, optical storage, tape storage, removable storage, and / or any other type of storage. In some examples, the memory units 140, 150 may be separate from the application processor 180 and / or image processor 190. In other embodiments, these memory units may be integrated into the application processor 180 and / or image processor 190.

[0068] In some embodiments, the system may include a location sensor 130. Location sensor 130 may include any type of device suitable for determining a location associated with at least one component of system 100. In some embodiments, location sensor 130 may include a GPS receiver. These receivers may determine user location and velocity by processing signals broadcast by global positioning system satellites. Location information from location sensor 130 may be made available to application processor 180 and / or image processor 190.

[0069] In some embodiments, the system 100 can be operably connected to various systems, devices, and units onboard a vehicle in which the system 100 can be installed, and the system 100 can communicate with the vehicle's systems via any suitable interface (e.g., a communication bus). Examples of vehicle systems with which the system 100 can cooperate include: a throttle system, a braking system, and a steering system.

[0070] In some embodiments, the system 100 may include a user interface 170. The user interface 170 may include any device suitable for providing information to or receiving input from one or more users of the system 100. In some embodiments, the user interface 170 may include user input devices, including, for example, a touch screen, a microphone, a keyboard, a pointing device, a track wheel, a camera, a joystick, buttons, etc. Through these input devices, the user may be able to provide information input or commands to the system 100 by entering instructions or information using buttons, a pointer, or eye tracking capabilities, or by any other suitable technique for conveying information to the system 100, providing voice commands, selecting menu options on a screen. Information may be provided to the user by the system 100 through the user interface 170 in a similar manner.

[0071] In some embodiments, system 100 may include a map database 160. Map database 160 may comprise any type of database for storing digital map data. In some examples, map database 160 may include data relating to the locations of various items (including roads, water features, geographic features, points of interest, etc.) in a reference coordinate system. Map database 160 may store not only the locations of these items, but also descriptors associated with those items (including, for example, names associated with any stored features). In some embodiments, map database 160 may be physically located with other components of system 100. Alternatively or additionally, map database 160 or portions thereof may be remotely located relative to other components of system 100 (e.g., processing unit 110). In these embodiments, information from map database 160 may be downloaded via a wired or wireless data connection to a network (e.g., via a cellular network and / or the internet, etc.).

[0072] Image capture devices 122, 124, and 126 may each comprise any type of device suitable for capturing at least one image from an environment. Furthermore, any number of image capture devices may be used to acquire images for input to the image processor. Some examples of the presently disclosed subject matter may include or be implemented with only a single image capture device, while other examples may include or be implemented with two, three, or even four or more image capture devices. Figure 2B-2E Image capture devices 122, 124, and 126 are further described.

[0073] It should be understood that the system 100 may include or be operatively associated with other types of sensors, including, for example, acoustic sensors, RF sensors (e.g., radar transducers), and LIDAR sensors. These sensors may be used independently of or in conjunction with the image acquisition device 120. For example, data from a radar system (not shown) may be used to validate processed information received from processed images acquired by the image acquisition device 120, for example, to filter a specific false alarm rate originating from the processed images acquired by the image acquisition device 120.

[0074] System 100 or its individual components may be incorporated into various different platforms. In some embodiments, system 100 may be included on vehicle 200, such as Figure 2A For example, vehicle 200 may be equipped with processing unit 110 and any other components of system 100, as described above with respect to Figure 1 While in some embodiments, the vehicle 200 may be equipped with only a single image capture device (e.g., a camera), in other embodiments (e.g., in combination with Figure 2B-2EIn the embodiment discussed above, multiple image capture devices may be used. For example, one of the image capture devices 122 and 124 of the vehicle 200 may be used. Figure 2A Shown may be part of an ADAS (Advanced Driver Assistance Systems) imaging suite.

[0075] The image capture device included as part of the image acquisition unit 120 on the vehicle 200 may be located at any suitable location. Figure 2A-2E as well as Figure 3A-3C As shown, the image capture device 122 can be located near the rearview mirror. This location can provide a line of sight similar to the line of sight of the driver of the vehicle 200, which can help determine what is visible and what is not visible to the driver.

[0076] Other locations for the image capture device used in image acquisition unit 120 may also be used. For example, image capture device 124 may be located on or in the bumper of vehicle 200. This location may be particularly suitable for image capture devices with a wide field of view. The line of sight of a bumper-located image capture device may differ from the driver's line of sight. Image capture devices (e.g., image capture devices 122, 124, and 126) may also be located in other locations. For example, the image capture device may be located on or in one or both of the sideview mirrors of vehicle 200, on the roof of vehicle 200, on the hood of vehicle 200, on the trunk of vehicle 200, on each side of vehicle 200, mounted on, in front of, or in front of any window of vehicle 200, mounted in or near a light figure on the front and / or rear of vehicle 200, etc. Image capture unit 120, or an image capture device that is one of multiple image capture devices used in image capture unit 120, may have a field of view (FOV) that differs from the driver's field of view (FOV) of the vehicle and may not always see the same objects. In one example, the FOV of the image acquisition unit 120 may extend beyond the FOV of a typical driver and may therefore image objects outside the driver's FOV. In another example, the FOV of the image acquisition unit 120 is a portion of the driver's FOV. In some embodiments, the FOV of the image acquisition unit 120 corresponds to a segment covering the road in front of the vehicle and possibly also the area surrounding the road.

[0077] In addition to the image capture device, the vehicle 200 may also include various other components of the system 100. For example, a processing unit 110 may be included on the vehicle 200, either integrated with or separate from the vehicle's engine control unit (ECU). The vehicle 200 may also be equipped with a position sensor 130 (e.g., a GPS receiver), and may also include a map database 160 and memory units 140 and 150.

[0078] Figure 2A is a diagrammatic side view representation of a vehicle imaging system according to an example of the presently disclosed subject matter. Figure 2B yes Figure 2A A diagrammatic top view illustration of the example shown. Figure 2B As shown, the disclosed example can include a vehicle 200 including a system 100 in its body, the system 100 having: a first image capture device 122 located near a rearview mirror and / or near a driver of the vehicle 200; a second image capture device 124 located on or in a bumper area (e.g., one of the bumper areas 210) of the vehicle 200; and a processing unit 110.

[0079] like Figure 2C As shown, image capture devices 122 and 124 may both be located near a rearview mirror and / or near the driver of vehicle 200. Additionally, although Figure 2B and Figure 2C Two image capture devices 122 and 124 are shown in FIG, but it should be understood that other embodiments may include more than two image capture devices. Figure 2D and Figure 2E In the illustrated embodiment, the system 100 in the vehicle 200 includes a first image capture device 122 , a second image capture device 124 , and a third image capture device 126 .

[0080] like Figure 2D As shown, image capture device 122 may be located near a rearview mirror of vehicle 200 and / or near the driver, and image capture devices 124 and 126 may be located on or in a bumper area (e.g., one of bumper areas 210) of vehicle 200. Figure 2E As shown, image capture devices 122, 124, and 126 may be located near the rearview mirror and / or near the driver's seat of vehicle 200. The disclosed examples are not limited to any particular number and configuration of image capture devices, and the image capture devices may be located in any suitable location in and / or on vehicle 200.

[0081] It should also be understood that the disclosed embodiments are not limited to a particular type of vehicle 200 and may be applicable to all types of vehicles, including automobiles, trucks, trailers, motorcycles, bicycles, self-balancing transporters, and other types of vehicles.

[0082] First image capture device 122 can include any suitable type of image capture device. Image capture device 122 can include an optical axis. In one example, image capture device 122 can include an Aptina M9V024W VGA sensor with a global shutter. In another example, a rolling shutter sensor can be used. Image acquisition unit 120 and any image capture devices implemented as part of image acquisition unit 120 can have any desired image resolution. For example, image capture device 122 can provide a resolution of 1280x960 pixels and can include a rolling shutter.

[0083] Image acquisition unit 120 and any image capture device implemented as part of image acquisition unit 120 may include various optical elements. In some embodiments, one or more lenses may be included, for example, to provide a desired focal length and field of view for image acquisition unit 120 and for any image capture device implemented as part of image acquisition unit 120. In some examples, for example, an image capture device implemented as part of image acquisition unit 120 may include or be associated with any optical element (e.g., a 6 mm lens or a 12 mm lens). In some examples, image capture device 122 may be configured to capture an image with a desired field of view (FOV) 202, such as Figure 2D shown.

[0084] The first image capture device 122 may have a scan rate associated with the acquisition of each of the first series of image scan lines. The scan rate may refer to the rate at which the image sensor may acquire image data associated with each pixel included in a particular scan line.

[0085] Figure 2F is a diagrammatic representation of an example vehicle control system according to the presently disclosed subject matter. Figure 2F As shown, vehicle 200 may include a throttle system 220, a brake system 230, and a steering system 240. System 100 may provide input (e.g., control signals) to one or more of throttle system 220, brake system 230, and steering system 240 via one or more data links (e.g., any one or more wired and / or wireless links for transmitting data). For example, based on analysis of images captured by image capture devices 122, 124, and / or 126, system 100 may provide control signals to one or more of throttle system 220, brake system 230, and steering system 240 to navigate vehicle 200 (e.g., by initiating acceleration, steering, lane changes, etc.). In addition, system 100 may receive input from one or more of throttle system 220, brake system 230, and steering system 240 indicating an operating condition of vehicle 200 (e.g., speed, whether vehicle 200 is braking and / or turning, etc.).

[0086] like Figure 3A As shown, the vehicle 200 may also include a user interface 170 for interacting with the driver or passengers of the vehicle 200. For example, the user interface 170 in a vehicle application may include a touch screen 320, a handle 330, buttons 340, and a microphone 350. The driver or passengers of the vehicle 200 may also use a handle (e.g., located on or near the steering column of the vehicle 200, including, for example, a turn signal handle), buttons (e.g., located on the steering wheel of the vehicle 200), etc. to interact with the system 100. In some embodiments, the microphone 350 may be located adjacent to the rearview mirror 310. Similarly, in some embodiments, the image capture device 122 may be located near the rearview mirror 310. In some embodiments, the user interface 170 may also include one or more speakers 360 (e.g., speakers of the vehicle's audio system). For example, the system 100 may provide various notifications (e.g., warnings) via the speakers 360.

[0087] Figure 3B-3D is an illustration of an exemplary camera mount 370 configured to be positioned in front of a rearview mirror (e.g., rearview mirror 310) and against a vehicle windshield, consistent with the disclosed embodiments. Figure 3B As shown, camera mount 370 may include image capture devices 122, 124, and 126. Image capture devices 124 and 126 may be located in front of glare shield 380, which may be flush against the vehicle windshield and include a film and / or a composite of anti-reflective materials. For example, glare shield 380 may be positioned such that it is aligned against the vehicle windshield with a matching bevel. In some embodiments, each of image capture devices 122, 124, and 126 may be located behind glare shield 380, such as, for example, Figure 3D The disclosed embodiments are not limited to any particular configuration of the image capture devices 122 , 124 , and 126 , the camera mount 370 , and the glare barrier 380 . Figure 3C From the front perspective Figure 3B An illustration of the camera mount 370 is shown.

[0088] Those skilled in the art who have the benefit of this disclosure will appreciate that numerous variations and / or modifications may be made to the aforementioned disclosed embodiments. For example, not all components are essential for the operation of system 100. Furthermore, any component may be located in any appropriate portion of system 100, and the components may be rearranged into various configurations while providing the functionality of the disclosed embodiments. Therefore, the aforementioned configurations are examples, and regardless of the configuration, system 100 may provide a wide range of functionality to analyze the surroundings of vehicle 200 and, in response to that analysis, navigate and / or otherwise control and / or operate vehicle 200. Navigation, control, and / or operation of vehicle 200 may include enabling and / or disabling various features, components, devices, modes, systems, and / or subsystems associated with vehicle 200 (directly or via an intermediate controller (e.g., the controllers mentioned above)). Navigation, control, and / or operation may alternatively or additionally include interacting with a user, driver, passenger, passerby, and / or other vehicles or users, which may be located inside or outside vehicle 200, for example, by providing visual, audio, tactile, and / or other sensory alerts and / or indications.

[0089] As discussed in further detail below, and consistent with the various disclosed embodiments, system 100 can provide various features related to autonomous driving, semi-autonomous driving, and / or driver assistance technologies. For example, system 100 can analyze image data, location data (e.g., GPS location information), map data, speed data, and / or data from sensors included in vehicle 200. System 100 can collect data for analysis from, for example, image acquisition unit 120, location sensor 130, and other sensors. Furthermore, system 100 can analyze the collected data to determine whether vehicle 200 should take a specific action, and then automatically take the determined action without human intervention. It should be understood that in some cases, the actions automatically taken by the vehicle are under human supervision, and the ability for human intervention to adjust, abort, or override the machine's actions is enabled in certain circumstances or at all times. For example, when vehicle 200 is navigating without human intervention, system 100 can automatically control the braking, acceleration, and / or steering of vehicle 200 (e.g., by sending control signals to one or more of throttle system 220, braking system 230, and steering system 240). In addition, the system 100 can analyze the collected data and issue warnings, instructions, recommendations, alerts, or instructions to the driver, passengers, users, or other persons inside or outside the vehicle (or to other vehicles) based on the analysis of the collected data. Additional details regarding various embodiments provided by the system 100 are provided below.

[0090] Multi-imaging system

[0091] As described above, system 100 can provide driver assistance functionality or semi-autonomous or fully autonomous driving functionality using a single-camera system or a multi-camera system. A multi-camera system can use one or more cameras facing the vehicle's forward direction. In other embodiments, the multi-camera system can include one or more cameras facing the side or rear of the vehicle. In one embodiment, for example, system 100 can use a dual-camera imaging system, wherein a first camera and a second camera (e.g., image capture devices 122 and 124) can be located at the front or side of a vehicle (e.g., vehicle 200). The first camera can have a field of view that is larger than, smaller than, or partially overlaps with the field of view of the second camera. Furthermore, the first camera can be connected to a first image processor to perform monocular image analysis of the image provided by the first camera, and the second camera can be connected to a second image processor to perform monocular image analysis of the image provided by the second camera. The outputs (e.g., processed information) of the first and second image processors can be combined. In some embodiments, the second image processor can receive images from both the first and second cameras to perform stereo analysis. In another embodiment, system 100 can use a three-camera imaging system, wherein each camera has a different field of view. The system can therefore make decisions based on information derived from objects located at varying distances to the front and sides of the vehicle. References to monocular image analysis may refer to instances where image analysis is performed based on images captured from a single viewpoint (e.g., from a single camera). Stereoscopic image analysis may refer to instances where image analysis is performed based on two or more images captured with one or more variations in image capture parameters. For example, captured images suitable for performing stereoscopic image analysis may include images captured from two or more different positions, from different fields of view, using different focal lengths, along with disparity information, and the like.

[0092] For example, in one embodiment, system 100 may implement a three-camera configuration using image capture devices 122-126. In this configuration, image capture device 122 may provide a narrow field of view (e.g., 34 degrees, or other values selected from a range of approximately 20 to 45 degrees), image capture device 124 may provide a wide field of view (e.g., 150 degrees, or other values selected from a range of approximately 100 to approximately 180 degrees), and image capture device 126 may provide a medium field of view (e.g., 46 degrees, or other values selected from a range of approximately 35 to approximately 60 degrees). In some embodiments, image capture device 126 may serve as the primary or master camera. Image capture devices 122-126 may be located behind rearview mirror 310 and positioned substantially side-by-side (e.g., 6 cm apart). Furthermore, in some embodiments, as described above, one or more of image capture devices 122-126 may be mounted behind glare shield 380 flush with the windshield of vehicle 200. This shielding may operate to minimize the effect of any reflections from the interior of the vehicle on the image capture devices 122 - 126 .

[0093] In another embodiment, as above combined Figure 3B and Figure 3C As discussed, a wide field of view camera (e.g., image capture device 124 in the above example) can be mounted lower than the narrow field of view camera and the main field of view camera (e.g., image devices 122 and 126 in the above example). This configuration can provide a clear line of sight from the wide field of view camera. To reduce reflections, the camera can be mounted closer to the windshield of vehicle 200, and a polarizer can be included on the camera to suppress reflected light.

[0094] A three-camera system can provide specific performance characteristics. For example, some embodiments may include the ability to verify the detection of an object by one camera based on the detection results from another camera. In the above three-camera configuration, the processing unit 110 may include, for example, three processing devices (e.g., three EyeQ series processor chips, as described above), where each processing device is dedicated to processing images captured by one or more of the image capture devices 122-126.

[0095] In a three-camera system, the first processing device can receive images from both the main camera and the narrow field of view camera and perform processing for the narrow FOV camera, or even a cropped FOV of the camera. In some embodiments, according to examples of the presently disclosed subject matter, the first processing device can be configured to use a trained system to estimate the future path ahead of the vehicle's current position. In some embodiments, the trained system can include a network (e.g., a neural network). In some other embodiments, the trained system can include a deep learning system using, for example, a machine learning algorithm.

[0096] The first processing device may be further adapted to perform image processing tasks, which may be intended to detect other vehicles, pedestrians, lane markings, traffic signs, traffic lights, and other road objects, for example. In addition, the first processing device may calculate the differences in pixels between the images from the main camera and the narrow camera and create a 3D reconstruction of the environment of the vehicle 200. The first processing device may then combine the 3D reconstruction with 3D map data (e.g., a depth map) or with 3D information calculated based on information from another camera. In some embodiments, according to an example of the presently disclosed subject matter, the first processing device may be configured to use a trained system on depth information (e.g., 3D map data) to estimate a future path ahead of the vehicle's current position. In this implementation, a system (e.g., a neural network, a deep learning system, etc.) may be trained on depth information (e.g., 3D map data).

[0097] The second processing device can receive images from the primary camera and can be configured to perform visual processing to detect other vehicles, pedestrians, lane markings, traffic signs, traffic lights, and other road objects. Additionally, the second processing device can calculate camera displacement and, based on the displacement, calculate pixel differences between consecutive images and create a 3D reconstruction of the scene (e.g., structure from motion). The second processing device can send the structure from motion-based 3D reconstruction to the first processing device to be combined with the stereo 3D image or with depth information obtained through stereo processing.

[0098] The third processing device can receive images from the wide FOV camera and process the images to detect vehicles, pedestrians, lane markings, traffic signs, traffic lights, and other road objects. The third processing device can execute additional processing instructions to analyze the images to identify moving objects in the images (e.g., vehicles changing lanes, pedestrians, etc.).

[0099] In some embodiments, enabling independent capture and processing of image-based information streams may provide an opportunity to provide redundancy in the system. This redundancy may include, for example, using a first image capture device and images processed from that device to verify and / or supplement information obtained by capturing and processing image information from at least a second image capture device.

[0100] In some embodiments, system 100 may utilize two image capture devices (e.g., image capture devices 122 and 124) in providing navigation assistance for vehicle 200, and utilize a third image capture device (e.g., image capture device 126) to provide redundancy and verify analysis of data received from the other two image capture devices. For example, in this configuration, image capture devices 122 and 124 may provide images for system 100 to perform stereo analysis for navigating vehicle 200, while image capture device 126 may provide images for system 100 to perform monocular analysis to provide redundancy and verification based on information captured from image capture device 122 and / or image capture device 124. In other words, image capture device 126 (and corresponding processing device) may be considered to provide a redundant subsystem for providing a check on analysis derived from image capture devices 122 and 124 (e.g., to provide an automatic emergency braking (AFB) system).

[0101] Those skilled in the art will appreciate that the above camera configurations, camera placement, number of cameras, camera positions, etc. are merely examples. These components, as well as other components described with respect to the overall system, can be assembled and used in a variety of different configurations without departing from the scope of the disclosed embodiments. Further details regarding the use of a multi-camera system to provide driver assistance and / or autonomous vehicle functionality are as follows.

[0102] Those skilled in the art who have the benefit of this disclosure will appreciate that a large number of variations and / or modifications may be made to the foregoing disclosed examples. For example, not all components are necessary for the operation of the system 100. Furthermore, any component may be located in any appropriate portion of the system 100, and the components may be rearranged into various configurations while still providing the functionality of the disclosed embodiments. Thus, the foregoing configurations are examples, and regardless of the configurations described above, the system 100 may provide a wide range of functionality to analyze the surroundings of the vehicle 200 and navigate the vehicle 200, or to alert the user of the vehicle in response to the analysis.

[0103] As discussed in further detail below, and according to examples of the presently disclosed subject matter, system 100 can provide various features related to autonomous driving, semi-autonomous driving, and / or driver assistance technologies. For example, system 100 can analyze image data, location data (e.g., GPS location information), map data, speed data, and / or data from sensors included in vehicle 200. System 100 can collect data for analysis from, for example, image acquisition unit 120, location sensor 130, and other sensors. Furthermore, system 100 can analyze the collected data to determine whether vehicle 200 should take a specific action and then automatically take the determined action without human intervention, or it can provide a warning, alert, or instruction to the driver indicating that a specific action is needed. Automatic actions can be performed under human supervision and can be subject to human intervention and / or override. For example, when vehicle 200 is navigating without human intervention, system 100 can automatically control the braking, acceleration, and / or steering of vehicle 200 (e.g., by sending control signals to one or more of throttle system 220, braking system 230, and steering system 240). Additionally, the system 100 may analyze the collected data and issue warnings and / or alerts to the vehicle occupants based on the analysis of the collected data.

[0104] Now refer to Figure 4 , Figure 4 The present invention is a flowchart illustrating a method for processing images to provide a trained system capable of estimating a future path ahead of a vehicle's current location based on images captured at the current location, according to an example of the presently disclosed subject matter. The method for processing images may include obtaining a first plurality of training images, each of the first plurality of training images being an image of an environment ahead of a vehicle navigating a road (block 410). In some embodiments, the first plurality of images is not limited to images of the environment ahead of the vehicle navigating a road and may include images of other sides of the vehicle navigating a road (e.g., the environment to the sides of the vehicle and / or in a rearward direction).

[0105] For each of the first plurality of training images, a pre-stored path of a vehicle preceding the corresponding current position of the vehicle may be obtained (block 420). Figures 5A-5C , Figures 5A-5C is a graphical illustration of features of a method of processing images to provide a trained system capable of estimating a future path ahead of a vehicle's current location based on images captured at the current location, according to an example of the presently disclosed subject matter. Figure 5A As can be seen in FIG, vehicle 510 is entering a section of road 520. Vehicle 510 includes a camera (not shown) that captures images. Figure 5AIn FIG, an image is shown by a cone 530 representing the FOV of a camera mounted in a vehicle 510. The image depicts any object in the camera's FOV. The image typically includes road objects (e.g., road signs, lane markings, curbs, other vehicles, etc.). In addition, at least some of the images used in the method of processing images to provide a trained system capable of estimating the future path ahead of the vehicle's current position based on images captured at the current position include other arbitrary objects (e.g., structures and trees at the side of the road, etc.).

[0106] exist Figures 5A-5C In the example shown, the images captured by the camera onboard the vehicle 510 are images of the environment in front of the vehicle 510 as it navigates a road 520 .

[0107] exist Figure 5B and Figure 5C As can be seen in FIG, vehicle 510 travels along (a segment of) road 520 and its path is recorded. The path of vehicle 510 is marked with pins 541-547. The path of vehicle 510 along road 520 is recorded from position 541 to position 547. Thus, the future path of the vehicle from point 541 down road 520 is available. It should be understood that the specific location of vehicle 510 and the 2D or 3D shape of the road are arbitrary and that examples of the presently disclosed subject matter are applicable to various locations on any road. Figure 5C As shown, as vehicle 510 travels along road 520, the camera captures multiple images represented by cones 551-554. One or more images may be captured for each of locations 541-547 or only for some of the recorded locations, however, for convenience, the images are not shown in FIG. Figure 5C , images are shown only for a subset of the recorded positions of vehicle 510 along road 520 .

[0108] Continue to Figure 4 As described above, a system (e.g., a neural network, a deep learning system, etc.) can be trained to provide a future path of a vehicle ahead of a corresponding current position of the vehicle for navigating a road given an image (block 430). Training the system (block 430) can include: providing a first plurality of training images as input to the trained system (block 440); at each iteration of training, calculating a loss function based on the corresponding interim future path estimated by the current state of the weights of the trained system and the corresponding pre-stored path (block 450); and updating the weights of the trained system according to the result of the loss function (block 460).

[0109] Typically, a very large number of images are provided to the trained system during a training phase, and for each image, a pre-stored path of a vehicle preceding the vehicle's corresponding current position is provided. The pre-stored path can be obtained by recording the vehicle's future position along the road the vehicle is traveling while capturing images. In another example, the pre-stored path can be generated manually or using image processing that visually or algorithmically identifies various objects in or near the road that indicate the vehicle's position on the road. The vehicle's position on the road can be the vehicle's actual position on the road during the session when the images were captured, or it can be an estimated or artificially generated position. For example, in one example, images can be captured every few meters or even tens of meters, and the vehicle's future path can be summarized by a technician based on lane markings or any other objects visually identified by the technician in each image. In the lane marking example, the technician can summarize the future path based on an image where a lane marking appears at a specific (e.g., predetermined) offset from the lane marking (e.g., in the middle of a lane separated by lane markings on either side).

[0110] Figure 9 and Figure 10 1 shows images that may be provided to a machine learning process during a training phase, for example, using a trained system (e.g., a neural network, a deep learning system, etc.), consistent with some disclosed embodiments. Figure 10 Additionally shown are a point 1010 on a lane marking 1012 detected by image processing or a technician, and a point 1020 on the center of the lane.

[0111] Now additionally refer to Figure 8A , Figure 8A An image 810 of the environment in front of a vehicle navigating a road according to an example of the presently disclosed subject matter and a pre-stored path 812 recorded with respect to the image using the vehicle's ego motion are shown; and reference is made to Figure 8B , Figure 8B An image 820 of the environment in front of a vehicle navigating a road according to an example of the presently disclosed subject matter is shown, the image including marked lane markings 822 marked by a technician or by a computer vision algorithm. Figure 8B The image shown generates and pre-stores a pre-stored path based on the marked lane markings. A vehicle ego-motion trajectory for a given image can be determined by processing subsequent images (e.g., subsequent images captured as the vehicle continues to move ahead of the road (e.g., ahead of the current position)) that can be used to define a path for the vehicle ahead of the vehicle's current position (the position where the image was captured).

[0112] As described above, at each iteration of training, a loss function is calculated based on the corresponding temporary future path estimated by the current state of the weights and the corresponding pre-stored path (block 450). The weights of the trained system can be updated according to the results of the loss function (block 460). Figure 11A and Figure 11B , Figure 11A and Figure 11B Provides a graphical illustration of certain aspects of training according to examples of the presently disclosed subject matter. Figure 11A In , the iterations of training are shown. Figure 11A , a pre-stored path 1112 is generated based on a detection (e.g., by a technician) of lane markings on the road in front of the location where the image 1110 was captured. The trained system calculates a temporary future path 1114 and calculates a loss function based on the corresponding temporary future path 1114 estimated by the current state of the weights and the corresponding pre-stored path 1112. According to an example of the presently disclosed subject matter, the loss function uses a top-view ("world") representation of the pre-stored path 1112 and the temporary future path 1114 (e.g., using camera focal length, camera height, and dynamic field of view), and an absolute loss is calculated. The loss function can be configured to impose a penalty on an error of a few meters in the real world (which focuses on a score for far errors). In some embodiments, according to an example of the presently disclosed subject matter, the marked objects (e.g., lane markings manually marked by a technician) can also include virtual objects (e.g., merging points, areas where lane markings disappear, unmarked highway exits or merges, etc.), such as, for example Figure 13 As shown, a virtual lane marker 1310 is added to mark a highway exit.

[0113] exist Figure 11B In FIG, a pre-stored path 1122 is generated based on the vehicle's ego-motion and its subsequent path along a road ahead of (in this case, in) the location where the image 1120 was captured. According to examples of the presently disclosed subject matter, for the pre-stored path data based on ego-motion, an optimized offset between the pre-stored path 1122 and the provisional future path 1124 provided by the trained system can be determined, and a loss function can be calculated after correction by the optimized offset.

[0114] According to examples of the presently disclosed subject matter, training of the system can be performed until a stopping condition is met. In some embodiments, the stopping condition can be a specific number of iterations. For example, the first plurality of training images can include a relatively high number of images of environments that are relatively sparsely represented on the road (e.g., images of environments that include curved roads). In another example, the first plurality of training images can include a relatively high number of images of environments that include lane splits, lane merges, highway exits, highway entrances, and / or merging points; in yet another example. In yet another example, the first plurality of training images can include a relatively high number of images of environments that include poor or no lane markings, Botts points, and / or shadows on the road in front of the vehicle.

[0115] According to another aspect of the presently disclosed subject matter, a system and method for estimating a future path ahead of a current position of a vehicle is provided. Figure 7 , Figure 7 7 is a flowchart illustrating a method for estimating a future path ahead of a vehicle's current location, according to an example of the presently disclosed subject matter. The method may be implemented by a processor. According to an example of the presently disclosed subject matter, the method for estimating a future path ahead of a vehicle's current location may include obtaining an image of an environment ahead of a current arbitrary location of a vehicle navigating a road (block 710). A system trained to estimate a future path based on a first plurality of images of the environment ahead of the vehicle navigating a road may be obtained (block 720). In some embodiments, the trained system may include a network (e.g., a neural network). In other embodiments, the trained system may be a deep learning system using, for example, a machine learning algorithm. The trained system may be applied to an image of the environment ahead of the vehicle's current arbitrary location (block 730). The trained system may provide an estimated future path ahead of the vehicle ahead of the current arbitrary location (block 740).

[0116] Now refer to Figure 6 , Figure 6 is a graphical illustration of certain aspects of a method of estimating a future path ahead of a vehicle's current position according to an example of the presently disclosed subject matter. Figure 6 As shown, vehicle 610 is entering a section of road 620. Road 620 is any road, and images from road 620 may or may not have been used in the training of a system (e.g., a neural network, a deep learning system, etc.). Vehicle 610 includes a camera (not shown) that captures images. The images captured by the camera onboard vehicle 620 may be cropped or uncropped, or processed in any other way (e.g., down-sampling), and then fed to the trained system. Figure 6In FIG. 6 , an image is shown by a cone 630 representing the FOV of a camera mounted in a vehicle 610. The image depicts any object within the camera's FOV. The image may, but need not, include road objects (e.g., road signs, lane markings, curbs, other vehicles, etc.). The image may also include any other objects (e.g., structures and trees on the side of the road, etc.).

[0117] The trained system can be applied to the image 630 of the environment in front of the current any location of the vehicle 610 and can provide an estimated future path of the vehicle 610 in front of the current any location. Figure 6 , the estimated future paths are represented by pins 641-647. Figures 12A-12D Further shown are images including estimated future paths 1210 - 1240 consistent with disclosed embodiments.

[0118] In some embodiments, the trained system may include a piecewise affine function of the global function. In some embodiments, the global function may include: convolution, max pooling, and / or rectified linear unit (ReLU).

[0119] In some embodiments, the method may further include: utilizing the estimated future path ahead of the current position of the vehicle to control at least one electronic or mechanical unit of the vehicle to change at least one motion parameter of the vehicle. In some embodiments, the method may further include: utilizing the estimated future path ahead of the current position of the vehicle to provide sensory feedback to a driver of the vehicle.

[0120] In some embodiments, the estimated future path of the vehicle ahead of the current position may be further based on identifying one or more predefined objects appearing in the image of the environment using at least one classifier.

[0121] The method may further include utilizing the estimated future path ahead of the vehicle's current position to provide a control point for a steering control function of the vehicle.

[0122] In some embodiments, applying the trained system to images of the environment ahead of the vehicle's current location provides two or more estimated future paths for the vehicle ahead of the current location.

[0123] In some embodiments, the method may further comprise utilizing the estimated future path ahead of the vehicle's current position in estimating a road profile ahead of the vehicle's current position.

[0124] In some embodiments, applying the trained system to an image of the environment in front of the vehicle's current position provides two or more estimated future paths of the vehicle in front of the current position, and may also include estimating a road profile along each of the two or more estimated future paths of the vehicle in front of the current position.

[0125] In some embodiments, the method may further include utilizing an estimated future path ahead of the current position of the vehicle in detecting one or more vehicles located in or near the future path of the vehicle.

[0126] In some embodiments, the method may further include causing at least one electronic or mechanical unit of the vehicle to change at least one motion parameter of the vehicle based on the position of one or more vehicles determined to be in or near the vehicle's future path.

[0127] In some embodiments, the method may further include triggering a perception alert to indicate to a user that the one or more vehicles are determined to be in or near the future path of the vehicle.

[0128] In some embodiments, in addition to processing images of the environment ahead of a vehicle navigating a road for training a system (e.g., a neural network, a deep learning system, etc.) to estimate the vehicle's future path based on the images and / or using the trained system to process images of the environment ahead of the vehicle navigating a road to estimate the vehicle's future path, confidence levels can be provided during the training phase or used during the navigation phase using the trained system. The overall path prediction (HPP) confidence is an output of the trained system (e.g., a neural network), similar to the neural network used for HPP. The concept can generate a classifier that attempts to guess the error of another classifier for the same image. One way to implement this is to use the trained system (e.g., a first neural network) to produce a used output (e.g., the location of a lane, the center of a lane, or the predicted future path) and train another system (e.g., a second neural network) using the same input data (or a subset of that data, or features extracted from that data) to estimate the error of the first trained system for that image (e.g., the absolute mean loss of the first trained system's predictions).

[0129] The foregoing description has been presented for purposes of illustration. It is not intended to be comprehensive and is not limited to the precise forms or embodiments disclosed. Modifications and alterations will be apparent to those skilled in the art from consideration of the specification and practice of the disclosed embodiments. Additionally, while various aspects of the disclosed embodiments are described as being stored in a memory, it will be understood by those skilled in the art that these aspects may also be stored on other types of computer-readable media, such as secondary storage devices (e.g., hard disks or CD ROMs) or other forms of RAM or ROM, USB media, DVDs, Blu-rays, 4K Ultra HD Blu-rays, or other optical drive media.

[0130] Computer programs based on the written description and disclosed methods are within the skill of experienced developers. Various programs or program modules can be created using any technology known to those skilled in the art, or can be designed in conjunction with existing software. For example, program segments or program modules can be designed using or with the .NET Framework, .NET Compact Framework (and related languages such as Visual Basic, C, etc.), Java, C++, Objective-C, HTML, HTML / AJAX combinations, XML, or HTML including Java applets.

[0131] In addition, although illustrative embodiments have been described herein, those skilled in the art will appreciate the scope of any and all embodiments with equivalent elements, modifications, omissions, combinations, adaptations, or alterations (e.g., throughout the various aspects of the various embodiments) based on this disclosure. The limitations in the claims are to be interpreted broadly based on the language employed in the claims and are not limited to the examples described in this specification or during the prosecution of the application. The examples are to be understood as non-exclusive. In addition, the steps of the disclosed methods may be modified in any manner, including by rearranging steps and / or inserting or deleting steps. Therefore, the description and examples are intended to be regarded as illustrative only, with the true scope and spirit being indicated by the full scope of the appended claims and their equivalents.

Claims

1. At least one non-transitory machine-readable storage medium comprising instructions that, when executed by a processor circuit of a computing device, cause the processor circuit to: obtaining an image representing a scene of an environment captured from a current position of the vehicle on a road; applying a neural network model to the image, the neural network model trained to identify a plurality of future trajectories based on the input image, wherein training the neural network model includes applying a loss function after correcting the temporary future paths by an offset between pre-stored paths and the temporary future paths, wherein the pre-stored paths are based on one or more features including lane markings provided in the training image, in, the neural network model, when applied to the image, generates two or more predicted future trajectories of the vehicle ahead of the current position of the vehicle, wherein the neural network model generates the two or more predicted future trajectories of the vehicle based on an offset from a particular point of the road, and wherein the particular point of the road is determined based on one or more features extracted from the image; as well as The two or more predicted future trajectories of the vehicle are utilized to alter operation of the vehicle on the road.

2. The machine-readable storage medium of claim 1, wherein: The instructions further cause the processor circuit to provide a command to the vehicle to change at least one motion parameter of the operation of the vehicle.

3. The machine-readable storage medium of claim 2, wherein: The at least one motion parameter relates to the steering or speed of the vehicle.

4. The machine-readable storage medium of claim 1, wherein: The instructions further cause the processor circuit to: obtain map data indicating characteristics of the road; Wherein, the neural network model further generates the two or more predicted future trajectories of the vehicle based on the characteristics of the road.

5. The machine-readable storage medium of claim 4, wherein: The map data is obtained from a remote system.

6. The machine-readable storage medium of claim 1, wherein: The instructions further cause the processor circuit to: retrieve data indicative of a past state of the vehicle; Wherein, the neural network model further generates the two or more predicted future trajectories of the vehicle based on the past state of the vehicle.

7. The machine-readable storage medium of claim 1, wherein: The training of the neural network model is performed using a plurality of training images.

8. The machine-readable storage medium according to any one of claims 1 to 7, wherein: The image is captured by a camera of the vehicle.

9. A computing device comprising: a memory storing an image representing a scene of an environment captured from a current position of the vehicle on a road; as well as a processor circuit configured to: applying a neural network model to the image, the neural network model trained to identify a plurality of future trajectories based on the input image, wherein training the neural network model includes applying a loss function after correcting the temporary future paths by an offset between pre-stored paths and the temporary future paths, wherein the pre-stored paths are based on one or more features including lane markings provided in the training image, wherein the neural network model, when applied to the image, generates two or more predicted future trajectories of the vehicle ahead of the current position of the vehicle, wherein the neural network model generates the two or more predicted future trajectories of the vehicle based on an offset from a particular point of the road, and wherein the particular point of the road is determined based on one or more features extracted from the image; as well as The two or more predicted future trajectories of the vehicle are utilized to alter operation of the vehicle on the road.

10. The computing device of claim 9, wherein: The processor circuit is further configured to provide a command to the vehicle to change at least one motion parameter of the operation of the vehicle.

11. The computing device of claim 10, wherein: The at least one motion parameter relates to the steering or speed of the vehicle.

12. The computing device of claim 9, wherein: The processor circuit is further configured to: acquiring map data indicating characteristics of the road; Wherein, the neural network model further generates the two or more predicted future trajectories of the vehicle based on the characteristics of the road.

13. The computing device of claim 12, wherein: The map data is obtained from a remote system.

14. The computing device of claim 9, wherein: The processor circuit is further configured to: acquiring data indicative of a past state of the vehicle; Wherein, the neural network model further generates the two or more predicted future trajectories of the vehicle based on the past state of the vehicle.

15. The computing device of claim 9, wherein: The training of the neural network model is performed using a plurality of training images.

16. The computing device according to any one of claims 9 to 15, wherein: The image is captured by a camera of the vehicle.

17. A device comprising: means for providing an image representing a scene of an environment captured from a current position of the vehicle on a road; means for applying a neural network model to the image, the neural network model being trained to identify a plurality of future trajectories based on an input image, wherein training the neural network model comprises applying a loss function after correcting the temporary future paths by an offset between pre-stored paths and the temporary future paths, wherein the pre-stored paths are based on one or more features including lane markings provided in the training image, wherein the neural network model, when applied to the image, generates two or more predicted future trajectories of the vehicle ahead of the current position of the vehicle, wherein the neural network model generates the two or more predicted future trajectories of the vehicle based on an offset from a particular point of the road, and wherein the particular point of the road is determined based on one or more features extracted from the image; as well as Means for using the two or more predicted future trajectories of the vehicle to alter operation of the vehicle on the road.

18. The apparatus of claim 17, further comprising: Means for transmitting a command to the vehicle to change at least one kinetic parameter of the operation of the vehicle.

19. The apparatus of claim 18, further comprising: Means for defining at least one motion parameter related to the steering or speed of said vehicle.

20. The apparatus of any one of claims 17 to 19, further comprising: means for capturing said image.

Images