Data pipelines and deep learning systems for autonomous driving

Abstract

An image captured using a sensor on a vehicle is received and decomposed into a plurality of component images. Each of the plurality of component images is provided as a different input to a different layer of a plurality of layers of an artificial neural network to determine a result. The result of the artificial neural network is used to at least partially autonomously operate the vehicle.

Description

Background Art

[0001] Deep learning systems used to implement autonomous driving typically rely on captured sensor data as input. In traditional learning systems, the captured sensor data can be made compatible with the deep learning system by converting the captured data from the sensor format to a format compatible with the initial input layer of the learning system. The conversion can include compression and downsampling, which can reduce the signal fidelity of the original sensor data. In addition, changing the sensor may require a new conversion process. Therefore, a customized data pipeline is needed that can maximize the signal information from the captured sensor data and provide a higher level of signal information to the deep learning network for deep learning analysis. BRIEF DESCRIPTION OF THE DRAWINGS

[0002] Various embodiments of the invention are disclosed in the following detailed description and accompanying drawings.

[0003] Figure 1 is a flow chart illustrating an embodiment of a process for performing machine learning processing using a deep learning pipeline.

[0004] Figure 2 is a flow chart illustrating an embodiment of a process for performing machine learning processing using a deep learning pipeline.

[0005] Figure 3 is a flow chart illustrating an embodiment of a process for performing a machine learning process using component data.

[0006] Figure 4 is a flow chart illustrating an embodiment of a process for performing machine learning processing using high pass and low pass component data.

[0007] Figure 5 is a flow chart illustrating an embodiment of a process for performing machine learning processing using high-pass, band-pass, and low-pass component data.

[0008] Figure 6 is a block diagram illustrating an embodiment of a deep learning system for autonomous driving. DETAILED DESCRIPTION

[0009] The present invention may be implemented in a variety of ways, including as a process; an apparatus; a system; a composition of matter; a computer program product embodied on a computer-readable storage medium; and / or a processor, such as a processor configured to execute instructions stored on and / or provided by a memory coupled to the processor. In this specification, these implementations or any other form that the present invention may take may be referred to as techniques. In general, the order of steps of the disclosed processes may be changed within the scope of the present invention. Unless otherwise stated, components such as processors or memories described as being configured to perform tasks may be implemented as general components that are temporarily configured to perform tasks at a given time or as specific components manufactured to perform tasks. As used herein, the term "processor" refers to one or more devices, circuits, and / or processing cores configured to process data such as computer program instructions.

[0010] A detailed description of one or more embodiments of the present invention and drawings illustrating the principles of the present invention are provided below. The present invention is described in conjunction with such embodiments, but the present invention is not limited to any embodiment. The scope of the present invention is limited only by the claims, and the present invention encompasses many alternatives, modifications and equivalents. Many specific details are set forth in the following description to provide a thorough understanding of the present invention. These details are provided for illustrative purposes, and the present invention can be practiced according to the claims without some or all of these specific details. For the sake of clarity, technical materials known in the technical field related to the present invention are not described in detail so as not to unnecessarily obscure the present invention.

[0011] A data pipeline is disclosed that extracts sensor data and provides it as a separate component to a deep learning network for autonomous driving. In some embodiments, autonomous driving is implemented using a deep learning network and input data received from a sensor. For example, a sensor fixed to a vehicle provides real-time sensor data (such as visual, radar, and ultrasonic data) of the vehicle's surroundings to a neural network to determine a vehicle control response. In some embodiments, the network is implemented using multiple layers. Based on the signal information of the data, the sensor data is extracted into two or more different data components. For example, feature and / or edge data can be extracted as different data components separately from global data such as global illumination data. Different data components retain target-related data, for example, data that will eventually be used by a deep learning network to identify edges and other features. In some embodiments, different data components are used as containers for storing data that is highly relevant to identifying certain target features, but they do not identify or detect features themselves. Different data components extract data to ensure accurate feature detection at the appropriate stage of the machine learning network. In some embodiments, the different data components can then be preprocessed to enhance the specific signal information they contain. The data components can be compressed and / or downsampled to increase resource and computational efficiency. Then, different data components are provided to the deep learning system at different layers of the system. The deep learning network is able to use the signal information maintained during the extraction process as input to accurately identify and detect features associated with the target data (e.g., edges, objects, etc.) of the data components. For example, feature and edge data are provided to the first layer of the network, and global data is provided to the later layers of the network. By extracting different data components that each maintain their corresponding target signal information, the network can process sensor data more efficiently. Instead of receiving sensor data as the initial input of the network, the most useful information is provided to the network at the most appropriate layer of the network. In some embodiments, a more complete version of the captured sensor data is analyzed by the network because different data components can make full use of the image resolution of their corresponding components for their intended purpose. For example, inputs for features and edges can utilize full resolution, bit range, and bit depth for feature and edge data, while inputs for global illumination can utilize full resolution, bit range, and bit depth for global illumination data.

[0012] In some embodiments, an image captured using a sensor on a vehicle is received. For example, an image is captured from a front camera with a high dynamic range. As another example, ultrasound data is captured from a side-facing ultrasound sensor. In some embodiments, the received image is decomposed into multiple component images. For example, feature data is extracted from the captured high dynamic range image. As another example, global illumination data is extracted from the captured high dynamic range image. As another example, high-pass, low-pass and / or band-pass filters can be used to decompose the image. In some embodiments, each of the multiple component images is provided as a different input to a different layer of multiple layers of an artificial neural network to determine the result. For example, an artificial neural network such as a convolutional neural network includes multiple layers for processing input data. Different component images decomposed from the captured image are provided as input to different layers of the neural network. For example, feature data is presented as input to the first layer of the network, and global data is presented as input to a later layer (e.g., the third layer) of the network. In some embodiments, the results of the artificial neural network are used to at least partially autonomously operate the vehicle. For example, the results of the deep learning analysis performed using the artificial neural network are used to control the steering, braking, lighting and / or warning systems of the vehicle. In some embodiments, the results are used to autonomously match the vehicle's speed to traffic conditions, guide the vehicle to follow a navigation path, avoid collisions when objects are detected, summon the vehicle to a desired location, and warn the user of potential collisions, among other autonomous driving applications.

[0013] In some embodiments, the vehicle is attached with multiple sensors for capturing data. For example, in some embodiments, eight surround cameras are attached to the vehicle and provide 360-degree visibility around the vehicle within a range of up to 250 meters. In some embodiments, the camera sensor includes a wide-angle front camera, a narrow-angle front camera, a rear-view camera, a front-view side camera, and / or a rear-view side camera. In some embodiments, ultrasonic and radar sensors are used to capture surrounding details. For example, twelve ultrasonic sensors can be fixed to the vehicle to detect both hard objects and soft objects. In some embodiments, forward radar is used to capture data of the surrounding environment. In various embodiments, despite heavy rain, fog, dust and other vehicles, the radar sensor is still able to capture surrounding details. Various sensors are used to capture the environment around the vehicle, and the captured images are provided for deep learning analysis.

[0014] Using data captured from sensors and analyzed using the disclosed deep learning system, machine learning results are determined for autonomous driving. In various embodiments, the machine learning results are provided to a vehicle control module to implement autonomous driving features. For example, a vehicle control module can be used to control the steering, braking, warning system, and / or lighting of a vehicle. In some embodiments, the vehicle is controlled to navigate on a road, match the speed of the vehicle with traffic conditions, keep the vehicle in a lane, automatically change lanes without driver input, convert the vehicle from one highway to another, exit the highway when approaching the destination, automatically park the vehicle, and summon the vehicle to and from a parking space, as well as other autonomous driving applications. In some embodiments, autonomous driving features include identifying opportunities to move the vehicle to a faster lane when slower traffic is behind. In some embodiments, machine learning results are used to determine when autonomous driving without driver interaction is appropriate and when it should be disabled. In various embodiments, machine learning results are used to assist the driver in driving the vehicle.

[0015] In some embodiments, the machine learning results are used to implement an automatic parking mode in which the vehicle will automatically search for a parking space and park the vehicle. In some embodiments, the machine learning results are used to navigate the vehicle using a destination from a user's calendar. In various embodiments, the machine learning results are used to implement autonomous driving safety features, such as collision avoidance and automatic emergency braking. For example, in some embodiments, a deep learning system detects objects that may collide with the vehicle, and the vehicle control module applies the brakes accordingly. In some embodiments, the vehicle control module uses deep learning analysis to implement side, front and / or rear collision warnings, which warn the user of the vehicle of possible collisions with obstacles next to, in front of or behind the vehicle. In various embodiments, the vehicle control module can activate a warning system, such as a collision alarm, an audio alarm, a visual alarm, and / or a physical alarm (such as a vibration alarm), etc., to notify the user of an emergency or a situation where driver attention is necessary. In some embodiments, the vehicle control module can initiate a communication response according to the situation, such as an emergency response call, a text message, a network update, and / or another communication response, for example, to notify another party of an emergency. In some embodiments, the vehicle control module can adjust lighting based on the deep learning analysis results, including high / low beams, brake lights, interior lights, emergency lights, etc. In some embodiments, the vehicle control module can also adjust the audio inside or around the vehicle based on the deep learning analysis results, including using the horn, modifying the audio played from the vehicle's sound system (e.g., music, phone calls, etc.), adjusting the volume of the sound system, playing an audio alarm, enabling a microphone, and the like.

[0016] Figure 1 is a flow chart illustrating an embodiment of a process for performing a machine learning process using a deep learning pipeline. For example, Figure 1 The process can be used to implement autonomous driving features of autonomous driving and driver assistance vehicles, thereby improving safety and reducing the risk of accidents. In some embodiments, Figure 1 The process of preprocessing the data captured by the sensor for deep learning analysis. By preprocessing the sensor data, the data provided for deep learning analysis can be enhanced and more accurate results can be provided for controlling the vehicle. In some embodiments, the preprocessing resolves data mismatches between the data captured by the sensor and the data that the neural network expects to use for deep learning.

[0017] At 101, sensor data is received. For example, the sensor data is captured by one or more sensors fixed to a vehicle. In some embodiments, the sensor is fixed to the environment and / or other vehicles, and the data is received remotely. In various embodiments, the sensor data is image data, such as RGB or YUV channels of an image. In some embodiments, the sensor data is captured using a high dynamic range camera. In some embodiments, the sensor data is radar, LiDAR, and / or ultrasonic data. In various embodiments, LiDAR data is data captured using lasers and may include techniques known as light detection and ranging and laser imaging, detection, and ranging. In various embodiments, the bit depth of the sensor data exceeds the bit depth of the neural network used for deep learning analysis.

[0018] At 103, data preprocessing is performed on the sensor data. In some embodiments, one or more preprocessing may be performed on the sensor data. For example, the data may first be preprocessed to remove noise, correct alignment issues and / or blur, and so on. In some embodiments, two or more different filtering passes are performed on the data. For example, a high-pass filter may be performed on the data, and a low-pass filter may be performed on the data. In some embodiments, one or more band-pass filters may be performed. For example, in addition to a high-pass and a low-pass, one or more band-passes may also be performed on the data. In various embodiments, the sensor data is divided into two or more data sets, such as a high-pass data set and a low-pass data set. In some embodiments, one or more band-pass data sets are also created. In various embodiments, the different data sets are different components of the sensor data.

[0019] In some embodiments, the different components created by preprocessing the data include feature and / or edge components and global data components. In various embodiments, the feature and / or edge components are created by performing a high pass or band pass filter on the sensor data, and the global data components are created by performing a low pass or band pass filter on the sensor data. In some embodiments, one or more different filtering techniques may be used to extract feature / edge data and / or global data.

[0020] In various embodiments, one or more components of the sensor data are processed. For example, the high-pass component can be processed by removing noise from the image data and / or enhancing the local contrast of the image data. In some embodiments, the low-pass component is compressed and / or downsampled. In various embodiments, different components are compressed and / or downsampled. For example, the components can be appropriately compressed, resized, and / or downsampled to adjust the size and / or resolution of the data for inputting the data into a layer of the machine learning model. In some embodiments, the bit depth of the sensor data is adjusted. For example, a data channel of camera-captured data at 20 bits or other suitable bit depth is compressed or quantized to 8 bits to prepare channels for an 8-bit machine learning model. In some embodiments, one or more sensors capture data at a bit depth of 12 bits, 16 bits, 20 bits, 32 bits, or another suitable bit depth greater than the bit depth used by the deep learning network.

[0021] In various embodiments, the preprocessing performed at 103 is performed by an image preprocessor. In some embodiments, the image preprocessor is a graphics processing unit (GPU), a central processing unit (CPU), an artificial intelligence (AI) processor, an image signal processor, a tone mapper processor, or other similar hardware processor. In various embodiments, different image preprocessors are used to extract and / or preprocess different data components in parallel.

[0022] At 105, a deep learning analysis is performed. For example, the deep learning analysis is performed using a machine learning model (such as an artificial neural network). In various embodiments, the deep learning analysis receives as input the processed sensor data for 103. In some embodiments, the processed sensor data is received at 105 as a plurality of different components (such as a high-pass data component and a low-pass data component). In some embodiments, the different data components are received as inputs to different layers of the machine learning model. For example, a neural network receives a high-pass component as an initial input to the first layer of the network, and receives a low-pass component as an input to subsequent layers of the network.

[0023] At 107, the results of the deep learning analysis are provided for vehicle control. For example, the results may be provided to a vehicle control module to adjust the speed and / or steering of the vehicle. In various embodiments, the results are provided to enable autonomous driving functionality. For example, the results may indicate objects that should be avoided by steering the vehicle. As another example, the results may indicate merging vehicles that should be avoided by braking and changing the position of the vehicle in the lane.

[0024] Figure 2 is a flow chart illustrating an embodiment of a process for performing a machine learning process using a deep learning pipeline. For example, Figure 2 The process can be used to pre-process sensor data, extract image components from the sensor data, pre-process the extracted image components, and then provide these components for deep learning analysis. The results of the deep learning analysis can be used to implement autonomous driving to improve safety and reduce the risk of accidents. In some embodiments, Figure 2 The process used to execute Figure 1 In some embodiments, step 201 Figure 1 101 is performed; steps 203, 205, 207 and / or 209 are performed at Figure 1 103 is executed; and / or step 211 is performed at Figure 1 105 and / or 107 of the embodiment. By processing the extracted components of the sensor data, the processed data provided to the machine learning model is enhanced, thereby obtaining superior results from the deep learning analysis rather than using other non-enhanced data. In the example shown, the results of the deep learning analysis are used for vehicle control.

[0025] At 201, sensor data is received. In various embodiments, the sensor data is image data captured from a sensor such as a high dynamic range camera. In some embodiments, the sensor data is captured from one or more different sensors. In some embodiments, the image data is captured using a bit depth of 12 bits or higher to increase the fidelity of the data.

[0026] At 203, the data is preprocessed. In some embodiments, the data is preprocessed using an image preprocessor such as an image signal processor, a graphics processing unit (GPU), a tone mapper processor, a central processing unit (CPU), an artificial intelligence (AI) processor, or other similar hardware processor. In various embodiments, linearization, demosaicing, and / or another processing technique may be performed on the captured sensor data. In various embodiments, preprocessing is performed on the high-resolution sensor data to enhance the fidelity of the captured data and / or to reduce the introduction of errors by subsequent steps. In some embodiments, the preprocessing step is optional.

[0027] At 205, one or more image components are extracted. In some embodiments, two image components are extracted. For example, a feature / edge data component of the sensor data is extracted, and a global data component of the sensor data is extracted. In some embodiments, a high-pass component and a low-pass component of the sensor data are extracted. In some embodiments, one or more additional bandpass components are extracted from the sensor data. In various embodiments, high-pass, low-pass and / or bandpass filters are used to extract different components of the sensor data. In some embodiments, a tone mapper is used to extract image components. In some embodiments, global data and / or low-pass component data are extracted by downsampling the sensor data using binning or similar techniques. In various embodiments, the extraction maintains and saves the target signal information as an image data component, but does not actually detect or identify features related to the target information. For example, the extraction of an image component corresponding to edge data produces an image component having target signal information for accurately identifying edges, but the extraction performed at 205 does not detect the presence of edges in the sensor data.

[0028] In some embodiments, image data components extracted for a first layer of a machine learning network are extracted using a process that preserves the response of the first layer of deep learning analysis. For example, relevant signal information for the first layer is preserved so that the results of analysis performed on the image components after the analysis of the first layer are similar to the analysis performed on the corresponding sensor data before extraction into image components. In various embodiments, the results are preserved for filters as small as a 5×5 matrix filter.

[0029] In some embodiments, the extracted data component is created by combining multiple channels of the captured image into one or more channels. For example, the red, green, and blue channels may be averaged to create a new channel for the data component. In various embodiments, the extracted data component may be constructed from one or more different channels of source captured data and / or one or more different captured images of different sensors. For example, data from multiple sensors may be combined into a single data component.

[0030] In some embodiments, an image preprocessor such as the preprocessor of step 203 is used to extract different components. In some embodiments, an image signal processor can be used to extract different components. In various embodiments, a graphics processing unit (GPU) can be used to extract different components. In some embodiments, different preprocessors are used to extract different components so that multiple components can be extracted in parallel. For example, an image signal processor can be used to extract a high-pass component, and a GPU can be used to extract a low-pass component. As another example, an image signal processor can be used to extract a low-pass component, and a GPU can be used to extract a high-pass component. In some embodiments, a tone mapper processor is used to extract image components (such as high-pass components), and a GPU is used to extract separate image components (such as low-pass components) in parallel. In some embodiments, a tone mapper is part of an image signal processor. In some embodiments, there are multiple instances of similar preprocessors for performing extraction in parallel.

[0031] At 207, component preprocessing is performed. In some embodiments, an image preprocessor such as the preprocessor of steps 203 and / or 205 is used to preprocess one or more components. In some embodiments, different preprocessors are used to preprocess different components so that preprocessing can be performed on different components in parallel. For example, an image signal processor can be used to process a high-pass component, and a graphics processing unit (GPU) can be used to process a low-pass component. In some embodiments, a tone mapper processor is used to process one image component, and the GPU is used to process separate image components in parallel. In some embodiments, there are multiple instances of a similar preprocessor for processing different components in parallel.

[0032] In various embodiments, preprocessing includes downsampling and / or compressing the image component data. In some embodiments, preprocessing includes removing noise from the component data. In some embodiments, preprocessing includes compressing or quantizing the captured data from 20 bits down to 8-bit data fields. In some embodiments, preprocessing includes converting the size of the image component to a lower resolution. For example, the image component can be half, one-quarter, one-eighth, one-sixteenth, one-thirty-second, one-sixty-fourth, or other appropriate scaling of the original sensor image size. In various embodiments, the image component is reduced to a size suitable for the input layer of the machine learning model.

[0033] At 209, the components are provided to appropriate network layers of the deep learning network. For example, different components may be provided to different layers of the network. In some embodiments, the network is a neural network having multiple layers. For example, the first layer of the neural network receives high-pass component data as input. One of the subsequent network layers receives low-pass component data corresponding to the global illumination data as input. In various embodiments, different components extracted at 205 and preprocessed at 207 are received at different layers of the neural network. As another example, feature and / or edge data components are provided as input to the first layer of a deep learning network such as an artificial neural network. The global data component is provided to subsequent layers and may be provided as a compressed and / or downsampled version of the data because the global data does not require as high precision as the feature and / or edge component data. In various embodiments, the global data is more easily compressed without losing information and may be provided at a later layer of the network.

[0034] In some embodiments, the machine learning model consists of multiple consecutive layers, wherein one or more subsequent layers receive input data having a size attribute that is smaller than the previous layer. For example, the first layer of the network may receive an image size similar to the size of the captured image. Subsequent layers may receive input data that is half or one-quarter the size of the captured image. The reduction in the size of the input data reduces the computation of subsequent layers and improves the efficiency of deep learning analysis. Computational efficiency can be improved by providing sensor input data as different components and at different layers. Earlier layers of the network require increased computation, especially because the amount of data and the size of the data are larger than subsequent layers. Since the input data has been compressed by the previous layers of the network and / or preprocessing at 207, subsequent layers can perform computations more efficiently.

[0035] At 211, the results of the deep learning analysis are provided for vehicle control. For example, the machine learning results using the processed image components can be used to control the movement of the vehicle. In some embodiments, the results correspond to vehicle control actions. For example, the results can correspond to the speed and steering of the vehicle. In some embodiments, the results are received by a vehicle control module for assisting in maneuvering the vehicle. In some embodiments, the results are used to improve the safety of the vehicle. In various embodiments, the results provided at 211 are determined by performing a deep learning analysis on the components provided at 209.

[0036] Figure 3 is a flow chart illustrating an embodiment of a process for performing a machine learning process using component data. In the illustrated example, Figure 3A process is used to extract feature and edge data from the sensor data that is separate from the global data. These two data sets are then fed into the deep learning network at different stages to infer vehicle control outcomes. By separating the two components and providing them at different stages, the initial layers of the network can dedicate computational resources to initial edge and feature detection. In some embodiments, the initial stage dedicates resources to the initial identification of objects, such as roads, lane markings, obstacles, vehicles, pedestrians, traffic signs, etc. Subsequent layers can utilize the global data in a more computationally efficient manner because the global data is less resource intensive. Since machine learning can be computationally and data intensive, a data pipeline that uses different image components at different stages is used to improve the efficiency of deep learning computations and reduce the data resource requirements required for analysis. In some embodiments, Figure 3 The process used to execute Figure 1 and / or Figure 2 In some embodiments, step 301 Figure 1 101 and / or Figure 2 Step 303 is executed at 201; Figure 1 103 and / or Figure 2 203 is executed; Steps 311 and / or 321 are performed at Figure 1 103 and / or Figure 2 205 is executed; Steps 313, 323 and / or 325 are performed at Figure 1 103 and / or Figure 2 207 and 209 are performed; Steps 315 and / or 335 are performed at Figure 1 105 and / or Figure 2 211 is executed; and / or step 337 is executed at Figure 1 107 and / or Figure 2 211 executions.

[0037] At 301, sensor data is received. In various embodiments, the sensor data is data captured by one or more sensors of a vehicle. In some embodiments, such as Figure 1 Step 101 and / or Figure 2 The sensor data is received as described in step 201 of FIG.

[0038] At 303, data preprocessing is performed. For example, sensor data is enhanced by preprocessing the data. In some embodiments, the data is cleaned, for example, by performing denoising, alignment, or other appropriate filters. In various embodiments, as described with respect to Figure 1 Step 103 and / or Figure 2The data is preprocessed as described in step 203 of FIG. In the example shown, processing continues to steps 311 and 321. In some embodiments, the processing at 311 and 321 runs in parallel to extract and process different components of the sensor data. In some embodiments, each branch of the processing (e.g., the branch starting at 311 and the branch starting at 321) runs sequentially or pipelined. For example, processing is performed starting at step 311 to prepare data for the initial layer of the network. In some embodiments, the preprocessing step is optional.

[0039] At 311, feature and / or edge data are extracted from the sensor data. For example, feature data and / or edge data are extracted from the captured sensor data as component data. In some embodiments, the component data maintains relevant signal information from the sensor data for identifying features and / or edges. In various embodiments, the extraction process maintains signal information that is critical for identifying and detecting features and / or edges, and does not actually identify or detect features or edges from the sensor data. In various embodiments, features and / or edges are detected during one or more analysis steps at 315 and / or 335. In some embodiments, the extracted feature and / or edge data has the same bit depth as the original captured data. In some embodiments, the extracted data is feature data, edge data, or a combination of feature and edge data. In some embodiments, a high pass filter is used to extract feature and / or edge data from the sensor data. In various embodiments, the tone mapper processor is calibrated to extract feature and / or edge data from the sensor data.

[0040] At 313, preprocessing is performed on the feature and / or edge data. For example, a denoising filter can be applied to the data to improve signal quality. As another example, before deep learning analysis, different preprocessing techniques such as local contrast enhancement, gain adjustment, thresholding, noise filtering, etc. can be applied to enhance feature and edge data. In various embodiments, preprocessing is customized to enhance the features and edge characteristics of the data, rather than applying more general preprocessing techniques to the sensor data as a whole. In some embodiments, preprocessing includes performing compression and / or downsampling on the extracted data. In some embodiments, the preprocessing step at 313 is optional.

[0041] At 315, an initial analysis is performed using the feature and / or edge data. In some embodiments, the initial analysis is a deep learning analysis using a machine learning model such as a neural network. In various embodiments, the initial analysis receives the feature and edge data as input to a first layer of the network. In some embodiments, the initial layer of the network preferentially detects features and / or edges in the captured image. In various embodiments, the deep learning analysis is performed using an artificial neural network such as a convolutional neural network. In some embodiments, the analysis is run on an artificial intelligence (AI) processor.

[0042] At 321, global data is extracted from the sensor data. For example, global data is extracted from the captured sensor data as component data. In some embodiments, the global data corresponds to global illumination data. In some embodiments, the extracted global data has the same bit depth as the original captured data. In some embodiments, a low pass filter is used to extract global data from the sensor data. In various embodiments, a tone mapper processor is calibrated to extract global data from the sensor data. Other techniques (such as merging, resampling, and downsampling) may also be used to extract global data. In various embodiments, the extraction process retains data that may be globally relevant, and global features are not identified and detected from the sensor data. In various embodiments, global features are detected by the analysis performed at 335.

[0043] At 323, preprocessing is performed on the global data. For example, a denoising filter may be applied to the data to improve signal quality. As another example, different preprocessing techniques such as local contrast enhancement, gain adjustment, thresholding, noise filtering, etc. may be applied to enhance the global data before deep learning analysis. In various embodiments, the preprocessing is customized to enhance the characteristics of the global data, rather than applying more general preprocessing techniques to the sensor data as a whole. In some embodiments, the preprocessing of the global data includes compressing the data. In some embodiments, the preprocessing step at 323 is optional.

[0044] At 325, the global data is downsampled. For example, the resolution of the global data is reduced. In some embodiments, the size of the global data is reduced to improve the computational efficiency of the analysis data and the global data is configured as the input to the later layer of the deep learning network. In some embodiments, the global data is downsampled by merging, resampling or another appropriate technique. In some embodiments, a graphics processing unit (GPU) or an image signal processor is used to perform downsampling. In various embodiments, downsampling is applicable to global data because global data does not have the same resolution requirements as feature and / or edge data. In some embodiments, when the global data is extracted at 321, the downsampling performed at 325 is performed.

[0045] At 335, additional deep learning analysis is performed using the results of the deep learning analysis of the feature and / or edge data and the global data as input. In various embodiments, the deep learning analysis receives the global data as input at a later layer of the deep learning network. In various embodiments, the expected input data size at the layer receiving the global data is smaller than the expected input data size of the initial input layer. For example, the input size of the global data input layer can be half or one-quarter of the input size of the initial layer of the deep learning network. In some embodiments, the later layers of the network utilize the global data to enhance the results of the initial layer. In various embodiments, a deep learning analysis is performed and a vehicle control result is determined. For example, a convolutional neural network is used to determine the vehicle control result. In some embodiments, the analysis is run on an artificial intelligence (AI) processor.

[0046] At 337, the results of the deep learning analysis are provided for vehicle control. For example, the machine learning results using the extracted and processed image components are used to control the movement of the vehicle. In some embodiments, the results correspond to vehicle control actions. In some embodiments, such as with respect to Figure 1 Step 107 and / or Figure 2 The result is provided as described in step 211.

[0047] Figure 4 is a flow chart illustrating an embodiment of a process for performing machine learning processing using high-pass and low-pass component data. In the illustrated example, Figure 4 A process is used to extract two data components from the sensor data and provide these components to different layers of a deep learning network such as an artificial neural network. The two components are extracted using high-pass and low-pass filters. In various embodiments, the results are used to achieve autonomous driving with improved accuracy, safety, and / or comfort results. In some embodiments, Figure 4 The process used to execute Figure 1 , 2 In some embodiments, step 401 is performed in Figure 1 103, Figure 2 203 and / or Figure 3 Step 403 is executed at 303; Figure 1 103, Figure 2 205 and / or Figure 3 311 is executed; step 413 is executed at Figure 1 103, Figure 2 205 and / or Figure 3 321 is executed; step 405 is executed at Figure 1 103, Figure 2 207 and 209 and / or Figure 3 313 is executed; steps 415 and 417 are performed at Figure 1 103, Figure 2 207 and 209, and / or Figure 3 323 and 325 are executed; step 407 is performed at Figure 1 105 Figure 2 211 and / or Figure 3 315 is executed; and / or step 421 is executed at Figure 1 105 Figure 2 211 and / or Figure 3 335 executions.

[0048] At 401, data is preprocessed. In some embodiments, the data is sensor data captured from one or more sensors such as a high dynamic range camera, radar, ultrasound, and / or LiDAR sensor. In various embodiments, as described with respect to Figure 1 , Figure 2 203 and / or Figure 3 The data is preprocessed as described in 303. Once the data is preprocessed, processing continues to 403 and 413. In some embodiments, steps 403 and 413 are run in parallel.

[0049] At 403, a high pass filter is performed on the data. For example, a high pass filter is performed on the captured sensor data to extract high pass component data. In some embodiments, the high pass filter is performed using a graphics processing unit (GPU), a tone mapper processor, an image signal processor, or other image preprocessor. In some embodiments, the high pass data component represents features and / or edges of the captured sensor data. In various embodiments, the high pass filter is constructed to maintain the response of the first layer of the deep learning process. For example, a high pass filter is constructed to maintain the response of a small filter at the top of the machine learning network. The relevant signal information for the first layer of the network is maintained so that the analysis results performed on the high pass component data after the first layer are similar to the analysis performed on the unfiltered data after the first layer. In various embodiments, the results are maintained for filters as small as a 5×5 matrix filter.

[0050] At 413, a low pass filter is performed on the data. For example, a low pass filter is performed on the captured sensor data to extract low pass component data. In some embodiments, the low pass filter is performed using a graphics processing unit (GPU), a tone mapper processor, an image signal processor, or other image preprocessor. In some embodiments, the low pass data component represents global data of the captured sensor data, such as global illumination data.

[0051] In various embodiments, the filtering performed at 403 and 413 may use the same or different image preprocessors. For example, a tone mapper processor is used to extract a high-pass data component, while a graphics processing unit (GPU) is used to extract a low-pass data component. In some embodiments, high-pass or low-pass data is extracted by subtracting one of the data components from the original captured data.

[0052] At 405 and 415, post-processing is performed on the corresponding high-pass and low-pass data components. In various embodiments, different post-processing techniques are used to enhance signal quality and / or reduce the amount of data required to represent the data. For example, denoising, demosaicing, local contrast enhancement, gain adjustment, and / or threshold processing processes, etc., can be performed on the corresponding high-pass and / or low-pass data components. In some embodiments, the data components are compressed and / or downsampled. For example, once the high-pass and / or low-pass data are extracted, the corresponding data components can be compressed to more effectively utilize the full bit depth range. In some embodiments, the corresponding data components are compressed or quantized from the higher bit depth captured by the sensor to a lower bit depth compatible with the deep learning network. For example, sensor data captured at 12 bits, 16 bits, 20 bits, 32 bits, or other suitable bit depths per channel can be compressed or quantized to a lower bit depth such as 8 bits per channel. In some embodiments, the post-processing steps at 405 and / or 415 are optional.

[0053] At 417, the low-pass data component is downsampled. In various embodiments, the low-pass data component is fed into the network at a later stage of the network and can be downsampled to a more efficient resource size. For example, the low-pass data component can be extracted at the full sensor size and reduced to half or a quarter of the original size. Other reduction percentages are also possible. In various embodiments, the low-pass data is downsampled but the relevant signal information is maintained. In many cases, the low-pass data can be easily downsampled without losing signal information. By downsampling the data, the data can be analyzed more easily and quickly at later layers of the deep learning network.

[0054] At 407, a deep learning analysis is performed on the high-pass data component. In some embodiments, the high-pass data component is fed into an initial layer of a deep learning network and represents the most important data for feature and edge detection. In various embodiments, the results of the deep learning analysis performed in the first layer using the high-pass data component are fed into subsequent layers of the network. For example, a neural network may include multiple layers, such as five or more layers. The first layer receives the high-pass data component as input, and the second layer receives the results of the deep learning analysis performed by the first layer. In various embodiments, the second or later layer receives the low-pass data component as an additional input to perform additional deep learning analysis.

[0055] At 421, additional deep learning analysis is performed using the analysis results performed at 407 and the low-pass data components downsampled at 417. In various embodiments, the deep learning analysis infers vehicle control results. For example, the results of the deep learning analysis at 407 and 421 are used to control the vehicle for autonomous driving.

[0056] Figure 5 is a flow chart illustrating an embodiment of a process for performing machine learning processing using high-pass, band-pass, and low-pass component data. In the illustrated example, Figure 5 The process is used to extract three or more data components from sensor data and provide them at different layers of a deep learning network such as an artificial neural network. Figure 4 The process extracts the high-pass and low-pass components. Figure 5 The process extracts one or more bandpass data components. In various embodiments, the sensor data is decomposed into multiple components that are provided to different layers of the deep learning network so that the deep learning analysis can emphasize different data sets at different layers of the network.

[0057] In some embodiments, Figure 5 The process used to execute Figure 1 , 2 , 3 and / or 4. In some embodiments, step 501 is Figure 1 103, Figure 2 203 Figure 3 303 and / or Figure 4 In some embodiments, step 503 is performed at 401. Figure 1 103, Figure 2 205 Figure 3 311 and / or Figure 4 403 is executed; step 513 is executed at Figure 1 103, Figure 2 205 and / or Figure 3 311 or 321 is performed; and / or step 523 is performed at Figure 1 103, Figure 2 205 Figure 3 321 and / or Figure 4 In some embodiments, step 505 is performed at 413. Figure 1 103, Figure 2 207 and 209, Figure 3 313 and / or Figure 4 Step 405 is executed; Step 515 is executed at Figure 1 103, Figure 2 207 and 209, Figure 3 313, 323 and / or 325, and / or Figure 4405, 415 and / or 417 are performed; and / or step 525 is performed at Figure 1 103, Figure 2 207 and 209, Figure 3 323 and 325, and / or Figure 4 In some embodiments, step 537 is performed at 415 and 417. Figure 1 105 Figure 2 211 Figure 3 315 and 335, and / or Figure 4 407 and 421 are executed.

[0058] At 501, data is preprocessed. In some embodiments, the data is sensor data captured from one or more sensors such as a high dynamic range camera, radar, ultrasound, and / or LiDAR sensor. In various embodiments, as described with respect to Figure 1 103, Figure 2 203 Figure 3 303 and / or Figure 4 The data is preprocessed as described in 401. Once the data is preprocessed, the process continues to 503, 513 and 523. In some embodiments, steps 503, 513 and 523 are run in parallel.

[0059] At 503, a high pass filter is performed on the data. For example, a high pass filter is performed on the captured sensor data to extract high pass component data. In some embodiments, the high pass filter is performed using a graphics processing unit (GPU), a tone mapper processor, an image signal processor, or another image preprocessor. In some embodiments, the high pass data component represents features and / or edges of the captured sensor data.

[0060] At 513, one or more bandpass filters are performed on the data to extract one or more bandpass data components. For example, a bandpass filter is performed on the captured sensor data to extract component data, including a mixture of feature, edge, intermediate and / or global data. In various embodiments, one or more bandpass components can be extracted. In some embodiments, a low-pass filter is performed using a graphics processing unit (GPU), a tone mapper processor, an image signal processor, or another image preprocessor. In some embodiments, the bandpass data component represents data that is neither primarily edge / feature data of the captured sensor data nor primarily global data. In some embodiments, the bandpass data is used to maintain data fidelity that may be lost using only high-pass data components and low-pass data components.

[0061] At 523, a low pass filter is performed on the data. For example, a low pass filter is performed on the captured sensor data to extract low pass component data. In some embodiments, the low pass filter is performed using a graphics processing unit (GPU), a tone mapper processor, an image signal processor, or other image preprocessor. In some embodiments, the low pass data component represents global data of the captured sensor data, such as global illumination data.

[0062] In various embodiments, the filtering performed at 503, 513, and 523 may use the same or different image preprocessors. For example, a tone mapper processor is used to extract high-pass data components, while a graphics processing unit (GPU) is used to extract bandpass and / or low-pass data components. In some embodiments, data components are extracted by subtracting one or more data components from the original captured data.

[0063] At 505, 515, and 525, post-processing is performed on the corresponding high-pass, band-pass, and low-pass data components. In various embodiments, different post-processing techniques are utilized to enhance signal quality and / or reduce the amount of data required to represent the data. In some embodiments, for a network layer that receives data components, the different components are compressed and / or downsampled to an appropriate size. In various embodiments, the high-pass data will have a higher resolution than the band-pass data, and the band-pass data will have a higher resolution than the low-pass data. In some embodiments, the different band-pass data components will also have different resolutions suitable for the network layer, and each data component is provided as input. In some embodiments, the corresponding data components are compressed or quantized from a higher bit depth captured by the sensor to a lower bit depth compatible with the deep learning network. For example, sensor data captured at 12 bits per channel can be compressed or quantized to 8 bits per channel. In various embodiments, as described with respect to Figure 2 207 and / or Figure 4 Apply a pre-processing filter as described in 405, 415 and / or 417.

[0064] At 537, deep learning analysis is performed using the data component results of 505, 515, and 525. In some embodiments, the high-pass data component is fed into the initial layer of the deep learning network and represents the most important data for feature and edge detection. One or more band-pass data components are fed into the (multiple) intermediate layers of the network and include additional data and / or useful intermediate or global information for identifying features / edges. The low-pass data component is fed into the later layers of the network and includes global information to improve the analysis results of the deep learning network. When performing deep learning analysis, as the analysis proceeds, other data components representing different sensor data are fed into different layers to improve the accuracy of the results. In various embodiments, deep learning analysis infers vehicle control results. For example, the results of deep learning analysis are used to control vehicles for autonomous driving. In some embodiments, machine learning results are provided to a vehicle control module to at least partially autonomously operate a vehicle.

[0065] Figure 6 is a block diagram illustrating an embodiment of a deep learning system for autonomous driving. In some embodiments, Figure 6 The deep learning system can be used to implement autonomous driving features for autonomous and driver-assisted vehicles. For example, using sensors fixed to the vehicle, sensor data can be captured, processed as different input components, and then fed into different stages of the deep learning network. The results of the deep learning analysis are used by the vehicle control module to assist in the operation of the vehicle. In some embodiments, the vehicle control module is used for autonomous driving or driver-assisted operation of the vehicle. In various embodiments, Figure 1-Figure 5 The process uses Figure 6 The deep learning system described in .

[0066] In the example shown, the deep learning system 600 is a deep learning network that includes a sensor 601, an image preprocessor 603, a deep learning network 605, an artificial intelligence (AI) processor 607, a vehicle control module 609, and a network interface 611. In various embodiments, the different components are communicatively connected. For example, sensor data from the sensor 601 is fed to the image preprocessor 603. The processed sensor data component of the image preprocessor 603 is fed to the deep learning network 605 running on the AI ​​processor 607. The output of the deep learning network 605 running on the AI ​​processor 607 is fed to the vehicle control module 609. In various embodiments, the network interface 611 is used to communicate with a remote server based on the autonomous operation of the vehicle, make calls, send and / or receive text messages, and the like.

[0067] In some embodiments, sensor 601 includes one or more sensors. In various embodiments, sensor 601 can be fixed to the vehicle at different locations of the vehicle and / or oriented in one or more different directions. For example, sensor 601 can be fixed to the front, side, back and / or roof of the vehicle in the forward, backward, side and other directions. In some embodiments, sensor 610 can be an image sensor such as a high dynamic range camera. In some embodiments, sensor 601 includes a non-visual sensor. In some embodiments, sensor 601 includes radar, LiDAR and / or ultrasonic sensors, etc. In some embodiments, sensor 601 is not installed on a vehicle with a vehicle control module 609. For example, sensor 601 can be installed on an adjacent vehicle and / or fixed to a road or environment, and is included as part of a deep learning system for capturing sensor data.

[0068] In some embodiments, the image preprocessor 603 is used to preprocess the sensor data of the sensor 601. For example, the image preprocessor 603 can be used to preprocess the sensor data, separate the sensor data into one or more components, and / or post-process one or more components. In some embodiments, the image preprocessor 603 is a graphics processing unit (GPU), a central processing unit (CPU), an image signal processor, or a dedicated image processor. In various embodiments, the image preprocessor 603 is a tone mapper processor for processing high dynamic range data. In some embodiments, the image preprocessor 603 is implemented as part of an artificial intelligence (AI) processor 607. For example, the image preprocessor 603 can be a component of the AI ​​processor 607.

[0069] In some embodiments, the deep learning network 605 is a deep learning network for implementing autonomous vehicle control. For example, the deep learning network 605 may be an artificial neural network such as a convolutional neural network (CNN), which is trained using sensor data and used to output vehicle control results to the vehicle control module 609. In various embodiments, the deep learning network 605 is a multi-stage learning network, and input data may be received at two or more different stages of the network. For example, the deep learning network 605 may receive feature and / or edge data at the first layer of the deep learning network 605, and receive global data at a later layer (e.g., the second or third layer, etc.) of the deep learning network 605. In various embodiments, the deep learning network 605 receives data at two or more different layers of the network, and may compress and / or reduce the size of the data when processing data through different layers. For example, the data size at layer 1 is a higher resolution than the data at subsequent stages. In some embodiments, the data size at layer 1 is the full resolution of the captured image data, while the data at subsequent layers is a lower resolution (e.g., a quarter of the size) of the captured image data. In various embodiments, input data received from the image preprocessor 603 at subsequent layer(s) of the deep learning network 605 matches the internal data resolution(s) of data processed by one or more previous layers.

[0070] In some embodiments, the artificial intelligence (AI) processor 607 is a hardware processor for running the deep learning network 605. In some embodiments, the AI ​​processor 607 is a dedicated AI processor for performing reasoning on sensor data using a convolutional neural network (CNN). In some embodiments, the AI ​​processor 607 is optimized for the bit depth of the sensor data. In some embodiments, the AI ​​processor 607 is optimized for deep learning operations, such as neural network operations, including convolution, dot product, vector and / or matrix operations, etc. In some embodiments, the AI ​​processor 607 is implemented using a graphics processing unit (GPU). In various embodiments, the AI ​​processor 607 is coupled to a memory that is configured to provide instructions to the AI ​​processor that, when executed, cause the AI ​​processor to perform deep learning analysis on the received input sensor data and determine the results of machine learning for at least partially autonomously operating the vehicle.

[0071] In some embodiments, the vehicle control module 609 is used to process the output of the artificial intelligence (AI) processor 607 and convert the output into a vehicle control operation. In some embodiments, the vehicle control module 609 is used to control the vehicle for autonomous driving. In some embodiments, the vehicle control module 609 can adjust the speed and / or steering of the vehicle. For example, the vehicle control module 609 can be used to control the vehicle by braking, steering, changing lanes, accelerating and merging into another lane. In some embodiments, the vehicle control module 609 is used to control vehicle lighting, such as brake lights, turn lights, headlights, etc. In some embodiments, the vehicle control module 609 is used to control vehicle audio conditions, such as the sound system of the vehicle, play audio alarms, enable microphones, enable speakers, etc. In some embodiments, the vehicle control module 609 is used to control a notification system, which includes a warning system to notify the driver and / or passenger of a driving event, such as a potential collision or the proximity of a predetermined destination. In some embodiments, the vehicle control module 609 is used to adjust a sensor, such as a sensor 601 of a vehicle. For example, the vehicle control module 609 can be used to change parameters of one or more sensors, such as modifying orientation, changing output resolution and / or format type, increasing or decreasing capture rate, adjusting captured dynamic range, adjusting the focus of the camera, enabling and / or disabling sensors, etc. In some embodiments, the vehicle control module 609 can be used to change parameters of the image preprocessor 603, such as modifying the frequency range of filters, adjusting feature and / or edge detection parameters, adjusting channels and bit depth, etc. In various embodiments, the vehicle control module 609 is used to implement autonomous driving and / or driver assistance control of a vehicle.

[0072] In some embodiments, the network interface 611 is a communication interface for sending and / or receiving data including voice data. In various embodiments, the network interface 611 includes a cellular or wireless interface for interfacing with a remote server, connecting and making voice calls, sending and / or receiving text messages, etc. For example, the network interface 611 can be used to receive instructions and / or operating parameters for the sensor 601, the image preprocessor 603, the deep learning network 605, the AI ​​processor 607, and / or the vehicle control module 609. Update. For example, the machine learning model of the deep learning network 605 can be updated using the network interface 611. As another example, the network interface 611 can be used to update the firmware of the sensor 601 and / or the operating parameters of the image preprocessor 603, such as image processing parameters. In some embodiments, the network interface 611 is used to make emergency contact with emergency services in the event of an accident or near an accident. For example, in the event of a collision, the network interface 611 can be used to contact emergency services for help, and the location and collision details of the vehicle can be informed to the emergency services. In various embodiments, network interface 611 is used to implement autonomous driving features, such as accessing calendar information to retrieve and / or update a destination location and / or expected arrival time.

[0073] Although the foregoing embodiments have been described in detail for the purpose of clear understanding, the invention is not limited to the details provided. There are many alternative ways to implement the invention. The disclosed embodiments are illustrative rather than restrictive.

Claims

1. A method comprising: receiving an image captured using a sensor on a vehicle; decomposing the received image into a plurality of component images, wherein two or more different filtering passes are performed on the data, wherein the plurality of component images include a first component image and a second component image, the first component image including a feature component image and / or an edge component image, the feature component image excluding the edge component image, and the second component image including a global data component image; providing the first component image as input to a first layer of a plurality of layers of an artificial neural network ANN; providing the second component image and an intermediate result as input to a second layer of the plurality of layers of the ANN, wherein the intermediate result is determined based on providing the first component image to the ANN; determining a result based on providing the second component image and the intermediate result as input to the second layer of the plurality of layers of the ANN; as well as The results of the ANN are used to at least partially autonomously operate the vehicle. The method of claim 1 , wherein the sensor is a high dynamic range camera. 3 . The method of claim 1 , wherein the sensor is a camera sensor, a radar sensor, an ultrasonic sensor, or a LiDAR sensor.

4. The method of claim 1 , wherein decomposing the received image into the plurality of component images comprises: Use a high pass filter or a low pass filter.

5. The method of claim 1 , wherein decomposing the received image into the plurality of component images comprises: Use a high-pass filter, a band-pass filter, or a low-pass filter. 6 . The method of claim 1 , wherein a graphics processing unit is used to decompose the received image into at least one component image of the plurality of component images.

7. The method of claim 1, wherein a tone mapper processor is used to decompose the received image into at least one component image of the plurality of component images.

8. The method of claim 1, wherein the plurality of component images include high-pass data components and low-pass data components.

9. The method of claim 8, wherein the high-pass data component is decomposed using a first image preprocessor and the low-pass data component is decomposed using a second image preprocessor.

10. The method of claim 9, wherein the first image preprocessor and the second image preprocessor are different processors. The method according to claim 8 , wherein the low-pass data component is down-sampled from the received image.

12. The method of claim 8, wherein the low-pass data component has a lower resolution than the high-pass data component.

13. The method of claim 1, wherein each component image of the plurality of component images has a lower bit depth than the received image.

14. The method of claim 1, wherein the received image is pre-processed prior to decomposing the received image into the plurality of component images.

15. The method of claim 1, wherein at least one of the plurality of component images is pre-processed before providing each of the plurality of component images to the different layers of the plurality of layers of the artificial neural network.

16. The method of claim 15, wherein the pre-processing comprises performing one or more of: compression, denoising, demosaicing, local contrast enhancement, gain adjustment, or thresholding.

17. A computer program product embodied in a non-transitory computer readable storage medium and comprising computer instructions for executing the method of any one of claims 1 to 16.

18. A system comprising: Sensors on vehicles; Vehicle control module; One or more image signal processors configured to: receiving an image captured using the sensor; as well as decomposing the received image into a plurality of component images, wherein two or more different filtering passes are performed on the data, wherein the plurality of component images include a first component image and a second component image, the first component image including a feature component image and / or an edge component image, the feature component image excluding the edge component image, and the second component image including a global data component image; Artificial intelligence processors; as well as A memory coupled to the artificial intelligence processor, wherein the memory is configured to provide instructions to the artificial intelligence processor, the instructions, when executed, causing the artificial intelligence processor to: Receiving the first component image as input to a first layer of a plurality of layers of an artificial neural network ANN; receiving the second component image as input to a second layer of the plurality of layers of the ANN; providing an intermediate result as input to a second layer of the plurality of layers of the ANN, wherein the intermediate result is determined based on providing the first component image to the ANN; determining a result based on receiving the second component image and providing the intermediate result as input to the second layer of the ANN; as well as The results of the ANN are provided to the vehicle control module to at least partially autonomously operate the vehicle.

Images