Method for reducing calculation costs of automatic operation system

Abstract

To provide a method that can reduce calculation costs of an automatic operation system.SOLUTION: A method for reducing calculation costs of an automatic operation system includes: step (a) of acquiring data related to tasks for a vehicle operation; step (b) of training a deep learning model by using the acquired data, the deep learning model including an encoder and a policy head with respect to the tasks; step (c) of transferring the data acquired in step (a) to the encoder to generate a compressed latent representation of the relevant data and thereby reduce the complexity of the relevant data; and step (d) of determining a driving operation by allowing the policy head to use the compressed latent representation of the relevant data.SELECTED DRAWING: Figure 5

Description

[Technical field]

[0001] The present disclosure relates to the field of computer technology, and more particularly, to a method and / or apparatus for reducing the computational costs of an autonomous driving system. [Background technology]

[0002] As computational and vehicle technologies develop, automation features become more powerful and widely available, enabling vehicles to control a wider variety of environments. For example, for automobiles, the Society of Automotive Engineers (SAE) established a standard (J3016) that identifies six levels of driving automation ranging from "no automation" to "full automation." The SAE standard defines Level 0 as "no automation," where the human driver performs all dynamic driving tasks full-time, even if augmented by warning or intervention systems. Level 1 is defined as "driver-assisted," where the operator performs all remaining dynamic driving tasks by controlling steering or acceleration / deceleration (but not both) for at least some driving modes. Level 2 is defined as "partial automation," where the operator performs all remaining dynamic driving tasks by controlling steering and acceleration / deceleration for at least some driving modes. Level 3 is defined as "conditional automation," where the automated driving system performs all dynamic driving tasks for at least some driving modes, and expects the human driver to respond appropriately to intervention requests. Level 4 is defined as "high automation," where the automated driving system performs all dynamic driving tasks only for certain conditions, even if the human driver does not respond appropriately to a request to intervene. The specific conditions for Level 4 may be, for example, a specific type of road (e.g., highways) and / or a specific geographic area (e.g., a properly mapped geographically isolated metropolitan area). Finally, Level 5 is defined as "full automation," where the vehicle can operate without operator input under all conditions.

[0003] A fundamental challenge of any autonomous related technology involves collecting and describing information about the environment around the vehicle, as well as planning and executing commands to appropriately control the vehicle's movements to safely navigate the vehicle within the current environment. Thus, ongoing efforts are underway to improve each of these aspects so that autonomous vehicles can operate reliably in increasingly complex environments and adapt to expected and unexpected interactions within the environment. For example, to operate safely, autonomous vehicles must take into account objects such as vehicles, people, trees, animals, buildings, signs, poles, etc., when planning a path through the environment.

[0004] Autonomous driving systems must constantly monitor the surrounding environment, resulting in a large amount of information to be processed, so it is important to develop algorithms that reduce computational complexity while maintaining the flexibility and safety of autonomous driving operations. Summary of the Invention [Problem to be solved by the invention]

[0005] One objective of the present disclosure is to propose a method and / or apparatus for reducing the computational cost of an autonomous driving system. To this end, the present invention discloses various end-to-end deep learning models for autonomous vehicle control systems. The end-to-end deep learning models receive raw data from various sensors (e.g., cameras, LIDAR, etc.). The raw data can be collected directly from the sensors of the controlled vehicle. The raw data can be real-time recorded perception data (e.g., recorded driving) from any vehicle, or real-time shared perception data by another vehicle. The end-to-end deep learning models generate driving control decisions as output. Furthermore, the method of the deep learning models can use richer data that is obtained more rationally (e.g., less expensively) without being manually annotated.

[0006] Specifically, it can provide intermediate compressed or dimension-reduced latent representations of raw data for using / training an autonomous driving system to output driving control decisions when data is scarce, such as in reinforcement learning. Each embodiment discloses an encoder that converts raw data into a compressed latent representation. The compressed latent representation is significantly reduced in terms of data volume compared to the raw data. This improves the computational efficiency of the end-to-end deep learning model of the autonomous driving system in training mode and / or use mode. In one example, the encoder extracts useful features from the raw data for a particular driving task (e.g., lane centering, lane changing, traffic sign reading, etc.) and ignores the rest of the data. In another example, the encoder's feature extraction can be completed by various machine vision recognition, curve fitting, pattern recognition, text recognition, etc.

[0007] In another embodiment, the mask can be used to further sparsify the compressed latent representation, further reducing the amount of data processed and improving computational efficiency.

[0008] In some embodiments, a method for reducing computational costs of an automated driving system is disclosed, the method including: (a) acquiring data related to a task for a vehicle operation; (b) training a deep learning model using the acquired data, the deep learning model including an encoder and a policy head for the task; (c) transferring the data acquired in (a) to the encoder to generate a compressed latent representation of the data, thereby reducing the complexity of the data; and (d) determining a driving maneuver by the policy head using the compressed latent representation of the data.

[0009] In some embodiments, the acquired data includes recorded human driving data from the same vehicle or another vehicle. In some embodiments, the acquired data includes artificially augmented data. In some embodiments, the data is acquired using a sensor of the same vehicle or another vehicle, the sensor including one or more of a laser radar sensor, a radar sensor, an infrared sensor, and / or an image sensor.

[0010] In some embodiments, step (c) further comprises applying a mask obtained by multiplying elements of the compressed latent representation to further reduce the complexity of the data obtained in step (a). In some embodiments, step (c) further comprises normalizing values ​​of the mask.

[0011] In some embodiments, the method further includes evaluating a discrimination between the driving maneuver determined by the policy head and a driving maneuver benchmark by applying a loss function.

[0012] In some embodiments, the method further includes configuring one or more duplicated elements of the compressed latent representation generated by a first encoder of a first deep learning model so that the compressed latent representation can be shared by a second encoder of a second deep learning model.

[0013] In some embodiments, another method for reducing computational costs of an automated driving system is disclosed, the method including: (a) acquiring data related to a task for maneuvering a vehicle; (b) operating a deep learning model using the acquired data, the deep learning model including a policy head for the task; (c) acquiring a condensed latent representation of the data acquired in (a); and (d) the policy head using the condensed latent representation of the data to determine a driving maneuver.

[0014] In some embodiments, another method for reducing computational costs of an autonomous driving system is disclosed, the method including: (a) acquiring data related to a task for a vehicle operation; (b) training a first deep learning model using the acquired data, the first deep learning model including a first encoder and a policy head; (c) identifying one or more overlapping elements between the data related to the task and a compressed latent representation related to another task, the compressed latent representation being generated by a second deep learning model having a second encoder, the compressed latent representation being configured to be shareable by the first encoder of the first deep learning model; and (d) determining a driving maneuver by the policy head using the compressed latent representation, the compressed latent representation being generated by the second deep learning model having a second encoder.

[0015] In some embodiments, the disclosed method may be operated by an apparatus of an automated driving system. The apparatus may include at least one processor and a memory storing instructions. The instructions, when executed by the at least one processor, cause the at least one processor to perform operations of the disclosed method for reducing computational costs of an automated driving system. For example, in some embodiments, the disclosed method may be programmed as computer-executable instructions stored on a non-transitory computer-readable medium. The non-transitory computer-readable medium, when loaded into a computer, instructs a processor of the computer to perform the disclosed method. The non-transitory computer-readable medium may include any one or more of a hard disk, a CD-ROM, an optical storage device, a magnetic storage device, a read-only memory, a programmable read-only memory, an erasable programmable read-only memory, an EPROM, an electrically erasable programmable read-only memory, and a flash memory.

[0016] All combinations of the above concepts and additional concepts described in more detail herein are to be considered as part of this disclosure, for example, all combinations of claimed subject matter appearing at the end of this disclosure are to be considered as part of the subject matter disclosed in the body. [Brief description of the drawings]

[0017] In order to more clearly illustrate the embodiments of the present disclosure and related art, the figures described in the following embodiments are briefly introduced. Obviously, these figures are only some embodiments of the present disclosure, and those skilled in the art can obtain other figures without creative effort by referring to these figures. Arrows in the figures indicate relationships, by which the components where the arrows start can be used to train / apply the components where the arrows point. These figures can be used in combination with the following detailed description to more fully understand the embodiments of the present disclosure. [Figure 1] FIG. 1 is a block diagram of an example implementation of a deep learning model when training or using according to some embodiments of the present disclosure. [Diagram 2] FIG. 13 is a block diagram of another example implementation of a deep learning model having a mask function when training or using the model in accordance with some embodiments of the present disclosure. [Diagram 3] FIG. 1 is a block diagram of an example implementation of a method for training or using multiple deep learning models with masking capabilities according to some embodiments of the present disclosure. [Figure 4] 1 is a table illustrating an example of identifying one or more overlapping elements associated with different tasks according to some embodiments of the present disclosure. [Diagram 5] 1 is a flowchart illustrating an example of operations when an end-to-end deep learning model trains according to some embodiments of the present disclosure. [Figure 6] 1 is a flowchart illustrating an example of operations when used by an end-to-end deep learning model according to some embodiments of the present disclosure. [Figure 7]1 is a flowchart illustrating an example of operations when multiple deep learning models are training according to some embodiments of the present disclosure. [Figure 8] FIG. 1 illustrates an example of operating an automated driving system without using a task function according to some embodiments of the present disclosure. [Figure 9] FIG. 1 illustrates an example of operating an automated driving system when using a mask function for lane centering / keeping tasks in accordance with some embodiments of the present disclosure. [Figure 10] FIG. 1 illustrates an example of operating an automated driving system when using a mask function for a traffic sign reading task according to some embodiments of the present disclosure. [Figure 11] FIG. 1 illustrates an example hardware and software environment for an autonomous vehicle according to some embodiments of the present disclosure. For simplicity and clarity of the drawings, the components illustrated in the figures are not necessarily drawn to scale. For example, the size of some components may be larger than others for clarity. Also, where appropriate, additional graphical symbols may be repeated to indicate corresponding or similar components. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0018] With reference to the drawings, the embodiments of the present disclosure are described in detail with respect to technical problems, structural characteristics, objectives to be achieved, and effects. Specifically, the terms in the embodiments of the present disclosure are used only to describe specific embodiments, and do not limit the present disclosure. In the following detailed description, many specific details are described in order to fully understand the present invention. However, it should be understood that those skilled in the art can practice the present invention without these specific details. In other circumstances, known methods, processes, and components are not described in detail to avoid confusion. The subject matter of the present invention is specifically pointed out and explicitly protected in the last part of this specification. However, the organization, operation method, objectives, features, and advantages of the present invention can be best understood by referring to the following detailed description and reading it in combination with the drawings. Because the illustrated embodiments of the present invention can be realized mainly using electronic components and circuits that are already known to those skilled in the art. Therefore, no more details than are deemed necessary to understand the basic concept of the present invention and not to confuse or scatter the teachings of the present invention will be described. For example, the specification and / or drawings may involve a processor or a processing circuit. The processor may be a processing circuit. The processing circuitry may be implemented as a central processing unit (CPU) and / or one or more other integrated circuits, such as an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a fully custom integrated circuit, or a combination of these integrated circuits.

[0019] The following specification and / or drawings may involve images. An image is an example of a media unit. Any reference to an image may be applied to a media unit as appropriate. A media unit may be an example of a sensing information unit (SIU). Any reference to a media unit may be applied to any type of natural signal as appropriate, such as, but not limited to, naturally generated signals, signals representing human actions, signals representing actions related to vehicular signals, geodetic signals, geophysical signals, text signals, digital signals, time series signals, etc. Any reference to a media unit may be applied to an SIU as appropriate. An SIU may be of any type and may be sensed by any type of sensor, for example, a vision camera. Acoustic sensors may sense infrared, radar imaging, ultrasonic, electro-optical, radiography, light detection and ranging (LIDAR), thermal sensors, passive sensors, active sensors, etc. Sensing may include generating samples (e.g., pixels, audio signals, etc.) representative of a signal transmitted or otherwise reaching the sensor. An SIU may contain one or more images, one or more video clips, text information about one or more images, or text describing motion information, and so on.

[0020] Any combination of any modules or units listed in any of the accompanying drawings, any part of the specification, and / or any claim may be provided. Any of the units and / or modules illustrated in this application may be realized using codes, commands and / or instructions stored in hardware and / or non-transitory computer readable media and may be included inside a vehicle, outside a vehicle, on a mobile device, on a server, etc. The vehicle may be any type of vehicle, such as, for example, a ground transportation vehicle, an air vehicle, or a water transportation tool. The vehicle may also be referred to as a private vehicle. It should be understood that autonomous driving includes at least partially autonomous (semi-autonomous) driving of a vehicle, including all L2 level types or higher level types defined by the SAE standard.

[0021] Referring now to the drawings, where like numbers in the several drawings represent like parts, FIGS. 1-3 are block diagrams illustrating different end-to-end deep learning models performing a method for reducing computational costs of a system according to some embodiments of the present disclosure. The system includes, but is not limited to, autonomous driving. The model may comprise a computer system for training and / or operating one or more models, which may be artificial intelligence (AI) models, including, but not limited to, deep learning models, such as deep neural networks having one or more latent layers / representations. It should be understood that these and other arrangements described herein are described by way of example only. Other arrangements and elements (e.g., equipment, interfaces, functions, sequences and groupings of functions, etc.) can be used in addition to or in place of those shown. And, some elements may be omitted entirely for clarity. Also, many elements described herein are functional entities that may be implemented as discrete or distributed components or combined with other components, and in any suitable combination and location. Various functions performed by one or more entities described herein may be implemented by hardware, firmware and / or software. For example, some functions may be performed by a processor executing commands stored in a memory. It should be understood that the deep learning models described below may be neural networks that include multiple layers, such as an input layer, a latent / hidden layer, and an output layer. Each neural network layer may include multiple nodes (or neurons), which are typically connected in series.

[0022] In some embodiments, FIG. 1 is a block diagram of a deep learning model 100 including an encoder 104 and a policy head 108. Each of the encoder 104 and the policy head 108 may be a trainable AI model. In one embodiment, the deep learning model is determined by a task, for example, lane centering, lane changing, traffic sign reading, etc. The encoder 104 receives raw data 102. The encoder 104 extracts features from the raw data 102 and accordingly reduces the high-dimensional raw data to a low-dimensional latent vector as a compressed latent representation 106. In this manner, the data body / complexity of the raw data 102 can be reduced by forming the compressed latent representation 106. The compressed latent representation 106 of the raw data 102 contributes to learning data characteristics and simplifying the data representation. The encoder 104 outputs the compressed latent representation 106 to the policy head 108. The policy head 108 receives the compressed latent representation 106 and outputs a driving maneuver decision 110 in response. In different embodiments, the compressed latent representation 106 is obtained by discarding redundant or worthless data and / or using different data representation and approximation techniques (i.e., transferring less data losslessly and transferring compact models instead of raw data). For example, in some embodiments, the compressed latent representation 106 can apply linear or non-linear transformations to the raw data 102 and is used to generate the output driving maneuver decision 110.

[0023] In some embodiments, the raw data 102 is raw data from one or more sensors of the same vehicle or another vehicle. For example, the raw data 102 may be an image including red, green, and blue (RGB) values ​​of pixels captured by a camera sensor. The raw data 102 may be raw SIUs, processed SIUs, text information, information derived from SIUs, and the like. In different embodiments, the loading of the raw data 102 may originate from a local disk, from a remote storage location via a suitable network location, and the like. Obtaining the raw data 102 may include receiving data, generating data, participating in processing the data, processing only a portion of the data, and / or receiving only another portion of the data. Processing the data 102 may include at least one of detection, noise reduction, improving the signal-to-noise ratio, defining a bounding box, and the like. The raw data 102 may be received from one or more sources, such as one or more sensors, one or more communication units, one or more memory units, one or more image processors, and the like.

[0024] In some embodiments, the encoder 104 may be configured to map the raw data 102 to a compressed latent representation 106, which may be stored in a database of semantic relations. In some embodiments, the encoder 104 learns a dimensionality reduction of the input data to encode a latent representation of features, while the policy head 108 recreates the encoded latent representation as a reconstructed output, such as an output maneuver decision 110. For example, the encoder 104 may be configured to generate the compressed latent representation 106 of the raw data 102 using a one-dimensional vector representing one or more elements of the raw data 102. In one embodiment, the compressed latent representation may be represented as a vector V, where V=[E1, E2, E3, ... E N ]. E1 is element 1, E2 is element 2, E3 is element 3, E Nrefers to element N. Each element may be a one-dimensional or multi-dimensional matrix, for example, referring to Figure 4. Each element may represent a potentially useful feature around the vehicle, such as lane boundaries, lane centerlines, nearby vehicles, traffic signs, tree outlines, etc.

[0025] The encoder 104 may be configured to encode meaningful information about various data attributes in the latent representation, which is then used to perform the associated task. In such an embodiment, the compressed latent representation 106 serves to reduce the dimensionality of the input data and remove irrelevant information. Thus, reducing the dimensionality of the input data reduces computational costs and helps to avoid over-fitting.

[0026] In some embodiments, given the compressed latent representation 106, the policy head 108 may be configured to determine an action for the vehicle to follow from a set of predefined tasks. The tasks determine the action that the autonomous vehicle should take based on the compressed latent representation 106. Some examples of these tasks are lane keeping, overtaking, lane changing, intersection handling, and signal handling.

[0027] In some implementations, the gradient of the loss function 112 may be configured to evaluate the discrimination between the driving maneuver decision 110 determined by the policy head 108 and the driving maneuver benchmark 114 to characterize the accuracy of the compressed latent representation 106. The loss function is a measure of how good a predictive model is at predicting the expected outcome. The parameters of the encoder 104 and / or the compressed latent representation 106 may be updated / adjusted based on the gradient of the loss function 112 to achieve an improved driving decision output. It should be understood that the loss function 112 may not be necessary if the model is used only for steering and not for training purposes. However, it is possible to simultaneously operate and train the system.

[0028] 2, deep learning model 200 may include a similar structure to deep learning model 100 and may further include a trainable mask 208 to further reduce the complexity of the input data and thereby reduce computational costs. Trainable mask 208 may be configured to generate a sparse latent representation 212 for policy head 214 based on compressed latent representation 206 to generate driving maneuver decision 216.

[0029] For example, in some implementations, the trainable mask 208 is obtained by multiplying elements to the compressed latent representation 206 generated by the encoder 204 based on the set of raw data 202. In one implementation, the trainable mask 208 may be a vector having elements that match the compressed latent representation 206. The trainable mask 208 may zero out or normalize less useful elements in the compressed latent representation 206 to further sparsify the data.

[0030] For example, in model 200 processing a lane change task, mask 208 may retain elements of compressed latent representation 206, namely lane boundary lines [E1], lane centerlines [E2], other vehicles [E3], and traffic sign text [E4], but may zero out tree contour [E5] because model 200 determines that tree contours are less useful for lane change tasks. Thus, in this example, if the compressed latent representation is a vector V=[E1, E2, E3, E4, E5], sparse latent representation 212 may be a vector V sparse = [E1, E2, E3, E4, 0], where "0" represents the zero matrix.

[0031] In some embodiments, the trainable mask 208 may be configured to determine the output driving maneuver decision 216 by mapping from the compressed latent representation 206 to generate a sparse latent representation 212 received by the policy head 214. For example, the mask values ​​of the trainable mask 208 may be normalized between 0 and 1 (e.g., transferring the trainable parameters to a sigmoidal function to promote data sparsity). In some embodiments, the loss function 212 may be configured to compare the output of the policy head 216 to the driving maneuver benchmark 214. The mask values ​​may be added to the loss function 212 in the form of an L1 regularization loss, which sets many of the mask values ​​of the trainable mask 208 to zero by adding the absolute values ​​of the mask elements for better data sparsity used for the sparse latent representation 212. It should be understood that the loss function 212 may be applied to the encoder 204 and / or the mask 208 to improve system performance.

[0032] In some embodiments, the compressed latent representation may be configured to be shared among multiple end-to-end deep learning models. For example, as shown in FIG. 3, two deep learning models 300a and 330b are shown in training. The deep learning model 300a includes an encoder 304a and a policy head 312a. In some embodiments, a trainable mask 308a is obtained by multiplying an element by a compressed latent representation 306a generated by the encoder 304a based on a set of raw data 302a. In such embodiments, the trainable mask 308a may be configured to map from the compressed latent representation 306a to generate a sparse latent representation 310a, which is received by the policy head 312a and used to determine an output maneuver decision 314a. In some implementations, the gradient of the loss function 324a may be configured to evaluate the discrimination between the driving maneuver decisions 314a determined by the policy head 312a and the driving maneuver benchmark 326a to characterize the accuracy of the compressed latent representation 306a and / or the sparse latent representation 310a. The loss function 324a may be applied to the encoder 304a, the mask 308a, or the policy head 312a to improve system performance.

[0033] The deep learning model 300b includes the same structure as the deep learning model 300a. In some embodiments, the data sharing module 318 may be configured to identify one or more overlapping elements between the data and the sparse latent representations 310a, 310b of the deep learning models 300a, 300b such that the one or more sparse latent representations 310a, 310b are configured to be shareable by the encoders 304a, 304b of the two deep learning models. The shared latent representations further improve the computational efficiency of the autonomous driving system.

[0034] In one embodiment, deep learning model 300a is used for lane changing and deep learning model 300b is used for lane centering. Data sharing module 308 can compare elements of sparse latent representations 310a and 310b. Sparse latent representation 310a for lane changing may include elements of lane lines, lane centerlines, other vehicles, and traffic sign text. Sparse latent representation 310b for lane centering may include elements of lane lines and lane centerlines. Data sharing module 318 determines overlapping elements of 310a and 310b, e.g., lane lines and lane centerlines. Data sharing module 318 then creates a shared latent representation 316 and sends it to encoders 304a, 304b, or any other encoders that may require such overlapping elements. Data sharing module 318 can also upload 322 or download 322 overlapping elements to / from network 320. One or more elements storing the compressed latent representations on the network 320 contribute to further sharing between deep learning models at different times.

[0035] The sharing capability may be configured to optimize the trainable latent representation and mask function. To better illustrate the characteristics of the shareable compressed / sparse latent representation, FIG. 4 is a table showing an example of identifying one or more overlapping elements related to different tasks (e.g., lane change task (task 1), lane centering task (task 2), and traffic sign reading task (task 3)). As shown, both lane change task and lane centering task require elements related to lane boundary and lane center to complete the task. Thus, in some embodiments, the overlapping elements (e.g., lane boundary and lane center) of the sparse latent representation 310 may be shared from one deep learning module to another deep learning model 300 by the data sharing module 318.

[0036] In some embodiments, the data sharing module 318 may provide a populated sparse latent representation 310 for one task and directly apply it to another task in the same or a different vehicle. This method of sharing overlapping elements further reduces computational costs, since the system receiving the shareable elements may not need to generate its own compressed or sparse latent representation. In some embodiments, element sharing may be performed over a network 320, including but not limited to Wi-Fi, DSRC connections, etc. Data sharing may be extended beyond autonomous vehicles to autonomous robots, autonomous transport robots, or other systems capable of autonomous navigation via machine learning models.

[0037] In some embodiments, one or more networks 320 (e.g., a LAN, a WAN, a wireless network, and / or the Internet, etc.) may be provided to enable communication of information with other data sharing modules 318, computers, and / or electronic devices (e.g., including a central service, such as a cloud service, from which the data sharing module 318 receives sharable compressed / sparse latent representations, environmental data, and other data for its automated control). For example, in some embodiments, one or more predefined latent representations 316 are configured to be shared by the data sharing module 318 and the network 320. In such embodiments, the shared latent representations 316 configured by the local model may be uploaded 322 to the network 320 (e.g., a cloud system) and stored on the network 320 for use by other models remotely. The data sharing module 318 may then download 322 the shared latent representations 316 from the network 320 for local use. In different embodiments, the data sharing module 318 may be a tangible or intangible entity, such as an entity that is physically constructed, specifically configured (e.g., hardwired), or otherwise configured (e.g., programming) to operate in a specified manner or to operate or perform some or all of any operations described herein. Data sharing is used for one type of data or multiple types of data, one-time use, multiple use, and / or permanent use. Shared data may be collected and distributed in the original uploaded format or may be further processed before sharing. Shared data may be transmitted in real time or near real time.

[0038] Now, referring to FIG. 5-FIG. 7, these figures show three methods 500, 600, 70 that can be used to reduce the computational cost corresponding to the above-mentioned model. It should be noted that the order of the methods 500, 600, and 700 is exemplary and does not indicate the order of steps performed by the methods 500, 600, and 700. As shown in FIG. 5, the operation method 500 can begin in block 502 by acquiring data related to a task for vehicle operation (e.g., lane change, lane centering, traffic sign reading, etc.) and training a deep learning model using the acquired data in block 504. As described above, the deep learning model may include an encoder and a policy head for the task. Thereafter, in block 506, the data can be transferred to an encoder to generate a compressed latent representation of the data, thereby reducing the complexity of the data acquired in block 502, and in block 510, the policy head uses the compressed latent representation of the data to determine the driving maneuver.

[0039] In some embodiments, the data acquired in block 502 may include recorded human driving data from the same vehicle or another vehicle and may be acquired from one or more sensors. For example, in some embodiments, the acquired data may be acquired from a storage device / memory that includes recorded human driving data. In some embodiments, the recorded human driving data may be a vehicle data log that has completed a driving session from a conventional vehicle, a driving simulation system, or an autonomous vehicle. For example, when an autonomous vehicle performs a driving session, the autonomous vehicle or an associated computing system may collect and store human driving data and / or vehicle data. After the session is completed, the log of recorded data may be transmitted to a computing system, such as a cloud system, for training or use by the autonomous system as described above.

[0040] In some embodiments, different types of sensors (e.g., laser radar sensors, radar sensors, infrared sensors, and / or image sensors) can be used to generate data capturing each aspect of the driving environment to collect and mark enough data to train a model to control vehicle actions as described above. However, not all data is equally useful or used to train a model. Some data may be noisy, incomplete, or unbalanced. To overcome these limitations, in some embodiments, the data used to train the model obtained in block 502 may be treated as artificially augmented data. Data augmentation techniques can improve the quality and diversity of the data by applying transformations such as cropping, flipping, rotating, scaling, adding noise, changing brightness, interpolating, creating 3D models or blending images. These techniques help the model learn more robust and generalizable features to improve its performance and accuracy. For example, some elements (e.g., animals, bad weather conditions, traffic signals, etc.) can be introduced into the input data to improve the training results.

[0041] In some embodiments, to further reduce computational costs, the method of operation 500 may further include block 508, applying a mask obtained by multiplying elements of the compressed latent representation to further reduce the complexity of the data obtained in block 502. In some embodiments, the values ​​of the mask may be normalized as described above. In some other embodiments, the method of operation 500 may further include evaluating discrimination between the driving maneuver determined by the policy head and a driving maneuver benchmark by applying a loss function in block 512 to improve system performance.

[0042] In some embodiments, as shown in FIG. 6, a method 600 illustrates the operation of applying the trained model. The method 600 can begin by obtaining data related to a task for a vehicle operation (e.g., lane changing, lane centering, traffic sign reading, etc.) in block 602, and training a deep learning model using the obtained data in block 604. The deep learning model may include a policy head for the task. A compressed latent representation is then obtained in block 606 for application. In one example, obtaining in 606 may be extracting useful features from raw data using an encoder. In another example, obtaining in 606 may be downloading the compressed latent representation from a network or any non-volatile electronic storage medium. Using the compressed latent representation, a driving maneuver can be determined by the policy head in block 610. Similar to method of operation 500, method of operation 600 includes block 608 of further reducing the complexity of the acquired data by applying a mask obtained by multiplying the elements of the compressed latent representation, and block 612 of evaluating the discrimination between the driving maneuver determined by the policy head and the driving maneuver benchmark as described above by applying a loss function to improve system performance.

[0043] In some embodiments, the compressed latent representation may be configured to be sharable among multiple deep learning models. For example, as shown in FIG. 7, method 700 may include the first two steps similar to method 500. At block 702, the method may include obtaining data related to a task for vehicle operation, and at block 704, training a first deep learning model using the obtained data with a first encoder and a policy head, the first deep learning model including the first encoder and the policy head. The operation may then continue at block 706 by identifying one or more overlapping elements between the data related to the task and a compressed latent representation related to another task (as shown in FIGS. 3-4), the compressed latent representation being generated by a second deep learning model having a second encoder. Thus, in such embodiments, the compressed latent representation generated by the second deep learning model having a second encoder is configured to be sharable by the first encoder of the first deep learning model. In block 708, the policy head determines the driving maneuver by using the compressed latent representation, where the compressed latent representation is generated by a second deep learning model having a second encoder, thereby reducing the computational cost of the first deep learning model.

[0044] Figures 8-10 are different views of different examples of operating an autonomous driving system for different tasks. Figures 8-10 show examples of raw data 102, 202, 302, compressed latent representations 106, 206, 306, and sparse latent representations 212, 310 of Figures 1-3, where applicable. Figures 8-10 show examples of raw data, compressed latent representations, and masked data (i.e., sparse latent representations) of Figures 5-7, where applicable.

[0045] As shown in FIGS. 8-10, the input images 802, 902, and 1002 may be raw data from a camera sensor, and the compressed latent representations 804, 904, and 1004 may be compact representations of the input images capturing useful features generated by the encoders 104, 204, and 304. As shown here, the input images may be high-dimensional, from which raw data (e.g., RGB images) for the surrounding environment of an autonomous vehicle are usually high-dimensional. The raw data images include not only images of the road, but also images of the scene around the road (e.g., other vehicles, trees, traffic signs, and sky). In comparison, in some implementations, the compressed latent representations only retain some regions of interest, and color images are usually converted to black / white images. For example, as shown in FIG. 8 , a compressed latent representation 804 may be converted from the original color image 802 to a black / white line image 804 that contains only tree contours 806, lane boundary lines 808, first lane centerline 810a, second lane centerline 810b, traffic sign contours 812, traffic sign text 814 and other vehicles 816.

[0046] In some embodiments, gamma correction may be performed on the input raw image 802 first to improve image adaptability, and then image binarization may be performed to convert the image from color to black and white. In some embodiments, after image binarization processing, morphological operations may be used to repair cavities to smooth the boundaries, and then a skeleton extraction algorithm may be used to extract the lane centerlines (e.g., 810a, 810b). In some embodiments, local filtering may be performed by using the Hough transform result to remove interference and obstructions. In different embodiments, the lane boundary lines 808 may be guardrails, an interface between asphalt and grass, or another indicator of lane boundaries. Although shown here as single dashed / solid lines, the lane markings 808, 810a, and 810b may be solid or double lines (e.g., double solid lines, dashed solid lines), etc. The purpose of the image downgrading / downsampling operation is to reduce the size of the image used for the compressed latent representation to reduce computational costs. In such an embodiment, extracting a latent representation from an input image refers to a compact and low-dimensional representation of the acquired image that embodies the fundamental features and patterns contained within the image.

[0047] As shown in FIG. 9-FIG. 10, in some embodiments, the sparse latent representations 918 and 1018 may be the sparse latent representations 212, 310. The sparse latent representations 918 and 1018 may be further simplified from the compressed latent representations 904 and 1004, respectively, to provide drawings with fewer elements. The fewer elements are only basic elements that maintain the policy head for determining driving maneuver decisions based on different tasks. The purpose of the masking operation is to selectively keep or discard some pixel values. For example, for the task of lane centering as shown in FIG. 9, only elements related to lane conditions and other vehicles, such as lane boundary lines 908, first lane centerline 910a, second lane centerline 910b, and other moving vehicles 916, are left in the sparse latent representation 918. From the compressed latent representation image 904, a sparse latent representation image 918 is generated to be used for better data sparsity, removing elements not needed for the lane centering task, such as tree contours 906, traffic sign contours 912, traffic sign text 914, and other irrelevant elements such as other vehicles 916. Similarly, for the task of traffic sign reading, only elements related to traffic signs, such as traffic sign contours 1012 and traffic sign text 1014, are retained in the sparse latent representation 1018 compared to the compressed latent representation 1004 used for this particular task.

[0048] In some embodiments, the above functions / features may be implemented in hardware, software, firmware, or any combination thereof. When implemented in software, the functions may be stored as one or more commands or codes in a non-transitory computer-readable storage medium or a non-transitory processor-readable storage medium. The blocks of the methods or algorithms disclosed herein may be implemented in a processor-executable software module, which may reside on a non-transitory computer-readable storage medium or a processor-readable storage medium. The non-transitory computer-readable storage medium or a processor-readable storage medium may be any storage medium accessible by a computer or a processor. By way of example and not limitation, such non-transitory computer-readable storage medium or a processor-readable storage medium may include RAM, ROM, EEPROM, flash memory, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage, or any other medium used to store desired program code in the form of commands or data structures and accessible by a computer. As used herein, disks and optical disks include compact disks (CDs), laser disks, optical disks, digital versatile disks (DVDs), floppy disks, and Blu-ray disks, where disks typically reproduce data magnetically, whereas disks reproduce data optically with laser light. Combinations of the above are also included within the scope of non-transitory computer-readable media and processor-readable media. Furthermore, the operations of a method or algorithm may reside as one or any combination or set of code and / or commands in a non-transitory processor-readable storage medium and / or computer-readable storage medium, which may be incorporated into a computer program product.

[0049] FIG. 11 illustrates an exemplary hardware and software environment for an autonomous vehicle 1100 in which various techniques disclosed herein can be implemented. For example, the vehicle 1100 is shown traveling on a road 1101, and includes a power system 1102 including a prime mover 1106 that can be powered by an energy source 1104 and provide power to a drivetrain 1108, and a vehicle operation system 1110 including directional control 1112, power system control 1114, and brake control 1116. The vehicle 1100 can be implemented as any number of different types of vehicles, including vehicles that can transport people and / or cargo, travel over land, over sea, travel in the air, underground, under the sea, and / or in space. And, it should be understood that the above components 1102-1116 can vary widely based on the type of vehicle in which these components are used.

[0050] For simplicity, the embodiments described below focus on wheeled land vehicles such as cars, vans, trucks, buses, motorcycles, all-terrain vehicles (ATVs), etc. In such embodiments, the energy source 1104 may include, for example, a fuel system (e.g., providing gasoline, diesel, hydrogen, etc.), a battery system, solar panels, or other renewable energy, and / or a fuel cell system. The prime mover 1106 may include one or more motors and / or internal combustion engines, etc. The drive train 1108 may include wheels and / or tires, a driveline and / or any other mechanical driving components suitable for converting the power output of the prime mover 1106 into vehicle motion, as well as one or more brakes configured to controllably stop or slow the vehicle 1100, and a steering or steering component suitable for controlling the trajectory of the vehicle 1100 (e.g., a rack gear and pinion steering linkage, whereby one or more wheels of the vehicle 1100 can pivot about a substantially vertical axis to change the angle of the plane of rotation of the wheels relative to the longitudinal axis of the vehicle). In some embodiments, a combination of power system and energy source may be used (e.g., in the case of an electric / gas hybrid vehicle), and in other embodiments, multiple motors (e.g., dedicated to a single wheel or axle) may be used as prime mover 1106. In the case of a hydrogen fuel cell embodiment, prime mover 1106 may include one or more motors, and energy source 1104 may include a fuel cell system powered by hydrogen fuel.

[0051] Directional control 1112 may include one or more actuators or sensors for controlling and receiving feedback from directional or steering components to enable vehicle 1100 to follow a desired trajectory. Powertrain control 1114 may be configured to control the speed and / or direction of vehicle 1100 by controlling the output of drivetrain 1102 (e.g., controlling the output power of prime mover 1106, controlling driveline gears in drivetrain 1108, etc.). Brake control 1116 may be configured to control one or more brakes, e.g., disc or drum brakes coupled to the wheels of the vehicle, to slow or stop vehicle 1100.

[0052] Other vehicle types (including, but not limited to, all-terrain or tracked vehicles, and construction equipment) may use different power systems, drive trains, energy sources, directional control, power system control, and brake control. Also, in some embodiments, some components may be combined, for example, directional control of a vehicle is handled primarily by modifying the power output of one or more prime movers. Thus, the embodiments disclosed herein are not limited to the specific application of the techniques described herein in autonomous, wheeled, or land vehicles.

[0053] In the illustrated embodiment, full or semi-automatic control of the vehicle 1100 is realized in a main vehicle control system 1118. The main vehicle control system 1118 may include one or more processors 1122 configured to execute program code commands 1126 stored in a memory 1124, and one or more memories 1124. The processor 1122 may include, for example, a graphic processing unit(s) (GPU) and / or a central processing unit (CPU). The processor 1122 may further include an application specific integrated circuit (ASIC), other chipset, logic circuitry, and / or data processing device. The memory 1124 is used, for example, to load and store data and / or commands for the control system 1118. The memory 1124 may include any combination of suitable volatile memory (e.g., read only memory (ROM), dynamic random access memory (DRAM), random access memory (RAM), non-volatile memory (e.g., flash memory, memory cards, storage media), and / or other storage devices. When an embodiment is implemented in software, the techniques described herein may be implemented with modules, processes, functions, entities, etc. that perform the functions described herein. Modules may be stored in memory and executed by a processor. Memory may be implemented within or external to the processor and may be communicatively coupled to the processor via various means known in the art.

[0054] The sensors 1130 may include various sensors suitable for collecting information from the vehicle's surrounding environment to control the operation of the vehicle 1100. For example, the sensors 1130 may include one or more detection and ranging sensors (e.g., RADAR sensor 1134, LIDAR sensor 1136, or both), satellite navigation (SATNAV) sensor 1132, such as those compatible with any of the various satellite navigation systems (e.g., Global Positioning System (GPS), Global Navigation Satellite System (GLONASS), Beidou Navigation Satellite System (BDS), Galileo, Compass), etc. The radio detection and ranging (RADAR) 1134, light detection and ranging (LIDAR) sensor 1136, and digital camera 1138 (which may include various types of image capture devices capable of capturing still and / or video images) are used to sense stationary and moving objects within the vehicle's immediate area. The camera 1138 may be a monochrome camera or a stereo camera and may record still and / or video images. The SATNAV sensor 1132 is used to determine the vehicle's position on Earth using satellite signals. The sensors 1130 may optionally include an inertial measurement unit (IMU) 1140. The IMU 1140 may include multiple gyroscopes and accelerometers capable of detecting linear and rotational motion in three directions of the vehicle 1100. One or more other types of sensors (e.g., wheel rotation sensor / encoder 1142) are used to monitor the rotation of one or more wheels of the vehicle 1100.

[0055] In various embodiments, the removable hardware pod is unknown to the vehicle and can be attached to a variety of non-autonomous vehicles, including cars, buses, vans, trucks, mopeds, tractor trailers, exercise vehicles, and the like. While autonomous vehicles typically include a full sensor suite, in many embodiments the removable hardware pod may include a dedicated sensor suite. This dedicated sensor suite typically has fewer sensors than a fully autonomous vehicle sensor suite and may include an IMU, a 3D positioning sensor, one or more cameras, a LIDAR unit, and the like. Additionally or alternatively, the hardware pod may collect data from the non-autonomous vehicle itself, such as by integrating with the vehicle's CAN bus to collect various vehicle data including vehicle speed data, braking data, steering control data, and the like. In some embodiments, the removable hardware pod may include computing facilities that aggregate data collected by the removable pod sensor suite and vehicle data collected from the CAN bus and upload the collected data to a computing system for further processing (e.g., uploading the data to the cloud). In many embodiments, the computing equipment in the removable pod can apply a timestamp to each instance of the data before uploading the data for further processing. Additionally or alternatively, one or more sensors in the removable hardware pod can apply a timestamp when the data is collected (e.g., a laser radar unit can provide its own timestamp). Similarly, the computing equipment in the autonomous vehicle can apply a timestamp to data collected by the autonomous vehicle's sensor suite and upload the time-stamped autonomous vehicle data to a computer system for additional processing.

[0056] The output of the sensors 1130 may be provided to a set of main control subsystems 1120, including, for example, a positioning subsystem, a perception subsystem, a planning subsystem, and a control subsystem. The positioning subsystem is primarily responsible for accurately determining the position and orientation (sometimes referred to as "attitude" or "attitude estimate") of the vehicle 1100 within its surrounding environment, typically within some frame of reference. In some embodiments, the attitude is stored in memory 1124 as positioning data. In some embodiments, a surface model is generated from a high-resolution map and stored in memory 1124 as surface model data. In some embodiments, detection and ranging sensors store their sensor data in memory 1124 (e.g., radar data point clouds are stored as radar data). In some embodiments, calibration data is stored in memory 1124. The perception subsystem is primarily responsible for detecting, tracking, and / or identifying objects within the environment surrounding the vehicle 1100. According to some embodiments, machine learning models such as those described above are used to plan the vehicle's trajectory. The control subsystem 1120 is primarily responsible for generating appropriate control signals to control various controls in the vehicle control system 1118 to achieve the planned trajectory of the vehicle 1100. Similarly, machine learning models are used to generate one or more signals to control the autonomous vehicle 1100 to achieve the planned trajectory.

[0057] It should be understood that the collection of components for the vehicle control system 1118 shown in FIG. 11 is merely an example. In some embodiments, individual sensors may be omitted. Additionally or alternatively, in some embodiments, multiple sensors of the same type shown in FIG. 11 are used for redundancy and / or coverage of different areas around the vehicle. Also, there may be additional sensors of other types to provide actual sensor data related to the operation and environment of the wheeled land vehicle, in addition to the types described above. Similarly, in other embodiments, different types of control subsystems and / or combinations of control subsystems may be used. Additionally, while the primary control subsystem 1120 is shown as separate from the processor 1122 and memory 1124, it will be understood that in some embodiments, some or all of the functionality of the primary control subsystem 1120 may be implemented using program code commands 1126 resident in one or more memories 1124 and executed by one or more processors 1122, and in some cases, the primary control subsystem 1120 may be implemented using the same processor and / or memory. The subsystems may be implemented, at least in part, using various special purpose circuit logic, various processors, various field programmable gate arrays (FPGAs), various application specific integrated circuits (ASICs), various real time controllers, etc. As discussed above, the multiple subsystems may utilize circuits, processors, sensors, and / or other components. Additionally, the various components within the vehicle control system 1118 may be networked in various ways.

[0058] For example, vehicle 1100 may include one or more network interfaces, e.g., network interface 1154 adapted to communicate with one or more network interfaces 1150 (e.g., a LAN, a WAN, a wireless network, and / or the Internet, etc.) to enable communication of information with other vehicles, computers and / or electronic devices, including a central service, such as a cloud service, from which vehicle 1100 receives environmental and other data for automatic control.

[0059] For additional storage, the vehicle 1100 may also include one or more mass storage devices, such as a floppy disk or other removable disk drive, a hard disk drive, a direct access storage device (DASD), an optical drive (e.g., CD drive, DVD drive, etc.), a solid state storage drive (SSD), network attached storage, a storage area network, and / or a tape drive. The vehicle 1100 may also include a user interface 1152 such that the vehicle 1100 can receive inputs from a user or operator and generate outputs for the user or operator. This user interface 1152 may be, for example, one or more displays, a touch screen, a voice and / or gesture interface, buttons and other tactile controls, etc. Alternatively, the user inputs are received from, for example, a remote operator, via another computer or electronic device, for example, via an application on a mobile device or a web interface.

[0060] Disclosed herein are systems and methods for object detection and detection confidence. The disclosed methods may be suitable for autonomous driving, but may also be used in other applications, such as robotics, video analytics, weather forecasting, medical imaging, and the like. The disclosure may describe an exemplary autonomous vehicle 1100. Although the disclosure primarily provides examples using autonomous vehicles, other types of devices may be used to implement the various methods described herein, such as robots, camera systems, weather forecasting devices, medical imaging devices, and the like. Additionally, the methods may be used to control an autonomous vehicle, or for other purposes, such as, but not limited to, video surveillance, video or image editing, video or image research or retrieval, object tracking, weather forecasting (e.g., using radar data), and / or medical imaging (e.g., using ultrasound or magnetic resonance imaging (MRI) data).

[0061] Those skilled in the art can understand that each of the units, algorithms, and steps described and disclosed in the embodiments of the present disclosure can be realized using electronic hardware or a combination of computer software and electronic hardware. Whether a function is implemented in hardware or software depends on the application conditions and the design requirements of the technical solution. Those skilled in the art can realize the functions for each specific application using different methods. And such realization should not exceed the scope of the present disclosure. Those skilled in the art can understand that the operation processes of the above systems, devices, and units are basically the same, so it can refer to the operation processes of the systems, devices, and units in the above embodiments. For ease and simplicity of description, these operation processes will not be described in detail.

[0062] The software functional unit may be stored in a computer-readable storage medium when embodied as a product and used and sold. Based on this understanding, the technical proposal proposed in the present disclosure may be substantially or partially implemented in the form of a software product. Alternatively, some of the technical proposals useful in the prior art may be realized in the form of a software product. The software product in the computer is stored in a storage medium containing a plurality of commands for a computing device, such as a personal computer, a server, or a network device, to execute all or part of the steps disclosed in the embodiments of the present disclosure. The storage medium includes a USB disk, a mobile hard disk, a ROM, a RAM, a floppy disk, or other types of media capable of storing program code. Although the present disclosure has been described in connection with what is considered to be the most practical and preferred embodiment, it should be understood that the present disclosure is not limited to the disclosed embodiment, but is intended to encompass various configurations made without departing from the broadest interpretation of the scope of the appended claims.

[0063] However, other modifications, variations, and substitutions are possible. Thus, the specification and figures are to be regarded as illustrative, not restrictive. In the claims, graphic symbols in parentheses are not to be construed as limiting the claims. The word "comprehensive" means that it does not exclude the presence of other parts or steps than those recited in the claims. Also, the terms "a" and "one" as used in this text define one or more. Furthermore, the introductory phrases "at least one" and "one or more" used in the claims should not be construed as limiting another claim element introduced by the indefinite article "a" or "one" to an invention containing only one of the introduced claim element. This is true even if the same claim includes the introductory phrases "one or more" and the indefinite article "a" or "one". This also applies to the use of definite articles. Unless otherwise stated, terms such as "first" and "second" are used to arbitrarily distinguish between the elements described by these terms and are not meant to be used to indicate a temporal or other priority of these elements. The fact that certain features are recited in different claims does not mean that a combination of these features cannot be used to advantage. While certain features of the invention have been illustrated and described herein, those skilled in the art may envision many modifications, substitutions, changes, and equivalents. It is therefore to be understood that the appended claims are intended to cover all such modifications and variations that fall within the true spirit of the invention.

[0064] It should be understood that various features of the embodiments of the present disclosure that are described for clarity in the context of individual embodiments may also be provided in combination in a single embodiment. Conversely, various features of the embodiments of the present disclosure that are described for brevity in the context of a single embodiment may also be provided alone or in any suitable subcombination. It should be understood by those skilled in the art that the embodiments of the present disclosure are not limited by what has been particularly shown and described above. Rather, the scope of the embodiments of the present disclosure is defined by the appended claims and their equivalents.

[0065] The foregoing description of the disclosed embodiments is provided to enable others to make or use the disclosed subject matter. Various modifications to these embodiments will be readily apparent, and the general principles defined herein may be applied to other embodiments without departing from the spirit or scope of the foregoing. Thus, the foregoing description is not intended to be limited to the embodiments set forth herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein. Thus, the claims are not intended to be limited to the embodiments set forth herein, but are to be accorded the fullest scope consistent with the language of the claims, and references to elements in the singular do not mean "one and only," unless expressly stated as such, but rather "one or more." Unless otherwise stated, the term "some" refers to one or more. All structural and functional equivalents of the elements of the various embodiments described in the foregoing description, whether known or later known, are expressly incorporated herein by reference and are intended to be encompassed by the claims. Moreover, the subject matter disclosed herein is not intended to be publicly specific, regardless of whether such disclosure is expressly set forth in the claims. No claim element should be construed as an apparatus functional unless the element is expressly recited using the phrase "apparatus for." It is understood that the specific order or hierarchy of blocks in the disclosed processes is an example of an exemplary method. Based on design preferences, it is understood that the specific order or hierarchy of blocks in the processes can be rearranged while remaining within the foregoing scope. The accompanying method claims present the elements of the individual blocks in a sample order, and are not meant to be limited to the specific order or hierarchy presented.

[0066] The various examples shown and described are provided as examples only to illustrate various features of the claims. However, features shown and described with respect to any given example are not necessarily limited to the associated example, and may be combined with or in any combination with other examples shown and described. Furthermore, the claims are not intended to be defined by any examples. The above method descriptions and process flow diagrams are provided as illustrative examples only, and are not intended to require or imply that the blocks of the various examples must be performed in the order presented. As will be understood, the order of blocks in the above examples may be performed in any order. Terms such as "then," "then," and "next" are not intended to limit the order of the blocks, and these words are merely used to guide the reader through the method description. Furthermore, any reference to a claim element in the singular, for example using the articles "a," "an," or "the," should not be construed as limiting the element to the singular. The various illustrative logic blocks, modules, circuits, and algorithm blocks described with respect to the examples disclosed herein may be implemented as electronic hardware, computer software, or a combination of both. To clearly illustrate this interchangeability of hardware and software, various exemplary components, blocks, modules, circuits, and blocks have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the particular application and design constraints imposed on the overall system. Those skilled in the art may implement the described functionality in various ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present disclosure. Hardware for implementing the various exemplary logic, logic blocks, modules, and circuits described in connection with the examples disclosed herein may be implemented or performed using a general-purpose processor, a DSP, an ASIC, an FPGA or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein.A general purpose processor may be a microprocessor, but in the alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. A processor may also be implemented as a combination of computing devices, e.g., a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration. Alternatively, some blocks or methods may be performed by circuitry that is specific to a given function. EXAMPLES

[0067] A method for reducing computational costs of an autonomous driving system includes the steps of: (a) acquiring data related to a task for vehicle operation; (b) training a deep learning model using the acquired data, the deep learning model including an encoder and a policy head for the task; (c) reducing the complexity of the data acquired in step (a) by transferring the data to the encoder and generating a compressed latent representation of the data; and (d) determining a driving maneuver by the policy head using the compressed latent representation of the data. EXAMPLES

[0068] According to the method described in Example 1, the acquired data includes recorded human driving data from the same vehicle or another vehicle. EXAMPLES

[0069] According to the method of Example 1 or 2, the acquired data includes artificially enhanced data. EXAMPLES

[0070] Based on the method according to any one of Examples 1 to 3, the data is acquired using sensors of the same vehicle or another vehicle, the sensors including one or more of a laser radar sensor, a radar sensor, an infrared sensor, and / or an image sensor. EXAMPLES

[0071] According to the method according to any one of the first to fourth embodiments, the step (c) further comprises: applying a mask obtained by multiplying elements by the compressed latent representation to further reduce the complexity of the data obtained in the step (a). EXAMPLES

[0072] According to the method described in Example 5, the method further includes normalizing the mask values. EXAMPLES

[0073] The method according to any one of Examples 1 to 6 further includes evaluating the discrimination between the driving maneuver determined by the policy head and a driving maneuver benchmark by applying a loss function based on the method according to any one of Examples 1 to 6. EXAMPLES

[0074] The method according to any one of Examples 1 to 7 further includes configuring one or more duplicated elements of the compressed latent representation generated by a first encoder of a first deep learning model so that the compressed latent representation can be shared by a second encoder of a second deep learning model. EXAMPLES

[0075] A method for reducing computational costs of an autonomous driving system includes the steps of: (a) acquiring data related to a task for vehicle operation; (b) operating a deep learning model using the acquired data, the deep learning model including a policy head for the task; (c) acquiring a compressed latent representation of the data acquired in step (a); and (d) the policy head determining a driving maneuver by using the compressed latent representation of the data. EXAMPLES

[0076] According to the method of Example 9, the acquired data includes recorded human driving data from the same vehicle or another vehicle. EXAMPLES

[0077] According to the method of Example 9 or 10, the obtained data includes artificially enhanced data. EXAMPLES

[0078] Based on the method according to any one of Examples 9 to 11, the data is acquired using sensors of the same vehicle or another vehicle, the sensors including one or more of a laser radar sensor, a radar sensor, an infrared sensor, and / or an image sensor. EXAMPLES

[0079] Based on the method according to any one of Examples 9 to 12, the step (c) further includes applying a mask obtained by multiplying an element by the compressed latent representation to further reduce the complexity of the data obtained in step (a). EXAMPLES

[0080] According to the method of Example 13, the method further includes normalizing the mask values. EXAMPLES

[0081] The method according to any one of Examples 9 to 14 further includes evaluating the discrimination between the driving maneuver determined by the policy head and a driving maneuver benchmark by applying a loss function based on the method according to any one of Examples 9 to 14. EXAMPLES

[0082] The method according to any one of Examples 9 to 15 further includes configuring one or more overlapping elements of the compressed latent representation generated by a first encoder of a first deep learning model so that the compressed latent representation can be shared by a second encoder of a second deep learning model. EXAMPLES

[0083] A method for reducing computational costs of an autonomous driving system includes: (a) acquiring data related to a task for a vehicle operation; (b) training a first deep learning model using the acquired data, the first deep learning model including a first encoder and a policy head; (c) identifying one or more overlapping elements between the data related to the task and a compressed latent representation related to another task, the compressed latent representation being generated by a second deep learning model having a second encoder, the compressed latent representation being configured to be shareable by the first encoder of the first deep learning model; and (d) determining a driving maneuver by the policy head using the compressed latent representation, the compressed latent representation being generated by the second deep learning model having a second encoder. EXAMPLES

[0084] According to the method of Example 17, the acquired data includes recorded human driving data from the same vehicle or another vehicle. EXAMPLES

[0085] According to the method of Example 17 or 18, the obtained data includes artificially enhanced data. EXAMPLES

[0086] Based on the method according to any one of Examples 17 to 19, the data is acquired using sensors of the same vehicle or another vehicle, the sensors including one or more of a laser radar sensor, a radar sensor, an infrared sensor, and / or an image sensor. EXAMPLES

[0087] An apparatus for operating an automated driving system includes at least one processor and a memory storing instructions that, when executed by the at least one processor, cause the at least one processor to perform operations including the method of any one of Examples 1 to 20. EXAMPLES

[0088] A non-transitory computer-readable storage medium having instructions stored thereon that, when executed by one or more processors, cause the one or more processors to perform a method for controlling an automated driving system and reducing computational costs associated therewith, including the method of any one of Examples 1 to 20.

Claims

1. 1. A method for reducing computational costs of an automated driving system, comprising: (a) acquiring data related to a task for operating a vehicle; (b) training a deep learning model using the acquired data, the deep learning model including an encoder and a policy head for the task; (c) reducing the complexity of the data obtained in step (a) by transferring the data to the encoder and generating a compressed latent representation of the data; (d) determining a driving maneuver by the policy head using the compressed latent representation of the data.

1. A method for reducing computational costs of an automated driving system, comprising:

2. The acquired data may include recorded human driving data from the same vehicle or another vehicle, or artificially augmented data; 2. The method of claim 1 .

3. The data is acquired using a sensor on the same vehicle or another vehicle; The sensor includes one or more of a laser radar sensor, a radar sensor, an infrared sensor, and / or an image sensor; 2. The method of claim 1 .

4. The step (c) and applying a mask obtained by multiplying the elements of the compressed latent representation to further reduce the complexity of the data obtained in step (a).

2. The method of claim 1 .

5. Further, the operation of normalizing the mask values ​​includes:

2. The method of claim 1 .

6. and applying a loss function to evaluate discrimination between the driving maneuver determined by the policy head and a driving maneuver benchmark.

2. The method of claim 1 .

7. and configuring one or more duplicated elements of the compressed latent representation generated by a first encoder of a first deep learning model so that the compressed latent representation can be shared by a second encoder of a second deep learning model.

2. The method of claim 1 .

8. A non-transitory computer-readable storage medium having instructions stored thereon, comprising: A non-transitory computer readable storage medium, the instructions of which, when executed by one or more processors, cause the one or more processors to perform the method of any one of claims 1 to 7.

9. 1. A method for reducing computational costs of an automated driving system, comprising: (a) acquiring data related to a task for operating a vehicle; (b) operating a deep learning model using the acquired data, the deep learning model including a policy head for the task; (c) obtaining a condensed latent representation of the data obtained in step (a); and (d) determining a driving maneuver by the policy head using the compressed latent representation of the data.

1. A method for reducing computational costs of an automated driving system, comprising:

10. 1. A method for reducing computational costs of an automated driving system, comprising: (a) acquiring data related to a task for operating a vehicle; (b) training a first deep learning model using the acquired data, the first deep learning model including a first encoder and a policy head; (c) identifying one or more overlapping elements between the data related to the task and a compressed latent representation related to another task, the compressed latent representation being generated by a second deep learning model having a second encoder, the compressed latent representation being configured to be shareable by the first encoder of the first deep learning model; and (d) the policy head determining a driving maneuver by using the compressed latent representation, the compressed latent representation being generated by the second deep learning model having a second encoder.

1. A method for reducing computational costs of an automated driving system, comprising:

Images