Method for reducing computational cost of automatic driving system

Abstract

The invention provides a method for reducing the calculation cost of an automatic driving system. The method may include the steps of: a) acquiring data relating to a task for operating the vehicle; b) training a deep learning model using the acquired data, the deep learning model comprising an encoder and a policy header for the task; c) reducing the complexity of the data obtained in step a) by passing the data to an encoder to produce a compressed potential representation of the data; and d) determining, by the policy header, a driving operation using the compressed potential representation of the data.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and more particularly to a method and / or device for reducing the computational cost of an autonomous driving system. Background Art

[0002] As computing and vehicle technologies continue to advance, features related to automation have become more powerful and widely available, and are able to control vehicles in a wider variety of environments. For example, for automobiles, the Society of Automotive Engineers (SAE) has established a standard (J3016) that identifies six levels of driving automation from "no automation" to "full automation." The SAE standard defines Level 0 as "no automation," where the human driver performs all aspects of the dynamic driving task full time, even when enhanced by warning or intervention systems. Level 1 is defined as "driver assistance," where the vehicle controls steering or acceleration / deceleration (but not both) in at least some driving modes, allowing the operator to perform all remaining aspects of the dynamic driving task. Level 2 is defined as "partial automation," where the vehicle controls steering and acceleration / deceleration in at least some driving modes, allowing the operator to perform all remaining aspects of the dynamic driving task. Level 3 is defined as "conditional automation," where, for at least some driving modes, the automated driving system performs all aspects of the dynamic driving task, with the expectation that the human driver will respond appropriately to intervention requests. Level 4 is defined as "high automation," where, only for specific conditions, the automated driving system performs all aspects of the dynamic driving task, even if the human driver does not respond appropriately to intervention requests. Specific conditions for Level 4 may be, for example, specific types of roads (e.g., highways) and / or specific geographic areas (e.g., geographically isolated metropolitan areas that have been appropriately mapped). Finally, Level 5 is defined as "full automation," where the vehicle is able to operate without operator input under all conditions.

[0003] The fundamental challenges of any auto-related technology involve collecting and interpreting information about the vehicle's surroundings, along with planning and executing commands for appropriately controlling vehicle motion to navigate the vehicle safely through its current environment. As a result, there is an ongoing effort to improve each of these aspects, and by doing so, autonomous vehicles are continually able to reliably operate in increasingly complex environments and adapt to both expected and unexpected interactions within the environment. For example, in order to operate safely, an autonomous vehicle should consider objects such as vehicles, people, trees, animals, buildings, signs, and poles when planning a path through the environment.

[0004] Since autonomous driving systems need to constantly monitor their surroundings, the amount of information to be processed is large. Therefore, it is important to develop algorithms that reduce computational complexity while maintaining the flexibility and safety of autonomous driving operations. Summary of the invention

[0005] One object of the present disclosure is to propose a method and / or device for reducing the computational cost of an active driving system. To this end, the present invention discloses various end-to-end deep learning models for autonomous vehicle control systems. The end-to-end deep learning model accepts raw data from various sensors (e.g., cameras, LIDAR, etc.). The raw data can be collected directly from the sensors of the controlled vehicle. The raw data can also be perception data recorded in real time from any vehicle (e.g., recorded driving) or perception data shared in real time by another vehicle. The end-to-end deep learning model produces driving control decisions as output. In addition, the deep learning model method can use data that is not manually annotated and is therefore more reasonably (e.g., cheaper) to obtain and richer.

[0006] Specifically, an intermediate compressed latent representation or a reduced dimensionality latent representation of the original data can be provided for use / training an autonomous driving system to output driving control decisions, for example, in data-scarce situations such as reinforcement learning. Various embodiments disclose an encoder that transforms the original data into a compressed latent representation. The compressed latent representation is significantly reduced in data volume compared to the original data. This allows the computational efficiency of the end-to-end deep learning model of the autonomous driving system to be improved in training and / or usage mode. In one example, the encoder extracts useful features from the raw data for a specific driving task (e.g., lane centering, lane changes, traffic sign reading, etc.) and ignores the rest of the data. In another example, the feature extraction of the encoder can be accomplished by various machine vision recognition, curve fitting, pattern recognition, text recognition, etc.

[0007] In other embodiments, masks can be used to further sparse the compressed latent representation. This further reduces the amount of data to be processed and improves computational efficiency.

[0008] In some embodiments, a method for reducing the computational cost of an autonomous driving system is disclosed. The method includes the following steps: a) acquiring data related to a task for operating a vehicle; b) using the acquired data to train a deep learning model, wherein the deep learning model includes an encoder and a policy head for the task; c) reducing the complexity of the data acquired in step a) by passing the data to the encoder to generate a compressed latent representation of the data; and d) determining a driving operation by the policy head using the compressed latent representation of the data.

[0009] In some embodiments, the acquired data includes recorded human driving data from the same vehicle or a separate vehicle. In some embodiments, the acquired data includes artificially enhanced data. In some embodiments, the data is acquired using sensors of the same vehicle or a separate vehicle, and the sensors include one or more lidar sensors, radar sensors, infrared sensors, and / or image sensors.

[0010] In some embodiments, step c) further comprises: applying a mask element-wise multiplied with the compressed latent representation to further reduce the complexity of the data obtained in step a). In some embodiments, step c) further comprises: normalizing the mask value.

[0011] In some embodiments, the method further comprises: applying a loss function to evaluate a difference between the driving maneuver determined by the policy head and a driving maneuver benchmark.

[0012] In some embodiments, the method further includes configuring one or more overlapping elements of the compressed latent representation generated by a first encoder of a first deep learning model such that the compressed latent representation is configured to be shareable by a second encoder of a second deep learning model.

[0013] In some embodiments, another method for reducing the computational cost of an autonomous driving system is disclosed. The method includes: a) acquiring data related to a task for operating a vehicle; b) using the acquired data to operate a deep learning model, wherein the deep learning model includes a policy head for the task; c) acquiring a compressed latent representation of the data acquired in step a); and d) determining a driving operation by the policy head using the compressed latent representation of the data.

[0014] In some embodiments, another method for reducing the computational cost of an autonomous driving system is disclosed. The method includes: a) acquiring data related to a task for operating a vehicle; b) using the acquired data to train a first deep learning model, wherein the first deep learning model includes a first encoder and a policy head; c) identifying one or more overlapping elements between the data related to the task and a compressed latent representation related to another task, wherein the compressed latent representation is generated by a second deep learning model having a second encoder, and the compressed latent representation is configured to be shareable with the first encoder of the first deep learning model; and d) determining a driving operation by the policy head using the compressed latent representation generated by the second deep learning model having the second encoder.

[0015] In some embodiments, the disclosed method may be operated by an apparatus of an autonomous driving system. The apparatus may include at least one processor and a memory storing instructions. The instructions, when executed by at least one processor, cause the at least one processor to perform the disclosed operation of the method for reducing the computational cost of an autonomous driving system. For example, in some embodiments, the disclosed method may be programmed as a computer executable instruction stored in a non-transitory computer-readable medium. The non-transitory computer-readable medium instructs the processor of the computer to perform the disclosed method when loaded into the computer. The non-transitory computer-readable medium may include at least one of the group consisting of a hard disk, a CD-ROM, an optical storage device, a magnetic storage device, a read-only memory, a programmable read-only memory, an erasable programmable read-only memory, an EPROM, an electrically erasable programmable read-only memory, and a flash memory.

[0016] It should be understood that all combinations of the above concepts and additional concepts described in more detail herein are considered to be part of the subject matter disclosed herein. For example, all combinations of the claimed subject matter appearing at the end of this disclosure are considered to be part of the subject matter disclosed herein. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the embodiments of the present disclosure or related technologies, the following drawings to be described in the embodiments are briefly introduced. Obviously, the drawings are only some embodiments of the present disclosure, and ordinary technicians in this field can obtain other drawings based on these drawings without creative work. The arrows in the drawings indicate the relationship, through which the component from which the arrow starts is used to train / apply the component pointed by the arrow. The embodiments of the present disclosure will be more completely understood and appreciated according to the following detailed description in conjunction with the drawings, in which:

[0018] Figure 1 is a block diagram illustrating an example implementation of a deep learning model when being trained or when being used according to some embodiments of the present disclosure;

[0019] Figure 2 is a block diagram illustrating another example implementation of a deep learning model with a mask function when training or when in use according to some embodiments of the present disclosure;

[0020] Figure 3 is a block diagram illustrating an example implementation of multiple deep learning models with mask functions when being trained or when being used according to some embodiments of the present disclosure;

[0021] Figure 4 is a table illustrating an example of identifying one or more overlapping elements associated with different tasks according to some embodiments of the present disclosure;

[0022] Figure 5is a flow chart illustrating an example of the operation of an end-to-end deep learning model when training according to some embodiments of the present disclosure;

[0023] Figure 6 is a flow chart illustrating an example of the operation of an end-to-end deep learning model when used according to some embodiments of the present disclosure;

[0024] Figure 7 is a flow chart illustrating an example of the operation of multiple deep learning models when training according to some embodiments of the present disclosure;

[0025] Figure 8 is an illustration of an example of operating an autonomous driving system without using a mask function according to some embodiments of the present disclosure;

[0026] Fig. 9 is an illustration of an example of operating an autonomous driving system using a mask function for a lane centering / keeping task according to some embodiments of the present disclosure;

[0027] Fig.10 is an illustration of an example of operating an autonomous driving system using a mask function for a traffic sign reading task according to some embodiments of the present disclosure; and

[0028] Fig.11 An exemplary hardware and software environment for an autonomous vehicle according to some embodiments of the present disclosure is illustrated.

[0029] It should be understood that for simplicity and clarity of illustration, the elements shown in the figures are not necessarily drawn to scale. For example, for clarity, the size of some elements may be exaggerated relative to other elements. In addition, where deemed appropriate, reference numerals may be repeated in the drawings to indicate corresponding or similar elements. DETAILED DESCRIPTION

[0030] With reference to the accompanying drawings, the embodiments of the present disclosure are described in detail with technical problems, structural features, purposes and effects of implementation, as follows. Specifically, the terms in the embodiments of the present disclosure are only used for the purpose of describing specific embodiments, rather than limiting the present disclosure. In the following detailed description, many specific details are set forth to provide a thorough understanding of the present invention. However, those skilled in the art will understand that the present invention can be implemented without these specific details. In other cases, well-known methods, processes and components are not described in detail to avoid confusing the present invention. The subject matter of the present invention is specifically pointed out and clearly claimed in the concluding part of the specification. However, with respect to the organization and operation method of the present invention and its purposes, features and advantages, the present invention can be best understood by referring to the following detailed description when read in conjunction with the accompanying drawings. Because the illustrated embodiments of the present invention can be implemented using electronic components and circuits known to those skilled in the art for the most part, in order to understand and appreciate the underlying concepts of the present invention, and in order not to confuse or disperse the teachings of the present invention, the details will not be explained to a greater extent than what is considered necessary as shown above. For example, the specification and / or the accompanying drawings may relate to a processor or a processing circuit. The processor may be a processing circuit. The processing circuit may be implemented as a central processing unit (CPU) and / or one or more other integrated circuits such as an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a full custom integrated circuit, etc., or a combination of these integrated circuits.

[0031] The following description and / or drawings may relate to images. Images are examples of media units. Any reference to images may be necessarily applied to media units. Media units may be examples of sensory information units (SIUs). Any reference to media units may be necessarily applied to any type of natural signal, such as but not limited to signals generated by nature, signals representing human behavior, signals representing operations related to vehicle signals, geodetic signals, geophysical signals, text signals, digital signals, time series signals, etc. Any reference to media units may be necessarily applied to SIUs. SIUs may be of any type and may be sensed by any type of sensor, such as a visual light camera. Audio sensors. Sensors that can sense infrared, radar imaging, ultrasound, electro-optical, radiography, LIDAR (light detection and ranging), thermal sensors, passive sensors, active sensors, etc. Sensing may include generating samples (e.g., pixels, audio signals, etc.) that represent the signals sent or otherwise reach the sensor. SIUs may have one or more images, one or more video clips, text information about one or more images, text describing motion information, etc.

[0032] Any combination of any modules or units listed in any of the drawings, any part of the specification and / or any claims may be provided. Any of the units and / or modules illustrated in the present application may be implemented with hardware and / or code, instructions and / or commands stored in a non-transitory computer-readable medium, and may be included in a vehicle, outside a vehicle, in a mobile device, in a server, etc. The vehicle may be any type of vehicle, such as a ground transportation vehicle, an aviation vehicle, or a watercraft. The vehicle is also referred to as an ego vehicle. It should be understood that autonomous driving includes at least partially automatic (semi-automatic) driving of a vehicle, including all L2 level types or higher level types defined in the SAE standard.

[0033] Referring now to the drawings, in which like numerals represent like parts throughout the several views, Figures 1 to 3 It is a block diagram illustrating different end-to-end deep learning models for executing a method for reducing the computational cost of a system according to some embodiments of the present disclosure, the system including but not limited to autonomous driving. The model may have a computer system for training and / or operating one or more models, one or more models may be artificial intelligence (AI) models, including but not limited to deep learning models, such as deep neural networks with one or more latent layers / representations. It should be understood that these and other arrangements described herein are only set forth as examples. In addition to or in place of those shown, other arrangements and elements (e.g., machines, interfaces, functions, groups of sequences and functions, etc.) may be used, and some elements may be completely omitted for clarity. In addition, many of the elements described herein are functional entities that may be implemented as discrete or distributed components or in combination with other components, and implemented in any appropriate combination and position. The various functions described herein as being performed by one or more entities may be performed by hardware, firmware, and / or software. For example, some functions may be performed by a processor executing instructions stored in a memory. It should be understood that the deep learning model described below may be a neural network including multiple layers such as an input layer, a potential layer / hidden layer, and an output layer. Each neural network layer may include multiple nodes (or neurons), which are typically connected in series.

[0034] In some embodiments, Figure 11 is a block diagram of a deep learning model 100 including an encoder 104 and a policy head 108. Each of the encoder 104 and the policy head 108 can be a trainable AI model. In one embodiment, the deep learning model is task-specific, such as lane centering, lane changing, traffic sign reading, etc. The encoder 104 receives raw data 102. The encoder 104 extracts features from the raw data 102, thereby correspondingly reducing the high-dimensional raw data to a low-dimensional latent vector as a compressed latent representation 106. In this way, the data volume / complexity of the raw data 102 can be reduced by forming the compressed latent representation 106. The compressed latent representation 106 of the raw data 102 helps to learn data characteristics and simplify data representation. The encoder 104 outputs the compressed latent representation 106 to the policy head 108. The policy head 108 receives the compressed latent representation 106 and outputs a driving operation decision 110 accordingly. In various embodiments, the compressed latent representation 106 may be obtained by discarding duplicate or unvalued data and / or using different data representation and approximation techniques (i.e., transmitting less data without loss, and transmitting a compact model instead of the original data). For example, in some embodiments, the compressed latent representation 106 may apply a linear or nonlinear transformation to the original data 102 for generating the output driving operation decision 110.

[0035] In some embodiments, the raw data 102 is raw data from one or more sensors of the same vehicle or a separate vehicle. For example, the raw data 102 may be an image captured by a camera sensor including red, green, and blue (RGB) values ​​of pixels. The raw data 102 may be a raw SIU, a processed SIU, text information, information derived from the SIU, and the like. In different embodiments, the loading of the raw data 102 may be from a local disk, from a remote storage location through an appropriate network location, and the like. Acquiring the raw data 102 may include receiving data, generating data, participating in the processing of data, processing only a portion of the data, and / or receiving only another portion of the data. The processing of the data 102 may include at least one of detection, noise reduction, improvement of the signal-to-noise ratio, defining a bounding box, and the like. The raw data 102 may be received from one or more sources such as one or more sensors, one or more communication units, one or more memory units, one or more image processors, and the like.

[0036] In some embodiments, the encoder 104 can be configured to map the raw data 102 to a compressed latent representation 106, which can be stored in a database of semantic relationships. In some embodiments, the encoder 104 learns the input data dimensionality compression to encode the latent representation of the feature, and the strategy head 108 recreates the encoded latent representation as a reconstructed output, such as an output driving operation decision 110. For example, the encoder 104 can be configured to generate a compressed latent representation 106 of the raw data 102 using a one-dimensional vector representing one or more elements of the raw data 102. In one embodiment, the compressed latent representation can be represented as a vector V, where V = [E1, E2, E3, ... E N ]. E1 refers to element 1, E2 refers to element 2, E3 refers to element 3, E N Refers to the element N. Each element can be a one-dimensional or multi-dimensional matrix. Each element can represent potentially useful features around the vehicle, such as lane boundaries, lane centerlines, nearby vehicles, traffic signs, tree outlines, etc., for example, see Figure 4 .

[0037] The encoder 104 can be configured to encode meaningful information about various data attributes in its potential manifold, which can then be utilized to perform related tasks. In such an embodiment, the compressed potential representation 106 helps reduce the dimensionality of the input data and eliminates irrelevant information. Therefore, the reduction in the dimensionality of the input data will reduce computational overhead and help avoid overfitting.

[0038] In some embodiments, given the compressed latent representation 106, the policy head 108 can be configured to determine the behavior that the vehicle needs to follow from a set of predetermined tasks. The tasks determine the actions that the automated vehicle needs to take based on the compressed latent representation 106. Some examples of these tasks are lane keeping, overtaking, changing lanes, intersection handling, and traffic light handling, etc.

[0039] In some embodiments, the gradient of the loss function 112 can be configured to evaluate the difference between the driving operation decision 110 and the driving operation benchmark 114 determined by the strategy head 108 to characterize the accuracy of the compressed potential representation 106. The loss function is a measure of how well the prediction model performs in being able to predict the expected results. The parameters of the encoder 104 and / or the compressed potential representation 106 can be updated / adjusted based on the gradient of the loss function 112 to achieve improved driving decision output. It should be understood that when the model is used only for operation and no training purposes, the loss function 112 may not be needed. However, operation and training of the system can be performed simultaneously.

[0040] In some other embodiments, such as Figure 2As shown, the deep learning model 200 may include a similar structure to the deep learning model 100, and further include a trainable mask 208 to further reduce the computational cost by further reducing the complexity of the input data. The trainable mask 208 may be configured to generate a sparse latent representation 212 for a policy head 214 based on the compressed latent representation 206 to generate a driving operation decision 216.

[0041] For example, in some embodiments, the trainable mask 208 may be element-wise multiplied with the compressed latent representation 206 generated by the encoder 204 based on the set of raw data 202. In one embodiment, the trainable mask 208 may be a vector having elements that match the compressed latent representation 206. The trainable mask 208 may zero or normalize less useful elements in the compressed latent representation 206 to further sparse the data.

[0042] For example, in the model 200 processing the lane change task, the mask 208 may keep the elements of the compressed latent representation 206, namely, the lane boundary line [E1], the lane center line [E2], other vehicles [E3], and the traffic sign text [E4], but zero out the tree outline [E5] because the model 200 determines that the tree outline is not very useful for the lane change task. Therefore, in this embodiment, if the compressed latent representation is the vector V = [E1, E2, E3, E4, E5], then the sparse latent representation 212 is the vector V sparse =[E1,E2,E3,E4,0], where “0” represents the zero matrix.

[0043] In some embodiments, the trainable mask 208 can be configured to map from the compressed latent representation 206 to generate a sparse latent representation 212 received by the policy head 214 to determine the output driving operation decision 216. For example, the mask values ​​of the trainable mask 208 can be normalized between 0 and 1 (e.g., passing the trainable parameters to the sigmoid function) to encourage data sparsity. In some embodiments, the loss function 212 can be configured to compare the output of the policy head 216 with the driving operation benchmark 214. The mask values ​​can be added to the loss function 212 in the form of an L1 regularization loss, which adds the absolute values ​​of the mask elements, resulting in many of the values ​​in the mask values ​​of the trainable mask 208 being set to zero for better data sparsity of the sparse latent representation 212. It should be understood that the loss function 212 can be applied to the encoder 204 and / or the mask 208 to improve system performance.

[0044] In some embodiments, the compressed latent representation can be configured to be shareable between multiple end-to-end deep learning models. Figure 3As shown, two deep learning models 300a and 330b in training are illustrated. The deep learning model 300a includes an encoder 304a and a strategy head 312a. In some embodiments, the trainable mask 308a can be element-wise multiplied with the compressed potential representation 306a generated by the encoder 304a based on a set of raw data 302a. In such an embodiment, the trainable mask 308a can be configured to map from the compressed potential representation 306a to generate a sparse potential representation 310a for receiving by the strategy head 312a to determine the output driving operation decision 314a. In some embodiments, the gradient of the loss function 324a can be configured to evaluate the difference between the driving operation decision 314a determined by the strategy head 312a and the driving operation benchmark 326a to characterize the accuracy of the compressed potential representation 306a and / or the sparse potential representation 310a. The loss function 324a can be applied to the encoder 304a, the mask 308a or the strategy head 312a to improve system performance.

[0045] The deep learning model 300b includes the same structure as the deep learning model 300a. In some embodiments, the data sharing module 318 can be configured to identify one or more overlapping elements between the data and the sparse latent representations 310a, 310b of the deep learning models 300a and 300b, so that the one or more sparse latent representations 310a, 310b are configured to be shared by the encoders 304a and 304b of the two deep learning models. The shared latent representation further improves the computational efficiency of the autonomous driving system.

[0046] In one embodiment, deep learning model 300a is used for lane change, while deep learning model 300b is used for lane centering. Data sharing module 308 can compare elements of sparse potential representations 310a and 310b. Sparse potential representation 310a for lane change may include elements of lane boundary lines, lane center lines, other vehicles, and traffic sign text. Sparse potential representation 310b for lane centering may include elements of lane boundary lines and lane center lines. Data sharing module 318 determines overlapping elements of 310a and 310b, such as lane boundary lines and lane center lines. Then, data sharing module 318 creates a shared potential representation 316 and sends it to encoders 304a, 304b or any other encoder that may need such overlapping elements. Data sharing module 318 can also upload or download 322 overlapping elements to / from network 320. Storing one or more elements of compressed potential representations on network 320 facilitates further sharing between deep learning models at different points in time.

[0047] Sharing capabilities can be configured to optimize trainable latent representations and mask functions. To better illustrate the shared compressed / sparse latent representation features, Figure 4 is a table illustrating examples of identifying one or more overlapping elements associated with different tasks, such as the task of lane changing (task 1), the task of lane centering (task 2), and the task of traffic sign reading (task 3). As shown here, both the task of lane changing and the task of lane centering require elements about lane boundaries and lane center lines to complete the task. Therefore, in some embodiments, the overlapping elements of the sparse latent representation 310 (e.g., lane boundaries and lane center lines) can be shared from one deep learning module to another deep learning model 300 by the data sharing module 318.

[0048] In some embodiments, a sparse latent representation 310 that has been configured for one task can be provided by the data sharing module 318 and directly applied to another task of the same or different vehicle. In this sharing of overlapping elements, the computational cost is further reduced because the system receiving the sharable element may not need to generate its own compressed or sparse latent representation. In some embodiments, the sharing of elements can be performed via a network 320, including but not limited to Wi-Fi, DSRC connections, etc. Data sharing can also be extended to autonomous robots, autonomous transport robots, or any other system capable of autonomous navigation via machine learning models beyond autonomous vehicles.

[0049] In some embodiments, one or more networks 320 (e.g., LAN, WAN, wireless network and / or Internet, etc.) may be provided to allow communication of information with other data sharing modules 318, computers and / or electronic devices (including, for example, a central service, such as a cloud service, from which the data sharing module 318 receives sharable compressed / sparse latent representations, environmental data and other data for its automatic control). For example, in some embodiments, one or more predefined latent representations 316 are configured to be shared by the data sharing module 318 and the network 320. In such embodiments, the shared latent representations 316 configured by the local model can be uploaded 322 to the network 320 (e.g., cloud system) and stored in the network 320 for use by other models in the remote, and the data sharing module 318 can also download 322 the shared latent representations 316 from the network 320 for local use. In various embodiments, the data sharing module 318 can be a tangible or intangible entity, e.g., one that is physically constructed, specifically configured (e.g., hardwired), or configured (e.g., programmed) to operate in a specified manner or to perform a portion or all of any of the operations described herein. Data sharing can be used for one type of data or multiple types of data, for one-time use, multiple uses, and / or persistent use. The shared data can be collected and distributed in the form of raw uploads, or can be further processed before sharing. The shared data can be sent in real time or near real time.

[0050] Now go to Figure 5-Figure 7 , these figures illustrate three methods 500, 600 and 70 that can be used to reduce computational costs corresponding to the models discussed above. Note that the order of methods 500, 600 and 700 is exemplary and does not represent the order in which the steps of methods 500, 600 and 700 are performed. Figure 5 As shown, the operation method 500 can be started by acquiring data related to a task for operating a vehicle (e.g., lane change, lane centering, traffic sign reading, etc.) in box 502 and using the acquired data to train a deep learning model in box 504. As discussed above, the deep learning model can include an encoder and a policy head for the task. Then, in box 506, the complexity of the data acquired in box 502 can be reduced by passing the data to the encoder to generate a compressed potential representation of the data, and in box 510, the compressed potential representation of the data can be used by the policy head to determine the driving operation.

[0051] In some embodiments, the data acquired in box 502 may include recorded human driving data from the same vehicle or a separate vehicle, and may be from one or more sensors. For example, in some embodiments, the acquired data may come from a storage device / memory that includes recorded human driving data. In some embodiments, the recorded human driving data may be a vehicle data log of a completed driving session from a conventional vehicle, a driving simulation system, or an autonomous vehicle. For example, while an autonomous vehicle performs a driving session, the autonomous vehicle or an associated computing system may collect and store human driving and / or vehicle data. After the session is completed, the log of recorded data may be transmitted to a computing system, such as a cloud system, for training or using an autonomous driving system as described above.

[0052] In some embodiments, in order to collect and label enough data to train the model for controlling vehicle behavior as described above, different types of sensors (e.g., lidar sensors, radar sensors, infrared sensors, and / or image sensors) can be used to generate data that captures various aspects of the driving environment. However, not all data are equally useful or can be used to train models. Some data may be noisy, incomplete, or unbalanced. To overcome these limitations, in some embodiments, the data obtained in box 502 for training models can be processed as artificially enhanced data. Data enhancement techniques can improve the quality and diversity of data by applying transformations such as cropping, flipping, rotating, scaling, adding noise, changing brightness, interpolating, creating or mixing 3-D models of images. These techniques can help the model learn more robust and generalizable features that improve its performance and accuracy. For example, certain elements (e.g., animals, adverse weather conditions, traffic lights, etc.) can be introduced into the input data to improve the training results.

[0053] In some embodiments, to further reduce computational costs, the method 500 may further include a block 508, which applies a mask that is element-wise multiplied with the compressed latent representation to further reduce the complexity of the data obtained in block 502. In some embodiments, the mask values ​​of the mask may be normalized as previously discussed. In some other embodiments, the method 500 may further include applying a loss function in block 512 to evaluate the difference between the driving maneuver determined by the strategy head and the driving maneuver benchmark to improve system performance.

[0054] In some embodiments, Figure 6 As shown, method 600 illustrates the operation of applying a trained model. Method 600 can be started by acquiring data related to a task for operating a vehicle (e.g., lane change, lane centering, traffic sign reading, etc.) in box 602 and operating a deep learning model using the acquired data in box 604. The deep learning model may include a policy head for the task. Then, in box 606, a compressed potential representation is acquired for application. In one example, in 606, acquisition may be to extract useful features from raw data using an encoder. In another example, at 606, acquisition may be to download a compressed potential representation from a network or any non-volatile electronic storage medium. Using the compressed potential representation, in box 610, a driving operation may be determined by the policy head. Similar to the operation method 500, the operation method 600 may also include: box 608, applying a mask multiplied by the element with the compressed potential representation to further reduce the complexity of the acquired data; and box 612, applying a loss function to evaluate the difference between the driving operation determined by the policy head and the driving operation benchmark as previously described to improve system performance.

[0055] In some embodiments, the compressed latent representation can be configured to be shared among multiple deep learning models. Figure 7 As shown, the operation method 700 may include the first two steps similar to the operation method 500: in box 702, data related to the task for operating the vehicle is obtained, and in box 704, a first deep learning model is trained using the first encoder and the policy head using the obtained data. The first deep learning model includes the first encoder and the policy head. Then, the operation may continue by: in box 706, identifying one or more overlapping elements (such as the latent representation of the data related to the task and the compressed data related to another task) between the latent representation. Figure 3 to Figure 4 ), wherein the compressed latent representation is generated by a second deep learning model having a second encoder. Thus, in such an embodiment, the compressed latent representation generated by the second deep learning model having the second encoder is configured to be shareable with the first encoder of the first deep learning model. In block 708, the driving maneuver is determined by the policy head using the compressed latent representation generated by the second deep learning model having the second encoder. Thus, the computational cost of the first deep learning model is reduced.

[0056] Figures 8 to 10 are different illustrations of different examples of operating an autonomous driving system for different tasks. Figures 8 to 10 Graphics where applicable Figures 1 to 3 Examples of original data 102, 202, 302, compressed latent representation 106, 206, 306, and sparse latent representation 212, 310 in FIG. Figures 8 to 10 Graphics where applicable Figures 5 to 7 Examples of original data, compressed latent representation, and masked generated data (i.e., sparse latent representation) in .

[0057] like Figures 8 to 10 As shown, the input images 802, 902, and 1002 can be raw data from a camera sensor, and the compressed latent representations 804, 904, and 1004 can be compact representations of the input images that capture useful features generated by the encoders 104, 204, 304. As shown here, the input images can be high-dimensional because the raw data (e.g., RGB images) of the surrounding environment of an autonomous driving vehicle are typically high-dimensional. The raw data images include not only images of the road, but also images of the scene around the road (e.g., other vehicles, trees, traffic signs, and the sky). In contrast, in some embodiments, the compressed latent representation only retains some regions of interest, and color images are typically converted into black / white images. For example, Figure 8As shown, the compressed latent representation 804 can be converted from the original color image 802 to a black / white line image 804, including only tree outlines 806, lane boundary lines 808, first lane centerlines 810a, second lane centerlines 810b, traffic sign outlines 812, traffic sign text 814 and other vehicles 816.

[0058] In some embodiments, gamma correction may first be performed on the input raw image 802 to improve the adaptability of the image, and then image binarization may be performed to convert the image from color to black and white. In some embodiments, after the image binarization process, cavities may be repaired by using morphological operations, boundaries may be smoothed, and then the center line of the lane (e.g., 810a, 810b) may be extracted by using a skeleton extraction algorithm. In some embodiments, local filtering may be performed using the Hough transform result to remove interference and faults. In different embodiments, the lane boundary line 808 may be a guardrail, an interface between asphalt and grass, or another indicator of a lane boundary. Although depicted here as a single dashed / solid line, lane markings 808, 810a, and 810b may be solid lines or double lines (e.g., double solid lines, dashed and solid lines), etc. The purpose of the image degradation / downsampling operation is to reduce the size of the image used for the compressed latent representation to reduce computational costs. In such an embodiment, extracting a latent representation from an input image involves obtaining a compact and low-dimensional representation of the image that embodies the basic features and patterns contained within the image.

[0059] In such Figures 9 and 10 In some embodiments shown, the sparse latent representations 918 and 1018 may be the sparse latent representations 212 and 310. The sparse latent representations 918 and 1018 may be further simplified from the compressed latent representations 904 and 1004, respectively, to provide a view with even fewer elements, which remain the only essential elements of the strategy header to determine driving maneuver decisions according to different tasks. The purpose of the masking operation is to selectively keep or discard certain pixel values. For example, for a Fig. 9For the lane centering task shown, only elements related to lane conditions and other vehicles are retained in the sparse latent representation 918, such as lane boundary lines 908, first lane center lines 910a, second lane center lines 910b, and other moving vehicles 916. Elements that are not useful for the lane centering task, such as tree outlines 906, traffic sign outlines 912, traffic sign text 914, and other irrelevant elements such as other vehicles 916, are removed from the compressed latent representation image 904 to generate a sparse latent representation image 918 for better data sparsity. Similarly, for the task of traffic sign reading, only elements related to traffic signs, such as traffic sign outlines 1012 and traffic sign text 1014, are retained in the sparse latent representation 1018, compared to the compressed latent representation 1004 for this particular task.

[0060] In some embodiments, the above functions / features can be implemented with hardware, software, firmware, or any combination thereof. If implemented in software, the functions can be stored as one or more instructions or codes on a non-transitory computer-readable storage medium or a non-transitory processor-readable storage medium. The blocks of the methods or algorithms disclosed herein can be implemented in a processor-executable software module, which can reside on a non-transitory computer-readable or processor-readable storage medium. A non-transitory computer-readable or processor-readable storage medium can be any storage medium that can be accessed by a computer or processor. As an example and not limitation, such a non-transitory computer-readable or processor-readable storage medium can include RAM, ROM, EEPROM, flash memory, CD-ROM or other optical disk storage, disk storage or other magnetic storage device, or any other medium that can be used to store the required program code in the form of instructions or data structures and can be accessed by a computer. As used herein, disks and optical disks include compact disks (CDs), laser disks, optical optical disks, digital versatile disks (DVDs), floppy disks, and Blu-ray disks, wherein disks generally reproduce data magnetically, and optical disks reproduce data optically with lasers. The above combinations are also included within the scope of non-transitory computer-readable and processor-readable media. Additionally, the operations of a method or algorithm may reside as one or any combination or set of codes and / or instructions on a non-transitory processor-readable storage medium and / or computer-readable storage medium, which may be incorporated into a computer program product.

[0061] Fig.11An exemplary hardware and software environment of an autonomous vehicle 1100 in which the various techniques disclosed herein may be implemented is illustrated. For example, the vehicle 1100 is shown traveling on a road 1101, and the vehicle 1100 may include: a powertrain 1102 including a prime mover 1106 powered by an energy source 1104 and capable of providing power to a drivetrain 1108; and a vehicle operating system 1110 including a directional control 1112, a powertrain control 1114, and a brake control 1116. The vehicle 1100 may be implemented as any number of different types of vehicles, including vehicles capable of transporting people and / or cargo and capable of traveling by land, by sea, by air, underground, under the sea, and / or in space, and it should be appreciated that the above-described components 1102-1116 may vary widely based on the type of vehicle in which they are used.

[0062] For simplicity, the embodiments discussed below will focus on wheeled land vehicles, such as cars, vans, trucks, buses, etc. In such embodiments, the prime mover 1106 may include one or more electric motors and / or internal combustion engines (etc.). The energy source 1104 may include, for example, a fuel system (e.g., providing gasoline, diesel, hydrogen, etc.), a battery system, a solar panel or other renewable energy source, and / or a fuel cell system. The drive train 1108 may include wheels and / or tires together with a transmission and / or any other mechanical driving components suitable for converting the output of the prime mover 1106 into vehicle motion and one or more brakes configured to controllably stop or slow the vehicle 1100 and a direction or steering component suitable for controlling the trajectory of the vehicle 1100 (e.g., a rack and pinion steering linkage that enables one or more wheels of the vehicle 1100 to pivot about a substantially vertical axis to change the angle of the rotation plane of the wheel relative to the longitudinal axis of the vehicle). In some embodiments, a combination of powertrain and energy source may be used (e.g., in the case of an electric / gas hybrid vehicle), and in other embodiments, multiple electric motors (e.g., dedicated to individual wheels or axles) may be used as prime movers 1106. In the case of a hydrogen fuel cell implementation, prime mover 1106 may include one or more electric motors, and energy source 1104 may include a fuel cell system powered by hydrogen fuel.

[0063] Steering control 1112 may include one or more actuators and / or sensors for controlling and receiving feedback from a steering or steering assembly to enable vehicle 1100 to follow a desired trajectory. Powertrain control 1114 may be configured to control the output of powertrain 1102, e.g., controlling the output power of prime mover 1106, controlling the gears of a transmission in drivetrain 1108, etc., thereby controlling the speed and / or direction of vehicle 1100. Braking control 1116 may be configured to control one or more brakes that slow or stop vehicle 1100, e.g., disc or drum brakes coupled to the wheels of the vehicle.

[0064] Other vehicle types (including but not limited to all-terrain vehicles or tracked vehicles, and construction equipment) may utilize different powertrains, drivetrains, energy sources, directional controls, powertrain controls, and brake controls. In addition, in some embodiments, some components may be combined, for example, where the directional control of the vehicle is primarily handled by changing the output of one or more prime movers. Therefore, the embodiments disclosed herein are not limited to the specific application of the technology described herein in autonomous vehicles, wheeled vehicles, and land vehicles.

[0065] In the illustrated embodiment, full or semi-automatic control of the vehicle 1100 is implemented in a main vehicle control system 1118, which may include: one or more processors 1122 and one or more memories 1124, each processor 1122 being configured to execute program code instructions 1126 stored in the memory 1124. The processor 1122 may include, for example, a (multiple) graphics processing unit (GPU) and / or a (multiple) central processing unit (CPU). The processor 1122 may also include an application-specific integrated circuit (ASIC), other chipsets, logic circuits and / or data processing devices. The memory 1124 may be used to load and store data and / or instructions, such as for the control system 1118. The memory 1124 may include any combination of the following: suitable volatile memory (e.g., read-only memory (ROM), dynamic random access memory (DRAM), random access memory (RAM)), non-volatile memory (e.g., flash memory, memory card, storage medium) and / or other storage devices. When the embodiment is implemented in software, the technology described herein may be implemented with modules, processes, functions, entities, etc. that perform the functions described herein. The module may be stored in the memory and executed by the processor.The memory may be implemented within the processor or external to the processor, where the memory may be communicatively coupled to the processor via various means known in the art.

[0066] Sensors 1130 may include various sensors suitable for collecting information from the surrounding environment of the vehicle for controlling the operation of the vehicle 1100. For example, sensors 1130 may include: one or more detection and ranging sensors (e.g., RADAR sensor 1134, LIDAR sensor 1136, or both), satellite navigation (SATNAV) sensor 1132, for example, compatible with any of various satellite navigation systems (e.g., GPS (Global Positioning System), GLONASS (Global Navigation Satellite System), BeiDou Navigation Satellite System (BDS), Galileo, compass), etc. Radio detection and ranging (RADAR) 1134 and light detection and ranging (LIDAR) sensors 1136 and digital cameras 1138 (which may include various types of image capture devices capable of capturing still images and / or video images) can be used to sense stationary objects and moving objects in the immediate vicinity of the vehicle. Camera 1138 can be a monochrome camera or a stereo camera, and can record still images and / or video images. SATNAV sensor 1132 can be used to determine the location of the vehicle on the earth using satellite signals. The sensors 1130 may optionally include an inertial measurement unit (IMU) 1140. The IMU 1140 may include multiple gyroscopes and accelerometers capable of detecting linear and rotational motion of the vehicle 1100 in three directions. One or more other types of sensors (e.g., wheel rotation sensors / encoders 1142) may be used to monitor the rotation of one or more wheels of the vehicle 1100.

[0067] In various embodiments, the removable hardware pod is vehicle agnostic and can therefore be installed on a variety of non-autonomous vehicles, including: cars, buses, vans, trucks, mopeds, tractor trailers, sport vehicles, etc. Although autonomous vehicles typically contain a full sensor suite, in many embodiments, the removable hardware pod can contain a dedicated sensor suite that typically has fewer sensors than a fully autonomous vehicle sensor suite and can include: an IMU, a 3D positioning sensor, one or more cameras, a LIDAR unit, etc. Additionally or alternatively, the hardware pod can collect data from the non-autonomous vehicle itself, such as by integrating with the vehicle's CAN bus to collect various vehicle data including: vehicle speed data, braking data, steering control data, etc. In some embodiments, the removable hardware pod can include a computing device that can aggregate the data collected by the removable pod sensor suite and the vehicle data collected from the CAN bus, and upload the collected data to a computing system for further processing (e.g., uploading the data to the cloud). In many embodiments, a computing device in a removable pod can apply a timestamp to each instance of data before uploading the data for further processing. Additionally or alternatively, one or more sensors within the removable hardware pod can apply a timestamp to the data as it is collected (e.g., a lidar unit can provide its own timestamp). Similarly, a computing device within an autonomous vehicle can apply a timestamp to data collected by the autonomous vehicle's sensor suite, and can upload the timestamped autonomous vehicle data to a computer system for additional processing.

[0068] The output of the sensor 1130 can be provided to a set of main control subsystems 1120, including, for example, a positioning subsystem, a perception subsystem, a planning subsystem, and a control subsystem. The positioning subsystem is primarily responsible for accurately determining the position and orientation (sometimes also referred to as "pose" or "pose estimation") of the vehicle 1100 in its surroundings, usually in a certain reference frame. In some embodiments, the pose is stored in the memory 1124 as positioning data. In some embodiments, the surface model is generated from a high-definition map and stored in the memory 1124 as surface model data. In some embodiments, the detection and ranging sensors store their sensor data in the memory 1124 (for example, the radar data point cloud is stored as radar data). In some embodiments, calibration data is stored in the memory 1124. The perception subsystem is primarily responsible for detecting, tracking and / or identifying objects in the environment around the vehicle 1100. Machine learning models, such as those discussed above according to some embodiments, can be used to plan vehicle trajectories. The control subsystem 1120 is primarily responsible for generating appropriate control signals for controlling various controls in the vehicle control system 1118 so as to achieve the planned trajectory of the vehicle 1100. Similarly, the machine learning model can be used to generate one or more signals to control the autonomous vehicle 1100 to achieve the planned trajectory.

[0069] You should understand that Fig.11 The collection of components for vehicle control system 1118 shown in is only an example. In some embodiments, individual sensors may be omitted. Additionally or alternatively, in some embodiments, Fig.11 The same type of multiple sensors shown can be used for redundancy and / or to cover different areas around the vehicle. In addition, in addition to the above types, there can also be other types of additional sensors to provide actual sensor data related to the operation and environment of the wheeled land vehicle. Similarly, different types of control subsystems and / or combinations of control subsystems can be used in other embodiments. In addition, although the main control subsystem 1120 is illustrated as being separated from the processor 1122 and the memory 1124, it will be understood that in some embodiments, some or all of the functions of the main control subsystem 1120 can be implemented with program code instructions 1126 that reside in one or more memories 1124 and are executed by one or more processors 1122, and in some cases, the main control subsystem 1120 can be implemented using (multiple) the same processors and / or memories. The subsystem can be implemented at least in part using various dedicated circuit logic, various processors, various field programmable gate arrays (FPGAs), various application-specific integrated circuits (ASICs), various real-time controllers, etc., as described above, multiple subsystems can utilize circuits, processors, sensors, and / or other components. In addition, the various components in the vehicle control system 1118 can be networked in various ways.

[0070] For example, the vehicle 1100 may include one or more network interfaces, such as network interface 1154, which is suitable for communicating with one or more networks 1150 (e.g., a LAN, WAN, wireless network and / or the Internet, etc.) to allow information to be transmitted with other vehicles, computers and / or electronic devices, including, for example, a central service such as a cloud service, from which the vehicle 1100 receives environmental data and other data for its automatic control.

[0071] In addition, for additional storage, the vehicle 1100 may also include one or more mass storage devices, such as floppy disks or other removable disk drives, hard disk drives, direct access storage devices (DASD), optical drives (e.g., CD drives, DVD drives, etc.), solid-state storage drives (SSD), network attached storage, storage area networks, and / or tape drives, etc. In addition, the vehicle 1100 may include a user interface 1152 to enable the vehicle 1100 to receive multiple inputs from a user or operator and generate outputs for the user or operator, such as one or more displays, touch screens, voice and / or gesture interfaces, buttons and other tactile controls, etc. Otherwise, the user input may be received via another computer or electronic device, such as via an application on a mobile device or via a web interface, such as from a remote operator.

[0072] Disclosed herein are systems and methods relating to object detection and detection confidence. The disclosed methods may be suitable for autonomous driving, but may also be used for other applications, such as robotics, video analysis, weather forecasting, medical imaging, and the like. The present disclosure may be described with respect to an example autonomous driving vehicle 1100. Although the present disclosure primarily provides examples using autonomous driving vehicles, the various methods described herein may be implemented using other types of devices, such as robots, camera systems, weather forecasting devices, medical imaging devices, and the like. In addition, these methods may be used to control autonomous driving vehicles, or for other purposes, such as, but not limited to, video surveillance, video or image editing, video or image search or retrieval, object tracking, weather forecasting (e.g., using radar data), and / or medical imaging (e.g., using ultrasound or magnetic resonance imaging (MRI) data).

[0073] It is understood by a person skilled in the art that each of the units, algorithms, and steps described and disclosed in the embodiments of the present disclosure is implemented using electronic hardware or a combination of software and electronic hardware for a computer. Whether the function runs in hardware or in software depends on the conditions of the application and the design requirements of the technical solution. A person skilled in the art can use different ways to implement the functions for each specific application, and such implementation should not exceed the scope of the present disclosure. It is understood by a person skilled in the art that since the working processes of the above-mentioned systems, devices, and units are basically the same, he / she can refer to the working processes of the systems, devices, and units in the above-mentioned embodiments. For ease of description and simplification, these working processes will not be described in detail.

[0074] If the software functional unit is implemented, used and sold as a product, it can be stored in a readable storage medium in a computer. Based on this understanding, the technical solution proposed in the present disclosure can be basically or partially implemented in the form of a software product. Alternatively, a part of the technical solution that is beneficial to traditional technology can be implemented in the form of a software product. The software product in the computer is stored in a storage medium, which includes multiple commands for a computing device (such as a personal computer, a server or a network device) to run all or some of the steps disclosed in the embodiments of the present disclosure. The storage medium includes a USB disk, a mobile hard disk, a ROM, a RAM, a floppy disk or other types of media capable of storing program code. Although the present disclosure has been described in conjunction with what are considered to be the most practical and preferred embodiments, it should be understood that the present disclosure is not limited to the disclosed embodiments, but is intended to cover various arrangements made without departing from the scope of the broadest interpretation of the attached claims.

[0075] However, other modifications, changes and substitutions are also possible. Therefore, the description and drawings are considered to be illustrative, not restrictive. In the claims, any reference numerals placed in brackets should not be interpreted as limiting the claims. The word "comprising" does not exclude the existence of other elements or steps other than those listed in the claims. In addition, the term "one" or "an" as used herein is defined as one or more than one. In addition, the use of introductory phrases such as "at least one" and "one or more" in the claims should not be interpreted as implying that any particular claim containing such introduced claim elements is limited to an invention containing only one such element by introducing another claim element by the indefinite article "one" or "one", even when the same claim includes the introductory phrases "one or more" or "at least one" and indefinite articles such as "one" or "one". The same applies to the use of definite articles. Unless otherwise specified, terms such as "first" and "second" are used to arbitrarily distinguish the elements described by these terms. Therefore, these terms are not necessarily intended to indicate the time or other priority of such elements. The fact that certain measures are stated in mutually different claims does not mean that the combination of these measures cannot be used advantageously. Although certain features of the present invention have been illustrated and described herein, many modifications, substitutions, changes and equivalents may now occur to those skilled in the art. It is therefore to be understood that the appended claims are intended to cover all such modifications and changes that fall within the true spirit of the invention.

[0076] It should be understood that various features of embodiments of the present disclosure described in the context of separate embodiments for clarity may also be provided in combination in a single embodiment. Conversely, various features of embodiments of the present disclosure described in the context of a single embodiment for brevity may also be provided individually or in any suitable sub-combination. Those skilled in the art will appreciate that embodiments of the present invention are not limited by what has been specifically shown and described above. Instead, the scope of the embodiments of the present disclosure is defined by the appended claims and their equivalents.

[0077] The previous description of the disclosed embodiments is provided to enable others to make or use the disclosed subject matter. Various modifications to these embodiments will be apparent, and the general principles defined herein can be applied to other embodiments without departing from the spirit or scope of the previous description. Therefore, the previous description is not intended to be limited to the embodiments shown herein, but to conform to the broadest scope consistent with the principles and novel features disclosed herein. Therefore, the claims are not intended to be limited to the aspects shown herein, but to conform to the full scope consistent with the language claims, wherein the reference to the elements in the singular form is not intended to mean "one and only one", unless explicitly stated so, but "one or more". Unless otherwise specifically stated, the term "some" refers to one or more. All structural and functional equivalents of the elements of the various aspects described in the previous description (which are known or will be known later) are expressly introduced herein as a reference and are intended to be covered by the claims. In addition, whether such disclosure is explicitly stated in the claims, the content disclosed herein is not intended to be dedicated to the public. Unless the phrase "device for..." is used to explicitly state the elements, the claim elements should not be interpreted as device plus function. It should be understood that the specific order or hierarchy of blocks in the disclosed process is an example of an illustrative method. Based on design preferences, it is understood that the specific order or hierarchy of blocks in the process can be rearranged while remaining within the scope of the previous description. The attached method claims present elements of the various blocks in a sample order and are not meant to be limited to the specific order or hierarchy presented.

[0078] The various examples shown and described are provided only as examples to illustrate the various features of the claims. However, the features shown and described with respect to any given example are not necessarily limited to the associated examples, and may be used or combined with other examples shown and described. In addition, the claims are not intended to be limited by any one example. The above method descriptions and process flow charts are provided only as illustrative examples, and are not intended to require or imply that the blocks of the various examples must be executed in the order presented. As will be understood, the order of the blocks in the above examples can be executed in any order. Words such as "thereafter", "then", "next", etc. are not intended to limit the order of the blocks; these words are simply used to guide the reader through the description of the method. In addition, any reference to a claim element in the singular form, for example, using the article "one", "an" or "the", should not be interpreted as limiting the element to the singular. The various illustrative logic blocks, modules, circuits, and algorithm blocks described in conjunction with the examples disclosed herein may be implemented as electronic hardware, computer software, or a combination of both. In order to clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and blocks have been generally described above according to their functions. Whether such functionality is implemented as hardware or software depends on the specific application and the design constraints imposed on the entire system. Those skilled in the art may implement the described functionality in different ways for each specific application, but such implementation decisions should not be interpreted as causing departure from the scope of the present invention. Hardware for implementing the various illustrative logics, logic blocks, modules, and circuits described in conjunction with the examples disclosed herein may be implemented or executed with a general-purpose processor, DSP, ASIC, FPGA, or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. A general-purpose processor may be a microprocessor, but in an alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration. Alternatively, some blocks or methods may be performed by circuits specific to a given function.

[0079] Further examples are listed below.

[0080] Embodiment 1. A method for reducing the computational cost of an autonomous driving system, the method comprising: a) acquiring data related to a task for operating a vehicle; b) using the acquired data to train a deep learning model, wherein the deep learning model includes an encoder and a strategy head for the task; c) reducing the complexity of the data by passing the data acquired in step a) to the encoder to generate a compressed latent representation of the data; and d) determining a driving operation using the compressed latent representation of the data through the strategy head.

[0081] Embodiment 2. A method according to embodiment 1, wherein the acquired data includes recorded human driving data from the same vehicle or a separate vehicle.

[0082] Embodiment 3. A method according to any one of embodiments 1-2, wherein the acquired data includes artificially enhanced data.

[0083] Embodiment 4. A method according to any one of embodiments 1-3, wherein the data is acquired using sensors of the same vehicle or a separate vehicle, and the sensors include one or more lidar sensors, radar sensors, infrared sensors and / or image sensors.

[0084] Embodiment 5. The method according to any one of embodiments 1-4, wherein step c) further comprises: applying a mask of element-wise multiplication of the compressed latent representation to further reduce the complexity of the data obtained in step a).

[0085] Embodiment 6. The method according to embodiment 5 further includes: normalizing the mask value.

[0086] Embodiment 7 The method according to any one of embodiments 1-6 further includes: applying a loss function to evaluate the difference between the driving operation determined by the strategy head and a driving operation benchmark.

[0087] Embodiment 8. The method according to any one of Embodiments 1-7 further includes: configuring one or more overlapping elements of the compressed latent representation generated by the first encoder of the first deep learning model so that the compressed latent representation is configured to be shareable by the second encoder of the second deep learning model.

[0088] Embodiment 9. A method for reducing the computational cost of a driving system, the method comprising: a) acquiring data related to a task for operating a vehicle; b) using the acquired data to operate a deep learning model, wherein the deep learning model includes a strategy head for the task; c) acquiring a compressed latent representation of the data acquired in step a); and d) determining a driving operation using the compressed latent representation of the data through the strategy head.

[0089] Embodiment 10. A method according to embodiment 9, wherein the acquired data includes recorded human driving data from the same vehicle or a separate vehicle.

[0090] Embodiment 11. A method according to any one of embodiments 9-10, wherein the acquired data includes artificially enhanced data.

[0091] Embodiment 12. A method according to any one of embodiments 9-11, wherein the data is acquired using sensors of the same vehicle or a separate vehicle, and the sensors include one or more lidar sensors, radar sensors, infrared sensors and / or image sensors.

[0092] Embodiment 13. A method according to any one of embodiments 9-12, wherein step c) further comprises: applying a mask that is element-wise multiplied with the compressed latent representation to further reduce the complexity of the data obtained in step a).

[0093] Embodiment 14. The method according to embodiment 13 further includes: normalizing the mask value.

[0094] Embodiment 15. The method according to any one of embodiments 9-14 further includes: applying a loss function to evaluate the difference between the driving operation determined by the strategy head and a driving operation benchmark.

[0095] Embodiment 16. The method according to any one of Embodiments 9-15 further includes: configuring one or more overlapping elements of the compressed latent representation generated by the first encoder of the first deep learning model, so that the compressed latent representation is configured to be shareable by the second encoder of the second deep learning model.

[0096] Embodiment 17. A method for reducing the computational cost of an autonomous driving system, the method comprising: a) acquiring data related to a task for operating a vehicle; b) using the acquired data to train a first deep learning model, wherein the first deep learning model comprises a first encoder and a policy head; c) identifying one or more overlapping elements between the data related to the task and a compressed latent representation related to another task, wherein the compressed latent representation is generated by a second deep learning model having a second encoder, and the compressed latent representation is configured to be shareable with the first encoder of the first deep learning model; and d) determining a driving operation by the policy head using the compressed latent representation generated by the second deep learning model having the second encoder.

[0097] Embodiment 18. A method according to Embodiment 17, wherein the acquired data includes recorded human driving data from the same vehicle or a separate vehicle.

[0098] Embodiment 19. A method according to any one of embodiments 17-18, wherein the acquired data includes artificially enhanced data.

[0099] Embodiment 20. A method according to any one of embodiments 17-19, wherein the data is acquired using sensors of the same vehicle or a separate vehicle, and the sensors include one or more lidar sensors, radar sensors, infrared sensors and / or image sensors.

[0100] Embodiment 21. A device for operating an autonomous driving system, the device comprising: at least one processor and a memory storing instructions, wherein the instructions, when executed by the at least one processor, cause the at least one processor to perform operations, the operations comprising the method described in any one of Embodiments 1-20.

[0101] Embodiment 22. A non-temporary computer-readable storage medium storing computer instructions, wherein the computer instructions are executable by one or more processors to perform a method for controlling an autonomous driving system and reducing the computational costs associated therewith, the method comprising any one of embodiments 1-20.

Claims

1. A method for reducing the computational cost of an autonomous driving system, the method comprising the following steps: a) acquiring data related to tasks used to operate the vehicle; b) using the acquired data to train a deep learning model, wherein: The deep learning model includes an encoder and a policy head for the task; c) reducing the complexity of the data acquired in step a by passing the data to the encoder to generate a compressed latent representation of the data; as well as d) determining, by the strategy head, a driving maneuver using the compressed latent representation of the data.

2. The method according to claim 1, wherein: The acquired data includes recorded human driving data from the same vehicle or a separate vehicle, or human-augmented data.

3. The method according to claim 1, wherein: The data is acquired using sensors of the same vehicle or a separate vehicle, including one or more lidar sensors, radar sensors, infrared sensors and / or image sensors.

4. The method according to claim 1, wherein: The step c further comprises: applying a mask which is element-wise multiplied with the compressed latent representation to further reduce the complexity of the data obtained in step a.

5. The method according to claim 4, further comprising: Normalizes the mask values.

6. The method according to claim 1, further comprising: A loss function is applied to evaluate the difference between the driving operation determined by the policy head and a driving operation benchmark.

7. The method according to claim 1, further comprising: One or more overlapping elements of the compressed latent representation produced by a first encoder of a first deep learning model are configured such that the compressed latent representation is configured to be shareable by a second encoder of a second deep learning model.

8. A non-transitory computer-readable storage medium having instructions stored thereon, which when executed by one or more processors cause the one or more processors to perform the method according to any one of claims 1-7.

9. A method for reducing the computational cost of an autonomous driving system, the method comprising the following steps: a) acquiring data related to tasks used to operate the vehicle; b) using the acquired data to operate a deep learning model, wherein: The deep learning model includes a policy head for the task; c) obtaining a compressed latent representation of the data obtained in step a; and d) determining, by the strategy head, a driving maneuver using the compressed latent representation of the data.

10. A method for reducing the computational cost of an autonomous driving system, the method comprising the following steps: a) acquiring data related to tasks used to operate the vehicle; b) using the acquired data to train a first deep learning model, wherein: The first deep learning model includes a first encoder and a strategy head; c) identifying one or more overlapping elements between the data associated with the task and a compressed latent representation associated with another task, wherein the compressed latent representation is produced by a second deep learning model having a second encoder, the compressed latent representation being configured to be shareable with the first encoder of the first deep learning model; and d) determining, by the policy head, a driving maneuver using the compressed latent representation produced by the second deep learning model with the second encoder.

Images