A method for operating a support system of a motor vehicle, a computer program product, a non-transitory computer-readable storage medium,

Abstract

A support system for a motor vehicle uses a base modeal for a plurality of operational design domains, but the weights of the base model are unchangeable. Training of the base model uses respective tr

Description

Mercedes-Benz Group AG Negi 2025-05-06 A METHOD FOR OPERATING A SUPPORT SYSTEM OF A MOTOR VEHICLE, A COMPUTER PROGRAM PRODUCT, A NON-TRANSITORY COMPUTER-READABLE STORAGE MEDIUM, AS WELL AS A SUPPORT SYSTEM FIELD OF THE INVENTION

[0001] The invention relates to the field of automobiles. More specifically, the present invention relates to a method for operating a support system of a motor vehicle. Furthermore, the present invention relates to a corresponding computer program product, a corresponding non-transitory computer-readable storage medium, as well as to a corresponding support system. BACKGROUND INFORMATION

[0002] The so-called operational design domain (ODD) adaption is a big topic, in particular in at least in part automatically operated motor vehicles or fully automatically operated motor vehicles. In particular, because of different markets, different weather conditions, different illumination conditions or different use cases the operational design domain adaption is needed. In particular, there is a need to develop a machine learning model or any model, so that it can be adapted to these different operational design domains with reasonable efforts and costs.

[0003] According to the state of the art, a single model can be trained and may include also operational design domains in the training data. The hope is that this single model trained with data from all operational design domains can cover all operational design domains. But this can result in huge model size and / or compromise model accuracy because the model is trying to do too much. Furthermore, training or pretraining different models can be provided. This is the most straightforward way to get great accuracy for all operational design domains but it is time consuming, and the effort is high. A real-world deployment may be challenging.

[0004] Therefore, there is a need in the art to provide a solution how to train a model for different operational design domains in an improved manner. SUMMARY OF THE INVENTION

[0005] It is an object of the present invention to provide a method, a corresponding computer program product, a corresponding non-transitory computer-readable storage medium, as well as a corresponding support system, by which the disadvantages of the state of the art are overcome.

[0006] In particular, it is an object of the present invention to provide a method, a corresponding computer program product, a corresponding non-transitory computer-readable storage medium, as well as a corresponding support system, by which a model for different operational design domains can be trained in an improved manner.

[0007] This object is solved by a method, a corresponding computer program product, a corresponding non-transitory computer-readable storage medium, as well as a corresponding support system according to the independent claims. Advantageous embodiments are presented in the dependent claims.

[0008] One aspect of the invention relates to a method for operating a support system of a motor vehicle. A mathematical base model for a plurality of operational design domains is provided for the motor vehicle by an electronic computing device of the support system, wherein weights of the base model are unchangeable. Respective training data for at least two different operational design domains of the plurality of operational design domains are provided by the electronic computing device. Model parameters for each of the at least two different operational design domains are provided by the electronic computing device. Respective models are generated for each of the at least two different operational design domains by training the base model with the respective training data and the respective model parameters.

[0009] Therefore, a single model is trained, but with different prediction head / path for a different operational design domain. The different operational design domains therefore share the same backbone. Therefore, a combination of the two solutions known from the state of the art is provided.

[0010] Therefore, an extra trainable parameter are used, in particular the model parameters, after the base model is trained, and these extra parameters are trained with the weights of the original base model being frozen / unchangeable. This can reduce memory-usage and speed-up training and generates these very light weight extra parameters that can be deployed with the original model in different operational design domains. These extra trained parameters can be implemented in the form of extra added layers, for example adapter layers, or low-rank compositions of the update of the weight’s matrixes, for example LoRA, or extra added vectors, for example IA3.

[0011] The training procedure may comprise the following steps: training the base model with the data including all operational design domains. Freezing the parameters of the base model and adding extra trainable parameters and fine-tune the new model in the different operational design domains. The trained extra parameter with the original base model is deployed as the new operational design domain adapted model for each operational design domain.

[0012] According to an embodiment, the mathematical base model is an artificial intelligence.

[0013] In another embodiment, the model parameters are provided as low-rank adaption weights and / or as added layers and / or as added vectors.

[0014] In another embodiment, the training data are provided for different countries and / or different weather conditions and / or different day times and / or different road conditions and / or different illumination conditions and / or different use cases.

[0015] In another embodiment, the training data are sensor data and / or camera image data and / or motion control data.

[0016] In another embodiment the model weights are stored in a cloud.

[0017] In another embodiment, the LoRA weights can be downloaded on the fly or at down time.

[0018] In particular, the method is a computer-implemented method. Therefore, another aspect of the invention relates to a computer program product comprising program code means for performing a method according to the preceding aspect.

[0019] Furthermore, the present invention relates to a non-transitory computer-readable storage medium comprising at least the computer program product according to the preceding aspect.

[0020] Furthermore, the present invention relates to a support system for a motor vehicle, wherein the support system at least comprises an electronic computing device, and wherein the support system is configured for performing a method according to the preceding aspect. In particular, the method is performed by the support system.

[0021] A still further aspect of the invention relates to a motor vehicle comprising at least the support system according to the preceding aspect. The motor vehicle may be configured as an at least in part automatically operated motor vehicle or a fully automatically operated motor vehicle.

[0022] Advantageous embodiment of the method are to be regarded as advantageous embodiments of the computer program product, the non-transitory computer-readable storage medium, the support system, as well as the motor vehicle. The support system, as well as the motor vehicle therefore comprises means for performing the method.

[0023] In particular, the computing unit / electronic computing device may include one or more computers, one or more microcontrollers, and / or one or more integrated circuits, for example, one or more application-specific integrated circuits, ASIC, one or more field-programmable gate arrays, FPGA, and / or one or more systems on a chip, SoC. The computing unit may also include one or more processors, for example one or more microprocessors, one or more central processing units, CPU, one or more graphics processing units, GPU, and / or one or more signal processors, in particular one or more digital signal processors, DSP. The computing unit may also include a physical or a virtual cluster of computers or other of said units.

[0024] In various embodiments, the computing unit includes one or more hardware and / or software interfaces and / or one or more memory units.

[0025] A memory unit may be implemented as a volatile data memory, for example a dynamic random access memory, DRAM, or a static random access memory, SRAM, or as a non-volatile data memory, for example a read-only memory, ROM, a programmable read-only memory, PROM, an erasable programmable read-only memory, EPROM, an electrically erasable programmable read-only memory, EEPROM, a flash memory or flash EEPROM, a ferroelectric random access memory, FRAM, a magnetoresistive random access memory, MRAM, ora phase-change random access memory, PCRAM.

[0026] In an example embodiment the electronic computing device is onboard the vehicle. For example, the electronic computing device and other components of the vehicle may be housed, located, or otherwise included on or within the vehicle. The electronic computing device may be configured to execute the computing functions and operations of the vehicle.

[0027] In some implementations, the one or more integrated circuits may be part of, or may form, a vehicle control unit (also referred to as a vehicle controller) that is embedded or otherwise disposed in a vehicle (e.g., a Mercedes-Benz® car or van). For example, the vehicle controller may be or may include an infotainment system controller (e.g., an infotainment head-unit), a telematics control unit (TCU), an electronic control unit (ECU), a central powertrain controller (CPC), a charging controller, a central exterior &interior controller (CEIC), a zone controller, or any other controller. In an embodiment, the one or more integrated circuits may be programmed by one or more computer-readable or computer-executable instructions stored on the non-transitory computer-readable medium.

[0028] Further advantages, features, and details of the invention derive from the following description of preferred embodiments as well as from the drawings. The features and feature combinations previously mentioned in the description as well as the features and feature combinations mentioned in the following description of the figures and / or shown in the figures alone can be employed not only in the respectively indicated combination but also in any other combination or taken alone without leaving the scope of the invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] The novel features and characteristic of the disclosure are set forth in the appended claims. The accompanying drawings, which are incorporated in and constitute a part of this disclosure, illustrate exemplary embodiments and together with the description, serve to explain the disclosed principles. The same numbers are used throughout the figures to reference like features and components. Some embodiments of system and / or methods in accordance with embodiments of the present subject matter are now described below, by way of example only, and with reference to the accompanying figures.

[0030] The drawings show in:

[0031] Fig 1 a schematic side view according to an embodiment of a motor vehicle comprising an embodiment of the support system;

[0032] Fig. 2 a schematic block diagram according to an embodiment of an electronic computing device; and

[0033] Fig. 3 another schematic block diagram according to an embodiment of the electronic computing device.

[0034] In the figures the same elements or elements having the same function are indicated by the same reference signs. DETAILED DESCRIPTION

[0035] In the present document, the word "exemplary" is used herein to mean "serving as an example, instance, or illustration". Any embodiment or implementation of the present subject matter described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other embodiments.

[0036] While the disclosure is susceptible to various modifications and alternative forms, specific embodiments thereof have been shown by way of example in the drawing and will be described in detail below. It should be understood, however, that it is not intended to limit the disclosure to the particular forms disclosed, but on the contrary, the disclosure is to cover all modifications, equivalents, and alternatives falling within the scope of the disclosure.

[0037] The terms “comprises”, “comprising”, or any other variations thereof, are intended to cover a non-exclusive inclusion so that a setup, device or method that comprises a list of components or steps does not include only those components or steps but may include other components or steps not expressly listed or inherent to such setup or device or method. In other words, one or more elements in a system or apparatus preceded by “comprises” or “comprise” does not or do not, without more constraints, preclude the existence of other elements or additional elements in the system or method.

[0038] In the following detailed description of the embodiment of the disclosure, reference is made to the accompanying drawing that forms part hereof, and in which is shown by way of illustration a specific embodiment in which the disclosure may be practiced. This embodiment is described in sufficient detail to enable those skilled in the art to practice the disclosure, and it is to be understood that other embodiments may be utilized and that changes may be made without departing from the scope of the present disclosure. The following description is, therefore, not to be taken in a limiting sense.

[0039] Fig. 1 shows a schematic side view according to an embodiment of a motor vehicle 10. The motor vehicle 10 is, for example, at least in part automatically operated or fully automatically operated. The motor vehicle 10 may comprise a support system 12. The support system 12 may at least comprise an electronic computing device 14.

[0040] Fig. 2 shows a schematic block diagram according to an embodiment of the electronic computing device 14, in particular of a so-called mathematical model 16, which may be stored inside the electronic computing device 14. In particular, Fig. 2 shows input data 18, pretrained weights 20, first data weights 22, second data weights 24 as well as a hidden layer 26. In particular, Fig. 2 shows an example of a so-called low rank adaption (LoRA).

[0041]

[0042] In particular, according to the shown embodiment a single foundation model is provided, which may be represented by the pretrained weights 20. Furthermore, the first delta weights 22 and the second delta weights 24 represent the updated weights, in particular the so-called LoRA-weights. Furthermore, instead of updating the weights for the entire model, only the delta weights 22, 24, represented by low rank decomposition matrices are updated. Therefore, fine tuning of the models, in particular of the different operational design domains, using different LoRA-weights is provided. The different LoRA-weights based on the operational design domains are applied during the so-called inference time.

[0043] Fig. 3 shows another schematic block diagram according to an embodiment of the electronic computing device 14. In particular, a base model 28 is shown, which is trained with all training data 30. Furthermore, three different operational design domains 32, 34, 36 are shown. In particular, a first operational design domain 32, a second operational domain 34, and a third operational domain 36 are shown. For each of the operational design domains 32, 34, 36 specific training data 38 are provided.

[0044] These training data 38 are then provided for the respective model 40, 42, 44. For example, a first respective model 40, a second respective model 42 and a third respective model 44 is generated. The respective models 40, 42, 44 may use the base model 20 and specific model parameters 46, 48, 50 for each of the respective models 40, 42, 44. In particular, the first respective model 40 uses first model parameters 46, the second respective model 42 uses second model parameters 48, and the third respective model 44 uses third respective model parameters 50.

[0045] According to the shown embodiment, the base model 28 is provided for a plurality of operational design domains 32, 34, 36 of the motor vehicle 10 by the electronic computing device 14, wherein the weights of the base model 28 are unchangeable. Respective training data 38 for at least two different operational design domains 32, 34, 36 of the plurality of operational design domains 32, 34, 36 are provided. The model parameters 46, 48, 50 for each of the at least two different operational design domains 32, 34, 36 are provided. The respective models 40, 42, 44 for each of the at least two different operational design domains 32, 34, 36 are generated by training the base model 28 with the respective training data 38 and the respective model parameters 46, 48, 50.

[0046] In particular, as shown, the model is an artificial intelligence. Furthermore, it is shown that the model parameters 46, 48, 50 are provided as low-rank adaption weights and / or as added layers and / or added vectors.

[0047] Furthermore, the training data 38 are provided for different countries and / or different weather conditions and / or different day times and / or different road conditions and / or different illumination conditions and / or different use cases. Furthermore, the training data 30 are sensor data and / or camera image data and / or motion control data.

[0048] Therfore, the present disclosure relates to a method, system, and computer program product for Operation Design Domain (ODD) adaptation for Deep Neural Networks (DNNs). More specifically, it relates to ODD adaptation for DNNs in semi or fully Autonomous Driving (AD) development.

[0049] One aspect of the present disclosure relates to one or more domain-specific Al (artificial intelligence) models, such as models used for scene understanding, semantic analysis, etc. In an embodiment, these models may be tailored for a specific ODD, or operational design domain (which is discussed in more detail below). In some implementations, this method may include a computing system (e.g., a set of servers or other computing devices) that trains a ODD-specific model. The training may include the computing system accessing a base model, and training separate parameters while keeping the base model frozen (or, more specifically, keeping weights of the base model unchanged). For example, the computing system may initialize these separate parameters (also referred to as extra parameters) with initial (e.g., random) values, and apply ODD-specific training data, so as to train these separate parameters to be suitable for specific ODD’s. The training data may be associated with a specific ODD (such as training data generated from a specific country, a specific time of day, a specific illumination condition, a specific weather condition, or some other domain). Examples of the training data may include sensor data, camera image data, motion control data, or other data associated with the vehicle and generated under conditions for a specific ODD. Thus, the computing system may adjust the values of the separate parameters based on the training data. It is to be noted that during the adaptation for different ODDs, the weight(s) of the base model is frozen (untrainable).

[0050] ODD is a term for a specific operating context for an autonomous vehicle. The context is defined by a set of conditions, including environmental, geographical, time of day, and other conditions. Some of the context also varies based on different markets, such as US, Europe, China and so on. Further, it may relate to different weather conditions, such as sunny, rainy, snowy and so on. Furthermore, it may relate to different illumination conditions, such as bright, dark and so on. In addition, it may also relate to different use cases, for example, highway driving, urban driving, low speed driving, low speed parking and so on. The AD system recognizes the ODD and modifies its behavior accordingly.

[0051] According to US Department of Transportation report, an ODD is defined in terms of physical infrastructure, operational constraints, objects, connectivity, environmental conditions, and zones. Physical infrastructure includes roadway types, surfaces, edges and geometry. Operational constraints include speed limits and traffic conditions. Environmental conditions include weather, illumination, etc. Zones include regions, states, school areas, and construction sites. For example, in 2022, Mercedes-Benz announced a product with an ODD of Level 3 autonomous driving at 130 km / hr.

[0052] The present disclosure relates to method, system and computer program products for developing models, for example, machine learning models, DNNs such that the models could be adapted to different ODDs (as explained above) within reasonable effort and cost.

[0053] One of the existing approaches includes training a single model for all the ODDs in the training data with the hope that this single model trained with data from all ODDs can cater to all ODDs. But this can result in huge model size, and / or compromised model accuracy because the model is trying to do too much. This approach may provide great accuracy for all ODDs, but it is time consuming and costs a lot. Further, real-world deployment might be challenging.

[0054] Another existing approach is training a single model, but with different prediction “heads / paths” for different ODDs, wherein all the “heads / paths” share the same “trunk / backbone”.

[0055] The present disclosure provides added extra trainable parameters after a base model that is trained and then trains the added (extra) parameters with the weights of the base model being frozen. This could reduce memory-usage and speeds up training and generates lightweight extra parameters that can be deployed with the original model in different ODDs. These extra trainable parameters can be implemented in the form of extra added layers (e.g., Adapters layers) or low-rank decompositions of the update of the weight matrices (e.g., LoRA), or extra added vectors (e.g., IA3).

[0056] In an example embodiment, the added trainable parameters can be in different forms, for example, extra weight matrices or extra DNN layer. The added parameters compared to original model could range from 0.1% to something high like 20% depending on how much adaptation is needed.

[0057] In an example embodiment, the added parameters will be applied on top of the base model and treated as a new model. It’s input and output will be different as compared to the base model.

[0058] This approach may be referred to as fine-tuning a new model for a specific ODD. This includes (i) taking the base model, (ii) initializing the extra parameters (e.g., with random values), (iii) applying training data that includes ODD-specific data (e.g., sensor data, image data, navigation data, motion planning data), (iv) adjusting the extra parameters based on the ODD-specific training data. It is to be noted that during the adaptation for different ODD, the weight(s) of the base model is frozen (untrainable).

[0059] Various example embodiment includes achieving ODD adaptation without regressing / degrading the performance of AD system based on DNNs.

[0060] As shown in Fig. 3, the ODD adaptation includes training a base model with all the data including all ODDs. This is performed by freezing the parameters of the base model, adding extra trainable parameters (different sets of trained extra parameters corresponding to the different ODDs) and then fine-tuning the new model in different ODDs.

[0061] The additional trainable parameters may be introduced using a mechanism known as LoRA (Low-Rank Adaptation). Specifically, two low-rank matrices, A and B, may be added to selected layers of the frozen pre-trained weight matrices such that the adapted weight becomes W_adapted = W_frozen + BA.

[0062] In this setup, matrix A may be typically initialized with random values and matrix B may be initialized with zeros. During training, only A and B may be updated using gradient descent, while the original weight matrix W remains fixed. The resulting adapted output of the layer is W x + (BA)x, where B A represents a low-rank approximation of task-specific adjustments for the given ODD. The fixed base weights W ensure that the original model knowledge is preserved, while the additional matrices A and B are optimized to minimize the task-specific loss. This enables a highly parameter-efficient fine-tuning process tailored to different operational design domains.

[0063] The trainable parameters are thus limited to these low-rank matrices A and B.

[0064] In an example embodiment, the adaptation of the base model to a specific operational design domain (ODD) may involve the creation of a corresponding dataset and the configuration of a training script. For the dataset, sensor data such as camera images, LiDAR point clouds, and radar measurements are collected under target ODD conditions, for example rain, night-time, or urban environments.

[0065] These data are annotated with task-specific labels, such as bounding boxes, drivable areas, or traffic rule indicators.

[0066] Rare and safety-critical scenarios, such as pedestrians at night, may be balanced through oversampling or the use of synthetic data, for example via simulation.

[0067] The collected and annotated data are preprocessed to match the input format of the base model, including adjustments for resolution and normalization.

[0068] The training script for fine-tuning the LoRA parameters is configured with domainspecific hyperparameters. In an example embodiment, the following parameters are set: - LoRA rank (r): Typically in the range of 4 to 16, to balance computational efficiency and adaptation capability. -Target layers: LoRA may be applied to selected layers of the base model, e.g. attention heads in transformer architectures or final convolutional blocks. - Learning rate: Higher than during original pretraining, e.g. between 1e-4 and 3e-4. - Batch size: Set according to available GPU memory, e.g. between 8 and 32 samples. - Loss function: Aligned with the ODD-specific task, such as object detection loss or trajectory prediction error. - Epochs: Fine-tuning may be typically conducted over 5 to 20 epochs to avoid overfitting. - Freeze base model: The weights of the base model remain unchanged, and only the LoRA matrices A and B are updated via gradient descent.

[0069] This approach enables a resource-efficient, task-specific adaptation of the base model to various ODDs while retaining the generalization capacity of the pretrained model.

[0070] Thereafter, the trained extra parameters with the original base model are deployed as the new ODD adapted model for each ODD specific scenario. In an example embodiment, the ODD adaptation is performed by a computing system (server, controllers, GPUs). With the proposed method, the computing system has a technical effect of computing and applying ODDs in a time and cost-efficient manner. This is explained in more detail below.

[0071] As shown in Fig. 3, multiple ODD-specific models are created, example 3 ODD-specific modules. The base model (shown in blue) along with the multiple ODD-specific modules (red ODD 1, green ODD2 and yellow ODD 3) may be stored in a vehicle. The multiple ODD-specific modules have been trained on extra parameters (as explained above). These different sets of trained extra parameters are corresponding to the different ODDs. The base model and different sets of trained extra parameters (based on ODD) are stored in the vehicle, thereby leveraging the memory optimally. If ODDs need to be switched, the corresponding red, green or yellow layers could be applied (insert layer) to the base model (blue color) and accordingly, the AD system plans it behavior. Such a distributed method of storing the base model and different sets of trained extra parameters corresponding to different ODDs resolve the problem of heavy costs and compute and could be applied in different ODDs in an efficient manner.

[0072] In an example embodiment, the electronic computing device onboard the vehicle may be implemented using a high-performance automotive system-on-chip (SoC), such as Nvidia Orin. This onboard computing platform enables real-time inference based on the adapted model by dynamically applying ODD-specific LoRA weights in the operational phase.

[0073] The loading of LoRA weights corresponding to a particular ODD may be triggered in multiple ways. For example, the selection may be based on a currently used function (e.g., if the user activates an automatic parking assist (APA), parking-specific weights are loaded); a detected market (e.g., regional configuration such as Ell, US, or ON); an onboard ODD monitoring module which detects the current operational conditions and selects the appropriate weights accordingly; a geolocation signal e.g derived from GPS data; or a user preference or user-selected driving mode (e.g., calm, assertive, preferred speed level).

[0074] Any model may be used as the base model, including end-to-end architectures that are composed of multiple interconnected modules. Such models are typically company-specific and may not follow a standardized naming convention. The separate parameters used for adaptation to different operational design domains may be implemented as matrix-based numerical data structures, such as LoRA matrices. These parameters may be stored onboard the vehicle and are loaded as needed during operation.

[0075] The values of the separate parameters, such as LoRA matrices, are determined during an offline pre-training phase for each target operational design domain (ODD). These domain-specific LoRA weights are stored onboard the vehicle together with the frozen base model weights. When a particular ODD is triggered, e.g., based on function, geography, or driving context, the corresponding LoRA weights are loaded and combined with the base model.

[0076] In another embodiment, prefix tuning may be used as an alternative to LoRA. In this approach, continuous task-specific vectors, referred to as prefixes, are learned and prepended to the model’s input or inserted at intermediate layers. In the context of autonomous driving (self-driving), such prefixes may represent operational design domain (ODD) factors, for example, weather or road type. The prefix vectors influence the output of the frozen base model without modifying its internal weights. The only trainable parameters in this case are the prefix vectors themselves, typically with a fixed length between 20 and 100 tokens. Reference signs 10 12 14 16 18 20 22 24 26 28 30 32 34 36 38 40 42 44 46 48 50 motor vehicle support system electronic computing device mathematical model input pretrained weights first delta weights second delta weights hidden layer base model training data first operational design domain second operational design domain third operational design domain training data first respective model second respective model third respective model first model parameters second model parameters third model parameters

Images