Device and method for training a neural network

By decomposing and grouping singular values and vectors with distinct learning rates and optimizers, the method efficiently tunes pre-trained neural networks, addressing computational inefficiencies and overfitting, achieving faster and more effective adaptation to downstream tasks.

DE102024211284A1Pending Publication Date: 2026-05-28ROBERT BOSCH GMBH
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
DE102024211284
Authority / Receiving Office
DE · DE
Patent Type
Applications
Current Assignee / Owner
Filing Date
2024-11-25
Publication Date
2026-05-28

AI Technical Summary

Technical Problem

Adapting large pre-trained neural networks to specific downstream tasks is computationally expensive and resource-intensive, often leading to overfitting, especially with limited training data, and existing methods like PiSSA fail to effectively utilize lower singular values and lack flexibility in parameter tuning.

Method used

Decompose weight matrices using SVD, group singular values and vectors, and apply distinct learning rates and optimizers to each group for tailored fine-tuning, allowing selective preservation of pre-trained knowledge and rapid adjustment to new tasks.

Benefits of technology

This approach reduces computational complexity, minimizes overfitting, and enhances performance by dynamically adjusting to the importance of different features during training, leading to faster convergence and better adaptation to new tasks.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 00000000_0000_ABST
    Figure 00000000_0000_ABST
Patent Text Reader

Abstract

A computer-implemented method for training a neural network (60), wherein the neural network is designed to accept an input (x) characterizing at least one sensor measurement (S) and to provide an output (x) characterizing a classification and / or regression result of the at least one sensor measurement (S) and / or a probability that the sensor measurement (S) occurs among a set of sensor measurements, or wherein the neural network (60) is designed to provide an output (x) characterizing a prediction of a sensor measurement for generating a training and / or test data set for training another machine learning system, wherein the training method comprises the following steps: • Performing a singular value decomposition of a weight matrix or weight tensor of the neural network (60); • Grouping singular values ​​and corresponding singular vectors into a plurality of groups based on the magnitudes of the singular values; • Training the neural network by adjusting each group according to a distinct non-zero learning rate and / or distinct optimizers.
Need to check novelty before this filing date? Find Prior Art