Method and device for object detection and classification in a vehicle
By modularizing neural networks into specialized modules with weighted voting, the system addresses catastrophic forgetting, enabling continuous learning and improved object recognition in autonomous vehicles with reduced validation efforts and enhanced scalability.
Patent Information
- Application Number
- DE102025101228
- Authority / Receiving Office
- DE · DE
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2025-01-15
- Publication Date
- 2025-04-03
AI Technical Summary
Conventional neural networks in autonomous vehicles face the challenge of 'catastrophic forgetting' during continuous learning, leading to loss of previously learned information when retrained on new object classes, necessitating extensive validation and verification efforts.
The neural network is divided into independent, specialized modules, each optimized for specific object classes, with an additional module trained for new classes, and a weighted voting process integrates their outputs to maintain existing knowledge and adapt to new conditions.
This approach prevents catastrophic forgetting, reduces validation efforts, enhances system flexibility and scalability, and ensures continuous improvement without affecting stability or security, allowing modules to be outsourced for additional computing resources.
Smart Images
Figure 00000000_0000_ABST
Abstract
Description
[0001] The present invention relates to a method for object detection and classification in a vehicle. The present invention also relates to a corresponding device, a corresponding computer program, and a corresponding storage medium. State of the art
[0002] In autonomous driving systems and similar applications, neural networks are used to reliably detect and classify objects in the vehicle environment. A key challenge here is "catastrophic forgetting." This refers to the effect that a neural network, once trained on specific objects, loses previously learned information when retraining with new data. This effect complicates ongoing, continuous learning, which is crucial for autonomous vehicles to be able to respond appropriately to constantly changing environmental conditions and new object types. Previous approaches, such as fine-grained retraining of the same neural network with additional regularization mechanisms, often lead to complex validation and verification measures for the entire system and do not offer a fully satisfactory solution. Disclosure of the invention
[0003] One problem is that conventional neural networks only inadequately retain previously learned information when retraining on new object classes, which can lead to the loss of safety-critical prior knowledge. Furthermore, revalidating and verifying the entire network each time the model is expanded or adapted requires considerable effort. The invention solves this problem by dividing the neural network into independent, specialized modules, each optimized for specific object classes. Instead of retraining an existing entire network with new data, an additional module (e.g., a separate neural subnetwork) is created and trained for each new object class to be learned. This means that what has been previously learned remains unchanged in the existing modules, preventing catastrophic forgetting.
[0004] Each module can be validated and verified separately without the need to re-validate the entire system. Furthermore, a weighted voting process is introduced, which combines the output of the individual modules specialized for specific object classes and arrives at a final classification. Modules with particularly high expertise in the current object detection are given greater weight. These weightings can be dynamically adjusted, for example, through reinforcement learning, to optimally account for changing environmental conditions and object classes.
[0005] The inventive approach thus enables neural networks to be continuously expanded with new object classes without overwriting existing knowledge. Modularization and the subsequent voting system prevent catastrophic forgetting, simplify validation, and increase the overall security of the system. Furthermore, individual modules can also be outsourced, for example, to the cloud, providing additional computing capacity and increasing real-time capability.
[0006] The proposed solution offers the following advantages: 1. Avoiding catastrophic forgetting by isolated extension of the neural network with new modules without retraining the entire network. 2. Reduced validation and verification effort, since only newly created modules need to be secured separately. 3. High flexibility and scalability, as any number of modules, such as for pedestrians, vehicles, traffic signs, weather or other object classes, can be added. 4. Dynamic weight adjustment of the modules through reinforcement learning or other methods for continuous performance improvement in real-time applications. 5. Easy integration into the electronic architecture of modern vehicles, since each module can be implemented as a software or hardware component (e.g., individual ECU).
[0007] In advantageous implementations, the weighting of the modules, i.e., the selection of the module, can not only be done statically based on historical performance (accuracy on validation data sets), but can also be self-optimized during operation. Thus, when an object is detected, the module trained on this object class is selected so that this specialized module can recognize as many properties / features as possible, since it is specifically trained on this object class. A reinforcement learning algorithm can be used here, which learns which weightings are optimal from the behavior of the modules in real-time applications.
[0008] Furthermore, it is possible to operate individual modules independently of the vehicle in the cloud, where they can perform complex calculations or initialize new modules. This approach relieves the load on the on-board computing units and enables the use of external computing and storage resources. Furthermore, the security of each module can be guaranteed long-term after initial validation, as it remains unaffected by other modules.
[0009] Through these measures, object detection in autonomous vehicles can be continuously improved without compromising system stability or safety. Short description of the drawings Fig. 1 shows several specialized modules and the initialization of a new module (17), which can be validated separately downstream due to the new recognition of the class “Trees”. Embodiments of the invention
[0010] The invention can be implemented, for example, as follows: In the control unit of a vehicle (e.g., in one or more ECUs), an overall system (10) is implemented, which consists of a plurality of specialized neural network modules (12, 13, 14, 15, 16, 17). Each module is designed for specific object types, such as lanes (12), open space (13), traffic signs (14), weather conditions (15), cars / trucks (16), or trees (17). A camera image of the vehicle's surroundings serves as input (18) for object detection, which is explained here using the example of the dash-dotted bounding box (car).
[0011] If the system detects a new, previously unknown object class, a new module—here, for trees (17)—is initialized and trained separately. This training preferably takes place without influencing the existing modules (11) to avoid changing their learned knowledge and the problem of catastrophic forgetting. The new module is validated and verified on suitable test data sets selected specifically for the new object class. Once the module (here: 17) is sufficiently validated separately, it is integrated into the operational workflow without the need for further testing of the overall system (10).
[0012] If multiple modules are available for object detection, a weighted, probabilistic voting procedure is used to determine the final classification (19). Modules with greater expertise for the currently detected object class receive higher weights. This weighting can be implemented using simple evaluation functions (e.g., accuracy values) or dynamically adapted to current environmental conditions using reinforcement learning. This ensures that the overall system (10) is robust to changing situations and is continuously optimized.
[0013] Through integrated decision-making (20), not only is a final classification derived, for example, "car" (21). This final decision regarding the classification of an object is made by a weighted sum of the probabilities or classification decisions of all modules. This can be formally expressed by the following equation: Pfinal(y|x)=ΣiNwi∗Pi(y|x)
[0014] In it, P final (y|x) the final probability for the class y given the input object x, P i (y|x) is the probability of the i-th and w i the weight of the i-th module, which reflects its expertise for object x.
[0015] In another advantageous embodiment, individual modules can be outsourced to external computing units, such as the cloud. This allows for the provision of additional computing capacity for complex calculations and also reduces the load on the vehicle system. Already validated modules retain their validity and safety rating because they are not retrained, thus eliminating the need for a new overall system validation. Only when a new module is created does a module-specific validation need to be performed. List of reference symbols 10 Entire neural network with various modules 11 Already validated 12 Module: Lanes 13 Module: Open Space 14 Module: Traffic Signs 15 Module: Weather 16 Module: Cars / Trucks 17 Module: Trees 18 Input of the object detection Probabilistic voting procedure for the evaluation of the 19 Detection quality based on conditional probability Integrated decision making for the dotted line 20 Bounding Box 21 Decision: Car (Safety: 95%)
Claims
[1] Method for object detection and classification in a vehicle, characterized by following features: - a neural network (10) is provided, which consists of several independent, already validated modules (11), each module (12, 13, 14, 15, 16, 17) being specialized in the recognition of a specific object class, - an input signal of the object detection (18) is fed to the modules (12, 13, 14, 15, 16, 17), - when a new, previously unlearned object class is detected, a new module (12, 13, 14, 15, 16, 17) is created and trained without retraining or modifying existing modules (11), - the output of the modules (12, 13, 14, 15, 16, 17) is weighted by means of a voting procedure (19), whereby the weighting is based on the respective expertise of the module for the recognized object and its previously validated performance, - by means of an integrated decision-making process (20), a final classification with an associated confidence probability is determined from the weighted module outputs, which is output as a bounding box with a security statement (21). [2] Method according to claim 1, characterized by following features: - the weightings of the individual modules (12, 13, 14, 15, 16, 17) are dynamically adapted to their performance on validation data sets, - a reinforcement learning approach is used to continuously optimize the weights (19) during operation, - each newly created module (12, 13, 14, 15, 16, 17) is validated and verified separately before use, without revalidation of the entire system (10). [3] Device, characterized by following features: - the device is designed to carry out a method according to one of claims 1 or 2, - the device comprises a control unit in which the modules (12, 13, 14, 15, 16, 17) as well as the voting procedure (19) and the integrated decision-making (20) are implemented, - the device has an interface for inputting object detection data (18) and output means for providing the final classification information with confidence probability (21). [4] Computer program which is arranged to carry out all the steps of a method according to one of claims 1 or 2. [5] A machine-readable storage medium having a computer program according to claim 4 stored thereon.