Method and device for active learning from multimodal input
By using an acquisition function to select inputs for labeling based on costs and model uncertainty, the method addresses the inefficiencies in existing active learning methods for multimodal inputs, optimizing network performance and reducing annotation costs.
Patent Information
- Application Number
- JP2024141109
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-09-28
- Filing Date
- 2024-08-22
- Publication Date
- 2025-04-09
AI Technical Summary
Existing active learning methods for multimodal inputs face challenges in efficiently selecting images for labeling, especially when dealing with diverse sensor data modes and varying annotation costs, which can lead to suboptimal network performance.
The method employs an acquisition function to select inputs for labeling from multimodal data, considering costs and model uncertainty, and allows for the fusion of different sensor data modes to maximize network performance while minimizing labeling costs.
This approach effectively selects the most informative inputs for labeling across different sensor modes, optimizing network performance and reducing annotation costs, thereby improving the efficiency of active learning from multimodal inputs.
Smart Images
Figure 2025058921000001_ABST
Abstract
Description
[Background technology]
[0001] The present invention relates to an apparatus and method for active learning from multi-modal input.
[0002] The active learning approach assumes a fixed budget for the number of acquired images and simulates this by iteratively learning to add a certain number of acquired images during each iteration. The images to be added are selected based on some criteria. Summary of the Invention [Means for solving the problem]
[0003] According to the independent claims, the method and apparatus for active learning from inputs use an acquisition function to select inputs to be labeled from the inputs. Multimodal input in this context refers to sensor data of different modes, such as digital images, LiDAR images, radar images, ultrasonic images, infrared images, sound, etc. In a sensor data fusion setup, the acquisition function exploits the modes from the multimodal input and their associated costs, minimizing the total cost spent and maximizing the network performance. Since different tasks from different modes may be complementary in a fusion setup, it may not be necessary to label inputs of all modes of the multimodal input. In fact, it may be undesirable considering different annotation costs.
[0004] A method for active learning from a multimodal input includes providing an input, the input comprising different modal inputs, and learning a model responsive to the input. Learning the model includes determining a modal input from the different modal inputs responsive to an acquisition function comprising a measure for a cost of labeling the modal input, labeling the modal input with a label, and learning the model responsive to the label.
[0005] The acquisition function may comprise a measure of the uncertainty of the model with respect to the input of the mode.
[0006] The method may include actively learning the model for different tasks, the acquisition function comprising a measure for synergy in learning different tasks with modal inputs.
[0007] The method may include determining the different modes of input from the different modes of input as a function of an acquisition function.
[0008] Providing the input may include providing an input comprising an input of a modal digital image, a LiDAR image, a radar image, an ultrasound image, an infrared image, or an acoustic signal.
[0009] Providing the input may include capturing the input with a sensor, in particular a digital imaging sensor, a LiDAR imaging sensor, a radar imaging sensor, an ultrasonic imaging sensor, an infrared imaging sensor, an acoustic signal sensor.
[0010] The method may include the steps of training a model to map inputs to outputs with the model, and operating a technical system in response to the outputs.
[0011] An apparatus for active learning from a multimodal input, the apparatus comprising at least one processor and at least one memory, the at least one memory configured to store instructions that, when executed by the at least one processor, cause the apparatus to perform the method of any one of claims 1 to 6. The processor is configured to execute the instructions.
[0012] The device may include sensors for capturing input, or an interface for sensors.
[0013] The device may comprise an actuator for actuating the technical system, or an interface for an actuator.
[0014] A technical system comprising the device has advantages corresponding to the advantages of the device.
[0015] The computer program comprises computer readable instructions which, when executed by a computer, cause the computer to perform steps of a method, the steps of the method having advantages corresponding to the advantages of the method.
[0016] Further advantageous embodiments can be derived from the following description and drawings. [Brief description of the drawings]
[0017] [Figure 1] FIG. 1 illustrates a schematic of an apparatus for active learning from multi-modal input. [Diagram 2] FIG. 1 shows a schematic diagram of a first embodiment of active learning including a step of fusing sensor data. [Diagram 3] FIG. 2 shows a schematic diagram of a second embodiment of active learning including fusing feature vectors representing sensor data. [Figure 4] FIG. 13 shows a schematic diagram of a third embodiment of active learning including fusing decisions resulting from sensor data. [Diagram 5] 1 is a flow chart including steps of a method for active learning from multi-modal input. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0018] FIG. 1 illustrates generally an apparatus 100 for active learning from multi-modal input.
[0019] The apparatus 100 comprises at least one processor 102 and at least one memory 104 .
[0020] The at least one memory 104 is configured to store instructions that, when executed by the at least one processor 102, cause the apparatus 100 to perform a method for active learning from multi-modal input.
[0021] At least one processor 102 is configured to execute instructions.
[0022] The instructions may be computer readable instructions. The computer program may comprise computer readable instructions. The computer readable instructions, when executed by a computer, are configured to cause the computer to perform the steps of a method.
[0023] The device 100 may also include an interface 106 for sensors 108-1, ..., 108-n to capture input. The device 100 may include sensors 108-1, ..., 108-n, for example, digital image sensors, LiDAR image sensors, radar image sensors, ultrasonic image sensors, infrared image sensors, or acoustic signal sensors.
[0024] The device 100 may comprise an interface 110 for an actuator 112 for actuating a technical system 114. The device 100 may comprise an actuator 112.
[0025] The technical system 114 may comprise the device 100. The technical system 114 may comprise sensors 108-1, . . . , 108-n and / or actuators 112.
[0026] 2 shows a schematic of a first embodiment of active learning including fusing sensor data. The sensor data in this example is input from sensors 108-1, ..., 108-n. The input may include input of modal digital images, LiDAR images, radar images, ultrasonic images, infrared images, or acoustic signals.
[0027] The active learning includes a signal processing step 202, a fusing step 204, a feature extraction step 206, and a determining step 208.
[0028] According to a first embodiment, the inputs are processed by a signal processing step 202 to determine processed data of the individual sensors. The processed data is processed in a fusing step 204. A feature extraction step 206 extracts information from the result of the fusing step. An output 210 of a determining step 208 is determined from the result of the feature extraction step 206.
[0029] FIG. 3 illustrates generally a second embodiment of active learning, which includes fusing feature vectors representing sensor data.
[0030] In contrast to the active learning according to the first embodiment, a feature extracting step 206 extracts features from the result of the signal processing step 202. A fusing step 204 processes the result of the feature extracting step 206. An output 210 of a determining step 208 is determined from the result of the fusing step 204.
[0031] 4 shows a schematic diagram of a third embodiment of active learning, which includes a step of fusing decisions resulting from sensor data. In contrast to active learning according to the first embodiment, a feature extraction step 206 extracts features from the result of the signal processing step 202. A determining step 208 determines a decision from the result of the feature extraction step 206. A fusing step 204 processes the decisions to determine an output 210.
[0032] FIG. 5 shows a flow chart including steps of a method for active learning from multimodal input.
[0033] The method includes step 502.
[0034] In step 502, an input is provided.
[0035] The input comprises different modes of input.
[0036] The input may comprise, for example, an input of a modal digital image, a LiDAR image, a radar image, an ultrasound image, an infrared image, or an acoustic signal.
[0037] The inputs may be captured by sensors 108-1, . . . , 108-n, including digital imaging sensors, LiDAR imaging sensors, radar imaging sensors, ultrasonic imaging sensors, infrared imaging sensors, and acoustic signal sensors, among others.
[0038] The method includes step 504.
[0039] In step 504, a model F is trained in response to the input.
[0040] The model F may comprise a neural network, in particular a deep neural network, with weights. The model F may be adapted to map inputs to outputs of the model according to the weights.
[0041] Active learning of the model F may include determining the weights.
[0042] Active learning is described as an example for one task and different modes.
[0043] In an embodiment, model F is trained on different modes with the aim of solving a single task. Example tasks are classification, image segmentation (pixel-level classification), 2D object detection, 3D object detection, depth prediction, free space detection.
[0044] As an example, model F comprises a backbone and a task-specific head. The backbone receives inputs of different modes as inputs. The head outputs the model's output.
[0045] The method is also applied to models F that are configured to solve different tasks.
[0046] Model F may contain a backbone and different task-specific heads for solving different tasks, allowing deep neural networks to run despite hardware constraints on resource-limited chips, for example in automobiles.
[0047] The backbone may be trained jointly with or with the task-specific head.
[0048] The backbone may be a convolutional backbone. The head may be configured for 2D road sign detection, or 2D object detection, or semantic segmentation, or 3D vehicle detection.
[0049] Training a model F includes step 504-1.
[0050] Step 504-1 includes determining an input of one mode m or multiple inputs of different modes from the inputs of different modes. This means that active learning may include selecting an input of one mode m for labeling or selecting multiple inputs of different modes for labeling. According to this embodiment, at least one mode is not selected for labeling.
[0051] The input of mode m is determined according to an acquisition function acq.
[0052] According to one embodiment, multiple inputs of different modes are provided, each of which is represented by an unlabeled data point X u Active learning focuses on selecting, annotating, and adding new data points to a training set X from a pool of unlabeled data.
[0053] An exemplary acquisition function:
number
[0054] Costs may be expressed in Euros. When considering real-world annotations, costs may depend on multiple factors, such as the difficulty of providing accurate annotations, the training required for annotating, and the time required to provide the annotations. For example, selecting pixels in a digital image that correspond to a cat requires no training for a human, whereas selecting pixels in a brain scan that correspond to a tumor requires extensive training for a human.
[0055] These factors lead to different real costs in labeling real data points, which results in potentially very different costs of labeling for different tasks and modalities, which is reflected in active learning.
[0056] The acquisition function acq is a measure of the uncertainty of the model M with respect to the input of the mode, Unc(F,x u ) The uncertainty measure makes it possible to select data points about which the trained network is uncertain. The uncertainty measure Unc(F,x u ) is the entropy of the output of the model M. The output of the model F for classification can be a SoftMax vector. The uncertainty measure Unc(F,x u ) may be replaced by another measure, for example an acquisition function that ignores the costs mentioned above. u ) may be ignored, e.g., set to zero.
[0057] In the case of active learning for multiple tasks, the cost function operates across tasks.
[0058] In the case of active learning for multiple tasks, the acquisition function may comprise a measure for the synergy of learning different tasks at the input of the mode.
[0059] This synergy can be positive or negative: mismatched loss directions may result in slower convergence and poorer performance for the individual tasks; or the presence of similar semantic content may result in better performance: individual tasks benefit from a shared and improved feature representation, resulting in increased performance for the individual tasks.
[0060] For example, synergy t (x u ,T) is added to the acquisition function:
number
[0061] The synergy function for mode p is:
number
[0062] Synergy Function Syn p is suitable for the different modes, assuming that there exists a set of data points in which the training set X comprises all the inputs of the different modes.
[0063] Conceptually, the synergy function Syn pcompares the activation of all shared weights for unlabeled inputs on the trained model in question, looking for situations where the same weight shows high activation for both input modes. Preferably, the input modes are the same semantic scene. The constraint requiring the input modes to be the same semantic scene may be relaxed. For example, synergies could be determined for a given mode pairwise with other inputs in the training data. For example, synergies of unlabeled and labeled images in training set X are determined for a given particular mode.
[0064] This is likely to be more expressive and useful, at the cost of increased computational or storage resources.
[0065] According to this embodiment, the input that minimizes the acquisition function is determined, which means selecting for labeling an input that is expected to maximize model performance with the least labeling cost. The method may include selecting an input that is expected to have a lower cost or better model performance than another input.
[0066] Training the model includes step 504-2.
[0067] Step 504-2 includes labeling the input of the mode with the label.
[0068] For modes that comprise images, the label may comprise a pixel-level annotation of the input of mode m.
[0069] Training the model includes step 504-3.
[0070] Step 504-3 involves training a model as a function of the labels.
[0071] According to this embodiment, model F is configured to map inputs to outputs in model F. The output may be the output 210 of the determining step 208 or may be the output 210 of the fusing step 204, which processes the decisions to determine the output 210.
[0072] A model F is trained on inputs and maps the inputs to outputs.
[0073] The model F may be trained in response to a batch of inputs comprising the inputs, which may be determined in iterations of steps 504-1 and 504-2.
[0074] For example, a fixed number of inputs to be labeled are iteratively selected based on the output of an acquisition function.
[0075] The selected inputs are then labeled and added to a training set X, either actually for a real system or simulated in an experimental setting.
[0076] A model F is trained with a loss, e.g.,
number
[0077] Entering a mode, e.g. a mode with images;
number
number
[0078] Model F may be retrained until some termination criterion is met. For example, model F is trained until either a performance threshold, e.g., 99% accuracy, or a budget limit, e.g., total number of inputs or amount of money spent, is met.
[0079] A model F may be trained on a training set X. Then, a number of unlabeled data points x u Inference on x may be performed on the trained model F. An acquisition function acq may be used to output the inference, and the unlabeled data points x u Determine the score for each unlabeled data point x with the highest score. u may be selected for labeling.
[0080] The unlabeled data points may be sent for annotated labeling and added to the training datasets X,Y. The model F may be retrained using the new, updated training datasets X,Y. The process is repeated to acquire new data points until a termination criterion is reached.
[0081] Referring to the examples depicted in Figures 2, 3 and 4, the model F may include a signal processing step 202, a fusing step 204, a feature extraction step 206 and a decision step 208. The synergy function may operate only on the shared weights. The shared weights in the first embodiment may be the fusing step 204 and the feature extraction step 206. The shared weights in the second embodiment may be the feature extraction step 206 and the fusing step 204. The shared weights in the third embodiment may be the decision fusing step 204.
[0082] The method may include step 506 .
[0083] In step 506, a technology system 114 may be actuated in response to the output.
[0084] Step 502 may be performed at a later time.
Claims
1. 1. A method for active learning from a multimodal input, comprising the steps of: providing (502) an input; and training (504) a model responsive to the input, the input comprising different modal inputs, the step of training (504) comprising the steps of: determining (504-1) a modal input from the different modal inputs responsive to an acquisition function comprising a measure for a cost of labeling the modal inputs; labeling (504-2) the modal inputs with a label; and training (504-3) a model responsive to the label.
2. 2. The method of claim 1, wherein the acquisition function comprises a measure of model uncertainty regarding the input of the mode.
3. 3. The method of claim 1 or 2, further comprising the step of actively learning (504) models for different tasks, wherein the acquisition function comprises a measure for synergy of learning the different tasks with the input of the mode.
4. The method according to any one of claims 1 to 3, characterized in that the method comprises a step (504-1) of determining, from said different modes of input, a different mode of input depending on said acquisition function.
5. 5. The method of claim 1, wherein the step of providing (502) an input comprises providing an input comprising an input of a modal digital image, a LiDAR image, a radar image, an ultrasound image, an infrared image, or an acoustic signal.
6. The method according to any one of claims 1 to 5, characterized in that the step of providing (502) an input comprises the step of capturing the input with a sensor (108-1, ..., 108-n), in particular a digital image sensor, a LiDAR image sensor, a radar image sensor, an ultrasonic image sensor, an infrared image sensor, an acoustic signal sensor.
7. The method according to any one of claims 1 to 6, characterized in that the method comprises a step (504-3) of learning a model in order to map inputs to outputs with the model, the method comprising a step (506) of operating a technical system depending on the outputs.
8. An apparatus (100) for active learning from a multimodal input, the apparatus (100) comprising at least one processor (102) and at least one memory (104), the at least one memory (104) configured to store instructions that, when executed by the at least one processor (102), cause the apparatus (100) to perform the method of any one of claims 1 to 7, the at least one processor (102) configured to execute the instructions.
9. The apparatus (100) of claim 8, wherein the apparatus (100) comprises sensors (108-1, ..., 108-n) for capturing inputs or an interface (106) for the sensors (108-1, ..., 108-n).
10. 10. The apparatus (100) according to claim 8 or 9, comprising an actuator (112) for actuating a technical system (114) or an interface (110) for said actuator (112).
11. A technical system (114), characterized in that it comprises an apparatus (100).
12. A computer program comprising computer readable instructions which, when executed by a computer, cause the computer to carry out the steps of the method according to any one of claims 1 to 7.