Method and apparatus for active learning from multi-modal input
By using the acquisition function in the active learning method, efficiently selecting the input to be marked from the multimodal input, the problems of waste of resources and poor model performance in the prior art are solved, and more efficient resource utilization and model performance optimization are achieved.
Patent Information
- Application Number
- CN202411290956.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2023-09-28
- Filing Date
- 2024-09-14
- Publication Date
- 2025-03-28
AI Technical Summary
When existing active learning methods deal with multimodal inputs, it is difficult to effectively select inputs that need to be marked, resulting in waste of resources and poor model performance.
The acquisition function is used to select the input to be marked from the multimodal input, and the total cost is minimized and network performance is maximized by measuring the marking cost and uncertainty of inputs in different modes.
It realizes efficient selection of inputs that need to be marked in multimodal input scenarios, optimizes resource utilization and model performance, and reduces marking costs.
Smart Images
Figure CN119721172A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to an apparatus and method for active learning from multimodal input. Background Art
[0002] Active learning methods assume a fixed budget in terms of the number of acquired images and simulate this by learning in iterations and adding a certain number of acquired images during each iteration. The added images are selected based on some criteria. Summary of the invention
[0003] A method and apparatus for active learning from inputs uses an acquisition function to select inputs to be labeled from the inputs. Multimodal inputs in this context refer to sensor data of different modes (e.g., digital images, LiDAR images, radar images, ultrasound images, infrared images, sound) to multimodal sensor data. In a sensor data fusion setting, the acquisition function utilizes modes from the multimodal inputs and their associated costs to minimize the total cost spent and maximize network performance. Since different tasks from different modes can be complementary in a fusion setting, it may not be necessary to label inputs of all modes of the multimodal input, and in fact it may not be desirable to consider different annotation costs.
[0004] The method for active learning based on multimodal input includes: providing input and learning a model based on the input, wherein the input includes inputs of different modes, and wherein learning the model includes: determining an input of a mode from the inputs of the different modes based on an acquisition function, the acquisition function including a measure of the cost of labeling the input of the mode; labeling the input of the mode with labels; and learning the model based on the labels.
[0005] The acquisition function may include a measure of the uncertainty of the model with respect to the inputs to the pattern.
[0006] The method may include actively learning models for different tasks, wherein acquiring a function includes learning a measure of synergy of the different tasks using inputs of the model.
[0007] The method may include determining a plurality of inputs of different modes from the inputs of different modes according to an acquisition function.
[0008] Providing the input may include providing the input to include an input of a modal digital image, a LiDAR image, a radar image, an ultrasound image, an infrared image, or an acoustic signal.
[0009] Providing the input may include capturing the input using a sensor, in particular a digital image sensor, a LiDAR image sensor, a radar image sensor, an ultrasound image sensor, an infrared image sensor, or an acoustic signal sensor.
[0010] The method may include learning a model to map inputs to outputs using the model, and actuating a technical system based on the outputs.
[0011] A device for active learning from multimodal input, characterized in that the device includes at least one processor and at least one memory, the at least one memory is configured to store instructions, and the instructions, when executed by the at least one processor, cause the device to perform the method according to the above, wherein the processor is configured to execute the instructions.
[0012] The device may include a sensor or an interface to a sensor for capturing input.
[0013] The device can comprise an actuator or an interface of an actuator for actuating the technical system.
[0014] A technical system comprising a device having advantages corresponding to those of the device.
[0015] A computer program comprising computer readable instructions which, when executed by a computer, cause the computer to perform the steps of a method having advantages corresponding to those of the method. BRIEF DESCRIPTION OF THE DRAWINGS
[0016] Further advantageous embodiments can be derived from the following description and the accompanying drawings. In the drawings:
[0017] Figure 1 A schematic depiction of a device for active learning from multimodal input;
[0018] Figure 2 A first implementation of active learning including fusion of sensor data is schematically depicted;
[0019] Figure 3 schematically depicts a second embodiment of active learning including fusion of feature vectors representing sensor data;
[0020] Figure 4 schematically depicts a third embodiment of active learning including fusion of decisions resulting from sensor data;
[0021] Figure 5 A flow chart comprising the steps of a method for active learning from multimodal input is depicted. DETAILED DESCRIPTION
[0022] Figure 1 An apparatus 100 for active learning from multimodal input is schematically depicted.
[0023] The device 100 includes at least one processor 102 and at least one memory 104 .
[0024] The at least one memory 104 is configured to store instructions that, when executed by the at least one processor 102 , cause the device 100 to perform a method for active learning from multimodal input.
[0025] The at least one processor 102 is configured to execute instructions.
[0026] The instructions may be computer readable instructions. The computer program may include computer readable instructions. The computer readable instructions are configured to, when executed by a computer, cause the computer to perform the steps of the method when executed.
[0027] The device 100 may include an interface 106 for sensors 108-1, ..., 108-n for capturing input. The device 100 may include sensors 108-1, ..., 108-n, such as digital image sensors, LiDAR image sensors, radar image sensors, ultrasonic image sensors, infrared image sensors, or acoustic signal sensors.
[0028] The device 100 may include an interface 110 for actuating an actuator 112 of a technical system 114 . The device 100 may include an actuator 112 .
[0029] The technical system 114 may include the device 100 . The technical system 114 may include sensors 108 - 1 , . . . , 108 - n and / or actuators 112 .
[0030] Figure 2 A first embodiment of active learning including fusion of sensor data is schematically depicted. The sensor data in this example is input from sensors 108-1, ..., 108-n. The input may include input of modal digital images, LiDAR images, radar images, ultrasound images, infrared images, or acoustic signals.
[0031] Active learning includes signal processing 202 , fusion 204 , feature extraction 206 , and decision making 208 .
[0032] According to a first embodiment, the input is processed by signal processing 202 to determine processed data for each sensor. The processed data is processed in fusion 204. Feature extraction 206 extracts information from the result of the fusion, and the output 210 of decision making 208 is determined from the result of feature extraction 206.
[0033] Figure 3 A second embodiment of active learning including fusion of feature vectors representing sensor data is schematically depicted.
[0034] Compared to the active learning according to the first embodiment, feature extraction 206 extracts features from the result of signal processing 202, and fusion 204 processes the result of feature extraction 206. Output 210 of decision 208 is determined from the result of fusion 204.
[0035] Figure 4 A third embodiment of active learning including fusion of decisions resulting from sensor data is schematically depicted.
[0036] Compared to the active learning according to the first embodiment, feature extraction 206 extracts features from the result of signal processing 202, and decision 208 determines a decision from the result of feature extraction 206. Fusion 204 processes the decision to determine output 210.
[0037] Figure 5 A flow chart comprising the steps of a method for active learning from multimodal input is depicted.
[0038] The method includes step 502 .
[0039] In step 502, input is provided.
[0040] The input includes inputs of different modes.
[0041] The input includes input such as a modal digital image, a LiDAR image, a radar image, an ultrasound image, an infrared image, or an acoustic signal.
[0042] The input may be captured using sensors 108 - 1 , . . . , 108 - n , in particular, a digital image sensor, a LiDAR image sensor, a radar image sensor, an ultrasound image sensor, an infrared image sensor, or an acoustic signal sensor.
[0043] The method includes step 504 .
[0044] In step 504, a model F is learned based on the input.
[0045] The model F may comprise a neural network, in particular a deep neural network, comprising weights. The model F may be adapted to map inputs to outputs of the model according to the weights.
[0046] Active learning of the model F may include determining weights.
[0047] Active learning is described using examples for a task and different modalities.
[0048] In this example, a model F is trained on different modalities for the purpose of solving a single task. Examples of tasks are classification, image segmentation (pixel-level classification), 2D object detection, 3D object detection, depth prediction, free space detection.
[0049] As an example, model F includes a backbone and a task-specific head. The backbone receives inputs of different modalities as input. The head outputs the output of the model.
[0050] The method is also applicable to models F that are configured to solve different tasks.
[0051] The model F for solving different tasks may include a backbone and different task-specific heads. This allows running deep neural networks despite hardware constraints on resource-constrained chips (e.g., in cars).
[0052] The backbone can be trained jointly with one task-specific head or multiple task-specific heads.
[0053] The backbone can be a convolutional backbone. The head can be configured for 2D road sign detection or 2D object detection or semantic segmentation or 3D vehicle detection.
[0054] Learning model F includes step 504 - 1 .
[0055] Step 504-1 includes determining an input of mode m or multiple inputs of different modes from inputs of different modes. This means that active learning may include selecting an input of mode m for labeling or selecting multiple inputs of different modes for labeling. According to this example, at least one mode is not selected for labeling.
[0056] The input of mode m is determined according to the acquisition function acq.
[0057] According to the example, the input provided in different modes is an unlabeled data point X u Active learning focuses on selecting a new data point from the pool of unlabeled data for annotation and adding the new data point to the training set X.
[0058] Exemplary acquisition function acq t =-C t (x u ) includes the input x for the pattern u (i.e., the unlabeled data points of the unlabeled data) t The measurement of.
[0059] The cost may be in Euros. When considering real-world annotations, the cost may be affected by multiple factors, such as the difficulty level of providing accurate annotations or the training required for the annotations or the time required to provide the annotations. For example, selecting pixels corresponding to a cat in a digital image does not require training for a person, while selecting pixels corresponding to a tumor in a brain scan does require extensive training for a person.
[0060] These factors can lead to different true costs when labeling real data points, leading to potentially very different costs for labeling for different tasks and modalities. This is reflected in active learning.
[0061] The acquisition function acq can include a measure of the uncertainty of the model M with respect to the input of the pattern Unc(F,x u The uncertainty measure Unc(F,x u ) can be the entropy on the output of model M. The output of model F for classification can be a SoftMax vector. The uncertainty measure Unc(F,x u ) can be replaced by another measure (e.g., an acquisition function that ignores the above costs). The uncertainty measure Unc(F,x u ) can be ignored, for example set to zero.
[0062] In the case of active learning for several tasks, the cost function works across tasks.
[0063] In the case of active learning for several tasks, the acquisition function may include a measure for the synergy of learning different tasks using the input of the pattern.
[0064] This synergy can be positive or negative, i.e., conflicting loss directions can lead to slower convergence and reduced performance for individual tasks, or the presence of similar semantic content can lead to better performance, i.e., individual tasks benefit from shared, improved feature representations, leading to increased performance for the individual tasks.
[0065] For example, Syn t (x u ,T) is added to the get function:
[0066] Unc(F,x u )-C t (x u )+Syn t (x u ,T)
[0067] An exemplary synergy function for mode p is:
[0068]
[0069] Among them, x p is the unlabeled input of pattern p, x m is the unlabeled input of pattern m, M is the set of patterns other than pattern p, where act is the activation function of the model (e.g., neural network), and W is the shared weight w of the model (e.g., neural network) i The activation function can be relu.
[0070] When it is assumed that there is a set of data points including all different modes of input for the training set X, the synergy function Syn p Suitable for these different modes.
[0071] Conceptually, the synergy function Syn p The activations of all shared weights for unlabeled inputs on the training model in question are compared, and cases where the same weight shows high activation for two input patterns are sought. Preferably, the input patterns belong to the same semantic scene. The constraint requiring input patterns to be of the same semantic scene can also be relaxed, e.g., synergies can be determined pairwise with other inputs of the training data for a given pattern. For example, synergies are determined for unlabeled images and labeled images from the training set X for a given pattern.
[0072] This may be more expressive and useful, at the expense of increased computational or storage resources.
[0073] According to this example, an input is determined that minimizes the acquisition function. This means that an input is selected for labeling with the lowest labeling cost and is expected to maximize model performance. The method may include selecting an input that has a lower cost or better expected model performance than another input.
[0074] Learning the model includes step 504 - 2 .
[0075] Step 504 - 2 includes labeling the input of the pattern with a label.
[0076] For modes that include images, the labels may include pixel-level annotations of the input to mode m.
[0077] Learning the model includes step 504 - 3 .
[0078] Step 504 - 3 includes learning a model based on the labels.
[0079] According to this example, model F is configured to map the input to the output using model F. The output may be the output 210 of decision 208 or the output 210 of fusion 204 , which processes the decision to determine the output 210 .
[0080] The model F is trained using the input to map the input to the output.
[0081] The model F may be trained based on a batch of inputs including the inputs. The batch of inputs may be determined in an iteration of step 504-1 and step 504-2.
[0082] For example, a specific number of inputs to be labeled are iteratively selected based on the output of the acquisition function.
[0083] The selected inputs are then labeled and added to the training set X (either the actual training set X for a real system or a simulated training set X in an experimental setting).
[0084] For example, to train model F using the loss:
[0085]
[0086] Where T is the number of tasks, w t is a task-specific weight, and L t is a task-specific loss function. Task-specific loss function L t Can be cross entropy or mean squared error.
[0087] The input of a model (e.g., a model including an image) is X = [x1, ..., x t ] For example, associated with a training dataset X,Y with a set of labels:
[0088] Y=[y1,…,y t ]
[0089] Model F may be retrained until a certain termination criterion is met. For example, model F is trained until a performance threshold (e.g., 99% accuracy) or a budget constraint (e.g., total number of inputs or money spent) is met.
[0090] The model F can be trained on the training set X, and then the trained model F can be used to predict multiple unlabeled data points x. u Perform inference. Obtain the output of the function acq that can be used for inference to determine for each unlabeled data point x u The score of , and the unlabeled data point x with the highest score u Can be selected for labeling.
[0091] The unlabeled data points may be sent for labeling to be annotated and added to the training dataset X, Y. The model F may be retrained using the new, updated training dataset X, T, and the process is repeated to acquire new data points until a termination criterion is reached.
[0092] refer to Figure 2 , Figure 3 and Figure 4 In the example described in , model F may include signal processing 202, fusion 204, feature extraction 206, and decision 208. The synergy function may only act on the shared weights. The shared weights in the first embodiment may be fusion 204 and feature extraction 206. The shared weights in the second embodiment may be feature extraction 206 and fusion 204. The shared weights in the third embodiment may be fusion 204 of decision.
[0093] The method may include step 506 .
[0094] In step 506 , the technology system 114 may be actuated based on the output.
[0095] Thereafter, step 502 may be performed.
Claims
1. A method for active learning from multimodal input, characterized in that The method comprises providing (502) the input and learning (504) a model based on the input, wherein the input comprises inputs of different modes, and wherein learning (504) the model comprises: determining (504-1) a mode of input from the different modes of input according to an acquisition function, the acquisition function comprising a measure of a cost of labeling the mode of input; labeling (504-2) the mode of input using labels; and learning (504-3) the model based on the labels.
2. The method according to claim 1, characterized in that: The acquisition function includes a measure of uncertainty of the model with respect to an input to the pattern.
3. The method according to claim 1 or 2, characterized in that: The method includes actively learning (504) the model for different tasks, wherein the acquisition function includes learning a measure of synergy of the different tasks using inputs of the model.
4. The method according to any one of the preceding claims, characterized in that The method includes: determining (504-1) a plurality of inputs of different modes from the inputs of different modes according to the acquisition function.
5. The method according to any one of the preceding claims, characterized in that The providing (502) the input includes: providing the input to include an input of a modal digital image, a LiDAR image, a radar image, an ultrasound image, an infrared image, or an acoustic signal.
6. The method according to any one of the preceding claims, characterized in that The providing (502) of the input includes: capturing the input using a sensor (108-1, ..., 108-n), in particular a digital image sensor, a LiDAR image sensor, a radar image sensor, an ultrasound image sensor, an infrared image sensor, or an acoustic signal sensor.
7. The method according to any one of the preceding claims, characterized in that The method comprises: learning (504-3) the model to map the input to an output using the model, and wherein the method comprises: actuating (506) a technical system according to the output.
8. A device (100) for active learning from multimodal input, characterized in that The device (100) comprises at least one processor (102) and at least one memory (104), wherein the at least one memory (104) is configured to store instructions, which, when executed by the at least one processor (102), cause the device (100) to perform a method according to any one of claims 1 to 7, wherein the at least one processor (102) is configured to execute the instructions.
9. The device (100) according to claim 8, wherein: The device (100) includes a sensor (108-1, ..., 108-n) or an interface (106) of sensors (108-1, ..., 108-n) for capturing the input.
10. The device (100) according to claim 8 or 9, wherein: The device (100) comprises an actuator (112) or an interface (110) of an actuator (112) for actuating a technical system (114).
11. A technical system (114), characterized in that The technical system comprises an apparatus (100) according to any one of claims 8 to 10.
12. A computer program, characterized in that The computer program comprises computer readable instructions which, when executed by a computer, cause the computer to perform the steps of the method according to any one of claims 1 to 7.