A Multi-Task Image Understanding Method and Apparatus Based on Parameter Propagation

Abstract

This invention discloses a multi-task image understanding method and apparatus based on parameter propagation. The method includes: acquiring several image data sets and preprocessing them to construct a training dataset; acquiring several image understanding tasks and constructing a task model for each task, wherein the image understanding tasks are related and each task model has independent parameters; training each task model using the training dataset based on a linear parameter propagation mechanism; and using each trained task model to perform image understanding on the image to be processed. This invention completely abandons the parameter sharing mechanism, assuming that the parameters of different image understanding tasks are completely independent, and achieving task association through parameter propagation between different image understanding tasks during training. This effectively avoids the negative transfer problem in traditional multi-image understanding tasks based on parameter sharing, and improves the prediction performance of each image understanding task.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and image understanding technology, and in particular to a multi-task image understanding method and apparatus based on parameter propagation. Background Technology

[0002] Image understanding technology can be applied to fields such as autonomous driving, security monitoring, and education. Image understanding refers to taking an RGB image as input and using a multi-layer convolutional neural network to directly output scene segmentation (which category each pixel belongs to) and depth estimation (the depth distance of objects from each pixel to the camera). Practical applications often require solving multiple image understanding tasks simultaneously. For example, in the field of autonomous driving, scene segmentation and depth estimation need to be performed simultaneously based on images captured by the current camera. The key to multi-task image understanding lies in leveraging the connections between different tasks to enhance the prediction performance of each task.

[0003] To achieve multi-task image understanding, traditional methods use parameter sharing among multi-layer convolutional neural networks for different tasks to establish connections between tasks (e.g., ...). Figure 1 (As shown). Specifically, in each training step, given multiple input-output pairs for various tasks, a loss function is established between the expected and actual outputs of the neural networks for each task. Then, gradient backpropagation is used to update the parameters of the neural networks, optimizing the output and improving performance. Because the parameters of the first M layers of the neural networks for different tasks are shared (the values ​​are the same), these shared parameters are updated simultaneously based on the backpropagated gradients of all tasks. However, if the backpropagated gradient directions for different tasks are opposite or conflicting—that is, improving the prediction performance of one task will decrease the prediction performance of another—the update of the shared parameters cannot simultaneously improve the prediction performance of multiple tasks, resulting in a negative transfer problem. Therefore, providing a multi-task image understanding method and apparatus based on parameter propagation is particularly necessary. Summary of the Invention

[0004] The purpose of this invention is to provide a multi-task image understanding method and apparatus based on parameter propagation to solve the problems of the prior art. It completely abandons the parameter sharing mechanism, assumes that the parameters between different image understanding tasks are completely independent, and achieves task association through parameter propagation between different image understanding tasks during the training process. This effectively avoids the negative transfer problem in traditional multi-image understanding tasks based on parameter sharing, while improving the prediction effect of each image understanding task.

[0005] To achieve the above objectives, the present invention provides the following solution: The present invention provides a multi-task image understanding method based on parameter propagation, comprising the following steps:

[0006] Acquire several image data, and construct a training dataset after preprocessing the image data;

[0007] Several image understanding tasks are obtained, and a task model is constructed for each image understanding task. The image understanding tasks are related to each other, and each task model has independent parameters.

[0008] Based on the linear parameter propagation mechanism, each task model is trained using the training dataset; the trained task models are then used to perform image understanding on the image data to be processed.

[0009] Preferably, a multi-layer convolutional neural network is used to construct a task model for each of the image understanding tasks.

[0010] Preferably, training the task model based on the linear parameter propagation mechanism includes two stages: self-learning and neighbor learning. In the self-learning stage, each task model independently trains its own parameters. In the neighbor learning stage, the parameters of all task models are fixed, and a parameter propagation matrix is ​​trained. The parameter propagation matrix is ​​used to represent the relationship between the various image understanding tasks.

[0011] Preferably, in the self-learning phase, the method of independently training the parameters of each task model includes:

[0012] The parameters of all the task models are randomly initialized, and the parameters of each task model are iteratively updated using the stochastic gradient method.

[0013] Preferably, the specific method for updating the parameters of each task model using the stochastic gradient method includes:

[0014] In each iterative update process, data is randomly sampled from the training dataset of the task model, the gradient of the parameters of the task model is calculated using the sampled data, and the parameters of the task model are updated based on the gradient of the parameters.

[0015] Preferably, in the neighbor learning phase, the method for training the parameter propagation matrix includes:

[0016] S3031. Initialize the parameter propagation matrix as an identity matrix;

[0017] S3032. Perform parameter propagation calculation based on the parameters of each task model after self-learning, and substitute the parameters obtained after parameter propagation calculation into each task model to calculate the gradient of the parameter propagation matrix;

[0018] S3033. Update the parameter propagation matrix based on the gradient of the parameter propagation matrix;

[0019] S3034. Based on the parameter propagation matrix updated by gradient, repeat steps S3032-S3033 to complete the training of the parameter propagation matrix.

[0020] The present invention also provides a multi-task image understanding device based on parameter propagation, including a data acquisition module, a model building module, and a model training module, wherein the data acquisition module and the model building module are respectively connected to the model training module;

[0021] The data acquisition module is used to acquire several image data and preprocess the image data to construct a training dataset;

[0022] The model building module is used to acquire several image understanding tasks, and to build a task model for each image understanding task. The several image understanding tasks are related to each other, and each task model has independent parameters.

[0023] The model training module is based on a linear parameter propagation mechanism and trains each task model using the training dataset; the trained task models are then used to perform image understanding on the image data to be processed.

[0024] Preferably, the model building module uses a multi-layer convolutional neural network to build a task model for each image understanding task.

[0025] Preferably, the model training module includes a self-learning module and a neighbor learning module connected in sequence;

[0026] The self-learning module is used to independently train the parameters of each task model.

[0027] The neighbor learning module is used to fix the parameters of all task models and train the parameter propagation matrix; the parameter propagation matrix is ​​used to represent the relationship between the various image understanding tasks.

[0028] Preferably, the self-learning module iteratively updates the parameters of each task model using the stochastic gradient method.

[0029] The present invention discloses the following technical effects:

[0030] This invention discloses a multi-task image understanding method and apparatus based on parameter propagation. Based on the correlation between image understanding tasks, a task model is constructed for each task, with each model having independent parameters. Training is performed on each task model using a combination of self-learning and neighbor learning based on a linear parameter propagation mechanism. This invention completely abandons the parameter-sharing mechanism, assuming that the parameters of different image understanding tasks are completely independent. During training, task correlation is achieved through parameter propagation between different image understanding tasks, effectively avoiding the negative transfer problem in traditional multi-task image understanding methods based on parameter sharing. This approach effectively utilizes the relationships between tasks while avoiding conflicts between them, significantly improving the prediction performance of each image understanding task. Attached Figure Description

[0031] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0032] Figure 1 Flowchart of a traditional multi-task image understanding method based on parameter sharing;

[0033] Figure 2 This is a flowchart of the multi-task image understanding method based on parameter propagation in Embodiment 1 of the present invention;

[0034] Figure 3 This is a flowchart of the method for training each task model based on the linear parameter propagation mechanism in Embodiment 1 of the present invention.

[0035] Figure 4 This is a schematic diagram of the multi-task image understanding device based on parameter propagation in Embodiment 2 of the present invention. Detailed Implementation

[0036] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0037] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0038] Example 1:

[0039] Reference Figure 2 As shown, this embodiment provides a multi-task image understanding method based on parameter propagation, including:

[0040] S1. Obtain several image data, and construct a training dataset after preprocessing the image data;

[0041] Suppose we need to solve M image understanding tasks, and the training dataset for task i is... Where, x ij y ij Let N be the j-th image in the training dataset for task i and the expected output label for that image, respectively. i Let f be the total number of images in the training dataset for task i. The goal of task i is to design model f given any input image. i The parameters are set such that the model's output is as close as possible to the desired output label.

[0042] S2. Obtain several image understanding tasks, and construct a task model for each image understanding task. The several image understanding tasks are related to each other, and each task model has independent parameters.

[0043] In this step, a multi-layer convolutional neural network is used to construct the task model. Currently, a popular approach in the field of artificial intelligence is to use multi-layer convolutional neural networks to implement model f. i In this application, it is assumed that the convolutional neural network f i Each layer consists of convolutional and pooling computations. For task i, in traditional methods, all tasks share certain layers, whose parameters are denoted as w, while the parameters of other non-shared layers are d. i Unlike traditional methods that share the parameter w, in this embodiment, each task model has an independent parameter w. i To characterize the relationships between different image understanding tasks, this embodiment introduces a linear parameter propagation mechanism, as shown in equation (1):

[0044] W′←WA (1)

[0045] Where W is the original parameter matrix, W = [w1, w2, ..., w MLet A be the parameter propagation matrix and W′ be the parameter matrix after propagation. Equation (1) means that the parameters of each task model after propagation are a weighted sum of the parameters before propagation, and the weighted parameters are given by the parameter propagation matrix A. A can effectively characterize the relationship between each image understanding task, and by adjusting the value of A, the negative transfer problem can be effectively overcome. There are many ways to adjust the value of A. For example, if there is a conflict between the training of task i and task i′, the value of the i-th row and i′ column of A can be set to 0. Therefore, how to determine A is crucial to improving the multi-task effect. In addition, for each task i, there is also the parameter d. i , parameter d i These are completely independent parameters for this task and do not participate in parameter propagation.

[0046] S3. Based on the linear parameter propagation mechanism, each task model is trained using the training dataset; the trained task models are then used to perform image understanding on the image data to be processed.

[0047] In this step, each task model is trained using the training dataset to correct the parameter propagation matrix A. Specifically, training the task models includes two stages: self-learning and neighbor learning. In the self-learning stage, each task model independently trains its own parameters. In the neighbor learning stage, the parameters of all task models are fixed, and the parameter propagation matrix A is trained. The specific process is as follows: Figure 3 As shown, it includes the following steps:

[0048] S301, For all the parameters w of the described task models i and d i Perform random initialization;

[0049] S302, Self-learning phase: The parameters of each task model are iteratively updated using the stochastic gradient descent method. Specifically, in each iteration, a small portion of data is randomly sampled from the training dataset for that task, denoted as... The parameters of the task model are updated using the sampled data, and the specific update process is shown in equations (2)-(5):

[0050]

[0051]

[0052]

[0053]

[0054] In the formula, The parameters w are respectively i and di The gradient of b, where λ1 and λ2 are the learning rates. i L represents the number of data points obtained from sampling. i This is the loss function.

[0055] S303, Neighbor Learning Phase, specifically includes:

[0056] S3031. Initialize the parameter propagation matrix A to the identity matrix A = I;

[0057] S3032. Based on the parameters of each task model after self-learning, perform parameter propagation to calculate W′←WA, and substitute the propagated parameters into each task model to calculate the gradient of the parameter propagation matrix A; wherein, the gradient of the parameter propagation matrix A... The calculation is shown in equation (6):

[0058]

[0059] In the formula, The parameters of the Mth task model obtained through parameter propagation calculation;

[0060] S3033, Gradient-based The gradient is updated on the parameter propagation matrix A, as shown in equation (7):

[0061]

[0062] In the formula, λ A The learning rate is used to update the gradient of the parameter propagation matrix A;

[0063] S3034. Based on the parameter propagation matrix A updated by gradient, repeat steps S3032-S3033 to complete the training of the parameter propagation matrix.

[0064] S304. Repeat steps S302-S303 until the total number of iterations reaches the preset threshold T, and complete the training of each task model.

[0065] To further verify the superiority of the method of the present invention, this embodiment compares the performance of the traditional parameter sharing method and the parameter propagation method of this aspect on a publicly available multi-task image understanding dataset, Taskonomy.

[0066] Dataset: The Taskonomy dataset contains images collected during the movement of robots in multiple indoor rooms, aiming to solve multiple image understanding tasks simultaneously using these images. Due to the large size of the original dataset, this embodiment only performs validation on a subset. Specifically, it only considers the Cauthron room dataset, consisting of 9464 RGB images, targeting five tasks: depth estimation, normal vector estimation, keypoint detection, edge detection, and re-deformation, and dividing the training and test sets in a 5:1 ratio.

[0067] Model: All task models constructed include an encoder and a decoder, corresponding to the multi-layer convolutional neural networks XceptionNet and ResNet34, respectively. For traditional parameter-sharing methods, all tasks share the encoder parameters, denoted as w, while the decoder parameters are not shared, denoted as p. i In the method of this invention, the parameters of all encoders and decoders are not shared; the encoder parameters participate in parameter propagation, denoted as w. i The decoder's parameters do not participate in parameter propagation, denoted as d. i .

[0068] Experimental setup: During stochastic gradient updates, the learning rate is fixed at 0.05, and the number of input images sampled each time is b. i =16. For the neighbor learning phase of the method of this invention, the learning rate of the neighbor learning phase is λ. A =0.1. Self-learning switches to neighbor learning every 10 epochs, and neighbor learning switches back to self-learning every 10 epochs. The total number of epochs for all updates is T = 300.

[0069] Experimental results: The prediction errors of the traditional parameter sharing method and the method of the present invention on the test set are shown in Table 1. As can be seen from Table 1, the prediction error of the method of the present invention is lower than that of the traditional method on all tasks.

[0070] Table 1

[0071]

[0072] Example 2:

[0073] Reference Figure 4 As shown, this embodiment provides a multi-task image understanding device based on parameter propagation, including a data acquisition module, a model building module, and a model training module, wherein the data acquisition module and the model building module are respectively connected to the model training module;

[0074] The data acquisition module is used to acquire several image data and preprocess the image data to construct a training dataset;

[0075] The model building module is used to acquire several image understanding tasks, and to build a task model for each image understanding task. The several image understanding tasks are related to each other, and each task model has independent parameters.

[0076] In this module, the model building module uses a multi-layer convolutional neural network to build a task model for each image understanding task. The structure of the task model has been described in detail in Embodiment 1, and will not be repeated here.

[0077] The model training module is based on a linear parameter propagation mechanism and trains each task model using the training dataset; the trained task models are then used to perform image understanding on the image data to be processed.

[0078] In this module, the model training module includes a self-learning module and a neighbor learning module connected in sequence;

[0079] The self-learning module is used to independently train the parameters of each task model.

[0080] The neighbor learning module is used to fix the parameters of all task models and train the parameter propagation matrix; the parameter propagation matrix is ​​used to represent the relationship between the various image understanding tasks.

[0081] The self-learning module iteratively updates the parameters of each task model using the stochastic gradient method.

[0082] The specific training methods for the self-learning module and the neighbor learning module have been described in detail in Example 1, and will not be repeated here.

[0083] The embodiments described above are merely preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Various modifications and improvements made by those skilled in the art to the technical solutions of the present invention without departing from the spirit of the present invention should fall within the protection scope defined by the claims of the present invention.

Claims

1. A method for multi-task picture understanding based on parameter propagation, characterized in that, The method comprises the following steps: Obtaining a plurality of picture data, and constructing a training data set after preprocessing the picture data; Obtaining a plurality of picture understanding tasks, and constructing a task model for each picture understanding task, wherein the plurality of picture understanding tasks are associated with each other, and each task model has independent parameters; Training each task model based on a linear parameter propagation mechanism through the training data set; and using the trained task model to perform picture understanding on picture data to be processed; The training of the task model based on the linear parameter propagation mechanism comprises two stages of self-learning and neighbor learning; In the self-learning stage, each parameter of each task model is trained independently; In the neighbor learning stage, the parameters of all task models are fixed, and a parameter propagation matrix is trained, wherein the parameter propagation matrix is used to represent the relationship between the plurality of picture understanding tasks, and comprises a weighting parameter of the parameters of the task model corresponding to each picture understanding task, and the weighting parameter is used to perform weighted summation on the parameters of each task model; The method for training the parameter propagation matrix comprises the following steps: S3031, initializing the parameter propagation matrix as an identity matrix; S3032, performing parameter propagation calculation based on the parameters of each task model after self-learning, and substituting the parameters obtained after the parameter propagation calculation into each task model to calculate the gradient of the parameter propagation matrix; S3033, updating the gradient of the parameter propagation matrix; and S3034, repeating steps S3032-S3033 based on the gradient updated parameter propagation matrix to complete the training of the parameter propagation matrix.

2. The parameter propagation based multi-task picture understanding method according to claim 1, characterized in that, A multi-layer convolutional neural network is used to construct a task model for each picture understanding task.

3. The parameter propagation based multi-task picture understanding method according to claim 1, characterized in that, In the self-learning stage, the method for training each parameter of each task model independently comprises the following steps: Randomly initializing the parameters of all task models, and iteratively updating the parameters of each task model through a stochastic gradient method.

4. The parameter propagation based multi-task picture understanding method according to claim 3, characterized in that, The specific method for updating the parameters of each task model through the stochastic gradient method comprises the following steps: In each iterative update process, randomly sampling data from the training data set of the task model, calculating the gradient of the parameters of the task model through the sampled data, and updating the parameters of the task model based on the gradient of the parameters.

5. A parameter propagation based multi-task picture understanding apparatus, characterized by, The method comprises a data acquisition module, a model construction module, and a model training module, wherein the data acquisition module and the model construction module are connected to the model training module; The data acquisition module is used to obtain a plurality of picture data, and construct a training data set after preprocessing the picture data; The model construction module is used to obtain a plurality of picture understanding tasks, and construct a task model for each picture understanding task, wherein the plurality of picture understanding tasks are associated with each other, and each task model has independent parameters; The model training module is used to train each task model based on a linear parameter propagation mechanism through the training data set; and use the trained task model to perform picture understanding on picture data to be processed. The model training module trains each of the task models based on a linear parameter propagation mechanism and the training data set; and each of the trained task models is used for picture understanding of the picture data to be processed. The model training module comprises a self-learning module and a neighbor learning module connected in sequence. The self-learning module is configured to independently train parameters of each of the task models. The neighbor learning module is configured to fix the parameters of all the task models and train a parameter propagation matrix; the parameter propagation matrix is used to represent relationships between the picture understanding tasks, including weighted parameters of the parameters of the task models corresponding to the picture understanding tasks, the weighted parameters being used to perform weighted summation on the parameters of the task models; and a method for training the parameter propagation matrix comprises: S3031, initializing the parameter propagation matrix as an identity matrix; S3032, performing parameter propagation calculation based on the parameters of each of the task models after self-learning, and substituting the parameters obtained after the parameter propagation calculation into each of the task models to calculate gradients of the parameter propagation matrix; S3033, performing gradient update on the parameter propagation matrix based on the gradients of the parameter propagation matrix; and S3034, repeating steps S3032-S3033 based on the parameter propagation matrix after the gradient update to complete training of the parameter propagation matrix.

6. The parameter propagation based multi-task picture understanding apparatus according to claim 5, characterized in that, The model construction module constructs a task model for each of the picture understanding tasks using a multi-layer convolutional neural network.

7. The parameter propagation based multi-task picture understanding apparatus according to claim 5, characterized in that, The self-learning module is configured to iteratively update the parameters of each of the task models using a stochastic gradient method.

Images