A multi-task learning-based transfer learning method, device and system

Abstract

Embodiments of the present application provide a kind of transfer learning method, device and system based on multi-task learning, by first difference and second difference value, auxiliary network and main network are trained, since auxiliary network is to the image of destruction and is repaired then first difference can not need to be additionally labeled to image, therefore only a small amount of image on target domain labeled in the present application is used.Auxiliary network and the main network of main network part are common, when first difference and second difference are based on and neural network is trained, first difference can be adjusted to improve its feature extraction capability on target domain by optimizing main network, and simultaneously, other network parts of main network are further adjusted by second difference, therefore, the image feature of image on target domain can be accurately extracted by the main network after training, therefore, the performance of target main network obtained by training can be improved in the case of reducing training cost.

Description

Technical Field

[0001] This invention relates to the field of transfer learning technology, and in particular to a transfer learning method, apparatus and system based on multi-task learning. Background Technology

[0002] Training deep neural networks often requires labeled sample images collected in a specific context (referred to as the target domain in this paper). On the one hand, collecting samples in the target domain is difficult in some application scenarios, such as in industrial quality inspection, where defects occur infrequently and there are few defect samples. On the other hand, labeling samples is time-consuming, labor-intensive, and costly. Therefore, the number of labeled sample images that can be collected in the target domain is limited, resulting in poor performance of the trained deep neural network.

[0003] In related technologies, labeled sample images collected from other locations (referred to as the source domain in this paper) can be used to train a deep neural network. This allows the deep neural network to learn relevant knowledge from the sample images in the source domain. Then, a small number of labeled sample images in the target domain are used to fine-tune the deep neural network. That is, the parameters of the backbone network or the first few layers of the backbone network are kept unchanged, and the parameters of other parts of the network are adjusted using a small number of samples in the target domain (a region with only a small amount of labeled data). This allows the deep neural network to transfer the knowledge learned from the samples in the source domain to the images in the target domain. In this way, the trained deep neural network can extract the features of the images in the target domain and perform detection, recognition and other processing based on the features.

[0004] However, when the difference between the target domain and the source domain is large, the knowledge learned by the deep neural network from the samples in the source domain is difficult to apply to images in the target domain. As a result, the trained deep neural network cannot accurately extract the features of the images in the target domain. Therefore, the trained deep neural network cannot accurately perform detection, recognition and other processing based on the extracted features, that is, the performance of the trained deep neural network is low. Summary of the Invention

[0005] The purpose of this invention is to provide a transfer learning method, apparatus, and system based on multi-task learning, to address the problem of low backbone network performance in transfer learning. The specific technical solution is as follows:

[0006] In a first aspect of the present invention, a transfer learning method based on multi-task learning is provided, characterized in that the method includes:

[0007] Obtain a first image dataset and a second image dataset on the target domain, wherein the second image dataset is obtained by destroying the images in the first image dataset;

[0008] The second image dataset is input into the auxiliary network for repair, resulting in the third image dataset;

[0009] The labeled fourth image dataset in the target domain is input into the main network to obtain the predicted values ​​of each image in the fourth image dataset;

[0010] The auxiliary network and the main network are trained based on the first difference and the second difference; wherein, the first difference is the difference between the third image dataset and the first image dataset; the second difference is the difference between the labeled values ​​of the fourth image dataset in the target domain and the predicted values ​​of each image in the fourth image dataset; the backbone networks in the auxiliary network and the main network are shared.

[0011] In one possible embodiment, obtaining the first image dataset in the target domain includes:

[0012] Obtain the target dataset in the target domain;

[0013] The images in the target dataset are standardized and / or augmented to obtain a first image dataset.

[0014] In one possible embodiment, obtaining the second image dataset over the target domain includes:

[0015] Noise blocks are superimposed on the images in the first image dataset to obtain the second image dataset.

[0016] In one possible embodiment, training the auxiliary network and the main network based on the first difference and the second difference includes:

[0017] The image restoration loss is calculated based on the first difference; the image restoration loss is negatively correlated with the difference.

[0018] The prediction loss is calculated based on the second difference; the prediction loss is negatively correlated with the difference.

[0019] Calculate the total loss based on the image restoration loss and the prediction loss;

[0020] The auxiliary network and the main network are trained based on the total loss.

[0021] In a second aspect of the present invention, a transfer learning apparatus based on multi-task learning is provided, characterized in that the apparatus comprises:

[0022] The acquisition module is used to acquire a first image dataset and a second image dataset on the target domain, wherein the second image dataset is obtained by destroying the images in the first image dataset;

[0023] The repair module is used to input the second image dataset into the auxiliary network for repair, so as to obtain the third image dataset;

[0024] The prediction module is used to input the labeled fourth image dataset in the target domain into the main network to obtain the predicted value of each image in the fourth image dataset;

[0025] The training module is used to train the auxiliary network and the main network based on a first difference and a second difference; wherein, the first difference is the difference between the third image dataset and the first image dataset; the second difference is the difference between the labeled values ​​of the fourth image dataset in the target domain and the predicted values ​​of each image in the fourth image dataset; the backbone network in the auxiliary network and the main network is shared.

[0026] In one possible embodiment, the acquisition module is specifically used to acquire a target dataset in a target domain; and to perform normalization and / or data augmentation on the images in the target dataset to obtain a first image dataset;

[0027] Noise blocks are superimposed on the images in the first image dataset to obtain the second image dataset;

[0028] The training module is specifically used to calculate the image restoration loss based on the first difference; the image restoration loss is negatively correlated with the difference.

[0029] The prediction loss is calculated based on the second difference; the prediction loss is negatively correlated with the difference.

[0030] Calculate the total loss based on the image restoration loss and the prediction loss;

[0031] The auxiliary network and the main network are trained based on the total loss.

[0032] In a third aspect of the present invention, a transfer learning system based on multi-task learning is provided, characterized in that the system includes an auxiliary network and a main network for implementing any of the method steps described in the first aspect above.

[0033] In a fourth aspect of the present invention, an electronic device is provided, comprising:

[0034] Memory, used to store computer programs;

[0035] When a processor executes a program stored in memory, it implements any of the steps described in the first aspect above.

[0036] In a fifth aspect of the present invention, a computer-readable storage medium is provided, wherein a computer program is stored therein, and when executed by a processor, the computer program implements the steps of any of the methods described in the first aspect above.

[0037] Beneficial effects of the embodiments of the present invention:

[0038] This invention provides a transfer learning method, apparatus, and system based on multi-task learning. An auxiliary network performs image inpainting on a second image dataset in the target domain. Since the second image dataset is obtained by damaging the first image dataset, the first image dataset can be considered the ground truth of the second image dataset for image inpainting. Therefore, the backbone network of the auxiliary network is trained using the inpainted third image dataset and the first image dataset. This allows the backbone network of the auxiliary network to learn how to accurately extract image features from the second image dataset during training. Because the backbone network parts of the auxiliary network and the main network are shared (i.e., the backbone network parameters of the auxiliary network and the main network are the same), the original backbone network can also accurately extract the features from the second image dataset. Regarding the image features of the image dataset, since the backbone of the original main network can already extract image features relatively accurately, the subsequent training of the original main network using the labeled fourth image dataset only requires a small number of labeled images from the target domain. Furthermore, since the training of the auxiliary network's backbone did not require labeled images from the target domain, this application only needs to use a small number of labeled images from the target domain. Simultaneously, since the second image dataset is an image dataset from the target domain, and as analyzed above, the backbone of the original main network can accurately extract image features from the second image dataset, the target main network trained based on the original main network can accurately extract image features from images in the target domain, thus exhibiting high performance. In other words, this application can improve the performance of the trained target main network while reducing training costs.

[0039] Of course, implementing any product or method of the present invention does not necessarily require achieving all of the advantages described above at the same time. Attached Figure Description

[0040] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other embodiments can be obtained based on these drawings.

[0041] Figure 1 A flowchart illustrating a transfer learning method based on multi-task learning provided in an embodiment of this application;

[0042] Figure 2 A schematic diagram illustrating a process for training an auxiliary network and a main network based on a first difference and a second difference, provided for an embodiment of this application;

[0043] Figure 3 This is a schematic diagram of a process for preprocessing image data to obtain a first image dataset, provided in an embodiment of this application.

[0044] Figure 4 A schematic diagram of a transfer learning process based on multi-task learning provided for an embodiment of this application;

[0045] Figure 5 A schematic diagram of the structure of a transfer learning device based on multi-task learning provided in an embodiment of this application;

[0046] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0047] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art based on this application are within the scope of protection of the present invention.

[0048] In related technologies, training deep neural networks requires labeled sample images collected in a specific context (referred to as the target domain in this paper). On the one hand, collecting samples in the target domain is difficult in some application scenarios, such as in industrial quality inspection, where the frequency of defects is low and defect samples are scarce. On the other hand, labeling samples is time-consuming, labor-intensive, and costly. Currently, transfer learning is often used, where a deep neural network is trained using labeled sample images from the source domain, and then fine-tuned using a small number of labeled sample images from the target domain. However, when the difference between the target domain and the source domain is significant, the knowledge learned by the deep neural network from the samples in the source domain is difficult to apply to images in the target domain. Consequently, the trained deep neural network cannot accurately extract features from images in the target domain, and therefore cannot accurately perform detection, recognition, or other processing based on the extracted features, resulting in low performance.

[0049] Based on this, this application provides a transfer learning method based on multi-task learning, such as... Figure 1As shown, the transfer learning method based on multi-task learning provided in this application includes:

[0050] S101. Obtain the first image dataset and the second image dataset on the target domain, wherein the second image dataset is obtained by destroying the images in the first image dataset.

[0051] S102. Input the second image dataset into the auxiliary network for repair to obtain the third image dataset.

[0052] S103. Input the labeled fourth image dataset in the target domain into the main network to obtain the predicted values ​​of each image in the fourth image dataset.

[0053] S104. Train the auxiliary network and the main network based on the first difference and the second difference; wherein, the first difference is the difference between the third image dataset and the first image dataset; the second difference is the difference between the labeled values ​​of the fourth image dataset in the target domain and the predicted values ​​of each image in the fourth image dataset; the backbone network in the auxiliary network and the main network is shared.

[0054] In this embodiment, an auxiliary network is used to perform image inpainting on a second image dataset in the target domain. Since the second image dataset is obtained by destroying the first image dataset, the first image dataset can be considered the ground truth of the second image dataset for image inpainting. Therefore, the backbone network of the auxiliary network is trained using the inpainted third image dataset and the first image dataset. This allows the backbone network of the auxiliary network to learn how to accurately extract image features from the second image dataset during training. Since the backbone network parts of the auxiliary network and the main network are shared, i.e., the backbone network parameters of the auxiliary network and the main network are the same, the backbone network of the original main network can also accurately extract image features from the second image dataset. The backbone of the original main network is already capable of accurately extracting image features. Therefore, during the subsequent training of the original main network using the labeled fourth image dataset, only a small number of labeled images from the target domain are needed. Furthermore, since the training of the auxiliary network's backbone did not require labeled images from the target domain, this application only requires a small number of labeled images from the target domain. Simultaneously, since the second image dataset is an image dataset from the target domain, and as analyzed above, the backbone of the original main network can accurately extract image features from the second image dataset, the target main network trained based on the original main network can accurately extract image features from images in the target domain. Therefore, the target main network exhibits high performance. In other words, this application can improve the performance of the trained target main network while reducing training costs.

[0055] The following will provide a detailed explanation of the aforementioned steps S101-S104:

[0056] In S101, the target domain is a pre-established dataset based on a preset task, including labeled and unlabeled images. Both labeled and unlabeled images can be used to train the auxiliary network. For example, the preset task is face recognition, virtual character face recognition, etc. Assuming the preset task is face recognition, the target domain is a pre-established image dataset related to various face images, where most images are unlabeled and a small portion are labeled with facial features. The first image dataset is a dataset of face images randomly selected from the target domain, or it can be a dataset of face images selected from every three storage locations in the target domain, or it can be the entire target domain. The second image dataset is a dataset of face images obtained by destructing the images in the first image dataset.

[0057] In one possible implementation, obtaining the second image dataset over the target domain can be specifically as follows:

[0058] Noise blocks are superimposed on the images in the first image dataset to obtain the second image dataset.

[0059] In this step, the images in the first image dataset can be corrupted by overlaying noise blocks of random quantity, grayscale value, and shape onto the images in the first image dataset, thus obtaining the second image dataset.

[0060] Since the second image dataset needs to be repaired subsequently to train the backbone network, the more random the image destruction, the greater the destruction effect, which places higher demands on the backbone network's performance and is also more beneficial for its training. Therefore, we can randomly superimpose noise blocks onto the images in the first image dataset.

[0061] This application does not limit the specific superposition method. It can superimpose noise blocks with random attributes, such as noise blocks with random grayscale values, noise blocks with random shapes, or noise blocks with both random grayscale values ​​and shapes. The specific method of superimposing noise blocks to degrade the image can be adjusted according to actual needs.

[0062] Taking face image 1 (first image dataset) as an example, face image 1' (second image dataset) can be obtained by superimposing a noise block on face image 1. Alternatively, face image 1' can be obtained by superimposing a noise block with a gray value of 2, a noise block with a gray value of 3, or a noise block with a gray value of 125 on face image 1. Alternatively, face image 1' can be obtained by superimposing a noise block of size 3×3 on face image 1. Alternatively, face image 1' can be obtained by superimposing a noise block with a gray value of 2 and a noise block of size 3×3 on face image 1.

[0063] In this application embodiment, the first image dataset can also be damaged in other ways to obtain the second image dataset. Any way that can damage the image can be applied to this application embodiment, and this application does not make any limitation.

[0064] In S102, the auxiliary network is a repair network that can repair the damaged image. In one possible implementation, the step of inputting the second image dataset into the auxiliary network for repair to obtain the third image dataset can be specifically as follows:

[0065] a. Input the second image dataset into the auxiliary network, which includes a backbone network and a decoding network corresponding to the backbone network.

[0066] In this step, the decoding network can be a decoder network whose structure is symmetrical to that of the backbone network.

[0067] b. The backbone network extracts image features from the second image dataset.

[0068] c. The decoding network decodes and repairs the images in the second image dataset based on the image features extracted in step b, thus obtaining the third image dataset.

[0069] Taking face image 1 (first image dataset) as an example, face image 1 is destroyed to obtain face image 1' (second image dataset). Face image 1' is input into the auxiliary network. The backbone network extracts feature a of face image 1'. The decoding network repairs face image 1' based on feature a to obtain repaired face image 1 (third image dataset).

[0070] In this step, the damaged image is repaired using the backbone network and decoding network in the auxiliary network. Better extracted image features result in better image repair, indicating higher backbone network performance. Therefore, by continuously extracting features and repairing the image, and then learning from the differences between the repaired image and the first image dataset, the backbone network's parameters are continuously updated, enabling it to better extract image features, thus improving its performance. Therefore, in this embodiment, training the backbone network through the auxiliary network improves its performance in extracting image features.

[0071] In S103, the fourth image dataset can be an image dataset selected from the first image dataset, or it can be an image dataset selected from the target domain, depending on the actual needs. One possible implementation involves inputting the labeled fourth image dataset from the target domain into the main network to obtain the predicted values ​​for each image in the fourth image dataset, specifically as follows:

[0072] S1031, Feature Extraction: Extract image features from the fourth image dataset through the backbone network of the main network.

[0073] In this step, since the backbones of the auxiliary and main networks are shared (i.e., their backbone parameters are identical), the main network performs well in feature extraction. Therefore, to avoid overfitting of the original main network, the fourth image dataset can be either an image dataset augmented from the first image dataset or an image dataset augmented from the target domain.

[0074] In one possible embodiment, the above steps may involve inputting the data-enhanced image into the original main network and extracting the image features through the backbone network of the original main network.

[0075] S1032. Based on the image features of the fourth image dataset, obtain the predicted value of the main task.

[0076] In this step, after obtaining the image features of the fourth image dataset, the other parts of the main network continue to run based on the image features of the fourth image dataset, and finally the predicted values ​​of each image in the fourth image dataset are obtained.

[0077] In S104, both the auxiliary network and the main network include a backbone network. During training, the network parameters of the backbone network in the auxiliary network are the same as those in the main network. Therefore, during training, the network parameters of the backbone network in the auxiliary network can be directly obtained, thereby improving training efficiency. Since the third image dataset is obtained by repairing the second image dataset, and the first image dataset can be considered the ground truth of the second image dataset, all three datasets can contain unlabeled images. The second difference is the difference between the labeled values ​​of the fourth image dataset in the target domain and the predicted values ​​of each image in the fourth image dataset. Therefore, at least one labeled image exists in the fourth image dataset.

[0078] In one possible implementation, such as Figure 2 As shown, the steps for training the auxiliary network and the main network based on the first and second differences can be specifically as follows:

[0079] S201. Calculate the image restoration loss based on the first difference; the image restoration loss is negatively correlated with the difference.

[0080] The first difference is the difference between the first image dataset and the third image dataset, which can be obtained from the differences in the pixel values ​​of the input image in the first image dataset and the corresponding repaired image in the third image dataset.

[0081] In this step, the image restoration loss can be calculated using the following formula:

[0082]

[0083] Among them, L aux For image inpainting loss, X i This represents the value of the i-th pixel in the input image of the first image dataset. Let X represent the i-th pixel value of the restored image in the third image dataset, and P represent P in the LP norm, where P = 1. Then the image restoration loss is X, i.e., X1 - X. n If the mean absolute error is P = 2, then the image inpainting loss is X, i.e., X1 - X n The mean square error is given by N, where N represents the total number of pixels in the image.

[0084] Taking the aforementioned face image 1 (first image dataset) as an example, after obtaining the repaired face image 1 (third image dataset), by inputting face image 1, face image 1" and the total number of pixels of the images into the formula, L can be calculated. aux 1.

[0085] S202. Calculate the predicted loss using the second difference; the predicted loss is negatively correlated with the difference.

[0086] In this step, the predicted values ​​of each image in the fourth image dataset obtained in step S103 are compared with their corresponding labeled values ​​to obtain the main task prediction loss value L. main The labeled values ​​are obtained based on the labels in the labeled images. Since the target domain includes both labeled and unlabeled images, and the unlabeled images do not have corresponding ground truth labels, only the labeled image portion is calculated when calculating the prediction loss value of the main task, and the unlabeled images are skipped.

[0087] S203. Calculate the total loss based on image inpainting loss and prediction loss.

[0088] In one possible implementation, the total loss can be calculated based on the following loss function:

[0089] L total =L main +γ·L aux

[0090] Among them, L total For the total loss, L main To predict losses, L aux Let γ be the image inpainting loss, and let γ be the weighting coefficient of the image inpainting loss in the total loss function.

[0091] S204. Based on the total loss, train the auxiliary network and the main network.

[0092] In this step, the auxiliary network and the main network can be trained using the backpropagation algorithm.

[0093] For example, the image inpainting loss is calculated as L through an auxiliary task. main 1. The prediction loss is calculated as L through the main task. aux 1. Assuming the weight coefficient of image loss in the total loss function is 'a', the calculated total loss is L. total 1 = L main 1+a·L aux 1. The auxiliary network and the main network are trained using the backpropagation algorithm.

[0094] In this embodiment, since the backbone network parts of the auxiliary task network and the main network are shared, that is, the backbone network parameters of the two tasks are the same, and the auxiliary network is for image inpainting, the first difference obtained through the auxiliary network can be used to train the performance of the backbone network in extracting image features. Therefore, the backbone network has good performance in extracting image features. Then, by using the difference between the predicted value and the ground truth label of the labeled image, the main network is adjusted so that the main network can be more suitable for its corresponding application scenario. In this way, the backbone network can extract image features well in the target domain, thereby improving the feature extraction capability of the main network on the target and the image.

[0095] In one feasible embodiment, during image data processing, in order to improve the processing efficiency of image data, the image data is preprocessed to obtain a first image dataset, such as... Figure 3 As shown, it specifically includes:

[0096] S301. Obtain the target dataset in the target domain.

[0097] In this step, the entire target domain is not needed when training the backbone network. Therefore, to reduce the amount of data processing, a portion of the images in the target domain can be selected to form the target dataset, and subsequent preprocessing operations can be performed only on the images in the target dataset. For example, the target dataset can be selected randomly or according to preset rules.

[0098] S302. Preprocess the images in the target dataset to obtain the first image dataset.

[0099] In this step, the images in the target dataset are preprocessed. Preprocessing includes standardization and / or data augmentation. That is, preprocessing may include only standardization, only data augmentation, or both. Standardization can be performed according to the following formula:

[0100]

[0101] Where X represents the original image data of the target dataset, X′ represents the standardized image data, and μ and σ represent the mean and standard deviation of the original image data in the three RGB channels, respectively.

[0102] In this step, after standardizing the images in the target dataset, all image data are in the same format. Therefore, when processing the data, we do not need to consider the data format conversion, which can improve the efficiency of data processing.

[0103] Data augmentation can be performed on the original image data in the target dataset by performing any one or a combination of operations such as random rotation, translation, and pixel value transformation. Augmented image data increases its diversity, allowing neural networks to be trained with a wider variety of images, thus preventing overfitting.

[0104] Taking the aforementioned face image 1 as an example, before obtaining face image 1 (the first image dataset), face image a (the target dataset) is first selected from the target domain. After standardizing face image a, face image 1 is obtained; alternatively, face image a can be data-enhanced to obtain face image 1; or face image a can be both standardized and data-enhanced simultaneously. The specific preprocessing operations performed on the image can be set according to actual needs, and this application embodiment does not impose too many limitations.

[0105] In another feasible embodiment, all data corresponding to the target domain can be preprocessed before selecting the first image dataset directly from the preprocessed target domain. This avoids subsequent preprocessing operations during data processing, improving the efficiency of data processing.

[0106] In this embodiment, a first image dataset is obtained by preprocessing the original image data in the target dataset. This includes standardization and / or data augmentation. Standardization can standardize various types of image data in the target dataset, thereby improving the efficiency of subsequent data processing. Data augmentation can increase the diversity of data, so that the neural network can be trained with as many diverse image data as possible, avoiding overfitting of the neural network.

[0107] This application provides a flowchart of a transfer learning process based on over-task learning, as shown in the embodiments below. Figure 4As shown, the remaining parts of the main network are those other than the backbone network. Since the auxiliary network calculates the image inpainting loss based on the first difference, and the image inpainting process is also an image reconstruction process, the image inpainting loss in this application can also be called the reconstruction loss. Since the main network calculates the prediction loss based on the difference between the final output of the main network and the fourth image dataset, i.e., the second difference, and the final output is also the running process of the main network, the prediction loss in this application can also be called the main network loss.

[0108] In one possible implementation, the target domain includes unlabeled images and labeled images. The specific content of the target domain is the same as described in S101-S102 above, and will not be repeated here. After preprocessing the image data in the target domain, the preprocessing process is the same as described in S301-S302 above, and will not be repeated here. The process of inputting the image data into the auxiliary network and the main network to train the main network and the auxiliary network is the same as described in S201-S204 above, and will not be repeated here.

[0109] Corresponding to the aforementioned transfer learning method based on multi-task learning, this application also provides a transfer learning device based on multi-task learning, such as... Figure 5 As shown, the device includes:

[0110] The acquisition module 501 is used to acquire a first image dataset and a second image dataset on the target domain, wherein the second image dataset is obtained by destroying the images in the first image dataset;

[0111] Repair module 502 is used to input the second image dataset into the auxiliary network for repair to obtain the third image dataset;

[0112] The prediction module 503 is used to input the labeled fourth image dataset in the target domain into the main network to obtain the predicted value of each image in the fourth image dataset;

[0113] Training module 504 is used to train the auxiliary network and the main network based on a first difference and a second difference; wherein, the first difference is the difference between the third image dataset and the first image dataset; the second difference is the difference between the labeled values ​​of the fourth image dataset in the target domain and the predicted values ​​of each image in the fourth image dataset; the backbone network parts of the auxiliary network and the main network are shared, that is, the backbone network parameters of the auxiliary network and the main network are the same.

[0114] In one possible implementation, the acquisition module 501 is specifically used to acquire a target dataset in the target domain; and to perform normalization and / or data augmentation on the images in the target dataset to obtain a first image dataset;

[0115] Noise blocks are superimposed on the images in the first image dataset to obtain the second image dataset;

[0116] In one possible implementation, the training module 504 is specifically used to calculate an image restoration loss based on the first difference; the image restoration loss is negatively correlated with the difference.

[0117] The prediction loss is calculated based on the second difference; the prediction loss is negatively correlated with the difference.

[0118] Calculate the total loss based on the image restoration loss and the prediction loss;

[0119] The auxiliary network and the main network are trained based on the total loss.

[0120] This application also provides an electronic device, such as... Figure 6 As shown, the system includes a processor 601 and a memory 602, wherein the memory 602 is used to store computer programs; when the processor 601 executes the program stored in the memory 602, it performs the following steps:

[0121] Obtain a first image dataset and a second image dataset on the target domain, wherein the second image dataset is obtained by destroying the images in the first image dataset;

[0122] The second image dataset is input into the auxiliary network for repair, resulting in the third image dataset;

[0123] The labeled fourth image dataset in the target domain is input into the main network to obtain the predicted values ​​of each image in the fourth image dataset;

[0124] The auxiliary network and the main network are trained based on the first difference and the second difference; wherein, the first difference is the difference between the third image dataset and the first image dataset; the second difference is the difference between the labeled values ​​of the fourth image dataset in the target domain and the predicted values ​​of each image in the fourth image dataset; the backbone network parts of the auxiliary network and the main network are shared, that is, the backbone network parameters of the auxiliary network and the main network are the same.

[0125] The communication bus mentioned in the above electronic devices can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not mean that there is only one bus or one type of bus.

[0126] The communication interface is used for communication between the aforementioned electronic devices and other devices.

[0127] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0128] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0129] In another embodiment of the present invention, a computer-readable storage medium is also provided, which stores a computer program that, when executed by a processor, implements the steps of the above-described transfer learning method based on multi-task learning.

[0130] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid state disk (SSD)).

[0131] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0132] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the apparatus embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0133] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention are included within the scope of protection of the present invention.

Claims

1. A multi-task learning based transfer learning method, characterized in that, The method comprises: obtaining a first image data set and a second image data set on a target domain, wherein the second image data set is obtained by destroying images in the first image data set; the target domain comprises labeled images and unlabeled images; inputting the second image data set into an auxiliary network for repairing to obtain a third image data set; inputting a fourth image data set labeled on the target domain into a main network to obtain predicted values of images in the fourth image data set; training the auxiliary network and the main network based on first and second differences; wherein the first difference is a difference between the third image data set and the first image data set; the second difference is a difference between labeled values of the fourth image data set on the target domain and predicted values of images in the fourth image data set; and a backbone network in the auxiliary network and the main network is shared.

2. The method of claim 1, wherein, The obtaining of the first image data set on the target domain comprises: obtaining a target data set on the target domain; standardizing and / or data enhancing images in the target data set to obtain the first image data set.

3. The method of claim 1, wherein, The obtaining of the second image data set on the target domain comprises: superimposing noise blocks on images in the first image data set to obtain the second image data set.

4. The method of claim 1, wherein, The training of the auxiliary network and the main network based on the first and second differences comprises: calculating an image repairing loss through the first difference; the image repairing loss and the difference are negatively correlated; calculating a prediction loss through the second difference; the prediction loss and the difference are negatively correlated; calculating a total loss based on the image repairing loss and the prediction loss; training the auxiliary network and the main network based on the total loss. 5.A device for transfer learning based on multi-task learning, characterized in that, The device comprises: an obtaining module configured to obtain a first image data set and a second image data set on a target domain, wherein the second image data set is obtained by destroying images in the first image data set; and the target domain comprises labeled images and unlabeled images; a repairing module configured to input the second image data set into an auxiliary network for repairing to obtain a third image data set; a predicting module configured to input a fourth image data set labeled on the target domain into a main network to obtain predicted values of images in the fourth image data set; a training module configured to train the auxiliary network and the main network based on first and second differences; wherein the first difference is a difference between the third image data set and the first image data set; the second difference is a difference between labeled values of the fourth image data set on the target domain and predicted values of images in the fourth image data set; and a backbone network in the auxiliary network and the main network is shared.

6. The apparatus of claim 5, wherein, The obtaining module is specifically configured to obtain a target data set on the target domain; and standardize and / or data enhance images in the target data set to obtain the first image data set; superimpose noise blocks on images in the first image data set to obtain the second image data set.

7. The apparatus of claim 5, wherein, The training module is specifically configured to calculate an image inpainting loss through the first difference; the image inpainting loss is negatively correlated with the difference; calculate a prediction loss through the second difference; the prediction loss is negatively correlated with the difference; calculate a total loss based on the image inpainting loss and the prediction loss; train the auxiliary network and the main network based on the total loss. 8.A multi-task learning based transfer learning system, characterized by, The system comprises an auxiliary network and a main network for implementing the method steps of any one of claims 1-4.

9. An electronic device, comprising: The system comprises a processor and a memory, wherein the memory is configured to store a computer program; The processor is configured to execute the program stored in the memory to implement the method steps of any one of claims 1-4.

10. A computer-readable storage medium, characterized in that, The computer program stored in the computer readable storage medium is executed by the processor to implement the method steps of any one of claims 1-4.

Images