Model training method, image processing method and related device
By adopting a shared encoder and two decoder architecture in the deep learning model, combining annotated public data and unlabeled actual data for training, the problem of deep learning model degradation when data distribution changes is solved, and high performance and low training costs in actual business scenarios are achieved.
Patent Information
- Application Number
- CN202311581497.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2023-08-24
- Filing Date
- 2023-11-23
- Publication Date
- 2025-02-25
AI Technical Summary
Deep learning models are prone to deterioration when data distribution changes, resulting in poor performance in actual business scenarios.
Using a shared encoder and two decoder architecture, the model is trained using annotated public data and unlabeled actual data during the model training stage. Through multiple iterative training, the model learns the actual data distribution.
It effectively improves the model degradation problem caused by inconsistent data distribution, improves the model's performance in actual business scenarios, and does not need to label the actual data during the training stage, which significantly saves model training costs.
Smart Images
Figure CN119514626A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of deep learning technology, and in particular to a model training method, an image processing method and related devices. Background Art
[0002] Deep learning (DL) technology is widely used in image processing business. For example, deep learning technology can be used to achieve various tasks such as image denoising, image super-resolution, image segmentation, target detection, etc. It can convert the input raw image data into the required output data by constructing a deep neural network model (NN).
[0003] When training deep learning neural network models, the training data used often comes from public data sets and is trained for specific prediction tasks of specific data sets. When the trained model is used in actual business scenarios, the model is prone to degradation due to changes in data distribution, and the performance cannot reach the effect shown in the training phase. Summary of the invention
[0004] The embodiments of the present application provide a model training method, an image processing method and related devices, which can solve the problem that the model is easily degraded due to changes in data distribution. To achieve the above purpose, the technical solutions provided by the embodiments of the present application are as follows:
[0005] In a first aspect, an embodiment of the present application provides a model training method for training a deep learning model, wherein the deep learning model includes a shared encoder, a first decoder, and a second decoder, and the model training method includes:
[0006] Acquire a training data set, and input each training data in the training data set into the shared encoder; wherein the training data includes first labeled training data and second unlabeled training data;
[0007] When the first training data is input to the shared encoder, a first output result is output through the first decoder, and a second output result is output through the second decoder;
[0008] When the second training data is input to the shared encoder, the second output result is output only through the second decoder;
[0009] The current model loss is calculated, and the deep learning model is iteratively trained according to the current model loss.
[0010] In a second aspect, an embodiment of the present application provides an image processing method, including:
[0011] Inputting the image to be processed into an encoder of a deep learning model to obtain features of the image to be processed;
[0012] Inputting the feature into a first decoder of the deep learning model to obtain an output result of the image to be processed;
[0013] Wherein, the deep learning model is trained according to any one of the model training methods described in claims 1-7.
[0014] In a third aspect, an embodiment of the present application provides a model training device, the model training device comprising:
[0015] An input module, used to obtain a training data set, and input each training data in the training data set into the shared encoder; wherein the training data includes first labeled training data and second unlabeled training data;
[0016] A first output module, configured to output a first output result through the first decoder when the first training data is input to the shared encoder;
[0017] A second output module, configured to output a second output result through the second decoder when the first training data or the second training data is input to the shared encoder;
[0018] The training module is used to calculate the current model loss and iteratively train the deep learning model according to the current model loss.
[0019] In a fourth aspect, an embodiment of the present application provides an image processing device, the image processing device comprising:
[0020] An input module inputs the image to be processed into an encoder of a deep learning model to obtain features of the image to be processed;
[0021] An output module, inputting the features of the image to be processed into a first decoder of the deep learning model to obtain an output result of the image to be processed;
[0022] Wherein, the deep learning model is trained according to any one of the model training methods described in claims 1-5.
[0023] In a fifth aspect, an embodiment of the present application provides an electronic device, comprising: a processor and a memory; wherein the memory stores a computer program, and the computer program is suitable for being loaded by the processor and executing the model training method described in the first aspect or the image processing method described in the second aspect.
[0024] In a sixth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores a plurality of instructions, and the instructions are suitable for being loaded by a processor and executed by the model training method described in the first aspect or the image processing method described in the second aspect.
[0025] The deep learning model of each embodiment of the present application adopts an architecture that adds a decoder on the basis of encoder-decoder (Encoder-Decoder), and the two decoders share an encoder. In the training process of the deep learning model, the pre-prepared training data set contains two kinds of data, one with annotations and one without annotations. For the public data set with annotations, there are two outputs of the first decoder and the second decoder; for the actual camera collection data without annotations, there is only one output of the second decoder. After that, based on the loss function corresponding to each of the two output lines, the deep learning model is iteratively trained for multiple times. After multiple iterative training, the encoder of the deep learning model will be able to learn the data distribution of the actual business scene. It can be understood that the annotated input data from the public data set can train the deep learning model for specific tasks, and the unlabeled input data from the actual business scene can reconstruct the model data distribution to make it closer to the real data distribution in the actual business scene. Furthermore, after the deep learning model converges, the required output data can be output through the first decoder in the reasoning stage, which improves the model degradation problem caused by inconsistent data distribution. BRIEF DESCRIPTION OF THE DRAWINGS
[0026] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.
[0027] Figure 1 A schematic diagram of the structure of a deep learning model provided in an embodiment of the present application;
[0028] Figure 2 A schematic diagram of the structure of another deep learning model provided in an embodiment of the present application;
[0029] Figure 3 A flowchart of a model training method provided in an embodiment of the present application;
[0030] Figure 4 A schematic diagram of the structure of a model training device provided in an embodiment of the present application;
[0031] Figure 5 A schematic diagram of the structure of an image processing device provided in an embodiment of the present application;
[0032] Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0033] In order to make the objectives, technical solutions and advantages of the present application more clear, the embodiments of the present application will be further described in detail below with reference to the accompanying drawings.
[0034] It should be clear that the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present application.
[0035] When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application. Instead, they are only examples of devices and methods consistent with some aspects of the present application as detailed in the attached claims.
[0036] In the description of the present application, it should be understood that the terms "first" and "second" etc. are only used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence, nor can they be understood as indicating or implying relative importance. For those of ordinary skill in the art, the specific meanings of the above terms in the present application can be understood according to the specific circumstances. In addition, in the description of the present application, unless otherwise specified, "multiple" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the objects associated before and after are in an "or" relationship.
[0037] The network model used in this application can be any model that uses an encoder-decoder architecture, such as CNN (convolutional neural networks), RNN (Recurrent Neural Network), LSTM (Long Short-Term Memory), etc. The model training method provided in this application can also be applied to various business scenarios. For example, in business scenarios such as image processing, natural language processing, and speech recognition, the training data can be selected according to the actual business scenario, and this application does not limit this. The following will take the image processing scenario as an example to describe the embodiments of this application in detail.
[0038] For image processing tasks, deep learning aims to construct pairs of input images and annotations (Groundtruth, GT for short) on a large scale. In the training phase, the neural network is driven by back propagation to learn the mapping from input images to annotations. In the model usage phase (also called the inference phase), the input image is input into the model to obtain the mapped model output result. Figure 1 , the deep learning model may include an encoder and decoder architecture. Based on this architecture, the input image is input into the encoder of the deep learning model, and the original high-dimensional feature map is converted into lower-dimensional features through several downsampling modules. Then, the low-dimensional features output by the encoder are input into the decoder, and after being processed by several upsampling modules, the reconstructed image is output.
[0039] However, the input data used by the model in the training phase often comes from public data sets, while in the model inference phase, the input data comes from actual business scenarios. For example, the input data in an actual business scenario can be collected by the camera used in the scenario. There is inevitably a distribution inconsistency between the data collected by the camera in the real scene and the data during training. This distribution inconsistency will cause the model performance to fail to reach the effect shown during training when the data collected by the camera is used for inference in the real scene.
[0040] In order to solve the model degradation problem caused by inconsistent data distribution, the following solutions can be adopted. One solution is to use the camera in actual use to collect large-scale input images and annotate them for training during the model training stage. This method directly solves the problem of inconsistent data distribution in the training and inference stages, but large-scale annotation leads to high model training costs. Another solution is to train on public data, then use the camera used in real scenes to collect small-scale input images for annotation, and fine-tune the model trained with public data. However, image processing tasks such as image super-resolution and image denoising require pixel-by-pixel annotation, which is difficult and costly.
[0041] Based on this, this application also provides another model training method. Figure 2 , the deep learning model provided by this method is Figure 1 A decoder is added on the basis of . For the convenience of description, the original decoder can be called the first decoder (ie, decoder 1), and the newly added decoder can be called the second decoder (ie, decoder 2). Based on such a model architecture, the encoder of the present application will be able to learn the data distribution of the actual business scenario, and then, after the deep learning model converges, the required output data can be output through the first decoder in the inference stage, which improves the model degradation problem caused by inconsistent data distribution. In addition, there is no need to annotate the data of the actual business scenario in the model training stage for use as training set data, which significantly saves the model training cost.
[0042] See also Figure 3 The model training method provided in this embodiment may specifically include the following steps:
[0043] S100, obtaining a training data set, and inputting each training data in the training data set into a shared encoder.
[0044] In implementation, a training data set can be prepared in advance. In the data preparation stage, a batch of data can be collected with the camera actually to be used, and randomly mixed with labeled data obtained from a public data set corresponding to a specific task in an actual business scenario. At this time, there are two types of training data in the training data set used to train the deep learning model, one is labeled training data from a public data set (which may be referred to as the first training data), and the other is unlabeled training data in an actual business scenario (which may be referred to as the second training data). Among them, the specific task can be any task, such as image denoising, image super-resolution, image segmentation, target detection, etc., and this application does not limit this.
[0045] It is worth mentioning that when training a deep learning model, the shared encoder is used to extract features from input data. Both the first decoder and the second decoder can perform subsequent processing based on the output features of the encoder. Therefore, the encoder provided in this embodiment can be referred to as a shared encoder.
[0046] S200: Determine whether the training data input to the shared encoder is the first training data.
[0047] In implementation, whether the input data is the first training data can be determined based on whether the input data is labeled. For example, when the training data input to the shared encoder is labeled data, it is determined to be the first training data, and step S300 can be executed; when the training data input to the shared encoder is not labeled data (i.e., unlabeled data), it is determined to be the second training data, and step S400 can be executed.
[0048] S300, outputting a first output result through a first decoder, and outputting a second output result through a second decoder.
[0049] S400: output the second output result through the second decoder.
[0050] In implementation, when the loaded data is a picture from a public data set, there may be two outputs, the first decoder and the second decoder. When the loaded data is actual collected data without annotations, there is only one output, the second decoder.
[0051] S500, calculating the current model loss, and iteratively training the deep learning model according to the current model loss.
[0052] In implementation, the training set data can be randomly input into the shared encoder. For the outputs of the first decoder and the second decoder, the task-dependent loss function and the reconstruction loss function can be used to guide the network model learning respectively.
[0053] When the input data is an annotated image from a public dataset, the first decoder can output, and the model loss corresponding to the shared encoder and the first decoder can be calculated based on the task-dependent loss function, and the model loss corresponding to the shared encoder and the first decoder can be back-propagated; at the same time, the second decoder can also output, and the model loss corresponding to the shared encoder and the second decoder can be calculated based on the reconstruction loss function, and the model loss corresponding to the shared encoder and the second decoder can be back-propagated.
[0054] When the input data is an unlabeled image in an actual business scenario, only the second decoder can output, and the first decoder does not output. At this time, the model loss corresponding to the shared encoder and the second decoder can be calculated, and the model loss corresponding to the shared encoder and the second decoder can be back-propagated. In this way, the shared encoder can learn the data distribution of the actual business scenario, and the first decoder can perform specific tasks. Furthermore, after the deep learning model converges, the accuracy and generalization of the model can be guaranteed in the inference stage, which improves the model degradation problem caused by inconsistent data distribution.
[0055] For the convenience of description, on the one hand, the loss function corresponding to the model architecture composed of the shared encoder and the first decoder can be called the first loss function, and correspondingly, the loss value calculated based on the first loss function can be called the first model loss. On the other hand, the loss function corresponding to the model architecture composed of the shared encoder and the second decoder can be called the second loss function, and correspondingly, the loss value calculated based on the second loss function can be called the second model loss.
[0056] In one embodiment, when the first output result is output through the first decoder, step S500 may include: calculating the current first model loss, and back-propagating the first model loss to the first decoder and the shared encoder.
[0057] In another embodiment, when the second output result is output through the second decoder, step S500 may include: calculating the current second model loss, and back-propagating the second model loss to the second decoder and the shared encoder.
[0058] In implementation, different types of task types may use different types of loss functions, so the first loss function may be any loss function selected according to a specific task. In addition, the second loss function may also be diversified, and may be any unsupervised loss function, such as the L1 loss function (or mean absolute error function, MAE), the L2 loss function (or least square error function, LSE), etc. This application does not limit the type of loss function.
[0059] In one embodiment, the second loss function may be an L2 loss function. The sum of squares of the difference between the output of the second decoder and the input data of the model may be minimized. The calculation formula of the L2 loss function may be as follows:
[0060]
[0061] Among them, x i represents input data, f represents the neural network, and n represents the total number of pixels.
[0062] It is worth mentioning that, in one embodiment, the input data x i It can be replaced by the result after some kind of image processing, such as the result after calculating the gradient.
[0063] In one embodiment, in step S500, before iteratively training the deep learning model according to the current model loss, the model training method may also include: determining whether the current model loss reaches a preset convergence condition; if the preset convergence condition is reached, ending the training of the deep learning model.
[0064] In implementation, the preset convergence condition may be that the first model loss and / or the second model loss reaches a preset loss value, or that the difference between the currently calculated first model loss (or second model loss) and the first model loss (or second model loss) calculated in the last iterative training is not higher than the preset difference. Of course, the preset convergence condition may also be any other termination iteration strategy, such as the number of iterative training reaching a preset number of training times, or the iterative training duration reaching a preset training duration, etc., and this application does not limit this.
[0065] Based on the above model training method, for the annotated data from the public database, both the first decoder and the second decoder will output, while for the unannotated data in the actual business scenario, only the newly added second decoder will output. In this way, the deep learning model of the present application can enable the encoder to learn the data distribution of the actual business scenario, and enable the first decoder to achieve specific tasks. Furthermore, the second decoder can be cancelled in the reasoning stage, and the required output data can be output through the first decoder, which improves the model degradation problem caused by inconsistent data distribution. In other words, the deep learning model provided by the present application has "seen" the data of the real use scenario during training, and uses the reconstruction loss function to guide the encoder to learn the data distribution of the real scenario, so as to reduce the reasoning performance degradation caused by the inconsistent distribution of training and reasoning data. At the same time, the scheme does not need to annotate the real data in the actual business scenario, and even in the face of pixel-by-pixel annotation tasks, it can avoid the model degradation problem caused by the inconsistency between the data distribution in the actual business scenario and the data distribution of the training set.
[0066] Based on the same technical concept, the embodiment of the present application also provides an image processing method, including:
[0067] Inputting the image to be processed into an encoder of a deep learning model to obtain features of the image to be processed;
[0068] Inputting the feature into a first decoder of the deep learning model to obtain an output result of the image to be processed;
[0069] Wherein, the deep learning model is trained according to the model training method described in any of the above embodiments.
[0070] Based on the same technical concept, the present application embodiment also provides a model training device. Figure 4 , the model training device comprises:
[0071] An input module, used to obtain a training data set, and input each training data in the training data set into the shared encoder; wherein the training data includes first labeled training data and second unlabeled training data;
[0072] A first output module, configured to output a first output result through the first decoder when the first training data is input to the shared encoder;
[0073] A second output module, configured to output a second output result through the second decoder when the first training data or the second training data is input to the shared encoder;
[0074] The training module is used to calculate the current model loss and iteratively train the deep learning model according to the current model loss.
[0075] Based on the same technical concept, the present application also provides an image processing device. Figure 5 , the image processing device comprises:
[0076] An input module inputs the image to be processed into an encoder of a deep learning model to obtain features of the image to be processed;
[0077] An output module, inputting the features of the image to be processed into a first decoder of the deep learning model to obtain an output result of the image to be processed;
[0078] Wherein, the deep learning model is trained according to the model training method described in any one of the above embodiments.
[0079] It should be noted that the model training device or image processing device provided in the above embodiment only uses the division of the above functional modules as an example when executing the corresponding method. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the model training device or image processing device provided in the above embodiment belongs to the same concept as the corresponding method embodiment, and its implementation principle and the technical effect to be achieved can be referred to the method embodiment, and this application will not repeat them here.
[0080] Based on the same technical concept, the present application embodiment also provides an electronic device. Figure 6 , the electronic device includes: a processor and a memory; wherein the memory stores a computer program, and the computer program is suitable for being loaded by the processor and executing the model training method or image processing method as described in any of the above embodiments. The implementation principle and technical effect to be achieved by the electronic device for model training or image processing can be referred to the method embodiment, and this application will not be repeated here.
[0081] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course can also be implemented by hardware. Based on this understanding, the above technical solution can essentially or contribute to the prior art in the form of a software product, and the software product of the model training method or image processing method can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including storing a number of instructions to enable an electronic device to execute the model training method or image processing method described in each embodiment or some parts of the embodiment.
[0082] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application should be included in the protection scope of the present application.
Claims
1. A model training method, characterized in that: Used to train a deep learning model, the deep learning model includes a shared encoder, a first decoder and a second decoder, and the model training method includes: Acquire a training data set, and input each training data in the training data set into the shared encoder; wherein the training data includes first labeled training data and second unlabeled training data; When the first training data is input to the shared encoder, a first output result is output through the first decoder, and a second output result is output through the second decoder; When the second training data is input to the shared encoder, the second output result is output only through the second decoder; The current model loss is calculated, and the deep learning model is iteratively trained according to the current model loss.
2. The model training method according to claim 1, characterized in that: The first training data are labeled training images from a public data set, and the second training data are unlabeled business images in actual business scenarios.
3. The model training method according to claim 1, characterized in that: When the first output result is output by the first decoder, The calculating the current model loss and iteratively training the deep learning model according to the current model loss includes: Calculate the current first model loss, and back-propagate the first model loss to the first decoder and the shared encoder.
4. The model training method according to claim 1, characterized in that: When the second output result is output by the second decoder, The calculating the current model loss and iteratively training the deep learning model according to the current model loss includes: Calculate the current second model loss, and back-propagate the second model loss to the second decoder and the shared encoder.
5. The model training method according to claim 3 or 4, characterized in that: Before iteratively training the deep learning model according to the current model loss, the method further includes: Determine whether the current model loss reaches a preset convergence condition; If the preset convergence condition is reached, the training of the deep learning model is terminated.
6. An image processing method, characterized in that: include: Inputting the image to be processed into an encoder of a deep learning model to obtain features of the image to be processed; Inputting the feature into a first decoder of the deep learning model to obtain an output result of the image to be processed; Wherein, the deep learning model is trained according to any one of the model training methods described in claims 1-7.
7. A model training device, characterized in that: The model training device comprises: An input module, used to obtain a training data set, and input each training data in the training data set into the shared encoder; wherein the training data includes first labeled training data and second unlabeled training data; A first output module, configured to output a first output result through the first decoder when the first training data is input to the shared encoder; A second output module, configured to output a second output result through the second decoder when the first training data or the second training data is input to the shared encoder; The training module is used to calculate the current model loss and iteratively train the deep learning model according to the current model loss.
8. An image processing device, characterized in that: The image processing device comprises: An input module inputs the image to be processed into an encoder of a deep learning model to obtain features of the image to be processed; An output module, inputting the features of the image to be processed into a first decoder of the deep learning model to obtain an output result of the image to be processed; Wherein, the deep learning model is trained according to any one of the model training methods described in claims 1-5.
9. An electronic device, characterized in that: The electronic device comprises: a processor and a memory; wherein the memory stores a computer program, and the computer program is suitable for being loaded by the processor and executing the method according to any one of claims 1-5 and 6.
10. A computer-readable storage medium, characterized in that: The computer storage medium stores a plurality of instructions, and the instructions are suitable for being loaded by a processor and executing the method according to any one of claims 1-5 and 6.