A migration learning method and device based on semantic differential diffusion
By constructing a semantic differential diffusion method in transfer learning, and utilizing the prior relationships and category dispersion loss function of the pre-trained model, the problem of ignoring the correlation information between upstream and downstream data is solved, which improves the accuracy and generalization ability of transfer learning, especially in the autonomous driving scenario.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- BEIHANG UNIV
- Filing Date
- 2022-08-01
- Publication Date
- 2026-05-15
AI Technical Summary
Existing transfer learning methods ignore the differential correlation information between upstream and downstream tasks when facing downstream tasks, resulting in poor transfer performance on datasets with finer granularity or greater differences in downstream tasks, and there is also the problem of excessive forgetting of pre-trained knowledge.
By constructing a transfer learning method based on semantic differential diffusion, a priori relations are extracted using a pre-trained model. A category dispersion loss function and a differential semantic dispersion loss function are constructed to guide the transfer learning process of the pre-trained model on downstream data, establish differential associations between upstream and downstream data, and explore association information at different semantic granularities to improve the model's representation and generalization capabilities.
It effectively solves the semantic collapse phenomenon in transfer learning, improves the transfer learning performance of pre-trained models, and significantly improves the accuracy in autonomous driving scenarios.
Smart Images
Figure CN115424087B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to a transfer learning method based on semantic differential diffusion, and also to a corresponding transfer learning device, belonging to the field of computer vision technology. Background Technology
[0002] In recent years, transfer learning methods have attracted widespread attention in autonomous driving scenarios, mainly because they can utilize pre-trained models trained with large datasets. These models can quickly adapt to different downstream tasks using only a small number of samples, relaxing the stringent requirements of large-scale data training. This allows for rapid generalization of existing knowledge to entirely new downstream tasks. The essence of autonomous driving is to use neural network models pre-trained with massive amounts of road data and apply them to entirely new downstream tasks (i.e., road scenarios never before traversed). During the processing of these downstream tasks, the neural network model is continuously fine-tuned using downstream data.
[0003] In the process of transfer learning, Girshick et al. found that fine-tuning the parameters of pre-trained networks can achieve better performance than fixing the pre-trained parameters. However, excessive fine-tuning of the pre-trained model can lead to the model excessively forgetting existing pre-trained knowledge. To address this issue, methods such as L2-SP and DELTA attempt to impose regularization constraints on the transfer learning process, thereby ensuring that the pre-trained model can retain the original knowledge to a certain extent. Specifically, the L2-SP method constrains the fine-tuned model parameters to maintain consistency with the pre-trained parameters as much as possible, ensuring that the pre-trained model does not excessively forget pre-trained knowledge. The DELTA method imposes a penalty on the feature dimension, constraining the consistency between the "behavior" of the fine-tuned network and the pre-trained network to ensure the network's generalization ability. However, while existing transfer learning methods have achieved good performance on some downstream tasks, they neglect to establish differentiated correlations between upstream and downstream tasks, resulting in poor transfer performance when faced with datasets with fine-grained downstream granularity or datasets with significant differences between upstream and downstream tasks.
[0004] Chinese invention patent ZL 202110068384.0 discloses an image processing method applicable to autonomous driving scenarios. Specifically, the method involves: acquiring a scene image and an image to be processed containing a target object; extracting target scene features from the scene image and target object features from the target object; fusing the target scene features and target object features to obtain a target scene transfer image, which is an image of the target object within the scene represented by the scene image. By fusing the scene features in the scene image with the object features of the target object contained in the image to be processed, the method achieves the replacement of the scene in the image to be processed while retaining the target object, thus obtaining an image of the target object in the new scene. This provides transfer learning samples for the detection model to be transferred to the new scene, reducing the cost of obtaining training samples in the new scene. Summary of the Invention
[0005] The primary technical problem to be solved by this invention is to provide a transfer learning method based on semantic differential diffusion.
[0006] Another technical problem to be solved by the present invention is to provide a transfer learning device based on semantic differential diffusion.
[0007] To achieve the above objectives, the present invention adopts the following technical solution:
[0008] According to a first aspect of the present invention, a transfer learning method based on semantic differential diffusion is provided, comprising the following steps:
[0009] S1: Use the pre-trained model to extract the prior relationships between the pre-trained categories;
[0010] S2: Construct a category dispersion loss function using the prior relationships between pre-trained categories to complete the pre-training of the pre-trained model;
[0011] S3: Using a pre-trained model, extract the average score code for each category in the downstream dataset;
[0012] S4: Based on the average score encoding of each category in the downstream dataset, the downstream categories are differentially distributed into different upstream category sequences;
[0013] S5: Construct a differentiated semantic dispersion loss function to guide the transfer learning process of the pre-trained model on downstream data.
[0014] Preferably, in step S1, for all categories of the pre-trained data, the average score of each sample in each category is predicted using the pre-trained model, and this is used as the prior relationship between upstream categories.
[0015] Preferably, the prior relation is used to obtain the set of similar categories corresponding to each upstream category i. Where i is a positive integer.
[0016] Preferably, in step S2, the categorical dispersion loss function is calculated using the following formula.
[0017]
[0018] in, This represents the set of similar categories corresponding to category i. Indicates the true class label of the sample, l i τ is the network prediction value, n is the introduced soft margin, and n is a positive integer. It is the Kronecker trigonometric function. When the input is not 0, it returns 1; when the input is 0, it returns 1.
[0019] Preferably, in step S3, the average score code of each category in the downstream dataset is extracted using the pre-trained model; then, the scores of each category in the downstream dataset are concatenated to construct an upstream-downstream correlation matrix.
[0020] Preferably, in step S4, the top-K most similar downstream categories are selected based on the average score encoding of each category m in the downstream dataset. Each downstream category is distributed into different upstream category sequences.
[0021] Preferably, in step S5, the differentiated semantic dispersion loss function Calculated using the following formula:
[0022]
[0023]
[0024] in, It is the score code extracted by the pre-trained classifier. It is a semantically dispersed label, and satisfies the following constraints:
[0025]
[0026] According to a second aspect of the present invention, a transfer learning apparatus based on semantic differential diffusion is provided, comprising a processor and a memory, wherein the processor reads a computer program in the memory for performing the following operations:
[0027] S1: Use the pre-trained model to extract the prior relationships between the pre-trained categories;
[0028] S2: Construct a category dispersion loss function using the prior relationships between pre-trained categories to complete the pre-training of the pre-trained model;
[0029] S3: Using a pre-trained model, extract the average score code for each category in the downstream dataset;
[0030] S4: Based on the average score encoding of each category in the downstream dataset, the downstream categories are differentially distributed into different upstream category sequences;
[0031] S5: Construct a differentiated semantic dispersion loss function to guide the transfer learning process of the pre-trained model on downstream data.
[0032] Preferably, the transfer learning method and transfer learning device are applied in autonomous driving scenarios; wherein, the score code of each sample in the training set of the vehicle recognition dataset is extracted to construct the upstream category sequence.
[0033] Compared with existing technologies, the transfer learning method and apparatus based on semantic differential diffusion provided by this invention addresses the common semantic collapse phenomenon in transfer learning by establishing differential associations between upstream and downstream data. It explores the guiding and constraining effects of association information at different semantic granularities on the hypothesis space of pre-trained models, effectively improving the representation and generalization capabilities of the pre-trained models. Experimental data show that this invention achieves superior transfer learning performance, with accuracy significantly higher than existing mature models, thus possessing significant practical value. Attached Figure Description
[0034] Figure 1 A flowchart illustrating the transfer learning method based on semantic differential diffusion provided in this embodiment of the invention;
[0035] Figure 2 This is a schematic diagram of the structure of a transfer learning device based on semantic differential diffusion provided in an embodiment of the present invention. Detailed Implementation
[0036] The technical content of the present invention will now be described in detail with reference to the accompanying drawings and specific embodiments.
[0037] Figure 1 This is an overall flowchart of the transfer learning method based on semantic differential diffusion in an embodiment of the present invention. Figure 1 As shown, this transfer learning method includes at least the following steps:
[0038] S1: Using a pre-trained model, extract the prior relationships between pre-trained categories.
[0039] In one embodiment of the present invention, for all categories of the pre-training data, the average score of each sample in each category is predicted using the pre-trained model, and this score is used as a priori relationship between upstream categories. Using this priori relationship, a set of similar categories corresponding to each upstream category (i.e., pre-training category) i is obtained. Where i is a positive integer.
[0040] Specifically, in this embodiment of the invention, ResNet50, the most commonly used deep learning network model in transfer learning (further details can be found at the following link: https: / / zhuanlan.zhihu.com / p / 353235794), is used as the classifier. For the selection of the pre-trained model, this embodiment selects ResNeXt (32×48d), which currently boasts outstanding performance, as the pre-trained model. To meet the actual needs of autonomous driving scenarios, in this embodiment, the upstream training data can be the ImageNet-1k image dataset commonly used in computer vision, and the downstream training data can be the Stanford Cars, FGVC Aircraft, Oxford Flowers102, Birdsnap, Oxford-IIIT Pets datasets widely used in autonomous driving research, as well as the DTD texture dataset and the SUN natural scene dataset.
[0041] S2: Construct a category dispersion loss function using the prior relationships between pre-trained categories to complete the pre-training of the pre-trained model.
[0042] During pre-training, by introducing prior relationships between pre-trained categories, the attention of the pre-trained model is broadened from focusing solely on the true label category to encompassing prior similarity categories to the true label category. Based on this, the category dispersion loss function is calculated using the following formula.
[0043]
[0044] in, This represents the set of similar categories corresponding to category i. Indicates the true class label of the sample, l i τ is the network prediction value, n is the introduced soft margin, and n is a positive integer. It is the Kronecker trigonometric function. When the input is not 0, it returns 1; when the input is 0, it returns 1.
[0045] Compared with the prior art, a significant feature of the present invention is the introduction of a soft margin τ, which relaxes the constraints of the pre-trained model on the similarity categories of the real label category, enabling the pre-trained model to adaptively learn the relationship between the real label category and its similar categories.
[0046] In category dispersion loss function Guided by this, the absolute values of the gradients for different categories are as follows:
[0047]
[0048] Where τ is the introduced soft margin, The set of similar categories corresponding to category i can be accurately obtained using various existing pre-trained models.
[0049] S3: Using a pre-trained model, extract the average score code for each category in the downstream dataset.
[0050] In one embodiment of the present invention, a pre-trained model is used to extract the average score code for each category in the downstream dataset. Where N is the number of upstream categories. Then, the scores of each category in the downstream dataset are concatenated to construct an upstream-downstream correlation matrix.
[0051] S4: Based on the average score encoding of each category in the downstream dataset, the downstream categories are differentially distributed into different upstream category sequences.
[0052] In one embodiment of the invention, encoding is based on the average score of each category m in the downstream dataset. Select the top-K most similar downstream categories. This is to encourage connections between downstream categories and pre-trained categories.
[0053] Furthermore, in order to better encourage downstream categories to pay attention to differentiated upstream category information, each downstream category is adaptively distributed to different upstream category sequences, thereby achieving the effect of basic semantic diffusion.
[0054] S5: Construct a differentiated semantic dispersion loss function to guide the transfer learning process of the pre-trained model on downstream data.
[0055] In one embodiment of the present invention, the differentiated semantic dispersion loss function Calculated using the following formula:
[0056]
[0057]
[0058] in, It is the score code extracted by the pre-trained classifier. It is a semantically dispersed label, and satisfies the following constraints:
[0059]
[0060] It can be seen that the above steps S1 to S5 solve the common semantic collapse phenomenon in transfer learning by establishing differentiated associations between upstream and downstream data, explore the guiding and constraining effects of association information with different semantic granularities on the hypothesis space of the pre-trained model, and effectively improve the representation ability and generalization ability of the pre-trained model.
[0061] As previously mentioned, transfer learning has attracted widespread attention in autonomous driving scenarios in recent years, mainly because it can utilize pre-trained models trained with large datasets. With only a small number of samples, these pre-trained models can quickly adapt to different downstream tasks, relaxing the stringent conditions of training with large-scale data. This allows for rapid generalization of existing knowledge to entirely new downstream tasks. The essence of autonomous driving is to use neural network models pre-trained with massive amounts of road data and apply them to entirely new downstream tasks (i.e., road scenarios never before traversed). During the processing of these downstream tasks, the pre-trained model is continuously fine-tuned using downstream data.
[0062] In one embodiment of the present invention, the pre-trained model optimized by the transfer learning method provided by the present invention was applied and verified in an autonomous driving scenario. The specific details are as follows:
[0063] In this embodiment of the invention, 196 types of vehicles were selected to construct a vehicle recognition dataset, which was divided into training and testing sets. For the upstream pre-training process, the SGD (Stochastic Gradient Descent) optimizer was selected, with a batch size of 1024 and a learning rate of 0.4, for a total of 100 generations of training. For the downstream dataset, the SGD optimizer was also used, with a batch size of 64 and a learning rate of 1e-3, for a total of 150 generations of training.
[0064] In this embodiment of the invention, a pre-trained model is first used to iteratively execute the transfer learning method described in steps S1 to S5 above, extracting the offline score encoding of each sample in the training set of the vehicle recognition dataset, which is then used to construct a differentiated upstream category sequence. Further, the SGD optimizer is trained on the vehicle recognition dataset with a batch size of 64 and a learning rate of 1e-3. After training, validation is performed on the test set. To ensure the accuracy of the validation results, the experiment was repeated three times under the same settings, and the mean and variance were calculated. The specific results are shown in Table 1.
[0065] Table 1
[0066] Model L2-SP BSS Co-Tuning This invention accuracy 90.78±0.06 90.73±0.02 90.99±0.22 91.62±0.19
[0067] As can be seen from Table 1, the present invention can achieve superior transfer learning performance, with an accuracy significantly higher than existing mature models, thus possessing good practical value.
[0068] Based on the above-mentioned transfer learning methods, this invention further provides a transfer learning device based on semantic differential diffusion. For example... Figure 2 As shown, the transfer learning apparatus includes one or more processors 21 and a memory 22. The memory 22 is coupled to the processors 21 and is used to store one or more programs. When the one or more programs are executed by the one or more processors 21, the one or more processors 21 implement the transfer learning method as described in the above embodiments.
[0069] The processor 21 controls the overall operation of the transfer learning device to complete all or part of the steps of the transfer learning method. The processor 21 can be a central processing unit (CPU), graphics processing unit (GPU), field-programmable gate array (FPGA), application-specific integrated circuit (ASIC), digital signal processing (DSP) chip, etc. The memory 22 stores various types of data to support the operation of the transfer learning device. This data may include, for example, instructions for any application or method operating on the transfer learning device, and application-related data. The memory 22 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, etc.
[0070] Compared with existing technologies, the transfer learning method and apparatus based on semantic differential diffusion provided by this invention addresses the common semantic collapse phenomenon in transfer learning by establishing differential associations between upstream and downstream data. It explores the guiding and constraining effects of association information at different semantic granularities on the hypothesis space of pre-trained models, effectively improving the representation and generalization capabilities of the pre-trained models. Experimental data show that this invention achieves superior transfer learning performance, with accuracy significantly higher than existing mature models, thus possessing significant practical value.
[0071] The foregoing has provided a detailed description of the transfer learning method and apparatus based on semantic differential diffusion provided by this invention. Any obvious modifications made by those skilled in the art without departing from the essential content of this invention will constitute an infringement of the patent rights of this invention and will incur corresponding legal liability.
Claims
1. A transfer learning method based on semantic differential diffusion, applied in autonomous driving scenarios; wherein, The upstream training data is selected from image datasets in the field of computer vision, and the downstream training data is selected from datasets, texture datasets, and natural scene datasets in autonomous driving research. The method is characterized by the following steps: S1: Use the pre-trained model to extract the prior relationships between the pre-trained categories; S2: Construct a class dispersion loss function using the prior relationships between pre-trained classes to complete the pre-training of the pre-trained model; the class dispersion loss function is calculated using the following formula. : in, This represents the set of similar categories corresponding to category i. Indicates the true class label of the sample. It is a network prediction value. This is the introduced soft margin, where n is a positive integer. It is the Kronecker trigonometric function. When the input is not 0, it returns 0; when the input is 0, it returns 1. S3: Use a pre-trained model to extract the average score code for each category in the downstream dataset; concatenate the scores of each category in the downstream dataset to construct an upstream-downstream correlation matrix; S4: Extract the score code of each sample in the training set of the vehicle recognition dataset to construct the upstream category sequence; based on the average score code of each category m in the downstream dataset, select the top-K most similar downstream categories. Each downstream category is distributed into different upstream category sequences; S5: Construct a differentiated semantic dispersion loss function to guide the transfer learning process of the pre-trained model on downstream data; the differentiated semantic dispersion loss function Calculated using the following formula: in, It is the score code extracted by the pre-trained classifier. It is a semantically dispersed label, and satisfies the following constraints: 。 2. The transfer learning method as described in claim 1, characterized in that: In step S1, for all categories of the pre-trained data, the average score of each sample in each category is predicted using the pre-trained model, and this is used as the prior relationship between upstream categories.
3. The transfer learning method as described in claim 2, characterized in that: Using the prior relation, obtain the set of similar categories corresponding to each upstream category i. , where i is a positive integer.
4. A transfer learning device based on semantic differential diffusion, applied in autonomous driving scenarios, characterized in that... It includes a processor and a memory, wherein the processor reads a computer program from the memory for executing the transfer learning method according to any one of claims 1 to 3.