A transfer learning method and device based on domain pair association

Abstract

The present invention provides a method and device for transfer learning based on domain pair association, which includes: upon receiving a target domain sample image and a category label corresponding to the target domain sample image, obtaining a source domain sample image; constructing a feature relationship graph between the target domain sample image and the source domain sample image to obtain a feature-similar image pair; inputting the feature-similar image pair into a pre-trained continuous domain adaptation model, using the category label corresponding to the target domain sample image as supervision information, and performing transfer learning on the continuous domain adaptation model with a preset learning objective based on a feature mutual correlation matrix as an optimization objective; inputting the target domain target image into the continuous domain adaptation model that has completed transfer learning to obtain a prediction result. The present invention achieves comparative pairing, and incrementally explores the local correlation of the new source domain on the basis of consolidating the model's prior relationships, thereby avoiding the occurrence of transfer drift and improving the accuracy of the prediction results.

Description

Technical Field

[0001] The present invention relates to the field of transfer learning technology, and in particular to a transfer learning method and device based on domain pair association. Background Art

[0002] Unsupervised domain adaptation has been widely explored to alleviate domain shift between a labeled source domain and an unlabeled target domain. It specifically transfers domain knowledge from a single source domain (such as single-source domain adaptation) or multiple domains (such as multi-source domain adaptation) to a target domain.

[0003] However, existing unsupervised domain adaptation research traditionally assumes that all data from the source domain is pre-collected and used to train the unsupervised domain adaptation model. This inevitably fails to cope with more practical situations: data from different source domains is collected sequentially, or due to security issues and data privacy, adapting the model to the target domain using all available source data is difficult in reality. Recently, some works have attempted to shift from continuous source domains. However, the problems raised in these works cannot be extended from standard unsupervised domain adaptation and lack comparability.

[0004] Most current unsupervised domain adaptation methods ignore local correlations between domains. Some samples in the target domain are more similar to parts of source domain 1, while other samples are more similar to source domain 2. When the local correlations between the historical source and target domains are not preserved, migration drift can occur, leading to inaccurate predictions.

[0005] Therefore, the existing technology has defects and needs to be improved and developed. Summary of the Invention

[0006] The technical problem to be solved by the present invention is to provide a transfer learning method and device based on domain pair association in response to the above-mentioned defects of the existing technology, aiming to solve the problem in the existing technology that when the local correlation between the historical source domain and the target domain is not preserved, migration drift may occur, thereby leading to inaccurate prediction results.

[0007] The technical solutions adopted by the present invention to solve the technical problems are as follows:

[0008] A transfer learning method based on domain pair association, the method comprising:

[0009] When receiving a target domain sample image and a category label corresponding to the target domain sample image, obtaining a source domain sample image;

[0010] Constructing a feature relationship graph between the target domain sample image and the source domain sample image, and obtaining feature-similar image pairs according to the feature relationship graph;

[0011] Inputting the feature-similar image pairs into a pre-trained continuous domain adaptation model, using the category labels corresponding to the target domain sample images as supervision information, and performing transfer learning on the continuous domain adaptation model with a preset learning objective based on a feature mutual correlation matrix as an optimization objective;

[0012] When a target domain target image is received, the target domain target image is input into the continuous domain adaptation model that has completed transfer learning to obtain a prediction result.

[0013] In one implementation, the training step of the continuous domain adaptation model includes:

[0014] Obtaining a source domain sample image and a training category label corresponding to the source domain sample image, and establishing a relationship candidate library based on the source domain sample image and the corresponding training category label;

[0015] Constructing divergent contrast image pairs based on source domain sample images with different semantic representations in the relationship candidate library;

[0016] Setting the objective function and consolidation function based on the cross-correlation matrix;

[0017] Inputting the divergent contrast image pair into an initial continuous domain adaptation model, using the training category label as supervision information, and taking the objective function and the consolidation function as optimization targets to train the initial continuous domain adaptation model;

[0018] After the training is completed, the trained continuous domain adaptation model is obtained.

[0019] In one implementation, before obtaining a source domain sample image and a training category label corresponding to the source domain sample image and establishing a relationship candidate library based on the source domain sample image and the corresponding training category label, the method further includes:

[0020] Obtaining an original sample image and an original category label corresponding to the original sample image;

[0021] Inputting the original sample image into the initial continuous domain adaptation model, and calculating the confidence according to the corresponding original category label;

[0022] If the confidence of the original sample image is greater than a preset confidence threshold, the original sample image is used as a source domain sample image, and the original category label of the original sample image is used as a training category label.

[0023] In one implementation, the objective function is:

[0024] Among them, the C iirepresents the cross-correlation matrix calculated based on the features of the two parameter-sharing network outputs for the divergent contrast image pair, represents a divergent contrast image pair, the Representing an image The training category label, Representing an image The training class labels of .

[0025] In one implementation, the calculation formula of the consolidation function is:

[0026]

[0027]

[0028] Wherein, x represents the input image, and ∑ b z b,i z' b,j Represents the sum of the feature vector products of the two sample images in the current batch. Represents the sum of the lengths of a certain dimension of the feature vector of an image in the image pair, Represents the sum of the lengths of a certain dimension of the feature vector of another sample image, represents a fully connected projection module independent of the classification task, Represents the feature encoder of the network, b represents the serial number of the small batch sample image, i and j represent the feature dimension index of the network output, and C is a square matrix with a size of the number of network output features.

[0029] In one implementation, constructing a feature relationship graph between the target domain sample image and the source domain sample image includes:

[0030] Pre-built hierarchical navigable small world;

[0031] The hierarchical navigable small world is used to construct a feature relationship graph between the target domain sample image and the source domain sample image.

[0032] In one implementation, the pre-construction of a hierarchical navigable small world includes:

[0033] Providing a hierarchical navigable small world, and importing the feature vector of the source domain sample image into the hierarchical navigable small world;

[0034] Establishing a hierarchical retrieval structure in the hierarchical navigable small world, wherein the retrieval structure has a proximity graph of hierarchical nested subsets of source elements, and randomly selecting a minimum layer where an element is located with an exponentially decaying probability distribution;

[0035] The construction of the hierarchical navigable small world is completed.

[0036] In one implementation, obtaining feature-similar image pairs according to the feature relationship graph includes:

[0037] Calculating the feature distance between the target domain sample image and the source domain sample image in the hierarchical navigable small world;

[0038] Calculating a sampling probability value according to the characteristic distance, wherein the sampling probability value is negatively correlated with the characteristic distance;

[0039] A feature-similar image pair is obtained according to the sampling probability value.

[0040] In one implementation, the calculation formula for the sampling probability value is:

[0041]

[0042]

[0043] Among them, the Represents the relationship candidate sample image x T and source domain sample images Feature distance in a hierarchical navigable small world, the Indicates the sampling basis value, the Indicates the maximum distance among the nearest neighbor elements. Represents the sampling probability value.

[0044] In one implementation, the preset learning goal is:

[0045]

[0046]

[0047] State p(x S |x T ) represents the sampling probability value, and λ1 represents L cpd The proportion of the total learning goal L The learning goal set by the cross-correlation matrix is represents the objective function set based on the cross-correlation matrix.

[0048] The present invention also provides a transfer learning device based on domain pair association, the device comprising:

[0049] an acquisition module, configured to acquire a source domain sample image upon receiving a target domain sample image and a category label corresponding to the target domain sample image;

[0050] A construction module, configured to construct a feature relationship graph between the target domain sample image and the source domain sample image, and obtain feature-similar image pairs according to the feature relationship graph;

[0051] a transfer module, configured to input the feature-similar image pairs into a pre-trained continuous domain adaptation model, use the category labels corresponding to the target domain sample images as supervision information, and perform transfer learning on the continuous domain adaptation model with a preset learning objective based on a feature mutual correlation matrix as an optimization objective;

[0052] The prediction module is used to input the target domain target image into the continuous domain adaptation model that has completed transfer learning when receiving the target domain target image to obtain a prediction result.

[0053] The present invention also provides a terminal, comprising: a memory, a processor, and a domain-pair-association-based transfer learning program stored in the memory and executable on the processor, wherein the domain-pair-association-based transfer learning program, when executed by the processor, implements the steps of the domain-pair-association-based transfer learning method described above.

[0054] The present invention also provides a computer-readable storage medium storing a computer program, wherein the computer program can be executed to implement the steps of the transfer learning method based on domain pair association as described above.

[0055] Beneficial effects of the present invention: When a target domain sample image and a category label corresponding to the target domain sample image are received, the embodiment of the present invention obtains a source domain sample image; constructs a feature relationship graph between the target domain sample image and the source domain sample image, and obtains a feature similarity image pair based on the feature relationship graph; inputs the feature similarity image pair into a pre-trained continuous domain adaptive model, uses the category label corresponding to the target domain sample image as supervision information, and performs transfer learning on the continuous domain adaptive model with a preset learning target based on a feature mutual correlation matrix as an optimization target; when a target domain target image is received, the target domain target image is input into the continuous domain adaptive model that has completed transfer learning to obtain a prediction result. The present invention achieves comparative pairing by constructing a feature similarity image pair and inputting the feature similarity image pair into a pre-trained continuous domain adaptive model, and uses the preset learning target based on the feature mutual correlation matrix as an optimization target, incrementally exploring the local correlation of the new source domain on the basis of consolidating the model's prior relationship, avoiding the occurrence of migration drift, and improving the accuracy of the prediction result. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1It is a flowchart of a preferred embodiment of the transfer learning method based on domain pair association in the present invention.

[0057] Figure 2 This is a schematic diagram of sample image filtering based on category prototypes in the present invention.

[0058] Figure 3 It is the target optimization of semantic difference sample pairs in this invention.

[0059] Figure 4 This is the process of building the feature relationship graph in the present invention.

[0060] Figure 5 It is a schematic diagram of target optimization of feature-similar sample pairs in the present invention.

[0061] Figure 6 It is a functional principle block diagram of a preferred embodiment of the domain pair association-based transfer learning device in the present invention.

[0062] Figure 7 It is a functional principle block diagram of the terminal in the present invention. DETAILED DESCRIPTION

[0063] In order to make the purpose, technical solutions and advantages of the present invention more clear and distinct, the present invention is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0064] Unsupervised domain adaptation algorithms have become a research hotspot for deep transfer learning algorithms. This type of algorithm requires migrating a labeled source domain to an unlabeled target domain and solving the challenge of domain shift. At present, unsupervised domain adaptation algorithms can be divided into single-source domain adaptation and multi-source domain adaptation algorithms according to the number of source domains. These methods are all trained in a closed set of source and target domains. In most practical applications, deep transfer models are often required to have the ability to learn continuously, but existing methods exhibit adaptive drift when incrementally learning the source domain. The present invention targets the phenomenon of local correlation between domains. Based on the development of source domain correlation theory and contrastive learning research, the present invention proposes representation learning domain pairs based on features and semantic associations and relational transfer targets. On the basis of consolidating the model's prior relationships, the local correlation of the new source domain is incrementally explored, avoiding the problem of domain negative transfer caused by distribution migration.

[0065] This paper considers a more practical continuous domain adaptation problem: training a model to adapt to an unlabeled target domain from a newly emerged source domain. The continuous domain adaptation model sequentially receives different labeled source domains and is increasingly trained on a single source domain. For example, healthcare resources are unevenly distributed around the world. This paper can easily collect labeled domains from resource-rich regions and continuously train them to adapt to unlabeled target domains in resource-poor medical facilities.

[0066] In the task of continuous source domain adaptation, existing unsupervised domain adaptation suffers from significant adaptive drift due to the network's sensitivity to the current source domain. Through quantitative experimental analysis, this paper finds that methods that achieve significant performance in single-source and multi-source domain adaptation have limited success in the task of continuous source domain adaptation. Because the target domain is unlabeled, the supervision information is primarily determined by the source domain label, and the domain difference distance is determined by the feature space dominated by the source domain, which can lead to overfitting of the source domain.

[0067] Most current unsupervised domain adaptation methods ignore local correlations between domains. Some samples in the target domain are more similar to parts of source domain 1, but other samples are more similar to source domain 2. When the local correlations between the historical source domain and the target domain are not preserved, migration drift may occur. Based on such local correlations, this paper proposes domain comparison pairs that contain semantic and feature connections between samples, and proposes an adaptive consolidation and exploration network based on the domain pair associations.

[0068] See Figure 1 In an embodiment of the present invention, the domain pair association-based transfer learning method includes the following steps:

[0069] Step S100: When a target domain sample image and a category label corresponding to the target domain sample image are received, a source domain sample image is obtained.

[0070] Specifically, before using a pre-trained continuous domain adaptation model, users need to input sample images from the target domain. If the target domain sample images do not belong to the same category as the source domain sample images, they also need to input the category label corresponding to the target domain sample images. This allows transfer learning to proceed without acquiring training data for the target domain. The number of target domain sample images is small, thus reducing the user's workload.

[0071] In one implementation, the present invention requires pre-training of a continuous domain adaptation model, and the training steps of the continuous domain adaptation model include:

[0072] Step S10: obtaining a source domain sample image and a training category label corresponding to the source domain sample image, and establishing a relationship candidate library based on the source domain sample image and the corresponding training category label;

[0073] Step S20: constructing divergent contrast image pairs based on source domain sample images with different semantic representations in the relationship candidate library;

[0074] Step S30: setting the objective function and the consolidation function based on the cross-correlation matrix;

[0075] Step S40: inputting the divergent contrast image pair into an initial continuous domain adaptive model, using the training category label as supervision information, and training the initial continuous domain adaptive model with the objective function and the consolidation function as optimization targets;

[0076] Step S50: After the training is completed, a trained continuous domain adaptation model is obtained.

[0077] Specifically, the main challenge of continuous domain adaptation is how to retain reliable discriminative features in the current network to prevent negative transfer due to lack of consolidation when exploring new areas. This paper proposes a method based on the divergent contrast image pairing of samples with different semantic representations in the relation candidate library. d ,like The training category labels of the divergent contrast image pairs satisfy

[0078] In one embodiment, before step S10, the method further includes:

[0079] Obtaining an original sample image and an original category label corresponding to the original sample image;

[0080] Inputting the original sample image into the initial continuous domain adaptation model, and calculating the confidence according to the corresponding original category label;

[0081] If the confidence of the original sample image is greater than a preset confidence threshold, the original sample image is used as a source domain sample image, and the original category label of the original sample image is used as a training category label.

[0082] Specifically, unlike object-based unsupervised representation learning in self-supervised learning, the training and prediction process is based on the semantics and feature associations between samples, and continuously learns feature representations that are generalizable to the target domain. Improving the model's prediction accuracy for the target domain is the ultimate goal of continuous domain adaptation, so sample images are also the core of all sample relationships. However, due to the lack of manual labels, the model's feature representation of sample images will inevitably contain noise, and the sample relationships established based on noise will also introduce errors. To address this problem, the present invention proposes a filtered relationship candidate library based on target samples to provide a basis for the subsequent construction of differentiated sample relationships, so as to learn robust representations and achieve adaptive incremental learning.

[0083] First, the present invention periodically uses a high confidence threshold τ to filter sample images that the model cannot clearly judge. Considering the deviation of the model, the present invention further proposes to use the relationship between sample images to filter the related sample images. Figure 2 As shown, the present invention calculates the semantic prototype of each category based on the samples after high confidence filtering, and then selects multiple samples with the closest distance to the prototype space to add to the final relationship candidate library, and uses the corresponding prototype category as the pseudo label of the sample, that is, the training category label. In other words, the source domain sample images in the relationship candidate library are classified, and the relationships between different categories are also marked. The relationship candidate library of the present invention is based on the filtering of target samples, providing a basis for the subsequent construction of differentiated sample relationships, so as to learn robust representations and achieve adaptive incremental learning.

[0084] In one embodiment, the objective function is Among them, the C ii represents the cross-correlation matrix calculated based on the features of the two parameter-sharing network outputs for the divergent contrast image pair, represents a divergent contrast image pair, the Representing an image The training category label, Representing an image The training class labels of .

[0085] Specifically, the present invention adopts a supervised approach, focusing on the feature differences of divergent contrast image pairs as the core of the transfer network consolidation. Rather than using different data augmentation methods for the same sample, the objective function of the present invention is based on the posterior semantic differences of different samples, making the embedding representations as different as possible, thereby preserving the reliable posterior information obtained by the historical model and making the prediction results of the trained model more accurate.

[0086] In one embodiment, the calculation formula of the consolidation function is:

[0087]

[0088]

[0089] Wherein, x represents the input image, and ∑ b z b,i z' b,j Represents the sum of the feature vector products of the two sample images in the current batch. Represents the sum of the lengths of a certain dimension of the feature vector of an image in the image pair, Represents the sum of the lengths of a certain dimension of the feature vector of another sample image, represents a fully connected projection module independent of the classification task, Represents the feature encoder of the network, b represents the serial number of the small batch sample image, i and j represent the feature dimension index of the network output, and C is a square matrix with a size of the number of network output features.

[0090] Specifically, the value of C ranges from -1 (completely uncorrelated) to 1 (completely correlated). By trying to make the diagonal elements of the cross-correlation matrix equal to zero, the embedding vectors of different semantics are made to be orthogonal, as shown in the figure. d The differential feature representation plays a crucial role in separating domain-specific attributes from feature distribution and rebalancing the target distribution. d The samples belong to the target domain, so the orthogonalization operation on the features helps to eliminate some domain-specific information, such as background, color, style, etc., thereby better assisting the migration from the source domain to the target domain. At the same time, since a fixed threshold is used to filter the relationship candidate samples, the distribution of the candidate may deviate from the overall target domain, so CP d The category distribution of the mini-batch is also properly corrected to make it closer to the overall distribution of the target domain.

[0091] like Figure 3 As shown, in the relationship consolidation stage (ie, the training stage), the present invention adopts a supervised approach and takes the feature differences of divergent pairs as the core of the migration network consolidation.

[0092] like Figure 1 As shown, the domain pair association-based transfer learning method further includes the following steps:

[0093] Step S200: construct a feature relationship graph between the target domain sample image and the source domain sample image, and obtain feature-similar image pairs according to the feature relationship graph.

[0094] Specifically, the transfer learning of the present invention aims to learn new information from the source domain to improve the prediction accuracy of the model for the target domain target image. The latest domain adaptation work has found that not all knowledge can be transferred across domains, and indiscriminate transfer may be detrimental to the generalization of features, so it is necessary to pay attention to the correlation of cross-domain samples. Based on the fact that the target samples are more dependent on some samples in the source domain rather than all samples, the present invention proposes a feature similarity image pair CP based on implicit spatial connections. c To describe the partial connection between the source domain and the target domain, it is used to obtain the cross-domain unique features of the class and strengthen the domain invariance of the semantic features. c It consists of target domain sample images and source domain samples similar to them. c There are domain differences and feature similarities between each pair of sample images.

[0095] The present invention constructs contrasting pairings, effectively alleviating the adaptive drift problem of existing continuous source domain adaptive methods, with a lighter structure and higher computational efficiency.

[0096] In one embodiment, the step S200 of “constructing a feature relationship graph between the target domain sample image and the source domain sample image” specifically includes:

[0097] Step S210: pre-constructing a hierarchical navigable small world;

[0098] Step S220: construct a feature relationship graph between the target domain sample image and the source domain sample image using the hierarchical navigable small world.

[0099] Specifically, in order to facilitate the construction of CP c ,This invention introduces a hierarchical navigable small world to draw a complete ,feature relationship graph.

[0100] like Figure 4 As shown in the figure, dark black nodes represent nodes to be inserted, and light black nodes represent already inserted nodes. The specific mapping process includes: (1) first input the maximum number of connections and the maximum number of nodes to be indexed into the hierarchical navigable small world; (2) the target domain sample images are sequentially input into the hierarchical navigable small world to establish indexes; (3) the input target domain sample images are extracted based on the feature extraction network; (4) the features are regarded as nodes, and the node level L is obtained; (5) the nodes are inserted, and no operation is performed on the first inserted node; (6) the first step is to find the node ep closest to the node x to be inserted in the L layer, and use it as the input of the next layer; the second step is that this layer and the following layers are the insertion layers of the element to be inserted, starting from ep, search for the ef nodes closest to the element to be inserted, select M nodes from the ef nodes to be connected to the node to be inserted, and use these M nodes as the input of the next layer; the third step is to start searching from the M nodes, find the ef nodes closest to the node to be inserted, and select M nodes to be connected to the element to be inserted x; the fourth step is the same as the third step. (7) The first step is to find the node ep closest to the node to be inserted x in the L layer and use it as the input of the next layer; the second step is to find the node ep closest to the node to be inserted in the L=2 layer and use it as the input of the next layer; the third step is that this layer and the following layers are the insertion layers of the element to be inserted, starting from ep, find the ef nodes closest to the element to be inserted, select M nodes from the ef nodes to connect to the node to be inserted, and use these M nodes as the input of the next layer; the fourth step is the same as the third step.

[0101] In one embodiment, step S210 specifically includes:

[0102] Step S211: providing a hierarchical navigable small world, and importing the feature vector of the source domain sample image into the hierarchical navigable small world;

[0103] Step S212: establishing a hierarchical retrieval structure in the hierarchical navigable small world, wherein the retrieval structure has a proximity graph of hierarchical nested subsets of source elements, and randomly selecting a minimum layer where an element is located using an exponentially decaying probability distribution;

[0104] Step S213: Complete the construction of the hierarchical navigable small world.

[0105] Specifically, during the construction phase, the hierarchical navigable small world imports all feature vectors of the source domain. It then gradually builds a hierarchical retrieval structure consisting of a hierarchy of proximity graphs of nested subsets of source elements, and randomly selects the minimum level where an element resides using an exponentially decaying probability distribution.

[0106] In one embodiment, the step S200 of "obtaining feature-similar image pairs according to the feature relationship graph" specifically includes:

[0107] Step S230: Calculate the feature distance between the target domain sample image and the source domain sample image in the hierarchical navigable small world;

[0108] Step S240: Calculate a sampling probability value according to the characteristic distance, wherein the sampling probability value is negatively correlated with the characteristic distance;

[0109] Step S250: Obtain feature-similar image pairs according to the sampling probability values.

[0110] Specifically, such as Figure 5 As shown, in the retrieval phase, a sample is randomly sampled from the candidate samples of the relationship, and the corresponding feature vector is obtained based on the prediction of the model. The search starts from the top level of the hierarchical navigable small world, and the scale separation is used to improve the performance, allowing logarithmic complexity expansion. The present invention queries the hierarchical navigable small world for the top K points that are most similar to each target domain sample image. Then, the present invention extracts a point based on the probability of the target domain sample image to form the CP of this round. c The probability of sampling is determined by the distance in the feature relationship graph.

[0111] In one embodiment, the calculation formula of the sampling probability value is:

[0112]

[0113]

[0114] Among them, the Represents the target domain sample image x T and source domain sample images Feature distance in a hierarchical navigable small world, the Indicates the sampling basis value, the Indicates the maximum distance among the nearest neighbor elements. Represents the sampling probability value.

[0115] Specifically, the present invention uses the maximum distance between the nearest neighbor elements minus the distance between the candidate sample and the element to obtain the sampling basis value The present invention obtains the final sampling probability value after the Softmax operation, and according to the sampling probability, the present invention obtains CP c , expressed as <x T , x s >.

[0116] like Figure 1 As shown, the domain pair association-based transfer learning method further includes the following steps:

[0117] Step S300: Input the feature-similar image pair into a pre-trained continuous domain adaptation model, use the category label corresponding to the target domain sample image as supervision information, and use the preset learning objective based on the feature mutual correlation matrix as the optimization objective to perform transfer learning on the continuous domain adaptation model.

[0118] In one embodiment, the preset learning objectives are:

[0119]

[0120]

[0121] Among them, the Represents the cross entropy loss, the (x T ,x S ) represents a pair of feature-similar images, and the p(x S |x T ) represents the sampling probability value, and λ1 represents L cpd The proportion of L in the entire learning goal, λ2 represents L cpc The proportion of the entire learning goal L, Represents the learning goal set based on the feature cross-correlation matrix, represents the objective function set based on the cross-correlation matrix.

[0122] and different, The goal is to make the embedding have no effect on the domain transfer by trying to make the diagonal elements of the cross-correlation matrix equivalent to 1, resulting in a more general feature space. Finally, the present invention further simply uses To calculate the cross entropy loss is

[0123] like Figure 1 As shown, the domain pair association-based transfer learning method further includes the following steps:

[0124] Step S400: When a target domain target image is received, the target domain target image is input into the continuous domain adaptation model that has completed transfer learning to obtain a prediction result.

[0125] The target domain target image and the target sample image belong to the same target domain. When a user needs a model for a certain category, if the category does not exist in the source domain, they only need to input a small number of target sample images and corresponding labels into the pre-trained model to perform transfer learning. After the transfer learning is completed, the user only needs to input the target domain target image to be predicted to obtain the prediction result. The relationship exploration phase (i.e., the prediction phase) of the present invention can obtain cross-domain unique features of the class and strengthen the domain invariance of semantic features.

[0126] The present invention designs a consolidation and exploration network to solve the domain drift problem in continuous source domain adaptation. In order to give the model incremental transferability, the consolidation and exploration network uses the prior knowledge of the model to construct contrasting pairings for memory consolidation and adaptive exploration, respectively. Based on the differential connections of these samples, the present invention designs a representation learning objective based on the cross-correlation matrix, which can obtain further domain adaptability in the current source domain while retaining reliable priors in the model. The consolidation and exploration network effectively alleviates the adaptive drift problem of existing continuous source domain adaptation methods, with a lighter structure and higher computational efficiency. Aiming at the phenomenon of local correlation between domains, the present invention proposes representation learning domain pairs based on features and semantic associations and relational transfer objectives based on the development of source domain correlation theory and contrastive learning research, and conducts incremental exploration of the local correlation of the new source domain on the basis of consolidating the prior relationship of the model, thereby avoiding the problem of domain negative transfer caused by distribution migration.

[0127] In this way, the present invention solves the domain drift problem in continuous source domain adaptive migration and endows the model with incremental transferability; the representation learning objective based on the cross-correlation matrix can obtain further domain adaptability in the current source domain while retaining the reliable priors in the model.

[0128] In one embodiment, if Figure 6 As shown, based on the above-mentioned domain pair association-based transfer learning method, the present invention also provides a domain pair association-based transfer learning device, including:

[0129] An acquisition module 100 is configured to acquire a source domain sample image upon receiving a target domain sample image and a category label corresponding to the target domain sample image;

[0130] A construction module 200 is configured to construct a feature relationship graph between the target domain sample image and the source domain sample image, and obtain feature-similar image pairs based on the feature relationship graph;

[0131] A transfer module 300 is configured to input the feature-similar image pairs into a pre-trained continuous domain adaptation model, use the category labels corresponding to the target domain sample images as supervision information, and perform transfer learning on the continuous domain adaptation model using a preset learning objective based on a feature mutual correlation matrix as an optimization objective;

[0132] The prediction module 400 is configured to, when receiving a target domain target image, input the target domain target image into the continuous domain adaptation model that has completed transfer learning to obtain a prediction result.

[0133] In one embodiment, if Figure 7 As shown, based on the above-mentioned domain pair association-based transfer learning method, the present invention also provides a terminal, including a processor 10 and a memory 20. Figure 7 Only some of the components of the terminal are shown, but it should be understood that implementation of all of the shown components is not required, and more or fewer components may be implemented instead.

[0134] In some embodiments, the memory 20 may be an internal storage unit of the terminal, such as a hard disk or memory of the terminal. In other embodiments, the memory 20 may also be an external storage device of the terminal, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the terminal. Furthermore, the memory 20 may also include both an internal storage unit of the terminal and an external storage device. The memory 20 is used to store application software and various types of data installed on the terminal, such as program codes for installing the terminal. The memory 20 may also be used to temporarily store data that has been output or is to be output. In one embodiment, a domain-pair association-based transfer learning program 30 is stored on the memory 20, and the domain-pair association-based transfer learning program 30 can be executed by the processor 10, thereby realizing the domain-pair association-based transfer learning method in the present application.

[0135] In some embodiments, the processor 10 may be a central processing unit (CPU), a microprocessor, or other data processing chip, configured to execute program codes or process data stored in the memory 20, such as executing the domain pair association-based transfer learning method.

[0136] In one embodiment, when the processor 10 executes the domain pair association-based transfer learning program 30 in the memory 20, the following steps are implemented:

[0137] When receiving a target domain sample image and a category label corresponding to the target domain sample image, obtaining a source domain sample image;

[0138] Constructing a feature relationship graph between the target domain sample image and the source domain sample image, and obtaining feature-similar image pairs according to the feature relationship graph;

[0139] Inputting the feature-similar image pairs into a pre-trained continuous domain adaptation model, using the category labels corresponding to the target domain sample images as supervision information, and performing transfer learning on the continuous domain adaptation model with a preset learning objective based on a feature mutual correlation matrix as an optimization objective;

[0140] When a target domain target image is received, the target domain target image is input into the continuous domain adaptation model that has completed transfer learning to obtain a prediction result.

[0141] The training steps of the continuous domain adaptation model include:

[0142] Obtaining a source domain sample image and a training category label corresponding to the source domain sample image, and establishing a relationship candidate library based on the source domain sample image and the corresponding training category label;

[0143] Constructing divergent contrast image pairs based on source domain sample images with different semantic representations in the relationship candidate library;

[0144] Setting the objective function and consolidation function based on the cross-correlation matrix;

[0145] Inputting the divergent contrast image pair into an initial continuous domain adaptation model, using the training category label as supervision information, and taking the objective function and the consolidation function as optimization targets to train the initial continuous domain adaptation model;

[0146] After the training is completed, the trained continuous domain adaptation model is obtained.

[0147] Before obtaining a source domain sample image and a training category label corresponding to the source domain sample image and establishing a relationship candidate library based on the source domain sample image and the corresponding training category label, the method further includes:

[0148] Obtaining an original sample image and an original category label corresponding to the original sample image;

[0149] Inputting the original sample image into the initial continuous domain adaptation model, and calculating the confidence according to the corresponding original category label;

[0150] If the confidence of the original sample image is greater than a preset confidence threshold, the original sample image is used as a source domain sample image, and the original category label of the original sample image is used as a training category label.

[0151] The objective function is:

[0152] Among them, the C ii represents the cross-correlation matrix calculated based on the features of the two parameter-sharing network outputs for the divergent contrast image pair, represents a divergent contrast image pair, the Representing an image The training category label, Representing an image The training class labels of .

[0153] The calculation formula of the consolidation function is:

[0154]

[0155]

[0156] Wherein, x represents the input image, and ∑ b z b,i z' b,j Represents the sum of the feature vector products of the two sample images in the current batch. Represents the sum of the lengths of a certain dimension of the feature vector of an image in the image pair, Represents the sum of the lengths of a certain dimension of the feature vector of another sample image, represents a fully connected projection module independent of the classification task, Represents the feature encoder of the network, b represents the serial number of the small batch sample image, i and j represent the feature dimension index of the network output, and C is a square matrix with a size of the number of network output features.

[0157] Constructing a feature relationship graph between the target domain sample image and the source domain sample image, including:

[0158] Pre-built hierarchical navigable small world;

[0159] The hierarchical navigable small world is used to construct a feature relationship graph between the target domain sample image and the source domain sample image.

[0160] The pre-built hierarchical navigable small world includes:

[0161] Providing a hierarchical navigable small world, and importing the feature vector of the source domain sample image into the hierarchical navigable small world;

[0162] Establishing a hierarchical retrieval structure in the hierarchical navigable small world, wherein the retrieval structure has a proximity graph of hierarchical nested subsets of source elements, and randomly selecting a minimum layer where an element is located with an exponentially decaying probability distribution;

[0163] The construction of the hierarchical navigable small world is completed.

[0164] Obtaining feature-similar image pairs according to the feature relationship graph includes:

[0165] Calculating the feature distance between the target domain sample image and the source domain sample image in the hierarchical navigable small world;

[0166] Calculating a sampling probability value according to the characteristic distance, wherein the sampling probability value is negatively correlated with the characteristic distance;

[0167] A feature-similar image pair is obtained according to the sampling probability value.

[0168] The calculation formula of the sampling probability value is:

[0169]

[0170]

[0171] Among them, the Represents the relationship candidate sample image x T and source domain sample images Feature distance in a hierarchical navigable small world, the Indicates the sampling basis value, the Indicates the maximum distance among the nearest neighbor elements. Represents the sampling probability value.

[0172] The preset learning objectives are:

[0173]

[0174]

[0175] Among them, the Represents the cross entropy loss, the (x T ,x S ) represents a pair of feature-similar images, and the p(x S |x T ) represents the sampling probability value, and λ1 represents L cpd The proportion of L in the entire learning goal, λ2 represents L cpc The proportion of the entire learning goal L, Represents the learning goal set based on the feature cross-correlation matrix, represents the objective function set based on the cross-correlation matrix.

[0176] The present invention also provides a computer-readable storage medium storing a computer program, wherein the computer program can be executed to implement the steps of the transfer learning method based on domain pair association as described above.

[0177] In summary, the present invention discloses a method and device for transfer learning based on domain pair association, the method comprising: upon receiving a target domain sample image and a category label corresponding to the target domain sample image, obtaining a source domain sample image; constructing a feature relationship graph between the target domain sample image and the source domain sample image, and obtaining a feature-similar image pair based on the feature relationship graph; inputting the feature-similar image pair into a pre-trained continuous domain adaptation model, using the category label corresponding to the target domain sample image as supervision information, and performing transfer learning on the continuous domain adaptation model with a preset learning target based on a feature mutual correlation matrix as an optimization target; upon receiving a target image in the target domain, inputting the target image into the continuous domain adaptation model that has completed transfer learning to obtain a prediction result. The present invention achieves comparative pairing by constructing feature-similar image pairs and inputting the feature-similar image pairs into the pre-trained continuous domain adaptation model, and uses the preset learning target based on the feature mutual correlation matrix as an optimization target, incrementally exploring the local correlation of the new source domain on the basis of consolidating the model's prior relationships, avoiding the occurrence of transfer drift, and improving the accuracy of the prediction results.

[0178] It should be understood that the application of the present invention is not limited to the above examples. For those skilled in the art, improvements or changes can be made based on the above description. All these improvements and changes should fall within the scope of protection of the claims attached to the present invention.

Claims

1. A transfer learning method based on domain pair association, characterized in that: The method comprises: When receiving a target domain sample image and a category label corresponding to the target domain sample image, obtaining a source domain sample image; Constructing a feature relationship graph between the target domain sample image and the source domain sample image, and obtaining feature-similar image pairs according to the feature relationship graph; Inputting the feature-similar image pairs into a pre-trained continuous domain adaptation model, using the category labels corresponding to the target domain sample images as supervision information, and performing transfer learning on the continuous domain adaptation model with a preset learning objective based on a feature mutual correlation matrix as an optimization objective; When a target domain target image is received, the target domain target image is input into the continuous domain adaptation model that has completed transfer learning to obtain a prediction result; Constructing a feature relationship graph between the target domain sample image and the source domain sample image, including: Pre-built hierarchical navigable small world; constructing a feature relationship graph between the target domain sample image and the source domain sample image using the hierarchical navigable small world; The pre-built hierarchical navigable small world includes: Providing a hierarchical navigable small world, and importing the feature vector of the source domain sample image into the hierarchical navigable small world; Establishing a hierarchical retrieval structure in the hierarchical navigable small world, wherein the retrieval structure has a proximity graph of hierarchical nested subsets of source elements, and randomly selecting a minimum layer where an element is located with an exponentially decaying probability distribution; Complete the construction of the hierarchical navigable small world; Obtaining feature-similar image pairs according to the feature relationship graph includes: Calculating the feature distance between the target domain sample image and the source domain sample image in the hierarchical navigable small world; Calculating a sampling probability value according to the characteristic distance, wherein the sampling probability value is negatively correlated with the characteristic distance; A feature-similar image pair is obtained according to the sampling probability value.

2. The domain pair association-based transfer learning method according to claim 1, characterized in that: The training steps of the continuous domain adaptation model include: Obtaining a source domain sample image and a training category label corresponding to the source domain sample image, and establishing a relationship candidate library based on the source domain sample image and the corresponding training category label; Constructing divergent contrast image pairs based on source domain sample images with different semantic representations in the relationship candidate library; Setting the objective function and consolidation function based on the cross-correlation matrix; Inputting the divergent contrast image pair into an initial continuous domain adaptation model, using the training category label as supervision information, and taking the objective function and the consolidation function as optimization targets to train the initial continuous domain adaptation model; After the training is completed, the trained continuous domain adaptation model is obtained.

3. The domain pair association-based transfer learning method according to claim 2, characterized in that: Before obtaining a source domain sample image and a training category label corresponding to the source domain sample image and establishing a relationship candidate library based on the source domain sample image and the corresponding training category label, the method further includes: Obtaining an original sample image and an original category label corresponding to the original sample image; Inputting the original sample image into the initial continuous domain adaptation model, and calculating the confidence according to the corresponding original category label; If the confidence of the original sample image is greater than a preset confidence threshold, the original sample image is used as a source domain sample image, and the original category label of the original sample image is used as a training category label.

4. The domain pair association-based transfer learning method according to claim 2, characterized in that: The objective function is: Among them, the C ii represents the cross-correlation matrix calculated based on the features of the two parameter-sharing network outputs for the divergent contrast image pair, represents a divergent contrast image pair, the Representing an image The training category label, Representing an image The training class labels of .

5. The domain pair association-based transfer learning method according to claim 2, characterized in that: The calculation formula of the consolidation function is: Wherein, the x represents the input image, the Represents the sum of the feature vector products of the two sample images in the current batch. Represents the sum of the lengths of a certain dimension of the feature vector of an image in the image pair, Represents the sum of the lengths of a certain dimension of the feature vector of another sample image. represents a fully connected projection module independent of the classification task, Represents the feature encoder of the network, b represents the serial number of the small batch sample image, i and j represent the feature dimension index of the network output, and C is a square matrix with a size of the number of network output features.

6. The domain pair association-based transfer learning method according to claim 1, characterized in that: The calculation formula of the sampling probability value is: Among them, the Represents the relationship candidate sample image x T and source domain sample images Feature distance in a hierarchical navigable small world, the Indicates the sampling basis value, the Indicates the maximum distance among the nearest neighbor elements. Represents the sampling probability value.

7. The domain pair association-based transfer learning method according to claim 4, characterized in that: The preset learning objectives are: Among them, the Represents the cross entropy loss, the (x T ,x S ) represents a pair of feature-similar images, and the p(x S |x T ) represents the sampling probability value, and λ1 represents L cpd The proportion of L in the entire learning goal, λ2 represents L cpc The proportion of the entire learning goal L, Represents the learning goal set based on the feature cross-correlation matrix, represents the objective function set based on the cross-correlation matrix.

8. A transfer learning device based on domain pair association, characterized in that: The device comprises: an acquisition module, configured to acquire a source domain sample image upon receiving a target domain sample image and a category label corresponding to the target domain sample image; A construction module, configured to construct a feature relationship graph between the target domain sample image and the source domain sample image, and obtain feature-similar image pairs based on the feature relationship graph; a transfer module, configured to input the feature-similar image pairs into a pre-trained continuous domain adaptation model, use the category labels corresponding to the target domain sample images as supervision information, and perform transfer learning on the continuous domain adaptation model with a preset learning objective based on a feature mutual correlation matrix as an optimization objective; a prediction module configured to, upon receiving a target domain target image, input the target domain target image into the continuous domain adaptation model that has completed transfer learning, to obtain a prediction result; Constructing a feature relationship graph between the target domain sample image and the source domain sample image, including: Pre-built hierarchical navigable small world; constructing a feature relationship graph between the target domain sample image and the source domain sample image using the hierarchical navigable small world; The pre-built hierarchical navigable small world includes: Providing a hierarchical navigable small world, and importing the feature vector of the source domain sample image into the hierarchical navigable small world; Establishing a hierarchical retrieval structure in the hierarchical navigable small world, wherein the retrieval structure has a proximity graph of hierarchical nested subsets of source elements, and randomly selecting a minimum layer where an element is located with an exponentially decaying probability distribution; Complete the construction of the hierarchical navigable small world; Obtaining feature-similar image pairs according to the feature relationship graph includes: Calculating the feature distance between the target domain sample image and the source domain sample image in the hierarchical navigable small world; Calculating a sampling probability value according to the characteristic distance, wherein the sampling probability value is negatively correlated with the characteristic distance; A feature-similar image pair is obtained according to the sampling probability value.

9. A terminal, characterized in that: include: A memory, a processor, and a domain pair association-based transfer learning program stored in the memory and executable on the processor, wherein the domain pair association-based transfer learning program, when executed by the processor, implements the steps of the domain pair association-based transfer learning method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and the computer program can be executed to implement the steps of the domain pair association-based transfer learning method according to any one of claims 1 to 7.

Images