A domain adaptation approach using adversarial learning and data selection strategies

Abstract

The present invention provides a domain adaptation method using adversarial learning and data selection strategies, and fuses the backbone network structure of the LCResNet network and the Vision Transformer as the backbone network structure to extract feature information of the input image. LCResNet adopts the LSE_At structure as the basic network module, and adds a convolution plus attention module CBAAM to the backbone network, which can generate attention feature map information in two dimensions of channel and space in a serial manner. LCResNet is also connected to ViT in parallel, and the different characteristics of LCResNet and ViT are used to obtain rich feature information. A domain classifier is added to enable the backbone network to perform adversarial training and learning through back propagation to distinguish different domains of data. An energy-based data selection strategy is also adopted, and the image recognition algorithm that is easy to be misidentified is selected from the target data set for annotation and retraining, and the image recognition algorithm that is easy to be correctly identified is selected for temporary storage to improve the utilization rate of limited data, thereby improving the classification performance of the network and improving the problem of domain drift.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a domain adaptation method using adversarial learning and data selection strategies. Background Art

[0002] Since the beginning of the 21st century, with the rapid development of the Internet, mankind has entered an era of the Internet of Everything. As people go about their daily lives, a huge amount of data is generated every day, which provides realistic material conditions for the development of artificial intelligence. In practical applications, image recognition models based on deep learning have promoted the accuracy of models in specific scenarios to reach or exceed human levels, freeing humans from boring and repetitive detection work, improving the safety and comfort of human work, and having a profound impact on daily learning, work and life.

[0003] Although deep learning technology has achieved remarkable results in image recognition, its application still faces some challenges. The good performance of deep learning models is based on a large number of annotated images as training sets and the test set data is required to obey the same probability distribution. However, in actual application scenarios, due to the appearance and color attributes of objects and the influence of natural factors such as weather and light, there will be huge differences between image data in different fields. This difference is called data set bias. Due to the bias in the data set, when a deep learning model trained in one field is applied to another field, the performance of the model will be significantly reduced. This phenomenon is called domain shift. When an image recognition model trained with real images is applied to a similar environment for classification and recognition, its classification performance will be significantly reduced; in addition to being affected by samples, the shortage of training samples will also affect the generalization performance of the image recognition model, which leaves a lot of room for development in the research of domain adaptation. In domain adaptation, the training set and the test set are called the source domain and the target domain respectively. The core purpose of training the model is to reduce the distribution difference between the two domains. When the data distribution of the source domain and the target is different but the two tasks are the same, this special transfer learning is called domain adaptation.

[0004] In computer vision, domain adaptation is widely used because labeled datasets are easy to obtain, but the domains of the obtained dataset and the dataset used in the final application are often different. Therefore, the degree of association between the source domain and the target domain usually determines the degree of success of domain adaptation. This paper takes the domain adaptation method based on adversarial learning as the starting point, aims to minimize the inter-domain differences between source domain data and target domain data, and proposes a domain adaptation method using adversarial learning and data selection strategy in combination with current technical research. Summary of the invention

[0005] The embodiment of the present invention provides a resource management method in a multimedia communication system, which is used to solve the problems existing in the prior art.

[0006] In order to achieve the above object, the present invention adopts the following technical scheme.

[0007] A domain adaptation method using adversarial learning and data selection strategy, including:

[0008] S1 configures the operating environment, dataset, and network model for domain adaptation;

[0009] S2 uses the data set to train the network model to obtain a recognition model; the recognition model has a backbone feature extraction network that integrates the LCResNet network structure and the ViT network structure;

[0010] S3 obtains the energy value of each initial sample by calculating the energy function of the recognition model;

[0011] S4 sets an iterative round number cycle for energy function calculation; if the current iterative round number is within the set iterative round number cycle for energy function calculation, a first sample is obtained by performing two energy function calculations;

[0012] S5 obtains energy values ​​of other samples other than the first sample by energy function calculation, selects other samples whose energy values ​​exceed the first preset threshold as candidate data sets; obtains activation samples whose energy value differences exceed the second preset threshold in the candidate data sets by calculation;

[0013] S6 updates and saves the parameters of the recognition model through a gradient descent algorithm;

[0014] S7 inputs the first sample and the activation sample into the recognition model obtained by executing step S6 for training;

[0015] S8 calculates and executes the loss function of the recognition model obtained in step S7;

[0016] S9 obtains the optimal model based on the loss function of the recognition model and the training result of the recognition model;

[0017] S10 performs classification processing of image datasets using the optimal model.

[0018] Preferably, the LCResNet network structure includes: a convolution layer, a pooling layer, 4 groups of stage groups arranged in sequence along the data flow direction, an average pooling layer, and a fully connected layer;

[0019] Each stage group includes multiple Bottleneck layers and a CBAAM block, and the CBAAM block is located at the end of the Stage group;

[0020] Multiple Bottleneck layers include a main path consisting of multiple relu function blocks, a left path containing the LSE_At attention mechanism, and a branch for the original image input.

[0021] Preferably, the CBAAM block includes: an input layer, a group of parallel maximum pooling layers and average pooling layers, a multilayer perceptron, a group of parallel maximum pooling layers and average pooling layers, a first activation function layer, a channel attention feature extraction layer, a group of convolutionally connected maximum pooling-average pooling layers and convolution layers, a second activation function layer, and a spatial attention feature extraction layer, which are arranged in sequence along the data flow direction;

[0022] The input layer is also jump-connected with the first activation function layer through the channel attention mechanism; the channel attention feature extraction layer is also jump-connected with the second activation function layer through the spatial attention mechanism.

[0023] Preferably, the ViT network includes a first encoder and a second encoder in a stacked arrangement; the first encoder includes a regularization module and a multi-head self-attention module, and the second encoder includes a regularization module and a multi-layer perceptron.

[0024] Preferably, the recognition model also has an adversarial network, including a generator and a discriminator; the generator is used to deceive the discriminator during the training process of the recognition model, and the discriminator is used to distinguish the accuracy of the difference between the target domain and the source domain during the training process of the recognition model.

[0025] Preferably, the calculation process of the energy function includes:

[0026] Pass-through

[0027]

[0028] Define a probability distribution P(x) of a random variable x∈X set; where f(x) is a function of x, Z=∑ x∈ X e f(x) is a regularization term, used to transform equation (1) into

[0029]

[0030] Substituting the state energy E into equation (2) to obtain

[0031]

[0032] Based on equations (2) and (3), we can obtain

[0033] f(x)=-E(x) (4);

[0034] For feature X = {x1, x2, ..., xn ,}, label y∈Y, through the formula

[0035] y * =argmin y∈Y E(x,y) (5)

[0036] Calculate the matching degree of x and y; where x is the matching degree of x and y.

[0037] Preferably, in step S5, by formula

[0038] U(x)=E(x,y * ;θ)-E(x,y';θ)

[0039] Calculate the difference in energy values ​​in the alternative data sets; where E(x,y) is the free energy of a sample, x represents the input image data, and y * and y′ are the minimum and second minimum energy values ​​respectively, and θ is a hyperparameter.

[0040] It can be seen from the technical solutions provided by the above embodiments of the present invention that the present invention provides a domain adaptation method using adversarial learning and data selection strategies, and fuses the backbone network structure of the LCResNet network and the Vision Transformer as the backbone network structure to extract the feature information of the input image. LCResNet adopts the LSE_At structure as the basic network module of ResNet50, and proposes to add another left path containing the SE attention mechanism on the basis of the basic residual block to improve the network structure and enhance the network's feature extraction ability for the input image. Secondly, LCResNet adds a convolution plus attention module CBAAM proposed in this topic to the backbone network, which can sequentially generate attention feature map information in two dimensions, channel and space, and on the basis of the original spatial channel attention, increases the participation of the original image in the entire feature extraction process, bridging the disadvantage that the original image information is easily lost during feature extraction, resulting in incomplete feature extraction. Unlike other deep models, the present invention connects LCResNet and ViT in parallel, and simultaneously utilizes the different characteristics of LCResNet and ViT in the feature extraction process to obtain richer feature information. At the same time, a domain classifier is added so that the backbone network can perform adversarial training and learning through back-propagation to distinguish different domains of data.

[0041] On the other hand, an energy-based data selection strategy is adopted. By selecting data that is easy to be misidentified from the target data set for labeling and retraining, and selecting data that is easy to be correctly identified for temporary storage, the image recognition algorithm improves the utilization rate of limited data, thereby improving the classification performance of the network and ultimately improving the problem of domain drift.

[0042] Additional aspects and advantages of the present invention will be given in part in the following description, which will become obvious from the following description, or may be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0044] Figure 1 A processing flow chart of a domain adaptation method using adversarial learning and data selection strategy provided by the present invention;

[0045] Figure 2 A LSE_At structure diagram of a domain adaptation method using adversarial learning and data selection strategy provided by the present invention;

[0046] Figure 3 A CBAAM module structure diagram of a domain adaptation method using adversarial learning and data selection strategy provided by the present invention;

[0047] Figure 4 The overall framework diagram of LCResNet of a domain adaptation method using adversarial learning and data selection strategy provided by the present invention;

[0048] Figure 5 A Transformer encoding layer of a domain adaptation method using adversarial learning and data selection strategy provided by the present invention;

[0049] Figure 6 A GAN network structure diagram of a domain adaptation method using adversarial learning and data selection strategy provided by the present invention;

[0050] Figure 7 A LCVTNet network model structure of a domain adaptation method using adversarial learning and data selection strategy provided by the present invention;

[0051] Figure 8 A flowchart of a preferred embodiment of a domain adaptation method using adversarial learning and data selection strategy provided by the present invention;

[0052] Fig. 9 Schematic diagram of the ResNet network structure in the prior art. DETAILED DESCRIPTION

[0053] The embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be interpreted as limiting the present invention.

[0054] It will be understood by those skilled in the art that, unless expressly stated, the singular forms "one", "said", and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present invention refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be intermediate elements. In addition, the "connection" or "coupling" used herein may include wireless connection or coupling. The term "and / or" used herein includes any unit and all combinations of one or more associated listed items.

[0055] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as those generally understood by those skilled in the art in the art to which the present invention belongs. It should also be understood that terms such as those defined in common dictionaries should be understood to have meanings consistent with the meanings in the context of the prior art, and will not be interpreted with idealized or overly formal meanings unless defined as herein.

[0056] To facilitate understanding of the embodiments of the present invention, several specific embodiments will be further explained below with reference to the accompanying drawings, and each embodiment does not constitute a limitation on the embodiments of the present invention.

[0057] The present invention provides a domain adaptation method using adversarial learning and data selection strategy, which is used to solve the following technical problems existing in the prior art.

[0058] The existing technology has made significant progress in the field of domain adaptation, especially in application scenarios in specific fields. For example, in agricultural image recognition, researchers have used domain adaptation technology to better adapt the model to different farmland environments and improve the accuracy of crop disease recognition in the field of agricultural image recognition. In medical image recognition, domain adaptation is widely used in medical image recognition tasks in different medical institutions to solve image differences under different equipment and acquisition conditions and improve the versatility of the model. In autonomous driving technology, domestic researchers focus on the adaptability of the model in different driving scenarios such as cities and rural areas to improve the performance of autonomous driving systems under diverse road conditions. In traffic monitoring and pedestrian re-identification tasks, researchers use domain adaptation methods to enable the model to adapt to traffic scenarios in different cities and improve the recognition ability of the model in new environments.

[0059] Internationally, domain adaptation has become one of the hot research directions in the field of deep learning and computer vision. Researchers are committed to solving the problem of performance degradation of deep learning models when facing new domain data. Common research directions and methods are as follows:

[0060] (1) Adversarial training: By using the idea of ​​adversarial training and introducing domain classifiers, the deep learning model can be trained to combat the differences between the source domain and the target domain in order to improve the generalization performance of the model.

[0061] (2) Inter-domain alignment: By minimizing the distribution difference between the source domain and the target domain, the feature representation of the model in different domains is made more consistent, thereby enhancing the adaptability of the model.

[0062] (3) Transfer learning: Using previously learned knowledge, the model parameters trained in the source domain are transferred to the target domain to improve the performance of the model in the target domain.

[0063] (4) Unsupervised learning: In the absence of label information in the target domain, unsupervised learning methods are used to maximize the use of data in the target domain for training to improve the generalization performance of the model.

[0064] Through the analysis of the above methods, adversarial learning method has the following advantages for domain adaptation, so adversarial learning is chosen to build the domain adaptation model:

[0065] (1) Effectively handle inter-domain distribution differences: Adversarial learning methods can effectively minimize the distribution differences between the source domain and the target domain by introducing adversarial training, thereby helping the model better adapt to the data distribution of the target domain.

[0066] (2) No need to rely on target domain label information: Adversarial learning methods are usually unsupervised or semi-supervised and do not require target domain label information. They only need to use target domain data for training, thus saving label acquisition costs.

[0067] (3) Improving model generalization ability: Through adversarial learning methods, the model can perform dynamic adversarial training between the source domain and the target domain, so that the model can be better generalized to the target domain, improving the generalization ability of the model.

[0068] (4) Wide applicability: Adversarial learning methods do not depend on specific domains or tasks and can be applied to a variety of domain adaptation problems, including image recognition, natural language processing, etc.

[0069] (5) Strong theoretical foundation: Adversarial learning methods are developed based on technologies such as generative adversarial networks (GANs). They have a solid theoretical foundation and can provide effective solutions to domain adaptation problems.

[0070] In summary, choosing adversarial learning methods for domain adaptation can effectively deal with distribution differences between domains, does not require target domain label information, improves model generalization ability, has wide applicability, and has a strong theoretical foundation, so it has great advantages in practical applications.

[0071] However, the existing methods for solving domain adaptation problems mostly rely on the annotated datasets used, and the commonly used public datasets are artificially constructed. The extraction of image features depends on the depth of the network and the accuracy of the algorithm used. Therefore, there are the following shortcomings:

[0072] (1) Use mature neural network structures as the feature extraction network of your own model to train the model. However, these established network model designs are not suitable for all classification task data, and it is difficult to extract the deep features of the input data more perfectly.

[0073] (2) Ignoring the fact that all data contribute differently to the training model, throwing all data into the model for repeated training will inevitably increase the computational cost and increase the training burden of the network.

[0074] See also Figure 1 The present invention provides a domain adaptation method using adversarial learning and data selection strategy, comprising the following steps:

[0075] S1 configures the operating environment, dataset, and network model for domain adaptation;

[0076] S2 uses the data set to train the network model and obtain the recognition model;

[0077] S3 obtains the energy value of each initial sample by calculating the energy function of the recognition model;

[0078] S4 sets an iterative round number cycle for energy function calculation; if the current iterative round number is within the set iterative round number cycle for energy function calculation, a first sample is obtained by performing two energy function calculations;

[0079] S5 obtains energy values ​​of other samples other than the first sample by calculating the energy function, and selects other samples whose energy values ​​exceed the first preset threshold as candidate data sets; obtains activation samples whose energy value differences in the candidate data sets exceed the second preset threshold by calculation;

[0080] S6 updates and saves the parameters of the recognition model through a gradient descent algorithm;

[0081] S7 inputs the first sample and the activation sample into the recognition model obtained by executing step S6 for training;

[0082] S8 calculates and executes the loss function of the recognition model obtained in step S7;

[0083] S9 obtains the optimal model based on the loss function of the recognition model and the training result of the recognition model;

[0084] S10 performs classification processing of image datasets using the optimal model.

[0085] The domain adaptation method provided by the present invention designs a new backbone vital sign extraction network LCVTNet and combines it with an energy-based data selection strategy to improve the accuracy of domain adaptation. The model structure mainly includes a feature extraction network LCVTNet and a data selection algorithm. In the preferred embodiment provided by the present invention, the specific setting method is as follows.

[0086] 1. Design LCVTNet structure

[0087] In deep learning, in order to enhance the learning ability of the model, the network layers will become deeper and deeper, but as the depth increases, some problems also arise. For example, network degradation, that is, the learning ability of the model has reached saturation, and increasing the number of network layers cannot improve the accuracy. In order to solve the problem of network degradation, deep residual networks came into being. This network uses skip connections or residual connections to skip several layers of residual blocks to solve the problem of accuracy saturation and ultimately degradation caused by deepening the number of layers.

[0088] In the preferred embodiment provided by the present invention, firstly, the LCResNet network structure is added to the network model, which specifically includes the following two structures:

[0089] (1) Based on the residual block ( Fig. 9 ) adds another left path containing the LSE_At attention mechanism to construct the LCResNet network structure, which can automatically obtain the importance of each channel of the feature map, so that the neural network can focus on certain feature channels. The LSE_At structure is as follows Figure 2 As shown in the figure, it includes: a conventional main path consisting of three relu function blocks, a left path containing the LSE_At attention mechanism, and a branch for the original image input. Finally, the outputs of the three network paths are stacked. It should be understood that the "left" and "right" here are based on the direction of data flow.

[0090] (2) A hybrid attention module CBAAM is proposed, which can sequentially generate attention feature map information in both channel and spatial dimensions. On the basis of the original spatial channel attention, it increases the participation of the original image in the entire feature extraction process, thus bridging the disadvantage of losing the original image information during feature extraction. The structure of the CBAAM module is shown in the figure. Figure 3 As shown, it includes the following arranged in sequence along the data flow direction: input layer, a group of parallel maximum pooling layers and average pooling layers, multi-layer perceptron, a group of parallel maximum pooling layers and average pooling layers, the first activation function layer, Channel-refined featureF' layer, a group of convolutionally connected maximum pooling-average pooling layers and convolution layers, the second activation function layer, Aspatial Attention (F') layer; wherein, the input layer is jump-connected to the first activation function layer through the channel attention mechanism; the Channel-refined featureF' layer is jump-connected to the second activation function layer through the spatial attention mechanism. The maximum pooling-average pooling layer is composed of a maximum pooling block and an average pooling block, which respectively output their respective calculated data to the convolution layer.

[0091] The entire network model structure of LCResNet is as follows Figure 4 As shown in the figure, ResNet is divided into 5 Stage groups, where the first convolution layer and the maximum pooling layer are regarded as Stage 0, which has a relatively simple structure and can be regarded as a preprocessing of the input layer. The last 4 Stages are composed of Bottleneck (the basic building block of ResNet) and have similar structures. Stage 1 contains 3 Bottlenecks, and the remaining 3 Stages include 4, 6, and 3 Bottlenecks respectively. Finally, the CBAAM block is added to the back of the first 3 Stage blocks to enhance feature extraction.

[0092] On the other hand, the ViT network structure is also introduced into the network model to solve the following technical problems discovered by the applicant:

[0093] Since different feature channels have different expressive abilities and functions in convolutional neural networks, obtaining the importance of each channel in the feature map and assigning different weights can enhance the feature channels that are useful for classification and suppress the feature channels that are not very useful for classification, thereby achieving the purpose of improving the feature representation ability. Because convolutional neural networks can combine lower-level local features into higher-level features through convolutional pooling operations, convolutional pooling operations are widely used in the field of image recognition in neural networks. However, the pooling layer often loses a lot of valuable information, thereby ignoring the correlation between the local and the whole. Research has found that ViT can obtain global information from the shallow layer and ViT uses the global self-attention mechanism of Transformer, which can consider all positions of the image at the same time, so that the model can better capture global features and long-range dependencies, but the local information obtained from the shallow layer is generally less. The main idea of ​​ViT is to divide the image into a series of image blocks and perform serialization processing, converting each image block into an element in the sequence, so that the two-dimensional structure of the image is flattened into a one-dimensional sequence, and then input this sequence into the Transformer network model for processing. The Transformer encoding layer repeatedly stacks the Encoder Block. The structure of Encoder Block mainly includes two parts: Multi-Head Self Attention Mechanism MHA (Multi-Head Self Attention) and Multi-Layer Perceptron MLP (Multi-Layer Perceptron). The structure of Transformer encoding layer is as follows: Figure 5 As shown, it includes a first encoder and a second encoder in a stacked arrangement. The first encoder (the lower area in the figure) includes a regularization module and a multi-head self-attention module, and the second encoder (the upper area in the figure) includes a regularization module and a multi-layer perceptron.

[0094] In some other preferred embodiments, the specific configuration of the adversarial learning framework of the network model is as follows:

[0095] The core idea of ​​the domain adaptation method based on adversarial learning is to use the adversarial learning method to train a generative adversarial model between the source domain and the target domain. The structure of the generative adversarial network GAN is as follows: Figure 6As shown. Specifically, the main parts of the model include the generator and discriminator of the training model. The purpose of training the generator is to enable the model to better learn the mapping from source domain data to target domain data. Its goal is to minimize the distribution gap between two or more domains and make the representation of source domain data closer to the distribution of target domain data. In the training process of the network model, the generator and the discriminator compete with each other, forming an adversarial learning process, that is, the generator tries to deceive the discriminator so that it cannot accurately distinguish between source and target domain data, while the discriminator strives to improve its accuracy in distinguishing the differences between the two domains. By continuously training the model, the representation of source domain data and target domain data will eventually become more similar, thereby improving the model's generalization ability for the target domain. Therefore, it is considered to apply the idea of ​​GAN to the domain adaptation method.

[0096] In summary, in order to make full use of the information of each feature channel, the present invention designs the LCVTNet network model structure based on the above three points, such as Figure 7 The network architecture is shown in .

[0097] 2. Designing an Energy-Based Data Selection Method

[0098] Energy-Based Model (EBM) is a machine learning model that uses energy functions to describe and learn data distribution. Its core idea or working point is to model and process data samples by defining a formal energy function. After the data is operated by this function, the energy generated for real samples is lower and the energy generated for unreal or fake samples is higher. The energy function is usually written as E(x, y), which is used to measure the compatibility of x and y. It can also be understood as whether x and y match. The smaller the energy, the higher the matching degree. E represents the energy function, x represents the input image data, and y represents the label data of the image. In order to minimize the energy, the label y is changed to adapt to the image data x, or x is changed to adapt to the label y. An energy-based model, also known as a random field, defines a probability distribution P(x) of a set of random variables x∈X, which can be implemented using an exponential function, as shown in formula (1):

[0099]

[0100] Where f(x) is a function of x, where Z = ∑ x∈X e f(x) is a regular term, so the probability distribution P(x) can be written as formula (2):

[0101]

[0102] As shown in formula (3), the Boltzmann distribution has a similar expression in statistical physics, where E represents the energy of the state:

[0103]

[0104] After comparing equation (2) and equation (3), the function f(x) mentioned above can be written as:

[0105] f(x)=-E(x) (4)

[0106] This modeling method is called the energy model. In machine learning, the energy function E(x) is usually expressed as a likelihood function or a log-likelihood function, so finding the maximum likelihood is expressed as finding the minimum energy. For example, in a classification problem, for feature X = {x1, x2, …, x n ,}, label y∈Y, we can establish the energy function E(x,y), which represents the matching degree of x and y. The lower the energy, the better the matching degree. As shown in formula (5), y * Indicates the minimum energy value:

[0107] y * =argmin y∈Y E(x,y) (5)

[0108] This method calculates the free energy of each sample through the energy function E(x,y) and uses the formula U(x)=E(x,y * ; θ)-E(x,y'; θ) calculates samples with large differences as activation samples for training, so that samples that are more likely to be misidentified have more training opportunities, where x represents the input image data, y * and y′ are the minimum and sub-minimum energy values, respectively, and θ is a hyperparameter. At the same time, the images with small calculated differences are taken as temporary samples, that is, images that are easier to be successfully recognized, and are placed in the cache area so that they are temporarily no longer involved in training, thereby improving the utilization of data, reducing the computational burden of the model, and improving the performance of the model.

[0109] The present invention also provides an embodiment for showing a preferred implementation process of the method provided by the present invention. Figure 8 As shown, it includes the following steps:

[0110] Step 1: Configure the corresponding operating environment and basic packages required by the domain adaptation program. The training model of the present invention is trained using the Pytorch architecture, using Python 3.8 and configuring the required virtual environment.

[0111] Step 2: Prepare a data set for training the model. The data sets selected in the present invention are Office31 and handwritten digital data sets.

[0112] Step 3: Set up configuration files and download pre-trained models. According to the different characteristics of different data sets, we set up their own configuration files and downloaded pre-trained model parameters in related fields to reduce the amount of training calculations and improve training efficiency.

[0113] Step 4: Image preprocessing: crop and normalize the input image to improve its usability.

[0114] Step 5: Load relevant parameters for model training. Input the input data into the network model for model training, and initially obtain a recognition model used on this experimental data set.

[0115] Step 6: Calculate the energy value of each sample through the energy function. For example, use the various sub-datasets of the Office dataset as source domain and target domain data input into the network model, and use the energy function to calculate the energy value (free energy) of each sub-dataset, so as to obtain the complete learning target loss function later.

[0116] Step 7: Record the recognition accuracy of the model. Before reaching the maximum number of training rounds (epochs), calculate the accuracy of the model every 5 epochs.

[0117] Step 8: Start the data selection algorithm phase. The data selection algorithm is performed every 5 epochs. If the current epoch belongs to the epoch in the set number of selection rounds, more valuable target samples are obtained through two selections, which are marked for training to improve generalization ability. At the same time, target samples that are easier to identify correctly are obtained through two selections for fixing to reduce the amount of data calculation for the model. The energy value of the unlabeled target data is calculated according to the energy function, and α1% of the data with relatively large free energy (which can be achieved by setting the first preset threshold) is selected as the alternative data set. Then, in the alternative data set, according to the formula U(x)=E(x,y * ;θ)-E(x,y';θ) calculates α2% of the free energy with a relatively large difference (which can be achieved by setting a second preset threshold) as the activation sample for training, where E(x,y) is the free energy of a sample, x represents the input image data, y * and y′ are the minimum and second minimum energy values ​​respectively, and θ is a hyperparameter.

[0118] Step 9: Update model parameters through gradient descent algorithm and save.

[0119] Step 10: Label the samples selected in step 8 and use them together with the source domain images and target domain images as input samples for model training.

[0120] Step 11: Calculate the loss function as an indicator to evaluate the model.

[0121] Step 12: Set different training times according to different data sets, and finally select the optimal training model.

[0122] Step 13: Save the loss function, the selected images, the fixed images, and the final network model with the best performance.

[0123] Step 14: End the entire cycle.

[0124] The present invention also provides an experimental example for demonstrating the domain adaptation effect of the present invention.

[0125] In this embodiment, the following baselines are experimentally evaluated on the handwritten digit dataset:

[0126] 1. The model is only trained on the source domain without using the target domain data (no domain classifier branch is added to the network).

[0127] 2. Train the model on a labeled target domain. When the target data volume is very large, this data can be used as the upper limit of the DA method.

[0128] The experimental results are then compared with the unsupervised visual domain adaptation method using subspace registration, and the specific results are summarized in Table 1. From Table 1, it can be seen that the average recognition performance of the model in the four groups of target domains on the SVHN and SYN datasets is better than other baseline methods, and the performance of the model on each dataset is at least 5% higher than the lowest baseline without domain adaptation method.

[0129] For the Office-31 dataset, the present invention uses several transfer tasks commonly used in model performance evaluation to conduct experimental verification, and the relevant results are summarized in Table 2. As can be seen from Table 2, on the Office dataset, the recognition accuracy of AMAZON as the source domain to WEBCAM as the target domain is improved by 10%. Although the experimental performance of the model on the Office dataset is not perfect, it has a stronger ability to adapt to unknown domains than the baseline model or the recognition model that does not use the domain adaptation method.

[0130] Table 1 Accuracy of the model in digital image classification

[0131]

[0132]

[0133] Table 2 Accuracy of the model on the Office-31 dataset

[0134]

[0135] In summary, the present invention provides a domain adaptation method using adversarial learning and data selection strategies. For the network model, the backbone network structure of the LCResNet network and the Vision Transformer are fused as the backbone network structure to extract the feature information of the input image. LCResNet adopts the LSE_At structure as the basic network module of ResNet50, and adds a convolution plus attention module CBAAM to the backbone network, which can generate attention feature map information in two dimensions of channel and space in a serialized manner. LCResNet is also connected in parallel with ViT, and the different characteristics of LCResNet and ViT are simultaneously utilized in the feature extraction process to obtain richer feature information. At the same time, a domain classifier is added, so that the backbone network is trained and learned adversarially by back propagation to distinguish different domains of data. An energy-based data selection strategy is also adopted. By selecting data that is easy to be misidentified from the target data set for annotation and retraining, and selecting data that is easy to be correctly identified for temporary storage, the fixed image recognition algorithm improves the utilization rate of limited data, thereby improving the classification performance of the network, and ultimately improving the domain drift problem.

[0136] Those skilled in the art can understand that the accompanying drawings are only schematic diagrams of an embodiment, and the modules or processes in the accompanying drawings are not necessarily required to implement the present invention.

[0137] It can be known from the description of the above implementation methods that those skilled in the art can clearly understand that the present invention can be implemented by means of software plus a necessary general hardware platform. Based on such an understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which can be stored in a storage medium such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in the various embodiments of the present invention or certain parts of the embodiments.

[0138] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device or system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. The device and system embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0139] The above is only a preferred specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed by the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A domain adaptation method using adversarial learning and data selection strategy, characterized in that: include: S1 configures the operating environment, dataset, and network model for domain adaptation; S2 uses the data set to train the network model to obtain a recognition model; The recognition model has a backbone feature extraction network that integrates the LCResNet network structure and the ViT network structure; The LCResNet network structure includes: a convolution layer, a pooling layer, 4 groups of stage groups arranged in sequence along the data flow direction, an average pooling layer, and a fully connected layer; Each stage group includes a plurality of Bottleneck layers and a CBAAM block, and the CBAAM block is located at the end of the stage group; The multiple Bottleneck layers include a main path consisting of multiple relu function blocks, a left path including an LSE_At attention mechanism, and a branch for original image input; The CBAAM block includes: an input layer, a group of parallel maximum pooling layers and average pooling layers, a multilayer perceptron, a group of parallel maximum pooling layers and average pooling layers, a first activation function layer, a channel attention feature extraction layer, a group of convolutionally connected maximum pooling-average pooling layers and convolution layers, a second activation function layer, and a spatial attention feature extraction layer, which are arranged in sequence along the data flow direction; The input layer is also jump-connected with the first activation function layer through a channel attention mechanism; the channel attention feature extraction layer is also jump-connected with the second activation function layer through a spatial attention mechanism; The ViT network includes a first encoder and a second encoder in a stacked arrangement; the first encoder includes a regularization module and a multi-head self-attention module, and the second encoder includes a regularization module and a multi-layer perceptron; S3 obtains the energy value of each initial sample by calculating the energy function of the recognition model; S4 sets an iterative round number cycle for energy function calculation; if the current iterative round number is within the set iterative round number cycle for energy function calculation, a first sample is obtained by performing two energy function calculations; S5 obtains energy values ​​of other samples other than the first sample by calculating the energy function, and selects other samples whose energy values ​​exceed the first preset threshold as candidate data sets; obtains activation samples whose energy value differences in the candidate data sets exceed the second preset threshold by calculation; S6 updates and saves the parameters of the recognition model by using a gradient descent algorithm; S7: inputting the first sample and the activation sample into the recognition model obtained by executing step S6 for training; S8 calculates and executes the loss function of the recognition model obtained in step S7; S9: obtaining an optimal model based on the loss function of the recognition model and the training result of the recognition model; S10 performs classification processing of the image data set using the optimal model.

2. The method according to claim 1, characterized in that The recognition model also has an adversarial network, including a generator and a discriminator; the generator is used to deceive the discriminator during the training process of the recognition model, and the discriminator is used to distinguish the accuracy of the difference between the target domain and the source domain during the training process of the recognition model.

3. The method according to claim 1, characterized in that The calculation process of the energy function includes: Pass-through Define a probability distribution P(x) of a random variable x∈X set; where f(x) is a function of x, Z=∑ x∈X e f(x) is a regularization term, used to transform equation (1) into Substituting the state energy E into equation (2) to obtain Based on equations (2) and (3), we can obtain f(x)=-E(x) (4); For feature X = {x1, x2, ..., x n ,}, label y∈Y, through the formula the * =argmin y∈Y E(x,y) (5) The matching degree of x and y is calculated; wherein the energy function E(x, y) represents the matching degree of x and y.

4. The method according to claim 1, characterized in that: In step S5, by formula U(x)=E(x,y * ;θ)-E(x,y'; θ) Calculate the difference in energy values ​​in the candidate data set; where E(x,y) is the free energy of a sample, x represents the input image data, and y * and y′ are the minimum and sub-minimum energy values ​​respectively, and θ is a hyperparameter.

Images