Neural network training method, data acquisition method and device

Abstract

The present application discloses a neural network training method, a data acquisition method and a device in the field of artificial intelligence. The neural network training method includes: acquiring training data; using the training data to train the neural network so that the neural network learns to decompose domain-invariant features and domain-specific features from the training data. By decomposing the domain-invariant features and domain-specific features from the training data, the domain-invariant features can be decoupled from the domain-specific features; wherein the domain-specific features are features that characterize the domain to which the training data belongs, and the domain-invariant features are features that are unrelated to the domain to which the training data belongs. Since the neural network trained by the method of the present application uses the domain-invariant features obtained by feature decoupling to perform tasks, the influence of domain-specific features on the neural network is avoided, thereby improving the migration performance of the neural network between different domains.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence, and specifically to a neural network training method, data acquisition method and device. Background Art

[0002] Artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a branch of computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making. Research in the field of artificial intelligence includes robotics, natural language processing, computer vision, decision-making and reasoning, human-computer interaction, recommendation and search, basic AI theory, etc.

[0003] For example, in computer vision-related application scenarios, neural networks trained using machine learning can be used to complete a variety of tasks such as target classification / detection / recognition / segmentation / prediction. In many application scenarios, training samples and test samples are likely to come from different domains, which can cause problems for the practical application of neural networks. For example, in the application scenario of vehicle detection, the source domain data may be traffic scene images taken on a sunny day, while the target domain data is traffic scene images taken on a foggy day. At this time, it is difficult for the target detection model trained using the source domain data to achieve good results in the target domain data scenario. In order to solve this model application problem caused by the domain deviation between training samples and test samples, domain adaptation (DA) learning has received widespread attention as an important research field of machine learning in recent years.

[0004] Domain adaptive learning usually uses distribution alignment methods to align the probability distribution between the source domain and the target domain data to alleviate the adverse effects of domain bias on domain adaptive learning tasks. Since this distribution alignment process is only performed at the overall feature representation level, domain adaptive learning tasks are inevitably affected by specific features of different domains. Therefore, the trained neural network still has the problem of poor transfer performance. Summary of the invention

[0005] The present application provides a neural network training method, a data acquisition method and a device, which can better improve the migration performance of the neural network between different fields.

[0006] In a first aspect, a method for training a neural network is provided, comprising: obtaining training data; using the training data to train the neural network so that the neural network learns to decompose domain-invariant features and domain-specific features from the training data; wherein the domain-specific features are features that characterize the domain to which the training data belongs, and the domain-invariant features are features that are unrelated to the domain to which the training data belongs.

[0007] By decomposing domain-invariant features and domain-specific features from the training data, the domain-invariant features can be decoupled from the domain-specific features. Since the neural network obtained by the training method of the present application uses domain-invariant features to perform tasks, the influence of domain-specific features on the neural network is avoided, thereby improving the migration performance of the neural network between different domains.

[0008] In combination with the first aspect of the present application, in a possible implementation method, the training of the neural network using the training data includes: decomposing domain-invariant features and domain-specific features from the training data; using the domain-invariant features to perform a task, obtaining a task loss, and calculating the mutual information loss between the domain-invariant features and the domain-specific features, the task loss being used to characterize the gap between a result obtained by performing a task using the domain-invariant features and a task label, and the mutual information loss being used to represent the difference between the domain-invariant features and the domain-specific features; and training the neural network based on the task loss and the mutual information loss.

[0009] By training the neural network based on task loss and mutual information loss, not only can the decomposed domain-invariant features correspond to instances more accurately, but the mutual information loss between domain-invariant features and domain-specific features can also be reduced during the training process, so as to promote the complete decoupling of domain-invariant features and domain-specific features, and further reduce the impact of domain-specific features on domain-invariant features.

[0010] In combination with the first aspect of the present application, in a possible implementation manner, the method further includes: using the domain-specific features to perform domain classification to obtain domain classification loss; wherein, training the neural network according to the task loss and the mutual information loss includes: training the neural network according to the task loss, the mutual information loss and the domain classification loss.

[0011] By introducing domain classification loss, it helps to extract domain invariant features from the features of training data.

[0012] In combination with the first aspect of the present application, in a possible implementation manner, the decomposing the domain-invariant features and the domain-specific features from the training data includes: extracting initial features from the training data; decomposing the initial features into the domain-invariant features and the domain-specific features, wherein the method further includes: training the neural network to reduce the difference between the information contained in the initial features and the information jointly contained in the domain-invariant features and the domain-specific features.

[0013] By reducing the difference between the information contained in the initial features and the information contained in both the domain-invariant features and the domain-specific features, the decoupled domain-invariant features and domain-specific features can contain all the feature information of the training data, thereby improving the completeness and rationality of feature decoupling.

[0014] In combination with the first aspect of the present application, in a possible implementation manner, before training the neural network to reduce the difference between the information contained in the initial feature and the information jointly contained in the domain-invariant feature and the domain-specific feature, the method also includes: reconstructing the initial feature using the domain-invariant feature and the domain-specific feature to obtain a reconstructed feature; comparing the initial feature and the reconstructed feature to determine the difference between the information contained in the initial feature and the information jointly contained in the domain-invariant feature and the domain-specific feature.

[0015] Using this reconstruction loss to train the neural network can make the decoupled domain-invariant features and domain-specific features contain all the feature information of the training data, so as to improve the completeness and rationality of feature decoupling.

[0016] In combination with the first aspect of the present application, in a possible implementation manner, the method further includes: reconstructing the initial features using the domain-invariant features and the domain-specific features to obtain reconstructed features, wherein the domain-invariant features and the domain-specific features are features decomposed from the initial features; comparing the initial features and the reconstructed features to obtain a reconstruction loss, wherein the reconstruction loss is used to characterize the difference between the information contained in the initial features and the information jointly contained in the domain-invariant features and the domain-specific features, wherein training the neural network according to the task loss and the mutual information loss includes: performing a first stage of training the neural network according to the task loss; performing a second stage of training the neural network according to the mutual information loss, wherein the method further includes: performing a third stage of training the neural network according to the reconstruction loss.

[0017] Dividing the neural network training process into stages can simplify the amount of training in each stage and speed up the convergence of the neural network parameters.

[0018] In combination with the first aspect of the present application, in a possible implementation, the neural network includes a first decoupler and a second decoupler, and the decomposing the domain-invariant features and the domain-specific features from the training data includes: extracting a first feature of the training data from the training data; using the first decoupler to extract preliminary domain-invariant features and preliminary domain-specific features from the first feature; fusing the preliminary domain-invariant features with the first feature to obtain a second feature; extracting a third feature of the training data from the second feature; and using the second decoupler to extract the domain-invariant features and the domain-specific features from the third feature.

[0019] By first obtaining the first feature, decoupling the preliminary domain-invariant feature based on the first decoupler, and fusing the preliminary domain-invariant feature with the first feature to obtain the second feature, the domain-invariant feature information is increased at the level of the first feature. Then, the second feature is used to decouple the domain-invariant feature based on the second decoupler, and the decoupling accuracy of the domain-invariant feature is further enhanced, which can make the trained neural network have stronger task execution performance and better domain adaptability.

[0020] In combination with the first aspect of the present application, in a possible implementation manner, the method further includes: training the neural network to reduce the difference between the information contained in the third feature and the information jointly contained in the domain-invariant feature and the domain-specific feature.

[0021] By reducing the difference between the information contained in the third feature and the information contained in both the domain-invariant feature and the domain-specific feature, the decoupled domain-invariant feature and the domain-specific feature can be further enabled to contain all the feature information of the training data, thereby improving the completeness and rationality of feature decoupling.

[0022] In combination with the first aspect of the present application, in a possible implementation manner, the neural network is used to perform domain adaptive learning, and the training data includes image data in different fields.

[0023] By extracting domain-invariant features and domain-specific features of image data from different fields, and training the neural network based on the task loss obtained by performing tasks using the domain-invariant features, the domain-invariant features can be decoupled from the domain-specific features. Since the domain-invariant features are used to perform tasks, the neural network obtained by the training method of the present application can adapt itself to the processing tasks of images in various different fields through domain adaptive learning, thereby realizing adaptive processing of image data in different fields.

[0024] In a second aspect, a data acquisition method is provided, comprising: acquiring data from a source domain and / or a target domain; inputting the data from the source domain and / or the data from the target domain into a neural network for training to obtain gradient information of a loss function; and perturbing the data from the source domain and / or the data from the target domain based on the gradient information to obtain data from an intermediate domain; wherein the source domain and the target domain are two domains having different data features, and the difference in data features between the intermediate domain and any one of the source domain and the target domain is smaller than the difference in data features between the source domain and the target domain.

[0025] The introduction of directional information between the source domain and the target domain makes the perturbation of the training data more targeted. The training data in the intermediate domain obtained through perturbation can fill the "domain gap" between the source domain and the target domain, alleviating the problem of large distribution differences between the training data in the source domain and the training data in the target domain.

[0026] In combination with the second aspect of the present application, in a possible implementation method, the inputting the data of the source domain and / or the data of the target domain into a neural network for training to obtain gradient information of the loss function includes: inputting the labeled data of the source domain into a first neural network for training to obtain first gradient information, wherein the first neural network is generated based on the training of the labeled data of the target domain.

[0027] The first neural network is generated by training with labeled data in the target domain. Therefore, the first gradient information obtained after inputting the labeled data in the source domain into the first neural network can well measure the direction from the source domain to the target domain.

[0028] In combination with the second aspect of the present application, in a possible implementation method, the inputting the data of the source domain and / or the data of the target domain into a neural network for training to obtain gradient information of the loss function includes: inputting the unlabeled data of the target domain into a second neural network, training in a virtual adversarial training manner, and obtaining second gradient information, wherein the second neural network is generated based on labeled data training.

[0029] The second neural network is generated by training with labeled data from the source domain. Therefore, after inputting the unlabeled data from the target domain into the second neural network and undergoing virtual adversarial training, the second gradient information obtained can well measure the direction from the target domain to the source domain.

[0030] In a third aspect, a neural network training device is provided, comprising: a module calculation module for executing the first aspect.

[0031] In a fourth aspect, a data acquisition device is provided, comprising: a module calculation module for executing the method described in the second aspect.

[0032] In a fifth aspect, a training device for a neural network is provided, comprising: a memory for storing programs; a processor for executing the programs stored in the memory, and when the programs stored in the memory are executed, the processor is used to execute the method described in the first aspect or the second aspect.

[0033] In a sixth aspect, a neural network is provided, comprising: a first feature extraction layer, used to extract a first feature based on input data; a first domain-invariant feature decoupling layer, used to extract a first domain-invariant feature based on the first feature; a feature fusion layer, used to fuse the first feature and the first domain-invariant feature to obtain a second feature; a second feature extraction layer, used to extract a third feature based on the second feature; a second domain-invariant feature decoupling layer, used to extract a second domain-invariant feature based on the third feature; wherein the first domain-invariant feature and the second domain-invariant feature are respectively features characterizing the domain to which the input data belongs, and the first domain-specific feature and the second domain-specific feature are respectively features that are independent of the domain to which the input data belongs.

[0034] In the seventh aspect, a data processing system is provided, comprising: a data acquisition network, used to obtain gradient information of a loss function based on first data, and perturb the first data according to the gradient information to obtain second data; a feature decoupling network, used to train a neural network using training data including the second data, so that the neural network learns to decompose domain-invariant features and domain-specific features from the training data; wherein the domain-specific features are features that characterize the domain to which the training data belongs, and the domain-invariant features are features that are independent of the domain to which the training data belongs.

[0035] In combination with the seventh aspect of the present application, in a possible implementation method, the feature decoupling network includes: a first feature extraction layer, used to extract a first feature based on the training data; a first domain-invariant feature extraction layer, used to extract a first domain-invariant feature based on the first feature; a first domain-specific feature extraction layer, used to extract a first domain-specific feature based on the first feature; a first mutual information loss acquisition layer, used to obtain a first mutual information loss based on the first domain-invariant feature and the first domain-specific feature; a feature fusion layer, used to fuse the first feature and the first domain-invariant feature to obtain a second feature; a second feature extraction layer, used to extract a third feature based on the second feature; a second domain-invariant feature decoupling layer, used to extract a second domain-invariant feature based on the third feature; a second domain-specific feature extraction layer, used to extract a second domain-specific feature based on the third feature; a second mutual information loss acquisition layer, used to obtain a second mutual information loss based on the second domain-invariant feature and the second domain-specific feature; a task loss acquisition layer, used to perform a task using the second domain-invariant feature to obtain a task loss.

[0036] In combination with the seventh aspect of the present application, in a possible implementation, the data processing system further includes: a first domain classifier, used to perform a classification task based on the first domain-specific feature to obtain a first classification loss; a first gradient reversal layer, used to invert the gradient information of the first classification loss; and / or, a second domain classifier, used to perform a classification task based on the second domain-specific feature to obtain a second classification loss; a second gradient reversal layer, used to invert the gradient information of the second classification loss.

[0037] In combination with the seventh aspect of the present application, in a possible implementation method, the data processing system further includes: a reconstruction loss acquisition layer, used to reconstruct the third feature using the second domain invariant feature and the second domain specific feature to obtain a reconstructed feature; compare the third feature and the reconstructed feature to obtain a reconstruction loss.

[0038] In conjunction with the seventh aspect of the present application, in a possible implementation, the first data includes data from a source domain and / or data from a target domain, wherein the data acquisition network includes: a first training network generated based on training with labeled data in the target domain; and / or a second training network generated based on training with labeled data.

[0039] In an eighth aspect, a security device is provided, comprising the neural network described in the sixth aspect.

[0040] In a ninth aspect, a computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the processor executes the method described in the first aspect or the second aspect.

[0041] In a tenth aspect, a computer program product is provided, comprising computer program instructions, which, when executed by a processor, cause the processor to execute the method described in the first aspect or the second aspect.

[0042] In an eleventh aspect, a chip is provided, comprising a processor and a data interface, wherein the processor reads instructions stored in a memory through the data interface to execute the method described in the first aspect or the second aspect.

[0043] Optionally, as an implementation method, the chip may also include a memory, in which instructions are stored, and the processor is used to execute the instructions stored in the memory. When the instructions are executed, the processor is used to execute the method described in the first aspect or the second aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 A schematic diagram of an artificial intelligence main framework.

[0045] Figure 2 A system architecture is provided for an embodiment of the present application.

[0046] Figure 3 A chip hardware structure diagram provided for an embodiment of the present application.

[0047] Figure 4 A system architecture is provided for an embodiment of the present application.

[0048] Figure 5 A flowchart of a neural network training method provided in one embodiment of the present application.

[0049] Figure 6 A flowchart of a neural network training method provided in one embodiment of the present application.

[0050] Figure 7 A schematic diagram of the structure of a neural network provided in one embodiment of the present application.

[0051] Figure 8 A schematic diagram of the feature decoupling principle provided in one embodiment of the present application.

[0052] Fig. 9 A schematic diagram of the structure of a neural network provided in another embodiment of the present application.

[0053] Fig.10 A schematic diagram of the structure of a neural network provided in one embodiment of the present application.

[0054] Fig.11 Based on Fig.10Schematic diagram of the process of extracting domain-invariant features and domain-specific features for the neural network architecture shown.

[0055] Fig.12 A schematic diagram of the principles of the training process provided in one embodiment of the present application.

[0056] Fig.13 Shown is a schematic diagram of the structure of a neural network provided in one embodiment of the present application.

[0057] Fig.14 A schematic diagram of a process for obtaining data in an intermediate domain according to an embodiment of the present application.

[0058] Fig.15 A schematic diagram of the principle of obtaining data in the intermediate domain provided in an embodiment of the present application.

[0059] Fig.16 A schematic diagram of bidirectional adversarial training provided in another embodiment of the present application.

[0060] Fig.17 A schematic diagram of the structure of a data processing system provided in one embodiment of the present application.

[0061] Fig.18 A schematic diagram of the structure of a neural network training device provided in one embodiment of the present application.

[0062] Fig.19 A schematic diagram of the structure of a data acquisition device provided in another embodiment of the present application.

[0063] Fig. 20 A schematic diagram of the hardware structure of a neural network training device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0064] The technical solution in this application will be described below in conjunction with the accompanying drawings.

[0065] Figure 1 This is a schematic diagram of an artificial intelligence main framework. This main framework describes the overall workflow of the artificial intelligence system and is applicable to general artificial intelligence field requirements.

[0066] The following is an explanation of the above artificial intelligence theme framework from two dimensions: "intelligent information chain" (horizontal axis) and "IT value chain" (vertical axis).

[0067] The "intelligent information chain" reflects a series of processes from data acquisition to processing. For example, it can be a general process of intelligent information perception, intelligent information representation and formation, intelligent reasoning, intelligent decision-making, intelligent execution and output. In this process, data undergoes a condensed process of "data-information-knowledge-wisdom".

[0068] The "IT value chain" reflects the value that artificial intelligence brings to the information technology industry, from the low-level infrastructure of human intelligence, information (providing and processing technology implementation) to the system's industrial ecological process.

[0069] (1) Infrastructure:

[0070] The infrastructure provides computing power support for the artificial intelligence system, enables communication with the outside world, and supports it through the basic platform. It communicates with the outside world through sensors; computing power is provided by smart chips (CPU, NPU, GPU, ASIC, FPGA and other hardware acceleration chips); the basic platform includes distributed computing frameworks and networks and other related platform guarantees and support, which can include cloud storage and computing, interconnected networks, etc. For example, sensors communicate with the outside world to obtain data, and these data are provided to the smart chips in the distributed computing system provided by the basic platform for calculation.

[0071] (2) Data

[0072] The data on the upper layer of the infrastructure is used to represent the data sources in the field of artificial intelligence. The data involves graphics, images, voice, text, and IoT data of traditional devices, including business data of existing systems and perception data such as force, displacement, liquid level, temperature, and humidity.

[0073] (3) Data processing

[0074] Data processing usually includes data training, machine learning, deep learning, search, reasoning, decision-making and other methods.

[0075] Among them, machine learning and deep learning can symbolize and formalize data for intelligent information modeling, extraction, preprocessing, and training.

[0076] Reasoning refers to the process of simulating human intelligent reasoning in computers or intelligent systems, using formalized information to perform machine thinking and solve problems based on reasoning control strategies. Typical functions are search and matching.

[0077] Decision-making refers to the process of making decisions after intelligent information is reasoned, usually providing functions such as classification, sorting, and prediction.

[0078] (4) General ability

[0079] After the data has undergone the data processing mentioned above, some general capabilities can be further formed based on the results of the data processing, such as an algorithm or a general system, for example, translation, text analysis, computer vision processing, speech recognition, image recognition, etc.

[0080] (5) Smart products and industry applications

[0081] Smart products and industry applications refer to the products and applications of artificial intelligence systems in various fields. They are the encapsulation of the overall artificial intelligence solution, which productizes intelligent information decision-making and realizes practical application. Its application areas mainly include: smart manufacturing, smart transportation, smart home, smart medical care, smart security, autonomous driving, safe cities, smart terminals, etc.

[0082] As mentioned above, for domain adaptive learning tasks, due to the distribution difference between the source domain and the target domain, if the model with good performance in the source domain is directly applied to the target domain scenario, the performance will be limited. When training a neural network model for domain adaptive learning, a distribution alignment strategy is adopted, that is, the data in the source domain and the data in the target domain are aligned at the level of feature representation. Since this distribution alignment process is only performed at the level of overall feature representation, the domain adaptive learning task is inevitably affected by the specific features of different fields. Therefore, the trained neural network model still has the problem of poor transfer performance.

[0083] In response to the above technical problems, the present application proposes a method for training a neural network model, which can decouple domain-invariant features (domain-invariant features can be understood as instance-level features that are independent of the domain) from the features of the data during the training process, so that the domain adaptive learning task will not be affected by the specific features of different domains, thereby improving the migration performance of the neural network model.

[0084] It should be understood that the neural network model trained by the embodiment of the present application can be applied to various application scenarios, and the neural network model may have different structures depending on the specific application scenario. For example, in the image classification application scenario (such as vehicle recognition, face recognition, etc.), the neural network model can be a convolutional neural network model, and in the regression prediction application scenario (such as energy consumption prediction of industrial production lines, weather prediction, landslide prediction, etc.), the neural network model may include a multi-layer perceptron architecture. The embodiment of the present application does not limit the specific application scenario and structure of the trained neural network model.

[0085] In this application, all actions to obtain signals, information or data are carried out in compliance with the relevant data protection laws and policies of the country where they are located and with the authorization of the corresponding device owner.

[0086] Since the embodiments of the present application involve applications in domain adaptive learning and neural networks, for ease of understanding, the following is a brief introduction to related terms and related concepts such as neural networks that may be involved in the embodiments of the present application.

[0087] (1) Domain Adaptive Learning

[0088] Domain adaptive learning is a machine learning method used to solve the problem of inconsistent probability distributions of training samples and test samples. It aims to overcome the difference in probability distributions of source domain samples and target domain samples during training to achieve learning tasks on the target domain.

[0089] (2) Neural Network

[0090] A neural network can be composed of neural units. A neural unit can be referred to as s The output of this operation unit can be expressed by the following formula (1).

[0091]

[0092] In formula (1), s = 1, 2, ... n, n is a natural number greater than 1, W s For x s The weight of the neural unit, b is the bias of the neural unit. f is the activation function of the neural unit, which is used to introduce nonlinear characteristics into the neural network to convert the input signal in the neural unit into an output signal. The output signal of the activation function can be used as the input of the next convolutional layer. The activation function can be a sigmoid function. A neural network is a network formed by connecting many of the above-mentioned single neural units together, that is, the output of one neural unit can be the input of another neural unit. The input of each neural unit can be connected to the local receptive field of the previous layer to extract the features of the local receptive field. The local receptive field can be an area composed of several neural units.

[0093] (3) Deep Neural Networks

[0094] A deep neural network (DNN), also known as a multi-layer neural network, can be understood as a neural network with many hidden layers. There is no special metric for "many" here. Based on the position of different layers in a DNN, the neural network inside a DNN can be divided into three categories: input layer, hidden layer, and output layer. Generally speaking, the first layer is the input layer, the last layer is the output layer, and the layers in between are all hidden layers. The layers are fully connected, that is, any neuron in the i-th layer must be connected to any neuron in the i+1-th layer. Although a DNN looks complex, the work of each layer can be expressed based on the linear relationship expression recorded in the following formula (2).

[0095]

[0096] In formula (2), represents the input vector, represents the output vector, b represents the offset vector, W represents the weight matrix (also called coefficient), and α(.) represents the activation function. Each layer is just an input vector. After such a simple operation, the output vector Since there are many DNN layers, there are many coefficients W and offset vectors b. The definitions of these parameters in DNN are as follows.

[0097] Take the coefficient W as an example: Assume that in a three-layer DNN, the linear coefficient from the 4th neuron in the second layer to the 2nd neuron in the third layer is defined as The superscript 3 represents the layer number of the coefficient W, and the subscript 24 corresponds to the output of the third layer index 2 and the input of the second layer index 4. In summary, the coefficients from the kth neuron in the L-1th layer to the jth neuron in the Lth layer are defined as It should be noted that the input layer does not have a W parameter. In a deep neural network, more hidden layers allow the network to better describe complex situations in the real world. Theoretically, the more parameters a model has, the higher its complexity and the greater its "capacity", which means it can complete more complex learning tasks. The process of training a deep neural network is also the process of learning the weight matrix, and its ultimate goal is to obtain the weight matrix of all layers of the trained deep neural network (the weight matrix formed by many layers of vector W).

[0098] (4) Loss Function

[0099] In the process of training a deep neural network, because we hope that the output of the deep neural network is as close as possible to the value we really want to predict, we can compare the predicted value of the current network with the target value we really want, and then update the weight vector of each layer of the neural network according to the difference between the two (of course, there is usually an initialization process before the first update, that is, pre-configuring parameters for each layer in the deep neural network). For example, if the predicted value of the network is high, adjust the weight vector to make it predict a lower value, and continue to adjust until the deep neural network can predict the target value we really want or a value very close to the target value we really want. Therefore, it is necessary to pre-define "how to compare the difference between the predicted value and the target value", which is the loss function or objective function, which are important equations used to measure the difference between the predicted value and the target value. Among them, taking the loss function as an example, the higher the output value (loss) of the loss function, the greater the difference, so the training of the deep neural network becomes a process of minimizing this loss as much as possible.

[0100] (5) Back propagation algorithm

[0101] Convolutional neural networks can use the error back propagation (BP) algorithm to correct the size of the initial neural network parameters during the training process, so that the reconstruction error loss of the neural network becomes smaller and smaller. Specifically, the forward transmission of the input signal to the output will generate error loss, and the error loss information is back-propagated to update the parameters in the initial neural network, so that the error loss converges. The back propagation algorithm is a back propagation movement dominated by error loss, aiming to obtain the optimal neural network parameters, such as the weight matrix.

[0102] (6) Adversarial Examples

[0103] Adversarial samples refer to input samples formed by adding perturbations to the data set, causing the neural network to give an incorrect output with high confidence. Since the ultimate goal of the neural network is to obtain the correct output result, adversarial samples are used to train the neural network with this adversarial training strategy, so that the neural network adapts to this perturbation and is robust to adversarial samples.

[0104] (7) Virtual adversarial training

[0105] Virtual adversarial training refers to an adversarial training method that does not rely on training labels. Virtual adversarial training generates perturbations based on the first output of the neural network. This perturbation makes the second output obtained by inputting the generated adversarial sample into the neural network different from the previous first output, thereby implementing the adversarial training strategy.

[0106] Combine the following Figure 2 The system architecture provided in the embodiments of the present application is introduced in detail.

[0107] Figure 2 A system architecture 200 is provided for an embodiment of the present application. Figure 2 As shown, the system architecture 200 includes an execution device 210, a training device 220, a database 230, a client device 240, a data storage system 250, and a data acquisition system 260. The execution device 210 includes a computing module 211, an I / O interface 212, a preprocessing module 213, and a preprocessing module 214. The computing module 211 may include a target model / rule 201, and the preprocessing module 213 and the preprocessing module 214 are optional.

[0108] The data acquisition device 260 is used to collect training data (or sample data for training) and store it in the database 230. The training data in the embodiment of the present application may include training data in different fields, such as training data in the source domain and training data in the target domain. The training device 220 trains the target model / rule 201 based on the training data maintained in the database 230, so that the target model / rule 201 has the function of decoupling domain-invariant features and domain-specific features from the input data, and the function of using the domain-invariant features to complete the tasks required by the actual application scenario, such as the function of completing tasks such as target classification / detection / recognition / segmentation.

[0109] The target model / rule 201 can be a neural network model. The work of each layer in the neural network model can be expressed by a mathematical expression: To describe: From a physical level, the work of each layer in the neural network model can be understood as completing the transformation from input space to output space (i.e., from row space to column space of a matrix) through five operations on the input space (a set of input vectors). These five operations include: 1. Dimension increase / decrease; 2. Enlarge / reduce; 3. Rotate; 4. Translate; 5. "Bend". Operations 1, 2, and 3 are represented by Completed, operation 4 is completed by +b, and operation 5 is implemented by a(). The word "space" is used here because the classified object is not a single thing, but a class of things, and space refers to the collection of all individuals of this class of things. Among them, W is a weight vector, and each value in the vector represents the weight value of a neuron in this layer of the neural network. The vector W determines the spatial transformation from the input space to the output space described above, that is, the weight W of each layer controls how to transform the space. The purpose of training a neural network model is to eventually obtain the weight matrix of all layers of the trained neural network (the weight matrix formed by many layers of vectors W). Therefore, the training process of a neural network is essentially to learn how to control spatial transformation, and more specifically to learn the weight matrix.

[0110] Because we want the output of the neural network model to be as close as possible to the value we really want to predict, we can compare the current network's predicted value with the target value we really want, and then update the weight vector of each layer of the neural network based on the difference between the two (of course, there is usually an initialization process before the first update, that is, pre-configuring parameters for each layer in the neural network model). For example, if the network's predicted value is high, adjust the weight vector to make it predict a lower value, and keep adjusting until the neural network can predict the target value we really want. Therefore, it is necessary to predefine "how to compare the difference between the predicted value and the target value", which is the loss function or objective function, which are important equations used to measure the difference between the predicted value and the target value. Among them, taking the loss function as an example, the higher the output value (loss) of the loss function, the greater the difference, so the training of the neural network model becomes a process of minimizing this loss as much as possible.

[0111] The target model / rule obtained by the training device 220 can be applied in different systems or devices. Figure 2 In the embodiment, the execution device 210 is configured with an I / O interface 212 for data interaction with external devices, and a “user” can input data into the I / O interface 212 through a client device 240 .

[0112] The execution device 210 can call data, codes, etc. in the data storage system 250 , and can also store data, instructions, etc. in the data storage system 250 .

[0113] The calculation module 211 processes the input data using the target model / rule 201. In the working process of the actual application scenario, the specific input data of the calculation module 211 is related to the specific application scenario. For example, in the application scenario of face recognition, the input data of the calculation module 211 may be image data including a face image. Since the calculation module 211 processes the input data using the target model / rule 201, the calculation module actually obtains instance-level features based on the input data, and then uses the instance-level features to perform specific tasks.

[0114] In an embodiment of the present application, the system architecture 200 may also include some management function modules connected to the calculation module 211 to complete more flexible segmentation tasks based on the output results of the calculation module 211. For example, when the data input by the "user" to the I / O interface 212 through the client device 240 is image data of a traffic scene, Figure 2The illustrated correlation function module 213 can be configured to further identify the vehicle license plate number and model information according to the characteristic information of the vehicle object output by the calculation module 211; and the correlation function module 214 can be configured to further identify the pedestrian's gender, height, age and other information according to the characteristics of the pedestrian output by the calculation module 211. However, the present application does not limit whether the system architecture includes these correlation function modules, and the specific functions performed by these correlation function modules.

[0115] Finally, the I / O interface 212 returns the processing result to the client device 240 and provides it to the user.

[0116] More deeply, the training device 220 can generate corresponding target models / rules 201 based on different data for different goals to provide users with better results.

[0117] In the attached Figure 2 In the case shown in , the user can manually specify the data to be input into the execution device 210, for example, by operating in the interface provided by the I / O interface 212. In another case, the client device 240 can automatically input data into the I / O interface 212 and obtain the result. If the client device 240 needs to obtain the user's authorization to automatically input data, the user can set the corresponding authority in the client device 240. The user can view the results output by the execution device 210 on the client device 240, and the specific presentation form can be a specific method such as display, sound, action, etc. The client device 240 can also serve as a data collection end to store the collected sample data into the database 230.

[0118] It is worth noting that Figure 2 This is only a schematic diagram of a system architecture provided by an embodiment of the present application. The positional relationship between the devices, components, modules, etc. shown in the figure does not constitute any limitation. Figure 2 In the embodiment, the data storage system 250 is an external memory relative to the execution device 210. In other cases, the data storage system 250 can also be placed in the execution device 210.

[0119] Combine the following Figure 3 A chip hardware structure provided in an embodiment of the present application is introduced.

[0120] Figure 3 This is a chip hardware structure diagram provided in one embodiment of the present application. Figure 3 As shown, the chip includes a neural-network processing unit (NPU) 300. The chip can be arranged in Figure 2 The execution device 210 shown in FIG. 1 is used to complete the calculation work of the calculation module 211. The chip can also be set in Figure 2The training device 220 shown in FIG. 2 is used to complete the training work of the training device 220 and output the target model / rule 201. In addition, the following Figure 4 , Fig. 9 and Fig.11 The training methods of the neural network shown in Figure 3 is implemented in the chip shown.

[0121] The neural network processor 300 is mounted on the host central processing unit (host CPU) as a coprocessor, and the host CPU assigns tasks. The core part of the neural network processor 300 is the operation circuit 303, and the controller 304 controls the operation circuit 303 to extract data from the memory (weight memory 302 or input memory 301) and perform operations.

[0122] In some implementations, the operation circuit 303 includes multiple processing units (process engines, PEs) inside. In some implementations, the operation circuit 303 is a two-dimensional systolic array. The operation circuit 303 can also be a one-dimensional systolic array or other electronic circuits capable of performing mathematical operations such as multiplication and addition. In some implementations, the operation circuit 303 is a general-purpose matrix processor.

[0123] For example, assume there is an input matrix A, a weight matrix B, and an output matrix C. The operation circuit 303 takes the corresponding data of the weight matrix B from the weight memory 302 and caches it on each PE in the operation circuit 303. The operation circuit 303 takes the input matrix A and the weight matrix B from the input memory 301 to perform matrix operations to obtain partial results or final results of the matrix, and stores them in the accumulator 308.

[0124] The vector calculation unit 307 can further process the output of the operation circuit 303, such as vector multiplication, vector addition, exponential operation, logarithmic operation, size comparison, etc. For example, the vector calculation unit 307 can be used for network calculations of non-convolutional / non-FC layers in a neural network, such as pooling, batch normalization, local response normalization, etc.

[0125] In some implementations, the vector calculation unit 307 can store the vector of processed outputs to the unified memory 306. For example, the vector calculation unit 307 can apply a nonlinear function to the output of the operation circuit 303, such as a vector of accumulated values, to generate an activation value. In some implementations, the vector calculation unit 307 generates a normalized value, a merged value, or both. In some implementations, the vector of processed outputs can be used as an activation input to the operation circuit 303, for example, for use in a subsequent layer in a neural network.

[0126] The unified memory 306 is used to store input data and output data.

[0127] The weight data is directly transferred from the external memory to the input memory 301 and / or the unified memory 306 through the direct memory access controller (DMAC) 305, the weight data in the external memory is stored in the weight memory 302, and the data in the unified memory 306 is stored in the external memory.

[0128] The bus interface unit (BIU) 310 is used to implement interaction between the main CPU, DMAC and instruction fetch memory 309 through the bus.

[0129] An instruction fetch buffer 309 connected to the controller 304 is used to store instructions used by the controller 304 .

[0130] The controller 304 is used to call the instructions cached in the instruction fetch memory 309 to control the working process of the computing accelerator.

[0131] Generally, the unified memory 306, the input memory 301, the weight memory 302 and the instruction fetch memory 309 are all on-chip memories, and the external memory is a memory outside the NPU, which can be a double data rate synchronous dynamic random access memory (DDR SDRAM), a high bandwidth memory (HBM) or other readable and writable memory.

[0132] Figure 4A system architecture 400 is provided for an embodiment of the present application. The execution device 410 is implemented by one or more servers arranged in the cloud, and the server may also cooperate with other computing devices, such as data storage, routers, load balancers and other devices; the execution device 410 may be arranged at one physical site, or distributed at multiple physical sites. In an embodiment of the present application, the execution device 410 may use the data in the data storage system 420, or call the program code in the data storage system 420 to implement the training method of the neural network provided in the embodiment of the present application; specifically, the execution device 410 may train the neural network according to the training data in the data storage system 420 using the method provided in the embodiment of the present application, and complete the corresponding intelligent task according to the request of the local device 401 / 402. In another embodiment of the present application, the execution device 410 may not have the function of training a neural network, but can complete the corresponding intelligent task with a neural network trained according to the neural network training method provided in the embodiment of the present application; specifically, after the execution device 410 is configured with a neural network trained according to the neural network training method provided in the embodiment of the present application, it can complete the corresponding intelligent task after receiving a request from the local device 401 / 402 and feed back the result to the local device 401 / 402.

[0133] Users can operate their respective user devices (such as local device 401 and local device 402) to interact with execution device 410. Each local device can represent any computing device, such as a personal computer, a computer workstation, a smart phone, a tablet computer, a smart camera, a smart car or other types of cellular phones, a media consumption device, a wearable device, a set-top box, a game console, etc. In an embodiment of the present application, the local device can be a security device, such as a surveillance camera device, a smoke alarm device, or a fire extinguishing device.

[0134] Each user's local device can interact with the execution device 410 through a communication network of any communication mechanism / communication standard. The communication network can be a wide area network, a local area network, a point-to-point connection, etc., or any combination thereof.

[0135] In another implementation, one aspect or multiple aspects of the execution device 410 may be implemented by each local device. For example, the local device 401 may provide local data or feedback calculation results to the execution device 410 .

[0136] In another implementation, all functions of the execution device 410 can also be implemented by a local device. The local device 401 executes the neural network training method provided in the embodiment of the present application, and uses the trained neural network to provide services to users.

[0137] Figure 5A flowchart of a neural network training method provided in one embodiment of the present application. Figure 5 The training method of the neural network shown can be obtained by Figure 2 The training device 220 shown in the figure is executed, and the target model / rule 201 obtained by training by the training device 220 is the neural network. Figure 5 As shown, the training method of the neural network includes the following steps:

[0138] Step 501: Obtain training data.

[0139] The training data is the input data of the training process. The training data can be collected by the user or an existing training database can be used. It should be understood that the training data can have different formats and forms according to the needs of the actual scenario. For example, in the target detection or target recognition scenario, the training data can be image data. In the regression prediction scenario, the training data can be the collected historical housing price data.

[0140] In one embodiment of the present application, the training data input to the neural network may include training data from different domains. For example, different domains may include a target domain and a source domain. Taking the domain adaptive learning task as an example, the training data may include training data from the source domain and training data from the target domain. In one embodiment of the present application, the difference in domains may be reflected in the difference in scenarios. Taking the application scenario of vehicle detection as an example, the training data in the source domain may be a large number of traffic scene images under sunny scenes, and the training data in the target domain may be a large number of traffic scene images under foggy scenes. However, it should be understood that, depending on the application scenario, the training data in the source domain and the training data in the target domain may also reflect this domain difference in other aspects. For example, in the regression prediction scenario, the training data in the source domain may be the energy consumption data collected from the production line last year, and the training data in the target domain may be the energy consumption data collected from the production line this year. At this time, the domain difference is reflected in the inconsistent numerical distribution of the energy consumption data due to time transformation.

[0141] Step 502: Train the neural network using the training data so that the neural network learns to decompose domain-invariant features and domain-specific features from the training data.

[0142] In the process of training a neural network, the neural network can learn the training data by any of the following methods: supervised learning, semi-supervised learning, or unsupervised learning. Taking the domain adaptive learning task as an example, the training data may include training data of the source domain with sufficient labels and training data of the target domain with a small number of labels. In this case, the neural network can learn the training data by semi-supervised learning; or, the training data may include training data of the source domain with sufficient labels and training data of the target domain without labels. In this case, the neural network can learn the training data by unsupervised learning.

[0143] Domain-invariant representation (DIR) is a feature that is independent of the domain to which the training data belongs, and is a feature that will not change due to domain differences. Domain-invariant features are sometimes also called instance-level features related to the task. Taking the application scenario of vehicle detection as an example, the domain difference is reflected in the image difference presented by the weather changes between the traffic scene images taken on a sunny day and the traffic scene images taken on a foggy day, while the characteristics of the vehicles in the traffic scene will not change with the weather changes. At the same time, the target object (i.e., instance) of the target detection task is the vehicle in the image, so the characteristics of the vehicle are the domain-invariant features to be extracted. In the domain adaptive learning scenario of target detection, what needs to be achieved is that no matter whether the image obtained in actual work is taken on a sunny day or a foggy day, the trained neural network can accurately extract the characteristics of the vehicle to complete the target detection task.

[0144] Domain-specific representation (DSR) is a feature that characterizes the domain to which the training data belongs. It is a feature that is unique to the domain to which the training data belongs and will change due to domain differences. At the same time, domain-specific features are also features that are not related to the instance and are not related to the goal of the task during the actual task execution. For example, in the aforementioned vehicle detection application scenario, the features of the environment (trees, sky, street scenes, etc.) around the vehicle in the traffic scene image are irrelevant to the features of the vehicle, because the recognition or detection of the vehicle does not require the understanding of the features of the surrounding environment, and the feature information of these surrounding environments (such as the sky) will change with domain differences (weather changes).

[0145] It can be seen that the embodiment of the present application decomposes the domain-invariant features and the domain-specific features from the training data, so that the domain-invariant features can be decoupled from the domain-specific features. Since the neural network obtained by the training method of the present application uses the domain-invariant features to perform tasks, the influence of the domain-specific features on the neural network is avoided, thereby improving the migration performance of the neural network between different fields.

[0146] Figure 6 A flowchart of a neural network training method provided in one embodiment of the present application. Figure 7 for Figure 6 Schematic diagram of the structure of the trained neural network. Figure 6 The training method of the neural network shown can be obtained by Figure 2 The training device 220 shown in the figure is executed, and the target model / rule 201 obtained by training by the training device 220 is the neural network. Figure 6 and Figure 7 As shown, the training method of the neural network includes the following steps:

[0147] Step 601: Decompose domain-invariant features and domain-specific features from training data.

[0148] like Figure 7 As shown in the figure, the process of extracting domain invariant features DIR and domain specific features DSR can be respectively performed by the domain invariant feature extractor E in the neural network DIR and domain-specific feature extractor E DSR After the training data is input into the neural network, the domain invariant feature extractor E can be used DIR and the domain-specific feature extractor E DSR The extraction process of domain invariant features DIR and domain specific features DSR is completed.

[0149] Combine the following Figure 8 To illustrate the decoupling of domain-invariant features and domain-specific features in the target detection scenario. Figure 8 In the target detection scenario shown, the goal of the task is to detect objects (including people and vehicles) in image data. Figure 8 The training data of the source domain on the left is a photo image, and the training data of the target domain on the right is a cartoon image. In the domain-invariant space, the domain-invariant features extracted from the training data of the source domain are the people and vehicles in the photo image, and the domain-invariant features extracted from the training data of the target domain are the people and vehicles in the cartoon image. Line C1 represents the classification boundary of the domain-invariant features of people and the domain-invariant features of vehicles in the domain-invariant space. In the domain-specific space, the domain-specific features extracted from the training data of the source domain are the other features in the photo image except people and vehicles, and the domain-specific features extracted from the training data of the target domain are the other features in the cartoon image except people and vehicles. Line C2 represents the distribution boundary of the domain-specific features from the source domain and the domain-specific features from the target domain in the domain-specific space.

[0150] Step 602: Use the domain-invariant features to perform the task, obtain the task loss, and calculate the mutual information loss between the domain-invariant features and the domain-specific features. The mutual information loss is used to represent the difference between the domain-invariant features and the domain-specific features.

[0151] As mentioned above, domain-invariant features are used to represent feature information at the instance level. Therefore, using domain-invariant features to perform tasks and obtaining task loss can improve the accuracy and completeness of domain-invariant features for task-related instance representation. Task loss is used to characterize the gap between the result obtained by performing a task using domain-invariant features and the task label. For example, when domain-invariant features are used to perform target detection tasks, the result obtained by performing the task may include the attribute features of the detected target object, while the task label corresponds to the standard attribute features of the target object to which the domain-invariant features actually correspond. In this way, the difference between the detected attribute features and the standard attribute features can be characterized by task loss.

[0152] Mutual information (MI) loss characterizes the mutual dependence of two variables. The mutual information loss I of two random variables X and Z can be defined by the following formula (3), where H(X) is the marginal entropy and H(X|Z) is the conditional entropy:

[0153] I(X, Z) = H(X) - H(X|Z) (3)

[0154] Mutual information loss is used to represent the difference between domain-invariant features and domain-specific features. By calculating the mutual information loss between domain-invariant features and domain-specific features, and training the neural network based on the mutual information loss, it can help to further distinguish domain-invariant features from domain-specific features, and play a role in forcing the decoupling of domain-invariant features and domain-specific features. It should be understood that the calculation method of the mutual information loss can be selected according to the actual scenario requirements. For example, the mutual information neural estimator (MINE) can be used to obtain the mutual information loss. This application does not strictly limit the specific calculation method of the mutual information loss.

[0155] Step 603: Train the neural network according to the task loss and the mutual information loss.

[0156] As mentioned earlier, the training process of a neural network is actually the process of adjusting the weight vector according to the value of the loss function. The task loss here represents the ability of the domain-invariant features extracted based on the training data to complete the task. If the domain-invariant features cannot correspond to the instances accurately enough, the task loss value will be relatively large. At this time, it is necessary to adjust the weight vector in the neural network so that the domain-invariant features can obtain a lower task loss in the next prediction process. Through the iteration of training, the domain-invariant features extracted by the domain-invariant feature extractor will correspond to the instances more and more accurately.

[0157] The process of training a neural network according to mutual information loss may be a process of training the neural network to reduce the mutual information loss between the domain-invariant features and the domain-specific features, for example, to minimize the mutual information loss. In order to ensure that the domain-invariant features can correspond to the instances more accurately, the mutual information loss between the domain-invariant features and the domain-specific features may be calculated, and the mutual information loss may be used to further improve the accuracy of domain-invariant feature extraction.

[0158] Since the mutual information loss represents the correlation between domain-invariant features and domain-specific features, adjusting the weight vector of the neural network according to the mutual information loss can make the extracted domain-invariant features better distinguishable from domain-specific features, thus forcing feature decoupling. If the mutual information loss is large, it means that the domain-invariant features and domain-specific features are currently more correlated, that is, the features extracted by the domain-invariant feature extractor are likely to still include the information content of the domain-specific features. In this case, the weight vector of the neural network needs to be adjusted to reduce the mutual information loss.

[0159] Since the extraction of domain-invariant features will be trained in the training process based on task loss, the features extracted by the domain-invariant feature extractor may have a certain correlation with the task. Therefore, the training process based on mutual information loss can also be regarded as the process of "eliminating" domain-specific features from domain-invariant features, so that the features extracted by the domain-invariant feature extractor are more and more consistent with the instance with the iteration of training, and the features extracted by the domain-specific feature extractor are less and less relevant to the instance with the iteration of training, thereby achieving feature decoupling of domain-invariant features and domain-specific features. It can be seen that as the training process iterates, the features extracted by the domain-specific feature extractor will become less and less relevant to the instance, that is, they will become closer and closer to the unique features of the representation domain itself. Therefore, the domain-specific feature extractor is also trained in this training process based on mutual information loss.

[0160] It should be understood that the above-mentioned task loss-based training process and mutual information loss-based training process are not necessarily performed simultaneously. In one embodiment of the present application, the mutual information loss-based training process can also be performed after the task loss-based training process starts. The present application does not strictly limit the specific execution order of these two training processes.

[0161] The present application trains the neural network according to task loss and mutual information loss, which not only makes the decomposed domain-invariant features correspond to instances more accurately, but also reduces the mutual information loss between domain-invariant features and domain-specific features during the training process, so as to promote the complete decoupling of domain-invariant features and domain-specific features, and further reduce the influence of domain-specific features on domain-invariant features.

[0162] It should be understood that although the correlation between domain-invariant features and domain-specific features is characterized by mutual information loss as an example above, in other embodiments of the present application, other forms of loss information may also be used to characterize the correlation between domain-invariant features and domain-specific features; then the neural network is trained based on the task loss and the other forms of loss information so that the extracted domain-invariant features can be better distinguished from the domain-specific features, thereby forcing the decoupling of features. In one embodiment of the present application, one or more combinations of the following loss information between domain-invariant features and domain-specific features may be calculated: mutual information loss, metric loss (e.g., L1 distance or L2 distance), loss that measures data distribution (e.g., KL (kullback-leibler) divergence), and Wasserstein distance. The present application does not strictly limit the form of the loss information used to characterize the correlation between domain-invariant features and domain-specific features.

[0163] In one embodiment of the present application, the neural network can be used for domain adaptive learning, and the training data can come from image data of different fields (for example, different styles), such as photorealistic style, comic style, etc. By extracting domain-invariant features and domain-specific features of image data of different styles, and training the neural network based on the task loss obtained by performing tasks using domain-invariant features, the domain-invariant features can be decoupled from the domain-specific features. Since domain-invariant features are used to perform tasks, the neural network obtained by the training method of the present application can adapt itself to processing tasks for images in various different fields through domain adaptive learning, such as target detection / recognition / segmentation, etc., thereby achieving adaptive processing of image data in different fields.

[0164] Although it is mentioned in the above description that the domain-specific feature extractor is also trained in the training process based on mutual information loss, considering that when the extraction accuracy of the domain-specific feature extractor for the domain-specific feature is improved, the training process based on mutual information loss can effectively distinguish the domain-specific features from the domain-invariant features, and the extraction accuracy of the domain-invariant features will be indirectly further improved. Therefore, it is necessary to further improve the extraction accuracy of the domain-specific features in order to indirectly improve the extraction accuracy of the domain-invariant feature extractor through the training process based on mutual information loss.

[0165] In some embodiments of the present application, the domain-specific features extracted by the domain-specific feature extractor may be subjected to domain classification to obtain domain classification loss, and then the neural network may be trained according to the task loss, mutual information loss and domain classification loss.

[0166] For example Fig. 9As shown in the figure, the domain-specific feature extractor can be connected to a domain classifier, and a gradient reversal layer (GRL) is set between the feature extractor and the domain classifier. The domain-specific features extracted by the domain-specific feature extractor are input into the domain classifier to distinguish whether the domain-specific features are really domain-specific features, so as to obtain the domain classification loss, which actually represents the accuracy of the extraction results of the domain-specific feature extractor; then the domain classification loss will pass through the gradient reversal layer during the back propagation process to the domain-specific feature extractor, so that the gradient direction of the domain classification loss during the back propagation process is automatically reversed to "confuse" the domain-specific feature extractor. Since the domain classification loss will be automatically reversed during the back propagation process, the goal of the domain classifier is actually to confuse the domain-specific feature extractor; and the goal of the specific feature extractor is to ensure that the extracted features are domain-specific features. Through this confrontation strategy between the domain classifier and the domain-specific feature extractor, the purpose of ultimately improving the accuracy of the domain-specific feature extractor in extracting domain-specific features is achieved.

[0167] This application helps to extract domain-invariant features from the features of training data by introducing domain classification loss.

[0168] In one embodiment of the present application, in order to further enable the decoupled domain-invariant features and domain-specific features to contain all feature information of the training data so as to improve the completeness and rationality of feature decoupling, initial features can be first extracted from the training data, and the initial features can be decomposed into domain-invariant features and domain-specific features, and then the neural network can be trained to reduce the difference between the information contained in the initial features and the information contained in both the domain-invariant features and the domain-specific features.

[0169] Specifically, if Fig. 9 As shown, after the domain-specific features and the domain-invariant features are extracted, the domain-invariant features and the domain-specific features can be used to reconstruct the initial features to obtain reconstructed features, and then the initial features and the reconstructed features are compared to determine the difference between the information contained in the initial features and the information contained jointly by the domain-invariant features and the domain-specific features, i.e., the reconstruction loss; the reconstruction loss is then used to train the neural network so that the domain-invariant features extracted by the domain-invariant features and the domain-specific features extracted by the domain-specific feature extractor can better cover the feature information of the training data.

[0170] The present application reduces the difference between the information contained in the initial features and the information contained in both the domain-invariant features and the domain-specific features, so that the decoupled domain-invariant features and domain-specific features can contain all the feature information of the training data, thereby improving the completeness and rationality of feature decoupling.

[0171] Combine the following Fig.10 and Fig.11 To further describe the process of extracting domain-invariant features and domain-specific features in the embodiments of the present application.

[0172] Fig.10 This is a schematic diagram of the structure of a neural network provided in one embodiment of the present application. Fig.10 As shown, the neural network includes a first decoupler U1 and a second decoupler U2, and the extraction process of domain-invariant features and domain-specific features is completed through the joint action of the first decoupler U1 and the second decoupler U2. Fig.11 Based on an embodiment of the present application Fig.10 Schematic diagram of the extraction process of domain-invariant features and domain-specific features of the neural network architecture shown in FIG. Fig.11 As shown, the process of extracting the domain-invariant features and domain-specific features may include the following steps:

[0173] Step 1101: extract the first feature of the training data from the training data.

[0174] like Fig.10 As shown, the neural network includes a feature extractor Feature Extractor Used to extract the first feature from the training data First feature It is the feature basis for subsequent domain invariant feature enhancement. It should be understood that the qualifier "first" in the first feature means that the feature extractor It is a “preliminary” feature extraction result of the training data. For example, when the training data is image data, the first feature is actually the result of feature extraction at the image texture level.

[0175] Step 1102: Use the first decoupler U1 to extract preliminary domain-invariant features and preliminary domain-specific features from the first features.

[0176] The first decoupler U1 includes a domain invariant feature extractor and domain-specific feature extractors Used to extract preliminary domain invariant features and preliminary domain-specific features Preliminary domain invariant features and preliminary domain-specific features The respective extraction processes can be expressed by the following formula (4):

[0177]

[0178] In one embodiment of the present application, Fig.10As shown, the first decoupler U1 can be trained using mutual information loss to ensure the extraction accuracy of the preliminary domain invariant features and preliminary domain specific features. As mentioned above, the mutual information (MI) loss represents the mutual dependence of two variables. Here, the mutual information loss represents the preliminary domain invariant features. and preliminary domain-specific features Therefore, adjusting the weight vector of the network structure in the first decoupler U1 according to the mutual information loss can make the extracted preliminary domain invariant features Able to better integrate with preliminary domain-specific features If the mutual information loss is large, it means that the initial domain invariant features and preliminary domain-specific features is more relevant, that is, the current domain invariant feature extractor The extracted features are likely to include preliminary domain-specific features information content, then it is necessary to adjust the weight vector of the network structure of the first decoupler U1 to reduce the mutual information loss.

[0179] In one embodiment of the present application, Fig.10 As shown, in order to further improve the preliminary domain invariant features In order to improve the extraction accuracy, a domain classifier and a gradient reversal layer (GRL) can also be used in the first decoupler U1. to improve domain-specific feature extractors For preliminary domain-specific features The extraction accuracy of the initial domain invariant features can be indirectly improved by combining the training process of mutual information loss. For the purpose of extraction accuracy.

[0180] Step 1103: Fuse the preliminary domain invariant feature with the first feature to obtain a second feature.

[0181] This will be the initial domain invariant feature With the first feature The fusion process to the second feature F 1 It can be expressed by the following formula (5):

[0182]

[0183] It should be understood that the specific method of feature fusion can be selected according to the needs of the actual application scenario. For example, the initial domain invariant features can be fused while keeping the number of channels unchanged. With the first feature Superposition is performed to form a second feature with unchanged channel number; the preliminary domain invariant feature With the first feature The second feature of increasing the number of channels is formed by "joining" in a connected manner. The present application does not strictly limit the specific implementation method of the fusion process.

[0184] Due to the preliminary domain invariant features The domain-invariant feature information corresponding to the instance is included in With the first feature Fusion achieves data enhancement of domain-invariant features at the first feature level, making the first feature It can include more domain-invariant feature information, so that the trained neural network can better adapt to the domain differences in actual application scenarios.

[0185] Step 1104: extract the third feature of the training data from the second feature.

[0186] like Fig.10 As shown, the neural network also includes a feature extractor Feature Extractor For specific completion from the second feature F 1 Extract the third feature The third characteristic It serves as the basis for the subsequent extraction of domain-invariant features and domain-specific features. It should be understood that the qualifier "third" in the third feature means that the third feature is extracted based on the second feature that includes the first feature, and the extraction process will be more refined. For example, when the training data is image data, the third feature may be a feature map extracted to represent the semantic level of the image. The feature extraction process can be expressed by the following formula (6):

[0187]

[0188] Step 1105: Use the second decoupler U2 to extract domain-invariant features and domain-specific features from the third features.

[0189] The second decoupler U2 includes a domain invariant feature extractor and domain-specific feature extractors Used to extract domain invariant features and domain-specific features Domain Invariant Features and domain-specific features The respective extraction processes can be expressed by the following formula (7):

[0190]

[0191] like Fig.10 As shown, the domain invariant features are obtained After that, we can use domain invariant features Perform the task to obtain the task loss and calculate the mutual information (MI) loss between the domain-invariant features and the domain-specific features. As mentioned before, the domain-invariant features Used to perform tasks and obtain task losses, which can improve domain invariant features For the accuracy and completeness of the instance representation related to the task. At the same time, in order to ensure the domain invariant features It can correspond to instances more accurately and can also calculate domain-invariant features during training. and domain-specific features The mutual information loss between them is used to further improve the domain invariant features. In one embodiment of the present application, during the training process of the neural network based on task loss and mutual information loss, the first decoupler U1 is used to extract the preliminary domain invariant features. Domain-invariant feature extractor and / or for extracting preliminary domain-specific features Domain-specific feature extractor for It can also participate in the parameter adjustment process during the training process to ensure that the first decoupler U1 is invariant to the initial domain features. The extraction accuracy is improved, thereby further improving the data enhancement effect of the domain invariant features achieved by the first decoupler U1.

[0192] In one embodiment of the present application, Fig.10 As shown, in order to further improve the domain invariant features To improve the extraction accuracy, domain classifier and gradient reversal layer can also be used in the second decoupler U2. to improve domain-specific feature extractors For domain specific features The extraction accuracy of the domain invariant features can be improved by combining the training process based on mutual information loss. The purpose of extraction accuracy.

[0193] In one embodiment of the present application, in order to further promote the decoupled domain invariant features and domain-specific features It can contain all the feature information of the training data to improve the completeness and rationality of feature decoupling, and can train neural networks to reduce the third feature The information contained and the domain-invariant features and domain-specific features The difference between the information contained in common. Specifically, Fig.10 As shown, domain invariant features and domain-specific features After being extracted, domain-invariant features can be used and domain-specific features For the third feature Reconstruct, obtain the reconstructed features, and then compare the third features and the reconstructed features to determine the third feature The information contained and the domain-invariant features and domain-specific features The difference between the commonly included information is the reconstruction loss. In one embodiment of the present application, the calculation process of the reconstruction loss can be expressed by the following formula (8):

[0194]

[0195] Among them, R represents the reconstruction network, F r is the feature after reconstruction, L recon is the reconstruction loss, which is reflected in the reconstruction feature F r With the third feature The reconstruction loss is used to train the neural network to make the domain invariant features and domain-specific features Can better cover the feature information of training data.

[0196] It should be understood that although the qualifiers such as "first", "second" and "third" are used in the above description, these qualifiers are only used to more clearly explain the technical solution and distinguish similar concepts. These qualifiers themselves cannot be used to limit the scope of protection of this application.

[0197] It can be seen that the embodiment of the present application can adopt the concept of "double-layer domain-invariant feature decoupling" to train the neural network to extract domain-invariant features. By first obtaining the first feature, and decoupling the preliminary domain-invariant feature based on the first decoupler U1, the preliminary domain-invariant feature is fused with the first feature to obtain the second feature, so that the domain-invariant feature information is enhanced at the level of the first feature. Then, the second feature is used to decouple the domain-invariant feature based on the second decoupler U2, and the decoupling accuracy of the domain-invariant feature is further enhanced, which can make the trained neural network have stronger task execution performance and better domain adaptability.

[0198] Combination of the above Fig.10 and Fig.11 , describes in detail the training process associated with neural networks. Fig.10 and Fig.11 It can be seen from the description that in some implementations, the training process of the neural network can include: (1) training related to task loss and domain classification loss; (2) training related to mutual information loss; (3) training related to reconstruction loss.

[0199] The above three types of training can be carried out simultaneously or in stages, which is not limited in the present embodiment. Fig.12 , the training sequence of the above three types of training is illustrated with examples.

[0200] like Fig.12 As shown, the training process of the neural network can be divided into the following three stages.

[0201] The first stage: Control the neural network to perform training related to task loss and domain classification loss. This training stage aims to enable the neural network to learn the ability to decompose domain-invariant features and domain-specific features from the training data. Therefore, this first stage can also be called the feature decomposition stage (abbreviated as stage-fd, where fd stands for feature decomposition).

[0202] The second stage: Control the neural network to perform training related to mutual information loss. This training stage aims to allow the first neural network to learn to increase the difference between domain-invariant features and domain-specific features. Therefore, this second stage can also be called the feature decomposition stage (abbreviated as stage-fs, where fs stands for feature separation). Fig.10 middle The parameters are fixed, or Fig.10 In and The parameters are fixed.

[0203] The third stage: Control the neural network to perform training related to the reconstruction loss. This training stage aims to make the domain-invariant features and domain-specific features decomposed by the neural network contain as much information as possible in the initial features. Therefore, the third stage can also be called the feature reconstruction stage (abbreviated as stage-fr, fr stands for feature reconstruction).

[0204] Dividing the neural network training process into stages can simplify the amount of training in each stage and speed up the convergence of the neural network parameters.

[0205] Fig.13 The figure shows a schematic diagram of the structure of a neural network provided in one embodiment of the present application. The neural network is trained by using the training method provided in the above embodiment of the present application, such as Fig.13 As shown, the neural network 130 includes:

[0206] The first feature extraction layer 1301 is used to extract the first feature based on the input data.

[0207] The first domain invariant feature decoupling layer 1302 is used to extract the first domain invariant feature based on the first feature.

[0208] The feature fusion layer 1303 is used to fuse the first feature and the first domain invariant feature to obtain the second feature.

[0209] The second feature extraction layer 1304 is used to extract the third feature based on the second feature.

[0210] The second domain invariant feature decoupling layer 1305 is used to extract the second domain invariant feature based on the third feature.

[0211] The first domain-invariant feature and the second domain-invariant feature are respectively features that characterize the domain to which the input data belongs, and the first domain-specific feature and the second domain-specific feature are respectively features that are independent of the domain to which the input data belongs.

[0212] It can be seen that although Fig.10 and Fig.11 In the training process shown in FIG, by extracting domain-specific features to calculate mutual information loss and domain classification loss, the neural network has the ability to decompose domain-invariant features and domain-specific features. Fig.13 In the actual use process of the trained neural network shown, it is actually not necessary to extract domain-specific features. After the first feature is extracted by the first feature extraction layer 1301, the first domain-invariant feature is extracted based on the first feature, and the domain-invariant feature is enhanced by fusing with the first feature; then the second domain-invariant feature is further extracted based on the second feature, and the extracted second domain-invariant feature can accurately correspond to the instance, so that the neural network has stronger performance and better domain adaptability when performing specific tasks.

[0213] In one embodiment of the present application, considering that in the scenario of domain adaptive learning, traditional training data often comes from the source domain and / or the target domain, and domain adaptive learning actually solves the cross-domain migration ability of the neural network, to improve the domain generalization ability of the neural network, it is necessary to train not only based on the feature information of the source domain, but also based on the feature information of the target domain. Therefore, when training the neural network, training data of the intermediate domain between the source domain and the target domain can be added to the training data. By generating training data in the intermediate domain to fill the "domain gap" between the source domain and the target domain, the problem of large distribution differences between the training data in the source domain and the training data in the target domain can be alleviated.

[0214] Fig.14 A schematic diagram of a process for obtaining data in an intermediate domain according to an embodiment of the present application. Fig.15The schematic diagram of the principle of obtaining data in the intermediate domain provided by an embodiment of the present application. Fig.14 and Fig.15 As shown, the process of obtaining data of the intermediate domain may include the following steps:

[0215] Step 1401: Acquire data in the source domain and / or data in the target domain.

[0216] The source domain and the target domain are two domains with different data features. The difference in data features between the intermediate domain and any of the source domain and the target domain is smaller than the difference in data features between the source domain and the target domain. The data in the intermediate domain is actually generated by adding perturbations to the data in the source domain and / or the target domain, so the data in the source domain and / or the target domain must be obtained first.

[0217] Step 1402: Input the data of the source domain and / or the data of the target domain into the neural network for training to obtain the gradient information of the loss function.

[0218] Since the data to be generated is in the intermediate domain between the source domain and the target domain, it is necessary to obtain the gradient information of the loss function to guide the subsequent perturbation process to generate data in the intermediate domain.

[0219] Step 1403: According to the gradient information, the data in the source domain and / or the data in the target domain are perturbed to obtain data in the intermediate domain.

[0220] Perturbations are performed on the data in the source domain or the target domain to generate new data, which can then be used as data in the intermediate domain.

[0221] In this application, the introduction of directional information between the source domain and the target domain makes the data perturbation more targeted. The data in the intermediate domain obtained through perturbation can fill the "domain gap" between the source domain and the target domain, alleviating the problem of large distribution differences between the data in the source domain and the data in the target domain. In one embodiment of the present application, the data in the source domain, the data in the target domain, and the data in the intermediate domain can be used as training data to train a neural network, so that the trained neural network can have better domain adaptability.

[0222] In one embodiment of the present application, Fig.15 As shown, the labeled data X in the source domain can be s Input the neural network TNet and train it to obtain the gradient information of the loss function.

[0223] Specifically, the neural network TNet is based on the labeled data X of the target domain l generated by training, which may include feature extractor F T and classifier C T During the training process, the feature extractor FT The extracted feature information P T is input to the classifier C T To obtain the cross entropy loss L for the classification task ce To guide the parameter adjustment process of TNet. Since the neural network TNet is based on the input X l The task loss is calculated and the network parameters are adjusted. Therefore, the neural network TNet is more suitable for the target domain. s Inputting the neural network TNet will generate the first gradient information from the source domain to the target domain. s As an optimizable object, according to the first gradient information of the task loss back propagation, in X s A certain amplitude of gradient perturbation is superimposed on the source domain. The new sample after superimposing this perturbation from the source domain to the target domain can be used as the intermediate domain data, such as Fig.15 As shown in AAT.

[0224] In the present application, the neural network TNet is generated by training with labeled data of the target domain. Therefore, the first gradient information of back propagation obtained after inputting labeled data of the source domain into the neural network can well measure the direction from the source domain to the target domain.

[0225] In another embodiment of the present application, Fig.15 As shown, the unlabeled data X in the target domain can be u Input neural network HNet, due to X u There is no label, so virtual adversarial training can be used to obtain gradient information.

[0226] Specifically, the neural network HNet can be based on the labeled data X in the source domain s Similar to the architecture of TNet, HNet can include a feature extractor F H and classifier C H , during the training process, the feature extractor F H The extracted feature information P H is input to the classifier C H To obtain the cross entropy loss L for the classification task ce To guide the parameter adjustment process of HNet. s Input HNet to calculate the task loss, and update the network parameters of HNet according to the task loss. In a further embodiment, the labeled data X of the target domain l It can also be compared with the labeled data X in the source domain s They are used together to train the neural network HNet to further improve the accuracy of the neural network HNet in performing tasks.

[0227] Unlabeled target domain data X u After inputting HNet, the virtual adversarial training method is used to generate predicted virtual labels, based on which the task loss is calculated, and the second gradient information back-propagated based on the task loss is used in X u A certain amplitude of gradient perturbation is generated on the source domain. The new sample after superimposing this perturbation from the target domain to the source domain can be used as the intermediate domain data, such as Fig.15 As shown in E-VAT.

[0228] In the present application, the neural network HNet is generated by training with labeled data of the source domain and labeled data of the target domain. Therefore, after inputting the unlabeled data of the target domain into the neural network, the second gradient information of back propagation obtained through virtual adversarial training can well measure the direction from the target domain to the source domain.

[0229] In another embodiment of the present application, when both the data in the source domain and the data in the target domain are labeled, the labeled data X in the target domain may be l An auxiliary neural network is fed into the network to obtain the gradient information of the loss function.

[0230] The auxiliary neural network is based on the labeled data X in the source domain. s Generated by training. Since the auxiliary neural network is based on the input X s The task loss is calculated and the network parameters are adjusted. Therefore, the auxiliary neural network is more suitable for the source domain. l Inputting the auxiliary neural network will generate gradient information from the target domain to the source domain. l As an optimizable object, according to the gradient information of the task loss back propagation, in X l A gradient perturbation of a certain amplitude is superimposed on the source domain. The new sample after superimposing this perturbation from the target domain to the source domain can also be used as intermediate domain data.

[0231] It can be seen from this that Fig.15 The embodiment shown actually proposes a "bidirectional adversarial training" method to generate data in the intermediate domain, that is, using the gradient information of the network to guide the perturbation direction of the sample, and using the sample generated after superimposing the perturbation as the data in the intermediate domain. Fig.16 As shown, circles and triangles represent different sample categories, and gradient information can be used to obtain the perturbation direction from the source domain to the target domain (such as Fig.16 Then, perturbations are added to the data in the source domain to generate data in the intermediate domain. At the same time, the gradient information can also be used to obtain the perturbation direction from the target domain to the source domain (as shown in the arrow direction from left to right in the figure). Fig.16), and then add perturbations to the data in the target domain to generate data in the intermediate domain. Specifically, the auxiliary network obtained through training can give the gradient direction from the source domain to the target domain or from the target domain to the source domain, and use the gradient direction to perturb the data in the source domain or the data in the target domain to generate adversarial samples; virtual adversarial training can also be used to generate adversarial samples from the target domain to the source domain, thereby generating adversarial samples in the "domain gap" between the source domain and the target domain in a bidirectional manner to construct the intermediate domain.

[0232] However, it should be understood that, depending on the scenario of domain adaptive learning, only the data with perturbations from the source domain to the target domain can be obtained as the data of the intermediate domain, or only the data with perturbations from the target domain to the source domain can be obtained as the data of the intermediate domain. For example, in the scenario of unsupervised learning, the data in the target domain has no labels, so it is impossible to obtain the data X based on the labeled data in the target domain. l When training the neural network TNet, in this case, only the data with the perturbations from the target domain to the source domain superimposed on them is obtained as the data of the intermediate domain.

[0233] In one embodiment of the present application, the acquired data of the intermediate domain, together with the data of the source domain and the data of the target domain, can be input into Fig. 9 In the neural network shown, the neural network is trained by the neural network training method provided in the embodiment of the present application, thereby realizing the combination of "bidirectional adversarial training" and "double-layer domain-invariant feature decoupling". Since the feature decoupling data includes the data of the intermediate domain, the data of the source domain and the data of the target domain can be effectively supplemented, narrowing the difference between the source domain and the target domain. Using the data of the intermediate domain as training data for feature decoupling training can well improve the domain-invariant feature decoupling capability, so that the domain generalization performance and cross-domain migration capability of the trained neural network are more significantly improved.

[0234] In one embodiment of the present application, in order to further improve the robustness of the trained neural network when performing tasks, Fig.15 As shown, based on the labeled data X in the source domain s After training the generated neural network HNet, it can also be used in X s Generate random noise disturbances near X and superimpose these noise disturbances on X s The adversarial samples in the neighborhood are also input into the neural network as part of the training data for training. In one embodiment of the present application, the adversarial samples in the neighborhood can be input into HNet, and the feature extractor F in HNet H The feature map extracted based on the adversarial sample in the neighborhood is input into the classifier C H To obtain the cross entropy loss L for the classification taskat To guide the adjustment process of the network parameters of HNet, so that HNet can be further trained. In a further embodiment, when the labeled data X in the target domain l Also involved in the training of the neural network HNet, can also be in X l Generate random noise disturbances near X and superimpose these noise disturbances on X l To supplement the adversarial examples in the neighborhood.

[0235] It can be seen that the embodiments of the present application can also generate adversarial samples in the neighborhood based on the data of the source domain and the target domain, so as to effectively supplement the data of the source domain and the target domain, narrow the difference between the source domain and the target domain, and further improve the domain generalization performance and cross-domain migration ability of the trained neural network.

[0236] Fig.17 This is a schematic diagram of the structure of a data processing system provided in one embodiment of the present application. Fig.17 The data processing system 170 shown is used to train a neural network, including: a data acquisition network 1701 and a feature decoupling network 1702.

[0237] The data acquisition network 1701 is used to obtain the gradient information of the loss function based on the first data, and perturb the input data according to the gradient information to obtain the second data, and obtain the adversarial samples that fill the "domain gap" of the first data as the new second data, so that the training process can have better domain adaptability.

[0238] The feature decoupling network 1702 is used to train a neural network according to training data including the second data, so that the neural network learns to decompose domain-invariant features and domain-specific features from the training data.

[0239] In one embodiment of the present application, the feature decoupling network 1702 includes: a first feature extraction layer 17021, used to extract a first feature based on training data; a first domain-invariant feature extraction layer 17022, used to extract a first domain-invariant feature based on the first feature; a first domain-specific feature extraction layer 17023, used to extract a first domain-specific feature based on the first feature; a first mutual information loss acquisition layer 17024, used to obtain a first mutual information loss based on the first domain-invariant feature and the first domain-specific feature; a feature fusion layer 17025, used to fuse the first feature and the first domain-invariant feature to obtain a second feature; a second feature extraction layer 17026, used to extract a third feature based on the second feature; a second domain-invariant feature decoupling layer 17027, used to extract a second domain-invariant feature based on the third feature; a second domain-specific feature extraction layer 17028, used to extract a second domain-specific feature based on the third feature; a second mutual information loss acquisition layer 17029, used to obtain a second mutual information loss based on the second domain-invariant feature and the second domain-specific feature; a task loss acquisition layer 17030, used to perform a task using the second domain-invariant feature to obtain a task loss.

[0240] In an embodiment of the present application, the data processing system 170 may further include: a first domain classifier 17031, configured to perform a classification task based on the first domain-specific feature to obtain a first classification loss; a first gradient reversal layer 17032, configured to reverse the gradient information of the first classification loss;

[0241] And / or, the data processing system 170 may further include: a second domain classifier 17033, used to perform a classification task based on the second domain specific features to obtain a second classification loss; a second gradient reversal layer 17034, used to invert the gradient information of the second classification loss.

[0242] In one embodiment of the present application, the data processing system 170 may further include: a reconstruction loss acquisition layer 17035, which is used to reconstruct the third feature using the second domain invariant feature and the second domain specific feature to obtain a reconstructed feature; and compare the third feature and the reconstructed feature to obtain the reconstruction loss.

[0243] In one embodiment of the present application, the first data includes data from a source domain and / or data from a target domain. The data acquisition network 1701 includes: a first training network generated based on training with labeled data from a target domain; and / or, a second training network generated based on training with labeled data. In one embodiment of the present application, the first training network or the second training network may include a feature extractor and a classifier. During the training process, the feature information extracted by the feature extractor is input into the classifier to obtain the cross entropy loss of the classification task to guide the parameter adjustment process of the first training network or the second training network.

[0244] The specific functions and operations of each module in the above-mentioned data processing system 170 have been described in detail in the training method of the neural network described above, so their repeated description will be omitted here.

[0245] It can be seen from this that Fig.17 The data processing system 170 shown realizes the combination of "adversarial training to fill the domain gap" and "double-layer domain-invariant feature decoupling". Since the feature decoupling training data includes data that can fill the domain gap of the first data, the original training data can be effectively supplemented, and the difference between training data in different fields is narrowed. The data output by the data acquisition network is used for feature decoupling training, which can well improve the domain-invariant feature decoupling ability, so that the domain generalization performance and cross-domain migration ability of the trained neural network are more significantly improved.

[0246] Fig.18 This is a schematic diagram of the structure of a neural network training device provided in one embodiment of the present application. Fig.18 As shown, the neural network training device 180 includes:

[0247] An acquisition module 1801 is configured to acquire training data;

[0248] The training module 1802 is configured to train the neural network using the training data, so that the neural network learns to decompose the domain-invariant features and the domain-specific features from the training data.

[0249] The neural network training device 180 provided in the embodiment of the present application decomposes domain-invariant features and domain-specific features from the training data. Since the neural network obtained by the training method of the present application uses domain-invariant features to perform tasks, the influence of domain-specific features on the neural network is avoided, thereby improving the migration performance of the neural network between different domains.

[0250] In one embodiment of the present application, the training module 1802 is configured to decompose domain-invariant features and domain-specific features from the training data; use the domain-invariant features to perform tasks, obtain task losses, and calculate the mutual information loss between the domain-invariant features and the domain-specific features, where the mutual information loss is used to represent the difference between the domain-invariant features and the domain-specific features; and train the neural network based on the task loss and the mutual information loss.

[0251] In one embodiment of the present application, the training module 1802 is further configured to use domain-specific features to perform domain classification to obtain domain classification loss; and train the neural network according to the task loss, mutual information loss and domain classification loss.

[0252] In one embodiment of the present application, the training module 1802 is further configured to extract initial features from the training data; decompose the initial features into domain-invariant features and domain-specific features; and train the neural network to reduce the difference between the information contained in the initial features and the information contained in both the domain-invariant features and the domain-specific features.

[0253] In one embodiment of the present application, the training module 1802 is configured to reconstruct the initial features using the domain-invariant features and the domain-specific features to obtain the reconstructed features; and compare the initial features and the reconstructed features to determine the difference between the information contained in the initial features and the information contained jointly by the domain-invariant features and the domain-specific features.

[0254] In an embodiment of the present application, the training module 1802 is further configured to reconstruct the initial features using the domain-invariant features and the domain-specific features to obtain reconstructed features, wherein the domain-invariant features and the domain-specific features are features decomposed from the initial features; compare the initial features and the reconstructed features to obtain a reconstruction loss, where the reconstruction loss is used to characterize the difference between the information contained in the initial features and the information contained in the domain-invariant features and the domain-specific features.

[0255] Among them, the training module is configured to perform the first stage training of the neural network according to the task loss; perform the second stage training of the neural network according to the mutual information loss; and perform the third stage training of the neural network according to the reconstruction loss.

[0256] In one embodiment of the present application, the neural network includes a first decoupler and a second decoupler, and the training module 1602 is configured to extract a first feature of the training data from the training data; use the first decoupler to extract a preliminary domain-invariant feature and a preliminary domain-specific feature from the first feature; fuse the preliminary domain-invariant feature with the first feature to obtain a second feature; extract a third feature of the training data from the second feature; and use the second decoupler to extract a domain-invariant feature and a domain-specific feature from the third feature.

[0257] In one embodiment of the present application, the training module 1802 is further configured to train the neural network to reduce the difference between the information included in the third feature and the information included in both the domain-invariant feature and the domain-specific feature.

[0258] Fig.19 This is a schematic diagram of the structure of a data acquisition device provided in one embodiment of the present application. Fig.19 As shown, the data acquisition device 190 includes:

[0259] The data acquisition module 1901 is configured to acquire data of a source domain and / or data of a target domain; wherein the source domain and the target domain are two domains with different data features, and the difference in data features between the intermediate domain and any domain of the source domain and the target domain is smaller than the difference in data features between the source domain and the target domain;

[0260] A gradient information acquisition module 1902 is configured to input data from a source domain and / or data from a target domain into a neural network for training to obtain gradient information of a loss function;

[0261] The intermediate domain data generating module 1903 is configured to perturb the data in the source domain and / or the data in the target domain according to the gradient information to obtain the data in the intermediate domain.

[0262] In one embodiment of the present application, the gradient information acquisition module 1902 is configured to input the labeled data of the source domain into the first neural network for training to obtain first gradient information, wherein the first neural network is generated based on the labeled data training of the target domain.

[0263] In one embodiment of the present application, the gradient information acquisition module 1902 is configured to input the unlabeled data of the target domain into the second neural network, train it in a virtual adversarial training manner, and obtain second gradient information, wherein the second neural network is generated based on the labeled data training.

[0264] The specific functions and operations of each module in the above-mentioned neural network training device 180 / 190 have been described in detail in the neural network training method described above, so their repeated description will be omitted here.

[0265] Fig. 20 A schematic diagram of the hardware structure of a neural network training device provided in one embodiment of the present application. Fig. 20 The neural network training device 2000 shown (the device 2000 may be a computer device) includes a memory 2001, a processor 2002, a communication interface 2003 and a bus 2004. The memory 2001, the processor 2002 and the communication interface 2003 are connected to each other through the bus 2004.

[0266] The memory 2001 may be a read only memory (ROM), a static storage device, a dynamic storage device or a random access memory (RAM). The memory 2001 may store a program. When the program stored in the memory 2001 is executed by the processor 2002, the processor 2002 and the communication interface 2003 are used to execute the various steps of the neural network training method of the embodiment of the present application.

[0267] Processor 2002 can adopt a general central processing unit (CPU), a microprocessor, an application specific integrated circuit (ASIC), a graphics processing unit (GPU) or one or more integrated circuits to execute relevant programs to implement the functions required to be performed by the units in the neural network training device of the embodiment of the present application, or to execute the neural network training method of the method embodiment of the present application.

[0268] Processor 2002 may also be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the training method of the neural network of the present application may be completed by an integrated logic circuit of hardware or software instructions in processor 2002. The above-mentioned processor 2002 may also be a general-purpose processor, a digital signal processor (digital signal processing, DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (field programmable gate array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application may be implemented or executed. A general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The steps of the method disclosed in conjunction with the embodiments of the present application may be directly embodied as being executed by a hardware decoding processor, or may be executed by a combination of hardware and software modules in a decoding processor. The software module may be located in a mature storage medium in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 2001, and the processor 2002 reads the information in the memory 2001, and combines its hardware to complete the functions required to be performed by the units included in the neural network training device of the embodiment of the present application, or executes the neural network training method of the method embodiment of the present application.

[0269] The communication interface 2003 uses a transceiver device such as, but not limited to, a transceiver to implement communication between the device 2000 and other devices or a communication network. For example, training data can be obtained through the communication interface 2003.

[0270] The bus 2004 may include a path for transmitting information between various components of the device 2000 (eg, the memory 2001 , the processor 2002 , and the communication interface 2003 ).

[0271] It should be understood that the acquisition module 1801 and the training module 1802 in the neural network training device 180, or the data acquisition module 1901, the gradient information acquisition module 1902, the intermediate domain data generation module 1903, and the training execution module 1904 in the neural network training device 190 can be equivalent to the processor 2002.

[0272] It should be noted that although Fig. 20 The device 2000 shown only shows a memory, a processor, and a communication interface, but in the specific implementation process, those skilled in the art should understand that the device 2000 also includes other devices necessary for normal operation. At the same time, according to specific needs, those skilled in the art should understand that the device 2000 may also include hardware devices for implementing other additional functions. In addition, those skilled in the art should understand that the device 2000 may also only include the devices necessary to implement the embodiments of the present application, and does not necessarily include Fig. 20 All devices shown in .

[0273] It can be understood that the device 2000 is equivalent to Figure 2 The training device 220 in the embodiment disclosed herein. A person of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0274] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0275] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0276] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0277] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0278] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory, a random access memory, a magnetic disk, or an optical disk.

[0279] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A neural network training method, characterized in that: include: Acquire training data, where the training data includes image data or text data, and the training data includes intermediate domain data, where the intermediate domain data is generated based on source domain data and / or target domain data; training a neural network using the training data so that the neural network learns to decompose domain-invariant features and domain-specific features from the training data; The domain-specific features are features that characterize the domain to which the training data belongs, and the domain-invariant features are features that are irrelevant to the domain to which the training data belongs. The using the training data to train the neural network includes: Decomposing domain-invariant features and domain-specific features from the features of the training data; Performing a task using the domain-invariant feature to obtain a task loss, and calculating a mutual information loss between the domain-invariant feature and the domain-specific feature, wherein the task loss is used to characterize the gap between a result obtained by performing the task using the domain-invariant feature and the task label, and the mutual information loss is used to represent the difference between the domain-invariant feature and the domain-specific feature; The neural network is trained according to the task loss and the mutual information loss.

2. The method according to claim 1, characterized in that Also includes: Using the domain-specific features to perform domain classification, obtaining a domain classification loss; Wherein, training the neural network according to the task loss and the mutual information loss includes: The neural network is trained according to the task loss, the mutual information loss, and the domain classification loss.

3. The method according to claim 1 or 2, characterized in that: Decomposing domain-invariant features and domain-specific features from the features of the training data includes: Extracting initial features from the training data; Decomposing the initial features into the domain-invariant features and the domain-specific features, Wherein, the method further comprises: The neural network is trained to reduce the difference between information contained in the initial features and information contained jointly by the domain-invariant features and the domain-specific features.

4. The method according to claim 3, characterized in that Before training the neural network to reduce the difference between the information included in the initial features and the information included in both the domain-invariant features and the domain-specific features, the method further includes: Reconstructing the initial features using the domain-invariant features and the domain-specific features to obtain reconstructed features; The initial features and the reconstructed features are compared to determine the difference between information contained in the initial features and information contained jointly by the domain-invariant features and the domain-specific features.

5. The method according to claim 1, characterized in that Also includes: Reconstructing the initial feature using the domain-invariant feature and the domain-specific feature to obtain a reconstructed feature, wherein the domain-invariant feature and the domain-specific feature are features decomposed from the initial feature; comparing the initial features and the reconstructed features to obtain a reconstruction loss, wherein the reconstruction loss is used to characterize the difference between information contained in the initial features and information contained jointly by the domain-invariant features and the domain-specific features, Wherein, training the neural network according to the task loss and the mutual information loss includes: Performing a first phase of training on the neural network according to the task loss; According to the mutual information loss, the neural network is trained in the second stage, Wherein, the method further comprises: The neural network is trained in the third stage according to the reconstruction loss.

6. The method according to claim 1 or 2, characterized in that: The neural network includes a first decoupler and a second decoupler, and the decomposing domain-invariant features and domain-specific features from the features of the training data includes: extracting a first feature of the training data from the training data; extracting preliminary domain-invariant features and preliminary domain-specific features from the first features using the first decoupler; fusing the preliminary domain invariant feature with the first feature to obtain a second feature; Extracting a third feature of the training data from the second feature; A second decoupler is employed to extract the domain-invariant feature and the domain-specific feature from the third feature.

7. The method according to claim 6, characterized in that Also includes: The neural network is trained to reduce the difference between information contained in the third feature and information contained jointly by the domain-invariant feature and the domain-specific feature.

8. The method according to any one of claims 1-2 or 4-5 or 7, characterized in that: The neural network is used for domain adaptive learning, and the training data includes image data in different fields.

9. A neural network training device, characterized in that: include: An acquisition module is configured to acquire training data, wherein the training data includes image data or text data, and the training data includes intermediate domain data, and the intermediate domain data is generated based on source domain data and / or target domain data; a training module configured to train a neural network using the training data so that the neural network learns to decompose domain-invariant features and domain-specific features from the training data; The domain-specific features are features that characterize the domain to which the training data belongs, and the domain-invariant features are features that are irrelevant to the domain to which the training data belongs; The training module is configured to decompose domain-invariant features and domain-specific features from the features of the training data; use the domain-invariant features to perform tasks, obtain task losses, and calculate mutual information losses between the domain-invariant features and the domain-specific features, wherein the task loss is used to characterize the gap between the results obtained by performing the tasks using the domain-invariant features and the task labels, and the mutual information loss is used to represent the differences between the domain-invariant features and the domain-specific features; and train the neural network based on the task loss and the mutual information loss.

10. The device according to claim 9, characterized in that The training module is further configured to use the domain-specific features to perform domain classification to obtain domain classification loss; and train the neural network according to the task loss, the mutual information loss and the domain classification loss.

11. The device according to claim 9 or 10, characterized in that The training module is further configured to extract initial features from the training data; decompose the initial features into the domain-invariant features and the domain-specific features; The neural network is trained to reduce the difference between information contained in the initial features and information contained jointly by the domain-invariant features and the domain-specific features.

12. The device according to claim 11, characterized in that The training module is configured to reconstruct the initial feature using the domain-invariant feature and the domain-specific feature to obtain a reconstructed feature; and compare the initial feature with the reconstructed feature to determine the difference between the information contained in the initial feature and the information jointly contained in the domain-invariant feature and the domain-specific feature.

13. The device according to claim 12, characterized in that The training module is further configured to reconstruct the initial feature using the domain-invariant feature and the domain-specific feature to obtain a reconstructed feature, wherein the domain-invariant feature and the domain-specific feature are features decomposed from the initial feature; compare the initial feature and the reconstructed feature to obtain a reconstruction loss, wherein the reconstruction loss is used to characterize the difference between information contained in the initial feature and information contained jointly by the domain-invariant feature and the domain-specific feature, Wherein, the training module is configured to perform a first stage of training on the neural network according to the task loss; perform a second stage of training on the neural network according to the mutual information loss; and perform a third stage of training on the neural network according to the reconstruction loss.

14. The device according to claim 9 or 10, characterized in that The neural network includes a first decoupler and a second decoupler, and the training module is configured to extract a first feature of the training data from the training data; extract a preliminary domain-invariant feature and a preliminary domain-specific feature from the first feature using the first decoupler; The preliminary domain-invariant feature is fused with the first feature to obtain a second feature; a third feature of the training data is extracted from the second feature; and a second decoupler is used to extract the domain-invariant feature and the domain-specific feature from the third feature.

15. The device according to claim 14, characterized in that The training module is further configured to train the neural network to reduce the difference between information included in the third feature and information jointly included in the domain-invariant feature and the domain-specific feature.

16. A neural network training device, comprising: Memory, used to store programs; A processor, configured to execute the program stored in the memory. When the program stored in the memory is executed, the processor is configured to execute the neural network training method according to any one of claims 1 to 8.

17. A data processing system, characterized in that: include: A data acquisition network, used to acquire gradient information of a loss function based on first data, and to perturb the first data according to the gradient information to acquire second data, wherein the first data includes image data or text data; a feature decoupling network, configured to train a neural network using training data including the second data, so that the neural network learns to decompose domain-invariant features and domain-specific features from the training data; The domain-specific features are features that characterize the domain to which the training data belongs, and the domain-invariant features are features that are irrelevant to the domain to which the training data belongs; The feature decoupling network comprises: A first feature extraction layer, used to extract a first feature based on the training data; A first domain invariant feature extraction layer, configured to extract a first domain invariant feature based on the first feature; a first domain-specific feature extraction layer, configured to extract a first domain-specific feature based on the first feature; a first mutual information loss acquisition layer, configured to acquire a first mutual information loss based on the first domain invariant feature and the first domain specific feature; A feature fusion layer, used for fusing the first feature and the first domain invariant feature to obtain a second feature; A second feature extraction layer, used for extracting a third feature based on the second feature; A second domain invariant feature decoupling layer, used for extracting second domain invariant features based on the third features; A second domain specific feature extraction layer, configured to extract a second domain specific feature based on the third feature; A second mutual information loss acquisition layer, configured to acquire a second mutual information loss based on the second domain invariant feature and the second domain specific feature; The task loss acquisition layer is used to perform a task using the second domain invariant features to obtain the task loss.

18. The data processing system according to claim 17, characterized in that: Further including: a first domain classifier, configured to perform a classification task based on the first domain specific feature to obtain a first classification loss; A first gradient reversal layer, used to reverse the gradient information of the first classification loss; and / or, a second domain classifier, configured to perform a classification task based on the second domain specific feature to obtain a second classification loss; The second gradient reversal layer is used to reverse the gradient information of the second classification loss.

19. The data processing system according to claim 17 or 18, characterized in that: Further including: A reconstruction loss acquisition layer, used to reconstruct the third feature using the second domain invariant feature and the second domain specific feature to obtain a reconstructed feature; The third feature is compared with the reconstructed feature to obtain a reconstruction loss.

20. The data processing system according to claim 17 or 18, characterized in that: The first data includes data in a source domain and / or data in a target domain, wherein the data acquisition network includes: A first training network generated by training based on labeled data in the target domain; and / or, The second training network is generated based on the labeled data.

Images