Incremental learning method, device, electronic device and machine-readable storage medium

Abstract

The present application provides an incremental learning method, device, electronic device and machine-readable storage medium, the method comprising: performing domain discrimination on input features according to a domain discrimination branch network in an incremental learning model to obtain a domain discrimination result; performing fusion processing on the output features of the backbone network and the output features of the new knowledge learning branch network according to the domain discrimination result to obtain a fusion output feature; performing prediction processing according to the fusion output feature. The method can optimize the performance of the incremental learning model.

Description

Technical Field

[0001] The present application relates to the field of basic machine learning algorithms, and in particular to an incremental learning method, device, electronic device, and machine-readable storage medium. Background Art

[0002] Incremental Learning, also known as Lifelong Learning, aims to enable machine learning models to have the ability to continuously learn and evolve.

[0003] The parameter fixing method is a main method of incremental learning, which aims to avoid catastrophic forgetting of old knowledge in incremental training. The parameter fixing method avoids catastrophic forgetting by fixing the parameters of the backbone network and then expanding new branches to train the new data.

[0004] In the prediction stage, in order to determine whether the input image is predicted using the backbone network or the new branch, an independent model is required to determine whether the input image comes from the basic data set or the new data set, and determine whether to use the backbone network or the new branch for prediction based on the judgment result. Summary of the invention

[0005] In view of this, the present application provides an incremental learning method, device, electronic device and machine-readable storage medium to optimize the effect of incremental learning.

[0006] Specifically, the present application is implemented through the following technical solutions:

[0007] According to a first aspect of an embodiment of the present application, there is provided an incremental learning method, comprising:

[0008] Performing domain discrimination on input features according to the domain discrimination branch network in the incremental learning model to obtain a domain discrimination result; the incremental learning model includes a backbone network, a new knowledge learning branch network and the domain discrimination branch network, and for each convolution layer of the backbone network, the new knowledge learning branch network includes a new knowledge learning branch corresponding to the convolution layer of the backbone network, and the domain discrimination branch network includes a domain discrimination branch corresponding to the convolution layer of the backbone network;

[0009] According to the domain discrimination result, the output features of the backbone network and the output features of the new knowledge learning branch network are fused to obtain fused output features;

[0010] Prediction processing is performed based on the fused output features.

[0011] According to a second aspect of an embodiment of the present application, there is provided an incremental learning device, comprising:

[0012] A domain discrimination unit, configured to perform domain discrimination on input features according to a domain discrimination branch network in an incremental learning model to obtain a domain discrimination result; the incremental learning model includes a backbone network, a new knowledge learning branch network, and the domain discrimination branch network; for each convolution layer of the backbone network, the new knowledge learning branch network includes a new knowledge learning branch corresponding to the convolution layer of the backbone network, and the domain discrimination branch network includes a domain discrimination branch corresponding to the convolution layer of the backbone network;

[0013] A feature fusion unit, used to fuse the output features of the backbone network and the output features of the new knowledge learning branch network according to the domain discrimination result to obtain a fused output feature;

[0014] A prediction unit is used to perform prediction processing based on the fused output features.

[0015] According to a third aspect of an embodiment of the present application, there is provided an electronic device, comprising a processor and a memory, wherein the memory stores machine executable instructions that can be executed by the processor, and the processor is used to execute the machine executable instructions to implement the method provided in the first aspect.

[0016] According to a fourth aspect of an embodiment of the present application, a machine-readable storage medium is provided, wherein the machine-readable storage medium stores machine-executable instructions, and when the machine-executable instructions are executed by a processor, the method provided in the first aspect is implemented.

[0017] The technical solution provided by this application can at least bring the following beneficial effects:

[0018] By adding a domain discrimination branch network for domain discrimination in the incremental learning model, and performing domain discrimination on the input features according to the domain discrimination branch network in the incremental learning model, a domain discrimination result is obtained. Compared with the traditional solution of using a separate classification network to discriminate whether the input features come from the basic data domain or the newly added data domain, by integrating the domain discrimination branch network inside the incremental learning model, end-to-end training can be achieved with a small number of added parameters. When the domain discrimination result is obtained, the output features of the backbone network and the output features of the new knowledge learning branch network are fused to obtain fused output features, and prediction processing is performed based on the fused output features. Compared with the traditional solution of either selecting backbone features or branch features, the dependence of the prediction processing on the domain discrimination result is reduced, the attenuation of the incremental learning performance when the domain discrimination result is wrong is reduced, and the performance of the incremental learning model is optimized. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 is a flowchart of an incremental learning method shown in an exemplary embodiment of the present application;

[0020] Figure 2 It is a structural schematic diagram of a modulator and a domain discrimination branch network shown in an exemplary embodiment of the present application;

[0021] Figure 3 is a structural schematic diagram of an incremental learning device shown in an exemplary embodiment of the present application;

[0022] Figure 4 is a schematic diagram of the structure of another incremental learning device shown in an exemplary embodiment of the present application;

[0023] Figure 5 It is a schematic diagram of the hardware structure of an electronic device shown in an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0024] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0025] The terms used in this application are for the purpose of describing specific embodiments only and are not intended to limit this application. The singular forms "a", "said" and "the" used in this application and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings.

[0026] In order to enable those skilled in the art to better understand the technical solutions provided by the embodiments of the present application and to make the above-mentioned purposes, features and advantages of the embodiments of the present application more obvious and understandable, the technical solutions in the embodiments of the present application are further described in detail below in conjunction with the accompanying drawings.

[0027] It should be noted that the serial numbers of the steps in the embodiments of the present application do not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0028] See also Figure 1 , is a flow chart of an incremental learning method provided in an embodiment of the present application, such as Figure 1 As shown, the incremental learning method may include the following steps:

[0029] Step S100, performing domain discrimination on the input features according to the domain discrimination branch network in the incremental learning model to obtain a domain discrimination result; the incremental learning model includes a backbone network, a new knowledge learning branch network and a domain discrimination branch network. For each convolution layer of the backbone network, the new knowledge learning branch network includes a new knowledge learning branch corresponding to the convolution layer of the backbone network, and the domain discrimination branch network includes a domain discrimination branch corresponding to the convolution layer of the backbone network.

[0030] In an embodiment of the present application, in order to achieve end-to-end training of the model, an independent classifier is no longer used to distinguish whether the input features come from the basic data domain or the newly added data domain. Instead, a domain discrimination branch network is added to the incremental learning model to distinguish whether the input features come from the basic data domain or the newly added data domain through the domain discrimination branch network.

[0031] That is, the incremental learning model can include a backbone network, a new knowledge learning branch network, and a domain discrimination branch network.

[0032] Among them, the backbone network stores the knowledge of basic data, the new knowledge learning branch network stores the knowledge of newly added data, and the domain discrimination branch network is used to discriminate whether the input features come from the basic data domain or the newly added data domain.

[0033] For example, in order to allow the model to learn new knowledge on the newly added data without forgetting the knowledge of the basic data, there is a new branch on each convolutional layer of the backbone network, which includes a new knowledge learning branch corresponding to the convolutional layer and a domain discrimination branch.

[0034] Step S110: Based on the domain discrimination result, the output features of the backbone network and the output features of the new knowledge learning branch network are fused to obtain fused output features.

[0035] Step S120: perform prediction processing based on the fused output features.

[0036] In the embodiment of the present application, considering that in the traditional incremental learning scheme, when it is identified that the input feature comes from the basic data domain, the backbone network is selected for prediction processing; when it is identified that the input feature comes from the newly added data domain, the branch network is selected for prediction processing, that is, the accuracy of the prediction is highly dependent on the domain discrimination result. If the domain discrimination result is wrong, it will cause the incremental learning performance to be significantly attenuated.

[0037] In order to reduce the dependence on the domain discrimination results and reduce the attenuation of incremental learning performance when the domain discrimination results are wrong, the prediction processing is no longer based on the output features of the backbone network or the output features of the branch network alone. Instead, the prediction processing is based on the fused output features obtained by fusing the output features of the backbone network and the output features of the branch network.

[0038] Accordingly, when the domain discrimination result is obtained in the manner described in step S100, the output features of the backbone network and the output features of the new knowledge learning branch network can be fused based on the domain discrimination result to obtain fused output features, and prediction processing can be performed based on the fused output features.

[0039] It can be seen that in Figure 1 In the method flow shown, a domain discrimination branch network for performing domain discrimination is added to the incremental learning model, and domain discrimination is performed on the input features according to the domain discrimination branch network in the incremental learning model to obtain a domain discrimination result. Compared with the traditional solution in which a separate classification network is used to discriminate whether the input features come from the basic data domain or the newly added data domain, by integrating the domain discrimination branch network inside the incremental learning model, end-to-end training can be achieved with a smaller number of added parameters. When the domain discrimination result is obtained, the output features of the backbone network and the output features of the new knowledge learning branch network are fused to obtain fused output features, and prediction processing is performed based on the fused output features. Compared with the traditional solution in which either backbone features or branch features are selected, the dependence of the prediction processing on the domain discrimination result is reduced, the attenuation of the incremental learning performance when the domain discrimination result is wrong is reduced, and the performance of the incremental learning model is optimized.

[0040] In some embodiments, in step S100, performing domain discrimination on the input features according to the domain discrimination branch network in the incremental learning model may include:

[0041] For the target convolution layer of the backbone network, the domain discrimination branch corresponding to the target convolution layer in the domain discrimination branch network is used to perform domain discrimination on the input features of the target convolution layer to obtain the target domain discrimination result; the target convolution layer is any convolution layer of the backbone network;

[0042] In step S110, based on the domain discrimination result, the output features of the backbone network and the output features of the new knowledge learning branch network are fused to obtain fused output features, which may include:

[0043] According to the target domain discrimination result, the first output feature of the target convolutional layer and the second output feature of the new knowledge learning branch corresponding to the target convolutional layer in the new knowledge learning branch network are fused to obtain the fused output feature corresponding to the target convolutional layer.

[0044] For example, in order to ensure that the incremental learning model learns new knowledge on the newly added data and does not forget the knowledge of the basic data, there is a corresponding new knowledge learning branch in the new knowledge learning branch network on each convolutional layer of the backbone network. The expressive power of the backbone network is combined with the expressive power of the new knowledge learning branch network itself to recombine new features to learn the knowledge of the new data.

[0045] For any convolutional layer of the backbone network (referred to as the target convolutional layer in this article), domain discrimination can be performed on the input features of the target convolutional layer according to the domain discrimination branch corresponding to the target convolutional layer in the domain discrimination branch network to obtain the corresponding domain discrimination result (referred to as the target domain discrimination result in this article).

[0046] Exemplarily, for any convolutional layer of the backbone network (i.e., the target convolutional layer mentioned above), the output features of the target convolutional layer of the backbone network (referred to as the first output features in this article) and the output features of the new knowledge learning branch corresponding to the target convolutional layer in the new knowledge learning branch network (referred to as the second output features in this article) can be fused according to the target domain discrimination results to obtain the fused output features corresponding to the target convolutional layer.

[0047] For example, for non-last convolutional layers of the backbone network, the corresponding fused output features can be used as input features of the next convolutional layer; for the last convolutional layer of the backbone network, the corresponding fused output features can be used for prediction processing.

[0048] In an example, the output features of the target convolution layer and the output features of the new knowledge learning branch corresponding to the target convolution layer in the new knowledge learning branch network are fused according to the target domain discrimination result to obtain the fused output features corresponding to the target convolution layer, which may include:

[0049] According to the target domain discrimination result, determine a first weight of the first output feature and a second weight of the second output feature; wherein, when it is determined according to the target domain discrimination result that the distance between the input feature and the basic data domain is less than the distance between the input feature and the newly added data domain, the first weight is greater than the second weight; when it is determined according to the target domain discrimination result that the distance between the input feature and the basic data domain is greater than the distance between the input feature and the newly added data domain, the first weight is less than the second weight;

[0050] According to the first weight and the second weight, the first output feature and the second output feature are weighted and summed to obtain the fused output feature corresponding to the target convolution layer.

[0051] Exemplarily, considering that in actual scenarios, the input image in the incremental learning process usually has the features of both the basic data domain and the newly added data domain, therefore, when performing domain discrimination on the input features, the domain discrimination results may no longer be limited to characterizing whether the input features come from the basic data domain or the newly added data domain, but can be used to characterize the distance between the input feature and the basic data domain (assuming the first distance), and the distance between the input feature and the newly added data domain (assuming the second distance), and the size relationship between the two distances (the first distance and the second distance), that is, to determine whether the input feature is closer to the basic data domain or the newly added data domain.

[0052] For example, the domain discrimination result can be a number between 0 and 1, where 0 represents the basic data domain and 1 represents the newly added data domain. When the domain discrimination result is closer to 0, it indicates that the input feature is closer to the basic data domain. When the domain discrimination result is closer to 1, it indicates that the input feature is closer to the newly added data domain.

[0053] Exemplarily, the output features of the trunk network and the output features of the new knowledge learning branch network can be fused by taking a weighted sum of the output features of the trunk network and the output features of the new knowledge learning branch network to obtain fused output features.

[0054] In order to improve the rationality and accuracy of feature fusion, when performing weighted summation on the output features of the backbone network and the output features of the new knowledge learning branch network, for input features that are closer to the basic data domain, the weight of the output features of the backbone network is higher during the fusion process; for input features that are closer to the newly added data domain, the weight of the output features of the new knowledge learning branch network is higher during the fusion process.

[0055] Accordingly, when the first output feature and the second output feature are fused, the weight of the first output feature (referred to as the first weight in this article) and the weight of the second output feature (referred to as the second weight in this article) can be determined based on the target domain discrimination result, and the first output feature and the second output feature are weightedly summed based on the first weight and the second weight to obtain the fused output feature corresponding to the target convolutional layer.

[0056] Exemplarily, when the first distance is smaller than the second distance, the first weight is larger than the second weight; when the first distance is larger than the second distance, the first weight is smaller than the second weight.

[0057] In one example, during the training process, the backbone network and the new knowledge learning branch network are feedback tuned based on the prediction loss of the incremental learning model, as well as the first weight and the second weight.

[0058] For example, since the incremental learning model obtains output features by weighted summing up the output features of the backbone network and the new knowledge learning branch network when making predictions, during the training process, when the model is feedback tuned based on the prediction loss of the incremental learning model, the feedback loss can be inversely processed based on the weights of the backbone network and the new knowledge learning branch network during the weighted summation.

[0059] Accordingly, the backbone network and the new knowledge learning branch network can be feedback tuned according to the prediction loss of the reinforcement learning model, as well as the first weight and the second weight.

[0060] For example, assuming that the prediction loss is 1, the first weight is 0.2, and the second weight is 0.8, when performing loss feedback, the feedback loss of the backbone network can be 0.2, and the feedback loss of the new knowledge learning branch network can be 0.8.

[0061] In some embodiments, during the training process, the loss value of the trunk network predicting the training samples is used as a supervisory signal to train the domain discrimination branch network.

[0062] Exemplarily, it is considered that the loss value of the backbone network predicting the training sample represents the distance between the training sample and the basic data domain.

[0063] For example, the larger the loss value of the backbone network for predicting the training sample, the larger the distance between the training sample and the basic data domain; the smaller the loss value of the backbone network for predicting the training sample, the smaller the distance between the training sample and the basic data domain.

[0064] The domain discrimination results can also characterize the distance between the training samples and the basic data domain and the distance between the training samples and the newly added data domain.

[0065] For any training sample, the smaller the loss value of the backbone network in predicting the training sample, the closer the domain discrimination result of the domain discrimination branch network for the training sample should be to 0 (taking the domain discrimination result of 0 as an example, indicating that the input feature comes from the basic data domain).

[0066] Therefore, the loss value of the backbone network predicting the training samples can be used as a supervisory signal to train the domain discrimination branch network to improve the performance of the domain discrimination branch network.

[0067] Exemplarily, when the value range of the domain discrimination result is [0, 1], the loss value of the trunk network predicting the training sample can be normalized, and the normalized loss value is used as a supervisory signal to train the domain discrimination branch network.

[0068] In one example, the above-mentioned training of the domain discrimination branch network using the loss value of the prediction of the training sample by the backbone network as the supervision signal may include:

[0069] For any training sample, the loss of the domain discrimination branch network corresponding to the training sample is determined based on the loss value predicted by the backbone network for the training sample and the domain discrimination result of the domain discrimination branch network for the training sample.

[0070] As an example, based on the loss value predicted by the backbone network for the training sample and the domain discrimination result of the domain discrimination branch network for the training sample, determining the loss of the domain discrimination branch network corresponding to the training sample may include:

[0071] Normalize the loss value of the backbone network predicting the training sample to obtain a normalized loss value;

[0072] The loss of the domain discrimination branch network corresponding to the training sample is determined based on the absolute value of the difference between the normalized loss value and the domain discrimination result of the domain discrimination branch network for domain discrimination of the training sample; wherein the value range of the domain discrimination result is [0, 1]. For any training sample, the smaller the domain discrimination result of the domain discrimination branch network for domain discrimination of the training sample, the smaller the distance between the training sample and the basic data domain; the larger the domain discrimination result of the domain discrimination branch network for domain discrimination of the training sample, the smaller the distance between the training sample and the newly added data domain.

[0073] Exemplarily, taking the example in which the value range of the above domain discrimination result is [0, 1], that is, 0 represents the basic data domain, and 1 represents the newly added data domain. The closer the domain discrimination result is to 0, the closer the input feature is to the basic data domain. The closer the domain discrimination result is to 1, the closer the input feature is to the newly added data domain.

[0074] For any training sample, the loss value of the backbone network predicting the training sample can be normalized, and the loss of the domain discrimination branch network corresponding to the training sample can be determined based on the normalized loss value and the domain discrimination result of the domain discrimination branch network performing domain discrimination on the training sample.

[0075] Exemplarily, the larger the absolute value of the difference between the normalized loss value and the domain discrimination result, the greater the loss of the domain discrimination branch network corresponding to the training sample; the smaller the absolute value of the difference between the normalized loss value and the domain discrimination result, the smaller the loss of the domain discrimination branch network corresponding to the training sample.

[0076] In some embodiments, during the training process, for any training sample, when it is determined that the training sample belongs to the basic data domain based on the domain discrimination result of the domain discrimination branch network on the training sample, the main network is updated with a first learning rate, and the new knowledge learning branch network is updated with a second learning rate, and the second learning rate is less than the first learning rate;

[0077] When the domain discrimination result of the training sample by the domain discrimination branch network is determined to belong to the newly added data domain, the main network is updated with the third learning rate, and the new knowledge learning branch network is updated with the fourth learning rate, and the third learning rate is less than the fourth learning rate.

[0078] Exemplarily, in order to optimize the performance of the incremental learning model, during the training process, for training samples from the basic data domain, the backbone network can update the network at a larger learning rate, and the new knowledge learning branch network can update the network at a smaller learning rate to consolidate the basic knowledge.

[0079] For training samples from the newly added data domain, the backbone network can update the network at a smaller learning rate, and the new knowledge learning branch network can update the network at a larger learning rate to better learn new knowledge.

[0080] In order to enable those skilled in the art to better understand the technical solutions provided in the embodiments of the present application, the technical solutions provided in the embodiments of the present application are described below with reference to specific examples.

[0081] In the embodiment of the present application, an incremental learning method based on multi-level domain discrimination is provided, which uses the model's own expressive power to hierarchically judge the domain to which the sample belongs, and then fuses the subsequent prediction results of the network according to the judgment results of each layer. The result fusion process does not directly choose to go to the trunk or trunk + branch, but adaptively fuses the two prediction results through a fusion module to effectively solve the problems of existing methods in terms of space occupancy and model performance. It allows users to use basic data and new data to directly perform end-to-end incremental training, which is simpler to use and has better results. At the same time, the soft fusion technology prevents domain discrimination errors from having a fatal impact, and improves the robustness of the algorithm to data distribution.

[0082] In this embodiment, the incremental learning process is as follows:

[0083] 1.1. Train the backbone model (i.e. the backbone network mentioned above).

[0084] Exemplarily, the basic data set can be used to perform supervised training on the backbone model to obtain a trained backbone network.

[0085] 1.2. The trunk model expands new branches (i.e., the new knowledge learning branch network mentioned above, which can be called a modulator in this embodiment) to learn new knowledge.

[0086] For example, in order to allow the model to learn new knowledge on the newly added data without forgetting the knowledge of the basic data, new branches can be developed (these new branches can be called "modulators"), whose role is to use the expressive power of the backbone network, combined with the expressive power of the modulator itself, to recombine new features to learn the knowledge of the new data.

[0087] Exemplarily, the modulator may include new knowledge learning branches corresponding to each convolutional layer of the backbone network.

[0088] 1.3. The main model expands into a new branch (i.e., the domain discrimination branch network mentioned above) to discriminate whether the input features come from the basic data domain or the newly added data domain.

[0089] 1.4. Train the incremental learning model (including the above-mentioned backbone model, modulator, and domain discrimination branch network).

[0090] Exemplarily, the training of the incremental learning model is performed end-to-end, and the basic data set and the newly added data set need to be used for training at the same time.

[0091] Exemplarily, the domain discrimination branch network can determine whether the input sample comes from the basic data domain or the newly added data domain during the model testing phase, and based on the domain discrimination result, fuse a certain proportion of the output features of the backbone network and the new knowledge learning branch network, and then pass the fused features to the next layer.

[0092] Therefore, the incremental learning model realizes the automatic fusion of the output features of the backbone network and the output features of the new knowledge learning branch network through the domain discrimination branch network, without the need for an additional classifier to dominate the process.

[0093] In this process, for training samples in the newly added data domain, in order to learn the newly added knowledge, the backbone network can update the network at a smaller learning rate, and the new knowledge learning branch network can update the network at a normal learning rate; for training samples in the basic data domain, in order to consolidate the basic knowledge, the backbone network can update the network at a normal learning rate, and the new knowledge learning branch network can update the network at a smaller learning rate.

[0094] 1.4. Use the trained incremental learning model for testing.

[0095] The trained incremental learning model can adaptively and dynamically adjust the route taken by the internal feature extraction according to the domain to which the image belongs (the features of the trunk and branches are fused in dynamic proportions and transmitted to the next layer). In other words, from the perspective of use, the model is similar to a static network, but the model performance is much higher than that of a static network (that is, a deep neural network without a mechanism for dynamically adjusting the feature transmission path).

[0096] Some details of the above steps are described below.

[0097] 2.1 Modulator and domain discrimination branch network

[0098] For example, the structure of the modulator and the domain discrimination branch network can be as follows: Figure 2 shown.

[0099] in, Figure 2 The convolutional layers in the structure diagram are only schematic diagrams, and the actual structure can be other combinations with different convolution sizes and different numbers of convolutional layers.

[0100] also, Figure 2Only a single convolutional layer of the backbone network is shown. For each convolutional layer of the backbone network, a new knowledge learning branch and a domain discrimination branch can be deployed accordingly.

[0101] Exemplarily, the domain discrimination branch may include multiple layers of convolution to convert input features into domain discrimination results.

[0102] Exemplarily, the domain discrimination result may be a number between 0 and 1 (including 0 or 1), where 0 may represent a basic data domain and 1 may represent a newly added data domain.

[0103] If the domain discrimination result corresponding to an image is between 0 and 1, it means that part of the image features match the basic data domain and part of the image features match the newly added data domain.

[0104] Therefore, the prediction result α of the domain discrimination branch can be used as an indicator of how far the image is from the two domains.

[0105] For example, the closer α is to 0, the closer the image is to the basic data domain; and the closer α is to 1, the closer the image is to the newly added data domain.

[0106] For example, in order to update the domain discrimination branch network, its loss function can be defined as:

[0107] L(θ)=∑||α i( θ)-l i || 2

[0108] Among them, l i is the loss of the backbone network, α i (θ) is the output of sample i, and its value is determined by θ, which is the parameter of the domain discrimination branch network. 2 is the square operation.

[0109] 2.2 Feature Fusion

[0110] Exemplarily, the purpose of feature fusion is to achieve the fusion of the output features of the backbone network and the modulator features (i.e., the output features of the new knowledge learning branch network) to avoid the selection of wrong branch results due to inaccurate domain discrimination in the dynamic network solution.

[0111] When fusion of features is performed, the domain discrimination result is used as the weight to perform weighted summation on the output features of the backbone network and the output features of the new knowledge learning branch network. The formula for weighted summation is as follows:

[0112] F i (x) = (1-α)·g i (x)+α·h i (x)

[0113] Among them, α is the domain discrimination result of the domain discrimination branch network, g i (x) is the output feature of the backbone network, h i (x) is the output feature of the new knowledge learning branch network.

[0114] For example, if the domain discrimination result shows that the current sample x is closer to the basic data domain (that is, the distance to the basic data domain is smaller than the distance to the newly added data domain), then the backbone network can be relied upon more to extract features; if the sample x is closer to the newly added data domain, then the new knowledge learning branch network can be relied upon more to extract features.

[0115] Therefore, for the input image, the features extracted by the network contain both the information of the backbone network and the information of the new knowledge learning branch network, and the amount of information contained in it is strongly correlated with the domain to which it belongs.

[0116] 2.3 Training Process

[0117] 2.3.1. Generate supervision information for the domain discrimination branch network.

[0118] For example, the basic model (a model that only includes the backbone network) can be used to predict each sample in the basic training set and the newly added training set, and the loss value of each sample can be recorded.

[0119] For example, a larger loss value indicates that the model's prediction result for the sample is less accurate, and the sample is closer to the newly added training set.

[0120] Exemplarily, all loss values ​​are mapped between 0 and 1, and the smaller the loss function is, the closer the mapped result is to 0, and the larger the loss function is, the closer the mapped result is to 1. The mapped result can be used as supervision information for the domain discrimination branch network.

[0121] 2.3.2. The output features of the backbone network are integrated with the output features of the new knowledge learning branch network.

[0122] 2.3.3. Use the basic training set + the newly added training set to train the incremental learning model.

[0123] Exemplarily, during the training process, the loss function may include two parts: one is the supervision loss of the head part of the model, and the other is the loss of each layer of the domain discrimination branch network.

[0124] The method provided by the present application is described above. The device provided by the present application is described below:

[0125] See also Figure 3 , is a schematic diagram of the structure of an incremental learning device provided in an embodiment of the present application, such as Figure 3 As shown, the incremental learning device may include:

[0126] A domain discrimination unit 310 is used to perform domain discrimination on the input features according to the domain discrimination branch network in the incremental learning model to obtain a domain discrimination result; the incremental learning model includes a backbone network, a new knowledge learning branch network and the domain discrimination branch network, and for each convolution layer of the backbone network, the new knowledge learning branch network includes a branch corresponding to the convolution layer of the backbone network, and the domain discrimination branch network includes a domain discrimination branch corresponding to the convolution layer of the backbone network;

[0127] A feature fusion unit 320 is used to fuse the output features of the backbone network and the output features of the new knowledge learning branch network according to the domain discrimination result to obtain a fused output feature;

[0128] The prediction unit 330 is used to perform prediction processing according to the fused output features.

[0129] In some embodiments, the domain discrimination unit 310 performs domain discrimination on the input features according to the domain discrimination branch network in the incremental learning model, including:

[0130] For the target convolution layer of the backbone network, domain discrimination is performed on the input features of the target convolution layer according to the domain discrimination branch corresponding to the target convolution layer in the domain discrimination branch network to obtain a target domain discrimination result; the target convolution layer is any convolution layer of the backbone network;

[0131] The feature fusion unit 320 fuses the output features of the backbone network and the output features of the new knowledge learning branch network according to the domain discrimination result to obtain fused output features, including:

[0132] According to the target domain discrimination result, the first output feature of the target convolutional layer and the second output feature of the branch corresponding to the target convolutional layer in the new knowledge learning branch network are fused to obtain the fused output feature corresponding to the target convolutional layer.

[0133] In some embodiments, the feature fusion unit 320 fuses the output features of the target convolution layer and the output features of the new knowledge learning branch corresponding to the target convolution layer in the new knowledge learning branch network according to the target domain discrimination result to obtain the fused output features corresponding to the target convolution layer, including:

[0134] According to the target domain discrimination result, determine a first weight of the first output feature and a second weight of the second output feature; wherein, when it is determined according to the target domain discrimination result that the distance between the input feature and the basic data domain is less than the distance between the input feature and the newly added data domain, the first weight is greater than the second weight; when it is determined according to the target domain discrimination result that the distance between the input feature and the basic data domain is greater than the distance between the input feature and the newly added data domain, the first weight is less than the second weight;

[0135] According to the first weight and the second weight, the first output feature and the second output feature are weightedly summed to obtain a fused output feature corresponding to the target convolutional layer.

[0136] In some embodiments, the backbone network and the new knowledge learning branch network are feedback tuned according to the prediction loss of the incremental learning model, and the first weight and the second weight.

[0137] In some embodiments, Figure 4 As shown, the device also includes:

[0138] The training unit 340 is used to train the domain discrimination branch network using the loss value of the trunk network predicting the training samples as a supervisory signal during the training process.

[0139] In some embodiments, the training unit 340 uses the loss value of the trunk network predicting the training sample as a supervisory signal to train the domain discrimination branch network, including:

[0140] For any training sample, the loss of the domain discrimination branch network corresponding to the training sample is determined according to the loss value predicted by the backbone network for the training sample and the domain discrimination result of the domain discrimination branch network for the training sample.

[0141] In some embodiments, the training unit 340 determines the loss of the domain discrimination branch network corresponding to the training sample according to the loss value predicted by the backbone network for the training sample and the domain discrimination result of the domain discrimination branch network for the training sample, including:

[0142] Normalizing the loss value of the training sample predicted by the backbone network to obtain a normalized loss value;

[0143] The loss of the domain discrimination branch network corresponding to the training sample is determined according to the absolute value of the difference between the normalized loss value and the domain discrimination result of the domain discrimination branch network performing domain discrimination on the training sample; wherein the value range of the domain discrimination result is [0, 1], and for any training sample, the smaller the domain discrimination result of the domain discrimination branch network performing domain discrimination on the training sample, the smaller the distance between the training sample and the basic data domain; the larger the domain discrimination result of the domain discrimination branch network performing domain discrimination on the training sample, the smaller the distance between the training sample and the newly added data domain.

[0144] In some embodiments, for any training sample, when it is determined that the training sample belongs to the basic data domain based on the domain discrimination result of the domain discrimination branch network on the training sample, the backbone network is updated with a first learning rate, and the new knowledge learning branch network is updated with a second learning rate, and the second learning rate is less than the first learning rate;

[0145] When it is determined that the training sample belongs to the newly added data domain based on the domain discrimination result of the domain discrimination branch network on the training sample, the backbone network is updated with a third learning rate, and the new knowledge learning branch network is updated with a fourth learning rate, and the third learning rate is less than the fourth learning rate.

[0146] An embodiment of the present application provides an electronic device, including a processor and a memory, wherein the memory stores machine executable instructions that can be executed by the processor, and the processor is used to execute the machine executable instructions to implement the incremental learning method described above.

[0147] See also Figure 5 , is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application. The electronic device may include a processor 501 and a memory 502 storing machine executable instructions. The processor 501 and the memory 502 may communicate via a system bus 503. In addition, by reading and executing the machine executable instructions corresponding to the incremental learning logic in the memory 502, the processor 501 may execute the incremental learning method described above.

[0148] The memory 502 mentioned herein may be any electronic, magnetic, optical or other physical storage device that may contain or store information, such as executable instructions, data, etc. For example, the machine-readable storage medium may be: RAM (RadomAccess Memory), volatile memory, non-volatile memory, flash memory, storage drive (such as hard disk drive), solid state drive, any type of storage disk (such as optical disk, DVD, etc.), or similar storage medium, or a combination thereof.

[0149] In some embodiments, a machine-readable storage medium is also provided. Figure 5The memory 502 in the machine readable storage medium stores machine executable instructions, and when the machine executable instructions are executed by the processor, the incremental learning method described above is implemented. For example, the storage medium can be ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, etc.

[0150] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0151] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. An incremental learning method, characterized in that: The method is applied to an electronic device, comprising: For the target convolution layer of the backbone network in the incremental learning model, domain discrimination is performed on the features of the input image of the target convolution layer according to the domain discrimination branch corresponding to the target convolution layer in the domain discrimination branch network in the incremental learning model to obtain a target domain discrimination result; the target convolution layer is any convolution layer of the backbone network; the incremental learning model includes a backbone network, a new knowledge learning branch network and the domain discrimination branch network, and for each convolution layer of the backbone network, the new knowledge learning branch network includes a new knowledge learning branch corresponding to the convolution layer of the backbone network, and the domain discrimination branch network includes a domain discrimination branch corresponding to the convolution layer of the backbone network; According to the domain discrimination result, the output features of the backbone network and the output features of the new knowledge learning branch network are fused to obtain fused output features; Prediction processing is performed based on the fused output features.

2. The method according to claim 1, characterized in that According to the domain discrimination result, the output features of the backbone network and the output features of the new knowledge learning branch network are fused to obtain fused output features, including: According to the target domain discrimination result, the first output feature of the target convolutional layer and the second output feature of the new knowledge learning branch corresponding to the target convolutional layer in the new knowledge learning branch network are fused to obtain the fused output feature corresponding to the target convolutional layer.

3. The method according to claim 2, characterized in that According to the target domain discrimination result, the first output feature of the target convolution layer and the second output feature of the new knowledge learning branch corresponding to the target convolution layer in the new knowledge learning branch network are fused to obtain the fused output feature corresponding to the target convolution layer, including: Determine a first weight of the first output feature and a second weight of the second output feature according to the target domain discrimination result; wherein, when it is determined according to the target domain discrimination result that the distance between the feature of the input image and the basic data domain is smaller than the distance between the feature of the input image and the newly added data domain, the first weight is greater than the second weight; when it is determined according to the target domain discrimination result that the distance between the feature of the input image and the basic data domain is larger than the distance between the feature of the input image and the newly added data domain, the first weight is less than the second weight; According to the first weight and the second weight, the first output feature and the second output feature are weightedly summed to obtain a fused output feature corresponding to the target convolutional layer.

4. The method according to claim 3, characterized in that During the training process, the backbone network and the new knowledge learning branch network are respectively feedback tuned based on the prediction loss of the incremental learning model, and the first weight and the second weight.

5. The method according to claim 1, characterized in that During the training process, the loss value of the training sample predicted by the backbone network is used as a supervisory signal to train the domain discrimination branch network.

6. The method according to claim 5, characterized in that The method of using the loss value of the training sample predicted by the backbone network as a supervisory signal to train the domain discrimination branch network includes: For any training sample, the loss of the domain discrimination branch network corresponding to the training sample is determined according to the loss value predicted by the backbone network for the training sample and the domain discrimination result of the domain discrimination branch network for the training sample.

7. The method according to claim 6, characterized in that The step of determining the loss of the domain discrimination branch network corresponding to the training sample based on the loss value predicted by the backbone network for the training sample and the domain discrimination result of the domain discrimination branch network for the training sample comprises: Normalizing the loss value of the training sample predicted by the backbone network to obtain a normalized loss value; The loss of the domain discrimination branch network corresponding to the training sample is determined according to the absolute value of the difference between the normalized loss value and the domain discrimination result of the domain discrimination branch network performing domain discrimination on the training sample; wherein the value range of the domain discrimination result is [0, 1], and for any training sample, the smaller the domain discrimination result of the domain discrimination branch network performing domain discrimination on the training sample, the smaller the distance between the training sample and the basic data domain; the larger the domain discrimination result of the domain discrimination branch network performing domain discrimination on the training sample, the smaller the distance between the training sample and the newly added data domain.

8. The method according to claim 1, characterized in that During the training process, for any training sample, when it is determined that the training sample belongs to the basic data domain based on the domain discrimination result of the domain discrimination branch network on the training sample, the backbone network is updated with a first learning rate, and the new knowledge learning branch network is updated with a second learning rate, wherein the second learning rate is less than the first learning rate; When it is determined that the training sample belongs to the newly added data domain based on the domain discrimination result of the domain discrimination branch network on the training sample, the backbone network is updated with a third learning rate, and the new knowledge learning branch network is updated with a fourth learning rate, and the third learning rate is less than the fourth learning rate.

9. An incremental learning device, characterized in that: The device is applied to electronic equipment, including: A domain discrimination unit is used to perform domain discrimination on the features of the input image of the target convolutional layer of the backbone network in the incremental learning model according to the domain discrimination branch corresponding to the target convolutional layer in the domain discrimination branch network in the incremental learning model, so as to obtain a target domain discrimination result; the target convolutional layer is any convolutional layer of the backbone network; the incremental learning model includes a backbone network, a new knowledge learning branch network and the domain discrimination branch network, and for each convolutional layer of the backbone network, the new knowledge learning branch network includes a new knowledge learning branch corresponding to the convolutional layer of the backbone network, and the domain discrimination branch network includes a domain discrimination branch corresponding to the convolutional layer of the backbone network; A feature fusion unit, used to fuse the output features of the backbone network and the output features of the new knowledge learning branch network according to the domain discrimination result to obtain a fused output feature; A prediction unit is used to perform prediction processing based on the fused output features.

10. The device according to claim 9, characterized in that The feature fusion unit fuses the output features of the backbone network and the output features of the new knowledge learning branch network according to the domain discrimination result to obtain fused output features, including: According to the target domain discrimination result, a first output feature of the target convolution layer and a second output feature of the new knowledge learning branch corresponding to the target convolution layer in the new knowledge learning branch network are fused to obtain a fused output feature corresponding to the target convolution layer; The feature fusion unit fuses the first output feature of the target convolution layer and the second output feature of the new knowledge learning branch corresponding to the target convolution layer in the new knowledge learning branch network according to the target domain discrimination result to obtain the fused output feature corresponding to the target convolution layer, including: Determine a first weight of the first output feature and a second weight of the second output feature according to the target domain discrimination result; wherein, when it is determined according to the target domain discrimination result that the distance between the feature of the input image and the basic data domain is smaller than the distance between the feature of the input image and the newly added data domain, the first weight is greater than the second weight; when it is determined according to the target domain discrimination result that the distance between the feature of the input image and the basic data domain is larger than the distance between the feature of the input image and the newly added data domain, the first weight is less than the second weight; According to the first weight and the second weight, performing weighted summation on the first output feature and the second output feature to obtain a fused output feature corresponding to the target convolutional layer; According to the prediction loss of the incremental learning model, as well as the first weight and the second weight, feedback tuning is performed on the backbone network and the new knowledge learning branch network respectively; and / or, The device also includes: A training unit, used to train the domain discrimination branch network using the loss value of the training sample predicted by the backbone network as a supervisory signal during the training process; The training unit uses the loss value of the training sample predicted by the backbone network as a supervisory signal to train the domain discrimination branch network, including: For any training sample, according to the loss value predicted by the backbone network for the training sample and the domain discrimination result of the domain discrimination branch network for the training sample, determining the loss of the domain discrimination branch network corresponding to the training sample; The training unit determines the loss of the domain discrimination branch network corresponding to the training sample according to the loss value predicted by the backbone network for the training sample and the domain discrimination result of the domain discrimination branch network for the training sample, including: Normalizing the loss value of the training sample predicted by the backbone network to obtain a normalized loss value; According to the absolute value of the difference between the normalized loss value and the domain discrimination result of the domain discrimination branch network for domain discrimination of the training sample, the loss of the domain discrimination branch network corresponding to the training sample is determined; wherein the value range of the domain discrimination result is [0, 1], and for any training sample, the smaller the domain discrimination result of the domain discrimination branch network for domain discrimination of the training sample, the smaller the distance between the training sample and the basic data domain; the larger the domain discrimination result of the domain discrimination branch network for domain discrimination of the training sample, the smaller the distance between the training sample and the newly added data domain; and / or, For any training sample, when it is determined that the training sample belongs to the basic data domain according to the domain discrimination result of the domain discrimination branch network on the training sample, the backbone network is updated with a first learning rate, and the new knowledge learning branch network is updated with a second learning rate, wherein the second learning rate is less than the first learning rate; When it is determined that the training sample belongs to the newly added data domain based on the domain discrimination result of the domain discrimination branch network on the training sample, the backbone network is updated with a third learning rate, and the new knowledge learning branch network is updated with a fourth learning rate, and the third learning rate is less than the fourth learning rate.

11. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory stores machine executable instructions that can be executed by the processor, and the processor is used to execute the machine executable instructions to implement the method according to any one of claims 1 to 8.

12. A machine-readable storage medium, characterized in that: The machine-readable storage medium stores machine-executable instructions, and when the machine-executable instructions are executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images