Methods and apparatus for performing tunable continual learning on deep neural network models

Abstract

The method for adjustable continuous learning of a deep neural network model using a selective deep generative replay module comprises: (a) (i) inputting first learning data and second learning data from an entire database and a sub-database into the selective deep generative replay module to generate first low-dimensional distribution features and second low-dimensional distribution features, (ii) inputting binary, random parameters and the second low-dimensional distribution features into a data generator to generate third learning data, (iii) inputting the first learning data into a solver to generate learning labeled data; (b) inputting the first learning data, the second learning data, the first low-dimensional distribution features, the second low-dimensional distribution features, the third learning data and the binary into a discriminator to generate first learning data scores, second learning data scores, first distribution feature scores, second distribution feature scores and third learning data scores; (c) training the discriminator, the data generator, a distribution analyzer and the solver.

Description

Technical Field

[0001] This invention claims priority and benefit to U.S. Patent Application No. 63 / 028779, filed May 22, 2020, and U.S. Patent Application No. 17 / 136847, filed December 29, 2020, the entire contents of which are incorporated herein by reference.

[0002] This invention relates to a method and apparatus for continuous learning of a deep neural network model, and more specifically to a method and apparatus for adjustable continuous learning of a deep neural network model using a Selective Deep Generative Replay Module, wherein the Selective Deep Generative Replay Module is a Database Replay Module using a Generative Adversarial Network (GAN). Background Technology

[0003] In neural network models that apply deep learning, various continuous learning techniques are used to continuously learn the neural network model that reflects existing and new databases.

[0004] In order to carry out effective and continuous learning, and to continuously accumulate databases for specific domain problems, a mechanism for efficiently learning neural network models is needed.

[0005] Figure 1 This is a schematic diagram of the configuration of a conventional, existing continuous learning method.

[0006] See Figure 1 In conventional tunable continuous learning methods, when there is an existing model that is trained on an existing task using an existing database, and a new task and a corresponding new database are given, a new model that can perform both the new task and the existing task can be generated by training the existing model using both the existing and new databases.

[0007] However, in existing tunable continuous learning methods, the database inevitably grows continuously when learning incrementally for a specific task, thus increasing the storage capacity required to store the data.

[0008] Furthermore, in existing tunable continuous learning methods, when performing incremental learning, the time required to learn a neural network model for that database increases as the database grows, leading to a decrease in learning efficiency.

[0009] In addition, such as Figure 1 As shown, when the task required by a neural network model changes with the input of a new database, a method is needed to adjust the learning for the given new task. However, existing tunable continuous learning methods cannot flexibly adjust the learning method. For example, if a neural network model is used for image classification of vehicles, an existing neural network model can be trained to improve lane detection accuracy on a general road map accordingly. Subsequently, a new task, such as fast lane detection on a highway, can be given during its continuous learning process. In this case, existing tunable continuous learning methods lack alternatives that can effectively adjust the learning method, such as adjusting the learning participation rate of existing and new data according to the new task.

[0010] Therefore, an improved solution is needed to address the aforementioned problem. Summary of the Invention

[0011] Technical issues

[0012] The purpose of this invention is to solve all the above-mentioned problems.

[0013] Furthermore, the purpose of this invention is to reduce the storage space required to store the existing database instead of the entire existing database used to train the neural network model.

[0014] Another objective of this invention is to reduce the learning time required by training a neural network model by generating a certain amount of data using a selective deep generative replay module, instead of using the entire existing database and the entire new database for learning.

[0015] Furthermore, another objective of this invention is to generate a neural network model optimized for a given task by training a neural network model instead of storing the entire existing database and using both the entire existing database and the entire new database to train the neural network model by using a selective deep generative replay module of a Generative Adversarial Network (GAN) to generate data with distributions similar to and dissimilar to the new database at a selective ratio.

[0016] Technical solution

[0017] In order to achieve the above-mentioned objectives of the present invention and to realize the characteristic effects of the present invention described later, the characteristic structure of the present invention is as follows.

[0018] According to one aspect of the present invention, a method for adjustable continuous learning of a deep neural network model using a Selective Deep Generative Replay Module is disclosed, comprising: (a) when first training data is obtained from the entire database and second training data is obtained from a training sub-database that is a subset of the entire database, the learning device performs or supports performing the following processing: inputting the first training data and the second training data into the Selective Deep Generative Replay Module such that the Selective Deep Generative Replay Module (i) generates a first low-dimensional distribution feature for training corresponding to the first training data and a second low-dimensional distribution feature for training corresponding to the second training data through a distribution analyzer located in the Selective Deep Generative Replay Module. (i) training), wherein the dimension of the first learning low-dimensional distribution feature is lower than the dimension of the first learning data, and the dimension of the second learning low-dimensional distribution feature is lower than the dimension of the second learning data; (ii) learning binary generated from the dynamic binary generator in the selective deep generation and replay module, learning random parameters generated from the random parameter generator in the selective deep generation and replay module, and the second learning low-dimensional distribution feature are input to the data generator in the selective deep generation and replay module, and the data generator generates a third learning data corresponding to the second learning low-dimensional distribution feature according to the learning binary and the learning random parameters; (iii) the first learning data is input to the solver in the selective deep generation and replay module, and the solver outputs learning labeled data based on deep learning to annotate the first learning data;(b) The learning device performs or supports performing the following process: inputting the first learning data, the second learning data, the first learning low-dimensional distribution feature, the second learning low-dimensional distribution feature, the third learning data, and the learning binary data into a discriminator, such that the discriminator outputs the first learning data score, the second learning data score, the first distribution feature score, and the second distribution feature score corresponding to the learning binary data. (i) the second learning data score and the third learning data score; and (c) the learning device performs or supports performing the following processing: (i) generating a first discrimination loss, a second discrimination loss, a first generation loss, and a second generation loss, with reference to the second distribution feature score and the third learning data score; and (ii) training the discriminator, the data generator, and the score using the second learning data score, the third learning data score, the second learning data score, the third learning data score, the third learning data score, the second distribution feature score, and the third learning data score, respectively. The analyzer generates third learning data that, when the learning binary is a first binary value, includes a distribution having at least one of the means and variances within each learning data threshold range determined based on the means and variances of the second learning low-dimensional distribution features; generates third learning data that, when the learning binary is a second binary value, includes a distribution having at least one of the means and variances outside the each learning data threshold range determined based on the means and variances of the second learning low-dimensional distribution features; (ii) further generates a solver loss with reference to the learning labeled data and the corresponding ground truth data, and further trains the solver using the solver loss.

[0019] As an example, in step (c), the learning device performs or supports performing the following processing: referring to the learning binary value, (i) when the learning binary is a first binary value, training the discriminator and the distribution analyzer using the first discriminant loss, and training the data generator and the distribution analyzer using the first generation loss, and (ii) when the learning binary is a second binary value, training the discriminator and the distribution analyzer using the second discriminant loss, and training the data generator and the distribution analyzer using the second generation loss.

[0020] As an example, the learning device performs or supports the following processing: (i) when the discriminator and the distribution analyzer are trained using the first discriminant loss and the second discriminant loss respectively, training is performed with the first parameter of the data generator fixed; (ii) when the data generator and the distribution analyzer are trained using the first generation loss and the second generation loss respectively, training is performed with the second parameter of the discriminator fixed.

[0021] As an example, the data generator includes at least one encoding layer and at least one decoding layer.

[0022] According to another aspect of the present invention, a method for adjusting continuous learning of a deep neural network model using a Selective Deep Generative Replay Module is disclosed, comprising: (a) when a learning device obtains first training data from a whole database and second training data from a sub-database which is a subset of the whole database, the learning device performs or supports performing the following processing: inputting the first training data and the second training data into the Selective Deep Generative Replay Module such that the Selective Deep Generative Replay Module (I) (i) generates a first low-dimensional distribution feature for training corresponding to the first training data and a second low-dimensional distribution feature for training corresponding to the second training data through a distribution analyzer located in the Selective Deep Generative Replay Module. (ii) The learning binary generated from the Dynamic Binary Generator in the Selective Depth Generation and Replay Module, the learning random parameters generated from the Random Parameter Generator in the Selective Depth Generation and Replay Module, and the second learning low-dimensional distribution feature are input to the Data Generator in the Selective Depth Generation and Replay Module. The Data Generator generates a third learning data corresponding to the second learning low-dimensional distribution feature based on the learning binary and the learning random parameters. (iii) The first learning data is input to the Solver in the Selective Depth Generation and Replay Module. The Solver outputs the processed learning labeled data based on deep learning to annotate the first learning data.(II) Input the first learning data, the second learning data, the first learning low-dimensional distribution feature, the second learning low-dimensional distribution feature, the third learning data, and the learning binary into a discriminator, such that the discriminator corresponds to the learning binary, and outputs the first learning data score, the second learning data score, the first distribution feature score, and the second distribution feature score. Processing of the second learning data score and the third learning data score; and (III)(i) generating a first discrimination loss, a second discrimination loss, a first generation loss, and a second generation loss with reference to the second learning data score, the second distribution feature score, and the third learning data score, to train the discriminator, the data generator, and the distribution analyzer, so that... The data generator generates the third learning data, which, when the learning binary is the first binary value, includes a distribution having at least one of the means and variances within each learning data threshold range determined based on the means and variances of the second learning low-dimensional distribution features; and generates the third learning data, which, when the learning binary is the second binary value, includes a distribution having at least one of the means and variances outside the each learning data threshold range determined based on the means and variances of the second learning low-dimensional distribution features; (ii) under the state of further generating a solver loss with reference to the learning labeled data and the corresponding ground truth, and further training the solver using the solver loss, the continuous learning device performs or supports the execution of processing to obtain new data from a newly collected new database;(b) The continuous learning device performs or supports performing the following processing: inputting the new data into the selective deep generation replay module such that the selective deep generation replay module (i) generates a new low-dimensional distribution feature corresponding to the new data through the distribution analyzer, wherein the dimension of the new low-dimensional distribution feature is lower than the dimension of the new data; (ii) inputting the test binary generated from the dynamic binary generator, the test random parameters generated from the random parameter generator, and the new low-dimensional distribution feature into the data generator, and generating test data corresponding to the new low-dimensional distribution feature through the data generator based on the test binary and the test random parameters, wherein the test data includes first regeneration data and second regeneration data. (iii) The first regeneration data, when the test binary is a first binary value, includes a distribution having at least one of the means and variances within each test data threshold range determined based on the new low-dimensional distribution characteristics, and the second regeneration data, when the test binary is a second binary value, includes a distribution having at least one of the means and variances outside the test data threshold range determined based on the new low-dimensional distribution characteristics, and (iii) the test data is input to the solver, and test annotation data labeled with the test data is output through the solver.

[0023] As an example, (c) when generating an old selective deep generative replay module by copying the selective deep generative replay module, and inputting an old low-dimensional distribution feature, including at least a portion of the first learning low-dimensional distribution feature and the second learning low-dimensional distribution feature, into the old selective deep generative replay module so that the old selective deep generative replay module generates old data corresponding to the old low-dimensional distribution feature and old labeled data corresponding to the old data, the continuous learning device performs or supports performing the following processing: (i) inputting the old data, the new data, the old low-dimensional distribution feature, the new low-dimensional distribution feature, the test data, and the test binary into the discriminator so that the discriminator outputs an old data score, a new data score, an old distribution feature score, and a new distribution feature score corresponding to the test binary. (ii) inputting the old data and the new data into the solver so that the solver outputs newly labeled data based on deep learning to annotate the old data and the new data; and (d) the continuous learning device performs or supports performing the following processes: (i) generating a first new discrimination loss, a second new discrimination loss, a third new discrimination loss, a fourth new discrimination loss, a fifth new discrimination loss, a sixth new discrimination loss, a seventh new discrimination loss, and a eighth new discrimination loss, a seventh new discrimination loss, and a eighth new discrimination loss, a ninth new discrimination loss, and a eleventh new discrimination loss, ... The new loss is generated to train the discriminator, the data generator, and the distribution analyzer, such that the data generator generates test data with features including a distribution having at least one of the mean and variance within the range of each test data threshold determined based on the mean and variance of the new low-dimensional distribution features when the test binary is a first binary value, and generates test data with features including a distribution having at least one of the mean and variance outside the range of each test data threshold determined based on the mean and variance of the new low-dimensional distribution features when the test binary is a second binary value, and (ii) generates a new solver loss with reference to the new labeled data and the old labeled data, and trains the solver using the new solver loss.

[0024] As an example, in step (b), the continuous learning device performs or supports the following process: causing the dynamic binary generator to generate the test binary, wherein a data generation ratio that sets the generation ratio of the first reproducible data to the second reproducible data is input to the dynamic binary generator, such that the dynamic binary generator generates multiple test binary values ​​for multiple first binary values ​​used to generate the first reproducible data and multiple second binary values ​​used to generate the second reproducible data, according to the data generation ratio.

[0025] As an example, in step (d), the continuous learning device performs or supports performing the following processing: referring to the test binary value, (i) when the test binary is a first binary value, training the discriminator and the distribution analyzer using the first new discriminant loss, and training the data generator and the distribution analyzer using the first new generation loss, (ii) when the test binary is a second binary value, training the discriminator and the distribution analyzer using the second new discriminant loss, and training the data generator and the distribution analyzer using the second new generation loss.

[0026] As an example, the continuous learning device performs or supports the following processing: (i) when the discriminator and the distribution analyzer are trained using the first new discriminant loss and the second new discriminant loss respectively, training is performed with the first parameter of the data generator fixed; (ii) when the data generator and the distribution analyzer are trained using the first new generation loss and the second new generation loss respectively, training is performed with the second parameter of the discriminator fixed.

[0027] As an example, the data generator includes at least one encoding layer and at least one decoding layer.

[0028] According to another aspect of the invention, a learning apparatus for adjustable continuous learning of a deep neural network model using a Selective Deep Generative Replay Module is disclosed, comprising: at least one memory storing instructions; and at least one processor for executing the instructions, the processor performing or supporting the following processing: (I) when first training data is obtained from the entire database and second training data is obtained from a sub-database which is a subset of the entire database, the first training data and the second training data are input into the Selective Deep Generative Replay Module such that the Selective Deep Generative Replay Module (i) generates a first low-dimensional distribution feature for training corresponding to the first training data and a second low-dimensional distribution feature for training corresponding to the second training data through a distribution analyzer located in the Selective Deep Generative Replay Module. (i) training), wherein the dimension of the first learning low-dimensional distribution feature is lower than the dimension of the first learning data, and the dimension of the second learning low-dimensional distribution feature is lower than the dimension of the second learning data; (ii) learning binary generated from the dynamic binary generator in the selective deep generation and replay module, learning random parameters generated from the random parameter generator in the selective deep generation and replay module, and the second learning low-dimensional distribution feature are input to the data generator in the selective deep generation and replay module, and the data generator generates a third learning data corresponding to the second learning low-dimensional distribution feature according to the learning binary and the learning random parameters; (iii) the first learning data is input to the solver in the selective deep generation and replay module, and the solver outputs learning labeled data based on deep learning to annotate the first learning data;(II) Input the first learning data, the second learning data, the first learning low-dimensional distribution feature, the second learning low-dimensional distribution feature, the third learning data, and the learning binary into the discriminator, and make the discriminator correspond to the learning binary, outputting the first learning data score, the second learning data score, the first distribution feature score, and the second distribution feature score. (i) generating a first discrimination loss, a second discrimination loss, a first generation loss, and a second generation loss, with reference to the second distribution feature score and the third learning data score; and (iii)(i) generating a first generation loss with reference to the second distribution feature score and the third learning data score; and a second generation loss with reference to the first distribution feature score, the second distribution feature score, and the third learning data score to train the discriminator, the data generator, and the distribution analyzer, so that the... The data generator generates the third learning data, which, when the learning binary is the first binary value, includes a distribution having characteristics of at least one of the means and variances within each learning data threshold range determined based on the means and variances of the second low-dimensional learning distribution features; it also generates the third learning data, which, when the learning binary is the second binary value, includes a distribution having characteristics of at least one of the means and variances outside the respective learning data threshold ranges determined based on the means and variances of the second low-dimensional learning distribution features; (ii) it further generates a solver loss with reference to the learning labeled data and the corresponding ground truth data, and further trains the solver using the solver loss.

[0029] As an example, in the (III) process, the processor performs or supports performing the following processes: referring to the learning binary value, (i) when the learning binary is a first binary value, training the discriminator and the distribution analyzer using the first discriminant loss, and training the data generator and the distribution analyzer using the first generation loss, (ii) when the learning binary is a second binary value, training the discriminator and the distribution analyzer using the second discriminant loss, and training the data generator and the distribution analyzer using the second generation loss.

[0030] As an example, the processor performs or supports performing the following processes: (i) when the discriminator and the distribution analyzer are trained using the first discriminant loss and the second discriminant loss respectively, training is performed with the first parameter of the data generator fixed; (ii) when the data generator and the distribution analyzer are trained using the first generation loss and the second generation loss respectively, training is performed with the second parameter of the discriminator fixed.

[0031] As an example, the data generator includes at least one encoding layer and at least one decoding layer.

[0032] According to another aspect of the invention, a continuous learning apparatus for performing adjustable continuous learning of a deep neural network model using a Selective Deep Generative Replay Module includes: at least one memory storing instructions; and at least one processor for executing the instructions, the processor performing or supporting the following process: (I) the learning apparatus performs or supports the following process: when first training data is obtained from the entire database and second training data is obtained from a sub-database that is a subset of the entire database, the first training data and the second training data are input into the Selective Deep Generative Replay Module such that the Selective Deep Generative Replay Module (i)(i-1) generates a first low-dimensional distribution feature for training corresponding to the first training data and a second low-dimensional distribution feature corresponding to the second training data through a distribution analyzer located in the Selective Deep Generative Replay Module. (i-2) The learning binary generated from the Dynamic Binary Generator in the Selective Deep Generation Replay Module, the learning random parameters generated from the Random Parameter Generator in the Selective Deep Generation Replay Module, and the learning low-dimensional distribution feature are input to the Data Generator in the Selective Deep Generation Replay Module. The Data Generator generates a third learning data corresponding to the learning low-dimensional distribution feature based on the learning binary and the learning random parameters. (i-3) The first learning data is input to the Solver in the Selective Deep Generation Replay Module. The Solver outputs the learning labeled data based on deep learning to annotate the first learning data.(ii) Input the first learning data, the second learning data, the first learning low-dimensional distribution feature, the second learning low-dimensional distribution feature, the third learning data, and the learning binary into the discriminator, and make the discriminator correspond to the learning binary, outputting the first learning data score, the second learning data score, the first distribution feature score, and the second distribution feature score. (iii) Generate a first discrimination loss, a second discrimination loss, a first generation loss, and a second generation loss, with reference to the second distribution feature score and the third learning data score; and (iii) Generate a first generation loss with reference to the second distribution feature score and the third learning data score, with reference to the second distribution feature score and the third learning data score, with reference to the first distribution feature score, the second distribution feature score, and the third learning data score, to train the discriminator, the data generator, and the distribution analyzer, so that the... The data generator generates the third learning data, which, when the learning binary is the first binary value, includes a distribution having at least one of the means and variances within each learning data threshold range determined based on the means and variances of the second learning low-dimensional distribution features. It also generates the third learning data, which, when the learning binary is the second binary value, includes a distribution having at least one of the means and variances outside each learning data threshold range determined based on the means and variances of the second learning low-dimensional distribution features. (iii-2) While further generating a solver loss with reference to the learning labeled data and the corresponding ground truth, and further training the solver using the solver loss, new data is obtained from a newly collected database.(II) The new data is input into the selective depth generation and replay module, and the selective depth generation and replay module (i) generates a new low-dimensional distribution feature corresponding to the new data through the distribution analyzer, wherein the dimension of the new low-dimensional distribution feature is lower than the dimension of the new data; (ii) the test binary generated from the dynamic binary generator, the test random parameters generated from the random parameter generator, and the new low-dimensional distribution feature are input into the data generator, and the data generator generates test data corresponding to the new low-dimensional distribution feature based on the test binary and the test random parameters, wherein the test data includes first regeneration data and second regeneration data. The first reproducible data, when the test binary is a first binary value, includes a distribution having at least one of the mean and variance within each test data threshold range determined based on the new low-dimensional distribution characteristics; the second reproducible data, when the test binary is a second binary value, includes a distribution having at least one of the mean and variance outside each test data threshold range determined based on the new low-dimensional distribution characteristics; (iii) the test data is input to the solver, and the solver outputs test labeled data annotated with the test data.

[0033] As an example, (III) when generating an old selective deep generative replay module by copying the selective deep generative replay module, and inputting an old low-dimensional distribution feature, including at least a portion of the first learning low-dimensional distribution feature and the second learning low-dimensional distribution feature, into the old selective deep generative replay module so that the old selective deep generative replay module generates old data corresponding to the old low-dimensional distribution feature and old labeled data corresponding to the old data, the processor performs or supports performing the following processing: (i) inputting the old data, the new data, the old low-dimensional distribution feature, the new low-dimensional distribution feature, the test data, and the test binary into the discriminator so that the discriminator outputs an old data score, a new data score, an old distribution feature score, and a new distribution feature score corresponding to the test binary. (ii) inputting the old data and the new data into the solver so that the solver outputs newly labeled data based on deep learning to annotate the old data and the new data; and (iv) the processor performs or supports performing the following processes: (i) generating a first new discrimination loss with reference to the new data score, the new distribution feature score and the test data score, a second new discrimination loss with reference to the old data score, the old distribution feature score and the test data score, a first new generation loss with reference to the new distribution feature score and the test data score, and a second generation loss with reference to the old distribution feature score, the new distribution feature score and the test data score. The new loss is generated to train the discriminator, the data generator, and the distribution analyzer, such that the data generator generates test data with features including a distribution having at least one of the mean and variance within the range of each test data threshold determined based on the mean and variance of the new low-dimensional distribution features when the test binary is a first binary value, and generates test data with features including a distribution having at least one of the mean and variance outside the range of each test data threshold determined based on the mean and variance of the new low-dimensional distribution features when the test binary is a second binary value, and (ii) generates a new solver loss with reference to the new labeled data and the old labeled data, and trains the solver using the new solver loss.

[0034] As an example, in the (II) process, the processor performs or supports performing the following process: causing the dynamic binary generator to generate the test binary, inputting a data generation ratio that sets the generation ratio of the first reproducible data to the second reproducible data into the dynamic binary generator, such that the dynamic binary generator generates multiple test binary values ​​for multiple first binary values ​​used to generate the first reproducible data and multiple second binary values ​​used to generate the second reproducible data, based on the data generation ratio.

[0035] As an example, in the (IV) process, the processor performs or supports performing the following processes: referring to the test binary value, (i) when the test binary is a first binary value, training the discriminator and the distribution analyzer using the first new discriminant loss, and training the data generator and the distribution analyzer using the first new generation loss, (ii) when the test binary is a second binary value, training the discriminator and the distribution analyzer using the second new discriminant loss, and training the data generator and the distribution analyzer using the second new generation loss.

[0036] As an example, the processor performs or supports the following processing: (i) when the discriminator and the distribution analyzer are trained using the first new discriminant loss and the second new discriminant loss respectively, training is performed with the first parameter of the data generator fixed; (ii) when the data generator and the distribution analyzer are trained using the first new generation loss and the second new generation loss respectively, training is performed with the second parameter of the discriminator fixed.

[0037] As an example, the data generator includes at least one encoding layer and at least one decoding layer.

[0038] In addition, the present invention also provides a computer-readable recording medium for recording a computer program for performing the method of the present invention.

[0039] Technical effect

[0040] The advantage of this invention is that it no longer stores the entire existing database used to train the neural network model, but only stores the SelectiveDeepGenerativeReplay Module, thereby reducing the storage space required to store the existing database.

[0041] Another advantage of this invention is that, instead of using the entire existing database and the entire new database for learning, a certain amount of data is generated by using a selective deep generative replay module to train the neural network model, thereby reducing the time required for learning.

[0042] Furthermore, another advantage of this invention is that instead of storing the entire existing database and using both the entire existing database and the entire new database to train the neural network model, it trains the neural network model by using a selective deep generative replay module of a Generative Adversarial Network (GAN) to generate data with distributions similar to and dissimilar to the new database at a selective ratio, thereby generating a neural network model optimized for a given task. Attached Figure Description

[0043] The following drawings, which are used to describe embodiments of the present invention, are only a part of the embodiments of the present invention, and those skilled in the art to which the present invention pertains (hereinafter referred to as "skilled persons") can obtain other drawings based on these drawings without any creative work.

[0044] Figure 1 A schematic diagram illustrating the configuration of existing continuous learning methods;

[0045] Figure 2 To compare with existing continuous learning methods, a schematic diagram of a method for adjusting continuous learning of a deep neural network model using a Selective Deep Generative Replay Module according to an embodiment of the present invention is provided.

[0046] Figure 3 This is a schematic diagram of a learning apparatus for initial learning of a selective deep generation and replay module according to an embodiment of the present invention.

[0047] Figure 4 A schematic diagram illustrating a method for initial learning of a selective deep generation replay module using a learning apparatus according to an embodiment of the present invention;

[0048] Figure 5 This is a schematic diagram of a continuous learning apparatus for testing and adjusting continuous learning of a selective deep generation replay module according to an embodiment of the present invention.

[0049] Figure 6 This is a schematic diagram of a method for testing a selective depth generation and replay module using a continuous learning device according to an embodiment of the present invention;

[0050] Figure 7 This is a schematic diagram of a method for a selective depth generation and replay module to be continuously learned by a continuous learning apparatus according to an embodiment of the present invention. Detailed Implementation

[0051] The following detailed description of the invention is given with reference to the accompanying drawings, which illustrate specific embodiments in which the invention can be practiced to explain the objectives, technical solutions, and advantages of the invention. These embodiments are described in sufficient detail to enable those skilled in the art to practice the invention.

[0052] Furthermore, in the description and claims of this invention, the term "comprising" and its variations are not intended to exclude other technical features, additions, components, or steps. Other objects, advantages, and features of this invention will be apparent to those skilled in the art, in part from this specification and in part from practice of the invention. The following illustrations and figures are provided as examples and are not intended to limit the scope of the invention.

[0053] Furthermore, the present invention includes all possible combinations of the embodiments shown in this specification. It should be understood that the various embodiments of the present invention, though different, are not necessarily mutually exclusive. For example, specific shapes, structures, and characteristics described herein may be implemented in one embodiment by other embodiments without departing from the spirit and scope of the invention. Furthermore, it should be understood that the position or configuration of the components in each disclosed embodiment may be varied without departing from the spirit and scope of the invention. Therefore, the detailed description that follows is not intended to limit the invention; the scope of the invention should be defined by all scopes equivalent to those of the claims and the appended claims, provided that appropriate description is possible. Similar reference numerals in the drawings indicate the same or similar functions in several respects.

[0054] To enable those skilled in the art to readily implement the present invention, preferred embodiments of the invention will now be described in detail with reference to the accompanying drawings.

[0055] Figure 2 To compare with existing continuous learning methods, this invention presents a schematic diagram of a method for performing adjustable continuous learning on a deep neural network model using a Selective Deep Generative Replay Module, according to an embodiment of the present invention.

[0056] from Figure 2 and Figure 1In comparison, it can be seen that in the tunable continuous learning method applying the selective deep generative replay module, the selective deep generative replay module acts as a database replay module for the Generative Adversarial Network (GAN), replacing the storage of the database. Furthermore, the selective deep generative replay module can adjust the ratio of Class 1 data to Class 2 data. Specifically, the feature distribution of Class 1 data can have a mean and variance within each threshold range determined based on the mean and variance of the sub-database or the new database. Further, the feature distribution of Class 2 data can have a mean and variance outside each threshold range determined based on the mean and variance of the sub-database or the new database. Here, the sub-database can be the entire database, i.e., a subset of the existing database.

[0057] That is, in the tunable continuous learning method using a selective deep generation and replay module, only the selective deep generation and replay module is stored, instead of the entire database. This reduces the required storage space. Furthermore, instead of using the entire existing database, a dataset consisting of Class 1 and Class 2 data is used to train the neural network model, thereby generating a neural network model optimized for a given task. The Class 1 and Class 2 data can be pre-set with reference to the given task, but this invention is not limited to this. In this invention, Class 1 data can refer to data with a similarity higher than a preset similarity threshold to the input data, and Class 2 data can refer to data with a similarity lower than another preset similarity threshold to the input data.

[0058] Among them, the selective deep generative replay module can be applied to devices that require neural network models, such as autonomous vehicles, autonomous aircraft, and robots, and can be used for continuous learning of neural network models.

[0059] Correspondingly, Figures 3 to 7 A method is schematically illustrated that performs initial learning on a selective deep generative replay module, tests the initially learned selective deep generative replay module, and performs continuous learning on the selective deep generative replay module to perform tunable continuous learning of a neural network model.

[0060] first, Figure 3 This is a schematic diagram of a learning apparatus 1000 for initial learning of a selective depth generation and replay module according to an embodiment of the present invention.

[0061] See Figure 3 The learning device may include: a memory 1001 storing instructions for initial learning of the selective depth generation and replay module; and a processor 1002 that responds to the instructions stored in the memory 1001 for initial learning of the selective depth generation and replay module.

[0062] Specifically, the learning device 1000 can typically achieve the required system performance using a combination of computing devices (e.g., computer processors, memory, storage devices, input and output devices, and other components that may include conventional computing devices; electronic communication devices, such as routers, switches, etc.; electronic information storage systems, such as network attached storage (NAS) and storage area networks (SAN)) and computer software (i.e., instructions that enable the computing device to operate in a particular manner).

[0063] Additionally, the processor of a computing device may include hardware configurations such as a microprocessor unit (MPU) or central processing unit (CPU), cache memory, and a data bus. Furthermore, the computing device may include a software configuration such as an operating system and applications for executing specific purposes.

[0064] However, it is not excluded that the computing device may include an integrated processor in the form of an integrated medium, processor, and memory for the purpose of implementing the present invention.

[0065] Reference Figure 4 A method for initial learning of a selective depth generation and replay module according to an embodiment of the present invention is described using a learning device 1000 configured as described above.

[0066] First, in order to perform initial learning for the selective deep generation and replay module 100, the learning device 1000 can obtain first learning data from the entire database, i.e., from the existing database, and second learning data from a sub-database, which is a subset of the entire database.

[0067] Next, the learning device 1000 can input the first learning data and the second learning data into the selective depth generation and playback module 100.

[0068] At this time, the Selective Deep Generative Replay Module 100 can input the first training data and the second training data into the Distribution Analyzer 110 located in the Selective Deep Generative Replay Module 100, respectively, so that the Distribution Analyzer 110 generates a first low-dimensional distribution feature for training corresponding to the first training data and a second low-dimensional distribution feature for training corresponding to the second training data. The distribution feature can represent the distribution of features for at least some pixels in the image, the distribution of the positions of objects in the image, the distribution of classification scores for objects in the image, etc. This invention is not limited to these; it can represent the distribution of various information that can be extracted from all data. Furthermore, among the features included in the input data to the selective deep generation and replay module 100, when capturing useful features for the tunable continuous learning of the deep neural network model, irrelevant / unnecessary features can lead to the "curse of dimensionality" problem. To prevent this problem, low-dimensional distributed features can be generated by projecting the input data to a lower-dimensional subspace, and dimensionality reduction can be achieved through methods such as feature extraction or feature selection.

[0069] Next, the selective depth generation and replay module 100 can input the learning binary generated from the dynamic binary generator 120, the learning random parameters generated from the random parameter generator 130, and the second learning low-dimensional distribution feature into the data generator 140, so that the data generator 140 generates the third learning data corresponding to the second learning low-dimensional distribution feature based on the learning binary and the learning random parameters. The data generator 140 may include at least one encoding layer and at least one decoding layer.

[0070] At this point, the data generator 140 generates third learning data by using the input learning random parameters and the second learning low-dimensional distribution features. The third learning data corresponds to the learning binary and can be data within a distribution similar to the distribution of the second learning low-dimensional distribution features or data outside a similar distribution. For example, when the learning binary is the first binary value, the mean and variance of the feature distribution of the third learning data can be within the threshold range of each learning data determined based on the mean and variance corresponding to the second learning low-dimensional distribution features. When the learning binary is the second binary value, the mean and variance of the feature distribution of the third learning data can be outside the threshold range of each learning data determined based on the mean and variance of the second learning low-dimensional distribution features. The learning random parameters can represent vectors or Gaussian distributions of image deformations such as random noise, image orientation, and attributes. Accordingly, third learning data with various distributions that can be used for learning can be generated based on the learning random parameters.

[0071] Additionally, the learning device 1000 can input the first learning data into the solver 150 located in the selective depth generation and playback module 100, so that the solver 150 outputs learning annotation data based on deep learning to annotate the first learning data.

[0072] Next, the learning device 1000 can input the first learning data, the second learning data, the first learning low-dimensional distribution feature, the second learning low-dimensional distribution feature, the third learning data, and the learning binary data into the discriminator 200, and make the discriminator 200 correspond to the learning binary data, and output the first learning data score, the second learning data score, the first distribution feature score, the second distribution feature score, and the third learning data score.

[0073] Next, in order to learn the distribution analyzer 110, the data generator 140, and the discriminator 200, the learning device 1000 can generate or support the generation of a first discriminator loss, a second discriminator loss, a first generator loss, and a second generator loss, all referencing the second learning data score, the second distribution feature score, and the third learning data score.

[0074] Accordingly, the learning device 1000 can train the discriminator 200, the data generator 140, and the distribution analyzer 110, such that the data generator 140 generates third learning data that, when the learning binary is the first binary value, includes a distribution having at least one of the means and variances within each learning data threshold range determined based on the means and variances of the second low-dimensional learning distribution characteristics; and generates third learning data that, when the learning binary is the second binary value, includes a distribution having at least one of the means and variances outside the each learning data threshold range determined based on the means and variances of the second low-dimensional learning distribution characteristics. The first and second binary values ​​can be binary input values, such as numbers like "1" and "0" or characters like "a" and "b".

[0075] Therefore, the learning device 1000 can perform or support performing the following processes: referring to the learning binary, (i) when the learning binary is the first binary value, training the discriminator 200 and the distribution analyzer 110 using the first discriminant loss, and training the data generator 140 and the distribution analyzer 110 using the first generative loss, (ii) when the learning binary is the second binary value, training the discriminator 200 and the distribution analyzer 110 using the second discriminant loss, and training the data generator 140 and the distribution analyzer 110 using the second generative loss.

[0076] At this time, the learning device 1000 can perform or support the following processing: (i) in order to perform the above learning, when the discriminator 200 and the distribution analyzer 110 are trained using the first discriminant loss and the second discriminant loss respectively, the training is performed with the first parameter of the data generator 140 fixed; (ii) when the data generator 140 and the distribution analyzer 110 are trained using the first generation loss and the second generation loss respectively, the training is performed with the second parameter of the discriminator 200 fixed.

[0077] Since each of the distribution analyzer 110, data generator 140, and discriminator 200, which are all composed of deep network models, is interconnected, they can be learned through back-propagation using the first discriminant loss, the second discriminant loss, the first generator loss, or the second generator loss. Therefore, the learning of the distribution analyzer 110, data generator 140, and discriminator 200 using the aforementioned losses can be performed using the following target function.

[0078] if binary=first binary value,

[0079]

[0080]

[0081] else

[0082]

[0083]

[0084] In the above formula, x represents the first or second learning data, z represents the learning random parameter, and DF ED The first learning method uses low-dimensional distribution characteristics, DF sub G(z) represents the low-dimensional distribution characteristics of the second learning, and G(z) represents the data of the third learning.

[0085] The learning rules described above are basically similar to the learning methods of existing Generative Adversarial Networks (GANs), but the difference lies in that, in order for the discriminator 200 to correspond to multiple sub-databases or existing databases, it further receives a first learning low-dimensional distribution feature and a second learning low-dimensional distribution feature. Furthermore, in the above learning process, the distribution analyzer 110 participates in the learning terms of both the data generator 140 and the discriminator 200, and only the part connected to that term receives back-propagation.

[0086] Through such learning, the data generator 140 can output, when the learning binary is a first binary value, a first type of data included in an existing database that has characteristics of a distribution having at least one of the means and variances within each learning data threshold range determined based on the means and variances of the second learning low-dimensional distribution characteristics as input data; and can output, when the learning binary is a second binary value, a second type of data included in an existing database that has characteristics of a distribution having at least one of the means and variances outside each learning data threshold range determined based on the means and variances of the second learning low-dimensional distribution characteristics as input data. Then, when new data based on continuous learning is input, the data generator 140 can output a first type of data included in an existing database, characterized by a distribution having at least one of the means and variances within a threshold range of each test data determined based on the means and variances of the new data as input data, and a second type of data included in an existing database, characterized by a distribution having at least one of the means and variances outside a threshold range of each test data determined based on the means and variances of the new data as input data, thus eliminating the need to store an existing database as in the prior art.

[0087] In addition, the learning device 1000 can perform the following processing: in order to improve the accuracy of the learning labeled data generated by the solver 150, a solver loss is further generated by referring to the learning labeled data and the corresponding ground truth, and the solver 150 is further trained using the solver loss.

[0088] After the initial learning of the selective depth generation and replay module 100 is completed, according to an embodiment of the present invention, the continuous learning device 2000 tests the initially learned selective depth generation and replay module and performs continuous learning on the selective depth generation and replay module in order to perform adjustable continuous learning on the neural network model.

[0089] See Figure 5The continuous learning device 2000 may include: a memory 2001 storing instructions for testing and continuously learning the selective depth generation and playback module 100; and a processor 2002 that responds to the instructions stored in the memory 2001 for testing and continuously learning the selective depth generation and playback module 100.

[0090] Specifically, the continuous learning device 2000 can typically achieve the required system performance using a combination of computing devices (e.g., computer processors, memory, storage devices, input and output devices, and other components that may include conventional computing devices; electronic communication devices such as routers, switches, etc.; electronic information storage systems such as network attached storage (NAS) and storage area networks (SAN)) and computer software (i.e., instructions that enable the computing device to operate in a particular manner).

[0091] Additionally, the processor of a computing device may include hardware configurations such as a microprocessor unit (MPU) or central processing unit (CPU), cache memory, and a data bus. Furthermore, the computing device may include a software configuration such as an operating system and applications for executing specific purposes.

[0092] However, it is not excluded that the computing device may include an integrated processor in the form of an integrated medium, processor, and memory for the purpose of implementing the present invention.

[0093] On the other hand, the learning device 1000 and the continuous learning device 2000 can be installed together on a device employing a neural network model that performs adjustable continuous learning by applying a selective deep generation replay module. Alternatively, they can be installed on a shared server or shared device, and the learning data and labeled learning data generated for adjustable continuous learning can be remotely transmitted to a server or device for performing adjustable continuous learning of the neural network model. However, the present invention is not limited thereto.

[0094] Reference Figure 6 and Figure 7 A method is described for testing and continuously learning a selective depth generation and playback module 100 according to an embodiment of the present invention using a continuous learning device 2000 configured as described above.

[0095] Figure 6 This is a schematic diagram illustrating a method for testing a selective depth generation and playback module 100 using a continuous learning device 2000 according to an embodiment of the present invention. Details will be omitted below. Figure 4 The description is a detailed description of the parts that are easy to understand.

[0096] First, as referenced Figure 4 The continuous learning device 2000 can acquire new data from the newly collected database while the learning device 1000 is in the state of initial learning of the selective depth generation and replay module 100.

[0097] Then, the continuous learning device 2000 can input the acquired new data into the selective depth generation and replay module 100.

[0098] Then, the selective deep generation and replay module 100 can input the new data into the distribution analyzer 110, causing the distribution analyzer 110 to generate a new low-dimensional distribution feature corresponding to the new data. Among the features included in the input data to the selective deep generation and replay module 100, when capturing useful features for tunable continuous learning of a deep neural network model, irrelevant / unnecessary features can lead to the "curse of dimensionality" problem. To prevent this problem, low-dimensional distribution features can be generated by projecting the input data into a lower-dimensional subspace, and dimensionality reduction can be achieved through methods such as feature extraction or feature selection.

[0099] Additionally, the selective deep generation and replay module 100 can input test binary data generated from the dynamic binary generator 120, test random parameters generated from the random parameter generator 130, and new low-dimensional distribution features into the data generator 140. This allows the data generator 140 to generate test data corresponding to the new low-dimensional distribution features based on the test binary data and the test random parameters. At this time, with the entire database and sub-databases used for initial learning deleted, the data generator 140 generates test data corresponding to the distribution of the new data. The test data can be data with the same or similar distribution as the data included in the entire database and sub-databases. Additionally, the test data may include first regeneration data and second regeneration data. The first regeneration data, when the test binary is a first binary value, includes features of a distribution having at least one of the means and variances within a threshold range for each test data determined based on the new low-dimensional distribution characteristics. The second regeneration data, when the test binary is a second binary value, includes features of a distribution having at least one of the means and variances outside the threshold range for each test data determined based on the new low-dimensional distribution characteristics. The first and second binary values ​​can be binary input values, such as numbers like "1" and "0" or characters like "a" and "b".

[0100] Next, the continuous learning device 200 can input the test data into the solver 150, so that the solver 150 outputs test annotation data that annotates the test data.

[0101] At this time, the continuous learning device 2000 generates or supports the following process: the dynamic binary generator 120 generates test binary data, and the data generation ratio that sets the generation ratio of the first reproducible data and the second reproducible data is input into the dynamic binary generator 120, so that the dynamic binary generator 120 generates multiple test binary data for multiple first binary values ​​used to generate the first reproducible data and multiple second binary values ​​used to generate the second reproducible data according to the data generation ratio.

[0102] Next, Figure 7 This is a schematic diagram of a method for a continuous learning device 2000 to continuously learn a selective depth generation and playback module 100 according to an embodiment of the present invention.

[0103] First, the continuous learning device 2000 first replicates the initially learned selective deep generative replay module 100 to generate an old selective deep generative replay module 300, and inputs an old low-dimensional distribution feature, including at least a portion of the first learning low-dimensional distribution feature and the second training low-dimensional distribution feature, into the old selective deep generative replay module 300 so that the old selective deep generative replay module 300 generates old data corresponding to the old low-dimensional distribution feature and old labeled data corresponding to the old data.

[0104] Thus, while generating old data and corresponding old labeled data, the continuous learning device 2000 inputs the old data, new data, old low-dimensional distribution features, new low-dimensional distribution features, test data, and test binary data into the discriminator 200, and makes the discriminator 200 output the old data score, new data score, old distribution feature score, new distribution feature score, and test data score to the test binary data.

[0105] In addition, the continuous learning device 2000 inputs old and new data into the solver 150, so that the solver 150 outputs new labeled data based on deep learning to annotate the old and new data.

[0106] Next, in order to learn the distribution analyzer 110, the data generator 140, and the discriminator 200, the continuous learning device 2000 can generate a first new discrimination loss based on the reference new data score, the new distribution feature score, and the test data; a second new discrimination loss based on the reference old data score, the old distribution feature score, and the test data score; a first new generation loss based on the reference new distribution feature score and the test data score; and a second new generation loss based on the reference old distribution feature score, the new distribution feature score, and the test data score.

[0107] Accordingly, the continuous learning device 2000 can train the discriminator 200, the data generator 140, and the distribution analyzer 110 such that the data generator 140 generates test data with characteristics of a distribution having at least one of the means and variances within each test data threshold range determined based on the new low-dimensional distribution characteristics when the test binary is a first binary value, and generates test data with characteristics of a distribution having at least one of the means and variances outside the respective test data threshold ranges determined based on the new low-dimensional distribution characteristics when the test binary is a second binary value.

[0108] Therefore, the continuous learning device 2000 can perform or support performing the following processes: referring to the test binary, (i) when the test binary is the first binary value, training the discriminator 200 and the distribution analyzer 110 using the first new discriminant loss, and training the data generator 140 and the distribution analyzer 110 using the first new generation loss, (ii) when the test binary is the second binary value, training the discriminator 200 and the distribution analyzer 110 using the second new discriminant loss, and training the data generator 140 and the distribution analyzer 110 using the second new generation loss.

[0109] At this time, the continuous learning device 2000 can perform or support the following processing: (i) in order to perform the above learning, when the discriminator 200 and the distribution analyzer 110 are trained using the first new discriminant loss and the second new discriminant loss respectively, the training is performed with the first parameter of the data generator 140 fixed; (ii) when the data generator 140 and the distribution analyzer 110 are trained using the first new generation loss and the second new generation loss respectively, the training is performed with the second parameter of the discriminator 200 fixed.

[0110] Since the distribution analyzer 110, data generator 140, and discriminator 200, each composed of a deep network model, are interconnected, they can be learned through back-propagation using the first new discriminant loss, the second new discriminant loss, the first new generator loss, or the second new generator loss. Therefore, the learning of the distribution analyzer 110, data generator 140, and discriminator 200 using the aforementioned losses can be performed using the following target function.

[0111] if binary=first binary value,

[0112]

[0113]

[0114] else

[0115]

[0116]

[0117] In the above formula, x represents old or new data, z represents the random parameter used in the test, and DF ED Represents the characteristics of the old low-dimensional distribution, DF sub G(z) represents the new low-dimensional distribution characteristics, and G(z) represents the test data.

[0118] The learning rules described above are essentially similar to the learning methods used for initial learning, but the difference lies in that old low-dimensional distribution features, including at least a portion of the stored first and second low-dimensional distribution features, are input into the discriminator 200. Furthermore, the difference in continuous learning is that the old low-dimensional distribution features are input into the old selective deep generation replay module, causing the old selective deep generation replay module to generate old-shape data corresponding to the old low-dimensional distribution features, which is then used as input values ​​for the distribution analyzer 110 and the discriminator 200.

[0119] In addition, the continuous learning device 2000 can perform the following processing: in order to improve the accuracy of the new labeled data generated by the solver 150, a new solver loss is generated with reference to the new labeled data and the old labeled data, and the solver 150 is trained using the new solver loss.

[0120] The continuous learning device 2000 can use the method described above to continuously learn the selective depth generation and replay module 100 for each iteration, but is not limited thereto. To perform continuous learning of the selective depth generation and replay module 100, the period for updating the old selective depth generation and replay module 300 can be complementaryly influenced with the period of the adjustable continuous learning of the neural network model. For example, when it is necessary to improve the performance of the neural network model, or when it is necessary to adjust the continuous learning of the neural network model to perform a new task, the selective depth generation and replay module 100 can be continuously learned to generate learning data with a new ratio.

[0121] On the other hand, although the continuous learning method is described above as being performed immediately after the initial learning, it can also be applied to consecutive rounds of continuous learning.

[0122] That is, in reference Figure 6 and Figure 7 In the described method, a new database can replace the reference database. Figure 4 The initial learning uses a sub-database, and continuous learning can be performed using the next database obtained for subsequent continuous learning.

[0123] The embodiments described above according to the present invention can be implemented and recorded in a computer-readable recording medium in the form of program instructions executable by various computer components. The computer-readable recording medium may include individual or combined program instructions, data files, data structures, etc. The program instructions recorded in the computer-readable recording medium may be specifically designed and configured for the present invention, or may be known and available to those skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical recording media such as CD-ROMs and DVDs; magneto-optical media such as floppy disks; and hardware devices specifically configured for storing and executing program instructions, such as ROMs, RAMs, flash memory, etc. Examples of program instructions include not only machine language code generated by a compiler, but also high-level language code that can be executed by a computer using an interpreter, etc. The hardware device may be configured to operate as at least one software module to perform the processing according to the present invention, and vice versa.

[0124] In the foregoing, the invention has been described with reference to specific matters such as particular components, as well as limited embodiments and accompanying drawings. However, this is only to help to understand the invention more fully, and the invention is not limited to the embodiments described above. Various modifications and variations can be designed by those skilled in the art based on these descriptions.

[0125] Therefore, the spirit of the present invention should not be limited to the above embodiments. Except for the appended claims, all modifications that are equivalent or analogous to these claims should be included within the spirit and scope of the present invention.

Claims

1. A method for tunable continuous learning of a deep neural network model using a selective deep generation and replay module for image classification in vehicles, the method comprising: (a) When first learning data is obtained from the entire database used for performing image classification in vehicles and second learning data is obtained from a sub-database that is a subset of the entire database, the learning device performs or supports performing the following process: inputting the first learning data and the second learning data into a selective depth generation and replay module such that the selective depth generation and replay module (i) generates a first learning low-dimensional distribution feature corresponding to the first learning data and a second learning low-dimensional distribution feature corresponding to the second learning data through a distribution analyzer located in the selective depth generation and replay module, wherein the dimension of the first learning low-dimensional distribution feature is lower than the dimension of the first learning data, and the dimension of the second learning low-dimensional distribution feature is lower than the dimension of the first learning data.

2. Dimensions of the learning data: (ii) The learning binary generated from the dynamic binary generator in the selective depth generation and replay module, the learning random parameters generated from the random parameter generator in the selective depth generation and replay module, and the second learning low-dimensional distribution feature are input to the data generator in the selective depth generation and replay module. The data generator generates a third learning data corresponding to the second learning low-dimensional distribution feature based on the learning binary and the learning random parameters. (iii) The first learning data is input to the solver in the selective depth generation and replay module. The solver outputs learning labeled data based on deep learning that annotates the first learning data. (b) The learning device performs or supports performing the following processing: inputting the first learning data, the second learning data, the first learning low-dimensional distribution feature, the second learning low-dimensional distribution feature, the third learning data, and the learning binary data into a discriminator, such that the discriminator outputs the first learning data score, the second learning data score, the first distribution feature score, the second distribution feature score, and the third learning data score corresponding to the learning binary data; and (c) The learning device performs or supports performing the following processes: (i) generating a first discriminant loss, a second discriminant loss, a first generation loss, and a second generation loss, referencing the second learning data score, the second distribution feature score, and the third learning data score; and a third generation loss, referencing the first learning data score, the first distribution feature score, and the third learning data score, to train the discriminator, the data generator, and the distribution analyzer, such that the data generator generates data in the learning data... The third learning data, when the binary is the first binary value, includes a distribution having characteristics of at least one of the means and variances within each learning data threshold range determined based on the means and variances of the second learning low-dimensional distribution features. The third learning data, when the learning binary is the second binary value, includes a distribution having characteristics of at least one of the means and variances outside the each learning data threshold range determined based on the means and variances of the second learning low-dimensional distribution features. (ii) A solver loss is further generated with reference to the learning labeled data and the corresponding real data, and the solver is further trained using the solver loss.

2. The method according to claim 1, characterized in that: In step (c), the learning device performs or supports performing the following processing: referring to the learning binary value, (i) when the learning binary is a first binary value, training the discriminator and the distribution analyzer using the first discriminant loss, and training the data generator and the distribution analyzer using the first generation loss, and (ii) when the learning binary is a second binary value, training the discriminator and the distribution analyzer using the second discriminant loss, and training the data generator and the distribution analyzer using the second generation loss.

3. The method according to claim 2, characterized in that: The learning device performs or supports the following processes: (i) when the discriminator and the distribution analyzer are trained using the first discriminant loss and the second discriminant loss respectively, training is performed with the first parameter of the data generator fixed; (ii) when the data generator and the distribution analyzer are trained using the first generation loss and the second generation loss respectively, training is performed with the second parameter of the discriminator fixed.

4. The method according to claim 1, characterized in that: The data generator includes at least one encoding layer and at least one decoding layer.

5. A method for tunable continuous learning of a deep neural network model using a selective deep generation and replay module for image classification in vehicles, the method comprising: (a) When the learning device obtains first learning data from the entire database used for performing image classification in vehicles and second learning data from a sub-database that is a subset of the entire database, the learning device performs or supports performing the following processing: (I) inputting the first learning data and the second learning data into a selective depth generation and replay module such that the selective depth generation and replay module (i) generates a first learning low-dimensional distribution feature corresponding to the first learning data and a second learning low-dimensional distribution feature corresponding to the second learning data through a distribution analyzer located in the selective depth generation and replay module, wherein the dimension of the first learning low-dimensional distribution feature is lower than the dimension of the first learning data, and the dimension of the second learning low-dimensional distribution feature is lower than the dimension of the second learning data; (ii) generating a learning binary from a dynamic binary generator located in the selective depth generation and replay module, and a random parameter from a random parameter generator located in the selective depth generation and replay module. (iii) The first learning data is input into the solver located in the selective deep generation and replay module. The solver outputs the learning labeled data based on the learning binary and the learning random parameters. (ii) The first learning data, the second learning data, the first learning low-dimensional distribution feature, the second learning low-dimensional distribution feature, the third learning data, and the learning binary are input into the discriminator. The discriminator outputs the learning binary scores, the second learning data scores, the first distribution feature scores, the second distribution feature scores, and the third learning data scores.And (III)(i) generating a first discriminant loss, a second discriminant loss, a first generation loss, and a second generation loss, referencing the second learning data score, the second distribution feature score, and the third learning data score, to train the discriminator, the data generator, and the distribution analyzer, such that the data generator generates a value when the learning binary is the first binary value, including a mean and variance determination based on the second learning low-dimensional distribution feature. The third learning data, which has the characteristics of a distribution of at least one of the means and variances within the threshold range of each learning data, is generated when the learning binary is the second binary value. This third learning data has the characteristics of a distribution of at least one of the means and variances outside the threshold range of each learning data determined based on the means and variances of the second learning low-dimensional distribution. (ii) The solver loss is further generated with reference to the learning labeled data and the corresponding real data, and the solver is further trained using the solver loss. The continuous learning device performs or supports the execution of processing to obtain new data from a newly collected new database while the selective deep generation replay module is initially learned by the learning device. as well as (b) The continuous learning device performs or supports the following processing: inputting the new data into the selective depth generation and replay module such that the selective depth generation and replay module (i) generates a new low-dimensional distribution feature corresponding to the new data through the distribution analyzer, wherein the dimension of the new low-dimensional distribution feature is lower than the dimension of the new data; (ii) inputting the test binary generated from the dynamic binary generator, the test random parameters generated from the random parameter generator, and the new low-dimensional distribution feature into the data generator, and generating test data corresponding to the new low-dimensional distribution feature through the data generator based on the test binary and the test random parameters, wherein the test data... The trial data includes first reproducible data and second reproducible data. The first reproducible data, when the test binary is a first binary value, includes a distribution having at least one of the mean and variance within each test data threshold range determined based on the new low-dimensional distribution characteristics. The second reproducible data, when the test binary is a second binary value, includes a distribution having at least one of the mean and variance outside the each test data threshold range determined based on the new low-dimensional distribution characteristics. (iii) The test data is input into the solver, and the solver outputs test annotation data annotated with the test data.

6. The method according to claim 5, characterized in that, Also includes: (c) In a state where an old selective depth generation and replay module is generated by copying the selective depth generation and replay module, and an old low-dimensional distribution feature including at least a portion of the first learning low-dimensional distribution feature and the second learning low-dimensional distribution feature is input into the old selective depth generation and replay module so that the old selective depth generation and replay module generates old data corresponding to the old low-dimensional distribution feature and old labeled data corresponding to the old data, the continuous learning device performs or supports performing the following processing: (i) inputting the old data, the new data, the old low-dimensional distribution feature, the new low-dimensional distribution feature, the test data, and the test binary into the discriminator so that the discriminator outputs old data score, new data score, old distribution feature score, new distribution feature score, and test data score corresponding to the test binary; (ii) inputting the old data and the new data into the solver so that the solver outputs new labeled data based on deep learning to annotate the old data and the new data; and (d) The continuous learning device performs or supports performing the following processing: (i) generating a first new discriminant loss with reference to the new data score, the new distribution feature score, and the test data score; a second new discriminant loss with reference to the old data score, the old distribution feature score, and the test data score; a first new generation loss with reference to the new distribution feature score and the test data score; and a second new generation loss with reference to the old distribution feature score, the new distribution feature score, and the test data score to train the discriminator, the data generator, and the distribution analyzer, such that the data generator generates data on the test data score. When the binary representation is a first binary value, the test data includes the distribution having characteristics of at least one of the mean and variance within the threshold range of each test data determined based on the mean and variance of the new low-dimensional distribution characteristics. When the binary representation is a second binary value, the test data includes the distribution having characteristics of at least one of the mean and variance outside the threshold range of each test data determined based on the mean and variance of the new low-dimensional distribution characteristics. (ii) A new solver loss is generated with reference to the new labeled data and the old labeled data, and the solver is trained using the new solver loss.

7. The method according to claim 5, characterized in that: In step (b), the continuous learning device performs or supports the following process: causing the dynamic binary generator to generate the test binary, wherein a data generation ratio that sets the generation ratio of the first reproducible data to the second reproducible data is input to the dynamic binary generator, such that the dynamic binary generator generates multiple test binary values ​​for multiple first binary values ​​used to generate the first reproducible data and multiple second binary values ​​used to generate the second reproducible data, according to the data generation ratio.

8. The method according to claim 6, wherein: In step (d), the continuous learning device performs or supports performing the following processing: referring to the test binary value, (i) when the test binary is a first binary value, training the discriminator and the distribution analyzer using the first new discriminant loss, and training the data generator and the distribution analyzer using the first new generation loss, (ii) when the test binary is a second binary value, training the discriminator and the distribution analyzer using the second new discriminant loss, and training the data generator and the distribution analyzer using the second new generation loss.

9. The method according to claim 8, wherein: The continuous learning device performs or supports the following processing: (i) when the discriminator and the distribution analyzer are trained using the first new discriminant loss and the second new discriminant loss respectively, training is performed with the first parameter of the data generator fixed; (ii) when the data generator and the distribution analyzer are trained using the first new generation loss and the second new generation loss respectively, training is performed with the second parameter of the discriminator fixed.

10. The method according to claim 5, characterized in that: The data generator includes at least one encoding layer and at least one decoding layer.

11. A learning apparatus for image classification in vehicles that uses a selective deep generation and replay module to perform tunable continuous learning of a deep neural network model, comprising: At least one memory for storing instructions; and At least one processor for executing the instructions, The processor performs or supports performing the following processing: (I) when obtaining first learning data from the entire database used for performing image classification in vehicles and second learning data from a sub-database that is a subset of the entire database, the first learning data and the second learning data are input into a selective depth generation and replay module such that the selective depth generation and replay module (i) generates a first learning low-dimensional distribution feature corresponding to the first learning data and a second learning low-dimensional distribution feature corresponding to the second learning data through a distribution analyzer located in the selective depth generation and replay module, wherein the dimension of the first learning low-dimensional distribution feature is lower than the dimension of the first learning data, and the dimension of the second learning low-dimensional distribution feature is lower than the dimension of the second learning data; (ii) the learning binary generated from the dynamic binary generator located in the selective depth generation and replay module, and the random parameters from the selective depth generation and replay module... The learning random parameters generated in the generator and the second learning low-dimensional distribution feature are input to the data generator located in the selective deep generation and replay module. The data generator generates the third learning data corresponding to the second learning low-dimensional distribution feature based on the learning binary and the learning random parameters. (iii) The first learning data is input to the solver located in the selective deep generation and replay module. The solver outputs the learning labeled data based on deep learning to annotate the first learning data. (II) The first learning data, the second learning data, the first learning low-dimensional distribution feature, the second learning low-dimensional distribution feature, the third learning data and the learning binary are input to the discriminator. The discriminator is made to correspond to the learning binary and outputs the first learning data score, the second learning data score, the first distribution feature score, the second distribution feature score and the third learning data score.And (III)(i) generating a first discriminant loss, a second discriminant loss, a first generation loss, and a second generation loss, referencing the second learning data score, the second distribution feature score, and the third learning data score, to train the discriminator, the data generator, and the distribution analyzer, such that the data generator generates data in the learning binary as the first second... The third learning data, when the learning binary value is a binary number, includes the distribution having characteristics of at least one of the means and variances within each learning data threshold range determined based on the means and variances of the second learning low-dimensional distribution characteristics. The third learning data, when the learning binary value is a second binary value, includes the distribution having characteristics of at least one of the means and variances outside the ranges of the learning data thresholds determined based on the means and variances of the second learning low-dimensional distribution characteristics. (ii) A solver loss is further generated with reference to the learning labeled data and the corresponding real data, and the solver is further trained using the solver loss.

12. The learning device according to claim 11, characterized in that: In the (III) process, the processor performs or supports performing the following processes: referring to the learning binary value, (i) when the learning binary is a first binary value, training the discriminator and the distribution analyzer using the first discriminant loss, and training the data generator and the distribution analyzer using the first generation loss, (ii) when the learning binary is a second binary value, training the discriminator and the distribution analyzer using the second discriminant loss, and training the data generator and the distribution analyzer using the second generation loss.

13. The learning device according to claim 12, characterized in that: The processor performs or supports the following processes: (i) when the discriminator and the distribution analyzer are trained using the first discriminant loss and the second discriminant loss respectively, training is performed with the first parameter of the data generator fixed; (ii) when the data generator and the distribution analyzer are trained using the first generation loss and the second generation loss respectively, training is performed with the second parameter of the discriminator fixed.

14. The learning device according to claim 11, characterized in that: The data generator includes at least one encoding layer and at least one decoding layer.

15. A continuous learning apparatus for image classification in vehicles, comprising a selective deep generation and replay module for tunable continuous learning of a deep neural network model, comprising: At least one memory for storing instructions; and At least one processor for executing the instructions, The processor executes or supports the following processing: (I) The learning device executes or supports the following processing: when obtaining first learning data from the entire database used for performing image classification in vehicles and obtaining second learning data from a sub-database that is a subset of the entire database, (i) the first learning data and the second learning data are input into a selective depth generation and replay module such that the selective depth generation and replay module (i-1) generates a first learning low-dimensional distribution feature corresponding to the first learning data and a second learning low-dimensional distribution feature corresponding to the second learning data through a distribution analyzer located in the selective depth generation and replay module, wherein the dimension of the first learning low-dimensional distribution feature is lower than the dimension of the first learning data, and the dimension of the second learning low-dimensional distribution feature is lower than the dimension of the second learning data, (i-2) the learning binary generated from the dynamic binary generator located in the selective depth generation and replay module, and the learning binary generated from the dynamic binary generator located in the selective depth generation and replay module, are used to generate a first learning low-dimensional distribution feature corresponding to the first learning data and a second learning low-dimensional distribution feature corresponding to the second learning data through a distribution analyzer located in the selective depth generation and replay module. The random parameters for learning generated in the random parameter generator of the replay module and the second low-dimensional distribution feature for learning are input to the data generator located in the selective depth generation and replay module. The data generator generates the third learning data corresponding to the second low-dimensional distribution feature for learning based on the learning binary and the learning random parameters. (i-3) The first learning data is input to the solver located in the selective depth generation and replay module. The solver outputs the learning labeled data based on deep learning to annotate the first learning data. (ii) The first learning data, the second learning data, the first low-dimensional distribution feature for learning, the second low-dimensional distribution feature for learning, the third learning data and the learning binary are input to the discriminator. The discriminator is made to correspond to the learning binary and outputs the first learning data score, the second learning data score, the first distribution feature score, the second distribution feature score and the third learning data score.(iii) (iii-1) Generate a first discriminant loss, a second discriminant loss, a first generation loss, and a second generation loss, referencing the second learning data score, the second distribution feature score, and the third learning data score, respectively, to train the discriminator, the data generator, and the distribution analyzer, such that the data generator generates data with the first binary value, including features based on the second learning low-dimensional distribution feature. The third learning data, which has the characteristics of a distribution of at least one of the means and variances within each learning data threshold range determined by the means and variances of the second learning data, is generated when the learning binary is the second binary value. This third learning data has the characteristics of a distribution having at least one of the means and variances outside each learning data threshold range determined by the means and variances of the second learning low-dimensional distribution. (iii-2) A solver loss is further generated with reference to the learning labeled data and the corresponding real data, and the solver is further trained using the solver loss. New data is obtained from the newly collected new database while the selective deep generation replay module is initially learned by the learning device. And (II) inputting the new data into the selective depth generation and replay module, and causing the selective depth generation and replay module to (i) generate a new low-dimensional distribution feature corresponding to the new data through the distribution analyzer, wherein the dimension of the new low-dimensional distribution feature is lower than the dimension of the new data, and (ii) inputting the test binary generated from the dynamic binary generator, the test random parameters generated from the random parameter generator, and the new low-dimensional distribution feature into the data generator, and generating test data corresponding to the new low-dimensional distribution feature through the data generator based on the test binary and the test random parameters, wherein the test data includes the first...

1. Reproducible data and 2. Reproducible data, wherein the first reproducible data, when the test binary is a first binary value, includes a distribution having at least one of the means and variances within each test data threshold range determined based on the means and variances of the new low-dimensional distribution characteristics, and the second reproducible data, when the test binary is a second binary value, includes a distribution having at least one of the means and variances outside the test data threshold range determined based on the means and variances of the new low-dimensional distribution characteristics, and (iii) inputting the test data into the solver and outputting test annotation data labeled with the test data through the solver.

16. The continuous learning device according to claim 15, further comprising: (III) In a state where an old selective depth generation and replay module is generated by copying the selective depth generation and replay module, and an old low-dimensional distribution feature including at least a portion of the first learning low-dimensional distribution feature and the second learning low-dimensional distribution feature is input into the old selective depth generation and replay module so that the old selective depth generation and replay module generates old data corresponding to the old low-dimensional distribution feature and old labeled data corresponding to the old data, the processor performs or supports performing the following processing: (i) inputting the old data, the new data, the old low-dimensional distribution feature, the new low-dimensional distribution feature, the test data, and the test binary into the discriminator so that the discriminator outputs old data score, new data score, old distribution feature score, new distribution feature score, and test data score corresponding to the test binary; (ii) inputting the old data and the new data into the solver so that the solver outputs new labeled data based on deep learning to annotate the old data and the new data; and (IV) The processor performs or supports performing the following processing: (i) generating a first new discriminant loss with reference to the new data score, the new distribution feature score, and the test data score; a second new discriminant loss with reference to the old data score, the old distribution feature score, and the test data score; a first new generation loss with reference to the new distribution feature score and the test data score; and a second new generation loss with reference to the old distribution feature score, the new distribution feature score, and the test data score to train the discriminator, the data generator, and the distribution analyzer, such that the data generator generates data in the test binary... When the test data is a first binary value, it includes the test data having the characteristics of a distribution having at least one of the mean and variance within the threshold range of each test data determined based on the mean and variance of the new low-dimensional distribution characteristics. When the test binary is a second binary value, it includes the test data having the characteristics of a distribution having at least one of the mean and variance outside the threshold range of each test data determined based on the mean and variance of the new low-dimensional distribution characteristics. (ii) A new solver loss is generated with reference to the new labeled data and the old labeled data, and the solver is trained using the new solver loss.

17. The continuous learning device according to claim 15, characterized in that: In the (II) process, the processor performs or supports performing the following process: causing the dynamic binary generator to generate the test binary, inputting a data generation ratio that sets the generation ratio of the first reproducible data and the second reproducible data into the dynamic binary generator, such that the dynamic binary generator generates multiple test binary values ​​for multiple first binary values ​​used to generate the first reproducible data and multiple second binary values ​​used to generate the second reproducible data according to the data generation ratio.

18. The continuous learning device according to claim 16, characterized in that: In the (IV) process, the processor performs or supports performing the following processes: referring to the test binary value, (i) when the test binary is a first binary value, training the discriminator and the distribution analyzer using the first new discriminant loss, and training the data generator and the distribution analyzer using the first new generation loss, (ii) when the test binary is a second binary value, training the discriminator and the distribution analyzer using the second new discriminant loss, and training the data generator and the distribution analyzer using the second new generation loss.

19. The continuous learning device according to claim 18, characterized in that: The processor performs or supports the following processing: (i) when the discriminator and the distribution analyzer are trained using the first new discriminant loss and the second new discriminant loss respectively, training is performed with the first parameter of the data generator fixed; (ii) when the data generator and the distribution analyzer are trained using the first new generation loss and the second new generation loss respectively, training is performed with the second parameter of the discriminator fixed.

20. The continuous learning device according to claim 15, characterized in that: The data generator includes at least one encoding layer and at least one decoding layer.

Images