Method for training a deep learning network based on artificial intelligence and learning device using the same
By extracting difficult samples through active learning networks and automatic annotation networks, and combining lifelong learning and interpretable analysis networks to optimize deep learning training, the problem of reduced training effectiveness of deep learning networks is solved, data annotation efficiency and learning effect are improved, and human intervention is reduced.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- STRADVISION
- Filing Date
- 2020-12-17
- Publication Date
- 2026-07-21
AI Technical Summary
In the training process of deep learning networks in existing technologies, the number of difficult samples gradually decreases, leading to a decline in training effectiveness. The return on investment in human-labeled data decreases, and it is difficult to identify the problems in the learning algorithm and data collection strategy. Reliance on human engineers' experience leads to inefficient trial and error.
We employ an AI-based approach, using an active learning network to extract difficult samples, an automatic annotation network to generate new labeled data, and a lifelong learning network and an interpretable analysis network to optimize the training process, providing insightful performance analysis results to improve the learning model.
It improves the return on investment for data annotation, maximizes the learning effect of deep learning networks, reduces human intervention, and improves the accuracy and efficiency of performance analysis.
Smart Images
Figure CN115244548B_ABST
Abstract
Description
[0001] This application is a non-provisional application of U.S. Provisional Application No. 63 / 000,202, filed on March 26, 2020, the entire contents of which are incorporated herein by reference. Technical Field
[0002] This invention relates to a method for training a deep learning network and a learning device using the method, and more specifically, to a method for training a deep learning network based on artificial intelligence (AI) and a learning device using the method. Background Technology
[0003] In recent years, research has been conducted on methods that utilize machine learning to perform object recognition and other tasks.
[0004] Deep learning is a type of machine learning that utilizes neural networks with multiple hidden layers between the input and output layers, and it has high recognition performance.
[0005] In addition, neural networks using deep learning typically learn by utilizing backpropagation of the loss.
[0006] To train such a deep learning network, existing techniques involve collecting raw data according to a data collection strategy. Human annotators then label the collected raw data to generate new training data. The deep learning network is then trained using both the new and existing training data. Human engineers then modify and improve the learning algorithm used to train the deep learning network based on performance analysis results. Additionally, the data collection strategy is modified based on the analysis results, and any incorrect annotations are re-checked and corrected accordingly.
[0007] However, in such existing methods, as the performance of deep learning networks improves, the number of difficult samples that are helpful for training decreases, thus reducing the effectiveness of using new training data to improve the performance of deep learning networks, and decreasing the return on investment for data annotation by human annotators.
[0008] Furthermore, in existing methods, when new training data labeled by human annotators is added to existing training data in a set ratio, the existing training data gradually increases, and the learning effect of deep learning networks using the new training data gradually decreases.
[0009] Furthermore, existing methods cannot determine which part of the learning model—the learning algorithm, data collection strategy, training data, etc.—is problematic and requires improvement solely based on performance analysis results from human engineers. In other words, they rely on inefficient trial-and-error methods that compare performance by changing a tiny element, or on directly examining the results of trained deep learning networks to infer the cause and find more specific reasons. Therefore, they depend on the experience and judgment of human engineers. Summary of the Invention
[0010] Technical issues
[0011] The purpose of this invention is to solve all the above-mentioned problems.
[0012] Another objective of this invention is to improve the return on investment for data labeling.
[0013] Another objective of this invention is to improve the learning performance of deep learning networks that utilize new training data.
[0014] Another object of the present invention is to provide insightful information for performance analysis by learning the output values of deep learning networks.
[0015] Another objective of this invention is to minimize human intervention in training deep learning networks.
[0016] Technical solution
[0017] The features of this invention are used to achieve the objectives of this invention described above and the characteristic effects of this invention described later, and its structure is as follows.
[0018] According to an embodiment of the present invention, a method for training a deep learning network based on artificial intelligence is provided, comprising: (a) a learning device inputting unlabeled data into an active learning network, so that the active learning network extracts sub-unlabeled data, and inputting the sub-unlabeled data into an automatic labeling network, so that the automatic labeling network labels each sub-unlabeled data, thereby generating new labeled data, wherein the sub-unlabeled data is determined to be a difficult sample in the unlabeled data that is helpful for training an existing learning network (which is a state in which it has been trained using existing labeled data); (b) the learning device causing a lifelong learning network to sample the new labeled data and the existing labeled data, thereby generating a small dataset including the sampled new labeled data and the sampled existing labeled data. (c) The learning device enables the existing learning network to generate a trained learning network by incorporating all the sampled new labeled data into the training process when training the existing learning network using the mini-batch, and only incorporating the sampled existing labeled data into the training process when the performance of the existing learning network degrades; and (d) the learning device enables the interpretable analysis network to generate insightful results corresponding to validation data through the trained learning network and transmit the insightful results to at least one human engineer, so that the human engineer can transmit analysis results analyzing the performance of the trained learning network by referring to the insightful results, thereby modifying and improving at least one of the active learning network and the lifelong learning network by referring to the analysis results.
[0019] In step (a), the learning device enables the automatic annotation network to annotate each of the sub-unlabeled data to generate intermediate labeled data, enables it to generate a predicted loss by evaluating the uncertainty of the intermediate labeled data, and enables it to transmit first intermediate labeled data with a predicted loss greater than or equal to a loss threshold to at least one human annotator, so that the human annotator can verify the first intermediate labeled data and generate verified labeled data, and enables it to generate the new labeled data by combining the verified labeled data and second intermediate labeled data with a predicted loss less than the loss threshold.
[0020] The learning device inputs learning data into the automatic labeling network, which generates learning labeled data by labeling the learning data, and generates a learning predicted loss by evaluating the learning uncertainty corresponding to the learning data. The state of the learning device is to train the automatic labeling network by backpropagating the learning labeled loss and the loss prediction error, wherein the learning labeled loss refers to the learning labeled data and its corresponding ground truth, and the loss prediction error is generated due to the difference between the learning labeled loss and the learning predicted loss.
[0021] The learning device uses accuracy loss to enable the automatic annotation network to learn throughout its life, with the accuracy loss referring to the verified annotation data and the first intermediate annotation data.
[0022] In step (b), where the average loss calculated by averaging existing losses corresponding to existing output information of the existing learning network is obtained as the base loss and where the existing learning network corresponds to the existing labeled data, the learning device causes the lifelong learning network to perform the following operations: generate the mini-batch to a ratio of 1:1 between the sampled new labeled data and the sampled existing labeled data; train the existing learning network using all new losses, wherein the new losses reference the sampled new output information of the existing learning network corresponding to the sampled new labeled data; and train the existing learning network using a specific existing loss that is greater than or equal to the base loss, wherein the base loss references the sampled existing output information of the existing learning network corresponding to the sampled existing labeled data.
[0023] In step (c), the learning device enables the interpretable analysis network verification pair to perform a conformity assessment on the verification output information of the trained learning network corresponding to the verification data to generate a pixel comparison description, and obtains the pixel comparison description as the insightful result.
[0024] In step (a), the learning device causes the active learning network to compare the first annotation output information of the automatic annotation network corresponding to the unlabeled data with the second annotation output information of the existing learning network, thereby extracting the specific unlabeled data corresponding to the specific difference as the sub-unlabeled data, wherein the specific difference is the difference that is greater than or equal to the difference threshold in the difference between the first annotation output information and the second annotation output information.
[0025] In step (a), the learning device converts the unlabeled data into first-transformed unlabeled data to nth-transformed unlabeled data using a first transformation method to an nth transformation method, and inputs the first-transformed unlabeled data to nth-transformed unlabeled data into the active learning network, so that the active learning network evaluates the variance of the first-transformed output information to the nth-transformed output information of the existing learning network corresponding to each first-transformed unlabeled data to nth-transformed unlabeled data, and extracts the specific unlabeled data corresponding to a specific variance that is greater than or equal to a variance threshold as the sub-unlabeled data.
[0026] In step (a), the learning device enables the active learning network to measure the confidence level of the unlabeled data through the existing learning network, and enables it to extract specific unlabeled data that reach a preset number in increasing confidence order as the sub-unlabeled data.
[0027] According to another embodiment of the present invention, a learning device for training a deep learning network based on artificial intelligence is provided, comprising: a memory storing instructions for training a deep learning network based on artificial intelligence; and a processor performing operations according to the instructions stored in the memory, the operations being for training a deep learning network based on artificial intelligence, wherein the processor performs the following processes: (I) inputting unlabeled data into an active learning network to cause the active learning network to extract sub-unlabeled data, and inputting the sub-unlabeled data into an automatic labeling network to cause the automatic labeling network to label each of the sub-unlabeled data, thereby generating new labeled data, wherein the sub-unlabeled data is determined to be a difficult sample in the unlabeled data that is helpful in training an existing learning network (which is a state that has been trained using existing labeled data); (II) causing a lifelong learning network to process the new labeled data. The annotation data and the existing annotation data are sampled to generate a mini-batch that includes newly sampled annotation data and existing sampled annotation data. When training the existing learning network using the mini-batch, the newly sampled annotation data is fully reflected in the training, while the existing learning network is only reflected in the training when its performance deteriorates, thereby generating a trained learning network; and (III) the interpretable analysis network generates insightful results corresponding to the validation data through the trained learning network and transmits the insightful results to at least one human engineer, so that the human engineer can transmit analysis results analyzing the performance of the trained learning network by referring to the insightful results, thereby modifying and improving at least one of the active learning network and the lifelong learning network by referring to the analysis results.
[0028] In process (I), the processor causes the automatic annotation network to annotate each of the sub-unlabeled data to generate intermediate labeled data, causes it to generate a predicted loss by evaluating the uncertainty of the intermediate labeled data, and causes it to transmit first intermediate labeled data with a predicted loss greater than or equal to a loss threshold to at least one human annotator, so that the human annotator can verify the first intermediate labeled data and generate verified labeled data, and causes it to generate the new labeled data by combining the verified labeled data and second intermediate labeled data with a predicted loss less than the loss threshold.
[0029] The processor inputs the learning data into the automatic labeling network, causing the automatic labeling network to generate learning labeled data by labeling the learning data, and to generate a learning predicted loss by evaluating the learning uncertainty corresponding to the learning data. The state of the learning device is to train the state of the automatic labeling network by backpropagating the learning labeled loss and the loss prediction error, wherein the learning labeled loss refers to the learning labeled data and its corresponding ground truth, and the loss prediction error is generated by the difference between the learning labeled loss and the learning predicted loss.
[0030] The processor uses accuracy loss to enable the automatic annotation network to learn throughout its lifespan, with the accuracy loss referencing the verified annotation data and the first intermediate annotation data.
[0031] In process (II), where the average loss calculated by averaging existing losses corresponding to existing output information of the existing learning network is obtained as the base loss and wherein the existing learning network corresponds to the existing labeled data, the processor causes the lifelong learning network to perform the following operations: generate the mini-batch to a ratio of 1:1 between the sampled new labeled data and the sampled existing labeled data; train the existing learning network using all new losses, wherein the new losses reference the sampled new output information of the existing learning network corresponding to the sampled new labeled data; and train the existing learning network using a specific existing loss that is greater than or equal to the base loss, wherein the base loss references the sampled existing output information of the existing learning network corresponding to the sampled existing labeled data.
[0032] In process (III), the processor causes the interpretable analysis network verification pair to perform a conformity assessment on the verification output information of the trained learning network corresponding to the verification data to generate a pixel comparison description, and obtains the pixel comparison description as the insightful result.
[0033] In process (I), the processor causes the active learning network to compare the first annotation output information of the automatic annotation network corresponding to the unlabeled data with the second annotation output information of the existing learning network, thereby extracting the specific unlabeled data corresponding to the specific difference as the sub-unlabeled data, wherein the specific difference is the difference that is greater than or equal to the difference threshold in the difference between the first annotation output information and the second annotation output information.
[0034] In process (I), the processor converts the unlabeled data into first-transformed unlabeled data to nth-transformed unlabeled data using a first transformation method to an nth transformation method, and inputs the first-transformed unlabeled data to nth-transformed unlabeled data into the active learning network, so that the active learning network evaluates the variance of the first-transformed output information to the nth-transformed output information of the existing learning network corresponding to each first-transformed unlabeled data to nth-transformed unlabeled data, and extracts the specific unlabeled data corresponding to a specific variance that is greater than or equal to a variance threshold as the sub-unlabeled data.
[0035] In process (I), the processor enables the active learning network to measure the confidence level of the unlabeled data through the existing learning network, and enables it to extract specific unlabeled data that reach a preset number in increasing confidence order as the sub-unlabeled data.
[0036] Beneficial effects
[0037] This invention extracts difficult samples from the collected raw data that can improve the performance of deep learning networks and annotates them, thereby improving the return on investment in data annotation.
[0038] In addition, this invention can maximize the learning effect of deep learning networks by automatically adjusting the optimal response rate between new training data and existing training data.
[0039] Furthermore, by providing insightful information for performance analysis from the output values of the learned deep learning network, this invention can improve the accuracy of performance analysis performed by human engineers and minimize the time required for performance analysis.
[0040] In addition, this invention can save costs and time for deep learning network learning by minimizing human intervention in the learning process. Attached Figure Description
[0041] The following drawings, which are used to illustrate the embodiments of the present invention, are only a part of the embodiments of the present invention. Those skilled in the art (hereinafter referred to as "the skill person") can obtain other drawings based on the following drawings without making creative work.
[0042] Figure 1 This is a schematic diagram illustrating a learning device for training deep learning networks based on artificial intelligence according to an embodiment of the present invention.
[0043] Figure 2 This is a schematic diagram illustrating a method for training a deep learning network based on artificial intelligence according to an embodiment of the present invention.
[0044] Figure 3 This diagram illustrates the operation process of an actively running network in a method for training a deep learning network based on artificial intelligence according to an embodiment of the present invention.
[0045] Figure 4 This diagram illustrates other operational processes of the active learning network in a method for training a deep learning network based on artificial intelligence according to an embodiment of the present invention.
[0046] Figure 5 This is a schematic diagram illustrating the operation process of automatically labeling a network in a method for training a deep learning network based on artificial intelligence according to an embodiment of the present invention.
[0047] Figure 6 This diagram illustrates the training process of an auto-labeling network in a method for training a deep learning network based on artificial intelligence according to an embodiment of the present invention.
[0048] Figure 7 This diagram illustrates the operation of a continuous learning network in a method for training a deep learning network based on artificial intelligence according to an embodiment of the present invention.
[0049] Figure 8 This diagram illustrates the operation of an explainable analysis network in a method for training a deep learning network based on artificial intelligence according to an embodiment of the present invention.
[0050] Explanation of reference numerals in the attached figures
[0051] 1000: Learning equipment
[0052] 1001: Memory Detailed Implementation
[0053] The present invention will now be described in detail with reference to the accompanying drawings, which schematically illustrate specific embodiments in which the invention may be practiced to clarify the objectives, technical solutions, and advantages of the invention. These embodiments have been described in sufficient detail to enable those skilled in the art to practice the invention.
[0054] Furthermore, in the description and claims of this invention, the term "comprising" and its variations are not intended to exclude other technical features, additions, components, or steps. Other objects, advantages, and characteristics of this invention will partly become apparent to those skilled in the art through this specification, and partly through embodiments of the invention. The following examples and drawings are provided as exemplary embodiments and are not intended to limit the invention.
[0055] Furthermore, this invention covers all combinations of the embodiments described in this specification. It should be understood that the various embodiments of the invention are different but not mutually exclusive. For example, specific shapes, structures, and characteristics described in this specification may be implemented in other embodiments without departing from the technical concept and scope of the invention. Additionally, it should be understood that the position or configuration of the various components in each disclosed embodiment may be changed without departing from the technical concept and scope of the invention. Therefore, the following detailed description is not intended to be limiting; the scope of the invention is defined by all scopes equivalent to the scope of the claims or by the appended claims, provided that such description is adequate. Similar reference numerals in the drawings refer to the same or similar functions in several respects.
[0056] Preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings, so that those skilled in the art can readily implement the present invention.
[0057] Figure 1 This is a schematic diagram illustrating a learning device for training deep learning networks based on artificial intelligence according to an embodiment of the present invention. (Reference) Figure 1 The learning device 1000 may include a memory 1001 and a processor 1002. Instructions for training a deep learning network based on artificial intelligence are stored in the memory 1001, and the processor 1002 performs operations according to the instructions stored in the memory 1001. The operations are used to train a deep learning network based on artificial intelligence.
[0058] Specifically, the learning device 1000 can typically utilize a combination of computing devices and computer software (i.e., instructions that enable the computing devices to function in a specific manner) to achieve the required system performance. For example, the computing device may be a device that includes a computer processor, memory, storage, input devices and output devices, as well as other existing computing device components; electronic communication devices such as routers and switches; and electronic information storage systems such as network attached storage (NAS) and storage area networks (SAN).
[0059] In addition, the processor of a computing device may include hardware configurations such as an MPU (Micro Processing Unit) or CPU (Central Processing Unit), cache memory, and data bus. Furthermore, the computing device may include an operating system and software configurations for executing purpose-specific applications.
[0060] However, this does not preclude the computing device from including the integrated form of the medium, processor, and memory used to implement the present invention, i.e., an integrated processor.
[0061] The following is for reference. Figure 2 A method for training a deep learning network based on artificial intelligence using a learning device 1000 configured as described above according to an embodiment of the present invention is described.
[0062] First, after unlabeled data, which serves as the raw data for training, is collected, the learning device 1000 can input the unlabeled data into the active learning network 100 so that the active learning network 100 extracts sub-unlabeled data, which is determined to be difficult sample data in the unlabeled data that is helpful in training the existing learning network. The existing learning network may be in a state where it has already been trained using existing labeled data.
[0063] At this point, a hard example that is useful for training an existing learning network can be data that differs from the training data that was originally collected and used to train the deep learning network.
[0064] In addition, Active Learning Network 100 can extract sub-unlabeled data that are difficult samples from unlabeled data through various methods such as uncertainty sampling, query-by-Committee, and expected-mode-change.
[0065] For example, refer to Figure 3 The process of extracting sub-unlabeled data from unlabeled data by the active learning network 100 is described as follows.
[0066] The learning device 1000 can input unlabeled data into the active learning network 100, so that the active learning network 100 compares the first annotation output information of the automatic annotation network 200 corresponding to the unlabeled data with the second annotation output information of the existing learning network 400, thereby extracting specific unlabeled data corresponding to a specific difference as sub-unlabeled data, wherein the specific difference refers to the difference between the first annotation output information and the second annotation output information that is greater than or equal to a difference threshold.
[0067] That is, the learning device 1000 can input the collected unlabeled data into the active learning network 100.
[0068] Therefore, the active learning network 100 can input unlabeled data into the automatic labeling network 200 and the existing learning network 400 respectively.
[0069] Therefore, the automatic annotation network 200 can annotate each unlabeled data point, thereby outputting the first annotation information corresponding to each unlabeled data point. At this time, the first annotation information can be the ground truth (GT) corresponding to the unlabeled data, and the GT can include various information such as object region information, object category information, pixel category information, object location information, and point location information.
[0070] Furthermore, the existing learning network 400 can perform learning operations on each unlabeled data point, thereby outputting second annotation information corresponding to each unlabeled data point. In this case, the second annotation information can be the recognition result of the unlabeled data, and can include various information such as object region information, object category information, pixel category information, object location information, and point location information, according to the task to be performed by the existing learning network 400.
[0071] Then, the active learning network 100 confirms the difference between the first and second labeled information by comparing the first and second labeled information. When the difference between the first and second labeled information is less than the difference threshold, that is, when the first and second labeled information are confirmed to be the same or similar, the corresponding unlabeled data can be excluded from the learning data. When the difference between the first and second labeled information is greater than or equal to the difference threshold, that is, when the difference between the first and second labeled information is large, the corresponding unlabeled data can be judged as hard samples and extracted as sub-unlabeled data.
[0072] As another embodiment, refer to Figure 4 Another process for the Active Learning Network 100 to extract sub-unlabeled data from unlabeled data is described below.
[0073] The learning device 1000 can convert unlabeled data into first-transformed unlabeled data into nth-transformed unlabeled data through the first transformation method to the nth transformation method, and input the first-transformed unlabeled data into the nth-transformed unlabeled data into the active learning network 100, so that the active learning network 100 measures / evaluates the variance of the first-transformed output information to the nth-transformed output information of the existing learning network 400 corresponding to each first-transformed unlabeled data into the nth-transformed unlabeled data, and extracts the specific unlabeled data corresponding to a specific variance that is greater than or equal to the variance threshold as sub-unlabeled data.
[0074] That is, the learning device 1000 can transform the unlabeled data through the first transformation method to the nth transformation method, thereby generating unlabeled data that has been transformed by the first transformation to unlabeled data that has been transformed by the nth transformation.
[0075] At this point, the conversion method can be implemented by adjusting the size of the unlabeled data, changing the aspect ratio of the unlabeled data, changing the color tone, and so on.
[0076] In addition, the learning device 1000 can input the first-transformed unlabeled data to the nth-transformed unlabeled data into the existing learning network 400.
[0077] Therefore, the existing learning network 400 can perform learning operations on the unlabeled data from the first transformation to the nth transformation and output the first transformation output information to the nth transformation output information corresponding to the unlabeled data from the first transformation to the nth transformation.
[0078] At this point, the output information from the first transformation to the nth transformation can be the recognition result of the unlabeled data from the first transformation to the nth transformation. It can include various information such as object region information, object category information, pixel category information, object location information, and point location information, based on the existing learning network 400's task to be performed.
[0079] Then, the active learning network 100 calculates the variance of the first transformed unlabeled data to the nth transformed unlabeled data and determines whether the variance corresponding to each unlabeled data is greater than or equal to a variance threshold. Then, the specific unlabeled data corresponding to the variance greater than or equal to the variance threshold is extracted as sub-unlabeled data, and the unlabeled data corresponding to the variance less than the variance threshold is excluded from the learning data.
[0080] In addition, except according to Figure 3 and Figure 4 In addition to the above methods, the learning device 1000 can extract sub-unlabeled data from unlabeled data through confidence sampling. For example, the learning device 1000 can enable the active learning network 100 to measure the confidence of unlabeled data through the existing learning network 400, and enable it to extract specific unlabeled data as sub-unlabeled data by reaching a preset number in ascending order of confidence.
[0081] In addition, the learning device 1000 inputs the unlabeled data into the automatic labeling network 200, thereby enabling the automatic labeling network 200 to label each unlabeled data, that is, to perform labeling to generate new labeled data.
[0082] That is, reference Figure 5 When unlabeled sub-data is input through the learning device 1000, the automatic annotation network 200 performs annotation on each unlabeled sub-data (i.e., performs annotation to generate GT), thereby generating intermediate labeled data.
[0083] In addition, the automatic labeling network 200 can generate a predicted loss by evaluating the uncertainty of intermediate labeled data.
[0084] That is, the automatic annotation network 200 can generate annotation information corresponding to the intermediate annotation data, and use the uncertainty corresponding to the annotation information to generate a predicted loss.
[0085] Then, the automatic annotation network 200 can compare each predicted loss with a loss threshold and transmit first intermediate annotation data with a predicted loss greater than or equal to the loss threshold to at least one human annotator, so that the human annotator can verify the first intermediate annotation data and generate verified annotation data. Additionally, the automatic annotation network 200 can generate new annotation data from verified annotation data whose annotation information has been modified after verification by a human annotator, and from second intermediate annotation data with a predicted loss less than the loss threshold. On the other hand, when the first intermediate annotation data is transmitted to a human annotator using the predicted loss, the learning device 1000 can perform the operation of generating new annotation data from the verified annotation data and the second intermediate annotation data.
[0086] At this point, the automatic labeling network 200 can be in a state where it has been trained to automatically label unlabeled data.
[0087] For example, refer to Figure 6 When acquiring training data including ground truth (GT), the learning device 2000 can input the training data into the automatic annotation network 200, causing the automatic annotation network 200 to generate labeled training data by annotating the training data, and to generate a predicted training loss by evaluating the training uncertainty corresponding to the training data. In this case, the learning device 2000 can be the same as the learning device 1000 that trains deep learning networks based on artificial intelligence, or it can be a separate learning device used only for training the automatic annotation network 200.
[0088] Furthermore, the learning device 2000 calculates the learning annotation loss by referring to the annotation information and corresponding ground truth (GT) in the learning annotation data generated by the automatic annotation network 200, and calculates the loss prediction error by referring to the learning prediction loss and the learning annotation loss. Then, the learning device 2000 can train the automatic annotation network 200 through backpropagation, which utilizes the learning annotation loss and the loss prediction error.
[0089] On the other hand, the learning device 1000 can utilize accuracy loss to enable the automatic annotation network 200 to learn throughout its lifecycle, the accuracy loss being referenced to validated annotation data and first intermediate annotation data. This allows for continuous improvement of the performance of the automatic annotation network 200.
[0090] Then, the learning device 1000 enables the lifelong learning network 300 to sample new labeled data and existing labeled data to generate a minibatch that includes the sampled new labeled data and the sampled existing labeled data. When training the existing learning network 400 using the minibatch, the existing learning network 400 is trained by reflecting all the sampled new labeled data into the training, and only reflecting the sampled existing labeled data into the training when the performance of the existing learning network 400 degrades, thereby generating a trained learning network (400a).
[0091] That is, the learning device 1000 can train the existing learning network 400 using minibatch to generate a trained learning network 400a. However, the learning device 1000 enables the trained learning network 400a to have almost the same performance as the existing learning network 400 for existing labeled data, while having better performance than the existing learning network 400 for new labeled data. In this case, the existing learning network 400 can be a new learning network used to generate new deep learning models, rather than an existing trained learning network.
[0092] For example, refer to Figure 7 The lifelong learning network 300 can input existing labeled data stored in the existing training database into the existing learning network 400, so that the existing learning network 400 can learn from the existing labeled data and output existing output information, and obtain the average loss of the existing output information as the base loss.
[0093] Then, in each iteration used to train the existing learning network 400, the lifelong learning network 300 can sample the new labeled data stored in the new training database and the existing labeled data stored in the existing training database, thereby generating a mini-batch that includes the sampled new labeled data and the sampled existing labeled data. At this time, when the lifelong learning network 300 samples the sampled new labeled data and the sampled existing labeled data included in the mini-batch, it can make their ratio reach 1:1.
[0094] Furthermore, the lifelong learning network 300 can input newly sampled labeled data and existing sampled labeled data into the existing learning network 400, enabling the existing learning network 400 to learn from the newly sampled labeled data and existing sampled labeled data, thereby outputting newly sampled output information corresponding to the newly sampled labeled data and existing sampled output information corresponding to the existing sampled labeled data. In this case, the existing learning network 400 can be a new learning network used to generate new deep learning models, rather than an existing trained learning network.
[0095] That is, the lifelong learning network 300 can generate a new loss by referring to the sampled new output information and its corresponding ground truth (GT), and generate an existing loss by referring to the sampled existing output information and its corresponding GT.
[0096] Then, the lifelong learning network 300 trains the existing learning network 400 through backpropagation (which utilizes the new loss and the existing loss). However, it can perform backpropagation on the new loss, while only performing backpropagation on specific existing losses greater than or equal to the base loss after comparing them with the base loss, and not performing backpropagation on other existing losses less than the base loss. That is, by reflecting all the new labeled data into the training, and only reflecting the existing labeled data into the training when performance degrades, the trained learning network 400a can have almost the same performance as the existing learning network 400 for existing labeled data, and better performance than the existing learning network 400 for new labeled data.
[0097] Then, the learning device 1000 enables the interpretable analysis network 500 to generate insightful results corresponding to the validation data through the trained learning network 400a, and transmits the insightful results to at least one human engineer so that the human engineer can transmit analysis results analyzing the performance of the trained learning network 400a by referring to the insightful results, thereby modifying and improving at least one of the active learning network 100 and the lifelong learning network 300.
[0098] That is, the learning device 1000 can input validation data into the trained learning network 400a, causing the trained learning network 400a to output validation output information that performs learning operations on the validation data. Furthermore, it can input the validation output information corresponding to the validation data into the interpretable analysis network 500, causing the interpretable analysis network 500 to output insightful results for analyzing the performance of the trained learning network 400a based on the validation output information. In contrast, the learning device 1000 inputs validation output information into the interpretable analysis network 500, but instead of the trained learning network 400a, the interpretable analysis network 500 can input validation data into the trained learning network 400a, causing the trained learning network 400a to output validation output information that performs learning operations on the validation data, thereby acquiring the validation output information.
[0099] At this point, the insightful results generated by the interpretable analysis network 500 can represent samples where the performance of the trained learning network 400a appears to be low, such as samples that are incorrectly or not detected, the reasons for the incorrect or undetected samples, such as regions in the input data that significantly affect the output, or features in the intermediate layers that significantly affect the output.
[0100] For example, refer to Figure 8As shown in (A), the validation data is input into the trained learning network 400a to classify the validation data. Then, as shown in (B), the validation output information of the trained learning network 400a is input into the interpretable analysis network 500 to perform conformity evaluation on the validation output information of the trained learning network 400a, thereby outputting a pixel-wise description. That is, when the validation data is a validation image, for each image pixel, the change in the "cat" score (i.e., the feature conformity change) when the i-th feature is increased by 1 can be generated as a pixel-wise description of the change in the "cat" score when the (x, y)-th input pixel in the image is increased by 1, thereby outputting an insightful result. On the other hand, in addition to the above conformity evaluation, the interpretable analysis network 500 can generate insightful results corresponding to the output values of the trained learning network 400a through various methods.
[0101] The learning device 1000 then transmits the insightful results generated by the interpretable analysis network 500 to the human engineer, enabling the human engineer to easily analyze the performance of the trained learning network 400a with reference to the insightful results.
[0102] In addition, the learning device 1000 can modify and improve the active learning network 100 and the learning model of the lifelong learning network 300 by referring to the performance analysis results generated by human engineers.
[0103] Furthermore, the embodiments of the present invention described above can be implemented and recorded in the form of program commands executed by various computer components on a computer-readable recording medium. The computer-readable recording medium may include program commands, data files, data structures, etc., individually or in combination. The program commands recorded on the computer-readable recording medium may be specifically designed or configured for the present invention, or may be program commands known and used by those skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical recording media such as CD-ROMs and DVDs; magneto-optical media such as floppy disks; and hardware devices specifically configured for storing and executing program commands, such as ROMs, RAMs, and flash memory. Examples of program commands include not only machine language code generated by a compiler, but also executable high-level language code such as that executed by a computer using an interpreter. The hardware device may be configured to operate as one or more software modules to perform the processing according to the present invention, and vice versa.
[0104] The present invention has been described above with reference to specific components and other specific matters, limited embodiments, and accompanying drawings. However, these descriptions are intended to help to provide a more comprehensive understanding of the present invention. The present invention is not limited to the described embodiments, and those skilled in the art can attempt various modifications and variations based on these descriptions.
[0105] Therefore, the technical concept of the present invention should not be limited to the embodiments described above, and all contents of the claims of the present invention and their equivalent or equivalent transformations are within the technical concept of the present invention.
Claims
1. A method for training a deep learning network for image analysis based on artificial intelligence, wherein, The method includes: (a) The learning device inputs unlabeled image data into an active learning network to enable the active learning network to extract sub-unlabeled image data, and inputs the sub-unlabeled image data into an automatic labeling network to enable the automatic labeling network to label each sub-unlabeled image data to generate new labeled image data, wherein the sub-unlabeled image data is determined to be a difficult sample in the unlabeled image data that is helpful for training the existing learning network, and the existing learning network is in a state that has been trained using existing labeled image data; (b) The learning device enables a lifelong learning network (i) to sample the newly labeled image data and the existing labeled image data to generate a mini-batch comprising the sampled newly labeled image data and the sampled existing labeled image data, (ii) and to train the existing learning network using the mini-batch, wherein all of the sampled newly labeled image data is included in the training, and the sampled existing labeled image data is included in the training only when the performance of the existing learning network degrades to train the existing learning network to generate a trained learning network; and (c) The learning device (i) enables the interpretable analysis network (i-1) to generate insightful results (i-2) about the verification image data through the trained learning network and transmits the insightful results to at least one human engineer, so that the human engineer transmits analysis results obtained by analyzing the performance of the trained learning network with reference to the insightful results, (ii) and modifies and improves at least one of the active learning network and the lifelong learning network with reference to the analysis results.
2. The method according to claim 1, wherein: In (a), the learning device causes the automatic annotation network (i) to annotate each of the sub-unannotated image data to generate intermediate annotated image data, (ii) to generate a predicted loss by evaluating the uncertainty of the intermediate annotated image data, and (iii) to transmit a first intermediate annotated image data with a predicted loss greater than or equal to a loss threshold to at least one human annotator so that the human annotator can verify the first intermediate annotated image data to generate verified annotated image data, and to generate the new annotated image data by combining the verified annotated image data and the second intermediate annotated image data with a predicted loss less than the loss threshold.
3. The method according to claim 2, wherein: The learning device (i) inputs learning image data into the automatic annotation network, such that the automatic annotation network (i-1) generates learning labeled image data by annotating the learning image data, (i-2) generates learning predicted loss by evaluating the learning uncertainty of the learning image data, and (ii) backpropagates (1) the learning labeled loss of the learning labeled image data and its corresponding Ground Truth, and (2) trains the state of the automatic annotation network based on the loss prediction error of the difference between the learning labeled loss and the learning predicted loss.
4. The method according to claim 2, wherein: The learning device enables the automatic annotation network to learn throughout its life by referencing the accuracy loss of the verified labeled image data and the first intermediate labeled image data.
5. The method according to claim 1, wherein: In (b), given that the average loss is obtained by averaging the existing loss as the base loss on the existing output information of the existing learning network with respect to the existing labeled image data, The learning device enables the lifelong learning network (i) to generate the mini-batch with the ratio of the sampled new labeled image data to the sampled existing labeled image data being 1:1, (ii) (ii-1) training the existing learning network using all new losses referencing the sampled new output information of the existing learning network with respect to the sampled new labeled image data, and (ii-2) training the existing learning network using a specific existing loss that is greater than or equal to the base loss among the existing losses referencing the sampled existing output information of the existing learning network with respect to the sampled existing labeled image data.
6. The method according to claim 1, wherein: In (c), the learning device causes the interpretable analysis network to perform a conformity assessment on the verification output information of the trained learning network regarding the verification image data to generate a pixel contrast description, and obtains the pixel contrast description as the insightful result.
7. The method according to claim 1, wherein: In (a), the learning device causes the active learning network (i) to compare the first annotation output information of the automatic annotation network and the second annotation output information of the existing learning network with respect to the unlabeled image data (ii), and extracts specific unlabeled image data corresponding to a specific difference as the sub-unlabeled image data, wherein the specific difference is a difference greater than or equal to a difference threshold in the difference between the first annotation output information and the second annotation output information.
8. The method according to claim 1, wherein: In (a), the learning device (i) converts the unlabeled image data into first converted unlabeled image data to nth converted unlabeled image data using a first conversion method to an nth conversion method, (ii) and inputs the first converted unlabeled image data to nth converted unlabeled image data into the active learning network, so that the active learning network (ii-1) evaluates the variance of the first converted output information to the nth converted output information of the existing learning network for each of the first converted unlabeled image data to the nth converted unlabeled image data, (ii-2) and extracts specific unlabeled image data corresponding to a specific variance greater than or equal to a variance threshold as the sub-unlabeled image data.
9. The method according to claim 1, wherein: In (a), the learning device enables the active learning network to measure the confidence level of the unlabeled image data through the existing learning network, and extracts a preset number of specific unlabeled image data as the sub-unlabeled image data in ascending order of confidence level.
10. A learning device based on artificial intelligence to train a deep learning network for image analysis, wherein, include: A memory that stores instructions for training deep learning networks based on artificial intelligence; as well as A processor, configured to execute operations for training a deep learning network based on artificial intelligence, according to instructions stored in the memory, wherein: The processor performs the following processes: (I) inputting unlabeled image data into an active learning network to extract sub-unlabeled image data, and inputting the sub-unlabeled image data into an automatic labeling network to label each sub-unlabeled image data to generate new labeled image data, wherein the sub-unlabeled image data is determined to be a difficult sample in the unlabeled image data that is helpful for training the existing learning network, and the existing learning network is in a state that has been trained using existing labeled image data; (II) causing the lifelong learning network (i) to sample the new labeled image data and the existing labeled image data to generate a mini-batch including the sampled new labeled image data and the sampled existing labeled image data; (ii) The existing learning network is trained using the mini-batch method, wherein all the sampled newly labeled image data is included in the training, and the sampled existing labeled image data is included in the training only when the performance of the existing learning network degrades, thereby generating a trained learning network; and (III) (i) (i-1) the interpretable analysis network generates insightful results about the verification image data through the trained learning network (i-2) and transmits the insightful results to at least one human engineer, so that the human engineer transmits analysis results obtained by analyzing the performance of the trained learning network with reference to the insightful results (ii), and modifies and improves at least one of the active learning network and the lifelong learning network with reference to the analysis results.
11. The learning device according to claim 10, wherein: In step (I), the processor causes the automatic annotation network (i) to annotate each of the sub-unannotated image data to generate intermediate annotated image data, (ii) to generate a predicted loss by evaluating the uncertainty of the intermediate annotated image data, and (iii) to transmit a first intermediate annotated image data with a predicted loss greater than or equal to a loss threshold to at least one human annotator so that the human annotator can verify the first intermediate annotated image data to generate verified annotated image data, and to generate the new annotated image data by combining the verified annotated image data and the second intermediate annotated image data with a predicted loss less than the loss threshold.
12. The learning device according to claim 11, wherein: The processor (i) inputs the learning image data into the automatic annotation network, such that the automatic annotation network (i-1) generates learning labeled image data by annotating the learning image data, (i-2) generates learning predicted loss by evaluating the learning uncertainty of the learning image data, and (ii) backpropagates (1) the learning labeled loss with reference to the learning labeled image data and its corresponding Ground Truth, and (2) trains the state of the automatic annotation network based on the loss prediction error of the difference between the learning labeled loss and the learning predicted loss.
13. The learning device according to claim 11, wherein: The processor enables the automatic annotation network to learn throughout its lifecycle by referencing the accuracy loss of the verified labeled image data and the first intermediate labeled image data.
14. The learning device according to claim 10, wherein: In step (II), with the average loss obtained by averaging the existing loss of the existing learning network with respect to the existing output information of the existing labeled image data as the base loss, the processor causes the lifelong learning network (i) to generate the mini-batch with the ratio of the sampled new labeled image data to the sampled existing labeled image data being 1:1, (ii) (ii-1) training the existing learning network using all new losses with reference to the sampled new output information of the existing learning network with respect to the sampled new labeled image data, and (ii-2) training the existing learning network using a specific existing loss that is greater than or equal to the base loss among the existing losses with reference to the sampled existing output information of the existing learning network with respect to the sampled existing labeled image data.
15. The learning device according to claim 10, wherein: In (III), the processor causes the interpretable analysis network to perform a conformity assessment on the verification output information of the trained learning network regarding the verification image data to generate a pixel contrast description, and obtains the pixel contrast description as the insightful result.
16. The learning device according to claim 10, wherein: In (I), the processor causes the active learning network (i) to compare the first annotation output information of the automatic annotation network and the second annotation output information of the existing learning network with respect to the unlabeled image data (ii), and extracts specific unlabeled image data corresponding to a specific difference as the sub-unlabeled image data, wherein the specific difference is a difference greater than or equal to a difference threshold in the difference between the first annotation output information and the second annotation output information.
17. The learning device according to claim 10, wherein: In (I), the processor (i) converts the unlabeled image data into first converted unlabeled image data to nth converted unlabeled image data using a first conversion method to an nth conversion method, (ii) and inputs the first converted unlabeled image data to nth converted unlabeled image data into the active learning network, so that the active learning network (ii-1) evaluates the variance of the first converted output information to the nth converted output information of the existing learning network for each of the first converted unlabeled image data to the nth converted unlabeled image data, (ii-2) and extracts specific unlabeled image data corresponding to a specific variance that is greater than or equal to a variance threshold as the sub-unlabeled image data.
18. The learning device according to claim 10, wherein: In (I), the processor enables the active learning network to measure the confidence level of the unlabeled image data through the existing learning network, and extracts a preset number of specific unlabeled image data as the sub-unlabeled image data in ascending order of confidence level.