Method for training a custom model based on a pre-trained base model and a learning device using the same.

Abstract

This provides a method for training a custom model based on a pre-trained base model. [Solution] The method includes the steps of: a learning device inputting training data into a custom model that performs a specific task to generate first to n intermediate features, and generating first to k transformation features that transform the source domain corresponding to the base model in the first to k specific intermediate features into a target domain corresponding to a specific task; a learning device generating a task output by performing a learning operation on the kth fusion feature output from the kth residual unit through the task head block; and a learning device generating a task loss by referring to the task output and the original correct answer corresponding to the training data, and backpropagating the task loss to train at least a portion of the task head block, the first to k residual units, and the first to k adaptation blocks.

Description

Technical Field

[0001] The present invention relates to a method for training a custom model based on a pre-trained base model and a training device using the same.

Background Art

[0002] Base models such as Grounding DINO trained using various large amounts of training datasets such as MSCOCO, Object365, and OpenImages have extensive and universal knowledge. However, such base models have a problem that they do not have or lack the detailed knowledge necessary for performing specific tasks such as detecting defects in circuit boards. Therefore, research has been conducted to learn the knowledge necessary for performing specific tasks using a relatively small amount of training data based on the universal knowledge possessed by the base model, and a conventional training model related thereto is shown in FIG. 1.

[0003] As an example, FIG. 1(a) is a drawing showing a training model trained by the Full Fine-Tuning method. For example, when training data is input into a pre-trained base model 100 including 5 layers, the training data is sequentially subjected to learning operations through the first layer 110 to the fifth layer 150, and then task data is output. Learning can be performed by sequentially updating the respective weight value parameters of the fifth layer 150 to the first layer 110 through backpropagation that performs gradient operations in the reverse direction like the dotted arrow for the loss generated by referring to the task data and the GT (Ground Truth) corresponding thereto. Therefore, the Full Fine-tuning method has the advantage of being able to learn all the weight value parameters of the base model 100 and having high performance. However, due to the characteristics of the base model 100 generally having a very large number of weight value parameters, there is a problem that astronomical costs are required for learning for a specific task.

[0004] As another example, Figures 1(b) and 1(c) show a trained model in which some layers of the base model 100 are frozen to solve the cost problem required in the Full Fine-Tuning method. The cost required for training is reduced by not training on the frozen weight parameters. For example, in (b), the weight parameters of the first layer 110 to the fourth layer 140 of the base model 100 are frozen, and in (c), the weight parameters of the first layer 110, the third layer 130, and the fourth layer 140 of the base model 100 are frozen. In this case, the process up to generating the loss is performed in the same way as in (a), but in (b), only the weighting parameter of the fifth layer 150 is updated through backpropagation, and in (c), the weighting parameters of the fifth layer 150 and the second layer 120 are updated through backpropagation. As a result, only the information for some layers needs to be stored in memory, so the cost can be reduced compared to the Full Fine-Tuning method.

[0005] However, the base model 100 performs training in which the first half of the layers (for example, the first layer 110 extracts primitive features from the training data, and the second half of the layers (for example, the fifth layer 150) combines primitive features to perform advanced inference) is not trained as in (b). However, if training is not performed on the first half of the layers, as in (b), it is not possible to generate primitive features for a specific task, and inference must be performed using only primitive features for a universal task, resulting in a significant decrease in performance. Furthermore, if training is performed on the first half of the layers as in (c) to minimize the decrease in performance, it is necessary not only to perform gradient operations up to the first half of the layers through backpropagation, but also to store information up to the first half of the layers in memory, resulting in a significant loss of cost benefits.

[0006] As another example, Figure 1(d) shows a learning model in which all layers of the base model 100 are frozen and a separate adapter layer is added. In (d), two adapter layers, the first adapter layer 210 and the second adapter layer 220, are exemplarily connected to the outputs of the second layer 120 and the fourth layer 140 of the base model 100, respectively. The outputs of the first adapter layer 210 and the second adapter layer 220 are fused through a predetermined fusion layer 300, and then task data is output through an output layer 400. Subsequently, the loss generated by referencing the task data and its corresponding GT is backpropagated, and gradient operations are performed on the output layer 400, the first adapter layer 210, and the second adapter layer 220 to update the weighting parameters. Since the first layer 110 to the fifth layer 150 of the base model 100 are all frozen and their weighting parameters are fixed, the base model 100 is not trained. This has the advantage of effectively reducing the cost required for training by training only the output layer 400, the first adapter layer 210, and the second adapter layer 220, which have relatively fewer weighting parameters than the base model 100. However, there is a problem in that it is difficult to improve performance for advanced inference for specific tasks by simply combining adapter layers.

[0007] Therefore, there is a need for improvement measures to resolve the aforementioned problems. Furthermore, Patent Document 1 discloses a method for evaluating a text generation model using a pre-trained model, and Patent Document 2 discloses a method for fine-tuning the parameters of a pre-trained language model and a pre-trained vocabulary list, as well as an electronic device using the same. In addition, Patent Document 3 discloses a method and system for tuning a deep learning model, and Patent Document 4 discloses a pre-trained evaluation model for evaluating the quality of natural language generation (NLG) output. [Prior art documents] [Patent Documents]

[0008] [Patent Document 1] Korean Registered Patent Publication No. 10-2693112 [Patent Document 2] Korean Registered Patent Publication No. 10-2618219 [Patent Document 3] Patent No. 7371188 [Patent Document 4] U.S. Patent Application Publication No. 2024 / 0012999 [Overview of the Initiative] [Problems that the invention aims to solve]

[0009] The purpose of this invention is to solve all of the problems mentioned above.

[0010] Furthermore, the present invention also aims to configure a custom model comprising a pre-trained base model including at least one feature extraction block, at least one adaptation block, at least one residual unit, and a task head block, and to update at least some of the parameters of the task head block, at least one residual unit, and at least one adaptation block through backpropagation using task loss, while not updating the parameters of the base model.

[0011] Another objective of the present invention is to enable learning that allows for advanced inference by fusing at least one residual feature generated through at least one residual unit with at least one transformed feature obtained by transforming at least one intermediate feature generated from a base model through at least one adaptation block. [Means for solving the problem]

[0012] According to one embodiment of the present invention, in a method for learning a custom model based on a pre-trained base model, (a) a learning device inputs learning data to a custom model that performs a specific task, comprising a base model including a pre-trained first feature extraction block to the nth feature extraction block (where n is an integer of 2 or more), a first adaptation block to the kth adaptation block (where k is an integer of 2 or more and less than or equal to n), a first residual unit to the kth residual unit, and a task head block, to generate first intermediate features to the nth intermediate features through each of the first feature extraction blocks to the nth feature extraction blocks of the base model, and generates first transformation features to the kth transformation features, each of which converts the source domain corresponding to the base model in each of the first specific intermediate features to the kth specific intermediate features selected from the first to the nth intermediate features, to the target domain corresponding to the specific task, through each of the first adaptation blocks to the kth adaptation blocks; (b) After the learning device generates a first fusion feature by fusing the first fusion feature extracted for the specific task with the first fusion feature through the first fusion unit, and then generates an i-th provisional feature by fusing the (i-1) fusion feature with the i-th fusion feature (where i is an integer greater than or equal to 2 and less than or equal to k), it generates an i-th fusion feature by fusing the i-th fusion feature extracted for the specific task with the i-th provisional feature through the i-th fusion unit, and generates a task output by performing a learning operation on the k-th fusion feature output from the k-th fusion unit through the task head block;A method is provided which includes (c) the learning device generates a task loss by referring to the task output and the original correct answer corresponding to the learning data, and backpropagates the task loss to train at least a portion of the task head block, the first to k-th register dual units, and the first to k-th adaptation blocks.

[0013] In one example, in step (a), the learning device generates a conversion feature by selecting the nth intermediate feature or the (n-1)th intermediate feature from the first intermediate feature to the nth intermediate feature through any one of the first to the kth adaptation blocks, and converting the source domain to the target domain; and by selecting an intermediate feature which is one of the second intermediate feature to the (n-2)th intermediate feature through another adaptation block from the first to the kth adaptation blocks, and converting the source domain to the target domain.

[0014] In one example, in step (a), each of the first to k-residual units includes a plurality of convolution layers, and is configured such that a first filter included in at least some of the plurality of convolution layers is decomposed into a plurality of second filters having a lower rank than the first filter.

[0015] In one example, in step (a), the parameters of each of the first to n feature extraction blocks are fixed through freezing, and in step (c), when the learning device backpropagates the task loss, it is characterized in that the parameters of each of the first to n feature extraction blocks are not updated.

[0016] In one example, in step (b), the learning device generates the first fusion feature by performing an add operation on the first resistive feature and the first transformation feature through the first element-wise add layer and fusing them, generates the i provisional feature by performing an add operation on the i transformation feature and the (i-1) fusion feature through the (i-1) element-wise add layer and fusing them, and generates the i fusion feature by performing an add operation on the i resistive feature and the i provisional feature through the element-wise add layer and fusing them.

[0017] Furthermore, according to another embodiment of the present invention, in a method for learning a custom model based on a pre-trained base model, (a) a learning device inputs learning data to a custom model that performs a specific task, comprising a base model including a pre-trained first to nth feature extraction block (wherein n is an integer of 1 or more), an adaptation block, a residual unit, and a task head block, to generate first to nth intermediate features through each of the first to nth feature extraction blocks of the base model, and generates a transformation feature through the adaptation block that transforms the source domain corresponding to the base model in a specific intermediate feature selected from the first to nth intermediate features to a target domain corresponding to the specific task; (b) the learning device generates a fusion feature by fusing the residual feature extracted for the specific task in the transformation feature through the residual unit and the transformation feature, and generates a task output by performing a learning operation on the fusion feature through the task head block; and (c) A method is provided which includes the steps of: (c) the learning device generating a task loss by referring to the task output and the original correct answer corresponding to the learning data; and backpropagating the task loss to train at least a portion of the task head block, the residual unit and the adaptation block.

[0018] In one example, in step (a), the learning device generates the conversion feature by selecting one of the second intermediate feature to the (n-1)th intermediate feature as the specific intermediate feature through the adaptation block and converting the source domain to the target domain.

[0019] In one example, in step (a) above, the resistive unit is configured to include a plurality of convolution layers, wherein a first filter included in at least some of the plurality of convolution layers is decomposed into a plurality of second filters having a lower rank than the first filter.

[0020] In one example, in step (a), the parameters of each of the first to n feature extraction blocks are fixed through freezing, and in step (c), when the learning device backpropagates the task loss, it is characterized in that the parameters of each of the first to n feature extraction blocks are not updated.

[0021] In one example, the learning device generates the fusion feature by performing an add operation on the residual feature and the transformation feature through an element-wise add layer and fusing them together.

[0022] Furthermore, according to yet another embodiment of the present invention, a learning device for learning a custom model based on a pre-trained base model includes: at least one memory for storing instructions; and at least one processor configured to execute the instructions, wherein the processor comprises: (i) a base model including a pre-trained first feature extraction block to the nth feature extraction block (where n is an integer of 2 or more); a first adaptation block to the kth adaptation block (where k is an integer of 2 or more and less than or equal to n); a first residual unit to the kth residual unit; and a task head block, which inputs learning data to a custom model that performs a specific task, generates first intermediate features to the nth intermediate features through each of the first feature extraction blocks to the nth feature extraction blocks of the base model, and corresponds to the base model in each of the first specific intermediate features to the kth specific intermediate features selected from the first to the nth intermediate features through each of the first adaptation blocks to the kth adaptation blocks. (II) A process to generate first to kth transformation features, each of which transforms a source domain into a target domain corresponding to the specific task; (II) After generating a first fusion feature by fusing the first transformation feature with the first restoration feature extracted for the specific task through the first restoration unit, when the i-th fusion feature and the ith transformation feature (where i is an integer greater than or equal to 2 and less than or equal to k) are fused to generate the ith provisional feature, the process to generate the ith fusion feature by fusing the ith restoration feature extracted for the specific task with the ith provisional feature through the ith restoration unit, and to generate a task output by performing a learning operation on the kth fusion feature output from the kth restoration unit through the task head block;and (III) generating a task loss by referring to the task output and the original correct answer corresponding to the learning data, and backpropagating the task loss to learn at least a part of the task head block, the first to the k-th residual units, and the first to the k-th adaptation blocks; A learning device that performs the process is provided.;

[0023] In one example, in the process (I), the processor selects the n-th intermediate feature or the (n-1)-th intermediate feature among the first to the n-th intermediate features through any one of the first to the k-th adaptation blocks, and converts the source domain into the target domain to generate any one conversion feature. And through another one of the first to the k-th adaptation blocks, one of the second to the (n-2)-th intermediate features is selected, and the source domain is converted into the target domain to generate another conversion feature.

[0024] In one example, in the process (I), each of the first to the k-th residual units includes a plurality of convolutional layers, and the first filter included in at least a part of the plurality of convolutional layers is decomposed into a plurality of second filters having a rank lower than the first filter.

[0025] In one example, in process (I), each of the first to n feature extraction blocks has its parameters fixed through freezing, and in process (III), when the processor backpropagates the task loss, it does not update the parameters of each of the first to n feature extraction blocks.

[0026] In one example, the (II) process is characterized in that the processor generates the first fusion feature by performing an add operation on the first resistive feature and the first transformation feature through a first element-wise add layer and fusing them, generates the i provisional feature by performing an add operation on the i transformation feature and the (i-1) fusion feature through a (i-1) element-wise add layer and fusing them, and generates the i fusion feature by performing an add operation on the i resistive feature and the i provisional feature through an element-wise add layer and fusing them.

[0027] Furthermore, according to yet another embodiment of the present invention, a learning device for learning a custom model based on a pre-trained base model includes: at least one memory for storing instructions; and at least one processor configured to execute the instructions, wherein the processor comprises (i) a base model including a pre-trained first to nth feature extraction block (where n is an integer of 1 or more), an adaptation block, a residual unit, and a task head block, inputting learning data to a custom model that performs a specific task, generating first to nth intermediate features through each of the first to nth feature extraction blocks of the base model, and a source domain corresponding to the base model in a specific intermediate feature selected from the first to nth intermediate features through the adaptation block. A learning device is provided that performs the following steps: (II) a process of generating a transformation feature that transforms the data into a target domain corresponding to the specific task; (II) a process of generating a fusion feature by fusing the transformation feature with the residual feature extracted for the specific task using the transformation feature through the residual unit, and generating a task output by performing a learning operation on the fusion feature through the task head block; and (III) a process of generating a task loss by referencing the task output and the original correct answer corresponding to the training data, and backpropagating the task loss to train at least a portion of the task head block, the residual unit, and the adaptation block.

[0028] In one example, the (I) process is characterized in that the processor generates the conversion feature by selecting one of the second to (n-1) intermediate features as the specific intermediate feature through the adaptation block and converting the source domain to the target domain.

[0029] In one example, the resistive unit in process (I) is configured to include a plurality of convolution layers, wherein a first filter included in at least some of the plurality of convolution layers is decomposed into a plurality of second filters having a lower rank than the first filter.

[0030] In one example, in process (I), each of the first to n feature extraction blocks has its parameters fixed through freezing, and in process (III), when the processor backpropagates the task loss, it does not update the parameters of each of the first to n feature extraction blocks.

[0031] In one example, the (II) process is characterized in that the processor generates the fusion feature by performing an add operation on the residual feature and the transformation feature through an element-wise add layer to perform a fusion. [Effects of the Invention]

[0032] The present invention provides a custom model comprising a pre-trained base model including at least one feature extraction block, at least one adaptation block, at least one residual unit, and a task head block, and has the effect of updating at least some of the parameters of the task head block, at least one residual unit, and at least one adaptation block through backpropagation using task loss, while not updating the parameters of the base model.

[0033] Furthermore, the present invention has the effect of enabling advanced inference by fusing at least one residual feature generated through at least one residual unit with at least one transformed feature obtained by transforming at least one intermediate feature generated from a base model through at least one adaptation block. [Brief explanation of the drawing]

[0034] The following drawings, attached for use in describing embodiments of the present invention, represent only a portion of embodiments of the present invention, and a person with ordinary skill in the art to which the present invention pertains (hereinafter referred to as "ordinary art") can obtain the other drawings from these drawings without performing any inventive work.

[0035] [Figure 1] Figure 1 is a simplified diagram illustrating a conventional learning model for performing a specific task based on a pre-trained base model. [Figure 2] Figure 2 is a simplified diagram of a learning device for learning a custom model based on a pre-trained base model according to the present invention. [Figure 3] Figure 3 is a simplified diagram illustrating the process of training a custom model based on a pre-trained base model in one embodiment of the present invention. [Figure 4] Figure 4 is a simplified drawing illustrating an example of a custom model in one embodiment of the present invention, which includes one adaptation block and one residual unit. [Figure 5] Figure 5 is a simplified diagram illustrating the process of training a custom model based on a pre-trained base model in another embodiment of the present invention. [Figure 6] Figure 6 is a simplified drawing illustrating an example of a custom model in another embodiment of the present invention, which includes two adaptation blocks and two residual units. [Figure 7]Figure 7 is a simplified drawing illustrating an example of a custom model in another embodiment of the present invention, which includes the maximum number of adaptation blocks and the maximum number of residual units. [Modes for carrying out the invention]

[0036] The detailed description of the present invention, as described below, will refer to the accompanying drawings illustrating specific embodiments in which the present invention may be carried out to illustrate each object, each technical solution, and each advantage of the present invention. These embodiments will be described in sufficient detail so that a person of the ordinary skill can carry out the present invention.

[0037] Furthermore, the word “including” and its variations thereof throughout the detailed description of the present invention and each claim are not intended to exclude any other technical features, appendices, components, or steps. Other purposes, advantages, and characteristics of the present invention will be apparent to an ordinary person, partly from this description and partly from the practice of the present invention. The following examples and drawings are provided as illustrative examples and are not intended to limit the present invention.

[0038] Furthermore, the present invention encompasses all possible combinations of the embodiments shown herein. It should be understood that while the diverse embodiments of the present invention differ from one another, they do not necessarily have to be mutually exclusive. For example, certain shapes, structures, and characteristics described herein may be embodied in other embodiments without departing from the spirit and scope of the invention in relation to one embodiment. It should also be understood that the position or arrangement of individual components within each disclosed embodiment may be modified without departing from the spirit and scope of the invention. Therefore, the detailed descriptions below should not be taken as restrictive, and the scope of the present invention is limited only by the appended claims, along with all equivalent claims, provided they are adequately described. Similar reference numerals in the drawings refer to parts that are identical or have similar functions across various aspects.

[0039] Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings, so that persons with ordinary skill in the art to which the present invention pertains can easily implement the present invention.

[0040] Figure 2 is a simplified diagram of a learning device for learning a custom model based on a pre-trained base model according to the present invention.

[0041] Referring to Figure 2, the learning device 500 may include a memory 510 that stores instructions for learning a custom model based on a pre-trained base model, and a processor 520 that performs operations for learning the custom model based on the pre-trained base model in response to the instructions stored in the memory 510. In this case, the learning device 500 may include various computing devices such as servers, PCs (Personal Computers), notebooks, workstations, tablets, mobile computers, PDAs / EDAs, mobile phones, smartphones, and IoT devices.

[0042] Specifically, the learning device 500 may achieve desired system performance by utilizing a combination of typical computing devices (e.g., devices that may include computer processors, memory, storage, input and output devices, and other components of existing computing devices; electronic communication devices such as routers and switches; and electronic information storage systems such as network-attached storage (NAS) and storage area networks (SANs)) and computer software (i.e., instructions that cause the computing devices to function in a particular manner).

[0043] Furthermore, the processor of a computing device may include hardware components such as an MPU (Micro Processing Unit) or CPU (Central Processing Unit), cache memory, and a data bus. The computing device may also further include an operating system and software components for applications that perform specific purposes.

[0044] However, this does not exclude cases where the computing device includes an integrated processor in which the medium, processor, and memory for carrying out the present invention are integrated.

[0045] Figure 3 is a simplified diagram illustrating the process of training a custom model based on a pre-trained base model in one embodiment of the present invention.

[0046] First, the learning device 500 consists of a base model including pre-trained first feature extraction blocks to nth feature extraction blocks, an adaptation block, a residual unit, and a task head block. It can input training data into a custom model that performs a specific task, generate first intermediate features to nth intermediate features through the first feature extraction blocks to nth feature extraction blocks of the base model, and generate a conversion feature S10_1 by converting the source domain corresponding to the base model in a specific intermediate feature selected from the first to nth intermediate features to a target domain corresponding to a specific task through the adaptation block. In this case, n can be an integer of 1 or more, but generally it can be assumed that n is 2 or more.

[0047] For reference, Figures 3 and 4 describe a custom model including one adaptation block and one residual unit, which is a minimized configuration that can be embodied as one embodiment, and Figures 5 to 7 describe a custom model including multiple adaptation blocks and multiple residual units, which can learn more advanced inference as another embodiment.

[0048] For example, the base model can be GroundingDINO, which has been pre-trained using various training datasets such as MSCOCO, Object365, and OpenImages. However, any model that has been pre-trained using a large number of training datasets can also be used as a base model.

[0049] Furthermore, when training data is input to the base model, all of the first to nth intermediate features are generated in each of the first to nth feature extraction blocks of the base model. However, since this embodiment uses only one adaptation block, a specific intermediate feature, which is one of the first to nth intermediate features, is selected and input into the adaptation block. Through the adaptation block, Down-Projection and Up-Projection are sequentially applied to the specific intermediate feature to convert the source domain to the target domain, thereby generating a converted feature.

[0050] The learning device 500 can then generate a fusion feature by fusing the transform feature with the transform feature extracted for a specific task via the transform feature unit, and generate a task output S20_1 by performing a learning operation on the fusion feature via the task head block.

[0051] In other words, when a transformation feature is input to a residual unit, a residual feature is generated by performing a learning operation through multiple convolution layers contained within the residual unit. A fusion feature can then be generated by fusing transformation features transmitted within the residual unit through skip connections with the residual feature. Furthermore, fusion between features can be performed through a separate fusion layer capable of operations such as add, concatenation, and weighted sum operations, but this is not the only option.

[0052] Thereafter, the learning device 500 generates a task loss by referring to the original correct answer corresponding to the task output and the learning data, and backpropagates the task loss to train at least a portion of the task head block, residual unit, and adaptation block S30_1.

[0053] The process of training a custom model that includes one adaptation block and one residual unit will be explained in more detail with reference to Figure 4.

[0054] Figure 4 is a simplified drawing illustrating an example of a custom model in one embodiment of the present invention, which includes one adaptation block and one residual unit.

[0055] Here, both Figure 4(a) and (b) are diagrams illustrating the structure of a custom model 1000, which includes a base model 1100, one adaptation block 1200, one residency unit 1300, and a task head block 1400. The process of learning the custom model 1000 will be explained by referring to (a), followed by an explanation of the differences between (a) and (b). For reference, the base model included in the custom model illustrated in Figures 4, 6, and 7 is shown as containing five feature extraction blocks, but in reality, this can be extended so that the base model contains n feature extraction blocks.

[0056] First, the learning device 500 inputs learning data in the form of image data to a pre-trained base model 1100 to perform a specific task, such as defect detection on a circuit board. The base model 1100 then performs learning operations on the input learning data sequentially through the first feature extraction block 1110 to the fifth feature extraction block 1150, thereby generating the first to fifth intermediate features through each of the first to fifth feature extraction blocks 1110 to the fifth feature extraction block 1150. In this case, each of the first to fifth feature extraction blocks 1110 to the fifth feature extraction block 1150 may be in a pre-frozen state to prevent parameter updates when backpropagation is subsequently performed through gradient operations. Also, although Figure 4 shows the base model 1100 as containing five feature extraction blocks for convenience, if it has n feature extraction blocks, the first to n intermediate features may be generated through each of the first to n feature extraction blocks.

[0057] Furthermore, since Figures 4(a) and (b) both include a single adaptation block 1200, the learning device 500 can select one of the first to fifth intermediate features as a specific intermediate feature and input it into the adaptation block 1200. By sequentially performing Up-projection and Down-Projection operations using the specific intermediate feature through the adaptation block 1200, a transformation feature can be generated that transforms the source domain corresponding to the base model 1100 in the specific intermediate feature into a target domain corresponding to a specific task (i.e., the domain corresponding to the task head block 1400). In this process, the learning device 500 selects one of the first feature extraction blocks 1110 to the fifth feature extraction block 1150 as a specific intermediate feature. However, the first intermediate feature generated through the first feature extraction block 1110 is likely to be a primitive feature possessing universal knowledge rather than a feature extracted for a specific task, and therefore may not be selected as a specific intermediate feature. Similarly, the fifth intermediate feature generated through the fifth feature block 1150 is similar in structure to the model described in Figure 1(b) and offers little practical benefit, so it may not be selected as a specific intermediate feature.

[0058] In other words, if the base model 1100 includes a first feature extraction block to the nth feature extraction block, the learning device 500 can select one of the second intermediate features to the (n-1)th intermediate features generated from the second feature extraction block to the (n-1)th feature extraction block as a specific intermediate feature, and input that specific intermediate feature into the adaptation block 1200.

[0059] Furthermore, the learning device 500 can input the conversion feature generated through the adaptation block 1200 into the residual unit 1300, generate a residual feature through the residual block 1310 included in the residual unit 1300, and then input it into the fusion layer 1320. Simultaneously, it can input the conversion feature generated through the adaptation block 1200 into the fusion layer 1320 included in the residual unit 1300 using the skip structure included in the residual unit 1300, and generate a fusion feature by fusing the residual feature and the conversion feature through the fusion layer 1320. In this case, the residual block 1310 includes multiple convolution layers, and the residual feature may be generated by sequentially performing convolution operations through multiple convolution layers on the transformation feature. The fusion layer 1320 can apply add operations, concatenation operations, weighted sum operations, etc., as methods of fusion between features. Figure 4 shows that an add operation was performed using the element-wise add layer 1320 as the fusion layer.

[0060] On the other hand, if cost savings are needed when training the custom model 1000, the number of parameters required for training can be reduced by configuring the rank of the resistive unit 1300, or more precisely, the rank of at least some of the convolution layers contained in the resistive block 1310, to be low-rank. Specifically, this involves configuring the first filter in the full-rank state contained in each of the convolution layers to be decomposed into multiple second filters having a lower rank than the first filter. For example, if the first filter is a d×k matrix, it can be decomposed into two second filters having a d×r matrix and an r×k matrix, where r is the rank value. For example, if the matrix has d=10 and k=10 and the rank is r=3, the first filter is 10×10 and requires training on 100 parameters, whereas the second filters are 10×3 and 3×10 and only require training on a total of 60 parameters. Thus, by implementing the residual block 1310 in a low-rank configuration, the cost required to train the custom model 1000 can be reduced. The above explanation only describes the case where the first filter is in the form of a 2D matrix, but even if the first filter is in the form of a 3D tensor, it can be similarly implemented in a low-rank configuration by decomposing it into multiple second filters. Of course, if required by those skilled in the art, further cost savings can be obtained by configuring the adaptation block 1200 in a low-rank configuration as well.

[0061] Thereafter, the learning device 500 can input fusion features to the task head block 1400, and the task head block 1400 can generate task outputs. For example, if a particular task is related to image classification, the task head block 1400 may consist of FC (Fully-connected) layers, and if a particular task is related to object detection, the task head block 1400 may consist of pooling layers and FC layers.

[0062] Then, the learning device 500 generates a task loss by referencing the original correct answer (GT, Ground Truth) corresponding to the task output and the training data, and backpropagates the task loss (the direction is shown by a dotted line) to train the task head block 1400, the residual unit 1300, and the adaptation block 1200 to update at least some of their parameters. Here, since the parameters of the first feature extraction block to the nth feature extraction block (first feature extraction block 1110 to fifth feature extraction block 1150 in Figure 4) are fixed in advance through freezing, a gradient operation may be performed on the first feature extraction block to the nth feature extraction block of the base model 1100 so as not to update the parameters. Therefore, since the base model 1100 does not train, the training costs are reduced compared to the conventional Full Fine-tuning method. However, by fusing the features generated from the adaptation block 1200 and the residual unit 1300, more advanced inference becomes possible, thereby improving the training performance for performing specific tasks.

[0063] On the other hand, Figures 4(a) and 4(b) are identical in the basic structure of the custom model 1000, but they differ in that (a) is configured so that the fourth intermediate feature generated through the fourth feature extraction block 1140 of the base model 1100 is selected as a specific intermediate feature and input to the adaptation block 1200, while (b) is configured so that the fifth intermediate feature generated through the fifth feature extraction block 1150 of the base model 1100 is selected as a specific intermediate feature and input to the adaptation block 1200. In other words, in (b), the output of the last feature extraction block of the base model 1100 is input to the adaptation block 1200. In such a case, the structure is similar to the model described in Figure 1(b) above, and the gain that can be substantially obtained may not be large. Therefore, when realizing the custom model 1000 with a minimal configuration, it is more preferable to use the output of one of the remaining feature extraction blocks, excluding the first and last feature extraction blocks, as in (a).

[0064] The above describes an example of a custom model implemented with a minimal configuration. Referring to Figures 5 to 7, the configuration of the custom model, which includes multiple adaptation blocks and multiple residual units, will be explained.

[0065] Figure 5 is a simplified diagram illustrating the process of training a custom model based on a pre-trained base model in another embodiment of the present invention.

[0066] First, the learning device 500 is composed of a base model including pre-trained first feature extraction blocks to nth feature extraction blocks, first adoption blocks to kth adaptation blocks, first residual units to kth residual units, and a task head block. By inputting learning data into a custom model that performs a specific task, the device can generate first intermediate features to nth intermediate features through each of the first feature extraction blocks to nth feature extraction blocks of the base model, and generate first transformation features to kth transformation features by transforming the source domain corresponding to the base model in each of the first specific intermediate features to kth specific intermediate features selected from the first to nth intermediate features through each of the first adoption blocks to kth adaptation blocks into a target domain corresponding to a specific task. In this case, n is an integer greater than or equal to 2, and k is an integer greater than or equal to 2 and less than or equal to n.

[0067] For reference, the custom models shown in Figures 6 and 7 are different embodiments from the custom models shown in Figures 3 and 4. Figure 6 describes a custom model that includes two adaptation blocks and two residual units, while Figure 7 describes a custom model that includes the maximum number of adaptation blocks and the maximum number of residual units to enable the training of the most advanced inference.

[0068] Furthermore, when training data is input to the base model, all of the first to nth intermediate features are generated in each of the first to nth feature extraction blocks of the base model. However, there are k adoption blocks, which is less than or equal to n, i.e., there are first to kth adoption blocks. Therefore, the learning device 500 selects the first specific intermediate features to the kth specific intermediate features as k specific intermediate features from the first to nth intermediate features and inputs them into each of the first to kth adoption blocks. Down-Projection and Up-Projection are then sequentially applied to each of the first specific intermediate features to the kth specific intermediate features through each of the first to kth adoption blocks to convert the source domain of the base model corresponding to each of the first specific intermediate features to the kth specific intermediate features into a target domain, thereby generating the first to kth transformed features.

[0069] The learning device 500 then generates a first fusion feature by fusing the first refractory feature extracted for a specific task with the first transformation feature via the first refractory unit. After fusing the (i-1) fusion feature with the i-th transformation feature to generate the i-th provisional feature, the learning device 500 repeats the process of generating the i-th fusion feature by fusing the i-th refractory feature extracted for a specific task with the i-th provisional feature via the i-th refractory unit. The learning device 500 then generates a task output S20_2 by performing a learning operation on the k-th fusion feature output from the k-th refractory unit via the task head block. In this case, i can be an integer between 2 and k.

[0070] To reiterate, the learning device 500 can generate the first residual feature by inputting the first transformation feature into the first residual unit and performing a learning operation through multiple convolution layers contained in the first residual unit, and generate the first fusion feature by fusing the first transformation feature transmitted through skip connections within the first residual unit with the first residual feature. In other words, the process up to the generation of the first fusion feature can be carried out in the same manner as in Figure 4.

[0071] Furthermore, since the learning process for the second to the kth residual unit is the same, and i is an integer between 2 and k, generalizing this to the learning process for the ith residual unit, the learning device 500 first performs the process of generating the ith provisional feature by fusing the ith transformation feature and the (i-1) fusion feature through the (i-1)th residual unit. The ith residual feature can then be generated by performing a learning operation on the ith provisional feature through the ith residual unit via multiple convolution layers contained within the ith residual unit. The ith fusion feature can then be generated by fusing the ith provisional feature, which is transmitted within the ith residual unit via skip linkage, with the ith residual feature. This process can be repeated until the kth fusion feature is finally output from the kth residual unit. In this case, when fusion occurs between features, the fusion is performed through a separate fusion layer capable of operations such as add, concatenation, and weighted sum. The fusion layer that performs the fusion to generate the i-th provisional feature may be a fusion layer included in the (i-1)th residual unit, or it may be a fusion layer separately added between the (i-1)th residual unit and the i-th residual unit, but it is not limited to these. However, for the sake of explanation, we will assume that the fusion layer that performs the fusion to generate the i-th provisional feature is a fusion layer included in the (i-1)th residual unit.

[0072] Thereafter, the learning device 500 generates a task loss by referring to the original correct answer corresponding to the task output and the learning data, and backpropagates the task loss to train at least a portion of the task head block, residual unit, and adaptation block S30_2.

[0073] Thus, the process of training a custom model that includes at least two adaptation blocks and at least two residual units will be explained in more detail with reference to Figures 6 and 7.

[0074] Figure 6 is a simplified drawing illustrating an example of a custom model in another embodiment of the present invention, which includes two adaptation blocks and two residual units.

[0075] Here, both Figure 6(a) and (b) are diagrams showing example structures of a custom model 2000, including a base model 2100, a first adaptation block 2200_1, a second adaptation block 2200_2, a first resistor unit 2300_1, a second resistor unit 2300_2, and a task head block 2400. The process of learning the custom model 2000 will be explained by referring to (a), followed by an explanation of the differences between (a) and (b).

[0076] First, the learning device 500 inputs learning data in the form of image data to a pre-trained base model 2100 to perform a specific task, such as defect detection on a circuit board. The base model 2100 then performs learning operations on the input learning data sequentially through the first feature extraction block 2110 to the fifth feature extraction block 2150, thereby generating the first to fifth intermediate features through each of the first to fifth feature extraction blocks 2110 to the fifth feature extraction block 2150. In this case, each of the first to fifth feature extraction blocks 2110 to the fifth feature extraction block 2150 may be in a pre-frozen state to prevent parameter updates when backpropagation is subsequently performed through gradient operations. Also, although Figure 6 shows the base model 2100 as containing five feature extraction blocks for convenience, if it has n feature extraction blocks, the first to n intermediate features may be generated through each of the first to n feature extraction blocks.

[0077] Furthermore, since Figures 6(a) and (b) commonly include the first adaptation block 2200_1 and the second adaptation block 2200_2, the learning device 500 can select one of the first to fifth intermediate features as the first specific intermediate feature and input it into the first adaptation block 2200_1, and select the other one of the first to fifth intermediate features as the second specific intermediate feature and input it into the second adaptation block 2200_2. In this case, since the first intermediate feature generated through the first feature extraction block 2110 of the base model 2100 may contain only excessively universal knowledge, the learning device 500 can select the first specific intermediate feature and the second specific intermediate feature from the second to fifth intermediate features generated through the second to fifth feature extraction blocks 2120 to 2150, excluding the first feature extraction block 2110. Furthermore, in selecting the first specific intermediate feature and the second specific intermediate feature, the first specific intermediate feature is selected by selecting an intermediate feature generated through the middle part feature extraction block of the base model 2100 (e.g., the second feature extraction block 2120 and the third feature extraction block 2130), and the second specific intermediate feature is selected by selecting an intermediate feature generated through the latter part feature extraction block of the base model 2100 (e.g., the fourth feature extraction block 2140 and the fifth feature extraction block 2150). This allows for the reflection of all relatively universal knowledge and knowledge applicable to a specific task, thereby enabling more advanced inference learning.

[0078] Extending this to a base model 2100 including a first feature extraction block to an nth feature extraction block, the learning device 500 can select one of the second intermediate features to the (n-2)th intermediate features generated through the second feature extraction block to the (n-2)th feature extraction block as the first specific intermediate feature and input it into the first adaptation block 2200_1, and can select one of the (n-1)th intermediate features and the nth intermediate feature generated through the (n-1)th feature extraction block and the nth feature extraction block as the second specific intermediate feature and input it into the second adaptation block 2200_2. For example, (a) is an example in which the second intermediate feature is selected as the first specific intermediate feature and the fourth intermediate feature is selected as the second specific intermediate feature, and (b) may be an example in which the third intermediate feature is selected as the first specific intermediate feature and the fifth intermediate feature is selected as the second specific intermediate feature.

[0079] Thereafter, by sequentially performing Up-projection and Down-Projection operations using the first specific intermediate feature through the first adoption block 2200_1, a first transformation feature is generated in which the source domain corresponding to the base model 2100 in the first specific intermediate feature is transformed to a target domain corresponding to a specific task (i.e., the domain corresponding to the task head block 2400). Subsequently, by sequentially performing Up-projection and Down-Projection operations using the second specific intermediate feature through the second adoption block 2200_2, a second transformation feature may be generated in which the source domain corresponding to the base model 2100 in the second specific intermediate feature is transformed to a target domain corresponding to a specific task (i.e., the domain corresponding to the task head block 2400).

[0080] Furthermore, the learning device 500 can input the first transformation feature generated from the first adoption block 2200_1 into the first resistive unit 2300_1, generate the first resistive feature through the first resistive block 2310_1 contained in the first resistive unit 2300_1, and then input it into the first fusion layer 2320_1. At the same time, it can input the first transformation feature generated through the first adoption block 2200_1 into the first fusion layer 2320_1 using a skip structure, and generate the first fusion feature by fusing the first resistive feature and the first transformation feature through the first fusion layer 2320_1.

[0081] The learning device 500 then fuses the second transformation feature generated through the second adaptation block 2200_2 with the first fusion feature generated through the first resistive unit 2300_1 to obtain a second provisional feature, inputs the second provisional feature into the second resistive unit 2300_2, generates a second resistive feature through the second resistive block 2310_2 contained in the second resistive unit 2300_2, and then inputs it into the second fusion layer 2320_2. At the same time, the second provisional feature is input into the second fusion layer 2320_2 using a skip structure, and the second fusion feature is generated by fusing the second resistive feature and the second provisional feature through the second fusion layer 2320_2. In this case, the second provisional feature can be acquired through the first fusion layer 2320_1, but an auxiliary fusion layer can also be added between the first residual unit 2300_1 and the second residual unit 2300_2 so that the second provisional feature is acquired through the auxiliary fusion layer.

[0082] Furthermore, each of the first residual unit 2300_1 and the second residual unit 2300_2, more specifically the first residual block 2310_1 and the second residual block 2310_2, includes multiple convolution layers, and the first residual feature and the second residual feature can be generated by sequentially performing convolution operations through multiple convolution layers in each of the first residual block 2310_1 and the second residual block 2310_2, respectively. The first fusion layer 2320_1 and the second fusion layer 2320_2 can each apply add operations, concatenation operations, weighted sum operations, etc., as methods of fusion between features, and in Figure 6, it can be seen that the add operation was performed using the first element-wise add layer 2320_1 and the second element-wise add layer 2320_2, respectively, as the first fusion layer and the second fusion layer.

[0083] On the other hand, if cost savings are needed when training the custom model 2000, the number of parameters required for training can be reduced by configuring at least some of the ranks of the first regenerative unit 2300_1 and the second regenerative unit 2320_2, or more precisely, at least some of the ranks of the multiple convolution layers contained in the first regenerative block 2310_1 and the second regenerative block 2310_2, to be low-rank. Specifically, this involves configuring the first filter in a full-rank state contained in each of the multiple convolution layers to be decomposed into multiple second filters having a lower rank than the first filter. For example, if the first filter is a d×k matrix, it can be decomposed into two second filters, one a d×r matrix and the other an r×k matrix, where r is the rank value. For example, if the matrix has d=5 and k=5 and the rank is r=1, the first filter is 5×5 and requires training on 25 parameters, whereas the second filters are 5×1 and 1×5 and only require training on a total of 10 parameters. By implementing the first and second resistor dual blocks 2310_1 and 2310_2 in a low-rank manner, the cost required to train the custom model 2000 can be reduced. The above explanation only describes the case where the first filter is in the form of a 2D matrix, but the same method can be used to implement the first filter in a low-rank manner by decomposing it into multiple second filters even if it is in the form of a 3D tensor. Of course, if required by those skilled in the art, further cost savings can be achieved by configuring the first adoption block 2200_1 and the second adoption block 2200_2 in a low-rank configuration.

[0084] Thereafter, the learning device 500 can input fusion features to the task head block 2400, and the task head block 2400 can generate task outputs. For example, if a particular task is related to image classification, the task head block 2400 may consist of FC (Fully-connected) layers, and if a particular task is related to object detection, the task head block 2400 may consist of pooling layers and FC layers.

[0085] Then, the learning device 500 generates a task loss by referencing the original correct answer (GT, Ground Truth) corresponding to the task output and the learning data, and backpropagates the task loss (the direction is shown by a dotted line) to train at least some of the parameters of the task head block 2400, the first resistor unit 2300_1, the second resistor unit 2300_2, the first adaptation block 2200_1, and the second adaptation block 2200_2 to be updated. Here, since the parameters of the first feature extraction block to the nth feature extraction block (first feature extraction block 2110 to the fifth feature extraction block 2150 in Figure 6) of the base model 2100 are fixed in advance through freezing, a gradient operation may be performed on the first feature extraction block to the nth feature extraction block of the base model 2100 so as not to update the parameters. Therefore, since the base model 2100 does not learn, the cost required for learning is reduced compared to the conventional Full Fine-tuning method. However, by fusing the features generated from the first adaptation block 2200_1, the second adaptation block 2200_2, the first resistor unit 2300_1, and the second resistor unit 2300_2, more advanced inference becomes possible, thereby improving the learning performance for performing specific tasks.

[0086] On the other hand, Figures 6(a) and (b) are identical in the basic structure of the custom model 2000, but differ in that in (a), the second intermediate feature and the fourth intermediate feature generated through the second feature extraction block 2120 and the fourth feature extraction block 2140 of the base model 2100 are selected as the first specific intermediate feature and the second specific intermediate feature, respectively, and input into the first adoption block 2200_1 and the second adoption block 2200_2, respectively, while in (b), the third intermediate feature and the fifth intermediate feature generated through the third feature extraction block 2130 and the fifth feature extraction block 2150 of the base model 2100 are selected as the first specific intermediate feature and the second specific intermediate feature, respectively, and input into the first adoption block 2200_1 and the second adoption block 2200_2, respectively.

[0087] Finally, the process of training the best-performing custom model by further extending the custom model in Figure 6 to include the maximum number of adaptation blocks and the maximum number of residual units will be explained with reference to Figure 7.

[0088] Figure 7 is a simplified drawing illustrating an example of a custom model in another embodiment of the present invention, which includes the maximum number of adaptation blocks and the maximum number of residual units.

[0089] For reference, the custom model 3000 shown in Figure 7 is composed of a base model 3100, first adoption blocks 3200_1 to 5th adoption blocks 3200_5, first resistor units 3300_1 to 5th resistor units 3300_5, and a task head block 3400. For convenience, the example shown includes the first feature extraction blocks 3110 to 5th feature extraction blocks 3150 in the base model 3100. Accordingly, the custom model 3000 is configured to include the first adoption blocks 3200_1 to 5th adoption blocks 3200_5, first resistor units 3300_1 to 5th resistor units 3300_5, and a task head block 3400 for the most advanced inference learning. For example, if the base model 3100 includes a first feature extraction block to the nth feature extraction block, the custom model 3000 may be configured such that it includes a maximum of n adaptation blocks and residual units, respectively.

[0090] In this case, the learning device 500 inputs learning data in the form of image data to a pre-trained base model 3100 to perform a specific task, such as defect detection on a circuit board. The base model 3100 then performs learning operations on the input learning data sequentially through the first feature extraction block 3110 to the fifth feature extraction block 3150, thereby generating the first to fifth intermediate features through each of the first to fifth feature extraction blocks 3110 to the fifth feature extraction block 3150. In this case, each of the first to fifth feature extraction blocks 3110 to the fifth feature extraction block 3150 may be in a pre-frozen state to prevent parameter updates when backpropagation is subsequently performed through gradient operations. If this is extended to assume that the base model 3100 has n feature extraction blocks, then the first to n intermediate features may be generated through each of the first to n feature extraction blocks.

[0091] The learning device 500 can input each of the first to fifth intermediate features into each of the first to fifth adaptation blocks 3200_1 to 5th adaptation blocks 3200_5. Extending this to assume that the base model 3100 has n feature extraction blocks, the number of adaptation blocks is the same as the number of feature extraction blocks, configured as first to nth adaptation blocks, and the learning device 500 can input each of the first to nth intermediate features into each of the first to nth adaptation blocks.

[0092] Thereafter, by sequentially performing Up-projection and Down-Projection operations using each of the first to fifth intermediate features through the first to fifth adaptation blocks 3200_1 to 3200_5, the first to fifth transformation features can be generated, which are obtained by converting the source domain corresponding to the base model 3100 in each of the first to fifth intermediate features to the target domain corresponding to a specific task (i.e., the domain corresponding to the task head block 3400). If this is extended assuming that the base model 3100 has n feature extraction blocks, then the first to n transformation features can be generated by converting the source domain corresponding to the base model in each of the first to n intermediate features to the target domain corresponding to a specific task through the first to n adaptation blocks.

[0093] Furthermore, the learning device 500 can input the first transformation feature generated from the first adoption block 3200_1 into the first resistive unit 3300_1, generate the first resistive feature through the first resistive block 3310_1 contained in the first resistive unit 3300_1, and then input it into the first fusion layer 3320_1. At the same time, it can input the first transformation feature generated through the first adoption block 3200_1 into the first fusion layer 3320_1 using a skip structure, and generate the first fusion feature by fusing the first resistive feature and the first transformation feature through the first fusion layer 3320_1.

[0094] The learning device 500 then fuses the second transformation feature generated through the second adaptation block 3200_2 with the first fusion feature generated through the first resistive unit 3300_1 to obtain a second provisional feature, inputs the second provisional feature into the second resistive unit 3300_2, generates a second resistive feature through the second resistive block 3310_2 contained in the second resistive unit 3300_2, and then inputs it into the second fusion layer 3320_2. At the same time, the second provisional feature is input into the second fusion layer 3330_2 using a skip structure, and the second fusion feature is generated by fusing the second resistive feature and the second provisional feature through the second fusion layer 3320_2. In this case, the second provisional feature can be acquired through the first fusion layer 3320_1, but an auxiliary fusion layer can be added between the first resistor dual unit 3300_1 and the second resistor dual unit 3300_2 so that the second provisional feature is acquired through the auxiliary fusion layer.

[0095] Thereafter, the learning device 500 can also generate the third to fifth fusion features for the third to fifth residual units 3300_3 to 5th residual units 3300_5 by repeatedly performing the same process as that used to generate the second fusion feature through the second residual unit 3300_2. Extending this to the case where the base model 3100 has n feature extraction blocks, the number of residual units is also n, the same as the number of adaptation blocks, and the system consists of first to nth residual units. The operation process of the second to nth residual units is repeated identically. A generalized representation of this is the same as the S20_2 process in Figure 5, but it is performed until i becomes n. Of course, in the S20_2 process in Figure 5, i is assumed to be a maximum of k, but in the S20_1 process, k is assumed to be a maximum of n, so i can be considered an integer ranging from 2 to a maximum of n.

[0096] Furthermore, each of the first to fifth registerdual units 3300_1 to 3300_5, more specifically the first to fifth registerdual blocks 3310_1 to 3310_5, contains multiple convolution layers, and each of the first to fifth registerdual features can be generated by sequentially performing convolution operations through multiple convolution layers in each of the first to fifth registerdual blocks 3310_1 to 3310_5. Each of the first to fifth fusion layers 3320_1 to 3320_5 can apply add operations, concatenation operations, weighted sum operations, etc., as methods of fusion between features, and in Figure 7, it can be seen that the add operation was performed using each of the first to fifth fusion layers, specifically the first element-wise add layer 3320_1 to 2320_5. Similarly, this can be extended to the first resistor dual unit to the nth resistor dual unit, and a generalized representation is explained in S20_2 in Figure 5, so please refer to that.

[0097] On the other hand, if it is necessary to reduce additional costs when training the custom model 3000, the number of parameters required for training can be reduced by configuring at least some of the ranks of the first regenerative unit 3300_1 to the fifth regenerative unit 3300_5, or more precisely, at least some of the ranks of the multiple convolution layers contained in the first regenerative block 3310_1 to the fifth regenerative block 3310_5, to be low-rank. Extending this to the first to the nth regenerative block, it means configuring at least some of the ranks of the multiple convolution layers contained in each of the first to the nth regenerative units to be low-rank, and configuring the first filter in a full-rank state contained in each of the multiple convolution layers to be decomposed into multiple second filters having a lower rank than the first filter. Of course, if required by those skilled in the art, further cost savings can be achieved by configuring the first to fifth adoption blocks 3200_1 (or the first to nth adoption blocks if extended to the maximum number) in a low-rank configuration.

[0098] Thereafter, the learning device 500 can input the fusion feature to the task head block 3400, causing the task head block 3400 to generate the task output. The learning device 500 then generates a task loss by referencing the original correct answer (GT, Ground Truth) corresponding to the task output and the training data, and backpropagates the task loss (the direction is shown by a dotted line) to train the task head block 3400, the first resistor dual unit 3300_1 to the fifth resistor dual unit 3300_5, and the first adaptation block 3200_1 to the third adaptation block 3200_5 to update at least some of their parameters. Extending this, it is possible to train the task head block, the first resistor dual unit to the nth resistor dual unit, and the first adaptation block to the nth adaptation block to update at least some of their parameters.

[0099] Here, the base model 3100 has its parameters fixed in advance through freezing for the first feature extraction block 3110 to the fifth feature extraction block 3150 (or, if extended, the first feature extraction block to the nth feature extraction block). Therefore, gradient operations are not performed on the first feature extraction block 3110 to the fifth feature extraction block 3150 of the base model 3100 to update the parameters. Consequently, since the base model 3100 does not learn, the cost required for learning is reduced compared to the conventional Full Fine-tuning method. However, by fusing the features generated from the first adoption block 3200_1 to the fifth adoption block 3200_5 and the first residual unit 3300_1 to the third residual unit 3300_5, it becomes possible to perform more advanced inference than the custom model 2000 explained in Figure 6, thereby improving the learning performance for performing specific tasks.

[0100] Furthermore, the embodiments of the present invention described above can be embodied in the form of program instructions that can be executed through a variety of computer components and stored on a computer-readable recording medium. The computer-readable recording medium may include program instructions, data files, data structures, etc., individually or in combination. The program instructions stored on the computer-readable recording medium may be specifically designed and configured for the present invention or may be publicly known and available to those skilled in the field of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks and magnetic tapes, optical recording media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and hardware devices specifically configured to store and execute program instructions, such as ROMs, RAMs, and flash memory. Examples of program instructions include not only machine code, such as that produced by a compiler, but also high-level language code that can be executed by a computer using an interpreter or the like. The hardware devices may be configured to operate as one or more software modules to perform the processing according to the present invention, and vice versa.

[0101] Although the present invention has been described above with specific details such as concrete components, limited embodiments, and drawings, these are provided only to aid in a more general understanding of the invention, and the invention is not limited to the above embodiments. A person with ordinary skill in the art to which the invention belongs can make various modifications and variations from this description.

[0102] Therefore, the concept of the present invention should not be limited to the embodiments described above, and it can be said that not only the claims described later, but also all modifications equivalent to or equivalent to the claims of this invention fall within the scope of the concept of the present invention.

Claims

1. In a method for training a custom model based on a pre-trained base model, (a) A learning device inputs learning data into a custom model that performs a specific task, comprising a base model including a pre-trained first feature extraction block to the nth feature extraction block (where n is an integer of 2 or more), a first adaptation block to the kth adaptation block (where k is an integer of 2 or more and less than or equal to n), a first residual unit to the kth residual unit, and a task head block, to generate first intermediate features to the nth intermediate features through each of the first feature extraction blocks to the nth feature extraction blocks of the base model, and generates first transformation features to the kth transformation features, each of which converts the source domain corresponding to the base model in each of the first specific intermediate features to the kth specific intermediate features selected from the first intermediate features to the nth intermediate features, into a target domain corresponding to the specific task; (b) After the learning device generates a first fusion feature by fusing the first fusion feature extracted for the specific task with the first fusion feature through the first fusion unit, and then generates an i-th provisional feature by fusing the (i-1) fusion feature with the i-th fusion feature (where i is an integer greater than or equal to 2 and less than or equal to k), the learning device generates an i-th fusion feature by fusing the i-th fusion feature extracted for the specific task with the i-th provisional feature through the i-th fusion unit, and generates a task output by performing a learning operation on the k-th fusion feature output from the k-th fusion unit through the task head block; and (c) The learning device generates a task loss by referring to the task output and the original correct answer corresponding to the learning data, and backpropagates the task loss to train at least a portion of the task head block, the first residual unit to the k residual unit, and the first adaptation block to the k adaptation block; A method that includes this.

2. In step (a) above, The method according to claim 1, characterized in that the learning device selects the nth intermediate feature or the (n-1)th intermediate feature from the first intermediate feature to the nth intermediate feature through any one of the first to the kth adaptation blocks, and generates one transformation feature by converting the source domain to the target domain; and selects an intermediate feature which is one of the second intermediate feature to the (n-2)th intermediate feature through another adaptation block from the first to the kth adaptation blocks, and generates another transformation feature by converting the source domain to the target domain.

3. In step (a) above, The method according to claim 1, characterized in that each of the first to k-residual units includes a plurality of convolution layers, and a first filter included in at least some of the plurality of convolution layers is decomposed into a plurality of second filters having a lower rank than the first filter.

4. In step (a) above, Each of the first to n feature extraction blocks has its parameters fixed through freezing. In step (c) above, The method according to claim 1, characterized in that when the learning device backpropagates the task loss, it does not update the parameters of each of the first feature extraction block to the n feature extraction blocks.

5. In step (b) above, The method according to claim 1, characterized in that the learning device generates the first fusion feature by performing an add operation on the first resistive feature and the first transformation feature through the first element-wise add layer and fusing them; generates the i provisional feature by performing an add operation on the i transformation feature and the (i-1) fusion feature through the (i-1) element-wise add layer and fusing them; and generates the i fusion feature by performing an add operation on the i resistive feature and the i provisional feature through the i element-wise add layer and fusing them.

6. In a method for training a custom model based on a pre-trained base model, (a) A learning device comprising a base model including a pre-trained first feature extraction block to the nth feature extraction block (where n is an integer of 1 or more), an adaptation block, a residual unit, and a task head block, inputs training data to a custom model that performs a specific task, generates first intermediate features to the nth intermediate features through each of the first feature extraction blocks to the nth feature extraction blocks of the base model, and generates a transformation feature through the adaptation block that transforms the source domain corresponding to the base model in a specific intermediate feature selected from the first to the nth intermediate features into a target domain corresponding to the specific task; (b) The learning device generates a fusion feature by fusing the transform feature with the transform feature and the transform feature through the transform unit, and generates a task output by performing a learning operation on the fusion feature through the task head block; and (c) The learning device generates a task loss by referring to the task output and the original correct answer corresponding to the learning data, and backpropagates the task loss to train at least a portion of the task head block, the residual unit and the adaptation block; A method that includes this.

7. In step (a) above, The method according to claim 6, characterized in that the learning device generates the conversion feature by selecting one of the second intermediate feature to the (n-1)th intermediate feature as the specific intermediate feature through the adaptation block and converting the source domain to the target domain.

8. In step (a) above, The method according to claim 6, characterized in that the resistive unit includes a plurality of convolution layers, and is configured such that a first filter included in at least some of the plurality of convolution layers is decomposed into a plurality of second filters having a lower rank than the first filter.

9. In step (a) above, Each of the first to n feature extraction blocks has its parameters fixed through freezing. In step (c) above, The method according to claim 6, characterized in that when the learning device backpropagates the task loss, it does not update the parameters of each of the first feature extraction block to the n feature extraction blocks.

10. In step (b) above, The method according to claim 6, characterized in that the learning device generates the fusion feature by performing an add operation on the residual feature and the transformation feature through the element-wise add layer and fusing them.

11. In a learning device that trains a custom model based on a pre-trained base model, At least one memory to store instructions; and Includes at least one processor configured to perform the aforementioned instructions; The processor comprises (I) a base model including pre-trained first to nth feature extraction blocks (where n is an integer greater than or equal to 2), first to kth adaptation blocks (where k is an integer greater than or equal to 2 and less than or equal to n), first to kth residual units, and a task head block, and inputs training data to a custom model that performs a specific task, generates first to nth intermediate features through each of the first to nth feature extraction blocks of the base model, and assigns a source domain corresponding to the base model in each of the first to kth specific intermediate features selected from the first to nth intermediate features through each of the first to kth adaptation blocks to the specific task. (II) A process to generate first to kth transformation features that have been transformed into a target domain; (II) After generating a first fusion feature by fusing the first transformation feature with the first transformation feature extracted for the specific task through the first residual unit, when the i-1th fusion feature and the ith transformation feature (where i is an integer greater than or equal to 2 and less than or equal to k) are fused to generate the ith provisional feature, the process to generate the ith fusion feature by fusing the ith provisional feature with the ith provisional feature extracted for the specific task through the ith residual unit, and to generate a task output by performing a learning operation on the kth fusion feature output from the kth residual unit through the task head block;(III) A learning device that performs the process of generating a task loss by referring to the task output and the original correct answer corresponding to the learning data, and backpropagating the task loss to train at least a portion of the task head block, the first resistor unit to the k resistor unit, and the first adaptation block to the k adaptation block.

12. In the above process (I), The learning device according to claim 11, characterized in that the processor generates a conversion feature by selecting the nth intermediate feature or the (n-1)th intermediate feature from the first intermediate feature to the nth intermediate feature through any one of the first to the kth adaptation blocks, and converting the source domain to the target domain; and generating another conversion feature by selecting an intermediate feature that is one of the second intermediate feature to the (n-2)th intermediate feature through another adaptation block from the first to the kth adaptation blocks, and converting the source domain to the target domain.

13. In the above process (I), The learning device according to claim 11, wherein each of the first to k-residual units includes a plurality of convolution layers, and is configured such that a first filter included in at least some of the plurality of convolution layers is decomposed into a plurality of second filters having a lower rank than the first filter.

14. In the above process (I), Each of the first to n feature extraction blocks has its parameters fixed through freezing. In the aforementioned (III) process, The learning device according to claim 11, characterized in that when the processor backpropagates the task loss, it does not update the parameters of each of the first feature extraction block to the n feature extraction block.

15. In the above process (II), The learning device according to claim 11, characterized in that the processor generates the first fusion feature by performing an add operation on the first resistive feature and the first transformation feature through the first element-wise add layer and fusing them; generates the i provisional feature by performing an add operation on the i transformation feature and the (i-1) fusion feature through the (i-1) element-wise add layer and fusing them; and generates the i fusion feature by performing an add operation on the i resistive feature and the i provisional feature through the i element-wise add layer and fusing them.

16. In a learning device that trains a custom model based on a pre-trained base model, At least one memory to store instructions; and Includes at least one processor configured to perform the aforementioned instructions; The processor comprises (I) a base model including a pre-trained first feature extraction block to the nth feature extraction block (where n is an integer of 1 or more), an adaptation block, a residual unit, and a task head block, inputting training data to a custom model that performs a specific task, generating first intermediate features to the nth intermediate features through each of the first feature extraction blocks to the nth feature extraction blocks of the base model, and through the adaptation block, the source domain corresponding to the base model in a specific intermediate feature selected from the first to the nth intermediate features to the target corresponding to the specific task. A learning device that performs the following steps: (II) a process of generating a transformation feature converted to a domain; (II) a process of generating a fusion feature by fusing the transformation feature with the residual feature extracted for the specific task using the transformation feature through the residual unit, and generating a task output by performing a learning operation on the fusion feature through the task head block; and (III) a process of generating a task loss by referring to the task output and the original correct answer corresponding to the training data, and backpropagating the task loss to train at least a portion of the task head block, the residual unit, and the adaptation block.

17. In the above process (I), The learning device according to claim 16, characterized in that the processor generates the conversion feature by selecting one of the second intermediate feature to the (n-1)th intermediate feature as the specific intermediate feature through the adaptation block and converting the source domain to the target domain.

18. In the above process (I), The learning device according to claim 16, wherein the resistive unit includes a plurality of convolution layers, and is configured such that a first filter included in at least some of the plurality of convolution layers is decomposed into a plurality of second filters having a lower rank than the first filter.

19. In the above process (I), Each of the first to n feature extraction blocks has its parameters fixed through freezing. In the aforementioned (III) process, The learning device according to claim 16, characterized in that when the processor backpropagates the task loss, it does not update the parameters of each of the first feature extraction block to the n feature extraction block.

20. In the above process (II), The learning device according to claim 16, characterized in that the processor generates the fusion feature by performing an add operation on the resistive feature and the conversion feature through the element-wise add layer and fusing them.

Images