METHOD FOR TRAINING CUSTOM MODEL BASED ON PRE-TRAINED BASE MODEL AND LEARNING DEVICE USING THE SAME

Abstract

A method is provided for training a custom model that is based on a pre-trained base model. [Solution] The method includes a step of a learning device inputting learning data into a custom model that performs a specific task to generate 1st to nth intermediate features, and generating 1st to kth transformation features by transforming a source domain corresponding to a base model in the 1st to kth specific intermediate features into a target domain corresponding to the specific task; a step of the learning device generating a task output by performing a learning calculation on the kth fusion feature output from the kth residual unit through a task head block; and a step of the learning device generating a task loss by referring to the task output and an original correct answer corresponding to the learning data, and backpropagating the task loss to train at least a portion of the task head block, the 1st to kth residual units, and the 1st to kth adaptation blocks.

Description

[Technical field]

[0001] The present invention relates to a method for training a custom model based on a pre-trained base model and a training device using the same. [Background technology]

[0002] Base models such as Grounding DINO, which are trained using various large-scale learning datasets such as MSCOCO, Object365, and OpenImages, have extensive and universal knowledge. However, such base models have the problem that they do not have or lack detailed knowledge required to perform specific tasks such as detecting defects in circuit boards. Therefore, research has been conducted to learn knowledge required to perform specific tasks using a relatively small amount of training data based on the universal knowledge held by the base model, and a related conventional learning model is shown in Figure 1.

[0003] As an example, FIG. 1(a) shows a learning model trained by the Full Fine-Tuning method, and when learning data is input to a pre-trained base model 100 including five layers, the learning data is sequentially learned through the first layer 110 to the fifth layer 150, and then task data is output. Learning can be performed by sequentially updating the weight parameters of the fifth layer 150 to the first layer 110 through backpropagation, which performs gradient calculation in the reverse direction as shown by the dotted arrow, based on the loss generated by referring to the task data and the corresponding ground truth (GT). Therefore, the Full Fine-tuning method has the advantage of being able to learn all the weight parameters of the base model 100 and having high performance, but has the disadvantage of requiring astronomical costs for learning for a specific task due to the characteristics of the base model 100, which generally has a large number of weight parameters.

[0004] 1(b) and (c) are diagrams showing a learning model in which some layers of the base model 100 are frozen to solve the problem of the cost required in the Full Fine-Tuning method, and the cost required for learning is reduced by not learning the frozen weight parameters. For example, (b) shows a state in which the weight parameters for the first layer 110 to the fourth layer 140 of the base model 100 are frozen, and (c) shows a state in which the weight parameters for the first layer 110, the third layer 130, and the fourth layer 140 of the base model 100 are frozen. In this case, the process of generating loss is performed in the same manner as (a), but in (b), only the weight parameters of the fifth layer 150 are updated through backpropagation, and in (c), the weight parameters of the fifth layer 150 and the second layer 120 are updated through backpropagation. Through this, only information for some layers needs to be stored in memory, which can reduce costs compared to the Full Fine-Tuning method.

[0005] However, the base model 100 performs learning to extract primitive features from learning data in the first layer (e.g., the first layer 110) and to perform advanced inference by combining the primitive features in the second layer (e.g., the fifth layer 150). However, if learning is not performed on the first layer as in (b), primitive features for a specific task cannot be generated, and inference must be performed only on primitive features for general tasks, resulting in a significant decrease in performance. In addition, if learning is performed on the first layer to minimize the performance decrease as in (c), not only must gradient operations be performed up to the first layer through backpropagation, but also information on the first layer must be stored in memory, resulting in a significant loss of cost benefits.

[0006] As another example, FIG. 1(d) shows a learning model in which a separate adapter layer is added while all layers of the base model 100 are frozen. In (d), by way of example, two adapter layers, a first adapter layer 210 and a second adapter layer 220, are connected to the outputs of the second layer 120 and the fourth layer 140 of the base model 100, and the outputs of the first adapter layer 210 and the second adapter layer 220 are fused through a predetermined fusion layer 300, and then task data is output through the output layer 400. Thereafter, the weight parameters are updated by backpropagating the loss generated by referring to the task data and the corresponding GT and performing gradient operations on the output layer 400, the first adapter layer 210, and the second adapter layer 220. Since the first layer 110 to the fifth layer 150 of the base model 100 are all frozen and the weight parameters are fixed, the base model 100 is not trained, and only the output layer 400, the first adapter layer 210, and the second adapter layer 220, which have relatively fewer weight parameters than the base model 100, are trained. This has the advantage of effectively reducing the cost required for training, but there is a problem in that it is difficult to improve performance for advanced inference for a specific task by only using and combining adapter layers.

[0007] Therefore, there is a need for an improved solution to the above problems. Patent Document 1 discloses a method for evaluating a text generation model using a pre-trained model, and Patent Document 2 discloses a method for fine-tuning parameters of a pre-trained language model and a pre-trained vocabulary book and an electronic device using the same. Patent Document 3 discloses a method and system for tuning a deep learning model, and Patent Document 4 discloses a trained evaluation model for evaluating the quality of natural language generation (NLG) output. [Prior art documents] [Patent documents]

[0008] [Patent Document 1] Korean Patent No. 10-2693112 [Patent Document 2] Korean Patent No. 10-2618219 [Patent Document 3] Patent No. 7371188 [Patent Document 4] US Patent Application Publication No. 2024 / 0012999 Summary of the Invention [Problem to be solved by the invention]

[0009] An object of the present invention is to solve all of the above problems.

[0010] Another object of the present invention is to construct a custom model including a pre-trained base model including at least one feature extraction block, at least one adaptation block, at least one residual unit, and a task head block, and to update at least some parameters of the task head block, at least one residual unit, and at least one adaptation block through backpropagation using task loss, without updating parameters of the base model.

[0011] Another object of the present invention is to learn to enable advanced inference by fusing at least one residual feature generated through at least one residual unit and at least one transformed feature converted from at least one intermediate feature generated from a base model through at least one adaptation block. [Means for solving the problem]

[0012] According to an embodiment of the present invention, a method for training a custom model based on a pre-trained base model includes: (a) a learning device inputs training data into a custom model for performing a specific task, the custom model including a pre-trained base model including first feature extraction block through n-th feature extraction block (where n is an integer equal to or greater than 2), first adaptation block through k-th adaptation block (where k is an integer equal to or greater than 2 and equal to or less than n), first residual unit through k-th residual unit, and a task head block, and generates first intermediate feature through n-th intermediate feature through each of the first feature extraction block through n-th feature extraction block of the base model, and generates first transformed feature through k-th transformed feature through each of the first adaptation block through k-th transformed feature by transforming a source domain corresponding to the base model in each of the first specific intermediate feature through the k-th specific intermediate feature selected from the first intermediate feature through the n-th intermediate feature into a target domain corresponding to the specific task through each of the first adaptation block through k adaptation block; (b) the learning device generates a first fusion feature by fusing a first residual feature extracted for the specific task with the first transformation feature using the first transformation feature through the first residual unit, and then generates an i-th tentative feature by fusing the (i-1)-th fusion feature with the i-th transformation feature (where i is an integer greater than or equal to 2 and less than or equal to the k), and generates an i-th fusion feature by fusing the i-th residual feature extracted for the specific task with the i-th tentative feature using the i-th tentative feature through the i-th residual unit, and generates a task output by performing a learning operation on the k-th fusion feature output from the k-th residual unit through the task head block;and (c) the learning device generates a task loss by referring to the task output and an original answer corresponding to the learning data, and backpropagates the task loss to train at least a part of the task head block, the first residual unit through the kth residual unit, and the first adaptation block through the kth adaptation block.

[0013] In one example, in step (a), the learning device selects the nth intermediate feature or the (n-1)th intermediate feature among the first intermediate feature through the nth intermediate feature through one of the first adaptation block through the kth adaptation block, and generates one of the transformed features by transforming the source domain to the target domain; and selects one of the second intermediate feature through the (n-2)th intermediate feature through another adaptation block among the first adaptation block through the kth adaptation block, and generates another transformed feature by transforming the source domain to the target domain.

[0014] In one example, in step (a), each of the first residual unit to the kth residual unit includes a plurality of convolution layers, and is configured to decompose a first filter included in at least a portion of the plurality of convolution layers into a plurality of second filters having a lower rank than the first filter.

[0015] In one example, in step (a), parameters of each of the first feature extraction block to the nth feature extraction block are fixed through freezing, and in step (c), the learning device prevents the parameters of each of the first feature extraction block to the nth feature extraction block from being updated when backpropagating the task loss.

[0016] In one example, in step (b), the learning device generates the first fusion feature by performing an add operation on the first residual feature and the first transformation feature through a first element-wise add layer to fuse them, generates the i-th tentative feature by performing an add operation on the i-th transformation feature and the (i-1)-th fusion feature through an (i-1)-th element-wise add layer to fuse them, and generates the i-th fusion feature by performing an add operation on the i-th residual feature and the i-th tentative feature through an element-wise add layer to fuse them.

[0017] According to another embodiment of the present invention, a method for learning a custom model based on a pre-trained base model includes: (a) a learning device inputs learning data to a custom model that performs a specific task, the custom model including a pre-trained base model including a first feature extraction block to an nth feature extraction block (where n is an integer equal to or greater than 1), an adaptation block, a residual unit, and a task head block, and generates first to nth intermediate features through the first to nth feature extraction blocks of the base model, respectively, and generates a transformation feature by transforming a source domain corresponding to the base model in a specific intermediate feature selected from the first to nth intermediate features through the adaptation block to a target domain corresponding to the specific task; (b) the learning device generates a fusion feature by fusing the transformation feature and a residual feature extracted for the specific task in the transformation feature through the residual unit, and generates a task output by performing a learning operation on the fusion feature through the task head block; and (c) the learning device generates a task loss by referring to the task output and an original answer corresponding to the learning data, and backpropagates the task loss to train at least a portion of the task head block, the residual unit, and the adaptation block.

[0018] In one example, in step (a), the learning device selects any one of the second intermediate feature to the (n-1)th intermediate feature as the specific intermediate feature through the adaptation block, and generates the transformed feature by transforming the source domain to the target domain.

[0019] In one example, in step (a), the residual unit includes a plurality of convolution layers, and is configured to decompose a first filter included in at least a portion of the plurality of convolution layers into a plurality of second filters having a lower rank than the first filter.

[0020] In one example, in step (a), parameters of each of the first feature extraction block to the nth feature extraction block are fixed through freezing, and in step (c), the learning device prevents the parameters of each of the first feature extraction block to the nth feature extraction block from being updated when backpropagating the task loss.

[0021] In one example, in step (b), the learning device generates the fusion feature by fusing the residual feature and the transformation feature through an element-wise add layer by performing an add operation.

[0022] According to another embodiment of the present invention, a learning device for learning a custom model based on a pre-trained base model includes at least one memory for storing instructions; and at least one processor configured to execute the instructions, the processor including: (I) a base model including pre-trained first feature extraction block through n-th feature extraction block (where n is an integer equal to or greater than 2), first adaptation block through k-th adaptation block (where k is an integer equal to or greater than 2 and equal to or less than n), first residual unit through k-th residual unit, and a task head block, the processor inputs learning data to a custom model performing a specific task, generates first intermediate features through the first feature extraction block through the n-th feature extraction block of the base model, respectively, and generates first intermediate features through the n-th intermediate features through the first adaptation block through the k-th adaptation block, respectively, corresponding to the base model in each of the first specific intermediate features through the k-th specific intermediate features selected from the first intermediate features through the n-th intermediate features. (II) a process for generating a first fusion feature by fusing a first residual feature extracted for the specific task with the first transformation feature using the first transformation feature through the first residual unit, and then fusing an (i-1)th fusion feature with an i-th transformation feature (where i is an integer greater than or equal to 2 and less than or equal to the k) to generate an i-th tentative feature, and generating an i-th fusion feature by fusing the i-th residual feature extracted for the specific task with the i-th tentative feature using the i-th tentative feature through the i-th residual unit, and generating a task output by performing a learning operation on the k-th fusion feature output from the k-th residual unit through the task head block;and (III) a process of generating a task loss by referring to the task output and an original answer corresponding to the learning data, and backpropagating the task loss to train at least a part of the task head block, the first residual unit through the kth residual unit, and the first adaptation block through the kth adaptation block.

[0023] In one example, in the (I) process, the processor selects the nth intermediate feature or the (n-1)th intermediate feature among the first intermediate feature through the nth intermediate feature through one of the first adaptation blocks through the kth adaptation blocks, and generates one of the transformed features by transforming the source domain to the target domain, and selects one of the second intermediate feature through the (n-2)th intermediate feature through another adaptation block among the first adaptation block through the kth adaptation block, and generates another transformed feature by transforming the source domain to the target domain.

[0024] In one example, in the process (I), each of the first residual unit to the kth residual unit includes a plurality of convolution layers, and is configured to decompose a first filter included in at least a portion of the plurality of convolution layers into a plurality of second filters having a lower rank than the first filter.

[0025] In one example, in the (I) process, the parameters of each of the first feature extraction block to the nth feature extraction block are fixed through freezing, and in the (III) process, the processor is configured not to update the parameters of each of the first feature extraction block to the nth feature extraction block when backpropagating the task loss.

[0026] In one example, in the (II) process, the processor generates the first fusion feature by performing an add operation to fuse the first residual feature and the first transformation feature through a first element-wise add layer, generates the i tentative feature by performing an add operation to fuse the i transformation feature and the (i-1) fusion feature through an (i-1)th element-wise add layer, and generates the i fusion feature by performing an add operation to fuse the i residual feature and the i tentative feature through an element-wise add layer.

[0027] According to yet another embodiment of the present invention, a learning device for learning a custom model based on a pre-trained base model includes at least one memory for storing instructions; and at least one processor configured to execute the instructions, the processor including: (I) a base model including pre-trained first feature extraction block to n-th feature extraction block (where n is an integer equal to or greater than 1), an adaptation block, a residual unit, and a task head block, and inputs learning data to a custom model performing a specific task, generates first intermediate features to n-th intermediate features through the first feature extraction block to n-th feature extraction block of the base model, respectively, and generates a source domain corresponding to the base model in a specific intermediate feature selected from the first intermediate features to the n-th intermediate features through the adaptation block. a process of generating transformed features by transforming the transformed features into a target domain corresponding to the specific task; (II) a process of generating fusion features by fusing residual features extracted for the specific task with the transformed features through the residual unit, and generating task outputs by performing a learning operation on the fusion features through the task head block; and (III) a process of generating task losses by referring to the task output and an original correct answer corresponding to the learning data, and backpropagating the task losses to train at least a portion of the task head block, the residual unit, and the adaptation block.

[0028] In one example, in the process (I), the processor selects any one of the second intermediate feature to the (n-1)th intermediate feature as the specific intermediate feature through the adaptation block, and generates the transformed feature by transforming the source domain to the target domain.

[0029] In one example, in the process (I), the residual unit includes a plurality of convolution layers, and is configured to decompose a first filter included in at least a portion of the plurality of convolution layers into a plurality of second filters having a lower rank than the first filter.

[0030] In one example, in the (I) process, the parameters of each of the first feature extraction block to the nth feature extraction block are fixed through freezing, and in the (III) process, the processor is configured not to update the parameters of each of the first feature extraction block to the nth feature extraction block when backpropagating the task loss.

[0031] In one example, in the (II) process, the processor generates the fusion feature by fusing the residual feature and the transformation feature through an element-wise add layer by performing an add operation. Effect of the Invention

[0032] The present invention has the effect of constructing a custom model including a pre-trained base model including at least one feature extraction block, at least one adaptation block, at least one residual unit, and a task head block, and updating at least some of the parameters of the task head block, at least one residual unit, and at least one adaptation block through backpropagation using task loss, while not updating the parameters of the base model.

[0033] In addition, the present invention has the effect of fusing at least one residual feature generated through at least one residual unit and at least one transformed feature converted from at least one intermediate feature generated from a base model through at least one adaptation block, thereby enabling learning to perform advanced inference. [Brief description of the drawings]

[0034] The following drawings, which are attached to be used for explaining the embodiments of the present invention, are merely a part of the embodiments of the present invention, and a person having ordinary knowledge in the technical field to which the present invention pertains (hereinafter, "ordinary engineers") can derive other drawings based on these drawings without performing any inventive work.

[0035] [Figure 1] FIG. 1 is a simplified diagram of a conventional learning model for performing a specific task based on a pre-trained base model. [Diagram 2] FIG. 2 is a simplified diagram of a learning device for learning a custom model based on a pre-trained base model of the present invention; [Diagram 3] FIG. 3 is a simplified diagram illustrating a process for training a custom model based on a pre-trained base model in accordance with one embodiment of the present invention; [Figure 4] FIG. 4 is a simplified diagram illustrating an example of a custom model including one adaptation block and one residual unit in accordance with an embodiment of the present invention; [Diagram 5] FIG. 5 is a simplified diagram illustrating a process of training a custom model based on a pre-trained base model in another embodiment of the present invention; [Figure 6] FIG. 6 is a simplified diagram illustrating an example of a custom model including two adaptation blocks and two residual units in another embodiment of the present invention; [Figure 7]FIG. 7 is a diagram illustrating an example of a custom model including a maximum number of adaptation blocks and a maximum number of residual units in another embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0036] The following detailed description of the present invention will be made with reference to the accompanying drawings, which show, by way of example, specific embodiments in which the present invention may be practiced, in order to clarify the objects, technical solutions, and advantages of the present invention. These embodiments are described in sufficient detail to enable those of ordinary skill in the art to practice the present invention.

[0037] Furthermore, throughout the detailed description of the present invention and the claims, the word "comprises" and variations thereof are not intended to exclude other technical features, additions, components or steps. Other objects, advantages and characteristics of the present invention will become apparent to those of ordinary skill in the art in part from the description and in part from the practice of the present invention. The following examples and drawings are provided as illustrations and are not intended to limit the present invention.

[0038] Moreover, the present invention covers all possible combinations of the embodiments shown herein. It should be understood that the various embodiments of the present invention are different from one another but are not necessarily mutually exclusive. For example, a particular shape, structure, and characteristic described herein may be embodied in another embodiment without departing from the spirit and scope of the present invention. It should also be understood that the location or arrangement of individual components within each disclosed embodiment may be changed without departing from the spirit and scope of the present invention. Therefore, the following detailed description is not intended to be taken in a limiting sense, and the scope of the present invention is limited only by the appended claims with the full scope of equivalents that such claims are entitled to, if properly interpreted. In the drawings, like reference numerals refer to the same or similar functions across the various aspects.

[0039] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings in order to enable those skilled in the art to easily practice the present invention.

[0040] FIG. 2 is a simplified diagram of a learning device for learning a custom model based on a pre-trained base model of the present invention.

[0041] 2, the learning device 500 may include a memory 510 that stores instructions for learning a custom model based on a pre-trained base model, and a processor 520 that performs an operation for learning a custom model based on a pre-trained base model in response to the instructions stored in the memory 510. In this case, the learning device 500 may include various computing devices such as a server, a personal computer (PC), a notebook, a workstation, a tablet, a mobile computer, a PDA / EDA, a mobile phone, a smartphone, an IOT device, and the like.

[0042] Specifically, the learning device 500 may typically utilize a combination of computing devices (e.g., devices that may include computer processors, memory, storage, input and output devices, and other components of existing computing devices; electronic communication devices such as routers, switches, etc.; electronic information storage systems such as network attached storage (NAS) and storage area networks (SAN)) and computer software (i.e., instructions that cause a computing device to function in a particular manner) to achieve the desired system performance.

[0043] The processor of the computing device may include hardware components such as a micro processing unit (MPU) or central processing unit (CPU), cache memory, data bus, etc. The computing device may further include software components such as an operating system and applications that perform specific purposes.

[0044] However, this does not exclude the case where the computing device includes an integrated processor in which the medium, processor and memory for implementing the present invention are integrated.

[0045] FIG. 3 is a simplified diagram illustrating the process of training a custom model based on a pre-trained base model in accordance with one embodiment of the present invention.

[0046] First, the learning device 500 may be configured with a base model including pre-trained first to nth feature extraction blocks, an adaptation block, a residual unit, and a task head block, input learning data to a custom model performing a specific task, generate first to nth intermediate features through the first to nth feature extraction blocks of the base model, respectively, and generate a conversion feature (S10_1) by converting a source domain corresponding to the base model in a specific intermediate feature selected from the first to nth intermediate features into a target domain corresponding to the specific task through the adaptation block. In this case, n may be an integer of 1 or more, but generally, n may be 2 or more.

[0047] For reference, Figures 3 and 4 describe a custom model including one adaptation block and one residual unit, which is a minimized configuration that can be implemented as one embodiment, and Figures 5 to 7 describe a custom model including multiple adaptation blocks and multiple residual units as another embodiment so as to learn more advanced inference.

[0048] For example, the base model can be GroundingDINO, which is pre-trained using various learning datasets such as MSCOCO, Object365, and OpenImages, but any other model that has been pre-trained using a large number of learning datasets can also be used as the base model.

[0049] In addition, when learning data is input to the base model, all of the first to nth intermediate features are generated in each of the first feature extraction block to the nth feature extraction block of the base model. However, since this embodiment is an embodiment that uses only one adaptation block, a specific intermediate feature from the first to nth intermediate features is selected and input to the adaptation block, and Down-Projection and Up-Projection are sequentially applied to the specific intermediate feature through the adaptation block to convert the source domain to the target domain, thereby generating a transformed feature.

[0050] The learning device 500 can then generate fusion features by fusing the residual features extracted for a specific task with the transformation features through the residual unit, and generate task outputs (S20_1) by performing learning calculations on the fusion features through the task head block.

[0051] That is, when a transformation feature is input to a residual unit, a learning operation is performed through a plurality of convolution layers included in the residual unit to generate a residual feature, and a fusion feature can be generated by fusing a transformation feature transferred through a skip connection in the residual unit with the residual feature. In addition, when fusing between features, fusion can be performed through a separate fusion layer capable of an add operation, a concatenation operation, a weighted sum operation, etc., but is not limited thereto.

[0052] Thereafter, the learning device 500 generates a task loss by referring to the task output and the original answer corresponding to the learning data, and backpropagates the task loss to learn at least a part of the task head block, the residual unit, and the adaptation block (S30_1).

[0053] The process of training a custom model including one adaptation block and one residual unit will now be described in more detail with reference to FIG.

[0054] FIG. 4 is a simplified diagram illustrating an example of a custom model including one adaptation block and one residual unit in one embodiment of the present invention.

[0055] Here, both (a) and (b) of Figure 4 are drawings showing an example of the structure of a custom model 1000 including a base model 1100, one adaptation block 1200, one residual unit 1300, and a task head block 1400. The process of learning the custom model 1000 will be described with reference to (a), and then the differences between (a) and (b) will be described. For reference, the base model included in the custom model exemplarily described in Figures 4, 6, and 7 is illustrated as including five feature extraction blocks, but in reality, this can be expanded so that the base model includes n feature extraction blocks.

[0056] First, the learning device 500 inputs learning data in the form of image data to a pre-trained base model 1100 to perform a specific task, such as defect detection of a circuit board, and sequentially performs learning operations on the learning data input by the base model 1100 through a first feature extraction block 1110 to a fifth feature extraction block 1150, so that a first intermediate feature to a fifth intermediate feature can be generated through each of the first feature extraction block 1110 to the fifth feature extraction block 1150. In this case, each of the first feature extraction block 1110 to the fifth feature extraction block 1150 may be in a frozen state in advance to prevent parameters from being updated when performing backpropagation through a gradient operation later. Also, in FIG. 4, the base model 1100 is shown to include five feature extraction blocks for convenience, but if the base model 1100 has n feature extraction blocks, the first intermediate feature to the nth intermediate feature can be generated through each of the first feature extraction block to the nth feature extraction block.

[0057] Moreover, since Figures 4(a) and (b) commonly include one adaptation block 1200, the learning device 500 can select one of the first to fifth intermediate features as a specific intermediate feature and input it to the adaptation block 1200, and by sequentially performing up-projection and down-projection operations using the specific intermediate feature through the adaptation block 1200, a transformed feature can be generated in which a source domain corresponding to the base model 1100 in the specific intermediate feature is transformed into a target domain corresponding to a specific task (i.e., a domain corresponding to the task head block 1400). In this case, when the learning device 500 selects one of the first feature extraction block 1110 to the fifth feature extraction block 1150 as a specific intermediate feature, the first intermediate feature generated through the first feature extraction block 1110 may not be selected as a specific intermediate feature because it is more likely to be a primitive feature having general knowledge rather than a feature extracted for a specific task, and the fifth intermediate feature generated through the fifth feature block 1150 may not be selected as a specific intermediate feature because it is similar to the structure of the model described in Figure 1 (b) and has little practical benefit.

[0058] In other words, when the base model 1100 includes the first feature extraction block to the nth feature extraction block, the learning device 500 can select one of the second intermediate feature to the (n-1)th intermediate feature generated from the second feature extraction block to the (n-1)th feature extraction block as a specific intermediate feature, and input the specific intermediate feature to the adaptation block 1200.

[0059] In addition, the learning device 500 inputs the transformation features generated through the adaptation block 1200 to the residual unit 1300, generates residual features through the residual block 1310 included in the residual unit 1300, and then inputs them to the fusion layer 1320. At the same time, the transformation features generated through the adaptation block 1200 can be input to the fusion layer 1320 included in the residual unit 1300 using a skip structure included in the residual unit 1300, and fusion features can be generated by fusing the residual features and the transformation features through the fusion layer 1320. In this case, the residual block 1310 may include a plurality of convolution layers, and the residual feature may be generated by sequentially performing convolution operations through a plurality of convolution layers on the transformed feature, and the fusion layer 1320 may apply an add operation, a concatenation operation, a weighted sum operation, etc. as a method of fusion between features. In FIG. 4, it can be seen that an add operation is performed using the element-wise add layer 1320 as the fusion layer.

[0060] Meanwhile, when additional cost savings are required in training the custom model 1000, the rank of the residual unit 1300, more precisely, the rank of at least some of the multiple convolution layers included in the residual block 1310, can be configured as low-rank to reduce the number of parameters required for training. Specifically, a first filter in a full-rank state included in each of the multiple convolution layers is configured to be decomposed into multiple second filters having a lower rank than the first filter. For example, when the first filter is a d×k matrix, it is decomposed into two second filters having a d×r matrix and an r×k matrix, where r is a rank value. For example, when the matrix has d=10 and k=10 and the rank is r=3, the first filter is 10×10 and needs to be trained for 100 parameters, whereas the second filter is 10×3 and 3×10 and needs to be trained for a total of 60 parameters. Therefore, by implementing the residual block 1310 in a low rank, it is possible to reduce the cost required for training the custom model 1000. Although only the case where the first filter is in the form of a 2D matrix has been described above, when the first filter is in the form of a 3D tensor, it can be implemented in a low rank by decomposing it into a plurality of second filters in the same manner. Of course, if necessary, the adaptation block 1200 can also be configured in a low rank to obtain further cost reduction effects.

[0061] Thereafter, the learning device 500 can input the fusion features to the task head block 1400 and generate a task output using the task head block 1400. For example, if a specific task is related to image classification, the task head block 1400 can be configured with a fully-connected (FC) layer, and if a specific task is related to object detection, the task head block 1400 can be configured with a pooling layer and an FC layer.

[0062] The learning device 500 can generate a task loss by referring to the task output and an original ground truth (GT) corresponding to the learning data, and can learn to update at least some parameters of the task head block 1400, the residual unit 1300, and the adaptation block 1200 by backpropagating (the direction is shown by a dotted line) the task loss. Here, since the base model 1100 has parameters of the first feature extraction block to the nth feature extraction block (the first feature extraction block 1110 to the fifth feature extraction block 1150 in FIG. 4) fixed in advance through freezing, the first feature extraction block to the nth feature extraction block of the base model 1100 can be subjected to a gradient operation to prevent the parameters from being updated. Therefore, since the base model 1100 is not trained, the cost required for training is reduced compared to the conventional full fine-tuning method, while at the same time, by fusing the features generated from the adaptation block 1200 and the residual unit 1300, more advanced inference is possible, thereby improving the learning performance for performing specific tasks.

[0063] Meanwhile, (a) and (b) of FIG. 4 are the same in the basic structure of the custom model 1000, but (a) is different in that the fourth intermediate feature generated through the fourth feature extraction block 1140 of the base model 1100 is selected as a specific intermediate feature and input to the adaptation block 1200, and (b) is different in that the fifth intermediate feature generated through the fifth feature extraction block 1150 of the base model 1100 is selected as a specific intermediate feature and input to the adaptation block 1200. That is, in (b), the output of the last feature extraction block of the base model 1100 is input to the adaptation block 1200, but in this case, since it is similar to the model structure described in (b) of FIG. 1 above and the actual gain that can be obtained may not be large, when implementing the custom model 1000 with a minimum configuration, it is more preferable to use the output of one of the feature extraction blocks remaining except for the first feature extraction block and the last feature extraction block as in (a).

[0064] The above describes an embodiment in which a custom model is embodied with a minimum configuration. Next, a configuration of a custom model including a plurality of adaptation blocks and a plurality of residual units will be described with reference to FIGS. 5 to 7.

[0065] FIG. 5 is a simplified diagram illustrating a process of training a custom model based on a pre-trained base model in another embodiment of the present invention.

[0066] First, the learning device 500 may input learning data to a custom model including a pre-trained first feature extraction block to an n-th feature extraction block, a first adaptation block to a k-th adaptation block, a first residual unit to a k-th residual unit, and a task head block, and generate first to n-th intermediate features through the first feature extraction block to the n-th feature extraction block of the base model, respectively, and generate first to k-th transformed features S10_2 by transforming a source domain corresponding to the base model in the first to k-th specific intermediate features selected from the first to n-th intermediate features through the first adaptation block to the k-th adaptation block, respectively. In this case, n may be an integer of 2 or more, and k may be an integer of 2 or more and n or less.

[0067] For reference, the custom models shown in Figures 6 and 7 are different embodiments from the custom models shown in Figures 3 and 4, and Figure 6 describes a custom model including two adaptation blocks and two residual units, and Figure 7 describes a custom model including the maximum number of adaptation blocks and the maximum number of residual units to learn the most advanced inference.

[0068] In addition, when learning data is input to the base model, all of the first to n-th intermediate features are generated in the first to n-th feature extraction blocks of the base model, respectively. However, since the number of adaptation blocks is k, i.e., the first to k-th adaptation blocks, the learning device 500 selects the first to k-th specific intermediate features as k specific intermediate features from the first to n-th intermediate features and inputs them to the first to k-th adaptation blocks, respectively. Down-Projection and Up-Projection are sequentially applied to the first to k-th specific intermediate features through the first to k-th adaptation blocks, respectively, to convert the source domain of the base model corresponding to each of the first to k-th specific intermediate features into the target domain, thereby generating the first to k-th transformed features, respectively.

[0069] The learning device 500 may generate a first fusion feature by fusing a first residual feature extracted for a specific task with the first conversion feature through the first residual unit, and then generate an i-th tentative feature by fusing the (i-1)-th fusion feature with the i-th conversion feature through the i-th residual unit, and repeat the process of generating an i-th fusion feature by fusing the i-th residual feature extracted for a specific task with the i-th tentative feature through the i-th residual unit, and generate a task output by performing a learning operation on the k-th fusion feature output from the k-th residual unit through the task head block (S20_2). In this case, i may be an integer greater than or equal to 2 and less than or equal to k.

[0070] To repeat, the learning device 500 inputs the first transformation feature to the first residual unit and performs a learning operation through a plurality of convolution layers included in the first residual unit to generate the first residual feature, and fuses the first transformation feature transferred through a skip connection in the first residual unit with the first residual feature to generate the first fusion feature. That is, the process up to the generation of the first fusion feature can be performed in the same manner as in FIG. 4.

[0071] The learning process of the second residual unit to the k-th residual unit is the same, and since i is an integer between 2 and k, when generalized as the learning process of the i-th residual unit, the learning device 500 first performs a process of generating an i-th tentative feature by fusing the i-th transformation feature and the (i-1)-th fusion feature through the (i-1)-th residual unit, and the i-th residual feature can be generated by performing a learning operation through a plurality of convolution layers included in the i-th residual unit through the i-th residual unit, and the i-th fusion feature can be generated by fusing the i-th tentative feature transferred through the skip connection in the i-th residual unit with the i-th residual feature. This can be repeatedly performed until the k-th fusion feature is finally output from the k-th residual unit. In this case, when fusion between features is performed, fusion is performed through a separate fusion layer capable of add, concatenation, weighted sum, etc., and the fusion layer performing fusion to generate the i-th tentative feature may be a fusion layer included in the (i-1)th residual unit, or may be a fusion layer separately added between the (i-1)th residual unit and the i-th residual unit, but is not limited thereto. However, for convenience of explanation, it is assumed that the fusion layer performing fusion to generate the i-th tentative feature is a fusion layer included in the (i-1)th residual unit.

[0072] Thereafter, the learning device 500 generates a task loss by referring to the task output and the original answer corresponding to the learning data, and backpropagates the task loss to learn at least a part of the task head block, the residual unit, and the adaptation block (S30_2).

[0073] The process of training a custom model including at least two adaptation blocks and at least two residual units in this manner will now be described in more detail with reference to FIGS. 6 and 7. FIG.

[0074] FIG. 6 is a simplified diagram illustrating an example of a custom model including two adaptation blocks and two residual units in another embodiment of the present invention.

[0075] Here, both (a) and (b) of Figure 6 are drawings showing an example structure of a custom model 2000 including a base model 2100, a first adaptation block 2200_1, a second adaptation block 2200_2, a first residual unit 2300_1, a second residual unit 2300_2 and a task head block 2400. The process of learning the custom model 2000 will be described with reference to (a), and then the differences between (a) and (b) will be described.

[0076] First, the learning device 500 inputs learning data in the form of image data to a pre-trained base model 2100 to perform a specific task such as defect detection of a circuit board, and sequentially performs learning operations on the input learning data through the first feature extraction block 2110 to the fifth feature extraction block 2150 using the base model 2100, thereby generating first to fifth intermediate features through each of the first feature extraction block 2110 to the fifth feature extraction block 2150. In this case, each of the first feature extraction block 2110 to the fifth feature extraction block 2150 may be in a frozen state in advance to prevent parameters from being updated when performing backpropagation through a gradient operation later. Also, in FIG. 6, the base model 2100 is shown to include five feature extraction blocks for convenience, but if the base model has n feature extraction blocks, each of the first to nth intermediate features may be generated through each of the first to nth feature extraction blocks.

[0077] 6(a) and 6(b) commonly include the first adaptation block 2200_1 and the second adaptation block 2200_2, the learning device 500 may select any one of the first to fifth intermediate features as a first specific intermediate feature and input it to the first adaptation block 2200_1, and may select another one of the first to fifth intermediate features as a second specific intermediate feature and input it to the second adaptation block 2200_2. In this case, since the first intermediate feature generated through the first feature extraction block 2110 of the base model 2100 may include only excessively general knowledge, the learning device 500 may select the first specific intermediate feature and the second specific intermediate feature from the second to fifth intermediate features generated through the second feature extraction block 2120 to the fifth feature extraction block 2150 excluding the first feature extraction block 2110. In selecting the first specific intermediate feature and the second specific intermediate feature, the first specific intermediate feature is selected from intermediate features generated through a feature extraction block in the middle part of the base model 2100 (e.g., the second feature extraction block 2120 and the third feature extraction block 2130), and the second specific intermediate feature is selected from intermediate features generated through a feature extraction block in the latter part of the base model 2100 (e.g., the fourth feature extraction block 2140 and the fifth feature extraction block 2150). This makes it possible to reflect relatively universal knowledge and all knowledge applicable to a specific task, thereby enabling more advanced inference learning.

[0078] When this is extended to the base model 2100 including the first feature extraction block to the nth feature extraction block, the learning device 500 can select one of the second intermediate feature to the (n-2)th intermediate feature generated through the second feature extraction block to the (n-2)th feature extraction block as a first specific intermediate feature and input it to the first adaptation block 2200_1, and can select one of the (n-1)th intermediate feature and the nth intermediate feature generated through the (n-1)th feature extraction block and the nth feature extraction block as a second specific intermediate feature and input it to the second adaptation block 2200_2. For example, (a) may be an example in which the second intermediate feature is selected as the first specific intermediate feature and the fourth intermediate feature is selected as the second specific intermediate feature, and (b) may be an example in which the third intermediate feature is selected as the first specific intermediate feature and the fifth intermediate feature is selected as the second specific intermediate feature.

[0079] Thereafter, by sequentially performing up-projection and down-projection operations using a first specific intermediate feature through the first adaptation block 2200_1, a first transformed feature is generated in which a source domain corresponding to the base model 2100 in the first specific intermediate feature is transformed into a target domain corresponding to a specific task (i.e., a domain corresponding to the task head block 2400), and by sequentially performing up-projection and down-projection operations using a second specific intermediate feature through the second adaptation block 2200_2, a second transformed feature is generated in which a source domain corresponding to the base model 2100 in the second specific intermediate feature is transformed into a target domain corresponding to a specific task (i.e., a domain corresponding to the task head block 2400).

[0080] In addition, the learning device 500 inputs the first conversion feature generated from the first adaptation block 2200_1 to the first residual unit 2300_1, generates the first residual feature through the first residual block 2310_1 included in the first residual unit 2300_1, and then inputs it to the first fusion layer 2320_1. At the same time, the first conversion feature generated through the first adaptation block 2200_1 can be input to the first fusion layer 2320_1 using a skip structure, and the first fusion feature can be generated by fusing the first residual feature and the first conversion feature through the first fusion layer 2320_1.

[0081] Then, the learning device 500 fuses the second conversion feature generated through the second adaptation block 2200_2 and the first fusion feature generated through the first residual unit 2300_1 to obtain a second tentative feature, inputs the second tentative feature to the second residual unit 2300_2, generates a second residual feature through the second residual block 2310_2 included in the second residual unit 2300_2, and then inputs it to the second fusion layer 2320_2. At the same time, the second tentative feature can be input to the second fusion layer 2320_2 using a skip structure, and the second fusion feature can be generated by fusing the second residual feature and the second tentative feature through the second fusion layer 2320_2. In this case, the second tentative feature can be obtained through the first fusion layer 2320_1, or an auxiliary fusion layer can be added between the first residual unit 2300_1 and the second residual unit 2300_2 so that the second tentative feature can be obtained through the auxiliary fusion layer.

[0082] The first residual unit 2300_1 and the second residual unit 2300_2, more specifically, the first residual block 2310_1 and the second residual block 2310_2, each include a plurality of convolution layers, and the first residual feature and the second residual feature may be generated by sequentially performing convolution operations through a plurality of convolution layers in the first residual block 2310_1 and the second residual block 2310_2, respectively. The first fusion layer 2320_1 and the second fusion layer 2320_2 may apply an add operation, a concatenation operation, a weighted sum operation, etc. as a method of fusion between features. In FIG. 6, it can be seen that an add operation is performed using the first element-wise add layer 2320_1 and the second element-wise add layer 2320_2 as the first fusion layer and the second fusion layer, respectively.

[0083] Meanwhile, when additional cost reduction is required in learning the custom model 2000, at least some ranks of the first residual unit 2300_1 and the second residual unit 2320_2, more precisely, at least some ranks of the plurality of convolution layers included in the first residual block 2310_1 and the second residual block 2310_2 may be configured as low-rank to reduce the number of parameters required for learning. Specifically, a first filter in a full-rank state included in each of the plurality of convolution layers may be configured to be decomposed into a plurality of second filters having a lower rank than the first filter. For example, when the first filter is a d×k matrix, it is decomposed into two second filters having a d×r matrix and an r×k matrix, where r is a rank value, and for example, when a matrix with d=5 and k=5 has a rank of r=1, the first filter is 5×5 and needs to be trained for 25 parameters, whereas the second filter is 5×1 and 1×5 and needs to be trained for a total of 10 parameters. Therefore, by implementing each of the first residual block 2310_1 and the second residual block 2310_2 in a low rank, it is possible to reduce the cost required for training the custom model 2000. Although only the case where the first filter is in the form of a 2D matrix has been described above, when the first filter is in the form of a 3D tensor, it can be implemented in a low rank by decomposing it into a plurality of second filters in the same manner. Of course, if necessary, the first adaptation block 2200_1 and the second adaptation block 2200_2 may also be configured with low ranks to obtain further cost reduction effects.

[0084] Thereafter, the learning device 500 can input the fusion features to the task head block 2400 and generate a task output using the task head block 2400. For example, if a specific task is related to image classification, the task head block 2400 can be configured with a fully-connected (FC) layer, and if a specific task is related to object detection, the task head block 2400 can be configured with a pooling layer and an FC layer.

[0085] The learning device 500 can generate a task loss by referring to the task output and an original answer (GT, Ground Truth) corresponding to the learning data, and can learn to update at least some parameters of the task head block 2400, the first residual unit 2300_1, the second residual unit 2300_2, the first adaptation block 2200_1, and the second adaptation block 2200_2 by backpropagating the task loss (the direction is shown by a dotted line). Here, since the base model 2100 has parameters of the first feature extraction block to the nth feature extraction block (the first feature extraction block 2110 to the fifth feature extraction block 2150 in FIG. 6) fixed in advance through freezing, the first feature extraction block to the nth feature extraction block of the base model 2100 can be subjected to a gradient operation to prevent the parameters from being updated. Therefore, since the base model 2100 is not trained, the cost required for training is reduced compared to the conventional full fine-tuning method, while at the same time, by fusing features generated from each of the first adaptation block 2200_1, the second adaptation block 2200_2, the first residual unit 2300_1, and the second residual unit 2300_2, more advanced inference is possible, thereby improving the learning performance for performing a specific task.

[0086] On the other hand, (a) and (b) of Figure 6 are identical in the structure of the basic custom model 2000, but differ in that (a) is configured such that the second and fourth intermediate features generated through the second feature extraction block 2120 and the fourth feature extraction block 2140, respectively, of the base model 2100 are selected as the first and second specific intermediate features, respectively, and input to the first adaptation block 2200_1 and the second adaptation block 2200_2, and (b) is configured such that the third and fifth intermediate features generated through the third feature extraction block 2130 and the fifth feature extraction block 2150, respectively, of the base model 2100 are selected as the first and second specific intermediate features, respectively, and input to the first adaptation block 2200_1 and the second adaptation block 2200_2, respectively.

[0087] Finally, the process of learning the best performing custom model by further extending the custom model in FIG. 6 to include the maximum number of adaptation blocks and the maximum number of residual units will be described with reference to FIG.

[0088] FIG. 7 is a diagram illustrating an example of a custom model including a maximum number of adaptation blocks and a maximum number of residual units in another embodiment of the present invention.

[0089] For reference, the custom model 3000 shown in FIG. 7 is configured to include a base model 3100, a first adaptation block 3200_1 to a fifth adaptation block 3200_5, a first residual unit 3300_1 to a fifth residual unit 3300_5, and a task head block 3400. For convenience, an example is shown in which the base model 3100 includes a first feature extraction block 3110 to a fifth feature extraction block 3150. Accordingly, for the most advanced inference learning, the custom model 3000 is configured to include the first adaptation block 3200_1 to the fifth adaptation block 3200_5, the first residual unit 3300_1 to the fifth residual unit 3300_5, and the task head block 3400. For example, if the base model 3100 includes a first feature extraction block through an nth feature extraction block, the custom model 3000 can be configured to include a maximum number of n adaptation blocks and residual units.

[0090] In this case, the learning device 500 inputs learning data in the form of image data to the pre-trained base model 3100 to perform a specific task such as defect detection of a circuit board, and sequentially performs learning operations on the input learning data through the first feature extraction block 3110 to the fifth feature extraction block 3150 using the base model 3100, thereby allowing the first intermediate feature to the fifth intermediate feature to be generated through each of the first feature extraction block 3110 to the fifth feature extraction block 3150. In this case, each of the first feature extraction block 3110 to the fifth feature extraction block 3150 may be in a frozen state in advance to prevent parameters from being updated when performing backpropagation through a gradient operation later. If this is expanded to include n feature extraction blocks, the first intermediate feature to the nth intermediate feature may be generated through each of the first feature extraction block to the nth feature extraction block.

[0091] Then, the learning device 500 can input the first to fifth intermediate features to the first to fifth adaptation blocks 3200_1 to 3200_5, respectively. If this is expanded assuming that the base model 3100 has n feature extraction blocks, the number of adaptation blocks is configured as the first to n-th adaptation blocks, the same as the number of feature extraction blocks, and the learning device 500 can input the first to n-th intermediate features to the first to n-th adaptation blocks, respectively.

[0092] Thereafter, by sequentially performing up-projection and down-projection operations using the first to fifth intermediate features through the first adaptation block 3200_1 to the fifth adaptation block 3200_5, respectively, a first transformed feature to a fifth transformed feature can be generated by transforming a source domain corresponding to the base model 3100 in each of the first to fifth intermediate features into a target domain corresponding to a specific task (i.e., a domain corresponding to the task head block 3400). If this is expanded to the base model 3100 having n feature extraction blocks, a first transformed feature to an nth transformed feature can be generated by transforming a source domain corresponding to the base model in each of the first to nth intermediate features into a target domain corresponding to a specific task through the first adaptation block to the nth adaptation block, respectively.

[0093] In addition, the learning device 500 inputs the first conversion feature generated from the first adaptation block 3200_1 to the first residual unit 3300_1, generates the first residual feature through the first residual block 3310_1 included in the first residual unit 3300_1, and then inputs it to the first fusion layer 3320_1. At the same time, the first conversion feature generated through the first adaptation block 3200_1 can be input to the first fusion layer 3320_1 using a skip structure, and the first fusion feature can be generated by fusing the first residual feature and the first conversion feature through the first fusion layer 3320_1.

[0094] Then, the learning device 500 fuses the second conversion feature generated through the second adaptation block 3200_2 and the first fusion feature generated through the first residual unit 3300_1 to obtain a second tentative feature, inputs the second tentative feature to the second residual unit 3300_2, generates a second residual feature through the second residual block 3310_2 included in the second residual unit 3300_2, and then inputs it to the second fusion layer 3320_2. At the same time, the second tentative feature can be input to the second fusion layer 3330_2 using a skip structure, and the second fusion feature can be generated by fusing the second residual feature and the second tentative feature through the second fusion layer 3320_2. In this case, the second tentative feature can be obtained through the first fusion layer 3320_1, or an auxiliary fusion layer can be added between the first residual unit 3300_1 and the second residual unit 3300_2 so that the second tentative feature can be obtained through the auxiliary fusion layer.

[0095] Thereafter, the learning device 500 can generate the third to fifth fusion features by repeatedly performing the same process of generating the second fusion feature through the second residual unit 3300_2 for the third residual unit 3300_3 to the fifth residual unit 3300_5. If this is expanded to the case where the base model 3100 has n feature extraction blocks, the number of the residual units is the same as the number of the adaptation blocks, that is, n, and the operation process of the second to nth residual units is repeated in the same manner. A generalized expression of this is the same as the S20_2 process in FIG. 5, but is performed until i becomes n. Of course, since i is maximum k in the S20_2 process in FIG. 5, but k is maximum n in the S20_1 process, i can be regarded as an integer ranging from 2 to maximum n.

[0096] Each of the first residual unit 3300_1 to the fifth residual unit 3300_5, more specifically, the first residual block 3310_1 to the fifth residual block 3310_5 includes a plurality of convolution layers, and the first residual feature to the fifth residual feature may be generated by sequentially performing convolution operations through a plurality of convolution layers in each of the first residual block 3310_1 to the fifth residual block 3310_5. Each of the first fusion layer 3320_1 to the fifth fusion layer 3320_5 may apply an add operation, a concatenation operation, a weighted sum operation, etc. as a method of fusion between features. In FIG. 7, it can be seen that an add operation is performed using each of the first element-wise add layer 3320_1 to the fifth element-wise add layer 3320_5 as each of the first fusion layer to the fifth fusion layer. Similarly, it can be extended to the first residual unit to the nth residual unit, and a generalized expression is explained in S20_2 of FIG. 5, so please refer to this.

[0097] Meanwhile, when additional cost reduction is required in learning the custom model 3000, at least some ranks of the first residual unit 3300_1 to the fifth residual unit 3300_5, more precisely, at least some ranks of the plurality of convolution layers included in the first residual block 3310_1 to the fifth residual block 3310_5 are configured as low-rank, thereby reducing the number of parameters required for learning. When this is extended to the first residual block to the nth residual block, at least some ranks of the plurality of convolution layers included in each of the first residual block to the nth residual block of the first residual unit to the nth residual unit are configured as low-rank, and a first filter in a full-rank state included in each of the plurality of convolution layers is configured to be decomposed into a plurality of second filters having a lower rank than the first filter. Of course, depending on the needs of those skilled in the art, further cost savings can be achieved by configuring the first adaptation block 3200_1 to the fifth adaptation block 3200_5 (if expanded to the maximum number, the first adaptation block to the nth adaptation block) as low-rank.

[0098] Thereafter, the learning device 500 can input the fusion feature to the task head block 3400 and generate a task output using the task head block 3400. The learning device 500 can generate a task loss by referring to the task output and the original answer (GT, Ground Truth) corresponding to the learning data, and can backpropagate the task loss (the direction is shown by a dotted line) to learn to update at least some parameters of the task head block 3400, the first residual unit 3300_1 to the fifth residual unit 3300_5, and the first adaptation block 3200_1 to the third adaptation block 3200_5. By expanding this, it is possible to learn to update at least some parameters of the task head block, the first residual unit to the nth residual unit, and the first adaptation block to the nth adaptation block.

[0099] Here, since the parameters of the first feature extraction block 3110 to the fifth feature extraction block 3150 (the first feature extraction block to the n-th feature extraction block when expanded) of the base model 3100 are fixed in advance through freezing, the first feature extraction block 3110 to the fifth feature extraction block 3150 of the base model 3100 may not be updated by performing a gradient operation. Therefore, since the base model 3100 does not learn, the cost required for learning is reduced compared to the conventional full fine-tuning method, and the features generated from each of the first adaptation block 3200_1 to the fifth adaptation block 3200_5 and the first residual unit 3300_1 to the third residual unit 3300_5 are fused to enable more advanced inference than the custom model 2000 described in FIG. 6, thereby improving the learning performance for performing a specific task.

[0100] In addition, the above-described embodiments of the present invention may be embodied in the form of program instructions that can be executed by various computer components and stored in a computer-readable recording medium. The computer-readable recording medium may include program instructions, data files, data structures, and the like, alone or in combination. The program instructions stored in the computer-readable recording medium may be those specially designed and constructed for the present invention, or may be those known and available to those skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical recording media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and hardware devices specially configured to store and execute program instructions, such as ROMs, RAMs, flash memories, and the like. Examples of program instructions include not only machine code, such as that produced by a compiler, but also high-level language code executed by a computer using an interpreter, and the like. The hardware devices may be configured to operate as one or more software modules to execute the processes of the present invention, and vice versa.

[0101] Although the present invention has been described above using specific details such as specific components and limited examples and drawings, this is provided merely to aid in a more general understanding of the present invention, and the present invention is not limited to the above examples. Various modifications and variations may be made from such descriptions by those having ordinary knowledge in the technical field to which the present invention pertains.

[0102] Therefore, the concept of the present invention should not be limited to the embodiments described above, and all modifications equivalent to or equivalent to the scope of the claims, as well as the scope of the claims described below, are within the scope of the concept of the present invention.

Claims

1. A method for training a custom model based on a pre-trained base model, comprising: (a) a learning device inputs learning data into a custom model for performing a specific task, the custom model including a pre-trained first feature extraction block through an n-th feature extraction block (where n is an integer equal to or greater than 2), a first adaptation block through a k-th adaptation block (where k is an integer equal to or greater than 2 and equal to or less than n), a first residual unit through a k-th residual unit, and a task head block, and generates first intermediate features through the first feature extraction block through the n-th feature extraction block of the base model, respectively, and generates first transformed features through the k-th transformed features by transforming a source domain corresponding to the base model in a first specific intermediate feature through a k-th specific intermediate feature selected from the first intermediate feature through the n-th intermediate feature through the first adaptation block through the k adaptation block, respectively; (b) the learning device generates a first fusion feature by fusing a first residual feature extracted for the specific task with the first transformation feature using the first transformation feature through the first residual unit, and then fuses an (i-1)th fusion feature with an i-th transformation feature (where i is an integer greater than or equal to 2 and less than or equal to the k) to generate an i-th tentative feature, the learning device generates an i-th fusion feature by fusing the i-th residual feature extracted for the specific task with the i-th tentative feature through the i-th residual unit, and generates a task output by performing a learning operation on the k-th fusion feature output from the k-th residual unit through the task head block; and (c) generating a task loss by referring to the task output and an original answer corresponding to the learning data, and backpropagating the task loss to train at least a part of the task head block, the first residual unit to the kth residual unit, and the first adaptation block to the kth adaptation block; The method includes:

2. In the step (a), 2. The method of claim 1, wherein the learning device generates one converted feature by converting the source domain to the target domain by selecting the nth intermediate feature or the (n-1)th intermediate feature among the first intermediate feature through the kth adaptation block, where n is an integer greater than or equal to 4, and generating one converted feature by converting the source domain to the target domain by selecting one of the second intermediate feature through the (n-2)th intermediate feature through another adaptation block among the first adaptation block through the kth adaptation block, where n is an integer greater than or equal to 4.

3. In the step (a), 2. The method of claim 1, wherein each of the first residual unit to the kth residual unit includes a plurality of convolution layers, and a first filter included in at least a portion of the plurality of convolution layers is decomposed into a plurality of second filters having a lower rank than the first filter.

4. In the step (a), Each of the first feature extraction block to the nth feature extraction block has a parameter fixed through freezing, In the step (c), 2. The method of claim 1, wherein the learning device does not update the parameters of each of the first feature extraction block through the nth feature extraction block when backpropagating the task loss.

5. In the step (b), The method according to claim 1, characterized in that the learning device generates the first fusion feature by fusing the first residual feature and the first transformation feature through an add operation through a first element-wise add layer, generates the i tentative feature by fusing the i transformation feature and the (i-1) fusion feature through an (i-1)th element-wise add layer, and generates the i fusion feature by fusing the i residual feature and the i tentative feature through an i element-wise add layer.

6. A method for training a custom model based on a pre-trained base model, comprising: (a) inputting learning data into a custom model for performing a specific task, the custom model including a base model including pre-trained first feature extraction block to n-th feature extraction block (where n is an integer equal to or greater than 1), an adaptation block, a residual unit, and a task head block, the learning device generating first to n-th intermediate features through the first to n-th feature extraction blocks of the base model, respectively, and generating a transformed feature by transforming a source domain corresponding to the base model in a specific intermediate feature selected from the first to n-th intermediate features through the adaptation block into a target domain corresponding to the specific task; (b) generating fusion features by fusing the residual features extracted for the specific task with the transformation features through the residual unit, and generating task outputs by performing learning operations on the fusion features through the task head block; and (c) generating a task loss by referring to the task output and an original answer corresponding to the learning data, and backpropagating the task loss to train at least a part of the task head block, the residual unit, and the adaptation block; Including, In the step (a), The learning device is characterized in that, when n is an integer equal to or greater than 3, any one of the second intermediate feature to the (n-1)th intermediate feature is selected as the specific intermediate feature through the adaptation block, and the source domain is transformed into the target domain to generate the transformed feature. method.

7. A method for training a custom model based on a pre-trained base model, comprising: (a) inputting learning data into a custom model for performing a specific task, the custom model including a base model including pre-trained first feature extraction block to n-th feature extraction block (where n is an integer equal to or greater than 1), an adaptation block, a residual unit, and a task head block, the learning device generating first to n-th intermediate features through the first to n-th feature extraction blocks of the base model, respectively, and generating a transformed feature by transforming a source domain corresponding to the base model in a specific intermediate feature selected from the first to n-th intermediate features through the adaptation block into a target domain corresponding to the specific task; (b) generating fusion features by fusing the residual features extracted for the specific task with the transformation features through the residual unit, and generating task outputs by performing learning operations on the fusion features through the task head block; and (c) generating a task loss by referring to the task output and an original answer corresponding to the learning data, and backpropagating the task loss to train at least a part of the task head block, the residual unit, and the adaptation block; Including, In the step (a), The residual unit includes a plurality of convolution layers, and is configured to decompose a first filter included in at least a part of the plurality of convolution layers into a plurality of second filters having a lower rank than the first filter. method.

8. In the step (a), Each of the first feature extraction block to the nth feature extraction block has a parameter fixed through freezing, In the step (c), 8. The method according to claim 6 or 7, characterized in that the learning device does not update the parameters of each of the first feature extraction block to the nth feature extraction block when backpropagating the task loss.

9. In the step (b), The method according to claim 6 or 7, characterized in that the learning device generates the fusion feature by fusing the residual feature and the transformation feature through an element-wise add layer by performing an add operation.

10. In a learning device for learning a custom model based on a pre-trained base model, at least one memory for storing instructions; and at least one processor configured to execute said instructions; The processor (I) is composed of a base model including pre-trained first feature extraction block through n-th feature extraction block (where n is an integer equal to or greater than 2), first adaptation block through k-th adaptation block (where k is an integer equal to or greater than 2 and equal to or less than n), first residual unit through k-th residual unit, and a task head block, and inputs learning data into a custom model performing a specific task, generates first intermediate features through the first feature extraction block through the n-th feature extraction block of the base model, respectively, and converts a source domain corresponding to the base model in a first specific intermediate feature through a k-th specific intermediate feature selected from the first intermediate feature through the n-th intermediate feature, respectively, into a source domain corresponding to the specific task through the first adaptation block through the k-th adaptation block, respectively. (II) a process for generating a first fusion feature by fusing a first residual feature extracted for the specific task with the first transformation feature in the first transformation feature through the first residual unit, and then fusing an (i-1)th fusion feature with an i-th transformation feature (where i is an integer greater than or equal to 2 and less than or equal to the k) to generate an i-th tentative feature, generating an i-th fusion feature by fusing the i-th residual feature extracted for the specific task with the i-th tentative feature through the i-th residual unit, and generating a task output by performing a learning operation on the k-th fusion feature output from the k-th residual unit through the task head block;and (III) a process of generating a task loss by referring to the task output and an original correct answer corresponding to the learning data, and backpropagating the task loss to train at least a part of the task head block, the first residual unit through the kth residual unit, and the first adaptation block through the kth adaptation block.

11. In the process (I), the processor selects the nth intermediate feature or the (n-1)th intermediate feature from among the first intermediate feature through the nth intermediate feature through one of the first adaptation block through the kth adaptation block, where n is an integer equal to or greater than 4, and converts the source domain into the target domain, 11. The learning device of claim 10, further comprising: a training block for training a training feature that generates one of the training feature and the kth training feature; a training feature that is one of the second intermediate feature through the (n-2)th intermediate feature is selected through another adaptation block among the first adaptation block through the kth adaptation block; and another training feature is generated by transforming the source domain to the target domain.

12. In the process (I), The learning device according to claim 10, characterized in that each of the first residual unit to the kth residual unit includes a plurality of convolution layers, and is configured to decompose a first filter included in at least a portion of the plurality of convolution layers into a plurality of second filters having a lower rank than the first filter.

13. In the process (I), Each of the first feature extraction block to the nth feature extraction block has a parameter fixed through freezing, In the process (III), 11. The learning device according to claim 10, wherein the processor is configured not to update the parameters of each of the first feature extraction block to the nth feature extraction block when backpropagating the task loss.

14. In the process (II), The learning device according to claim 10, wherein the processor generates the first fusion feature by fusing the first residual feature and the first transformation feature through an add operation through a first element-wise add layer, generates the i tentative feature by fusing the i transformation feature and the (i-1) fusion feature through an (i-1)th element-wise add layer, and generates the i fusion feature by fusing the i residual feature and the i tentative feature through an i element-wise add layer.

15. A learning device that learns a custom model based on a pre-trained base model, At least one memory storing instructions; and at least one processor configured to execute said instructions; The processor includes: (I) a process of inputting learning data to a custom model that performs a specific task, the custom model including a base model including a pre-trained first feature extraction block to an nth feature extraction block (where n is an integer equal to or greater than 1), an adaptation block, a residual unit, and a task head block, generating first to nth intermediate features through the first to nth feature extraction blocks of the base model, respectively, and generating a transformation feature by transforming a source domain corresponding to the base model in a specific intermediate feature selected from the first to nth intermediate features to a target domain corresponding to the specific task through the adaptation block; (II) a process of generating a fusion feature by fusing a residual feature extracted for the specific task and the transformation feature through the residual unit in the transformation feature, and generating a task output by performing a learning operation on the fusion feature through the task head block; and (III) a process of generating a task loss by referring to the task output and an original answer corresponding to the learning data, and backpropagating the task loss to generate a task loss for the task head block. , a process of training at least a portion of the residual unit and the adaptation block; In the process (I), The processor is characterized in that, when n is an integer equal to or greater than 3, any one of the second intermediate feature to the (n-1)th intermediate feature is selected as the specific intermediate feature through the adaptation block, and the source domain is transformed into the target domain to generate the transformed feature. Learning device.

16. A learning device for learning a custom model based on a pre-trained base model, comprising: At least one memory storing instructions; and at least one processor configured to execute said instructions; The processor includes: (I) a process of inputting learning data to a custom model that performs a specific task, the custom model including a base model including a pre-trained first feature extraction block to an nth feature extraction block (where n is an integer equal to or greater than 1), an adaptation block, a residual unit, and a task head block, generating first to nth intermediate features through the first to nth feature extraction blocks of the base model, respectively, and generating a transformation feature by transforming a source domain corresponding to the base model in a specific intermediate feature selected from the first to nth intermediate features to a target domain corresponding to the specific task through the adaptation block; (II) a process of generating a fusion feature by fusing a residual feature extracted for the specific task and the transformation feature through the residual unit in the transformation feature, and generating a task output by performing a learning operation on the fusion feature through the task head block; and (III) a process of generating a task loss by referring to the task output and an original answer corresponding to the learning data, and backpropagating the task loss to generate a task loss for the task head block. , a process of training at least a portion of the residual unit and the adaptation block; In the process (I), The residual unit includes a plurality of convolution layers, and is configured to decompose a first filter included in at least a part of the plurality of convolution layers into a plurality of second filters having a lower rank than the first filter. Learning device.

17. In the process (I), Each of the first feature extraction block to the nth feature extraction block has a parameter fixed through freezing, In the process (III), 17. The learning device according to claim 15 or 16, wherein the processor is configured not to update the parameters of each of the first feature extraction block to the nth feature extraction block when backpropagating the task loss.

18. In the process (II), The learning device according to claim 15 or 16, wherein the processor generates the fusion feature by fusing the residual feature and the transformation feature through an element-wise add layer by performing an add operation.

Images