Method, server and system for deep metric learning per hierarchical steps of multi-labels and few-shot inference using thereof

Abstract

A method for stepwise deep neural network scaling based on hierarchical multi-labels and few-shot inference using the same is provided. The method comprises the steps of: sampling a plurality of N multiples from a training dataset represented by hierarchical multi-labels; configuring the N multiples into triples or quadruplets corresponding to the number of classes in each layer; performing layer-by-layer scaling for a deep neural network model based on the configured triples or quadruplets; and distributing the deep neural network model with the loss function removed to an embedded device as the scaling is completed.

Description

Technology Field

[0001] The present invention relates to a hierarchical multi-label based stepwise deep neural network scale learning for embedded devices and a few-shot inference method using the same, a server, and a system. Background Technology

[0002] Scale learning in conventional neural networks applied a method of training the neural network model so that the Euclidean distance between two samples in the embedding space represents the semantic distance.

[0003] However, the embedding space obtained through conventional neural network scaling learning has the problem of being able to distinguish between classes but failing to explain the relationships between them, so a technical solution to resolve this is currently required. Prior art literature

[0004] Published Patent Application No. 10-2019-0056009 (May 24, 2019) The problem to be solved

[0005] The problem that the present invention aims to solve is to provide a hierarchical multi-label based stepwise deep neural network scaling learning method, a few-shot inference method using the same, a server, and a system that enables class classification with only a small amount of data in an embedded device, and in particular, provides that the Euclidean distance in the embedding space through scaling learning of a deep learning neural network has a substantial semantic distance.

[0006] However, the problems that the present invention aims to solve are not limited to those described above, and other problems may exist. means of solving the problem

[0007] A method for hierarchical multi-label-based stepwise deep neural network scaling and few-shot inference using the same according to the first aspect of the present invention for solving the above-described problem comprises: a step of sampling a plurality of N-weights from a training dataset represented by hierarchical multi-labels; a step of configuring the N-weights into triples or quadruplets corresponding to the number of classes in each layer; a step of performing scaling learning for each layer of a deep neural network model based on the configured triples or quadruplets; and a step of distributing a deep neural network model with the loss function removed to an embedded device as the scaling learning is completed.

[0008] Additionally, a server for performing stepwise deep neural network scale learning based on hierarchical multi-label inference for an embedded device according to the second aspect of the present invention includes a communication module that transmits and receives data to and from the embedded device, a memory in which a program for performing stepwise deep neural network scale learning based on hierarchical multi-label is stored, and a processor that, by executing the program stored in the memory, samples a plurality of N multiples from a learning dataset represented by hierarchical multi-labels, configures the N multiples into triples or quadruplets corresponding to the number of classes in each layer, performs stepwise scale learning for each layer of a deep neural network model based on the configured triples or quadruplets, and distributes the deep neural network model with the loss function removed to the embedded device through the communication module as the scale learning is completed.

[0009] In addition, a system for stepwise deep neural network scale learning and few-shot inference based on a hierarchical multi-label according to the third aspect of the present invention includes a deep neural network scale learning server comprising a processor that samples a plurality of N-weights from a training dataset represented by hierarchical multi-labels, configures the N-weights into triples or quadruples corresponding to the number of classes in each layer, performs scale learning for each layer of a deep neural network model based on the configured triples or quadruples, and distributes the deep neural network model with the loss function removed to an embedded device as the scale learning is completed, and an embedded device that performs few-shot inference on input reference samples by adding a classifier to the deep neural network model as the deep neural network model is distributed.

[0010] A computer program according to another aspect of the present invention for solving the above-mentioned problem is combined with a computer, which is hardware, to execute a hierarchical multi-label-based stepwise deep neural network scale learning and a few-shot inference method using the same, and is stored in a computer-readable recording medium.

[0011] Other specific details of the present invention are included in the detailed description and drawings. Effects of the invention

[0012] According to one embodiment of the present invention described above, through hierarchical multi-label-based stepwise deep neural network scale learning, the embeddings extracted from each sample can not only distinguish between individual classes but also possess substantial semantic distances between classes.

[0013] Through this, even when you want to apply recognition targets to completely new classes according to the needs of the operating field of the embedded device, you can predict the characteristics of the class, cluster classes of similar properties together, or provide systematic prediction values.

[0014] The effects of the present invention are not limited to those mentioned above, and other unmentioned effects will be clearly understood by a person skilled in the art from the description below. Brief explanation of the drawing

[0015] FIG. 1 is a flowchart of a deep neural network scale learning and few-shot inference method according to one embodiment of the present invention. FIG. 2 is a diagram illustrating an example of the structure of a deep neural network in one embodiment of the present invention. FIG. 3 is a diagram illustrating an example of a hierarchical label-based dataset in one embodiment of the present invention. Figure 4 is a diagram illustrating an example of sampling six items from the hierarchical label structure of Figure 3. Figure 5 is a diagram illustrating the performance of scale learning based on triplets. Figure 6 is a diagram illustrating the process of performing scale learning based on quadruple terms. Figure 7 is a diagram showing an example of a configuration formed by sampling quadrants. Figure 8 is a diagram illustrating the gradient descent technique. Figure 9 is a diagram illustrating a deep neural network model deployed on an embedded device. Referring to FIG. 10, this is a block diagram of a hierarchical multi-label based stepwise deep neural network scale learning server and system according to one embodiment of the present invention. Specific details for implementing the invention

[0016] The advantages and features of the present invention and the methods for achieving them will become clear by referring to the embodiments described below in detail together with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below but may be implemented in various different forms. These embodiments are provided merely to ensure that the disclosure of the present invention is complete and to fully inform those skilled in the art of the scope of the present invention, and the present invention is defined only by the scope of the claims.

[0017] The terms used in this specification are for describing embodiments and are not intended to limit the invention. In this specification, the singular form includes the plural form unless specifically stated otherwise in the text. The terms "comprises" and / or "comprising" used in this specification do not exclude the presence or addition of one or more other components in addition to the components mentioned. Throughout the specification, the same reference numerals refer to the same components, and "and / or" includes each of the mentioned components and all combinations of one or more. Although terms such as "first," "second," etc., are used to describe various components, these components are not limited by these terms. These terms are used merely to distinguish one component from another. Therefore, the first component mentioned below may be the second component within the technical scope of the invention.

[0018] Unless otherwise defined, all terms used herein (including technical and scientific terms) may be used in a meaning commonly understood by those skilled in the art to which the present invention pertains. Additionally, terms defined in commonly used dictionaries are not to be interpreted ideally or excessively unless explicitly and specifically defined otherwise.

[0020] In the following, the background of the conception of the present invention will be explained first to aid the understanding of a person skilled in the art, and then the present invention will be described in detail.

[0021] To classify classes using only a small number of reference samples in embedded devices, an embedding technique using scale learning of a deep learning neural network can be used.

[0022] Generally, deep learning neural networks typically consist of neural networks for both feature extractors and classifiers, but their application in embedded devices is limited. This is due to several reasons: firstly, it is difficult to learn new classes other than the defined ones; secondly, even if learning is possible, modifying the neural network's weights reduces the recognition rate of existing classes; and thirdly, the network is vulnerable to outliers because it attempts to find the correct answer among the target classes even when a non-target class is input. For these reasons, using embedding techniques is useful for few-shot learning in embedded devices.

[0023] Classification between classes can be facilitated by making samples of the same class cluster within a certain distance in the embedding space. Similarly, training a neural network model so that the Euclidean distance between two samples in the embedding space represents semantic distance is called scale learning in neural networks.

[0024] Training a neural network model refers to updating the model's weights and biases to reduce the difference between the output value obtained by passing input data through the network and the correct answer for the actual input data. In this context, the error between the correct answer and the network output is called loss; while loss can simply represent a difference, it can also be expressed as a function of that difference, known as the loss function. The loss function for scaling learning is related to the distance between samples in the embedding space.

[0025] However, scale learning in conventional neural networks focused on distinguishing between classes, so the actual distance between classes in the embedding space could not hold special significance. For example, when passing unknown input data through a neural network, even if a feature vector located between Class 1 and Class 2 in the embedding space is extracted, it cannot be definitively concluded that it possesses characteristics intermediate between Class 1 and Class 2. It can only predict which of Class 1 or Class 2 it is closer to.

[0026] In other words, the embedding space obtained through conventional neural network-based scale learning can distinguish between classes, but it cannot explain the relationships between classes. However, when using few-shot learning and inference for embedding techniques in embedded devices, datasets that are not included in the training dataset are often applied.

[0027] In other words, due to the nature of actual operating environments in embedded devices, there are often cases where predictions must be made for classes that are completely different from the datasets used to train neural networks on servers or in the cloud.

[0028] At this time, embedding techniques using neural network scale learning have the advantage of facilitating class classification with a small amount of data (hereinafter few-shot). However, existing methods do not fully represent semantic distances, such as position in the embedding space or distance from other classes.

[0029] Accordingly, the hierarchical multi-label-based stepwise deep neural network scale learning and few-shot inference method using the same, server (100), and system (1) according to one embodiment of the present invention are characterized by providing that the Euclidean distance in the embedding space has a substantial semantic distance by learning embeddings stepwise for a dataset represented by multiple labels having a hierarchical structure in order to overcome this problem.

[0031] Hereinafter, with reference to FIGS. 1 to 9, a hierarchical multi-label-based stepwise deep neural network scale learning and a few-shot inference method using the same according to an embodiment of the present invention (hereinafter, deep neural network scale learning and few-shot inference method) will be described.

[0032] FIG. 1 is a flowchart of a deep neural network scale learning and few-shot inference method according to an embodiment of the present invention. FIG. 2 is a diagram illustrating an example of the structure of a deep neural network in an embodiment of the present invention.

[0033] A deep neural network scale learning and few-shot inference method according to one embodiment of the present invention comprises the steps of: sampling a plurality of N-weights from a learning dataset represented by hierarchical multiple labels (S110); configuring the N-weights into triples or quadruplets corresponding to the number of classes in each layer (S120); performing scale learning for each layer of a deep neural network model based on the configured triples or quadruplets (S130); and distributing the deep neural network model with the loss function removed to an embedded device (200) as the scale learning is completed (S140).

[0034] Meanwhile, each step illustrated in FIG. 1 can be understood as being performed by the hierarchical multi-label-based stepwise deep neural network scale learning server (100) and system (1) described later, but is not necessarily limited thereto.

[0035] An example of stepwise deep neural network scaling learning based on a hierarchical multi-label in one embodiment of the present invention is illustrated in FIG. 2. In this case, FIG. 2 assumes a label structure composed of three layers, but one embodiment of the present invention is not limited thereto, and the number of layers can be sufficiently expanded.

[0036] Scaling learning to extract features for each layer utilizes triplet and quadruplet loss functions.

[0037] First, the server (100) samples multiple N multiples from a training dataset represented by hierarchical multiple labels in a training database (S110). Here, an N multiple means composed of N different samples from the training dataset. For example, if a reference sample is randomly selected, one positive sample of the same class and N-2 negative samples from a different class are selected according to appropriate criteria, and a total of N samples are used as a single tuple.

[0038] Furthermore, multi-labels refer to a situation where a single object is represented by multiple correct label answers. And, hierarchical labels refer to a label structure in which a parent label includes child labels.

[0039] After the server (100) constructs N multiples, multiple N multiples are grouped into a single batch and used for one training session. If there are only 2 classes in the corresponding layer, scale learning is performed using a triplet consisting of a reference sample, a positive sample, and a negative sample. In contrast, if there are 3 or more classes in the corresponding layer, scale learning is performed using a quadruplet consisting of a reference sample, a positive sample, a first negative sample, and a second negative sample.

[0040] In scale learning, data sampling is much more important than in general deep neural network learning. In general deep neural network learning, data is randomly shuffled and then sequentially divided by batch size for training. On the other hand, in scale learning, a fixed number of samples (for example, 4 in the case of a quadruplet) that meet specific criteria are randomly selected and then divided by batch size for scale learning.

[0041] Scale learning in one embodiment of the present invention undergoes a more complex sampling process than in the case where all labels are independent and in an equitable relationship. In one embodiment, the specific conditions for sampling for each layer are as follows.

[0042] - Condition (1): A completely positive sample is a sample that has the same label as the reference sample.

[0043] - Condition (2): A completely negative sample is a sample that has a different label from the reference sample at each layer.

[0044] - Condition (3): All speech samples consist of samples that have a different class from the reference sample at the lowest level.

[0045] - Condition (4): Partial speech samples consist of samples from clusters different from the full speech samples at each layer.

[0046] At this time, according to conditions (2) to (4), generalized In a multi-label structure consisting of layers, speech samples include both complete speech samples and partial speech samples. Dogs appear. This is because, excluding the lowest class, the remaining upper classes (i.e., This is because there are two possibilities in ) either positive or negative. In this case, since the lowest level must be negative, the number of possibilities is 1.

[0047] If, in the case of a multi-label system consisting of three layers as shown in Fig. 2, a total of four cases of speech samples can be considered according to the above sampling conditions. Accordingly, a batch can be constructed by sampling the following six-word combination.

[0048] ① Reference Sample

[0049] ② Completely positive samples (for all strata)

[0050] ③ Full voice samples (for all layers)

[0051] ④ Partial voice sample (for the 1st floor)

[0052] ⑤ Partial voice samples (for the 1st and 2nd floors)

[0053] ⑥ Partial voice samples (for the 1st and 3rd floors)

[0054] Next, the server (100) configures the N-fold into a triplet or a quadruplet corresponding to the number of classes in each layer (S120). That is, in the case of the example described above, a triplet or a quadruplet is taken from the 6-fold according to the number of classes in each layer, and then scale learning is performed based on this. For convenience of explanation, let us assume a dataset consisting of hierarchical labels as shown in FIG. 3.

[0055] FIG. 3 is a diagram illustrating an example of a hierarchical label-based dataset in one embodiment of the present invention. FIG. 4 is a diagram illustrating an example of sampling six items in the hierarchical label structure of FIG. 3.

[0056] For example, as illustrated in FIG. 4, if a sample belonging to {Male, White, Class 5} is selected as the reference sample, the completely positive sample is selected as another object of Class 5. Since this positive sample also has multiple labels of {Male, White, Class 5}, condition (1) is satisfied.

[0057] Next, the completely negative sample must have a different label from the reference sample in all layers according to condition (2). Thus, as shown in Fig. 4, a sample belonging to {female, black, class 16} can be selected. Of course, there are many other options. For example, it can be selected as {female, Asian, class 11}.

[0058] Next, the selection of partial voice samples can be made according to each condition. As shown in Figure 4, partial voice samples for the lowest level (Level-1) can be selected from {Male, White, Class 1}, partial voice samples for Levels-1 & 2 from {Male, Asian, Class 12}, and partial voice samples for Levels-1 & 3 from {Female, White, Class 4}.

[0059] At this point, it should be noted that since the complete negative sample already has the label “Black,” the Level-1 & 2 negative samples cannot be selected from the same class as {Male, Black, Class 12}. This is because, at Level 2, the two negative samples must belong to different clusters to form a quadruplet.

[0060] After sampling the six-fold in this way, a triple or quadruple is taken for each stratum for scale learning.

[0061] At this time, if there are three or more classes in the corresponding layer, a quadruplet may be formed including a reference sample having any class, a positive sample having the same class as the reference sample, a first negative sample having a different class from the reference sample, and a second negative sample having a class different from the reference sample and the first negative sample, respectively.

[0062] In addition, if there are two classes in the corresponding layer, a triplet can be formed including a reference sample having any class, a positive sample having the same class as the reference sample, and a negative sample having a different class from the reference sample.

[0063] Accordingly, when sampling as shown in Fig. 4, the first layer can be composed of a reference sample, a positive sample, a level-1 negative sample, and a completely negative sample. The second layer can be composed of a reference sample, a positive sample, a level-1 & 2 negative sample, and a completely negative sample.

[0064] Also, since the third layer has only two classes, it is impossible to form a quadruplet. Therefore, it is formed as a triplet, but the voice sample that is closer to the reference sample among the Level-1 & 3 voice samples and the complete voice sample can be selected.

[0065] In this way, when the configuration of the triplet or quadruplet based on the N-multiplet is completed, the server (100) performs scale learning for each layer of the deep neural network model based on the configured triplet or quadruplet (S130).

[0066] Figure 5 is a diagram illustrating the performance of scale learning based on triplets.

[0067] First, when a triplet including a reference sample, a positive sample, and a negative sample is formed, the relationship between the triplets of each sample is defined as Equation 1 below.

[0068] [Equation 1]

[0069]

[0070] At this time, in Equation 1 represents the first margin value, and the function is a vector and vector It refers to the Euclidean distance between them. Also, represents the forward propagation of a neural network, and the input Outputs a vector with extracted features for. And , , is input data sampled from the training database, which in turn represent the reference sample, positive sample, and negative sample-1.

[0071] In this case, speech sample-1 basically refers to a partial speech sample at each layer. Therefore, at layer 1, it becomes a Level-1 speech sample, and at layer 2, it becomes a Level 1 & 2 speech sample. However, in cases where a quadruplet cannot be formed because it consists of only two classes, such as layer 3 in Fig. 3, the sample that is closer to the reference sample between the complete speech sample and the partial speech sample is selected. That is, the one that is closer between the Level-1 & 3 speech sample and the complete speech sample is selected as speech sample-1.

[0072] To explain Equation 1 above more intuitively, scale learning is performed through the sinusoidal propagation operation of the deep neural network model so that among the configured triplets, positive samples in the same class as the reference sample are located relatively closer in Euclidean distance, and negative samples in a different class are located relatively farther in Euclidean distance from the reference sample. At this time, at least positive samples than negative samples have a pre-set first margin value ( Scale learning is performed to position the sample closer to the reference sample by more than )

[0073] Figure 6 is a diagram illustrating the process of performing scale learning based on quadruple terms.

[0074] Quarterial scale learning aims for the result shown in Fig. 6 in addition to the triplet scale learning above. In this case, speech sample-2 in Fig. 6 refers to a complete speech sample. However, speech sample-2 does not exist separately in the triplet-structured hierarchy.

[0075] First, the relationship for the quadruplets of the reference sample, positive sample, negative sample-1, and negative sample-2 is defined as Equation 2.

[0076] [Equation 2]

[0077]

[0078] Here is the second margin value and must satisfy the relationship in Equation 3. That is, the second margin value is set to be smaller than the first margin value.

[0079] [Equation 3]

[0080]

[0081] To explain Equation 2 above, through the sinusoidal propagation operation of the deep neural network model, the goal is to ensure that positive samples in the same class as the reference sample among the configured quadruplets are located relatively closer in Euclidean distance, while first and second negative samples having different classes from the reference sample are separated further apart. Accordingly, scaling learning is performed so that the Euclidean distance between the first negative sample and the second negative sample is located relatively farther apart. In other words, while the distance between different classes is sufficiently separated, samples of the same class converge in a direction that brings them together at a single point.

[0082] At this time, scale learning is performed such that the sum of the Euclidean distance value of the positive sample relative to the reference sample and the pre-set second margin value is smaller than the Euclidean distance value between the first and second negative samples.

[0083] On the other hand, since Equation 1 considers only relative distance, the distribution of samples by class resulting from the final learning outcome may not be uniform. However, by adding the condition of Equation 2, the sample distribution can be formed to be close to a circular shape. In other words, one can expect the effect of making the difference between classes large and the difference between each sample within a class small. This result is very useful for applying and extending embeddings to new classes that have not been learned in the future.

[0084] In addition, one embodiment of the present invention defines a quadruple loss function to perform scale learning in a deep learning-based deep neural network model to satisfy the conditions of Equation 1 and Equation 2. First, a general quadruple loss function will be explained, and then the loss function defined in one embodiment of the present invention will be explained in detail.

[0085] Figure 7 is a diagram showing an example of a configuration formed by sampling quadrants.

[0086] The loss functions for triple and quadruplets are already widely known. Since the loss function for the triplet is included in the quadruplet, a separate explanation will be omitted.

[0087] When the configuration is composed of N quadruplets as shown in Fig. 7, the loss function of the quadruplets is defined as shown in the equation below.

[0088] [Equation 4]

[0089]

[0090]

[0091] In Equation 4 above, is the input sample Neural network regarding It is a feature vector extracted by passing it through. And, Each refers to the reference sample, positive sample, negative sample-1, and negative sample-2 within the batch, and there are N of them for each batch.

[0092] Based on Equation 4 above, in one embodiment of the present invention, the loss function for one set of quadruple terms in the batch is as follows Equation 5.

[0093] [Equation 5]

[0094]

[0095]

[0096] Meanwhile, scale learning for the quadruplets can be performed using Stochastic Gradient Descent (SGD).

[0097] Figure 8 is a diagram illustrating the gradient descent technique.

[0098] Gradient Descent is a method for finding the optimal point by calculating the difference (loss) between the result (predicted value) passed through the neural network and the correct answer, and updating the network parameters by calculating the inverse (descent) gradient (gradient) of the calculated loss. By simplifying the loss function and plotting it for a single component, the Gradient Descent method can be depicted as shown in Fig. 8. In other words, it is a method that approaches the optimal component value with the minimum loss by moving in the reverse direction along the graph by an amount proportional to the partial derivative of the loss function with respect to that component. Here, is a constant that indicates how much to move, and is a hyperparameter commonly called the learning rate.

[0099] Hyperparameters refer to values ​​arbitrarily determined by experts based on their experience with various experiments, rather than mathematically proven calculated values. Generally, it is common practice to set the learning rate high initially and decrease it as the model approaches the optimal point. Setting the learning rate too low requires excessive training time to find the optimal point, while setting it too high may cause the model to overshoot the optimal point; therefore, an appropriate value must be set.

[0100] In one embodiment of the present invention, scaling learning is performed by dividing it into each layer. That is, the server (100) performs scaling learning for each layer of the deep neural network model to calculate a layer-by-layer loss function, and can calculate the total loss function through the weighted sum of the layer-by-layer loss functions. At this time, the total loss function is as shown in Equation 6 below.

[0101] [Equation 6]

[0102]

[0103] In the example shown in Figure 4 above, the loss function of each layer based on Equation 6 is given by the following Equation 7.

[0104] [Equation 7]

[0105]

[0106]

[0107]

[0108]

[0109]

[0110] In Equation 7 , , , , represents the embedding outputs obtained by passing the reference sample, positive sample, Level-1 negative sample, Level-1 & 2 negative sample, Level-1 & 3 negative sample, and completely negative sample through their respective feature extraction neural networks, in that order. Also, silver between It means something closer to.

[0111] In such cases, the loss function for scaling learning at each level can be expressed as shown in Equation 8 below.

[0112] [Equation 8]

[0113]

[0114] At this time, in Equation 8 is a hyperparameter determined by the task as a weight constant for the hierarchical loss.

[0115] Once the loss function is calculated, the server (100) then updates the weights and biases of the deep neural network model through the back propagation operation of the deep neural network model.

[0116] At this time, the server (100) measures the accuracy of the current deep neural network model based on the KNN (K-Nearest Neighbors) algorithm at predetermined intervals, and if the measured accuracy satisfies a preset accuracy (sufficient accuracy), the scale learning of the deep neural network model is terminated.

[0117] When the training is finished, the server (100) stores the weights and biases of the deep neural network model as model information.

[0118] Referring again to FIG. 1, the server (100) distributes the deep neural network model with the loss function removed as scale learning is completed to the embedded device (200) (S140). That is, when the server (100) distributes the deep neural network model to the embedded device (200), it fixes the weights and biases to the model values ​​before distribution.

[0119] FIG. 9 is a diagram illustrating a deep neural network model deployed to an embedded device (200).

[0120] The embedded device (200) can obtain embeddings of classes to be inferred through few-shot learning by adding a classifier to a deep neural network model distributed by the server (100). In this case, if there are reference samples (M) with a predetermined number (N) of classes, a total of N * M embeddings can be obtained. Accordingly, the embedded device (200) only needs to store the embedding results in text or binary form instead of input data that occupies a large amount of storage space.

[0121] Meanwhile, a predetermined number of N classes may be classes that are not used for neural network scale learning by the server (100). That is, since data suitable for each situation exists in most operating environments of the embedded device (200), there may be many cases where embeddings are obtained using only classes that are not actually used for learning.

[0122] The classifier is added to the position of the removed loss function in Fig. 2. Since the Euclidean distance in the embedding space is learned to have semantic distance through scale learning in the server (100), the classifier can use the KNN algorithm, which is a distance-based class classification algorithm.

[0123] Meanwhile, the operation in the embedded device (200) can be considered in the following three cases.

[0124] Case 1) Select one from the pre-reference samples

[0125] When reference samples having a predetermined number of classes are passed through a deep neural network model added to an embedded device (200), they can be projected as a single point in the embedding space. In this case, class classification is possible using only the embeddings of the lowest layer.

[0126] Next, a classifier detects K output samples from the already embedded reference samples (N*M) based on the feature vectors of the reference samples. Then, among the K detected output samples, a majority vote determines which class they belong to most frequently, providing this as the few-shot inference result.

[0127] Case 2) Extending by adding a new class

[0128] If M reference samples having a new class are passed through a deep neural network model added to the embedded device (200), the result is added to the embedding. Subsequently, the classification of the class is the same as in case 1. In this case, the characteristics of the class can be estimated by analyzing the embeddings for each layer, which is one of the advantages of the present invention.

[0129] Case 3) Reject outliers

[0130] In one embodiment of the present invention, when a reference sample that does not belong to a class in a training dataset is input into a deep neural network model of an embedded device (200), it can be rejected. That is, unlike in case 1, it is not inferred as one of the pre-embedded classes by unconditional majority vote, but if the Euclidean distance from the average value of the reference sample for the class that does not belong exceeds a preset threshold, it is determined that it does not belong to that class, and the reference sample can be determined and rejected as an outlier.

[0131] One embodiment of the present invention has the advantage that, unlike existing methods, when an object with a completely new class other than the class used for training, such as in cases 2 and 3, is input through hierarchical multi-label-based stepwise scale learning, it is possible to predict the characteristics of an input object by analyzing its position in the hierarchical embeddings to determine which category among the upper class categories it belongs to.

[0132] Referring to the examples in FIGS. 2 and 3, let us assume that two new people have entered as inputs to the embedded device (200), in a three-layer structure consisting of male / female, white / Asian / black, and individual names.

[0133] In this case, the existing method could classify the two new individuals into classes included in the training dataset. However, the distance between new classes or the distance from other classes in the embedding space does not carry any specific meaning. For example, even if two classes are closer than other classes in the embedding space, they may possess completely different characteristics as male and female, or conversely, the same class of males may be further away.

[0134] However, according to one embodiment of the present invention, through layer-by-layer embeddings, it is possible to provide a few-shot inference result that both new classes appear to be male, one white and one Asian.

[0135] Meanwhile, in the above description, steps S110 to S140 may be further divided into additional steps or combined into fewer steps according to an embodiment of the present invention. Also, some steps may be omitted as necessary, and the order between steps may be changed. Furthermore, even if other omitted details are included, the contents of FIGS. 1 to 9 may also be applied to the system (1) and server (100) of FIG. 10.

[0136] Hereinafter, a hierarchical multi-label based stepwise deep neural network scale learning server (100) and a system (1) according to an embodiment of the present invention will be described with reference to FIG. 10.

[0137] A system (1) according to one embodiment of the present invention includes a deep neural network scale learning server (100) and at least one embedded device (200).

[0138] The server (100) is configured to include a communication module (110), memory (120), and a processor (130). The communication module (110) transmits and receives data with an embedded device (200), and the memory (120) stores a program for performing hierarchical multi-label-based stepwise deep neural network scale learning.

[0139] The processor (130) executes a program stored in memory (120) to sample multiple N-multiples from a learning dataset represented by hierarchical multi-labels, configures the N-multiples into triplets or quadruplets corresponding to the number of classes in each layer, and then performs layer-by-layer scaling learning for the deep neural network model based on the configured triplets or quadruplets. Then, through a communication module, the deep neural network model with the loss function removed is distributed to an embedded device (200) as the scaling learning is completed.

[0140] The embedded device (200) performs few-shot inference on input reference samples by adding a classifier to the deep neural network model as the deep neural network model is deployed.

[0141] One embodiment of the present invention described above may be implemented as a program (or application) and stored on a medium to be executed in combination with a computer, which is hardware.

[0142] The aforementioned program may include code encoded in computer languages ​​such as C, C++, JAVA, Ruby, and machine language, which can be read by the computer's processor (CPU) through the computer's device interface, in order for the computer to read the program and execute the methods implemented in the program. Such code may include functional code related to functions that define the necessary functions for executing the methods, and may include control code related to execution procedures necessary for the computer's processor to execute the functions according to a predetermined procedure. Additionally, such code may further include memory reference code regarding where (address) additional information or media necessary for the computer's processor to execute the functions should be referenced in the computer's internal or external memory. In addition, if the processor of the computer needs to communicate with any other computer or server located remotely in order to execute the above functions, the code may further include communication-related code regarding how to communicate with any other computer or server located remotely using the communication module of the computer, and what information or media to transmit or receive during communication.

[0143] The above-mentioned storage medium refers to a medium that stores data semi-permanently and is readable by a device, rather than a medium that stores data for a short period of time, such as a register, cache, or memory. Specifically, examples of the above-mentioned storage medium include, but are not limited to, ROM, RAM, CD-ROM, magnetic tape, floppy disk, and optical data storage device. That is, the above-mentioned program may be stored on various recording media on various servers that the computer can access, or on various recording media on the user's computer. Additionally, the above-mentioned medium may be distributed across networked computer systems, and computer-readable code may be stored in a distributed manner.

[0144] The foregoing description of the present invention is for illustrative purposes only, and those skilled in the art will understand that other specific forms can be easily modified without altering the technical spirit or essential features of the present invention. Therefore, the embodiments described above should be understood as illustrative in all respects and not restrictive. For example, each component described as a single unit may be implemented in a distributed manner, and components described as distributed may likewise be implemented in a combined form.

[0145] The scope of the present invention is defined by the claims set forth below rather than by the detailed description above, and all modifications or variations derived from the meaning and scope of the claims and equivalent concepts thereof should be interpreted as being included within the scope of the present invention. Explanation of the symbols

[0146] 1: Deep Neural Network Scale Learning and Few-Shot Inference System 100: Deep Neural Network Scale Learning Server. 110: Communication module 120: Memory 130: Processor 200: Embedded devices

Claims

Claim 1 A method performed by a computer comprises: sampling a plurality of N-weights from a training dataset represented by hierarchical multiple labels; configuring the N-weights into triples or quadruplets corresponding to the number of classes at each layer; performing layer-by-layer scaling learning for a deep neural network model based on the configured triples or quadruplets; and distributing the deep neural network model with the loss function removed to an embedded device upon completion of the scaling learning, wherein the step of performing layer-by-layer scaling learning for the deep neural network model based on the configured triples or quadruplets includes: performing scaling learning for each layer of the deep neural network model to calculate a layer-by-layer loss function; and calculating an overall loss function through the weighted sum of the layer-by-layer loss functions, and wherein the step of sampling a plurality of N-weights from the training dataset represented by hierarchical multiple labels comprises: configuring a completely positive sample as a sample having the same label as a reference sample; and configuring a completely negative sample as a sample having a label different from the reference sample at each layer. A hierarchical multi-label based stepwise deep neural network scale learning and few-shot inference method using the same, comprising the steps of: configuring all speech samples in the lowest layer as samples having a different class from the reference sample; and configuring partial speech samples in each layer as samples of a cluster different from the complete speech sample. Claim 2 delete Claim 3 A hierarchical multi-label based stepwise deep neural network scale learning and few-shot inference method using the same, wherein the step of configuring the N-multiplets into triplets or quadruplets corresponding to the number of classes in each layer comprises, when there are two classes in the layer, configuring a triplet including a reference sample having an arbitrary class, a positive sample having the same class as the reference sample, and a negative sample having a different class from the reference sample. Claim 4 In paragraph 3, the step of performing scale learning for each layer of a deep neural network model based on the configured triplet or quadruplet is to perform scale learning through the forward propagation operation of the deep neural network model such that, among the configured triplets, positive samples are located relatively closer to the reference sample in terms of Euclidean distance, and negative samples are located relatively farther away in terms of Euclidean distance, thereby performing scale learning. This describes a hierarchical multi-label based stepwise deep neural network scale learning and a few-shot inference method using the same. Claim 5 In claim 4, the step of performing scale learning for each layer of a deep neural network model based on the configured triplet or quadruplet is to perform scale learning such that the positive sample is located closer to the reference sample by at least a preset first margin value than the negative sample, a hierarchical multi-label based stepwise deep neural network scale learning and few-shot inference method using the same. Claim 6 A hierarchical multi-label based stepwise deep neural network scale learning and few-shot inference method using the same, wherein, in claim 1, the step of configuring the N-weights into triples or quadruplets corresponding to the number of classes in each layer comprises, when there are three or more classes in the layer, configuring the quadruplets to include a reference sample having any class, a positive sample having the same class as the reference sample, a first negative sample having a different class from the reference sample, and a second negative sample having a different class from the reference sample and the first negative sample, respectively. Claim 7 In claim 6, the step of performing scale learning for each layer of a deep neural network model based on the configured triplet or quadruplet is to perform scale learning through the forward propagation operation of the deep neural network model such that, among the configured quadruplets, the positive sample is located relatively closer in Euclidean distance to the reference sample, and the Euclidean distance between the first negative sample and the second negative sample is located relatively farther apart, thereby performing scale learning. This describes a hierarchical multi-label based stepwise deep neural network scale learning and a few-shot inference method using the same. Claim 8 In claim 7, the step of performing scale learning for each layer of a deep neural network model based on the configured triplet or quadruplet is to perform the scale learning such that the sum of the Euclidean distance value of a positive sample for the reference sample and a preset second margin value is smaller than the Euclidean distance value between the first and second negative samples, a hierarchical multi-label based stepwise deep neural network scale learning and few-shot inference method using the same. Claim 9 delete Claim 10 A method for hierarchical multi-label-based stepwise deep neural network scaling and few-shot inference using the same, wherein the step of performing scaling learning for each layer of a deep neural network model based on the configured triplet or quadruplet comprises: updating the weights and biases of the deep neural network model through the back propagation operation of the deep neural network model; measuring the accuracy of the current deep neural network model based on the KNN algorithm at predetermined intervals; terminating the scaling learning of the deep neural network model when the measured accuracy satisfies a preset accuracy; and storing the weights and biases of the deep neural network model that satisfy the preset accuracy as model information. Claim 11 A hierarchical multi-label based stepwise deep neural network scale learning and few-shot inference method using the same, further comprising: adding a classifier to the distributed deep neural network model in the embedded device; inputting reference samples having a predetermined number of classes into the added deep neural network model; detecting output samples for the reference samples embedded as outputs for the inputs of the deep neural network model through the classifier; and obtaining a few-shot inference result for the class of the output samples through the classifier. Claim 12 In claim 11, the step of inputting reference samples having a predetermined number of classes into the added deep neural network model is to input the reference samples having a new class into the added deep neural network model to further expand the new class, a hierarchical multi-label based stepwise deep neural network scale learning and few-shot inference method using the same. Claim 13 In claim 11, when a reference sample not belonging to a class in the training dataset is input to the deep neural network model, the step of obtaining a few-shot inference result for the class of an output sample through the classifier is to determine that the reference sample does not belong to the corresponding class and to determine and reject the reference sample as an outlier when the Euclidean distance from the mean value of the reference sample to the class not belonging to the class exceeds a preset threshold. This describes a hierarchical multi-label based stepwise deep neural network scale learning and a few-shot inference method using the same. Claim 14 A server for performing stepwise deep neural network scaling based on hierarchical multi-labels for few-shot inference of an embedded device, comprising: a communication module for transmitting and receiving data with said embedded device; a memory storing a program for performing stepwise deep neural network scaling based on said hierarchical multi-labels; and, upon executing the program stored in said memory, sampling a plurality of N-weights from a training dataset represented by hierarchical multi-labels, configuring the N-weights into triples or quadruples corresponding to the number of classes in each layer, and then performing scaling learning for each layer of a deep neural network model based on said configured triples or quadruples, wherein the scaling learning for each layer comprises performing scaling learning for each layer of said deep neural network model to calculate a layer-specific loss function and calculating an overall loss function through the weighted sum of said layer-specific loss functions, wherein the sampling of the plurality of N-weights comprises configuring a completely positive sample as a sample having the same label as a reference sample and configuring a completely negative sample as a sample having a label different from the reference sample in each layer, and all negative samples at the lowest layer are the reference sample and A hierarchical multi-label based stepwise deep neural network scaling server comprising a processor that comprises samples having different classes and samples of a cluster different from the complete speech samples at each layer, and a deep neural network model with the loss function removed as scaling learning is completed through the communication module. Claim 15 A system for hierarchical multi-label-based stepwise deep neural network scaling learning and few-shot inference, comprising a deep neural network scaling learning server including a processor that samples multiple N-multiples from a training dataset represented by hierarchical multi-labels by configuring fully positive samples as samples having the same label as a reference sample, configuring fully negative samples as samples having a different label from the reference sample at each layer, configuring all negative samples as samples having a different class from the reference sample at the lowest layer, and configuring partial negative samples as samples of a cluster different from the fully negative sample at each layer, configuring the N-multiples into triplets or quadruplets corresponding to the number of classes at each layer, performing scaling learning for each layer of a deep neural network model based on the configured triplets or quadruplets, and distributing the deep neural network model with the loss function removed to an embedded device upon completion of scaling learning; and a hierarchical multi-label-based stepwise deep neural network including an embedded device that performs few-shot inference on input reference samples by adding a classifier to the deep neural network model upon distribution of the deep neural network model. System for scale learning and few-shot inference. Claim 16 A system for hierarchical multi-label based stepwise deep neural network scale learning and few-shot inference according to claim 15, wherein the embedded device inputs reference samples having a predetermined number of classes into the added deep neural network model, detects output samples for the reference samples embedded as outputs for the input of the deep neural network model, and obtains a few-shot inference result for the class of the output samples through the classifier. Claim 17 A system for hierarchical multi-label based stepwise deep neural network scale learning and few-shot inference, wherein the embedded device inputs reference samples having a predetermined number of new classes into the added deep neural network model to further expand the new classes. Claim 18 A system for hierarchical multi-label based stepwise deep neural network scale learning and few-shot inference according to claim 16, wherein when a reference sample not belonging to a class in the above-mentioned training dataset is input to the above-mentioned deep neural network model, the embedded device determines that the reference sample does not belong to the class and rejects the reference sample as an outlier if the Euclidean distance from the mean value of the reference sample to the above-mentioned non-belonging class exceeds a preset threshold.

Images