Expression learning device, method, and program

Abstract

To provide an expression learning device, a method and a program that can improve accuracy of a loss function which is calculated in expression learning.SOLUTION: An expression learning device according to an embodiment comprises: a sample acquisition unit; a label acquisition unit; a similarity calculation unit; and a loss function calculation unit. The sample acquisition unit acquires M second mixed feature representations that are extracted from M first mixed feature representations. The label acquisition unit acquires M mixed labels in which N labels are mixed. The similarity calculation unit calculates L similarities of L label combinations selected from M mixed labels. Each of L label combinations has K mixed labels selected from M mixed labels, each of L similarities expresses a degree of similarity among K mixed labels, and L similarities are formulated on the basis of qualitative and quantitative constraint condition to be satisfied by the similarities. The loss function calculation unit calculates a loss function on the basis of M second mixed feature representations and L similarities.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] An embodiment of the present invention relates to an expression learning device, method, and program. [Background technology]

[0002] In statistical machine learning models, especially deep learning models, many attempts have been made to improve model performance by using representation learning, which designs loss functions using feature vectors themselves. It is also known that mixed data augmentation, which augments samples by mixing multiple samples, is effective in ensuring sample diversity during training. Attempts have also been made to introduce mixed data augmentation with label information into representation learning. However, it has not been possible to properly evaluate the similarity between the mixed labels of the mixed samples, and therefore it has not been possible to properly design loss functions. [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] Kibok Lee, Yian Zhu, Kihyuk Sohn, Chun-Liang Li, Jinwoo Shin, and Honglak Lee, “i-Mix: A Domain-Agnostic Strategy for Contrastive Representation Learning,” in Proceedings of the Ninth International Conference on Learning Representations (ICLR 2021). [Non-patent document 2] Yangkai Du, Tengfei Ma, Lingfei Wu, Fangli Xu, Xuhong Zhang, Bo Long, and Shouling Ji, “Constructing Contrastive Samples via Summarization for Text Classification with Limited Annotations,” in Findings of the Association for Computational Linguistics: EMNLP 2021, pages 1365-1376, Punta Cana, Dominican Republic. Association for Computational Linguistics. Summary of the Invention [Problem to be solved by the invention]

[0004] The problem to be solved by the present invention is to provide an expression learning device, method, and program that can improve the accuracy of a loss function calculated in expression learning. [Means for solving the problem]

[0005] An expression learning device according to an embodiment includes a sample acquisition unit, a label acquisition unit, a similarity calculation unit, and a loss function calculation unit. The sample acquisition unit acquires M first mixed feature representations obtained by mixing feature representations of N samples, M second mixed feature representations extracted from the M first mixed feature representations, M third mixed feature representations that are feature representations extracted from M mixed samples obtained by mixing N samples, or M fourth mixed feature representations that are feature representations extracted from the M third mixed feature representations. The label acquisition unit acquires M mixed labels obtained by mixing N labels corresponding to the N samples. The similarity calculation unit calculates L similarities of L label combinations selected from the M mixed labels, where each of the L label combinations has K mixed labels selected from the M mixed labels, and each of the L similarities represents a degree of similarity between the K mixed labels, and the L similarities are formulated based on qualitative and quantitative constraints that the similarities must satisfy. The loss function calculation unit calculates a loss function based on any one of the first to fourth mixed feature representations and the L similarities. [Brief explanation of the drawings]

[0006] [Figure 1] FIG. 1 is a diagram showing an example of the configuration of an expression learning device according to an embodiment of the present invention; [Figure 2] FIG. 1 is a diagram showing the procedure of an expression learning process according to the first embodiment; [Figure 3] FIG. 1 is a diagram schematically illustrating an overall image of an expression learning process according to the first embodiment. [Figure 4] FIG. 1 is a diagram schematically illustrating a design example of a loss function according to the present embodiment. [Figure 5] Diagram showing the matrix structure of the similarity measure cSimCLR when non-supervised labels are used. [Figure 6] Diagram showing the matrix structure of the similarity measure cSupCLR when teacher labels are used. [Figure 7] FIG. 10 is a diagram showing a comparison result between similarity Z according to this embodiment and similarity X according to Comparative Example 1. [Figure 8] FIG. 10 is a diagram showing a comparison result between similarity Z according to this embodiment and similarity Y according to Comparative Example 2. [Figure 9] FIG. 10 is a diagram showing classification accuracy rates of deep learning models according to the present embodiment, comparative example 1, and comparative example 2. [Figure 10] A diagram showing the scores for out-of-distribution detection for multiple detection targets. [Figure 11] FIG. 10 is a diagram showing the procedure of an expression learning process according to the second embodiment. [Figure 12] FIG. 10 is a diagram showing the procedure of an expression learning process according to the third embodiment. [Figure 13] FIG. 10 is a diagram showing the procedure of an expression learning process according to the fourth embodiment. [Figure 14] FIG. 10 is a diagram showing the procedure of an expression learning process according to the present embodiment. [Figure 15] FIG. 10 is a diagram showing an example of the configuration of an expression learning device according to an application example. [Figure 16] FIG. 1 is a diagram showing a schematic example of a design of a loss function related to an application example. DETAILED DESCRIPTION OF THE INVENTION

[0007] Hereinafter, an expression learning device, method, and program according to this embodiment will be described with reference to the drawings.

[0008] Fig. 1 is a diagram showing an example of the configuration of an expression learning device 100 according to this embodiment. As shown in Fig. 1, the expression learning device 100 is a computer having a processing circuit 1, a storage device 2, an input device 3, a communication device 4, and a display device 5. Data communication between the processing circuit 1, the storage device 2, the input device 3, the communication device 4, and the display device 5 is performed via a bus.

[0009] The processing circuit 1 includes a processor such as a CPU (Central Processing Unit) and a memory such as a RAM (Random Access Memory). The processing circuit 1 includes a sample acquisition unit 11, a label acquisition unit 12, a similarity calculation unit 13, a loss function calculation unit 14, an optimization unit 15, and an output control unit 16. The processing circuit 1 executes an expression learning program to realize the functions of the above-mentioned units 11 to 16. The expression learning program is stored in a non-transitory computer-readable recording medium such as a storage device 2. The expression learning program may be implemented as a single program that describes all the functions of the above-mentioned units 11 to 16, or may be implemented as multiple modules divided into several functional units. Furthermore, the above-mentioned units 11 to 16 may be implemented by an integrated circuit such as an application-specific integrated circuit (ASIC). In this case, they may be implemented on a single integrated circuit or individually on multiple integrated circuits.

[0010] The sample acquisition unit 11 acquires M (M is a natural number) mixed feature representations obtained by mixing feature representations of N (N is a natural number) samples. 、 The processing circuit 1 acquires M other mixed feature representations extracted from the M mixed feature representations, M mixed feature representations that are feature representations extracted from M mixed samples obtained by mixing the M samples, or M other mixed feature representations that are feature representations extracted from the M mixed feature representations. The sample acquisition unit 11 can also acquire N samples, mixed samples, or feature representations. A method of extracting a feature representation from a sample or a method of extracting a mixed feature representation from a mixed sample is assumed to use a statistical machine learning model such as a deep learning model. The sample refers to data to be processed by the processing circuit 1. The sample may be any data that can be labeled, and examples of such data include image data, audio data, character data, and waveform data.

[0011] The label acquiring unit 12 acquires M mixed labels by mixing N labels corresponding to the N samples, respectively. The label acquiring unit 12 can also acquire N labels.

[0012] The similarity calculation unit 13 calculates L similarities of L (L is a natural number) combinations (hereinafter referred to as label combinations) selected from M mixed labels. Each of the L label combinations has K (K is a natural number) mixed labels selected from the M mixed labels. Each of the L similarities represents the degree of similarity between the K mixed labels. The L similarities are formulated based on qualitative and quantitative constraints that the similarities must satisfy.

[0013] The loss function calculation unit 14 calculates a loss function based on any one of the first to fourth M mixed feature representations and the L similarities. More specifically, the loss function calculation unit 14 selects L combinations (hereinafter, sample combinations) from any one of the first to fourth M mixed feature representations. Here, each of the L sample combinations has K mixed feature representations. The loss function calculation unit 14 calculates correlations between the K mixed feature representations for each of the L sample combinations. The loss function calculation unit 14 calculates the loss function based on the L correlations and the L similarities.

[0014] The optimization unit 15 updates the deep learning model based on the loss function. The optimization unit 15 determines whether a stopping condition for the update is satisfied, and repeats updating the deep learning model until it determines that the stopping condition is satisfied. If the optimization unit 15 determines that the stopping condition is satisfied, it outputs the deep learning model at the current stage as a trained deep learning model.

[0015] The output control unit 16 outputs various data. For example, the output control unit 16 may display samples, mixed samples, feature representations, mixed feature representations, similarities, learning results of deep learning models, trained deep learning models, etc. on the display device 5, store them in the storage device 2, or transmit them to another computer via the communication device 4.

[0016] The storage device 2 is configured by a ROM (Read Only Memory), an HDD (Hard Disk Drive), an SSD (Solid State Drive), an integrated circuit storage device, etc. The storage device 2 stores an expression learning program and the like.

[0017] The input device 3 inputs various commands from a user. Examples of the input device 3 that can be used include a keyboard, a mouse, various switches, a touchpad, and a touch panel display. An output signal from the input device 3 is supplied to the processing circuit 1. Note that the input device 3 may also be an input device of a computer connected to the processing circuit 1 via a wired or wireless connection.

[0018] Communication device 4 is an interface for performing data communication between expression learning device 100 and external devices connected to the device via a network.

[0019] The display device 5 displays various information. For example, the display device 5 displays various data under the control of the output control unit 16. As the display device 5, a CRT (Cathode-Ray Tube) display, a liquid crystal display, an organic EL (Electro Luminescence) display, an LED (Light-Emitting Diode) display, a plasma display, or any other display known in the art can be used as appropriate. The display device 5 may also be a projector.

[0020] Various examples according to this embodiment will be described below.

[0021] [Example 1] Fig. 2 is a diagram illustrating a procedure of an expression learning process according to Example 1. Fig. 3 is a diagram illustrating an overall image of the expression learning process according to Example 1.

[0022] 2 and 3, the sample acquisition unit 11 acquires N samples 310 (S201). The N samples 310 constitute one mini-batch. The N samples 310 may all be from different sources, or may be a base sample and a sample that has undergone data augmentation such as geometric transformation or brightness value transformation.

[0023] After step S201, the label acquisition unit 12 acquires N labels 340 (step S202). The N labels 340 correspond one-to-one to the N samples 310. The type of the labels 340 is not particularly limited, and at least one of supervised labels, self-supervised labels (or unsupervised labels), and semi-supervised labels can be used. In other words, this embodiment can be implemented regardless of the label type. A supervised label refers to a label artificially assigned to the sample 310. A self-supervised label refers to a pseudo-label assigned based on the distribution of feature representations of the sample 310 or the type of the sample 310. For example, it refers to a pseudo-label assigned to data without a supervised label by clustering feature representations. A typical self-supervised representation learning method, called SimCLR, applies multiple transformations T1 and T2 to each data set, and considers data that have undergone different transformations to belong to the same class, while considering data generated from different data sets to belong to different classes. A semi-supervised label is a pseudo-label assigned to a sample without a supervised label based on the distribution of feature representations of other samples with supervised labels, the distribution of feature representations of samples without supervised labels, and the type of sample. For example, it is a label assigned to data without a supervised label by clustering feature representations while referring to data with a small number of supervised labels. Semi-supervised labels and self-supervised labels are sometimes called non-supervised labels.

[0024] After step S202 is performed, the sample acquisition unit 11 uses the deep learning model 30 to extract N feature representations from the N samples 310 (step S203). The deep learning model 30 is a pre-learning model. The deep learning model 30 is not particularly limited and may perform any task, such as out-of-distribution (OOD) detection, image classification, object detection, speech recognition, or VQA (Visual Question Answering).

[0025] When step S203 is performed, the sample acquisition unit 11 uses the deep learning model 30 to generate M first mixed feature representations 320 from the N feature representations (step S204). To ensure the diversity of the mixed feature representations, it is desirable that the number of mixed feature representation candidates is larger than N, while to avoid computational complexity, it is desirable that M is a value similar to N. In the embodiment, two of the N feature representations are randomly extracted without overlap to form a unit, and two mixed feature representations are obtained for one unit. When forming another unit after the formation of a unit, a constraint is imposed so that a previously extracted feature representation is not extracted more than once. Therefore, in the embodiment, the number of mixed feature representation candidates is N The number of mixed feature representations actually used is M = N. The method for mixing feature representations is not particularly limited, and it is possible to use methods such as Mixup, CutMix, and Manifold Mixup. Furthermore, the sample combinations used to generate a mixed feature representation are not limited to those described above, and it is also possible to mix the same feature representations, to form multiple units without any constraints, or to extract three or more of the N feature representations to form one unit.

[0026] After step S204, the label acquisition unit 12 generates M mixed labels 350 from the N labels 340 (step S205). The label acquisition unit 12 involved in step S205 is realized by the label mixing module 31. The label mixing module 31 is a program module independent of the deep learning model 30. The label mixing module 31 mixes the labels 340 using a method corresponding to the mixing of feature representations in step S204. In the embodiment, N / 2 units are formed from the N labels 340 using a method similar to the unit formation performed for the feature representations, and two mixed labels 350 are obtained for each unit, thereby generating N mixed labels 350. The mixing method is not particularly limited, and methods described in Mixup, CutMix, Manifold Mixup, etc. can be used. However, the label combination method needs to be performed using a method similar to the sample combination method.

[0027] After step S205 is performed, the sample acquisition unit 11 extracts M second mixed feature representations 330 from the M first mixed feature representations 320 using the deep learning model 30 (step S206).

[0028] Here, we will explain the configuration of deep learning model 30. As shown in Fig. 3, deep learning model 30 has feature extraction layer 301, feature representation mixing module 302, and feature extraction layer 303. In the training stage, the network layer (head) subsequent to feature extraction layer 303, which contributes greatly to the execution of the task of deep learning model 30, is removed.

[0029] The feature extraction layer 301 is a network layer that inputs samples 310 and extracts feature vectors from the samples 310. The feature vectors are used as feature representations. The feature extraction layer 301 has one or more arbitrary network layers, such as a convolutional layer, a fully connected layer, a multilayer perceptron, a pooling layer, or a DENS layer. ResNet, VGGNet, or the like may be used as such a network layer. In step S203, the feature representation mixing module 302 extracts feature representations from each of the N samples 310.

[0030] The feature representation mixing module 302 is provided after the feature extraction layer 301. The feature representation mixing module 302 is a program module that generates M first mixed feature representations from N feature representations in step S204.

[0031] The feature extraction layer 303 is provided after the feature representation mixing module 302. The feature extraction layer 303 is a network layer that receives the first mixed feature representation 320 and extracts a feature vector from the first mixed feature representation 320. The feature vector is used as a second mixed feature representation 330, which is a feature representation of the first mixed feature representation. The feature extraction layer 303 has one or more arbitrary network layers, such as a convolutional layer, a fully connected layer, a multilayer perceptron, a pooling layer, or a DENS layer. As such a network layer, ResNet, VGGNet, or the like may be used. In step S206, the feature extraction layer 303 extracts the second mixed feature representation 330 from each of the M first mixed feature representations 320.

[0032] As described above, in step S204, the sample acquisition unit 11 inputs N samples to the first network layer (feature extraction layer 301) included in the deep learning model 30 to extract N first feature representations, and in step S206, inputs M first mixed feature representations to the second network layer (feature extraction layer 303) located after the first network layer (feature extraction layer 301) included in the deep learning model 30 to extract M second mixed feature representations.

[0033] After step S206 is performed, the similarity calculation unit 13 calculates L similarities 360 from the M mixed labels 350 (step S207). The similarity calculation unit 13 involved in step S207 is realized by the similarity calculation module 32. The similarity calculation module 32 is assumed to be a program module independent of the deep learning model 30.

[0034] The similarity calculation module 32 calculates L similarities 360 for L label combinations selected from M mixed labels 350. Each of the L label combinations has K mixed labels selected from the M mixed labels. Each of the L similarities represents the degree of similarity between the K mixed labels in each of the L label combinations. In this embodiment, when combining labels, K labels are randomly extracted from the M mixed labels without overlapping, and all combinations of these labels are created. In this case, there are L label combinations = M C K The value of K is equal to the number of mixed feature representations for which correlation is calculated in the loss function calculation unit 14. In the loss function used in representation learning, K=2 is typical, but this is not limited to this. Furthermore, it is also possible to create the loss function by taking into consideration not only the combination but also the order, or by extracting and creating the loss function with overlaps.

[0035] Here, we will explain label combinations and similarities assuming that K=2. A label combination consisting of two mixed labels is denoted as (MLi, MLj). Note that i and j are subscripts that represent the mixed labels. For the label combination (MLi, MLj), the similarity of MLi to MLj is calculated. Specifically, consider the case of three (M=3) mixed labels ML1, ML2, and ML3. From the three mixed labels ML1, ML2, and ML3, L=3 (=3C2) label combinations (MLi, MLj)=(ML1, ML2), (ML1, ML3), (ML2, ML3) are created, and the similarity is calculated for each label combination.

[0036] After step S207 is performed, the loss function calculation unit 14 calculates the loss function (step S208). The loss function calculation unit 14 in step S208 is realized by the loss function calculation module 33. The loss function calculation module 33 is assumed to be a program module independent of the deep learning model 30.

[0037] In step S208, the loss function calculation module 33 calculates a loss function based on the M second mixed feature representations 330 and the L similarities 360 extracted in step S206, and calculates a loss 370. More specifically, the loss function calculation module 33 selects L sample combinations from the M second mixed feature representations 330. Here, each of the L sample combinations has K mixed feature representations. The loss function calculation module 33 calculates correlations between the K mixed feature representations for each of the L sample combinations. The loss function calculation module 33 calculates the loss function based on the L correlations and the L similarities.

[0038] Here, the label combinations and similarities will be described assuming that K=2. Each of the L correlations represents the degree of correlation between two second mixed feature representations in each of the L sample combinations. Here, a sample combination consisting of two second mixed feature representations is denoted as (MFRi, MFRj). i and j are subscripts representing the second mixed feature representations. For a sample combination (MFRi, MFRj), the correlation of MFRi with MFRj is calculated. Specifically, consider the case of three (M=3) mixed feature representations MFR1, MFR2, and MFR3. From the three mixed feature representations MFR1, MFR2, and MFR3, L=3 (=3C2) sample combinations (MFRi, MFRj) = (MFR1, MFR2), (MFR1, MFR3), and (MFR2, MFR3) are created, and the correlation is calculated for each sample combination.

[0039] Next, the design of the loss function according to this embodiment will be described in detail. FIG. 4 is a diagram schematically illustrating an example of the design of the loss function according to this embodiment. As shown in FIG. 4, the loss function is expressed by the product of the similarity and the contrastive loss term. As described above, the similarity represents the degree of similarity between the K mixed labels in each of the L label combinations. The contrastive loss term is described by the correlation between the K second mixed feature representations in each of the L sample combinations.

[0040] As shown in Figure 4, similarity is formulated based on qualitative and quantitative constraints that the similarity must satisfy. . exchange In other words, the L similarities are derived based on constraints that an ideal similarity must satisfy. Specific contents of the constraints vary. For example, the constraints include: the similarity takes a lower limit when the K mixed labels are identical (first condition); the similarity takes an upper limit when the K mixed labels are most similar, and the similarity takes a lower limit when the K mixed labels are least similar (second condition); and / or the similarity takes the same value when the K mixed labels are symmetric (third condition). Regarding the first and second conditions, the specific values ​​of the lower and upper limits are not particularly limited, but may be set, for example, as the lower limit = 0 and the upper limit = 1. The symmetry regarding the third condition means, for example, that when K=2, the similarity of MLj to MLi and the similarity of MLi to MLj are interchangeable.

[0041] The above first to third conditions can be expressed by the following formulas: ij represents the similarity of mixed label i to mixed label j. i ´ represents the value of the mixed label i. i ' is typically represented as a vector in 1-of-K format.

[0042]

number

[0043] The similarity may be calculated by a matrix structure depending on whether the label is a teacher label or not. Figure 5 shows the similarity c when a non-teacher label is used. SimCLR Fig. 6 shows the matrix structure of the similarity c when the teacher label is used. SupCLR 1 is a diagram showing the matrix structure of the deep learning model. The non-supervised label refers to a semi-supervised label or a self-supervised label. When the label is a non-supervised label, unsupervised contrast learning is performed as the learning algorithm of the deep learning model. When the label is a supervised label, supervised contrast learning is performed as the learning algorithm of the deep learning model.

[0044] The examples in Figures 5 and 6 show the matrix structure of similarity between labels corresponding to four sample images Img1, Img2, Img3, and Img4. The labels of sample images Img1 and Img2 are classified into class 1, and the labels of sample images Img3 and Img4 are classified into class 2. The similarity c SimCLR and similarity c SupCLR Each matrix is ​​divided into four 2x2 matrix blocks for each row and column. The first row of the matrix block represents the reference sample, and the second row represents the sample obtained by applying a transformation to the reference sample. Similarly, the matrix block The first column of represents the reference sample, and the second column represents the sample obtained by modifying the reference sample. The diagonal elements (matrix blocks surrounded by dotted lines, e.g., row 1, column 1, row 2, column 2) represent the similarity between the same labels, and are therefore set to a value of 0 according to the first condition. The non-diagonal elements (e.g., row 2, column 1, row 1, column 2) represent the similarity between different labels belonging to the same class, and are therefore set to a value of 1 according to the second condition. In addition, the similarity c SimCLR and similarity c SupCLR According to the third condition, each of them has the same similarity with respect to the exchangeable matrix blocks.

[0045] In the case of unsupervised control learning shown in FIG. 5, as shown in the matrix blocks in the off-diagonal elements, the similarities between different samples, whether they are in the same class or different classes, are all designed to be 0. In the case of supervised control learning shown in FIG. 6, for example, as shown in the matrix blocks illustrated by the dashed dotted line (e.g., rows 2 and column 1, and rows 1 and column 2), the similarities between different samples belonging to the same class are designed to be 1 due to the second condition, since both labels are most similar. For example, as shown in the matrix block of rows 3 and column 1, the similarities between different samples belonging to different classes are designed to be 0 due to the second condition, since both mixed labels are least similar.

[0046] Similarity c that satisfies the first to third conditions ij For example, it can be formulated based on the Jensen-Shannon divergence shown in the following equation (4) or the cosine similarity shown in the following equation (5). H(·) in equation (4) represents the Shannon entropy.

[0047]

number

[0048] As an example of the loss function, the loss function L of supervised control learning expressed by the following equation (6) is used. (sup) The loss function L (sup) is the similarity c ij and correlation s(f i ,f j ) and the correlation s(f i ,f j ) is the feature vector f i The feature vector f j represents the correlation with c ij Normalized c ij ´=c ij / Σ j c ij may be used as the similarity.

[0049]

number

[0050] The loss function is not limited to equation (6). For example, the loss function L shown in the following equation (7), which is a generalization of MoCo (momentum contrast), can be used. (MoCo) may be used. i is the feature representation obtained by inputting the data subjected to the transformation T1 into the deep learning model. j ~EMA is f i It is obtained by inputting samples from a deep learning model with the same initial values ​​as those used to obtain μ into a model updated by an exponential moving average. The input samples are the same as the original samples but are modified by a different transformation T2 than T1. l is the f obtained from the previous iteration step j ~EMA μ l is stored in the storage device 2.

[0051]

number

[0052] The loss function may be, for example, a generalized version of BYOL (bootstrap your own latent), as shown in the following equation (8): g(f i ) is f i represents the output from the accompanying multilayer perceptron after extracting

[0053]

number

[0054] Any other loss function may be used as long as it utilizes the similarity between multiple feature vectors. Furthermore, the constraints are not limited to the first to third conditions, and may be designed according to the type of task of the deep learning model or the type of loss function.

[0055] After step S208 is performed, the optimization unit 15 updates the deep learning model 30 (step S209). The optimization unit 15 involved in step S209 is realized by the optimization module 34. The optimization module 34 updates weight parameters set in a network layer to be trained among the network layers included in the deep learning model 30. An example of the network layer to be trained is the feature extraction layer 303. The optimization module 34 updates the weight parameters according to an adaptively set update width (step width) so as to minimize the loss function 370 calculated in step S208. The optimization method is not particularly limited, and any method such as stochastic gradient descent, AdaGrad, or Adam may be used.

[0056] After step S209, the optimization unit 15 determines whether to terminate the update (step S210). The optimization unit 15 involved in step S210 is realized by the optimization module 34. The optimization module 34 determines whether a stopping condition for the update is satisfied. The stopping condition may be, for example, that the number of iterations of the processes from step S201 to step S210 has reached a predetermined number, or that the loss 370 has fallen below a predetermined value. If it is determined that the stopping condition is not satisfied (step S210: NO), steps S201 to S210 are executed using new N samples 310 and N labels 340, and steps S201 to S210 are repeated until it is determined that the stopping condition is satisfied. If it is determined that the stopping condition is satisfied (step S210: YES), the optimization module 34 terminates the expression learning process. The output control unit 16 outputs the deep learning model in the iteration step in which it is determined that the stopping condition is satisfied as a trained deep learning model.

[0057] This completes the expression learning process according to the first embodiment.

[0058] The procedure of the expression learning process shown in Fig. 2 above is an example, and is not limited to the procedure shown in Fig. 2. As one example, step S202 may be performed before step S205, and does not necessarily have to be performed between steps S201 and S203. As another example, step S205 may be performed between steps S202 and S207, and does not necessarily have to be performed before step S206.

[0059] Next, the difference in accuracy between the similarity according to this embodiment and the similarity according to the comparative examples will be evaluated. Comparative Example 1 according to Non-Patent Document 1 and Comparative Example 2 according to Non-Patent Document 2 will be given as the comparative examples.

[0060] Comparative Example 1 attempts to improve the accuracy of the loss function by combining representation learning and sample mixing. Comparative Example 1 uses the pre-mixing label and the post-mixing label to calculate L in the following equation (9): (1) We introduce hybrid data augmentation into representation learning using a loss function equivalent to

[0061]

number

[0062] Here, the original sample x i and x j is subjected to deformation T, and the deformed sample T(x i ) and T(x k ) is weighted by the coefficient λ to obtain the mixed sample x i By generating mixed samples for all samples and inputting them into the deep learning model, the feature vector f i and the correlation s(f i ,f j ) is calculated. In Comparative Example 1, the method of reflecting the sample mixing in the loss function is to calculate the label y i ´ and the pre-mixed label y j Finally, we normalize the correlation between feature vectors using a hyperparameter called temperature, τ, and calculate the loss function L as the sum of the cross entropies of the softmax probabilities.(1) is given.

[0063] The loss function L according to this embodiment (sup) and the loss function L according to Comparative Example 1 (1) When compared with the above, the similarity according to Comparative Example 1 is expressed by the following formula (10).

[0064]

number

[0065] FIG. 7 is a diagram showing the results of comparing the similarity Z according to this embodiment with the similarity X according to Comparative Example 1. The similarity Z is expressed by equation (4), and the similarity X is expressed by equation (10). The samples are images, and their labels y are expressed by three-dimensional vectors. As shown in the left column of FIG. 7, sample A is an image of a car, and its label y A Similarly, sample B is an image of a bird, and its label y B is (0,1,0), sample C1 is an image of a cat, and its label y C1 is (0,0,1), sample C2 is an image of a cat, and its label y C2 is (0,0,1). Sample C1 and sample C2 are both images labeled "cat", but they are images of different cats. The mixed label y' is calculated by dividing the pre-mixed label y i and y j The weight value λ is called the mixture ratio.

[0066]

number

[0067] As shown in the middle column of Figure 7, according to equation (11), the mixed label y A ´ is (λ,0,1-λ), and the mixed label y of sample B´ is the mixed sample of sample C2 against sample B. B´ is (0,λ,1-λ), and the mixed label y of sample C1´ is the mixed sample of sample A with respect to sample C1. C1 ´ is (1-λ,0,λ), and the mixed label y of sample C2´ is the mixed sample of sample B for sample C2. C2 ´ is expressed as (0,1-λ,λ).

[0068] As shown in the right column of Figure 7, the mixed label y A Mixed Labels B Similarity C to ´ A´B´ , mixed label y A Mixed Labels C1 Similarity C to ´ A´C1´ and mixed label y A Mixed Labels C2 Similarity C to ´ A´C2´ was calculated using the similarity Z (formula (4)) according to this embodiment and the similarity X (formula (11)) according to Comparative Example 1. The mixture ratio λ was set to 0.5.

[0069] As shown in the upper right column of Figure 7, the mixed label y A ´ and mixed labels B ´ contains a cat, so C A´B´ ≠0, but in Comparative Example 1, C A´B´ = 0. This is qualitatively inappropriate. Also, the mixed label y A ´ and mixed labels C1 ´ includes cars and mixed labels C2 ´ does not include cars, so C A´C1´ >C A´C2´ However, in Comparative Example 1, C A´C1´ =C A´C2´ =0.5, which is quantitatively inappropriate. Such inappropriate results are obtained because, as shown in equation (11), in Comparative Example 1, the label y i On the other hand, as shown in the lower right column of FIG. 7, in this embodiment, the similarity is designed using C A´B´ =0.25, and as expected, C A´B´ ≠0. In this embodiment, C A´C1´ =0.5,C A´C2´=0.25, and as expected, C A´C1´ >C A´C2´ As described above, it can be seen that the similarity Z according to this embodiment is appropriate both qualitatively and quantitatively. This is because the similarity Z according to this embodiment is formulated based on qualitative and quantitative constraints.

[0070] In Comparative Example 2, the inner product between the mixed labels is used to calculate the loss function L shown in the following equation (12): (2) We introduce hybrid data augmentation to representation learning by i and mixed label y i The treatment for obtaining the above-mentioned compound is the same as that in Comparative Example 1.

[0071]

number

[0072] The loss function L according to this embodiment (sup) and the loss function L according to Comparative Example 2 (2) When compared with the above, the similarity according to Comparative Example 2 is expressed by the following formula (13).

[0073]

number

[0074] FIG. 8 is a diagram showing the results of comparing similarity Z according to this embodiment with similarity Y according to comparative example 2. Similarity Z is expressed by equation (4), and similarity Y is expressed by equation (12). The symbols in FIG. 8 are assigned according to the same rules as those in FIG. 7. Note that sample C1, sample C2, and sample C3 are all images labeled "cat," but are images of different cats.

[0075] As shown in the middle column of Fig. 8, the mixed label y A ´ is (λ,0,1-λ), and the mixed label y of sample B´ of sample A with respect to sample C1 C1´ is (1-λ,0,λ), and the mixed label y of sample C2´ is the mixed sample of sample C3 with sample C2. C2 ´ is (0,0,1), and the mixed label y C3 ´ is represented as (0,0,1).

[0076] As shown in the right column of Figure 8, the mixed label y A Mixed Labels C1 Similarity C to ´ A´C1´ , mixed label y A Mixed Labels C2 Similarity C to ´ A´C2´ and mixed label y A Mixed Labels C3 Similarity C to ´ A´C3´ was calculated using the similarity Z (formula (4)) according to this embodiment and the similarity Y (formula (13)) according to Comparative Example 2.

[0077] As shown in the upper right column of Figure 8, the mixed label y A ´ and mixed labels y C1 ´ includes cars and mixed labels C2 ´ and mixed labels y C3 ´ does not include cars, so C A´C1´ >C A´C2´ =C A´C3´ However, in Comparative Example 2, C A´C1´ =C A´C2´ =C A´C3´ =0.33. This is quantitatively inappropriate. Such an inappropriate result is obtained because, as shown in equation (13), the similarity Y is given by the inner product between the mixed labels. On the other hand, as shown in the bottom right column of FIG. 8, in this embodiment, C A´C1´ ~0.63, C A´C2´ ~0.37,C A´C3´ ~0.37, as expected, C A´C1´ >C A´C2´ =C A´C3´ Therefore, it can be seen that the similarity Z according to this embodiment is quantitatively appropriate. This is because the similarity Z according to this embodiment is formulated based on qualitative and quantitative constraints.

[0078] Next, the performance of the deep learning model trained by the expression learning process according to this embodiment will be compared with the deep learning models according to Comparative Example 1 and Comparative Example 2.

[0079] FIG. 9 is a diagram showing the classification accuracy rates of deep learning models according to this embodiment, Comparative Example 1, and Comparative Example 2. CIFAR-10 was used as the training dataset, which is a collection of samples. Supervised controlled learning was used as the learning method. The data augmentation method indicates the type of data augmentation method for the samples, with Base using four types: Random Horizontal Flip, Random Crop, Color Jitter, and Random Gray Scale. Base+Mixup uses Mixup in addition to Base, and Base+CutMix uses CutMix in addition to Base. The mixture ratio range indicates the range of possible mixture ratios λ in Mixup and CutMix. When generating each mixture feature representation and mixture label, the mixture ratio λ is determined from the mixture ratio range according to an appropriate probability distribution such as a Gaussian distribution. As shown in FIG. 9, by using mixture-type data augmentation, Classification accuracy rate It is clear that the improvement

[0080] Figure 10 shows the scores for out-of-distribution detection for multiple detection targets. CIFAR-10 was used as the training dataset. Supervised controlled learning was used as the learning method. The task of the deep learning model was out-of-distribution (OOD) detection. The score is an index used to evaluate the accuracy of out-of-distribution detection, and AUROC (Area Under Receiver Operating Characteristic curve) and AUPR (Area Under the Precision-Recall curve) were used. CIFAR-100, SVHN, MNIST, and Random were the detection targets. Base and Base+CutMix were used as data augmentation methods during deep learning training. As shown in Figure 10, it can be seen that mixed data augmentation improves out-of-distribution detection performance.

[0081] [Example 2] An expression learning device 100 according to Example 2 will be described. Unlike Example 1, the expression learning device 100 according to Example 2 does not include the step of extracting a second mixed feature representation from a first mixed feature representation (step S206). That is, the deep learning model according to Example 2 does not include a feature extraction layer 303.

[0082] 11 is a diagram showing the procedure of the expression learning process according to Example 2. Steps S1101 to S1105 are the same as steps S201 to S205 in FIG.

[0083] After step S1105 is performed, the similarity calculation unit 13 calculates L similarities from the M mixed labels (step S1106). Step S1106 is the same as step S207 in FIG.

[0084] After step S1106 is performed, the loss function calculation unit 14 calculates a loss function (step S1107). In step S1107, the loss function calculation unit 14 calculates a loss function based on the M mixed feature representations generated in step S1104 and the L similarities calculated in step S1106, thereby calculating a loss. The loss function according to the second embodiment can be calculated by replacing the second mixed feature representation according to the first embodiment with the mixed feature representation generated in step S1104.

[0085] After step S1107 is performed, the optimization unit 15 updates the deep learning model based on the loss function calculated in step S1107 (step S1108). In step S1108, the optimization unit 15 updates the weight parameters of the feature extraction layer 301 in the deep learning model. The optimization method may be the same as that in step S209.

[0086] After step S1108 is performed, the optimization unit 15 determines whether to end the update (step S1109). In step S1109, the optimization unit 15 determines whether the stop condition for the update is satisfied. If it is determined that the stop condition is not satisfied (step S1109: NO), steps S1101 to S1109 are executed using new N samples and N labels, and steps S1101 to S1109 are repeated until it is determined that the stop condition is satisfied. Then, if it is determined that the stop condition is satisfied (step S1109: YES), the optimization unit 15 ends the expression learning process. The output control unit 16 outputs the deep learning model in the iteration step where it is determined that the stop condition is satisfied as a trained deep learning model.

[0087] This completes the expression learning process according to the second embodiment.

[0088] The procedure of the expression learning process shown in Fig. 11 is an example, and is not limited to the procedure shown in Fig. 11. As an example, step S1102 may be performed before step S1105, and does not necessarily have to be performed between step S1101 and step S1103.

[0089] [Example 3] An expression learning device 100 according to a third embodiment will be described. Unlike the first embodiment, the expression learning device 100 according to the third embodiment mixes samples instead of mixing first feature expressions.

[0090] 12 is a diagram showing the procedure of the expression learning process according to Example 3. Steps S1201 to S1202 are the same as steps S201 to S202 in FIG.

[0091] After step S1202, the sample acquisition unit 11 generates M mixed samples from the N samples using a deep learning model (step S1203). K Create N sample combinations. KThe sample combination is made up of K samples selected from N samples. It is also permitted that the K samples are selected from the same sample. The sample acquisition unit 11 acquires N K The K samples that make up each of the sample combinations are mixed to obtain N K The method for mixing the samples is not particularly limited, and methods such as CutMix and Mixup can be used.

[0092] After step S1203 is performed, the sample acquisition unit 11 extracts M first mixed feature representations from the M mixed samples using the deep learning model (step S1204).

[0093] Steps S1205 to S1210 are the same as steps S205 to S210 shown in FIG. 2 . That is, in step S1205, the label acquisition unit 12 generates M first mixed feature representations from N feature representations using the deep learning model 30. In step S1206, the sample acquisition unit 11 extracts M second mixed feature representations from the M first mixed feature representations using the deep learning model. In step S1207, the similarity calculation unit 13 calculates L similarities from the M mixed labels. In step S1208, the loss function calculation unit 14 calculates a loss function based on the M second mixed feature representations and the L similarities. In step S1209, the optimization unit 15 updates the deep learning model based on the loss function. The loss function according to the third embodiment can be calculated by replacing the second mixed feature representation according to the first embodiment with the second mixed feature representation generated in step S1304. Then, in step S1210, the optimization unit 15 determines whether the stopping condition is satisfied, and repeats steps S1201 to S1210 until it is determined that the stopping condition is satisfied, and if it is determined that the stopping condition is satisfied, ends the expression learning process.

[0094] This completes the expression learning process according to the third embodiment.

[0095] The procedure of the expression learning process shown in Fig. 12 is an example, and is not limited to the procedure shown in Fig. 12. As one example, step S1202 may be performed before step S1205, and does not necessarily have to be performed between step S1201 and step S1203. As another example, step S1205 may be performed between step S1202 and step S1207, and does not necessarily have to be performed before step S1206.

[0096] [Example 4] Hereinafter, an expression learning device 100 according to Example 4 will be described. Unlike Example 3, the expression learning device 100 according to Example 4 does not include the step of extracting a second mixed feature representation from a first mixed feature representation (step S1206). That is, the deep learning model according to Example 4 does not include a feature extraction layer 303.

[0097] Fig. 13 is a diagram showing the procedure of the expression learning process according to Example 4. Steps S1301 to S1305 are the same as steps S1201 to S1205 in Fig. 12, and therefore a description thereof will be omitted.

[0098] After step S1305 is performed, the similarity calculation unit 13 calculates L similarities from the M mixed labels (step S1306). Step S1106 is the same as step S207 in FIG.

[0099] After step S1306 is performed, the loss function calculation unit 14 calculates a loss function (step S1307). In step S1307, the loss function calculation unit 14 calculates a loss function based on the M mixed feature representations generated in step S1304 and the L similarities calculated in step S1306, thereby calculating a loss. The loss function according to the fourth embodiment can be calculated by replacing the second mixed feature representation according to the first embodiment with the mixed feature representation generated in step S1304.

[0100] After step S1307 is performed, the optimization unit 15 updates the deep learning model based on the loss function calculated in step S1307 (step S1308). In step S1308, the optimization unit 15 updates the weight parameters of the feature extraction layer 301 in the deep learning model. The optimization method may be the same as that in step S209.

[0101] After step S1308, the optimization unit 15 determines whether to end the update (step S1309). In step S1309, the optimization unit 15 determines whether the stop condition for the update is satisfied. If it is determined that the stop condition is not satisfied (step S1309: NO), steps S1301 to S1309 are executed using new N samples and N labels, and steps S1301 to S1309 are repeated until it is determined that the stop condition is satisfied. If it is determined that the stop condition is satisfied (step S1309: YES), the optimization unit 15 ends the expression learning process. The output control unit 16 outputs the deep learning model in the iteration step where it is determined that the stop condition is satisfied as a trained deep learning model.

[0102] This completes the expression learning process according to the fourth embodiment.

[0103] The procedure of the expression learning process shown in Fig. 13 is an example, and is not limited to the procedure shown in Fig. 13. As an example, step S1302 may be performed before step S1305, and does not necessarily have to be performed between step S1301 and step S1303.

[0104] [Summary] In the above Examples 1 to 4, the loss function was used to update the deep learning model. However, the use of the loss function according to this embodiment is not limited to this. Furthermore, although the mixed feature representation was generated by the expression learning device 100 based on samples, it may also be generated by an external computer different from the expression learning device 100. In this case, the expression learning device 100 receives data of the mixed feature representation generated by the external computer via the communication device 4 or a portable recording medium.

[0105] FIG. 14 is a diagram showing the procedure of the expression learning process according to this embodiment. As shown in FIG. 14, the sample acquisition unit 11 acquires M mixed feature representations based on N samples (step S1401). The term "acquire" according to this embodiment refers to acquisition in a broad sense, generally referring to the expression learning device 100 acquiring the data. That is, the term "acquire" according to this embodiment also includes generation or calculation by the expression learning device 100 and reception from an external computer. The mixed feature representation acquired in step S1401 may be any of the first mixed feature representation according to Example 1, the second mixed feature representation according to Example 1, the mixed feature representation according to Example 2, the first mixed feature representation according to Example 3, the second mixed feature representation according to Example 3, and the mixed feature representation according to Example 4.

[0106] After step S1401 is performed, the label obtaining unit 12 obtains M mixed labels based on the N labels (step S1402). The label obtaining unit 12 in step S1402 may generate a mixed label by mixing labels as shown in the above first to fourth embodiments, or may receive a mixed label from an external computer.

[0107] After step S1402 is performed, the similarity calculation unit 13 calculates the similarity of L label combinations from the M mixed labels acquired in step S1402 (step S1403). The method of calculating the similarity in step S1403 is the same as the method of calculating the similarity in the above-mentioned first to fourth embodiments.

[0108] After step S1403 is performed, the loss function calculation unit 14 calculates a loss function based on the M mixed feature representations acquired in step S1401 and the L similarities calculated in step S1403 (step S1404). The method of calculating the loss function in step S1404 is the same as the method of calculating the loss function in the above-mentioned first to fourth embodiments.

[0109] When step S1404 is performed, the expression learning process according to this embodiment ends. The loss obtained by calculating the loss function may be used to update the deep learning model as shown in Examples 1 to 4, or the loss data may be transferred to another computer when the other computer updates the deep learning model. The loss may also be used for other purposes. For example, when extracting abnormal samples for an existing deep learning model, the magnitude of the loss may be used to determine whether the sample is abnormal.

[0110] The expression learning process according to this embodiment is not limited to the various examples described above. As an example, in the above example, the similarity calculation unit 13 calculates L similarities for L label combinations selected from M mixed labels. However, the similarity calculation unit 13 calculates L similarities for L label combinations selected from M mixed labels and N labels. In this case, each of the L label combinations includes a mixed label and a label. In other words, not only the similarity between mixed labels but also the similarity between a mixed label and a label may be calculated.

[0111] (Application example) In the above embodiment, the loss function was assumed to be designed in advance. The expression learning device 100 according to the application example designs the loss function according to various requirements. The expression learning device 100 according to the application example will be described below. In the following description, components having substantially the same functions as those in this embodiment will be given the same reference numerals and will be described only when necessary.

[0112] Fig. 15 is a diagram showing an example of the configuration of an expression learning device 100 according to an application example. As shown in Fig. 15, the processing circuit 1 further includes a design unit 17. The design unit 17 designs a loss function.

[0113] FIG. 16 is a diagram schematically illustrating a design example of a loss function according to an application example. As illustrated in FIG. 16, the loss function according to the application example is a function in which the similarity multiplied by the weight value acts on the contrast loss term. The design unit 17 designs the weight value based on the label type. The label type refers to the type of mixed label, specifically, any of a supervised label, a semi-supervised label, and an auto-supervised label. The design unit 17 sets the weight value to a value according to the label type. More specifically, the design unit 17 designs the weight value according to the combination of mixed label types regarding the first mixed label and the second mixed label that constitute the label combination. When the first mixed label and the second mixed label are both supervised labels or auto-supervised labels, the weight value of the similarity of the first mixed label to the second mixed label and the weight value of the similarity of the second mixed label to the first mixed label are set to the same value. When the first mixed label is a teacher label and the second mixed label is an auto-supervised label or a semi-supervised label, the weight value of the similarity of the first mixed label to the second mixed label is set to a larger value than the weight value of the similarity of the second mixed label to the first mixed label, because the teacher label is expected to be more reliable than the auto-supervised label or the semi-supervised label.

[0114] By designing similarity weights according to label type in this way, the accuracy of the loss function is improved, and ultimately, the accuracy of the deep learning model is expected to improve.

[0115] As shown in FIG. 16 , the design unit 17 designs the contrast loss term based on a correlation calculation method. Specifically, the correlation calculation method refers to the type of correlation definition formula in the contrast loss term, such as the above formulas (6), (7), and (8). The type of correlation definition formula may be set to any type by the user via the input device 3, or may be automatically set according to the task of the deep learning model, the characteristics of the samples, etc. Furthermore, the design unit 17 may design the contrast loss term based on the number K of elements constituting the sample combination and the label combination, as a correlation calculation method. The correlation definition formula in the contrast loss term is determined according to the number K. The number K may be set to any type by the user via the input device 3, or may be automatically set.

[0116] As described above, according to the application example, the weights, similarities, and contrast loss terms that constitute the loss function can be arbitrarily designed by the design unit 17. By providing the design unit 17, it becomes possible to flexibly design the loss function according to the samples, labels, task of the deep learning model, etc. This is expected to improve the accuracy of the similarities and loss functions, and ultimately improve the accuracy of representation learning.

[0117] Although several embodiments of the present invention have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These novel embodiments can be embodied in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their modifications are included within the scope and spirit of the invention, and are also included in the scope of the invention and its equivalents as defined in the claims. [Explanation of symbols]

[0118] 1...processing circuit, 2...memory device, 3...input device, 4...communication device, 5...display device, 11...sample acquisition unit, 12...label acquisition unit, 13...similarity calculation unit, 14...loss function calculation unit, 15...optimization unit, 16...output control unit, 17...design unit, 100...representation learning device.

Claims

1. a sample acquisition unit that acquires M first mixed feature representations obtained by mixing feature representations of N samples, M second mixed feature representations extracted from the M first mixed feature representations, M third mixed feature representations that are feature representations extracted from M mixed samples obtained by mixing N samples, or M fourth mixed feature representations that are feature representations extracted from the M third mixed feature representations; a label acquisition unit that acquires M mixed labels by mixing N labels corresponding to the N samples, respectively; a similarity calculation unit that calculates L similarities of L label combinations selected from the M mixed labels, each of the L label combinations having K mixed labels selected from the M mixed labels, each of the L similarities representing a degree of similarity between the K mixed labels, and the L similarities being formulated based on qualitative and quantitative constraints that the similarities should satisfy; a loss function calculation unit that calculates a loss function based on any one of the first to fourth mixed feature representations and the L similarities; Equipped with the similarity calculation unit calculates the L similarities formulated based on Jensen-Shannon divergence or cosine similarity. Expression learning device.

2. The constraint is: the similarity takes a lower limit value when the K mixed labels are identical; the similarity takes an upper limit value when the K mixed labels are most similar; The similarity takes a lower limit value when the K mixed labels are least similar, and / or When the K mixed labels have symmetry, the similarity has the same value.

2. The expression learning device according to claim 1.

3. The sample acquisition unit taking the N samples; extracting N first feature representations from the N samples; blending the N first feature representations to generate the M first blended feature representations; extracting the M second mixed feature representations from the M first mixed feature representations; The label acquisition unit obtaining the N labels corresponding to the N samples; mixing the N labels to generate the M mixed labels; The loss function calculation unit calculating the loss function by applying the L similarities to a contrastive loss term describing correlations between the K second mixed feature representations in each of the L sample combinations selected from the M second mixed feature representations; 2. The expression learning device according to claim 1.

4. The sample acquisition unit inputting the N samples into a first network layer included in a deep learning model to extract the N first feature representations; inputting the M first mixed feature representations into a second network layer located after the first network layer included in the deep learning model, and extracting the M second mixed feature representations; 4. The expression learning device according to claim 3.

5. The representation learning device according to claim 4 , further comprising an optimization unit that updates the deep learning model based on the loss function.

6. 2. The expression learning device according to claim 1, wherein the label acquisition unit acquires, as each of the N labels, a teacher label artificially assigned to the sample, a self-supervised label assigned based on a distribution of feature representations of the sample or a type of the sample, a distribution of feature representations of other samples having teacher labels, a distribution of feature representations of samples not having teacher labels, and a semi-supervised label assigned based on a type of sample.

7. the similarity calculation unit calculates L integrated values ​​based on the L weight values ​​and the L similarities; the loss function calculation unit calculates the loss function based on any one of the first to fourth mixed feature representations and the L integrated values; the L weight values ​​have values ​​corresponding to the label types of the K mixed labels, The type is any one of the truth label, the self-supervised label, and the semi-supervised label.

7. The expression learning device according to claim 6.

8. The expression learning device according to claim 1 , wherein the similarity calculation unit calculates the L similarities of the L label combinations selected from the M mixed labels and the N labels.

9. a sample acquiring step of acquiring M first mixed feature representations obtained by mixing feature representations of N samples, M second mixed feature representations extracted from the M first mixed feature representations, M third mixed feature representations which are feature representations extracted from M mixed samples obtained by mixing N samples, or M fourth mixed feature representations which are feature representations extracted from the M third mixed feature representations; a label acquisition step of acquiring M mixed labels by mixing N labels corresponding to the N samples; a similarity calculation step of calculating L similarities of L label combinations selected from the M mixed labels, each of the L label combinations having K mixed labels selected from the M mixed labels, each of the L similarities representing a degree of similarity between the K mixed labels, and the L similarities being formulated based on qualitative and quantitative constraints that the similarities must satisfy; a loss function calculation step of calculating a loss function based on any one of the first to fourth mixed feature representations and the L similarities; Equipped with the similarity calculation step calculates the L similarities formulated based on Jensen-Shannon divergence or cosine similarity. A computer-implemented method for learning representations.

10. On the computer, a sample acquisition function for acquiring M first mixed feature representations obtained by mixing feature representations of N samples, M second mixed feature representations extracted from the M first mixed feature representations, M third mixed feature representations which are feature representations extracted from M mixed samples obtained by mixing N samples, or M fourth mixed feature representations which are feature representations extracted from the M third mixed feature representations; a label acquisition function for acquiring M mixed labels obtained by mixing N labels corresponding to the N samples; a similarity calculation function that calculates L similarities of L label combinations selected from the M mixed labels, each of the L label combinations having K mixed labels selected from the M mixed labels, each of the L similarities representing a degree of similarity between the K mixed labels, and the L similarities being formulated based on qualitative and quantitative constraints that the similarities must satisfy; a loss function calculation function that calculates a loss function based on any one of the first to fourth mixed feature representations and the L similarities; To achieve this, the similarity calculation function calculates the L similarities formulated based on Jensen-Shannon divergence or cosine similarity; Expression learning program.

Images