Text-based final layer learning method and apparatus for debias removal in image classifiers

Abstract

This invention provides a final layer training method and apparatus for a learning model that utilizes text as a substitute for images, reduces annotation costs for data group information where class sets and pseudo-attribute sets are matched, does not require the collection of separate image datasets balanced between data groups, and mitigates bias in image classifiers. [Solution] The final layer learning method of a learning model, performed by the final layer learning device of a learning model, includes the step of inputting training text into the second learning model and training the final layer of the first learning model, based on a projection model that connects the first embedding space of the first learning model and the second embedding space of the second learning model.

Description

[Technical Field]

[0001] The embodiments disclosed herein relate to a method and apparatus for training the final layer of a learning model, and more specifically, to a method and apparatus for training the final layer based on text for debiasing an image classifier.

[0002] This research was conducted as a result of the "Support for Graduate School of Artificial Intelligence (Seoul National University)" project (IITP-II211343) of the Information and Communications Broadcasting Innovation Human Resource Development Program of the Ministry of Science and ICT and the Information and Communications Planning and Evaluation Service (IITP).

[0003] This research was conducted as a result of the project "Research on a New Continuous Learning Algorithm for Handling Data and Environments with Practical Constraints" (NRF-2021R1A2C2007884) of the Individual Basic Research Program of the Ministry of Science and ICT and the Korea Research Foundation (NRF).

[0004] This research was conducted as a result of the project "(2. Details) Development of Causal Relationship Inference Technology for Vision and Language Based on Fusion Shot Learning for Decision Support" (IITP-II220959) of the Human-Centered Artificial Intelligence Core Source Technology Development Project of the Ministry of Science and ICT and the Institute of Information and Communications Planning and Evaluation (IITP). [Background technology]

[0005] The training datasets used to train an image classifier contain spurious attributes that have a high correlation with the labels being predicted, even though they are not directly related. This can lead to problems where the image classifier makes incorrect predictions due to its reliance on such spurious attributes. This is known as the spurious attribute bias problem in image classifiers.

[0006] A representative bias mitigation algorithm for solving the bias problem in image classifiers is a method (Non-Patent Literature 1) that retrains only the final layer of an image classifier that has completed training based on a balanced set of data groups. Here, a data group can be defined based on the set of classes and pseudo-attributes to be predicted. Although the method in Non-Patent Literature 1 is effective in mitigating bias, it has two problems. First, it is costly to label all the data group information for each image. Second, it is difficult to secure images belonging to rare groups in order to construct a balanced dataset among data groups of sufficient size. Due to these problems, Non-Patent Literature 1 is difficult to apply to real-world situations.

[0007] Another algorithm for solving the bias problem in image classifiers is a method (Non-Patent Literature 2) that retrains only the final layer of the image classifier through comparative learning based on an image-text set. However, the method described in Non-Patent Literature 2 has the limitation that it is only applicable to the embedding space obtained through comparative learning based on an image-text set.

[0008] Therefore, a practical method is needed that can effectively mitigate the bias problem in learning models such as image classifiers. [Prior art documents] [Non-patent literature]

[0009] [Non-Patent Document 1] Polina Kirichenko et al., (DFR) Last Layer Re-Training is Sufficient for Robustness to Spurious Correlations, 30 Jun 2023 (https: / / arxiv.org / pdf / 2204.02937) [Non-Patent Document 2] Yuhui Zhang et al., (DrML) Diagnosing and Rectifying Vision Models using Language, 8 Feb 2023 (https: / / arxiv.org / pdf / 2302.04269) [Non-Patent Document 3] Alec Radford et al., (CLIP) Learning Transferable Visual Models From Natural Language Supervision, 26 Feb 2021 (https: / / arxiv.org / pdf / 2103.00020) [Overview of the project] [Problems that the invention aims to solve]

[0010] The embodiments disclosed herein aim to improve the bias of an image classifier by constructing a dataset based on text data and inputting text data into a projection model coupled to an image classifier having a feature extractor and a final layer, thereby retraining only the final layer.

[0011] Other objectives and advantages of the present invention can be understood from the following description and will be more clearly demonstrated by one embodiment. Furthermore, it will be readily apparent that the objectives and advantages of the present invention can be achieved by the means and combinations thereof described in the claims. [Means for solving the problem]

[0012] As a technical means for achieving the technical challenges described above, the final layer learning method of a learning model performed by a final layer learning device of a learning model includes the step of inputting learning text into the second learning model based on a projection model that connects the first embedding space of the first learning model and the second embedding space of the second learning model, thereby training the final layer of the first learning model.

[0013] According to another embodiment, the final layer learning device for a learning model includes a memory for storing the first learning model and the second learning model, and a control unit that inputs learning text to the second learning model based on a projection model that connects the first embedding space of the first learning model and the second embedding space of the second learning model, causing the final layer of the first learning model to be learned.

[0014] In yet another embodiment, the recording medium is a computer-readable recording medium on which a program that performs the final layer learning method of the learning model is recorded.

[0015] In yet another embodiment, the computer program is a computer program that is executed by the final layer learning device of the learning model and is stored on a recording medium to execute the final layer learning method of the learning model. [Effects of the Invention]

[0016] By using any one of the aforementioned problem-solving methods, it is possible to present a method and apparatus for learning the final layer of a learning model that utilizes text as a substitute for images, reduces the cost of annotating data group information where class sets and pseudo-attribute sets are matched, does not require the collection of a separate image dataset balanced between data groups, and mitigates the bias of the image classifier.

[0017] Furthermore, by using any one of the aforementioned problem-solving means, a projection model is generated that connects the first embedding space of the first learning model and the second embedding space of the second learning model, and this projection model is trained. By inputting training text data into the second learning model, which is connected to the projection model connected to the first learning model, it is possible to present a final layer learning method and apparatus for a learning model that improves versatility so that it can be applied to a variety of learning models to mitigate bias, rather than being limited to a specific learning model.

[0018] The effects obtained in the disclosed embodiments are not limited to the effects mentioned above, and other effects not mentioned will be clearly understandable to those skilled in the art in the technical field to which the disclosed embodiments belong from the following description.

[0019] Hereinafter, the accompanying drawings illustrate preferred embodiments disclosed in this specification, and achieve the role of further understanding the technical idea disclosed in this specification together with the specific content for implementing the invention. Therefore, the content disclosed in this specification should not be construed as being limited only to the matters described in the drawings.

Brief Description of the Drawings

[0020] [Figure 1] It is a diagram for explaining the functional configuration of the first learning model for classifying images. [Figure 2] It is a diagram illustrating the operation of training the final layer of the first learning model by an existing algorithm. [Figure 3] It is a block diagram for explaining the functional configuration of the final layer training device of the learning model according to an embodiment. [Figure 4] It is a diagram for explaining the functional configuration of the second learning model for processing image text sets. [Figure 5] It is a diagram illustrating the operation of the final layer training device of the learning model according to an embodiment training the final layer of the first learning model using training text and a projection model. [Figure 6] It is a diagram illustrating the operation of the final layer training device of the learning model according to an embodiment training the projection model. [Figure 7] It is a diagram illustrating the operation of the final layer training device of the learning model according to an embodiment generating a training text set. [Figure 8] It is a diagram illustrating the operation of the final layer training device of the learning model according to an embodiment training the final layer of the first learning model. [Figure 9]This flowchart illustrates a final layer learning method for a learning model according to one embodiment. [Figure 10] This figure illustrates the model performance simulated by the example. [Modes for carrying out the invention]

[0021] The following describes various embodiments in detail based on the attached drawings. The embodiments described below can be modified and implemented in various forms. In order to more clearly describe the characteristics of the embodiments, detailed explanations of matters that are commonly known to those who have ordinary skill in the art to which the embodiments belong are omitted. Furthermore, parts of the drawings that are not relevant to the description of the embodiments are omitted, and similar parts are denoted by similar reference numerals throughout the specification.

[0022] Throughout the specification, when we say that one configuration is "linked" to another, this includes not only cases where they are "directly linked," but also cases where they are "linked with another configuration in between." Furthermore, when we say that one configuration "includes" another, unless otherwise stated, this does not mean that it excludes other configurations, but rather that it may include even more configurations.

[0023] The following describes an embodiment in detail based on the attached drawings.

[0024] Figure 1 is a diagram illustrating the functional configuration of the first learning model for classifying images.

[0025] A "learning model" is a model that can learn to process input data to suit the purpose of a task and predict the outcome, and can include a neural network with multiple layers connected together.

[0026] Supervised learning is a method of training a model using input data that has labels indicating the correct answers for each data point.

[0027] "Label" refers to the class assigned to each piece of data, while "Class" refers to the group to which the data belongs within the dataset. "Ground truth label" refers to the actual label considered correct, while "predicted label" refers to the label inferred by the model.

[0028] "First Learning Model 100" is a learning model that operates as an image classifier, receiving image input and classifying the images.

[0029] An "image classifier" is a neural network that can detect features from an input image and classify the input image based on those features. Various types of deep learning networks can be applied as needed. For example, when processing image or video data, a convolutional neural network (CNN) can be used, and other types of neural networks can also be applied.

[0030] A "neural network" is formed as a network structure in which multiple layers are connected, and each layer contains nodes, which are constituent units. A learning model can have parameters to be learned, and these parameters can include weights and biases.

[0031] "Weight" is a parameter that adjusts the degree to which the input influences the output in a layer node, while "bias" is a parameter that adjusts how easily a layer node is activated (outputs as 1).

[0032] An "activation function" is a function that transforms a linear input, taking into account weights and biases, into a nonlinear output. Layers that output linear values ​​without applying an activation function are also possible.

[0033] The first learning model 100 may include a feature extractor 110 and a final layer 120.

[0034] The feature extractor 110 processes the input data through layers of a learning model and outputs features, which are core information appropriately represented to perform a specific task. Here, features contain information that well represents the subject to suit a specific task and can be represented in a vector form or other form that can be recognized by a computer.

[0035] Embedding is the process of transforming data using a vector space so that a learning model can understand the relationships between the data. An embedding vector is the information represented as a vector through embedding. For example, embedding can be understood as a form of dimensionality reduction or data compression. The embedding space is a distribution space of features that well represent the subject, and can be represented as a vector space that shows the relationships between classes. The embedding space is also called the latent space, and each class is mapped to a specific position in this space.

[0036] The final layer 120 is a classification layer that outputs the predicted classification results as probability values, etc. Classification layers are mainly attached to the end of the learning model. In this specification, the layer that performs the classification function is referred to as the final layer, but other layers can be further attached to the end of the layer that performs the classification function when necessary. In other words, depending on the configuration of the learning model, the position of the classification layer that is to be retrained may not be the last, but for the sake of explanation, it is referred to as the final layer.

[0037] To address the bias problem in the image classification process where an image classifier outputs incorrect predictions due to its reliance on the pseudo-attributes of the training data, it is necessary to define data groups. The data distribution can be defined by data groups G, which are defined by the Cartesian product of the label set Y and the pseudo-attribute set A. That is, G := Y × A. For example, in the Waterbirds dataset, the class labels indicate whether the bird in the image is a land bird or a water bird, and the pseudo-attributes indicate the background of the image. For the Waterbirds dataset, the data group G can be specified as G = {land birds, waterbirds} × {land background, water background}. Waterbirds with a water background and land birds with a land background are predominantly widespread, with a small number of groups being (land birds, water background) and (waterbirds, land background). The image classifier's dependence on pseudo-attributes can generally be evaluated using the Worst Group Accuracy (WGA).

[0038] In the process of classifying waterfowl datasets, image classifiers may infer land birds as waterfowl based on a water background, or vice versa. To mitigate this problem of erroneous correlations, algorithms have emerged that retrain only the final linear classification layer of the image classifier.

[0039] Figure 2 illustrates the process of training the final layer of the first learning model using an existing algorithm.

[0040] Existing algorithms that retrain only the final layer 120 of the feature extractor 110 and final layer 120 included in the first learning model 100 are effective in mitigating bias, but they have many drawbacks. Among the existing algorithms, Non-Patent Literature 1 requires labeling all the data group information for each image during the learning process, which incurs high annotation costs. In particular, it is difficult to collect images belonging to rare groups. Furthermore, Non-Patent Literature 2 has the limitation that it is only applicable to embedding spaces obtained through comparative learning based on image text sets.

[0041] To improve the difficulty of labeling training data and image collection, and to enhance versatility applicable to various training models, the final layer training device of the training model according to this embodiment solves the problems by the following core technical means.

[0042] First, a projection model is trained that connects the joint embedding space and the embedding space of the image classifier through image-text comparative learning. This embodiment overcomes the limitation that the model is only applicable to specific embedded spaces and can be applied to the structure of a general image classifier by generating and training a projection model.

[0043] Secondly, a large-scale language model is used to generate text, and the text is filtered based on cosine similarity and the predicted values ​​of an image classifier. This embodiment allows for the construction of a dataset at a much lower cost than collecting images and labeling each image, by utilizing text data as input data. In particular, the accuracy of the learning model can be improved when retraining the classification layer.

[0044] Third, the classification layer of the image classifier is retrained based on the projection model and text dataset. This embodiment can effectively improve the bias problem of the image classifier by utilizing the projection model and text dataset.

[0045] Figure 3 is a block diagram illustrating the functional configuration of the final layer learning device of a learning model according to one embodiment.

[0046] Referring to Figure 3, the final layer learning device 300 of the learning model according to one embodiment may include an input / output unit 310, a memory 320, a control unit 330, and a communication unit 340.

[0047] The input / output unit 310 may include an input unit for receiving input from a user and an output unit for displaying information such as the result of executing a task or the status of the final layer learning device 300 of the learning model. In other words, the input / output unit 310 is configured to receive data input and output the result of processing it. The final layer learning device 300 of the learning model according to this embodiment can receive final layer learning requests and the like via the input / output unit 310.

[0048] Memory 320 is configured to store files and programs and can be composed of various types of memory. In particular, memory 320 can store data and programs that enable the control unit 330, described later, to perform calculations for layer learning using the algorithm presented below.

[0049] Memory 320 can store a first learning model that processes images and a second learning model that processes image text sets. Memory 320 can store a projection model that connects the first embedding space of the first learning model and the second embedding space of the second learning model. Memory 320 can also store a large language model and the text generated by the large language model.

[0050] The control unit 330 is configured to include at least one processor, such as a CPU or GPU, and can control the overall operation of the final layer learning device 300 of the learning model. That is, the control unit 330 can control other components included in the final layer learning device 300 of the learning model to perform operations for layer learning. The control unit 330 can perform operations to extract representations by the algorithm presented below by executing a program stored in memory 320.

[0051] The communication unit 340 can perform wired or wireless communication with other devices or networks. For example, if a server providing a service for a specific online platform that collects or processes training datasets is embodied as a separate device, the communication unit 340 can receive training datasets by communicating with the server providing the service for the online platform, and can provide a retrained training model based on the received datasets to the server or the user's terminal.

[0052] For this purpose, the communication unit 340 may include a communication module that supports at least one of a variety of wired and wireless communication methods, and the communication module may be embodied in the form of a chipset. The mobile or wireless communication supported by the communication unit 340 may be, for example, N-generation mobile communication protocols, WiFi (Wireless Fidelity), Wi-Fi Direct, Bluetooth®, UWB (Ultra-Wide Band), or NFC (Near Field Communication).

[0053] The control unit 330 trains a specific learning model layer using a projection model that connects multiple learning models. Based on a projection model that connects the first embedding space of the first learning model that processes images and the second embedding space of the second learning model that processes image text sets, the control unit 330 inputs training text into the second learning model and trains the final layer (e.g., the classification layer) of the first learning model.

[0054] Prior to the stage of training the final layer of the first learning model, the control unit 330 can generate a projection model that connects the first embedding space of the first learning model that processes images and the second embedding space of the second learning model that processes the image text set.

[0055] The first learning model, which is linked to the projection model generated by the control unit 330, may include a final layer that classifies the image based on a first embedding space on which features extracted from the image by the feature extractor are mapped.

[0056] The second learning model, which is linked to the projection model generated by the control unit 330, can predict results based on a second embedding space in which features extracted by the image encoder from the images in the image text set and features extracted by the text encoder from the text in the image text set are mapped.

[0057] Figure 4 illustrates the functional configuration of the second learning model that processes the image text set.

[0058] The second learning model 200 may include an image encoder 210 and a text encoder 220. The image encoder 210 can extract features from an input image and represent them as a single vector value, and the text encoder 220 can extract features from input text and represent them as a vector value.

[0059] The second learning model 200 may be a learning model to which self-supervised learning is applied. Here, self-supervised learning is a learning method in which learning proceeds without labeling data and optimizes the distances between features mapped in the embedding space.

[0060] In particular, self-supervised learning utilizing contrastive learning can use positive samples as training data to minimize distances, and negative samples can also be used in the learning process. Positive sample data consists of data that is identical or similar to the target data, while negative sample data consists of data that is not identical or similar to the target data. Identicality or similarity can be measured based on various distances between values. The loss function of contrastive learning can be trained to minimize the distance between positive samples and maximize the distance between negative samples.

[0061] The second learning model 200 can map embedding vectors based on image features and embedding vectors based on text features together into the second embedding space 250 through comparative learning. The second embedding space 250 can be called a joint embedding space.

[0062] Figure 5 illustrates the operation of a final layer learning device for a learning model according to one embodiment, which uses a learning text and a projection model to train the final layer of the first learning model.

[0063] The control unit 330 can input training text into the second learning model 200 based on a projection model 350 that connects the first embedding space of the first learning model 100, which processes images, and the second embedding space of the second learning model 200, which processes the image text set, thereby training the final layer 120 of the first learning model 100.

[0064] In the process of generating the projection model 350, the control unit 330 can train a projection model 350 that projects features extracted by the feature extractor 110 of the first learning model 100 onto the embedding vectors of the first embedding space 150, which are mapped to the embedding vectors of the second embedding space 250, which are mapped to the embedding vectors of the second embedding space 250, which are mapped to the embedding vectors of the second embedding space 200, which are mapped to the embedding vectors of the first embedding space 100, using the data used to train the first learning model 100.

[0065] The control unit 330 can train the projection model 350 by considering the weight values ​​of the projection model 350 and the orthogonality of the modality gap in the second embedding space 250 during the training process of the projection model 350. Here, the modality gap represents the alignment state between the embedding vectors corresponding to the images in the image text set and the embedding vectors corresponding to the text in the image text set in the second embedding space 250.

[0066] The control unit 330 can train the projection model 350 in such a way that the modality gap satisfies the condition that the transpose matrix of the weights of the projection model 350 lies in the zero space (nullspace).

[0067] The control unit 330 can select training texts based on the first embedding space 150 of the first learning model 100 and the second embedding space 250 of the second learning model 200.

[0068] The control unit 330 can use a language model to generate a first training text relating to class labels and a second training text relating to spurious attributes that cause spurious correlations during the process of selecting training texts.

[0069] The control unit 330 can perform a primary filter on the first learning text based on the second embedding space 250. The control unit 330 can also perform a primary filter on the second learning text based on the second embedding space 250.

[0070] In the process of selecting training texts, the control unit 330 can perform a secondary filter on the first training texts that have been primary filtered based on the final layer 120 of the first embedding space 150 projected from the second embedding space 250 via the projection model 350.

[0071] During the process of training the final layer 120 of the first learning model 100, the control unit 330 projects the average of multiple embedding vectors mapped to the second embedding space 250 using the quadratic filtered first learning text and the linearly filtered second learning text onto the embedding vectors in the first embedding space 150 via the projection model 350, and then trains the final layer using the projected embedding vectors.

[0072] The main operations of this embodiment will be described in more detail below with reference to Figures 6 to 8.

[0073] Figure 6 illustrates the operation of the final layer learning device of a learning model according to one embodiment, in which it trains a projection model.

[0074] Referring to Figure 6, the final layer learning device 300 of the learning model in this embodiment learns a projection model 350 that connects the second embedding space 250 obtained by image-text comparison learning and the first embedding space 150 of the image classifier.

[0075] The final layer learning device 300 of the learning model selects the second learning model 200, which has completed comparative learning based on the image-text set. To retrain the image classifier based on text, a model that has undergone comparative learning using a large image-text pair dataset, such as the CLIP model described in Non-Patent Literature 3, can be used. Since the embedding space learned by comparative learning cannot be directly utilized for final layer retraining of the image classifier, a process is required to project the text embeddings present in the second embedding space 250 obtained by image-text comparative learning onto the first embedding space 150 of the image classifier.

[0076] The first learning model 100's feature extractor 110 and the final linear classification layer 120 are f θ and h Φ The parameters are displayed as θ and Φ.

[0077] JPEG2026068659000002.jpg53170

[0078] In the joint embedding space, there is a modality gap that represents a constant gap between image embedding and text embedding. An instance-specific modality gap can be considered.

[0079] If (I, T) represents an image-text set or an image-text pair, then the modality gap g in the joint embedding space of the second learning model is expressed as follows:

[0080] JPEG2026068659000003.jpg834

[0081] This relates to the embedding space of the CLIP model, where g can be defined in all joint embedding spaces where both modalities are well aligned.

[0082] JPEG2026068659000004.jpg42155

[0083] JPEG2026068659000005.jpg40155

[0084] JPEG2026068659000006.jpg30170

[0085]

number

[0086]

number

[0087]

number

[0088] JPEG2026068659000010.jpg27170

[0089] JPEG2026068659000011.jpg31156

[0090] The final layer learning device 300 of the learning model utilizes the data used to train the image classifier to learn a projection model that takes into account the characteristics of the embedding space of the control learning. This method effectively reflects the data distribution considered by the image classifier, does not require a learning process based on gradient descent, and can determine the optimal parameters in a closed form.

[0091] JPEG2026068659000012.jpg43155

[0092] JPEG2026068659000013.jpg29170

[0093] The linear relationship between X and Y is expressed as shown in mathematical equation 4.

[0094]

number

[0095] Then, constraint condition W ┬ The ridge regression estimates for (W, b) with g=0 are expressed as shown in mathematical formulas 5 and 6.

[0096]

number

[0097]

number

[0098] JPEG2026068659000017.jpg23158

[0099] JPEG2026068659000018.jpg19170

[0100] JPEG2026068659000019.jpg22170

[0101] Figure 7 illustrates the operation of the final layer learning device of a learning model according to one embodiment, which generates a training text set.

[0102] The final layer learning device 300 of the learning model utilizes the language model 400 to generate the necessary words in order to retrain the final layer of the image classifier.

[0103] The final layer learning device 300 of the learning model generates synonyms for class and pseudo-attribute categories using a Large Language Model (LLM). That is, it generates synonyms for the names of categories belonging to a set (Y, A) via the Large Language Model.

[0104] Given class labels and pseudo-attribute category names for Y and A, the system generates a variety of words corresponding to those names by generating synonyms.

[0105] JPEG2026068659000020.jpg20170

[0106] The generated text and the embedded text projected onto it are out-of-distribution (OOD) examples for the second learning model and feature extractor, which can degrade the performance of layer learning. The generated words may contain inaccurate words due to the language model's hallucination problem, so these inaccurate words need to be filtered based on two metrics.

[0107] The primary metric is the cosine similarity of the text embedding in the contrast learning latent space. A word is only adopted if the similarity between the generated word and the word in that category is higher than the similarity with other categories. This allows for accurate word selection.

[0108] The second metric involves projecting words generated using a projection model into the image classifier's embedding space, obtaining predicted values ​​through the trained final classification layer, and accepting a word only if the predicted value corresponds to a category that matches the generated word. This allows the image classifier to select words that it understands well.

[0109] Therefore, it is necessary to implement Embedding Alignment (VEA) for the generated words to retain only those words that are compatible with the embedding space of the second learning model and feature extractor. The final layer learning device 300 of the learning model checks whether the words generated via the large-scale language model are well aligned with the embedding spaces of the first and second learning models. The final layer learning device 300 of the learning model performs first-order filtering on the generated words based on category names and cosine similarity in the embedding space. Then, it performs second-order filtering based on the predicted values ​​of the first learning model, the image classifier.

[0110] Specifically, for Y={y1, y2}, the word t generated as a synonym for y1 is adopted only if it satisfies the condition cos(y1, t)>cos(y2, t), and the same process is then applied to A.

[0111] For words generated as synonyms for category names in set Y, additional filtering is performed based on the predictions of the trained image classifier.

[0112] Specifically, for a word t generated as a synonym for y1, it is projected into the embedding space of the image classifier, and then, if the predicted value obtained by feeding it into the existing trained final classification layer is y1, it is adopted.

[0113] JPEG2026068659000021.jpg28170

[0114] JPEG2026068659000022.jpg20155

[0115]

number

[0116] For example, P1 is a prompt template where P1(t) = "Aphotoofa{t}".

[0117] T a The embedding sorting validation of the generated words is performed similarly.

[0118] In the case of embedding alignment verification for the feature extractor, if the embedding alignment verification for the second learning model is completed, the filtered words are semantically well-aligned with the embedding space of the second learning model.

[0119] However, for the filtered words of T y the projected embeddings of the second learning model may be out-of-distribution (OOD) examples within the embedding space of the feature extractor f θ and thus may not still be well-aligned with the embedding space of f θ

[0120] Such alignment errors may cause additional confusion when constructing a new classification boundary by layer learning. To solve such problems, implement embedding alignment verification based on logits for the feature extractor fθ to additionally remove words that were not aligned from T y Logits represent the predicted values that were not normalized to the probability range.

[0121] JPEG2026068659000024.jpg14157

[0122]

Number

[0123] JPEG2026068659000026.jpg15153

[0124] The final layer learning device 300 of the learning model constructs a word number dataset based on the words filtered after the embedding alignment verification. All possible combinations can be considered for each data group. That is, for the data group (y, a), it is T y ×T a where T y and T a only contain the words that have been verified. ​

[0125] Figure 8 illustrates the operation of a final layer learning device for a learning model according to one embodiment, in which the device trains the final layer of the first learning model.

[0126] The final layer learning device 300 of the learning model embeds the first learning text for each class and the second learning text for pseudo-attributes using the text encoder 220 of the second learning model.

[0127] The final layer learning device 300 of the learning model averages the text embeddings for each class and pseudo-attribute in the second embedding space 250, and then processes them through the projection model 350 f θ Project onto the first embedding space 150. The projected embedding is h Φ It is supplied and used for retraining. The final layer learning device 300 of the learning model retrains the final classification layer of the image classifier with the projection model 350 and the text dataset. The final layer learning device 300 of the learning model performs text-based layer learning using projected embeddings of text prompts obtained from validated words.

[0128] The final layer learning device 300 of the learning model constructs a text dataset based on the filtered words. Here, the data belonging to each group consists of the average text embeddings of synonyms of y and synonyms of a. Specifically, the data belonging to the (y1, a1) group is the average embedding of the text embeddings of synonyms of y1 and synonyms of a1.

[0129] After constructing a text-based dataset using this method, each text embedding is projected onto the first embedding space 150 of the image classifier based on the projection model 350. Then, the final layer is retrained. When retraining the final classification layer, a group-equipped text dataset can be sampled at each epoch to perform the training.

[0130] JPEG2026068659000027.jpg88155

[0131] The final classification layer can be retrained using mini-batch optimization. Furthermore, to reduce training costs, model ensembles can be avoided, and a group equilibrium training set can be sampled at each epoch to maximize the use of available data. Basically, early stopping is adopted based on WGA validation.

[0132] Figure 9 is a flowchart illustrating the final layer learning method of a learning model according to one embodiment.

[0133] The final layer learning method of the learning model according to the embodiment shown in Figure 9 includes a stage that is processed chronologically by the final layer learning device of the learning model shown in Figures 1 to 6. Therefore, even if we omit some details below, the information described above regarding the final layer learning device of the learning model shown in Figures 3 to 8 can also be applied to the final layer learning method of the learning model according to the embodiment shown in Figure 9.

[0134] Referring to Figure 9, at step S930, the final layer learning device of the learning model inputs training text into the second learning model based on a projection model that connects the first embedding space of the first learning model and the second embedding space of the second learning model, thereby training the final layer of the first learning model.

[0135] Prior to the stage of training the final layer of the first learning model (S930), the final layer learning device of the learning model can perform the step (S910) of generating a projection model that connects the first embedding space of the first learning model that processes images and the second embedding space of the second learning model that processes the image text set.

[0136] Prior to the stage of training the final layer of the first learning model (S930), the final layer learning device of the learning model can perform the step of selecting training texts based on the first embedding space of the first learning model and the second embedding space of the second learning model (S920).

[0137] The first learning model, applied to the final layer learning method of the learning model, may include a final layer that classifies images based on a first embedding space where features extracted from images by a feature extractor are mapped.

[0138] The second learning model, applied to the final layer learning method of the learning model, can predict results based on a second embedding space where features extracted by the image encoder from the images in the image-text set and features extracted by the text encoder from the text in the image-text set are mapped.

[0139] The step of generating a projection model (S910) may include training a projection model that uses the data used to train the first learning model to project the features extracted by the feature extractor of the first learning model onto the embedding vectors of the first embedding space, which are mapped to the embedding vectors of the second embedding space, which are mapped to the embedding vectors of the second embedding space, which are mapped to the embedding vectors of the second embedding space, which are mapped to the embedding vectors of the first learning model, which are mapped to the embedding vectors of the first embedding space, which are mapped to the embedding vectors of the first learning model, using the data used to train the first learning model.

[0140] The stage of training the projection model may include a stage of training the projection model considering the weights of the projection model and the orthogonality of the modality gap in the second embedding space. Here, the modality gap can represent the alignment state between the embedding vectors corresponding to the images in the image text set and the embedding vectors corresponding to the text in the image text set in the second embedding space.

[0141] The stage of training the projection model may include training the projection model such that the modality gap exists in the zero space of the transpose matrix of the projection model's weights.

[0142] The step of selecting training texts (S920) may include a step of generating a first training text about class labels and a second training text about spurious attributes that cause spurious correlations using a language model.

[0143] The step of selecting learning texts (S920) may include a step of primary filtering the first learning texts based on the second embedding space.

[0144] The step of selecting training texts (S920) may include a step of secondary filtering of the first training texts, which have been first filtered based on the final layer of the first embedding space projected from the second embedding space via the projection model.

[0145] The step of training the final layer of the first learning model (S930) may include a step of projecting the average of multiple embedding vectors mapped to the second embedding space using the quadratic filtered first learning text and the linearly filtered second learning text onto the embedding vectors in the first embedding space via a projection model, and then training the final layer using the projected embedding vectors.

[0146] Figure 10 illustrates the model performance simulated by the embodiment.

[0147] This example uses a projection model to input text data and retrain only the final layer of the image classifier. This example uses TLDR(T ext-based L ast-layer retraining for D Image classification R It can be said that...

[0148] The comparative examples are ERM (Empirical Risk Minimization), DFR (Deep Feature Reweighting), AFR (Automatic Feature Reweighting), and SELF (Selective Last-layer Finetuning). ERM is a method for minimizing the risk of limited training data sampled from the population. DFR is an algorithm described in Non-Patent Literature 1, which takes a balanced data group as input and retrains only the final layer of the image classifier. AFR integrates final layer retraining and inference of data group information, and retrains the final layer of the ERM model with a weighted loss function that assigns higher importance to instances with poor predictions. SELF uses checkpoints to infer data belonging to a small population on a holdout dataset consisting of half of the validation set, and further trains the overall model.

[0149] The TLDR according to this embodiment can be confirmed to have superior WGA evaluation performance for small groups compared to the comparative examples ERM, DFR, AFR, and SELF.

[0150] The poor performance of DFR and SELF is due to the limited data available for each group in a balanced data group set. Smaller group numbers affect retraining and WGA validation evaluation. This is because DFR and SELF randomly split the validation set in half for WGA evaluation, leading to suboptimal hyperparameter search. TLDR, on the other hand, avoids this problem because it neither splits the validation set nor uses it for final layer retraining.

[0151] The TLDR method used in this embodiment is efficient and practical because it can be applied to a pre-trained model without additional training of the overall model.

[0152] The TLDR in this embodiment demonstrates robust performance in diverse, small proportions, making it usable even in situations where there is a shortage of images belonging to small groups.

[0153] JPEG2026068659000028.jpg43170

[0154] JPEG2026068659000029.jpg30170

[0155] Regarding modality gaps, varying the number of pairs used to estimate them reveals that without gap information, the WGA and average accuracy decrease significantly. However, even with only 10 image-text pairs, it is sufficient to estimate the modality gap and shows similar performance to using a larger number of pairs.

[0156] In relation to embedding alignment validation, DrML according to Non-Patent Literature 2 simply uses words already provided from metadata. Therefore, it has the problem of not being able to be perfectly applied to general datasets where the metadata does not contain appropriate text. Also, using only words from metadata may be insufficient due to a limited quantity or alignment errors with the embedding space, which can reduce the debiasing performance. Embedding alignment validation also has a positive impact on the text data of DrML, improving both the WGA test and the mean accuracy. Furthermore, it can be confirmed that the TLDR text generation method using embedding alignment validation is considerably superior to DrML. In other words, the embedding alignment validation process is important because relying only on all the generated words is not sufficiently effective.

[0157] The word generation and embedding alignment validation process according to this embodiment enables text-based layer retraining in situations where the dataset metadata is not provided with appropriate words. The gain of embedding alignment validation is greater when early interruption is not applied, which may be practical in situations where the proportion of the minority group is too small to justify appropriate early interruption.

[0158] The TLDR provided in this embodiment can be utilized in a variety of industrial fields.

[0159] First, it can be applied to medical image analysis and diagnostic support systems in the field of medical AI. For example, by removing bias in systems that analyze medical images such as X-rays, MRIs, and CT scans to diagnose diseases, the accuracy of the diagnosis can be improved. Furthermore, by minimizing bias in medical data, it can support fairer and more reliable diagnoses.

[0160] Next, it can play a significant role in the field of autonomous driving. It helps in accurately classifying and recognizing road conditions, pedestrians, and other vehicles perceived by the cameras of autonomous vehicles. It contributes to improving the safety of autonomous vehicles by preventing misjudgments due to biased data. Since bias can lead to accidents in autonomous driving technology, eliminating bias is essential.

[0161] Furthermore, it can play a crucial role in security and surveillance systems. Facial recognition systems in airports, train stations, and large event venues can enhance fairness by eliminating biases such as race and gender. It can also improve security reliability by providing unbiased and accurate surveillance data for crime prevention and investigation.

[0162] As used in the above embodiments, the term "~part" refers to software or hardware components such as FPGAs (field programmable gate arrays) or ASICs, and "~part" performs a certain role. However, "~part" is not limited to software or hardware. "~part" can be configured to reside in an addressable storage medium, or to regenerate one or more processors. Thus, as an example, "~part" includes components such as software components, object-oriented software components, class components, and task components, as well as processes, functions, attributes, procedures, subroutines, segments of programmatic code, drivers, firmware, microcode, circuits, data, databases, data structures, tables, arrays, and variables.

[0163] Components and the functions provided within a "~part" can be combined with a smaller number of components and "~parts" or separated from additional components and "~parts".

[0164] Furthermore, the components and "~parts" can also be embodied to regenerate one or more CPUs within the device or security multimedia card.

[0165] On the other hand, the final layer learning method of the learning model according to one embodiment described herein can also be embodied in the form of a computer-readable medium that stores computer-executable instructions and data. Here, the instructions and data can be stored in the form of program code, which, when executed by a processor, can generate a predetermined program module and perform a predetermined operation. The computer-readable medium may be any available medium accessible by a computer, and may include volatile and non-volatile media, and isolated and non-isolated media. The computer-readable medium may also be a computer recording medium. The computer recording medium may include any volatile and non-volatile, isolated and non-isolated media embodied by any method or technique for storing information such as computer-readable instructions, data structures, program modules, or other data. For example, the computer recording medium may be a magnetic storage medium such as an HDD and an SSD, an optical recording medium such as a CD, DVD, and a Blu-ray disc, or memory contained in a server accessible via a network.

[0166] Furthermore, the final layer learning method of the learning model according to one embodiment described herein can also be embodied in a computer program (or computer program product) containing computer-executable instructions. A computer program contains programmable machine instructions processed by a processor and can be embodied in a high-level programming language, object-oriented programming language, assembly language, or machine language. The computer program can also be recorded on a type of computer-readable recording medium (e.g., memory, hard disk, magnetic / optical media, or SSD (Solid-State Drive)).

[0167] Therefore, the final layer learning method of the learning model according to one embodiment described herein can be realized by executing the above-described computer program on a computing device. The computing device may include a processor, memory, storage device, a high-speed interface connected to the memory and high-speed expansion port, and at least a portion of a low-speed bus and a low-speed interface connected to the storage device. Each of these components is connected to one another using various buses and can be mounted on a common motherboard or in other suitable manner.

[0168] Here, the processor can process instructions within the computing device. Such instructions may include, for example, instructions stored in memory or storage devices for displaying graphic information to provide a GUI (Graphical User Interface) on an external input and output device, such as a display connected to a high-speed interface. In other embodiments, a number of processors and / or a number of buses can be appropriately used together with a number of memories and memory forms. The processor can also be embodied in a chipset consisting of chips containing a number of independent analog and / or digital processors.

[0169] Memory also stores information within a computing device. For example, memory can consist of volatile memory units or a collection thereof. Alternatively, memory can consist of non-volatile memory units or a collection thereof. Furthermore, memory may be other forms of computer-readable media, such as magnetic or optical discs.

[0170] Furthermore, the storage device can provide computing devices with a large-capacity storage space. The storage device may be a computer-readable medium or a configuration including such a medium, and may include, for example, a device in a SAN (Storage Area Network) or other configurations, and may be a floppy disk drive, a hard disk drive, an optical disk drive, or a tape drive, flash memory, or other similar semiconductor memory device or device array.

[0171] The embodiments described above are illustrative, and those with ordinary skill in the art to which the embodiments belong will understand that they can be easily modified into other specific forms without altering the technical idea or essential features of the embodiments. Therefore, it should be understood that the embodiments described above are illustrative in all respects and not limiting. For example, each component described as a single type can be implemented in a distributed manner, and similarly, components described as distributed can be implemented in a combined manner.

[0172] The scope of protection sought by this specification shall be determined by the claims set forth below rather than by the detailed description above, and all forms of modification or variation derived from the meaning and scope of the claims and their equivalent concepts shall be interpreted as being included within the scope of the invention. [Explanation of Symbols]

[0173] 300 Learning Model Final Layer Learning Device 310 Input / output section 320 memory 330 Control Unit 340 Communications Department

Claims

1. A method for learning the final layer of a learning model, which is performed by a final layer learning device of a learning model, A method for learning the final layer of a learning model, comprising the step of inputting training text into the second learning model based on a projection model that connects the first embedding space of the first learning model and the second embedding space of the second learning model, thereby training the final layer of the first learning model.

2. The final layer learning method of the aforementioned learning model is: Prior to the stage of training the final layer of the first learning model, The steps include generating a projection model that connects the first embedding space of the first learning model that processes images and the second embedding space of the second learning model that processes an image text set, The final layer learning method for a learning model according to claim 1, further comprising the step of selecting the learning text based on the first embedding space of the first learning model and the second embedding space of the second learning model.

3. The first learning model includes a final layer that classifies the image based on the first embedding space on which features extracted from the image by a feature extractor are mapped. The final layer learning method for a learning model according to claim 2, wherein the second learning model predicts results based on the second embedding space, which maps features extracted by an image encoder from the images of the image text set and features extracted by a text encoder from the text of the image text set.

4. The step of generating the aforementioned projection model is: The final layer learning method for a learning model according to claim 3, further comprising the step of training a projection model that projects features extracted by the feature extractor of the first learning model onto an embedding vector in the first embedding space, which is mapped to features extracted by the image encoder of the second learning model, using data used to train the first learning model.

5. The step of training the aforementioned projection model is: This includes a step of training the projection model, taking into account the weights of the projection model and the orthogonality of the modality gap in the second embedding space. The final layer learning method of a learning model according to claim 4, wherein the modality gap indicates the alignment state between the embedding vector corresponding to the image of the image text set and the embedding vector corresponding to the text of the image text set in the second embedding space.

6. The step of training the aforementioned projection model is: The final layer learning method for a learning model according to claim 5, comprising the step of training the projection model such that the modality gap satisfies the condition that it exists in the zero space of the transpose matrix of the weights of the projection model.

7. The step of selecting the aforementioned learning texts is: The process involves generating a first training text for class labels and a second training text for pseudo-attributes that cause spurious correlation using a language model, A step of performing a primary filter on the first learning text based on the second embedding space, A method for learning the final layer of a learning model according to claim 2, comprising the step of performing a primary filter on the second learning text based on the second embedding space.

8. The step of selecting the aforementioned learning texts is: The final layer learning method for a learning model according to claim 7, further comprising the step of second-filtering the first learning text that has been first-filtered based on the final layer of the first embedding space projected from the second embedding space via the projection model.

9. The stage of training the final layer of the first learning model is: A method for learning the final layer of a learning model according to claim 8, comprising the steps of: projecting the average of a plurality of embedding vectors mapped to the second embedding space using the second-order filtered first learning text and the first-order filtered second learning text onto the embedding vectors of the first embedding space via the projection model; and learning the final layer using the projected embedding vectors.

10. Memory for storing the first and second learning models, A final layer learning device for a learning model, comprising: a control unit that inputs learning text into the second learning model based on a projection model connecting the first embedding space of the first learning model and the second embedding space of the second learning model, causing the second learning model to learn the final layer of the first learning model.

11. A computer-readable recording medium on which a program for performing the method described in claim 1 is recorded.

12. A computer program executed by a final layer learning device of a learning model and stored on a recording medium to perform the method according to claim 1.

Images